Copied files take up more space than originals
in [Solaris]
|
Prev: Value too large for defined data type...
Next: Problem when multiple SCP connections in Solaris 10 - ssh_exchange_identification: Connection closed by remote host
From: Dave on 22 Dec 2009 16:24 Richard B. Gilbert wrote: > Dave wrote: >> I'm trying to copy a directory from one computer to another using scp. >> >> The 'source' machine is a Sun Netra T1 with Solaris 10 (03/2005). >> >> The 'destination' machine is a 'zone' configured on a Sun Blade 1000 >> running Solaris 10 update 7 (05/2009), using the following command: >> >> drkirkby(a)kestrel:~$ scp -pr sage-4.2 sage(a)t2nb: >> >> (The time stamp on the 'scp' command is September of this year, so the >> version of Sun's SSH is not that old - it must have been recently >> patched.) >> >> >> Executing: >> >> drkirkby(a)kestrel:~$ du -ks sage-4.2 >> 6923547 sage-4.2 >> >> on the 'source' machine, I see I have about 7 GB of data. >> >> Running the same command on the destination machine: >> >> -bash-3.00$ du -ks sage-4.2 >> 10692359 sage-4.2 >> >> I see I have around 10 GB, and the copy has not completed yet! >> >> I'm puzzled why this should be. There are no core files, and as far as >> I am aware, no sparse files. There are however lots of symbolic links, >> and one hard link. >> >> Any thoughts as to why the disk spaced used on the destination is much >> greater than on the source? I thought the '-p' option on scp would >> have copied everything unchanged. Clearly something is amiss. >> >> I've got 41 GB disk space free, so I hope I'm not going to fill up the >> whole disk copying over 7 GB! >> > > Have you tried comparing file sizes between the originals and the > copies? If they are not the same, what does diff tell you? The problem is the symbolic links are not being copied properly: On the source system, we see there is a file libsqlite3.so.0.8.6, and there are two symbolic links to it. lrwxrwxrwx 1 drkirkby other 19 Nov 21 12:22 libsqlite3.so -> libsqlite3.so.0.8.6 lrwxrwxrwx 1 drkirkby other 19 Nov 21 12:22 libsqlite3.so.0 -> libsqlite3.so.0.8.6 -rwxr-xr-x 1 drkirkby other 1397028 Nov 21 12:22 libsqlite3.so.0.8.6 On the destination system, I see: -rwxr-xr-x 1 sage other 1397028 Nov 21 12:22 libsqlite3.so -rwxr-xr-x 1 sage other 1397028 Nov 21 12:22 libsqlite3.so.0 -rwxr-xr-x 1 sage other 1397028 Nov 21 12:22 libsqlite3.so.0.8.6 So for all practical purposes, this one file has taken up 3x as much space on the destination as it did on the source. Is there any way to get scp to not do this? I've had the same problem using the Solaris 'cp' command. Copying a similar lot of files and making a tar file is creating one about 50% larger than using the GNU 'cp' command with the '-a' (archive) option. The '-a' option is not POSIX. From what I gather, POSIX is not too explicit about how 'cp' handles links - though it might be only be hard links. Either way, copying files in Solaris using the Solaris tools, seems to take up a lot more disk space if there are links. Dave -- I respectfully request that this message is not archived by companies as unscrupulous as 'Experts Exchange' . In case you are unaware, 'Experts Exchange' take questions posted on the web and try to find idiots stupid enough to pay for the answers, which were posted freely by others. They are leeches.
From: Richard B. Gilbert on 22 Dec 2009 16:35 Dave wrote: > Richard B. Gilbert wrote: >> Dave wrote: >>> I'm trying to copy a directory from one computer to another using scp. >>> >>> The 'source' machine is a Sun Netra T1 with Solaris 10 (03/2005). >>> >>> The 'destination' machine is a 'zone' configured on a Sun Blade 1000 >>> running Solaris 10 update 7 (05/2009), using the following command: >>> >>> drkirkby(a)kestrel:~$ scp -pr sage-4.2 sage(a)t2nb: >>> >>> (The time stamp on the 'scp' command is September of this year, so >>> the version of Sun's SSH is not that old - it must have been recently >>> patched.) >>> >>> >>> Executing: >>> >>> drkirkby(a)kestrel:~$ du -ks sage-4.2 >>> 6923547 sage-4.2 >>> >>> on the 'source' machine, I see I have about 7 GB of data. >>> >>> Running the same command on the destination machine: >>> >>> -bash-3.00$ du -ks sage-4.2 >>> 10692359 sage-4.2 >>> >>> I see I have around 10 GB, and the copy has not completed yet! >>> >>> I'm puzzled why this should be. There are no core files, and as far >>> as I am aware, no sparse files. There are however lots of symbolic >>> links, and one hard link. >>> >>> Any thoughts as to why the disk spaced used on the destination is >>> much greater than on the source? I thought the '-p' option on scp >>> would have copied everything unchanged. Clearly something is amiss. >>> >>> I've got 41 GB disk space free, so I hope I'm not going to fill up >>> the whole disk copying over 7 GB! >>> >> >> Have you tried comparing file sizes between the originals and the >> copies? If they are not the same, what does diff tell you? > > The problem is the symbolic links are not being copied properly: > > On the source system, we see there is a file libsqlite3.so.0.8.6, and > there are two symbolic links to it. > > lrwxrwxrwx 1 drkirkby other 19 Nov 21 12:22 libsqlite3.so -> > libsqlite3.so.0.8.6 > lrwxrwxrwx 1 drkirkby other 19 Nov 21 12:22 libsqlite3.so.0 -> > libsqlite3.so.0.8.6 > -rwxr-xr-x 1 drkirkby other 1397028 Nov 21 12:22 libsqlite3.so.0.8.6 > > On the destination system, I see: > > -rwxr-xr-x 1 sage other 1397028 Nov 21 12:22 libsqlite3.so > -rwxr-xr-x 1 sage other 1397028 Nov 21 12:22 libsqlite3.so.0 > -rwxr-xr-x 1 sage other 1397028 Nov 21 12:22 libsqlite3.so.0.8.6 > > So for all practical purposes, this one file has taken up 3x as much > space on the destination as it did on the source. > > Is there any way to get scp to not do this? > > I've had the same problem using the Solaris 'cp' command. Copying a > similar lot of files and making a tar file is creating one about 50% > larger than using the GNU 'cp' command with the '-a' (archive) option. > The '-a' option is not POSIX. From what I gather, POSIX is not too > explicit about how 'cp' handles links - though it might be only be hard > links. Either way, copying files in Solaris using the Solaris tools, > seems to take up a lot more disk space if there are links. > > Dave > I think it did EXACTLY what you told it to do. It looks as if it copied each file pointed to by the links you gave it. N links gives N copies. Just what you asked for!
From: Richard B. Gilbert on 22 Dec 2009 15:02 Dave wrote: > I'm trying to copy a directory from one computer to another using scp. > > The 'source' machine is a Sun Netra T1 with Solaris 10 (03/2005). > > The 'destination' machine is a 'zone' configured on a Sun Blade 1000 > running Solaris 10 update 7 (05/2009), using the following command: > > drkirkby(a)kestrel:~$ scp -pr sage-4.2 sage(a)t2nb: > > (The time stamp on the 'scp' command is September of this year, so the > version of Sun's SSH is not that old - it must have been recently patched.) > > > Executing: > > drkirkby(a)kestrel:~$ du -ks sage-4.2 > 6923547 sage-4.2 > > on the 'source' machine, I see I have about 7 GB of data. > > Running the same command on the destination machine: > > -bash-3.00$ du -ks sage-4.2 > 10692359 sage-4.2 > > I see I have around 10 GB, and the copy has not completed yet! > > I'm puzzled why this should be. There are no core files, and as far as I > am aware, no sparse files. There are however lots of symbolic links, and > one hard link. > > Any thoughts as to why the disk spaced used on the destination is much > greater than on the source? I thought the '-p' option on scp would have > copied everything unchanged. Clearly something is amiss. > > I've got 41 GB disk space free, so I hope I'm not going to fill up the > whole disk copying over 7 GB! > Have you tried comparing file sizes between the originals and the copies? If they are not the same, what does diff tell you?
From: unruh on 22 Dec 2009 16:47 ["Followup-To:" header set to comp.security.ssh.] On 2009-12-22, Dave <foo(a)coo.com> wrote: > Richard B. Gilbert wrote: >> Dave wrote: >> >> Have you tried comparing file sizes between the originals and the >> copies? If they are not the same, what does diff tell you? > > The problem is the symbolic links are not being copied properly: > > On the source system, we see there is a file libsqlite3.so.0.8.6, and there are > two symbolic links to it. > > lrwxrwxrwx 1 drkirkby other 19 Nov 21 12:22 libsqlite3.so -> > libsqlite3.so.0.8.6 > lrwxrwxrwx 1 drkirkby other 19 Nov 21 12:22 libsqlite3.so.0 -> > libsqlite3.so.0.8.6 > -rwxr-xr-x 1 drkirkby other 1397028 Nov 21 12:22 libsqlite3.so.0.8.6 > > On the destination system, I see: > > -rwxr-xr-x 1 sage other 1397028 Nov 21 12:22 libsqlite3.so > -rwxr-xr-x 1 sage other 1397028 Nov 21 12:22 libsqlite3.so.0 > -rwxr-xr-x 1 sage other 1397028 Nov 21 12:22 libsqlite3.so.0.8.6 > > So for all practical purposes, this one file has taken up 3x as much space on > the destination as it did on the source. > > Is there any way to get scp to not do this? Not as far as I know. Use something else--eg, rsync ( which uses ssh to actually transfer the files) > > I've had the same problem using the Solaris 'cp' command. Copying a similar lot > of files and making a tar file is creating one about 50% larger than using the > GNU 'cp' command with the '-a' (archive) option. The '-a' option is not POSIX. > From what I gather, POSIX is not too explicit about how 'cp' handles links - > though it might be only be hard links. Either way, copying files in Solaris > using the Solaris tools, seems to take up a lot more disk space if there are links. rsync will also handle hard links properly. > > Dave >
From: ChrisS on 23 Dec 2009 10:21
I would definitely use the tar solution or the rsync solution. Since it's a relatively small amount of data 'tar' may be easiest. I live for rsync these days too. I'm not sure rsync comes with Solaris 10 by default. I've added it via Blastwave or OpenCSW, but SunFreeware tools is good for that too. Good luck. Chris |