Enable rsh or other way for fast file transfer
in [Solaris]
|
Prev: MPxIO and stmsboot
Next: Scrambled display
From: Dave on 6 Nov 2009 06:35 I have two Sun workstations which I want to copy a lot of data between - one Sun is quite old, the other is new. 1) Sun Blade 2000, 2 x 1200 MHz SPARC 2) Sun Ultra 27. Quad core 3.33 GHz Xeon. I've stuck a Gigabit Ethernet card in the Blade 2000, so these have a gigabit connection between them. Transferring a 1000 MB file via ssh is taking 49 seconds (20 MB/s), compared to the 11 seconds (91 MB/s) I can achieve with ftp. It's clear the Blade is limiting the transfer speed, due to the speed of its CPU. What are my options for a faster transfer between these two machines? I want to do a recursive copy, so normal ftp is not convenient. I can't seem to work out how to enable rcp to work. Can anyone tell me how? Perhaps I can find an ftp client which can work recursively. Security is not an issue - these is between two machines on a home LAN. Any other suggestions? Dave
From: Chris Ridd on 6 Nov 2009 06:47 On 2009-11-06 11:35:32 +0000, Dave <foo(a)coo.com> said: > I have two Sun workstations which I want to copy a lot of data between > - one Sun is quite old, the other is new. > > 1) Sun Blade 2000, 2 x 1200 MHz SPARC > 2) Sun Ultra 27. Quad core 3.33 GHz Xeon. > > I've stuck a Gigabit Ethernet card in the Blade 2000, so these have a > gigabit connection between them. > > Transferring a 1000 MB file via ssh is taking 49 seconds (20 MB/s), > compared to the 11 seconds (91 MB/s) I can achieve with ftp. It's clear > the Blade is limiting the transfer speed, due to the speed of its CPU. > > What are my options for a faster transfer between these two machines? I > want to do a recursive copy, so normal ftp is not convenient. > > I can't seem to work out how to enable rcp to work. Can anyone tell me > how? Perhaps I can find an ftp client which can work recursively. Does using an efficient algorithm (eg blowfish) help? -- Chris
From: Dave on 6 Nov 2009 07:45 Chris Ridd wrote: > On 2009-11-06 11:35:32 +0000, Dave <foo(a)coo.com> said: > >> I have two Sun workstations which I want to copy a lot of data between >> - one Sun is quite old, the other is new. >> >> 1) Sun Blade 2000, 2 x 1200 MHz SPARC >> 2) Sun Ultra 27. Quad core 3.33 GHz Xeon. >> >> I've stuck a Gigabit Ethernet card in the Blade 2000, so these have a >> gigabit connection between them. >> >> Transferring a 1000 MB file via ssh is taking 49 seconds (20 MB/s), >> compared to the 11 seconds (91 MB/s) I can achieve with ftp. It's >> clear the Blade is limiting the transfer speed, due to the speed of >> its CPU. >> >> What are my options for a faster transfer between these two machines? >> I want to do a recursive copy, so normal ftp is not convenient. >> >> I can't seem to work out how to enable rcp to work. Can anyone tell me >> how? Perhaps I can find an ftp client which can work recursively. > > Does using an efficient algorithm (eg blowfish) help? > I tried using blowfish, but I get a message there is no matching ciper. (See end of posts for some outputs with various cipers tried). I can't see why blowfish should not work, as I've not added anything to sshd_config to limit the cipers that can be used: The only ciper I tried that appears to work is 'aes128-ctr' and that took almost identical amount of time to the secure copy where I did not specify the ciper. This rather makes me think aes128-ctr' might be the default ciper. I'm sure I've used 'blowfish' as a ciper on Solaris 10 machines before. But I somewhat doubt the overhead will be so much lower that it will give a dramatic increase in performance. I have noticed 'blowfish' faster in the past, but not by a factor of 4, which is what I need to achieve here. That said, none of the stuff had Gigabit connections then, so perhaps it was limited by the 100 Mbit/s connection, rather than the CPU speed. Clearly here the CPU of the Blade 2000 can't keep up. I'm going to try wget, to see if that works. I think if I enable the ftp server on the machine with the data on, I can pull it off via wget. I'd rather use scp if I could find out how to get it to work! In the copies below, the machine 'swan' is the Blade 2000 running Solaris 10 update 7. The machine 'hawk' where the file is being transfered to is the Ultra 27 running OpenSolaris 06/2009. drkirkby(a)swan:[~] $ scp fox drkirkby(a)hawk:. fox 100% |*************************************************************************************************************************************************| 1000 MB 00:48 drkirkby(a)swan:[~] $ time scp fox drkirkby(a)hawk:. fox 100% |*************************************************************************************************************************************************| 1000 MB 00:48 real 0m49.177s user 0m41.599s sys 0m11.323s drkirkby(a)swan:[~] $ time scp -c blowfish fox drkirkby(a)hawk:. no matching cipher found: client blowfish-cbc server aes128-ctr,aes192-ctr,aes256-ctr,arcfour128,arcfour256,arcfour lost connection real 0m0.072s user 0m0.003s sys 0m0.006s drkirkby(a)swan:[~] $ time scp -c blowfish-cbc fox drkirkby(a)hawk:. no matching cipher found: client blowfish-cbc server aes128-ctr,aes192-ctr,aes256-ctr,arcfour128,arcfour256,arcfour lost connection real 0m0.072s user 0m0.002s sys 0m0.006s drkirkby(a)swan:[~] $ time scp -c aes128-ctr fox drkirkby(a)hawk:. fox 100% |*************************************************************************************************************************************************| 1000 MB 00:48 real 0m49.117s user 0m41.504s sys 0m11.430s drkirkby(a)swan:[~] $ time scp -c aes192-ctr fox drkirkby(a)hawk:. Unknown cipher type 'aes192-ctr' lost connection real 0m0.019s user 0m0.007s sys 0m0.013s drkirkby(a)swan:[~] $ time scp -c aes256-ctr fox drkirkby(a)hawk:. Unknown cipher type 'aes256-ctr' lost connection real 0m0.020s user 0m0.007s sys 0m0.013s
From: solx on 6 Nov 2009 07:56 Dave wrote: > I have two Sun workstations which I want to copy a lot of data between - > one Sun is quite old, the other is new. > > 1) Sun Blade 2000, 2 x 1200 MHz SPARC > 2) Sun Ultra 27. Quad core 3.33 GHz Xeon. > > I've stuck a Gigabit Ethernet card in the Blade 2000, so these have a > gigabit connection between them. > > Transferring a 1000 MB file via ssh is taking 49 seconds (20 MB/s), > compared to the 11 seconds (91 MB/s) I can achieve with ftp. It's clear > the Blade is limiting the transfer speed, due to the speed of its CPU. > > What are my options for a faster transfer between these two machines? I > want to do a recursive copy, so normal ftp is not convenient. > > I can't seem to work out how to enable rcp to work. Can anyone tell me > how? Perhaps I can find an ftp client which can work recursively. > > Security is not an issue - these is between two machines on a home LAN. > > Any other suggestions? > > Dave Hi Dave, If you use FileZilla on your Sun Blade 2000 to transfer the files using port 22 (scp) it will create up to 10 simultaneous file transfers. Filezilla will also handle recursive directory copying and creation.
From: Dave on 6 Nov 2009 08:18
solx wrote: > Dave wrote: >> I have two Sun workstations which I want to copy a lot of data between >> - one Sun is quite old, the other is new. >> >> 1) Sun Blade 2000, 2 x 1200 MHz SPARC >> 2) Sun Ultra 27. Quad core 3.33 GHz Xeon. >> >> I've stuck a Gigabit Ethernet card in the Blade 2000, so these have a >> gigabit connection between them. >> >> Transferring a 1000 MB file via ssh is taking 49 seconds (20 MB/s), >> compared to the 11 seconds (91 MB/s) I can achieve with ftp. It's >> clear the Blade is limiting the transfer speed, due to the speed of >> its CPU. >> >> What are my options for a faster transfer between these two machines? >> I want to do a recursive copy, so normal ftp is not convenient. >> >> I can't seem to work out how to enable rcp to work. Can anyone tell me >> how? Perhaps I can find an ftp client which can work recursively. >> >> Security is not an issue - these is between two machines on a home LAN. >> >> Any other suggestions? >> >> Dave > Hi Dave, > > If you use FileZilla on your Sun Blade 2000 to transfer the files using > port 22 (scp) it will create up to 10 simultaneous file transfers. > Filezilla will also handle recursive directory copying and creation. > > Thank you. That might help a bit, as the Blade 2000 has two 1200 MHz CPUs. But even if I could double the throughput from 20 to 40 MB/s, it is still less than half of what I get via ftp (91 MB/s). I think wget will solve me problem. I initially had trouble, as the ftp server would not accept the password. The password has a '$' symbol in it, and despite putting the password in quotes wget --mirror --ftp-user=drkirkby --ftp-password="yyyyy$x" ftp://swan/ the password would not work. I then tried escaping the '$' character, and that seems to have solved it, so: wget --mirror --ftp-user=drkirkby --ftp-password="yyyyy\$x" ftp://swan/ does look like it will copy from the host 'swan', at a little over 90 MB/s, which is all I can reasonably expect from Gigabit ethernet. I'm actually surprise it is achieving that, as I would have thought the SCSI bus in the Blade 2000 was not able to do that, but it seems to be ok. Dave |