Why is network reading slow?
in [MFC]
|
Prev: Is it a bug in mciSendCommand function?
Next: AFX_MANAGE_STATE() macro disturbs the startup position of the ATL dialog
From: Hector Santos on 14 Jan 2010 00:09 Peter Olcott wrote: > > I am doing block I/O, and it is very fast on the local drive > and much slower on the 1.0 gb LAN, yet file copies to and > from the LAN are still fast. > > (1) File copy to and from the LAN is faster than local drive > copies, 20 seconds for LAN, 25 seconds for local. > (2) Block I/O is fast on the local drive, 12 seconds for 632 > MB. > (3) Block I/O is slow on the LAN, 80 seconds for 632 MB. > I also tried changing the block size from 4K to 1500 bytes > and 9000 bytes (consistent with Ethernet frame size), this > did not help. > By File Copy, you mean DOS copy command or the CopyFile() API? To me, the above appears to be consistent with a caching issue that your code is not enabling when the file is first open. The "File Copy" is doing it, but you are not. Probably showing how you are opening the file will help, i.e. the CreateFile() function or fopen(). Another thing is maybe to check google Search: SAMBA Slow File Copy There are some interesting hits, in particular if you are using Vista, this MS Samba hotfix for Vista, http://support.microsoft.com/kb/931770 There was another 2007 thread that the original poster said turning off indexing improved the speed. -- HLS
From: Peter Olcott on 14 Jan 2010 00:55 "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message news:%23OQCOfNlKHA.1824(a)TK2MSFTNGP04.phx.gbl... > Peter Olcott wrote: > >> >> I am doing block I/O, and it is very fast on the local >> drive and much slower on the 1.0 gb LAN, yet file copies >> to and from the LAN are still fast. >> >> (1) File copy to and from the LAN is faster than local >> drive copies, 20 seconds for LAN, 25 seconds for local. >> (2) Block I/O is fast on the local drive, 12 seconds for >> 632 MB. >> (3) Block I/O is slow on the LAN, 80 seconds for 632 MB. >> I also tried changing the block size from 4K to 1500 >> bytes and 9000 bytes (consistent with Ethernet frame >> size), this did not help. > > By File Copy, you mean DOS copy command or the CopyFile() > API? I am using the DOS command prompt's copy command. This is fast. > > To me, the above appears to be consistent with a caching > issue that your code is not enabling when the file is > first open. The "File Copy" is doing it, but you are not. > Probably showing how you are opening the file will help, > i.e. the CreateFile() function or fopen(). > > Another thing is maybe to check google > > Search: SAMBA Slow File Copy The problem is the contradiction formed by the fact that reading and writng the file is fast, while reading and not wrting this same file is slow. I am currently using fopen() and fread(); I am using Windows XP. > > There are some interesting hits, in particular if you are > using Vista, this MS Samba hotfix for Vista, > > http://support.microsoft.com/kb/931770 > > There was another 2007 thread that the original poster > said turning off indexing improved the speed. > > -- > HLS
From: Tom Serface on 14 Jan 2010 02:28 Not sure if this helps or not, but I've also noticed that network access speeds up if you map a drive letter to the share rather than using a UNC. I've seen as much as 3x speed up. I'm not sure why. Tom "Peter Olcott" <NoSpam(a)SeeScreen.com> wrote in message news:ut-dnZaUeubM69PWnZ2dnUVZ_vWdnZ2d(a)giganews.com... > I can copy a file to and from my Samba network in 20 seconds, yet it takes > 80 seconds to compute its MD5 value. The delay is not related to the MD5 > processing time because it can compute this in 12 seconds from the local > drive. > > What is going on here? >
From: Hector Santos on 14 Jan 2010 03:10 Peter Olcott wrote: > "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message > news:%23OQCOfNlKHA.1824(a)TK2MSFTNGP04.phx.gbl... >> Peter Olcott wrote: >> >> By File Copy, you mean DOS copy command or the CopyFile() >> API? > > I am using the DOS command prompt's copy command. This is > fast. > > > The problem is the contradiction formed by the fact that > reading and writng the file is fast, while reading and not > wrting this same file is slow. > I am currently using fopen() and fread(); I am using > Windows XP. True, if the DOS copy command is fast,then I believe the code you are using is not optimal. The DOS Copy is using the same CreateFile() API which fopen() also finally uses in the RTL. So you should be able to match the same performance of the DOS Copy command. Have you tried using setvbuf to set a buffer cache? Here is a small test code that opens a 50 meg file: // File: V:\wc7beta\testbufsize.cpp // Compile with: cl testbufsize.cpp #include <stdio.h> #include <windows.h> void main(char argc, char *argv[]) { char _cache[1024*16] = {0}; // 16K cache BYTE buf[1024*1] = {0}; // 1K buffer FILE *fv = fopen("largefile.dat","rb"); if (fv) { int res = setvbuf(fv, _cache, _IOFBF, sizeof(_cache)); DWORD nTotal = 0; DWORD nDisks = 0; DWORD nLoops = 0; DWORD nStart = GetTickCount(); while (!feof(fv)) { nLoops++; memset(&buf,sizeof(buf),0); int nRead = fread(buf,1,sizeof(buf),fv); nTotal +=nRead; if (nRead > 0 && !fv->_cnt) nDisks++; } fclose(fv); printf("Time: %d | Size: %d | Reads: %d | Disks: %d\n", GetTickCount()-nStart, nTotal, nLoops, nDisks); } } What this basically shows is the number of disk hits it makes by checking the fv->_cnt value. It shows that as long as the cache size is larger than the read buffer size, you get the same number of disk hits. I also spit out the milliseconds. Subsequent runs, of course, is faster since the OS API CreateFile() is used by the RTL in buffer mode. Also do you know what protocol you have Samba using? -- HLS
From: Hector Santos on 14 Jan 2010 03:34
Tom Serface wrote: > Not sure if this helps or not, but I've also noticed that network access > speeds up if you map a drive letter to the share rather than using a > UNC. I've seen as much as 3x speed up. I'm not sure why. Thats a good point. That was differently something I had to do under NT and always keep that in mind with other OSes. But after running out of drive letters and using UNC in certain new scenarios, I have not seen a slow down. I figured MS addressed this old problem. I seem to recall this was explained once back in the day. -- HLS |