Why is network reading slow?
in [MFC]
|
Prev: Is it a bug in mciSendCommand function?
Next: AFX_MANAGE_STATE() macro disturbs the startup position of the ATL dialog
From: Hans-J. Ude on 14 Jan 2010 04:11 Peter Olcott wrote: >I can copy a file to and from my Samba network in 20 >seconds, yet it takes 80 seconds to compute its MD5 value. >The delay is not related to the MD5 processing time because >it can compute this in 12 seconds from the local drive. > >What is going on here? You can use a network sniffer like Wireshark to see what's going on. It gives you detailed information, includes filters, timestamps, etc. Hans
From: Peter Olcott on 14 Jan 2010 09:15 "Tom Serface" <tom(a)camaswood.com> wrote in message news:uWT%23vsOlKHA.1536(a)TK2MSFTNGP06.phx.gbl... > Not sure if this helps or not, but I've also noticed that > network access speeds up if you map a drive letter to the > share rather than using a UNC. I've seen as much as 3x > speed up. I'm not sure why. > > Tom I am already doing that. > > "Peter Olcott" <NoSpam(a)SeeScreen.com> wrote in message > news:ut-dnZaUeubM69PWnZ2dnUVZ_vWdnZ2d(a)giganews.com... >> I can copy a file to and from my Samba network in 20 >> seconds, yet it takes 80 seconds to compute its MD5 >> value. The delay is not related to the MD5 processing >> time because it can compute this in 12 seconds from the >> local drive. >> >> What is going on here? >>
From: Peter Olcott on 14 Jan 2010 10:01 "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message news:OzySgEPlKHA.2132(a)TK2MSFTNGP05.phx.gbl... > Peter Olcott wrote: > >> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >> message news:%23OQCOfNlKHA.1824(a)TK2MSFTNGP04.phx.gbl... >>> Peter Olcott wrote: >>> >>> By File Copy, you mean DOS copy command or the >>> CopyFile() API? > > > > >> I am using the DOS command prompt's copy command. This >> is fast. >> >> >> The problem is the contradiction formed by the fact that >> reading and writng the file is fast, while reading and >> not wrting this same file is slow. >> I am currently using fopen() and fread(); I am using >> Windows XP. > > True, if the DOS copy command is fast,then I believe the > code you are using is not optimal. The DOS Copy is using > the same CreateFile() API which fopen() also finally uses > in the RTL. So you should be able to match the same > performance of the DOS Copy command. > > Have you tried using setvbuf to set a buffer cache? > > Here is a small test code that opens a 50 meg file: > > // File: V:\wc7beta\testbufsize.cpp > // Compile with: cl testbufsize.cpp > > #include <stdio.h> > #include <windows.h> > > void main(char argc, char *argv[]) > { > char _cache[1024*16] = {0}; // 16K cache > BYTE buf[1024*1] = {0}; // 1K buffer > > FILE *fv = fopen("largefile.dat","rb"); > if (fv) { > int res = setvbuf(fv, _cache, _IOFBF, > sizeof(_cache)); > DWORD nTotal = 0; > DWORD nDisks = 0; > DWORD nLoops = 0; > DWORD nStart = GetTickCount(); > while (!feof(fv)) { > nLoops++; > memset(&buf,sizeof(buf),0); > int nRead = fread(buf,1,sizeof(buf),fv); > nTotal +=nRead; > if (nRead > 0 && !fv->_cnt) nDisks++; > } > fclose(fv); > printf("Time: %d | Size: %d | Reads: %d | Disks: > %d\n", > GetTickCount()-nStart, > nTotal, > nLoops, > nDisks); > } > } > > What this basically shows is the number of disk hits it > makes > by checking the fv->_cnt value. It shows that as long as > the cache size is larger than the read buffer size, you > get the same number of disk hits. I also spit out the > milliseconds. Subsequent runs, of course, is faster since > the OS API CreateFile() is used by the RTL in buffer mode. > > Also do you know what protocol you have Samba using? I am guessing that the code above will work with a file of any size? If that is the case, then you solved my problem. The only Samba protocol that I am aware of is smb. > > > -- > HLS
From: Hector Santos on 14 Jan 2010 12:14 Peter Olcott wrote: > "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message > I am guessing that the code above will work with a file of > any size? If that is the case, then you solved my problem. > The only Samba protocol that I am aware of is smb. Should be fine as long at you keep in 32 bit chucks. For the benefit of the future mail archive researchers coming across the same issue, I'm going to assume the solution was using rtl setvbuf() to set a user-defined cache buffer of reasonable size. -- HLS
From: Joseph M. Newcomer on 14 Jan 2010 14:27
See below... On Wed, 13 Jan 2010 23:55:56 -0600, "Peter Olcott" <NoSpam(a)SeeScreen.com> wrote: > >"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >news:%23OQCOfNlKHA.1824(a)TK2MSFTNGP04.phx.gbl... >> Peter Olcott wrote: >> >>> >>> I am doing block I/O, and it is very fast on the local >>> drive and much slower on the 1.0 gb LAN, yet file copies >>> to and from the LAN are still fast. >>> >>> (1) File copy to and from the LAN is faster than local >>> drive copies, 20 seconds for LAN, 25 seconds for local. >>> (2) Block I/O is fast on the local drive, 12 seconds for >>> 632 MB. >>> (3) Block I/O is slow on the LAN, 80 seconds for 632 MB. >>> I also tried changing the block size from 4K to 1500 >>> bytes and 9000 bytes (consistent with Ethernet frame >>> size), this did not help. >> >> By File Copy, you mean DOS copy command or the CopyFile() >> API? >I am using the DOS command prompt's copy command. This is >fast. > >> >> To me, the above appears to be consistent with a caching >> issue that your code is not enabling when the file is >> first open. The "File Copy" is doing it, but you are not. >> Probably showing how you are opening the file will help, >> i.e. the CreateFile() function or fopen(). >> >> Another thing is maybe to check google >> >> Search: SAMBA Slow File Copy > >The problem is the contradiction formed by the fact that >reading and writng the file is fast, while reading and not >wrting this same file is slow. >I am currently using fopen() and fread(); I am using >Windows XP. **** Use of fopen/fread would certainly indicate that you are not doing this in anything like an optimal fashion. If you want to read a file that is under 100MB, it is usually best just to allocate a buffer the size of the file, CreateFile, do a single ReadFile, do your computation, do a WriteFile, and you are done. You are comparing two completely unrelated concepts: fopen/fread and a copy command; what you didn't ask was "what is the fastest way to read a file"; instead, you observe that two completely different technologies have different performance. You did not actually state this in your original question; you just used a generic concept of "copy". Details matter! Note that fread, called thousands of times, is amazingly slow in comparison to a single ReadFile. By failing to supply all the critical information, you essentially asked "Why is it that I can get from city A to city B in 20 minutes, but my friend takes two hours?" and neglected to mention you took the high-speed train while your friend went by bicycle. joe **** > >> >> There are some interesting hits, in particular if you are >> using Vista, this MS Samba hotfix for Vista, >> >> http://support.microsoft.com/kb/931770 >> >> There was another 2007 thread that the original poster >> said turning off indexing improved the speed. >> >> -- >> HLS > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm |