Why is network reading slow?
in [MFC]
|
Prev: Is it a bug in mciSendCommand function?
Next: AFX_MANAGE_STATE() macro disturbs the startup position of the ATL dialog
From: Hector Santos on 14 Jan 2010 22:42 Our application server product (DOS version) was introduced in 1984, the 32 bit version introduced in 1996. I don't recall if these extended 64 bit file I/O functions were available for the original design engineers to use. Obviously 32 bit was the NEW THING and equivalent to the same or near 32 bit to 64 bit design pressures as it was for the 16 bit to 32 bit design pressues. I suspect 64 bit was the LAST THING on anyone's mind nor a consideration. Yes, we need to do more serious testing with the ss64lib code. Its not in production. What I posted was first "drafts" with limited proof of concept testing. I do appreciate your input on proper c/c++ syntaxing or casting. Overall, we were still stuck in a badly quoted proverbial: "To do full 64 bit or not do full 64 bit (only partial)" design dilemma :) Going full 64 bit would be a major, I mean major RPC client/server framework redesign with a 250 function client API that is still based on 32 bit I/O and structures. Changing it is not an option without breaking thousands of 3rd party client applications and customer embedded server side code applications and we need to still deal with the "Shim" idea for backward compatibility 32 bit clients connecting to a new 64 bit server over RPC! -- Joseph M. Newcomer wrote: > It wouldn't have occurred to me that seeking was a problem, because if I were doing any > serious file system, I would not be using anything with a 32-bit limitation. Haven't > really worried about this problem since about 1997, when I did my first database that was > expected to hit terabytes. We just ignored the primitive and outdated C library, since it > did nothing useful for us. I notice you are still using the obsolete SetFilePointer, > instead of the more modern SetFilePointerEx, which has been around for a decade. > > Also, returning the .QuadPart component should fail in any environment in which TINT is > declared as a DWORD, since the compiler should complain about truncation of a value (you > should always build at /W4). You should use the same cast as for the file size, (TINT). > joe > > On Thu, 14 Jan 2010 21:51:24 -0500, Hector Santos <sant9442(a)nospam.gmail.com> wrote: > >> Joseph M. Newcomer wrote: >> >>> Yes, but the file size was given as 50MB. >>> joe >> Its amazing how "big" is relative these days. My first computer had a >> $1,500 Micropolis 10 meg drive! :) >> >> BTW, it (the code posted) should not be an issue when just streaming >> in a file of any size. >> >> The problem begins when seeking a file. >> >> When seeking is required, we know by using the 64 bit WIN32 file I/O >> functions that it works for large +2gig files. >> >> In one part of our server product file handing, a ISAM database with >> four key index files have DWORD position indices. Issues occur as the >> database grows and a index goes beyond a 32 bit value. A documented >> limitation but a limitations that is not outdated. >> >> A simple solution in the works was to use the 64 bit extended file I/O >> functions which offer QUAD (double DWORD) positions. Its about to be >> implemented in new major revision of our server. >> >> For backward single source compatibility, I produced a header and >> wrapper functions. Here is the *.h and *.cpp files: >> >> //------------------------------------------------------------ >> // File Name : ss64lib.h >> //------------------------------------------------------------ >> >> #ifndef __SS64LIB_H >> #define __SS64LIB_H >> >> #ifndef _WINDOWS_ >> #include <windows.h> >> #endif >> >> #define SUPPORT_FILEIO_64BIT >> >> #ifdef SUPPORT_FILEIO_64BIT >> # define TINT INT64 >> # define TWORD QWORD >> # define TFILESIZE INT64 >> # define MAXQWORD _UI64_MAX >> # define MAXINT64 _I64_MAX >> #else >> # define TINT DWORD >> # define TWORD DWORD >> # define TFILESIZE DWORD >> #endif >> >> >> TINT ssFileSeek(HANDLE hf, TINT distance, >> WORD MoveMethod = FILE_BEGIN); >> TINT ssFileEnd(HANDLE hf); >> TINT ssFilePos(HANDLE hf); >> BOOL ssFileRewind(HANDLE hf); >> BOOL ssFileLock(HANDLE hf, TINT Offset, TINT nBytes); >> BOOL ssFileUnlock(HANDLE hf, TINT Offset, TINT nBytes); >> TFILESIZE ssFileSize(HANDLE hf); >> >> #endif // __SS64LIB_H >> >> //------------------------------------------------------------ >> // File Name : ss64lib.cpp >> //------------------------------------------------------------ >> >> #include "ss64lib.h" >> >> TINT ssFileSeek(HANDLE hf, TINT dist, WORD method) >> { >> LARGE_INTEGER li; >> li.QuadPart = dist; >> li.LowPart = SetFilePointer (hf, li.LowPart, &li.HighPart, method); >> if (li.LowPart == 0xFFFFFFFF && GetLastError() != NO_ERROR) { >> li.QuadPart = -1; >> } >> return li.QuadPart; >> } >> >> BOOL ssFileRewind(HANDLE hf) >> { >> return ssFileSeek(hf,0,FILE_BEGIN) == 0; >> } >> >> TINT ssFilePos(HANDLE hf) >> { >> return ssFileSeek(hf,0,FILE_CURRENT); >> } >> >> TINT ssFileEnd(HANDLE hf) >> { >> return ssFileSeek(hf,0,FILE_END); >> } >> >> BOOL ssFileLock(HANDLE hf, TINT Offset, TINT nBytes) >> { >> LARGE_INTEGER fp, nb; >> fp.QuadPart = Offset; >> nb.QuadPart = nBytes; >> return LockFile (hf,fp.LowPart,fp.HighPart,nb.LowPart,nb.HighPart); >> } >> >> BOOL ssFileUnlock(HANDLE hf, TINT Offset, TINT nBytes) >> { >> LARGE_INTEGER fp, nb; >> fp.QuadPart = Offset; >> nb.QuadPart = nBytes; >> return UnlockFile(hf,fp.LowPart,fp.HighPart,nb.LowPart,nb.HighPart); >> } >> >> TFILESIZE ssFileSize(HANDLE hf) >> { >> LARGE_INTEGER li; >> li.LowPart = GetFileSize(hf,(DWORD *)&li.HighPart); >> return (TINT)li.QuadPart; >> } > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm -- HLS
From: Alexander Grigoriev on 14 Jan 2010 22:43 Historically, I/O sizes in the kernel drivers has been stored in ULONG. And Length in IO_STACK_LOCATION is ULONG, too. That would be a bit too much hassle to convert everything to SIZE_T... "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:686vk59l8chun24uceekvdc8pt2uj4n811(a)4ax.com... > By the way, did anyone really notice that ReadFile and WriteFile in Win64 > cannot read or > write more than 4.2GB? Seems really, really strange the length and bytes > read did not > become DWORD_PTR values... > joe > > On Thu, 14 Jan 2010 16:37:26 -0500, Joseph M. Newcomer > <newcomer(a)flounder.com> wrote: > >>Yes, but the file size was given as 50MB. >> joe >> >>On Thu, 14 Jan 2010 14:24:30 -0600, Stephen Myers >><""StephenMyers\"@discussions(a)microsoft.com"> wrote: >> >>>Just to verify my (admittedly limited) understanding... >>> >>>I assume that the code posted will fail for files greater than 2GB or so >>>with a 32 bit OS due to available address space. >>> >>>Steve >>> >>>Joseph M. Newcomer wrote: >>>> See below... >>>> On Thu, 14 Jan 2010 09:01:47 -0600, "Peter Olcott" >>>> <NoSpam(a)SeeScreen.com> wrote: >>>> >>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >>>>> news:OzySgEPlKHA.2132(a)TK2MSFTNGP05.phx.gbl... >>>>>> Peter Olcott wrote: >>>>>> >>>>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>>>>> message news:%23OQCOfNlKHA.1824(a)TK2MSFTNGP04.phx.gbl... >>>>>>>> Peter Olcott wrote: >>>>>>>> >>>>>>>> By File Copy, you mean DOS copy command or the >>>>>>>> CopyFile() API? >>>>>>> I am using the DOS command prompt's copy command. This >>>>>>> is fast. >>>>>>> >>>>>>> >>>>>>> The problem is the contradiction formed by the fact that >>>>>>> reading and writng the file is fast, while reading and >>>>>>> not wrting this same file is slow. >>>>>>> I am currently using fopen() and fread(); I am using >>>>>>> Windows XP. >>>>>> True, if the DOS copy command is fast,then I believe the >>>>>> code you are using is not optimal. The DOS Copy is using >>>>>> the same CreateFile() API which fopen() also finally uses >>>>>> in the RTL. So you should be able to match the same >>>>>> performance of the DOS Copy command. >>>>>> >>>>>> Have you tried using setvbuf to set a buffer cache? >>>>>> >>>>>> Here is a small test code that opens a 50 meg file: >>>>>> >>>>>> // File: V:\wc7beta\testbufsize.cpp >>>>>> // Compile with: cl testbufsize.cpp >>>>>> >>>>>> #include <stdio.h> >>>>>> #include <windows.h> >>>>>> >>>>>> void main(char argc, char *argv[]) >>>>>> { >>>>>> char _cache[1024*16] = {0}; // 16K cache >>>>>> BYTE buf[1024*1] = {0}; // 1K buffer >>>> **** >>>> Reading a 50MB file, why such an incredibly tiny buffer? >>>> **** >>>>>> FILE *fv = fopen("largefile.dat","rb"); >>>>>> if (fv) { >>>>>> int res = setvbuf(fv, _cache, _IOFBF, >>>>>> sizeof(_cache)); >>>>>> DWORD nTotal = 0; >>>>>> DWORD nDisks = 0; >>>>>> DWORD nLoops = 0; >>>>>> DWORD nStart = GetTickCount(); >>>>>> while (!feof(fv)) { >>>>>> nLoops++; >>>>>> memset(&buf,sizeof(buf),0); >>>> **** >>>> The memset is silly. Wastes time, accomplishes nothing. You are >>>> setting a buffer to 0 >>>> right before completely overwriting it! This is like writing >>>> int a; >>>> >>>> a = 0; // make sure a is 0 before assigning b >>>> a = b; >>>> **** >>>>>> int nRead = fread(buf,1,sizeof(buf),fv); >>>>>> nTotal +=nRead; >>>>>> if (nRead > 0 && !fv->_cnt) nDisks++; >>>>>> } >>>>>> fclose(fv); >>>>>> printf("Time: %d | Size: %d | Reads: %d | Disks: >>>>>> %d\n", >>>>>> GetTickCount()-nStart, >>>>>> nTotal, >>>>>> nLoops, >>>>>> nDisks); >>>>>> } >>>>>> } >>>> **** >>>> If I were reading a small 50MB file, I would do >>>> >>>> void tmain(int argc, _TCHAR * argv[]) >>>> { >>>> HANDLE h = CreateFile(_T("largefile.dat"), GENERIC_READ, 0, NULL, >>>> OPEN_EXISTING, >>>> FILE_ATTRIBUTE_NORMAL, NULL); >>>> >>>> LARGE_INTEGER size; >>>> >>>> GetFileSizeEx(h, &size); >>>> >>>> // This code assumes file is < 4.2GB! >>>> LPVOID p = VirtualAlloc(NULL, (SIZE_T)size.LowPart, MEM_COMMIT, >>>> PAGE_READWRITE); >>>> DWORD bytesRead; >>>> ReadFile(h, p, size.LowPart, &bytesRead, NULL); >>>> ... process data >>>> VirtualFree(p, (SIZE_T)size.LowPart, MEM_DECOMMIT); >>>> return 0; >>>> } >>>> >>>> Note that the above does not do any error checking; the obvious error >>>> checking is left as >>>> an Exercise For The Reader. No read loops, no gratuitous memsets, just >>>> simple code that >>>> does exactly ONE ReadFile. >>>> joe >>>> >>>>>> What this basically shows is the number of disk hits it >>>>>> makes >>>>>> by checking the fv->_cnt value. It shows that as long as >>>>>> the cache size is larger than the read buffer size, you >>>>>> get the same number of disk hits. I also spit out the >>>>>> milliseconds. Subsequent runs, of course, is faster since >>>>>> the OS API CreateFile() is used by the RTL in buffer mode. >>>>>> >>>>>> Also do you know what protocol you have Samba using? >>>>> I am guessing that the code above will work with a file of >>>>> any size? >>>>> If that is the case, then you solved my problem. >>>>> The only Samba protocol that I am aware of is smb. >>>>> >>>>>> >>>>>> -- >>>>>> HLS >>>> Joseph M. Newcomer [MVP] >>>> email: newcomer(a)flounder.com >>>> Web: http://www.flounder.com >>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>Joseph M. Newcomer [MVP] >>email: newcomer(a)flounder.com >>Web: http://www.flounder.com >>MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Hector Santos on 14 Jan 2010 23:21 Joseph M. Newcomer wrote: > Note that fread, called thousands of times, is amazingly slow in > comparison to a single ReadFile. Hmmm, both are limited to what the backend protocol driver is using for packet size. I would never recommend using a single ReadFile read with a large block such as a 100mb. Now if the file was local as a memory map, that would be a different story and even when we know the OS itself (by default) actually using memory maps when opening files, a large single read like 100mb IMO definitely adds design pressures. What if there is a failure or some condition he wants to detect during the very large block read? IOW, he now needs to make it asynchronous anyway! Again, I can see your design point if the file was LOCAL and knowing that Windows itself actually opens file in page mode. But not over a network. I would naturally suspect engineering issues there. > By failing to supply all the critical information, you essentially asked "Why is it that I > can get from city A to city B in 20 minutes, but my friend takes two hours?" and neglected > to mention you took the high-speed train while your friend went by bicycle. I didn't have a problem myself. It was obvious what the issue was when stating a DOS file copy was faster than his code - although it did take a few mail tags. -- HLS
From: Hector Santos on 14 Jan 2010 23:55 Alexander Grigoriev wrote: > Historically, I/O sizes in the kernel drivers has been stored in ULONG. And > Length in IO_STACK_LOCATION is ULONG, too. That would be a bit too much > hassle to convert everything to SIZE_T... > To add to this, the other issue with the misconception of using ULONG or DWORD, is that even if your own application is "Unsigned Ready" with the hope to handing the positive range representations, i.e, 0 to 4 MEG with support an extended a file size of an ideal 4 meg size in a 32 bit memory space, the problem I've seen is interfacing "other" API functions, libraries and/or WIN32 API itself where its still working and sometimes naturally so in an +/- signed integer world. In short, for example, once upon a time when we documented our requirements and limits, it use to say among its list: o Up to 4 GigaByte file sizes That was based on our persistent and consistent usage of DWORD within our own code. It was a theoretical limit, but empirically the limit was 2 GIG because of the various external interfaces. So today, we use the 2 gig limit in our support docs and don't bother taking about what we theoretically can handle. My Point? As we move into the 64 bit world, the positive range would still be a theory and not one to rely on in new 64 bit designs. I guess that will design on how an application interfaces with the outside world. Of course, one might might suggest "Even dealing in a signed 64 bit world is limitless and no one should have problems. So design with INT64 in mind" Two things come to mind: - We thought that was the case when moving to 32 bit, - Quoting the late George Carlin: "More Space, More Junk" :) As as side issue. One of my beefs with Microsoft and .NET, and I have to review this issue again, but under .NET, natural LONG type is 64 bit! I have to review this again to see if it was all the .NET languages or one or more, C# C++/.NET and/or VB, I seem to recall it was not consistent and I thought that was a mistake to create confusion by using a long time established LONG type keyword to be 64 bit still within a 32 bit compiled world. I can certainly understand the thinking of the design team but it definitely was absent of established C/C++ 32 bit engineering considerations. -- HLS
From: Tom Serface on 15 Jan 2010 01:38
Joe, My experience has been that if the buffer gets too large it starts to slow down the operation. In my case, I have to read all sizes of files and I've found the optimal buffer to be around 16K (I think that's what OP was using). I use the SDK functions CreateFile, ReadFile, WriteFile, rather than MFC and my copy routine is around 2-3x what the DOS copy command does (or CopyFile). I had to write my own for a special purpose though because I have to glue files back together that span multiple volumes, but I was happy to get better performance than I was getting with CopyFile previously. I think there is a trade-off somewhere, but I'm not entirely sure where. I just did trial and error with different scenarios until I got it the best I could. Tom "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:f2suk5dtuqe1drflfbk6e5ol0io4i7qocf(a)4ax.com... > See below... > On Thu, 14 Jan 2010 09:01:47 -0600, "Peter Olcott" <NoSpam(a)SeeScreen.com> > wrote: > >> >>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >>news:OzySgEPlKHA.2132(a)TK2MSFTNGP05.phx.gbl... >>> Peter Olcott wrote: >>> >>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in >>>> message news:%23OQCOfNlKHA.1824(a)TK2MSFTNGP04.phx.gbl... >>>>> Peter Olcott wrote: >>>>> >>>>> By File Copy, you mean DOS copy command or the >>>>> CopyFile() API? >>> >>> > >>> >>>> I am using the DOS command prompt's copy command. This >>>> is fast. >>>> >>>> >>>> The problem is the contradiction formed by the fact that >>>> reading and writng the file is fast, while reading and >>>> not wrting this same file is slow. >>>> I am currently using fopen() and fread(); I am using >>>> Windows XP. >>> >>> True, if the DOS copy command is fast,then I believe the >>> code you are using is not optimal. The DOS Copy is using >>> the same CreateFile() API which fopen() also finally uses >>> in the RTL. So you should be able to match the same >>> performance of the DOS Copy command. >>> >>> Have you tried using setvbuf to set a buffer cache? >>> >>> Here is a small test code that opens a 50 meg file: >>> >>> // File: V:\wc7beta\testbufsize.cpp >>> // Compile with: cl testbufsize.cpp >>> >>> #include <stdio.h> >>> #include <windows.h> >>> >>> void main(char argc, char *argv[]) >>> { >>> char _cache[1024*16] = {0}; // 16K cache >>> BYTE buf[1024*1] = {0}; // 1K buffer > **** > Reading a 50MB file, why such an incredibly tiny buffer? > **** >>> >>> FILE *fv = fopen("largefile.dat","rb"); >>> if (fv) { >>> int res = setvbuf(fv, _cache, _IOFBF, >>> sizeof(_cache)); >>> DWORD nTotal = 0; >>> DWORD nDisks = 0; >>> DWORD nLoops = 0; >>> DWORD nStart = GetTickCount(); >>> while (!feof(fv)) { >>> nLoops++; >>> memset(&buf,sizeof(buf),0); > **** > The memset is silly. Wastes time, accomplishes nothing. You are setting > a buffer to 0 > right before completely overwriting it! This is like writing > int a; > > a = 0; // make sure a is 0 before assigning b > a = b; > **** >>> int nRead = fread(buf,1,sizeof(buf),fv); >>> nTotal +=nRead; >>> if (nRead > 0 && !fv->_cnt) nDisks++; >>> } >>> fclose(fv); >>> printf("Time: %d | Size: %d | Reads: %d | Disks: >>> %d\n", >>> GetTickCount()-nStart, >>> nTotal, >>> nLoops, >>> nDisks); >>> } >>> } > **** > If I were reading a small 50MB file, I would do > > void tmain(int argc, _TCHAR * argv[]) > { > HANDLE h = CreateFile(_T("largefile.dat"), GENERIC_READ, 0, NULL, > OPEN_EXISTING, > FILE_ATTRIBUTE_NORMAL, NULL); > > LARGE_INTEGER size; > > GetFileSizeEx(h, &size); > > // This code assumes file is < 4.2GB! > LPVOID p = VirtualAlloc(NULL, (SIZE_T)size.LowPart, MEM_COMMIT, > PAGE_READWRITE); > DWORD bytesRead; > ReadFile(h, p, size.LowPart, &bytesRead, NULL); > ... process data > VirtualFree(p, (SIZE_T)size.LowPart, MEM_DECOMMIT); > return 0; > } > > Note that the above does not do any error checking; the obvious error > checking is left as > an Exercise For The Reader. No read loops, no gratuitous memsets, just > simple code that > does exactly ONE ReadFile. > joe > >>> >>> What this basically shows is the number of disk hits it >>> makes >>> by checking the fv->_cnt value. It shows that as long as >>> the cache size is larger than the read buffer size, you >>> get the same number of disk hits. I also spit out the >>> milliseconds. Subsequent runs, of course, is faster since >>> the OS API CreateFile() is used by the RTL in buffer mode. >>> >>> Also do you know what protocol you have Samba using? >> >>I am guessing that the code above will work with a file of >>any size? >>If that is the case, then you solved my problem. >>The only Samba protocol that I am aware of is smb. >> >>> >>> >>> -- >>> HLS >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm |