will FileLen give an error on very large files?
in [Visual Basic]
|
From: RB Smissaert on 17 Nov 2009 03:05 > You seemed to be using it as way to tell if a file exists. No, I do use GetAttr for that, just as in the function you posted. The idea was to try the simpler (and faster?) FileLen first before using the API method. In any case have this all sorted now and after a minor modification (see comment in below code) the API method is now about as fast as the FileLen method: Public Function FileSizeAPI(strFilePath As String) As Variant Dim hSearch As Long Dim WFD As WIN32_FIND_DATA On Error GoTo ERROROUT hSearch = FindFirstFile(strFilePath, WFD) If hSearch = -1 Then FileSizeAPI = -1 Else FindClose hSearch 'doing this before the next line makes it faster, not sure why FileSizeAPI = LargeInteger(WFD.nFileSizeLow, WFD.nFileSizeHigh) End If Exit Function ERROROUT: FileSizeAPI = -1 End Function RBS "Karl E. Peterson" <karl(a)exmvps.org> wrote in message news:ORpeOexZKHA.740(a)TK2MSFTNGP04.phx.gbl... > RB Smissaert wrote: >> Thanks for looking at that and yes, should get a virtual machine really. >> >>> Why are you using that? >> Great majority of files passed to the function will be small, so I >> thought I >> might save some time to try the FileLen first. > > Not the point. You seemed to be using it as way to tell if a file exists. > The most commonly accepted method for that is to use GetAttr. > >>> I guess what you're asking is whether FileLen() will toss an error with >>> a >>> really huge file? >> Exactly, as that would be no good. > > Nope, it definitely wouldn't. > >>> I'll try... Nope! Looks like it just wraps around and around... >> OK, thanks and that is no good either, so will need to chuck FileLen out >> then. >> Maybe FileLen uses that API in any case, so trying FileLen first won't >> gain >> any time in that case. >> Will do some timing tests to see if that might be the case. > > is this what you're looking for? > > Public Function FileExists(ByVal FileName As String) As Boolean > Dim nAttr As Long > ' Grab this files attributes, and make sure it isn't a folder. > ' This test includes cases where file doesn't exist at all. > On Error GoTo NoFile > nAttr = GetAttr(FileName) > If (nAttr And vbDirectory) <> vbDirectory Then > FileExists = True > End If > NoFile: > End Function > > -- > .NET: It's About Trust! > http://vfred.mvps.org >
From: Karl E. Peterson on 17 Nov 2009 13:41 RB Smissaert wrote: >> You seemed to be using it as way to tell if a file exists. > > No, I do use GetAttr for that, just as in the function you posted. > The idea was to try the simpler (and faster?) FileLen first before using the > API method. Ah, I see. Sorry for the quick (mis)reading of the code, there. > In any case have this all sorted now and after a minor modification (see > comment in below code) the API method is now about as fast as the FileLen method: Yeah, I would think. You'd probably see some gains using the W version, too, if you're so inclined. How many files are you dealing with, that speed is a concern here? > Public Function FileSizeAPI(strFilePath As String) As Variant > > Dim hSearch As Long > Dim WFD As WIN32_FIND_DATA > > On Error GoTo ERROROUT > > hSearch = FindFirstFile(strFilePath, WFD) > > If hSearch = -1 Then > FileSizeAPI = -1 > Else > FindClose hSearch 'doing this before the next line makes it faster, not > sure why > FileSizeAPI = LargeInteger(WFD.nFileSizeLow, WFD.nFileSizeHigh) > End If > > Exit Function > > ERROROUT: > FileSizeAPI = -1 > > End Function Another minor, perhaps even negligble, optimization would be to stop comparing a Long with an Integer in your If test. Hmmm, also, what's with the error trapping? If speed is an issue, get rid of that, too. :-) -- ..NET: It's About Trust! http://vfred.mvps.org
From: RB Smissaert on 17 Nov 2009 15:41 Usually this will be used just to get the size of one file, so speed is not an issue, but I also use it in a loop to read many files, so the faster the better. This is the optimized version then using your suggestions and it is some 15% faster than the posted code: Private Declare Sub CopyMemory _ Lib "kernel32" _ Alias "RtlMoveMemory" (Dest As Any, _ Src As Any, _ ByVal cBytes As Long) Private Declare Function FindFirstFile Lib "kernel32" _ Alias "FindFirstFileW" _ (ByVal lpFileName As Long, _ lpFindFileData As WIN32_FIND_DATA) As Long Private Declare Function FindClose Lib "kernel32" _ (ByVal hFindFile As Long) As Long Private Const MAX_PATH = 260 Private Type FILETIME dwLowDateTime As Long dwHighDateTime As Long End Type Private Type WIN32_FIND_DATA dwFileAttributes As Long ftCreationTime As FILETIME ftLastAccessTime As FILETIME ftLastWriteTime As FILETIME nFileSizeHigh As Long nFileSizeLow As Long dwReserved0 As Long dwReserved1 As Long cFileName(0 To (MAX_PATH * 2) - 1) As Byte cAlternate(0 To 27) As Byte End Type Private Function FileSizeAPI(strFilePath As String) As Variant Dim hSearch As Long Dim WFD As WIN32_FIND_DATA hSearch = FindFirstFile(StrPtr(strFilePath), WFD) If hSearch = -1& Then FileSizeAPI = -1& Else FindClose hSearch FileSizeAPI = LargeInteger(WFD.nFileSizeLow, WFD.nFileSizeHigh) End If End Function Private Function LargeInteger(ByVal LoPart As Long, _ ByVal HiPart As Long) As Variant Dim nResult As Currency 'Copy the parts, appropriately. CopyMemory ByVal VarPtr(nResult), LoPart, 4 CopyMemory ByVal VarPtr(nResult) + 4, HiPart, 4 'Return the result as VarType Decimal(14). LargeInteger = nResult * CDec(10000) End Function Just one thing I don't get and that is that I thought I could do this (modification suggested by Peter Thornton) to gain some speed: If WFD.nFileSizeHigh Then FileSizeAPI = LargeInteger(WFD.nFileSizeLow, WFD.nFileSizeHigh) Else FileSizeAPI = WFD.nFileSizeLow End If But that fails now on large files, giving negative figures due to WFD.nFileSizeHigh = 0 Any idea what I might be doing wrong there? RBS "Karl E. Peterson" <karl(a)exmvps.org> wrote in message news:OFM3mW7ZKHA.4688(a)TK2MSFTNGP06.phx.gbl... > RB Smissaert wrote: >>> You seemed to be using it as way to tell if a file exists. >> >> No, I do use GetAttr for that, just as in the function you posted. >> The idea was to try the simpler (and faster?) FileLen first before using >> the >> API method. > > Ah, I see. Sorry for the quick (mis)reading of the code, there. > >> In any case have this all sorted now and after a minor modification (see >> comment in below code) the API method is now about as fast as the FileLen >> method: > > Yeah, I would think. You'd probably see some gains using the W version, > too, if you're so inclined. How many files are you dealing with, that > speed is a concern here? > > >> Public Function FileSizeAPI(strFilePath As String) As Variant >> >> Dim hSearch As Long >> Dim WFD As WIN32_FIND_DATA >> >> On Error GoTo ERROROUT >> >> hSearch = FindFirstFile(strFilePath, WFD) >> >> If hSearch = -1 Then >> FileSizeAPI = -1 >> Else >> FindClose hSearch 'doing this before the next line makes it faster, >> not >> sure why >> FileSizeAPI = LargeInteger(WFD.nFileSizeLow, WFD.nFileSizeHigh) >> End If >> >> Exit Function >> >> ERROROUT: >> FileSizeAPI = -1 >> >> End Function > > Another minor, perhaps even negligble, optimization would be to stop > comparing a Long with an Integer in your If test. > > Hmmm, also, what's with the error trapping? If speed is an issue, get rid > of that, too. :-) > -- > .NET: It's About Trust! > http://vfred.mvps.org > > >
From: Karl E. Peterson on 17 Nov 2009 16:26 RB Smissaert wrote: > Usually this will be used just to get the size of one file, so speed is not > an issue, but > I also use it in a loop to read many files, so the faster the better. > This is the optimized version then using your suggestions and it is some 15% > faster than the posted code: Nice! > > Private Declare Sub CopyMemory _ > Lib "kernel32" _ > Alias "RtlMoveMemory" (Dest As Any, _ > Src As Any, _ > ByVal cBytes As Long) > > Private Declare Function FindFirstFile Lib "kernel32" _ > Alias "FindFirstFileW" _ > (ByVal lpFileName As Long, _ > lpFindFileData As WIN32_FIND_DATA) As Long > > Private Declare Function FindClose Lib "kernel32" _ > (ByVal hFindFile As Long) As Long > > Private Const MAX_PATH = 260 > > Private Type FILETIME > dwLowDateTime As Long > dwHighDateTime As Long > End Type > > Private Type WIN32_FIND_DATA > dwFileAttributes As Long > ftCreationTime As FILETIME > ftLastAccessTime As FILETIME > ftLastWriteTime As FILETIME > nFileSizeHigh As Long > nFileSizeLow As Long > dwReserved0 As Long > dwReserved1 As Long > cFileName(0 To (MAX_PATH * 2) - 1) As Byte > cAlternate(0 To 27) As Byte > End Type > > Private Function FileSizeAPI(strFilePath As String) As Variant > > Dim hSearch As Long > Dim WFD As WIN32_FIND_DATA > > hSearch = FindFirstFile(StrPtr(strFilePath), WFD) > > If hSearch = -1& Then > FileSizeAPI = -1& > Else > FindClose hSearch > FileSizeAPI = LargeInteger(WFD.nFileSizeLow, WFD.nFileSizeHigh) > End If > > End Function > > Private Function LargeInteger(ByVal LoPart As Long, _ > ByVal HiPart As Long) As Variant > > Dim nResult As Currency > > 'Copy the parts, appropriately. > CopyMemory ByVal VarPtr(nResult), LoPart, 4 > CopyMemory ByVal VarPtr(nResult) + 4, HiPart, 4 > > 'Return the result as VarType Decimal(14). > LargeInteger = nResult * CDec(10000) > > End Function > > > Just one thing I don't get and that is that I thought I could do this > (modification suggested by Peter Thornton) > to gain some speed: > > If WFD.nFileSizeHigh Then > FileSizeAPI = LargeInteger(WFD.nFileSizeLow, WFD.nFileSizeHigh) > Else > FileSizeAPI = WFD.nFileSizeLow > End If > > But that fails now on large files, giving negative figures due to > WFD.nFileSizeHigh = 0 > Any idea what I might be doing wrong there? You haven't accounted for the sign bit. You may be able to optimize it, but you'd need to add 2^32 if it's <0. In the end, the extra tests and math may not pay off. One possible compromise might be: If (WFD.nFileSizeHigh = 0) And (WFD.nFileSizeLow > 0) Then FileSizeAPI = WFD.nFileSizeLow Else FileSizeAPI = LargeInteger(WFD.nFileSizeLow, WFD.nFileSizeHigh) End If -- ..NET: It's About Trust! http://vfred.mvps.org
From: RB Smissaert on 17 Nov 2009 16:49
One possible compromise might be: If (WFD.nFileSizeHigh = 0) And (WFD.nFileSizeLow > 0) Then FileSizeAPI = WFD.nFileSizeLow Else FileSizeAPI = LargeInteger(WFD.nFileSizeLow, WFD.nFileSizeHigh) End If Yes, that does knock a few percent off. Actually, let's make that: If (WFD.nFileSizeHigh = 0&) And (WFD.nFileSizeLow > 0&) Then Looks we may have the perfect replacement then for FileLen ... RBS "Karl E. Peterson" <karl(a)exmvps.org> wrote in message news:%23y%23xVy8ZKHA.2160(a)TK2MSFTNGP02.phx.gbl... > RB Smissaert wrote: >> Usually this will be used just to get the size of one file, so speed is >> not >> an issue, but >> I also use it in a loop to read many files, so the faster the better. >> This is the optimized version then using your suggestions and it is some >> 15% >> faster than the posted code: > > Nice! > >> >> Private Declare Sub CopyMemory _ >> Lib "kernel32" _ >> Alias "RtlMoveMemory" (Dest As Any, _ >> Src As Any, _ >> ByVal cBytes As Long) >> >> Private Declare Function FindFirstFile Lib "kernel32" _ >> Alias "FindFirstFileW" _ >> (ByVal lpFileName As Long, _ >> lpFindFileData As WIN32_FIND_DATA) As Long >> >> Private Declare Function FindClose Lib "kernel32" _ >> (ByVal hFindFile As Long) As Long >> >> Private Const MAX_PATH = 260 >> >> Private Type FILETIME >> dwLowDateTime As Long >> dwHighDateTime As Long >> End Type >> >> Private Type WIN32_FIND_DATA >> dwFileAttributes As Long >> ftCreationTime As FILETIME >> ftLastAccessTime As FILETIME >> ftLastWriteTime As FILETIME >> nFileSizeHigh As Long >> nFileSizeLow As Long >> dwReserved0 As Long >> dwReserved1 As Long >> cFileName(0 To (MAX_PATH * 2) - 1) As Byte >> cAlternate(0 To 27) As Byte >> End Type >> >> Private Function FileSizeAPI(strFilePath As String) As Variant >> >> Dim hSearch As Long >> Dim WFD As WIN32_FIND_DATA >> >> hSearch = FindFirstFile(StrPtr(strFilePath), WFD) >> >> If hSearch = -1& Then >> FileSizeAPI = -1& >> Else >> FindClose hSearch >> FileSizeAPI = LargeInteger(WFD.nFileSizeLow, WFD.nFileSizeHigh) >> End If >> >> End Function >> >> Private Function LargeInteger(ByVal LoPart As Long, _ >> ByVal HiPart As Long) As Variant >> >> Dim nResult As Currency >> >> 'Copy the parts, appropriately. >> CopyMemory ByVal VarPtr(nResult), LoPart, 4 >> CopyMemory ByVal VarPtr(nResult) + 4, HiPart, 4 >> >> 'Return the result as VarType Decimal(14). >> LargeInteger = nResult * CDec(10000) >> >> End Function >> >> >> Just one thing I don't get and that is that I thought I could do this >> (modification suggested by Peter Thornton) >> to gain some speed: >> >> If WFD.nFileSizeHigh Then >> FileSizeAPI = LargeInteger(WFD.nFileSizeLow, WFD.nFileSizeHigh) >> Else >> FileSizeAPI = WFD.nFileSizeLow >> End If >> >> But that fails now on large files, giving negative figures due to >> WFD.nFileSizeHigh = 0 >> Any idea what I might be doing wrong there? > > You haven't accounted for the sign bit. You may be able to optimize it, > but you'd need to add 2^32 if it's <0. In the end, the extra tests and > math may not pay off. One possible compromise might be: > > If (WFD.nFileSizeHigh = 0) And (WFD.nFileSizeLow > 0) Then > FileSizeAPI = WFD.nFileSizeLow > Else > FileSizeAPI = LargeInteger(WFD.nFileSizeLow, WFD.nFileSizeHigh) > End If > > -- > .NET: It's About Trust! > http://vfred.mvps.org > |