Prev: NtDisplayString is not able to display unicode string
Next: Open Handle in one Process and Close in another?
From: Motonari Ito on 26 Feb 2010 10:32
Hi Alexander and Pavel,

The FILE_FLAG_SEQUENTIAL_SCAN flag seems to still use the file system
cache at least in my environment. Actually, the 3rd party library
(QuickTime) uses the flag already.

The idea of Detours is interesting. I will study the way to add FFNB
to CreateFile and also modify some other file I/O functions to
workaround the alignment constraints.

Thank you.

On Feb 25, 7:25 pm, "Alexander Grigoriev" <al...(a)earthlink.net> wrote:
> Welcome to windows file cache hell. I was hoping the cache bloat problem was
> fixed, but apparently I was wrong.
>
> You could use something like Detours (which works in user mode) to intercept
> CreateFile call, then add FILE_FLAG_SEQUENTIAL_SCAN to dwFlagsAndAttributes.
> I don't recommend setting FILE_FLAG_NO_BUFFERING, because it places
> additional constraints on the arguments of ReadFile/SetFilePosition.
>
> "Motonari Ito" <motonari....(a)gmail.com> wrote in message
>
> news:5bdd3cf4-c364-453f-9af1-97722e2860b6(a)b36g2000pri.googlegroups.com...
>
> >I have a 3rd party library which reads a huge media file (7GB+).
> > Obviously the library uses ReadFile() Win32 API function without
> > FILE_FLAG_NO_BUFFERING flag and the size of the file system cache is
> > increased while the file is being read.
>
> > This behavior invalidates any valuable data existing in the file
> > system cache. After the file reading completes, the whole system is
> > slowed down for a while. The problem doesn't happen when the media
> > file is located on remote machine because the file system cache is not
> > used for remote files.
>
> > I don't have an access to the source code of the library (In fact, the
> > library is Apple Quick Time). Is there any way to disable the use of
> > file system cache for a particular file? I start to think writing a
> > file system driver would be the only option, but I want to confirm
> > before jumping into that.
>
> > Environment:
> > RAM: 8GB
> > OS: Windows 7 64bits edition
>
> > Motonari Ito

From: Motonari Ito on 26 Feb 2010 10:46
HI Pavel,

> Or... why not just report this problem to Apple and let them sort it out?

It's interesting idea. I somehow didn't think about that...

Thank you for your suggestion.

On Feb 26, 6:53 am, "Pavel A." <pave...(a)12fastmail34.fm> wrote:
> "Alexander Grigoriev" <al...(a)earthlink.net> wrote in message
>
> news:eomFGKvtKHA.732(a)TK2MSFTNGP06.phx.gbl...
>
> > FFNB may not work because of alignment constraints.
>
> You're right. Then this needs a more sophisticated  patch, with
> intermediate buffer.
>
> Or... why not just report this problem to Apple and let them sort it out?
> --pa
>
>
>
> > "Pavel A." <pave...(a)12fastmail34.fm> wrote in message
> >news:C1EA6D79-992E-4FCF-862B-B9F4B384CFE4(a)microsoft.com...
> >> Try to hook ReadFile() in usermode and set FILE_FLAG_NO_BUFFERING, using
> >> something like Detours or VMware ThinApp?
> >> --pa
>
> >> "Motonari Ito" <motonari....(a)gmail.com> wrote in message
> >>news:5bdd3cf4-c364-453f-9af1-97722e2860b6(a)b36g2000pri.googlegroups.com....
> >>> I have a 3rd party library which reads a huge media file (7GB+).
> >>> Obviously the library uses ReadFile() Win32 API function without
> >>> FILE_FLAG_NO_BUFFERING flag and the size of the file system cache is
> >>> increased while the file is being read.
>
> >>> This behavior invalidates any valuable data existing in the file
> >>> system cache. After the file reading completes, the whole system is
> >>> slowed down for a while. The problem doesn't happen when the media
> >>> file is located on remote machine because the file system cache is not
> >>> used for remote files.
>
> >>> I don't have an access to the source code of the library (In fact, the
> >>> library is Apple Quick Time). Is there any way to disable the use of
> >>> file system cache for a particular file? I start to think writing a
> >>> file system driver would be the only option, but I want to confirm
> >>> before jumping into that.
>
> >>> Environment:
> >>> RAM: 8GB
> >>> OS: Windows 7 64bits edition
>
> >>> Motonari Ito

From: Jonathan de Boyne Pollard on 26 Feb 2010 18:15
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
<blockquote
cite="mid:21a20133-afbb-46ae-88ae-ae9f5069a8d5(a)15g2000yqi.googlegroups.com"
type="cite">
<p>The <code>FILE_FLAG_SEQUENTIAL_SCAN</code> flag seems to still
use the file system cache at least in my environment. </p>
</blockquote>
<p>Of course it does.&nbsp; Think about what the flag means.&nbsp; It's telling
the cache about the expected usage pattern of the file, so that the
cache can tailor its behaviour accordingly.&nbsp; It wouldn't make any sense
to have the flag without the cache.&nbsp; There's even a Microsoft
KnowledgeBase article, 98756, explaining the operation of the cache in
response to this flag, in detail.&nbsp; Note from the explanation that this
flag doesn't actually do what you want to do (which is to flush already
read data pages from the cache maps).<br>
</p>
</body>
</html>
From: Liviu on 26 Feb 2010 20:27
Just curious, but why do you reply in HTML to text posts, and use
strange indentation with no attribution for the quoted parts?

> "Jonathan de Boyne Pollard" <J.deBoynePollard-newsgroups(a)NTLWorld.COM>
> wrote...
>
>> The FILE_FLAG_SEQUENTIAL_SCAN flag seems to still use the file
>> system cache at least in my environment.
>
> Of course it does. [...] There's even a Microsoft KnowledgeBase
> article, 98756, explaining the operation of the cache in response to
> this flag, in detail. Note from the explanation that this flag
> doesn't actually do what you want to do (which is to flush already
> read data pages from the cache maps).

The http://support.microsoft.com/kb/98756 you mention does seem to
imply that FILE_FLAG_SEQUENTIAL_SCAN caches forward, only.

|| There is a minor savings because the Cache Manager dispenses
|| with keeping a history of reads on the file, and tries to maintain a
|| high-water mark on read ahead, which is always a certain delta
|| from the most recent read.

How else do you read "dispenses with keeping a history of reads"?

Liviu




From: Hugh Moran on 26 Feb 2010 21:11


"Motonari Ito" wrote:

> I have a 3rd party library which reads a huge media file (7GB+).
> Obviously the library uses ReadFile() Win32 API function without
> FILE_FLAG_NO_BUFFERING flag and the size of the file system cache is
> increased while the file is being read.
>
> This behavior invalidates any valuable data existing in the file
> system cache. After the file reading completes, the whole system is
> slowed down for a while. The problem doesn't happen when the media
> file is located on remote machine because the file system cache is not
> used for remote files.
>
> I don't have an access to the source code of the library (In fact, the
> library is Apple Quick Time). Is there any way to disable the use of
> file system cache for a particular file? I start to think writing a
> file system driver would be the only option, but I want to confirm
> before jumping into that.
>
> Environment:
> RAM: 8GB
> OS: Windows 7 64bits edition
>
> Motonari Ito
> .
>

Get the source code or write a new app, this amount of disk based data
should be accessed using a file mapping.

Hugh

First  |  Prev  |  Next  |  Last
Pages: 1 2 3 4
Prev: NtDisplayString is not able to display unicode string
Next: Open Handle in one Process and Close in another?