Disable file system cache for a certain file [Win32 Kernel]

Prev: NtDisplayString is not able to display unicode string
Next: Open Handle in one Process and Close in another?

From: Jonathan de Boyne Pollard on 26 Feb 2010 23:26

From: Liviu on 27 Feb 2010 01:15

Oh, the other thing, too... Please appreciate that I am well aware of
the comp.os.ms-windows.programmer.win32 newsgroup, and that I
have deliberately trimmed it off the reply list in my previous post,
since it wasn't copied there by the OP or any of the posters in this
thread other than yourself. Don't know why you added it right back.
IMHO you should stop these uninvited crossposts, even more so given
the rather peculiar way you cut off the relevant context.

Looking at it now, you account for almost a dozen broken threads in
comp.os.ms-windows.programmer.win32 for the past month or so.
I fail to see your point in doing this, if any at all.

"Jonathan de Boyne Pollard" <J.deBoynePollard-newsgroups(a)NTLWorld.COM>
wrote...
>>>> The FILE_FLAG_SEQUENTIAL_SCAN flag seems to still use the file
>>>> system cache at least in my environment.
>>>
>>> Of course it does. [...] There's even a Microsoft KnowledgeBase
>>> article, 98756, explaining the operation of the cache in response to
>>> this flag, in detail. Note from the explanation that this flag
>>> doesn't actually do what you want to do (which is to flush already
>>> read data pages from the cache maps).
>>
>> The http://support.microsoft.com/kb/98756 you mention does seem to
>> imply that FILE_FLAG_SEQUENTIAL_SCAN caches forward, only.
>>
>> || There is a minor savings because the Cache Manager dispenses
>> || with keeping a history of reads on the file, and tries to maintain
>> || a high-water mark on read ahead, which is always a certain delta
>> || from the most recent read.
>>
>> How else do you read "dispenses with keeping a history of reads"?
>
> One reads it properly. Dispensing with the history of reads means
> that the cache manager doesn't do what it normally does, which is
> attempt to auto-detect sequential read behaviour by comparing the
> current read request to prior requests, to see whether they are
> (roughly) sequential. It dispenses with keeping that history of reads.
> This is because, of course, it doesn't need it. It's already been
> told explicitly what it would otherwise auto-detect. Once again, note
> that this doesn't actually do what is wanted, and indeed is nothing to
> do with what is wanted, which isn't to do with any saved history of
> read requests, but is to do with the read data pages and the cache
> maps.

My reading of that MS KB paragraph is more along the line of "the
look-ahead cache is filled-in whenever the file pointer moves, and the
cached data is discarded as soon as it is actually requested/read by the
client application". If you have some authoritative source to support
your quite different interpretation of the same sentence, please provide
a pointer to such documentation.

Liviu

From: Hector Santos on 27 Feb 2010 08:45

Motonari Ito wrote:

> I have a 3rd party library which reads a huge media file (7GB+).
> Obviously the library uses ReadFile() Win32 API function without
> FILE_FLAG_NO_BUFFERING flag and the size of the file system cache is
> increased while the file is being read.
>
> This behavior invalidates any valuable data existing in the file
> system cache. After the file reading completes, the whole system is
> slowed down for a while. The problem doesn't happen when the media
> file is located on remote machine because the file system cache is not
> used for remote files.

It might not be a file system cache (which across the network, its a
function of the network driver), but the QT API does do buffering, in
fact, Apple has a frivolous patent pending feature called Skip
Protection. So you might want to see if QT API has a option to
disable Skip Protection or Instant On on the local machine where such
requirements are less.

--
HLS

From: Jonathan de Boyne Pollard on 27 Feb 2010 10:50

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
<blockquote cite="mid:O3oUiR3tKHA.4568(a)TK2MSFTNGP05.phx.gbl" type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
The <code>FILE_FLAG_SEQUENTIAL_SCAN</code> flag
seems to still use the file system cache at least in my environment. 

</blockquote>
Of course it does. [...] There's even a Microsoft
KnowledgeBase article, 98756, explaining the operation of the cache in
response to this flag, in detail. Note from the explanation that this
flag doesn't actually do what you want to do (which is to flush already
read data pages from the cache maps). 

</blockquote>
The [MSKB article] you mention does seem to imply that
<code>FILE_FLAG_SEQUENTIAL_SCAN</code> caches forward, only. 
 
How else do you read "dispenses with keeping a history of reads"? 

</blockquote>
One reads it properly. Dispensing with the
history of reads means that the cache manager doesn't do what it
normally does, which is attempt to auto-detect sequential read
behaviour by comparing the current read request to prior requests, to
see whether they are (roughly) sequential. It dispenses with keeping
that history of reads. This is because, of course, it doesn't
need it. It's already been told explicitly what it would otherwise
auto-detect. Once again, note that this doesn't actually do what is
wanted, and indeed is nothing to do with what is wanted, which isn't to
do with any saved history of read requests, but is to do with the read
data pages and the cache maps. 

</blockquote>
My reading of that MS KB paragraph is more along the line
of "the look-ahead cache is filled-in whenever the file pointer moves,
and the cached data is discarded as soon as it is actually
requested/read by the client application". If you have some
authoritative source to support your quite different interpretation of
the same sentence, please provide a pointer to such documentation. 

</blockquote>
Your reading is wrong; and it doesn't take much knowledge of how
mechanisms such as auto-detecting sequential read behaviour work to
figure this out from first principles.  There are only so many ways to
implement such a mechanism.  (This makes it most amusing that Compaq
Computer Corporation in 1998, some years after the creation of Windows
NT and OS/2, both of which have sequential I/O pattern detection and
API flags for processes to specify intended access patterns, filed a
U.S. patent claiming to have invented a method for detecting sequential
read patterns by keeping a history of read requests in a table.) You
can even figure it out from looking at the parameters to the <code>CcReadAhead()</code>
function.  You can also read plenty of documentation on the internals
of the cache manager in Windows NT that tells you this, including
Rajeev Nagar's book.  And this documentation tells you, as does the
MSKB article, that the <code>FO_SEQUENTIAL_ONLY</code> flag controls
read-ahead, not free-behind. 

</body>
</html>

From: Motonari Ito on 27 Feb 2010 13:00

Hi Hugh,

> Get the source code or write a new app, this amount of disk based data
> should be accessed using a file mapping.

My quick test shows file I/O through memory mapped file stills uses
system cache. I say so because the "Cached" physical memory value
reported by task manager is increased by the file size after the
entire file is read through file mapping.

Am I missing something?

Thank you.

First | Prev | Next | Last
Pages: 1 2 3 4
Prev: NtDisplayString is not able to display unicode string
Next: Open Handle in one Process and Close in another?