Prev: IEEE1284 / USB converter
Next: Sending an IPI to a target CPU
From: Le Chaud Lapin on 22 Oct 2009 15:04
On Oct 22, 12:51 pm, "Chris M. Thomasson" <n...(a)spam.invalid> wrote:
> "Le Chaud Lapin" <jaibudu...(a)gmail.com> wrote in messagenews:74647255-58cc-4a84-b8af-bdc6a93e166a(a)d4g2000vbm.googlegroups.com...
> > Hi All,
>
> > I doth seek to remain a sloth...
>
> > I have an application that does a bunch of ReadFile's against a 470MB
> > file. I have to read the entire file from end to end, always
> > sequentially. Each read can be anywhere from a few bytes to several
> > kilobytes. Process Explorer shows roughly 41,000,000 reads during an 8-
> > minute run.  Mean is roughly 50 bytes, with stand-dev I'm guessing
> > maybe 10 bytes.
>
> Use a memory mapped file, and process the memory in sequential order (e.g..,
> base to base + size_of_file). BTW, what type of processing are you doing?
> Does processing of one part of the file always depend on the processing
> results of a previous portion of the file?

Yes, this should work.

I am doing serialization to/from the file. I did some tests this
morning to make sure that the storage subsystem could keep up, and it
does. I do 70MB user-mode read 0.22 seconds, so I started thinking
about overall architecture and realized that 41,000,000 true ReadFiles
is likely the killer.

Thanks to all,

-Le Chaud Lapin-
From: Alexander Grigoriev on 22 Oct 2009 22:36
One MDL can only map a little below 64MB on x86 sstem and less than 32MB on
x64 system.

If an IO doesn't fit to one MDL, things get more complicated.

"Paul Baker [MVP, Windows Desktop Experience]"
<paulrichardbaker(a)community.nospam> wrote in message
news:eBHQYgxUKHA.5584(a)TK2MSFTNGP05.phx.gbl...
>
> I never saw any improvement under any circumstances with a buffer over 64
> KB. Don't make the mistake of using a huge buffer in a single call to
> ReadFile. This could actually negatively impact performance as well as
> system resources and stability. We saw this problem recently. On some
> systems I could use a buffer of hundreds of megabytes without issue,
> whereas on others I could not use a buffer size over about 64 MB. After
> digging through low level documentation, I am unable to explain it fully,
> but it is something to do with the fact that the device driver needs
> contiguous physical memory and depending on the type of buffer management
> used, may allocate a new buffer equal in size to the callers. And it may
> be from kernel memory (nonpaged pool), which is a scarce resource.
>


From: Pavel Lebedinsky [MSFT] on 23 Oct 2009 00:35
> One MDL can only map a little below 64MB on x86 sstem and less than
> 32MB on x64 system.

That's an artificial limitation of IoAllocateMdl (which was removed in
Vista,
by the way). MDLs created manually (ExAllocatePool+MmInitializeMdl)
or using APIs like MmAllocatePagesForMdlEx can describe up to 4 GB
worth of pages.

--
Pavel Lebedinsky/Windows Kernel Test
This posting is provided "AS IS" with no warranties, and confers no rights.


From: Pavel Lebedinsky [MSFT] on 23 Oct 2009 00:47
> I never saw any improvement under any circumstances with a buffer
> over 64 KB. Don't make the mistake of using a huge buffer in a single
> call to ReadFile. This could actually negatively impact performance
> as well as system resources and stability.

Using sizes larger than 64 KB can definitely improve performance,
especially for non-buffered IO. In the buffered case, the size of the actual
IOs that hit the disk will be determined by the cache manager, but it might
still be beneficial to use larger chunks.

Something like 1 MB is probably a good starting point.

--
Pavel Lebedinsky/Windows Kernel Test
This posting is provided "AS IS" with no warranties, and confers no rights.


From: Paul Baker [MVP, Windows Desktop Experience] on 23 Oct 2009 16:36
When I saw a buffer size well over 64 MB working, it was Windows XP, on
which a MDL can only map a little under 64 MB. So it could not have been
using a MDL in this case. I observed it allocating a buffer in physical
memory equal in size to the caller's buffer. Apparently, kernel memory was
not needed.

On the machine that had a limited buffer size, the error was
ERROR_NO_SYSTEM_RESOURCES. I guessed it was trying to use the kernel
nonpaged pool and allocating a MDL. But I didn't know why it chose to do
that. I wondered if it did not have a contiguous block of physical memory.
It couldn't be memory fragmentation, because it happened right after reboot
as well.

Paul

"Pavel Lebedinsky [MSFT]" <pavel(a)online.microsoft.com> wrote in message
news:%23IXryo5UKHA.5208(a)TK2MSFTNGP05.phx.gbl...
>> One MDL can only map a little below 64MB on x86 sstem and less than
>> 32MB on x64 system.
>
> That's an artificial limitation of IoAllocateMdl (which was removed in
> Vista,
> by the way). MDLs created manually (ExAllocatePool+MmInitializeMdl)
> or using APIs like MmAllocatePagesForMdlEx can describe up to 4 GB
> worth of pages.
>
> --
> Pavel Lebedinsky/Windows Kernel Test
> This posting is provided "AS IS" with no warranties, and confers no
> rights.
>
>


First  |  Prev  |  Next  |  Last
Pages: 1 2 3
Prev: IEEE1284 / USB converter
Next: Sending an IPI to a target CPU