MmAllocateContiguousMemory and non-paged pool size
in [Device Drivers]
|
From: Don Burn on 4 Mar 2010 07:49 MmAllocateContiguousMemory will fail because the physical memory needs to be Contiguous! So the odds of getting a contiguous block of memory of size X are much higher than getting size X random pages. You should never go near MmAllocateContiguousMemory unless your adapter needs it, because if you do you are taking memory the next device may need. On the non-paged versus paged on Windows 7 it won't create a difference, on earlier OS'es there are limits to the pools. If you need to go that extreme look at other approaches. Don Burn (MVP, Windows DKD) Windows Filesystem and Driver Consulting Website: http://www.windrvr.com Blog: http://msmvps.com/blogs/WinDrvr > -----Original Message----- > From: mirage2k2 [mailto:mirage2k2(a)discussions.microsoft.com] > Posted At: Thursday, March 04, 2010 7:33 AM > Posted To: microsoft.public.development.device.drivers > Conversation: MmAllocateContiguousMemory and non-paged pool size > Subject: Re: MmAllocateContiguousMemory and non-paged pool size > > The help for MmAllocateContiguousMemory says that it will try non-paged > pool first and if the allocation fails then it will allocate from > unused pages (physical I guess). So why do you think that > MmAllocateContiguousMemory could fail in a situation where > NdisAllocateMemoryWithTag would not? > > My requirement is large allocations of memory that cannot be paged out, > so if MmAllocateContiguousMemory can get me these allocations what is > there for me to worry about? > > Is there another alternative ... can I allocate from paged memory and > then call some memory manager function to get the memory paged in and > locked in? > > Mirage2k2 > > "Maxim S. Shatskih" wrote: > > > > To be honest, I don't really know the difference between physically > > > contiguous and virtually contiguous ... > > > > Any allocation call (except MmAllocatePagesForMdl which does not > allocate virtual addresses at all) will give you virtually contiguous > memory. > > > > Physically contiguous means that the underlying physical pages are > also contiguous. This is only important for DMA, since no other code > (except DMA and MM's internals) even care about the underlying physical > addresses. > > > > And yes, MmAllocateContiguousMemory is provided for _DMA adapter > object implementors_ to implement ->AllocateCommonBuffer. The DMA- > capable drivers must use ->AllocateCommonBuffer. > > > > >I guess virtually contiguous is > > > fragmented blocks that are somehow made to look like one big block. > > > > Do you know what are physical and virtual addresses? > > > > Virtual addresses are the pointers, or address values in the machine > code. > > > > Physical addresses are the addresses set to the physical CPU's > addressing bus. > > > > The CPU has the page tables to perform transparent automatic > translations virtual -> physical. These page tables are maintained by > the OS's MM, and there are APIs which you can use to govern this > translation. > > > > This is how per-process address spaces are implemented in the OS, and > also some other things like > MmMapLockedPages/MmGetSystemAddressForMdlSafe. > > > > > Anyway, I'm using the memory as a packet data history buffer ... I > > > use offsets into it, I search for recurring patterns in it, I might > > > also need to iterate all the way through it (by incrementing a > UCHAR pointer) ... > > > > Bad idea. Implement a parser state machine instead - like the GNU > grep or sed does (see the sources). > > > > > this sounds like it needs to be physically contiguous. > > > > No. > > > > -- > > Maxim S. Shatskih > > Windows DDK MVP > > maxim(a)storagecraft.com > > http://www.storagecraft.com > > > > . > > > > > __________ Information from ESET Smart Security, version of virus > signature database 4914 (20100304) __________ > > The message was checked by ESET Smart Security. > > http://www.eset.com >
From: Scott Noone on 4 Mar 2010 11:37 > My requirement is large allocations of memory that cannot be paged out, so > if MmAllocateContiguousMemory can get me these allocations what is there > for > me to worry about? The fact that you're asking for physically contiguous memory, which is more likely to fail. > Is there another alternative ... can I allocate from paged memory and then > call some memory manager function to get the memory paged in and locked > in? Allocating from paged pool and locking the memory down via an MDL works (IoAllocateMdl, MmProbeAndLockPages). You might also want to look into MmAllocatePagesForMdl and then mapping the MDL in smaller chunks as you need them. That would on the surface appear to be a more conservative approach since you wouldn't be hammering the general use pools and you wouldn't be asking the Mm for contiguous memory. You would have to worry about the resources consumed by mapping the MDL into the kernel virtual address space though (system PTEs, namely) and the time needed to build/teardown the mappings (which can be expensive if you're doing it often). In the end, you're going to need to strike a nice balance between performance and resource consumption. This is the sort of thing that makes driver writing hard (and fun!). -scott -- Scott Noone Consulting Associate OSR Open Systems Resources, Inc. http://www.osronline.com "mirage2k2" <mirage2k2(a)discussions.microsoft.com> wrote in message news:63B24366-D0DB-4C41-9C72-D1278B4049A4(a)microsoft.com... > The help for MmAllocateContiguousMemory says that it will try non-paged > pool > first and if the allocation fails then it will allocate from unused pages > (physical I guess). So why do you think that MmAllocateContiguousMemory > could fail in a situation where NdisAllocateMemoryWithTag would not? > > My requirement is large allocations of memory that cannot be paged out, so > if MmAllocateContiguousMemory can get me these allocations what is there > for > me to worry about? > > Is there another alternative ... can I allocate from paged memory and then > call some memory manager function to get the memory paged in and locked > in? > > Mirage2k2 > > "Maxim S. Shatskih" wrote: > >> > To be honest, I don't really know the difference between physically >> > contiguous and virtually contiguous ... >> >> Any allocation call (except MmAllocatePagesForMdl which does not allocate >> virtual addresses at all) will give you virtually contiguous memory. >> >> Physically contiguous means that the underlying physical pages are also >> contiguous. This is only important for DMA, since no other code (except >> DMA and MM's internals) even care about the underlying physical >> addresses. >> >> And yes, MmAllocateContiguousMemory is provided for _DMA adapter object >> implementors_ to implement ->AllocateCommonBuffer. The DMA-capable >> drivers must use ->AllocateCommonBuffer. >> >> >I guess virtually contiguous is >> > fragmented blocks that are somehow made to look like one big block. >> >> Do you know what are physical and virtual addresses? >> >> Virtual addresses are the pointers, or address values in the machine >> code. >> >> Physical addresses are the addresses set to the physical CPU's addressing >> bus. >> >> The CPU has the page tables to perform transparent automatic translations >> virtual -> physical. These page tables are maintained by the OS's MM, and >> there are APIs which you can use to govern this translation. >> >> This is how per-process address spaces are implemented in the OS, and >> also some other things like >> MmMapLockedPages/MmGetSystemAddressForMdlSafe. >> >> > Anyway, I'm using the memory as a packet data history buffer ... I use >> > offsets into it, I search for recurring patterns in it, I might also >> > need to >> > iterate all the way through it (by incrementing a UCHAR pointer) ... >> >> Bad idea. Implement a parser state machine instead - like the GNU grep or >> sed does (see the sources). >> >> > this sounds like it needs to be physically contiguous. >> >> No. >> >> -- >> Maxim S. Shatskih >> Windows DDK MVP >> maxim(a)storagecraft.com >> http://www.storagecraft.com >> >> . >>
From: Maxim S. Shatskih on 4 Mar 2010 14:18 > The help for MmAllocateContiguousMemory says that it will try non-paged pool > first and if the allocation fails then it will allocate from unused pages > (physical I guess). Nonpaged pool itself will try unused pages :-) >So why do you think that MmAllocateContiguousMemory > could fail in a situation where NdisAllocateMemoryWithTag would not? Because NPP has relaxed requirement of not being physically contiguous. On a loaded machine, you can only call MmAllocateContiguousMemory successfully for a large amount _at boot_. > if MmAllocateContiguousMemory can get me these allocations It will not give you this under load. > Is there another alternative ... can I allocate from paged memory and then > call some memory manager function to get the memory paged in and locked in? Yes. -- Maxim S. Shatskih Windows DDK MVP maxim(a)storagecraft.com http://www.storagecraft.com
From: mirage2k2 on 4 Mar 2010 17:54 Thanks everybody for all your help. So it seems that the best option for me is to use paged memory and then use mm functions to page it in and lock it down so that it can never be paged out. Does anyone see a reason why this will not work? Also, is it definately ok to access paged memory at DISPATCH_LEVEL ... provided that it is paged in and locked down? Mirage2k2. "mirage2k2" wrote: > The documentation on MmAllocateContiguousMemory says that it attempts to > allocate memory from the non-paged pool, and if this fails then "it attempts > to perform the allocation from available unused pages". > > Firstly, what on earth does that mean? Secondly, will the memory "from > available unused pages" still be non-paged? I will be accessing it at > DISPATCH_LEVEL so I need to be certain. > > My guess is that the memory available in "unused pages" is somehow mapped > into the non-paged pool ... thus increasing the size of the pool. > > > My next question is about the size of the non-paged pool. I've read a few > articles that suggest that there is a limit of 250MB, irrespective of total > RAM. Does anyone know if there is a way of changing the configuration of > the system to make this value higher, i.e. some weird registry setting or > some system API call. > > Basically I need to allocate between 32MB - 64MB of non-paged memory, at any > given time, and I need it to work. Allocating the memory in DriverEntry at > startup is probably my best shot but there will be times when I don't need to > allocate anything at all ... I don't want to allocate 32MB and then never use > it ... or hog it until I do need it. > > Does anyone have any ideas? > Thanks > Mirage2k >
From: m on 4 Mar 2010 20:16 None at all - but the better question is why you want to scan a pattern buffer at DISPATCH! If you find something bad, then, since the upper level has already received some of it, your response, in general, has no different efficacy than if you were scanning on a passive thread / work item, but the scanning has much less impact on overall system perf. "mirage2k2" <mirage2k2(a)discussions.microsoft.com> wrote in message news:DA1CC7E1-1C50-4189-9326-053F9418CC84(a)microsoft.com... > Thanks everybody for all your help. So it seems that the best option for > me > is to use paged memory and then use mm functions to page it in and lock it > down so that it can never be paged out. Does anyone see a reason why this > will not work? Also, is it definately ok to access paged memory at > DISPATCH_LEVEL ... provided that it is paged in and locked down? > > Mirage2k2. > > "mirage2k2" wrote: > >> The documentation on MmAllocateContiguousMemory says that it attempts to >> allocate memory from the non-paged pool, and if this fails then "it >> attempts >> to perform the allocation from available unused pages". >> >> Firstly, what on earth does that mean? Secondly, will the memory "from >> available unused pages" still be non-paged? I will be accessing it at >> DISPATCH_LEVEL so I need to be certain. >> >> My guess is that the memory available in "unused pages" is somehow mapped >> into the non-paged pool ... thus increasing the size of the pool. >> >> >> My next question is about the size of the non-paged pool. I've read a >> few >> articles that suggest that there is a limit of 250MB, irrespective of >> total >> RAM. Does anyone know if there is a way of changing the configuration >> of >> the system to make this value higher, i.e. some weird registry setting or >> some system API call. >> >> Basically I need to allocate between 32MB - 64MB of non-paged memory, at >> any >> given time, and I need it to work. Allocating the memory in DriverEntry >> at >> startup is probably my best shot but there will be times when I don't >> need to >> allocate anything at all ... I don't want to allocate 32MB and then never >> use >> it ... or hog it until I do need it. >> >> Does anyone have any ideas? >> Thanks >> Mirage2k >>
|
Next
|
Last
Pages: 1 2 3 4 Prev: driver installation failed in xp with 7600.16385.0 build environme Next: kernel base address |