MmAllocateContiguousMemory and non-paged pool size
in [Device Drivers]
|
From: Maxim S. Shatskih on 5 Mar 2010 09:01 > the huge megabytes are for packet history, the larger the history the better > the compression. This is true for PPM algorithms only (all other algos are fine with <1M buffer), and sorry, but PPM is so slow, so that offloading to worker thread will not give you any significant loss. Also, I cannot understand what are you trying to implement. Is is an implementation of some RFC, or some design of your own? If the latter - then I would all WinSock LSP in user mode which will compress all per-socket data flow. Much simpler. Probably even not a LSP, but an explicit tunnel tool like ZeeBeeDee. This would be the simplest solution. Deep compression on _network_ level is a major problem. For instance, some packet can disappear _forever_, and the retransmitted packet will not save you - it is different. Also, the packets can be reordered, and so on. -- Maxim S. Shatskih Windows DDK MVP maxim(a)storagecraft.com http://www.storagecraft.com
From: Don Burn on 5 Mar 2010 09:06 So you think it is acceptable to take most of the memory that can be allocated in the kernel for your driver to keep packet history? Remember, in most systems out there you can have 128MB total of non-paged pool and 256MB of paged pool. Consider that XP still runs on a lot of systems with 512MB total. Even if the systems have a ton of memory you are taking a large portion on 32-bit of the address space of the kernel since that is 2GB. So no matter how you look at this your design is flawed. Don Burn (MVP, Windows DKD) Windows Filesystem and Driver Consulting Website: http://www.windrvr.com Blog: http://msmvps.com/blogs/WinDrvr > -----Original Message----- > From: mirage2k2 [mailto:mirage2k2(a)discussions.microsoft.com] > Posted At: Friday, March 05, 2010 7:16 AM > Posted To: microsoft.public.development.device.drivers > Conversation: MmAllocateContiguousMemory and non-paged pool size > Subject: Re: MmAllocateContiguousMemory and non-paged pool size > > the huge megabytes are for packet history, the larger the history the > better the compression. Anyway, thanks for all your help. I now do > this and it > works ok ... > > PMDL mdl; > UCHAR *testPagedMem = ( > (UCHAR *)ExAllocatePoolWithTag(PagedPool, MEM_SIZE, 'aaa1') ); if > (testPagedMem == 0) { > return; > } > > mdl = IoAllocateMdl(testPagedMem, MEM_SIZE, FALSE, FALSE, 0); if (mdl > == 0) { > return; > } > > MmProbeAndLockPages(mdl, KernelMode, IoWriteAccess); > > Mirage2k > > "Maxim S. Shatskih" wrote: > > > >I need to decompress a compressed inbound packet in my receive > packet > > >handler > > > > Then why do you need these huge megabytes? > > > > Allocate the space enough for, say, 100 packets - i.e. 150KB. > > > > -- > > Maxim S. Shatskih > > Windows DDK MVP > > maxim(a)storagecraft.com > > http://www.storagecraft.com > > > > . > > > > > __________ Information from ESET Smart Security, version of virus > signature database 4917 (20100305) __________ > > The message was checked by ESET Smart Security. > > http://www.eset.com >
From: mirage2k2 on 5 Mar 2010 15:32 Firstly, I do not need 100's of MB, I need 32-64MB. Secondly, it is not my design, I have been provided with a compression/decompression library which requires me to allocate memory for packet history (1 history buffer for compressing outbound packets and 1 for decompressing inbound packets). I do not think it is acceptable for my driver to use a large percentage of available resources ... and this is why I've posted ... I'm looking for alternatives to non-paged pool. What I will probably do is limit my history buffer to about 16MB on xp 32-bit. I might allocate 1 history buffer from non-paged pool and the other from paged pool ... one buffer can typically be larger than the other ... the biggest one will come from paged pool. I will try to limit total allocation (on xp 32-bit) to about 32MB. Mirage2k2. "Don Burn" wrote: > So you think it is acceptable to take most of the memory that can be > allocated in the kernel for your driver to keep packet history? Remember, > in most systems out there you can have 128MB total of non-paged pool and > 256MB of paged pool. Consider that XP still runs on a lot of systems with > 512MB total. Even if the systems have a ton of memory you are taking a > large portion on 32-bit of the address space of the kernel since that is > 2GB. > > So no matter how you look at this your design is flawed. > > > Don Burn (MVP, Windows DKD) > Windows Filesystem and Driver Consulting > Website: http://www.windrvr.com > Blog: http://msmvps.com/blogs/WinDrvr > > > > > -----Original Message----- > > From: mirage2k2 [mailto:mirage2k2(a)discussions.microsoft.com] > > Posted At: Friday, March 05, 2010 7:16 AM > > Posted To: microsoft.public.development.device.drivers > > Conversation: MmAllocateContiguousMemory and non-paged pool size > > Subject: Re: MmAllocateContiguousMemory and non-paged pool size > > > > the huge megabytes are for packet history, the larger the history the > > better the compression. Anyway, thanks for all your help. I now do > > this and it > > works ok ... > > > > PMDL mdl; > > UCHAR *testPagedMem = ( > > (UCHAR *)ExAllocatePoolWithTag(PagedPool, MEM_SIZE, 'aaa1') ); if > > (testPagedMem == 0) { > > return; > > } > > > > mdl = IoAllocateMdl(testPagedMem, MEM_SIZE, FALSE, FALSE, 0); if (mdl > > == 0) { > > return; > > } > > > > MmProbeAndLockPages(mdl, KernelMode, IoWriteAccess); > > > > Mirage2k > > > > "Maxim S. Shatskih" wrote: > > > > > >I need to decompress a compressed inbound packet in my receive > > packet > > > >handler > > > > > > Then why do you need these huge megabytes? > > > > > > Allocate the space enough for, say, 100 packets - i.e. 150KB. > > > > > > -- > > > Maxim S. Shatskih > > > Windows DDK MVP > > > maxim(a)storagecraft.com > > > http://www.storagecraft.com > > > > > > . > > > > > > > > > __________ Information from ESET Smart Security, version of virus > > signature database 4917 (20100305) __________ > > > > The message was checked by ESET Smart Security. > > > > http://www.eset.com > > > > . >
From: Maxim S. Shatskih on 5 Mar 2010 15:50 > Firstly, I do not need 100's of MB, I need 32-64MB. Secondly, it is not my > design, I have been provided with a compression/decompression library which > requires me to allocate memory for packet history (1 history buffer for > compressing outbound packets and 1 for decompressing inbound packets). This does not work. Packet history is unreliable. Packets are often lost on the network (due to router overflow usually) and can be reordered. And, if you're compressing, then surely offloading to a thread (in a paged pool) will not be so large a burden. -- Maxim S. Shatskih Windows DDK MVP maxim(a)storagecraft.com http://www.storagecraft.com
From: Don Burn on 5 Mar 2010 16:11 Whether you were provided with the library or not, you are using it so it is your design. In fields with professional engineering credentials it is the person who is responsible for the delivery whose design it is. Yes it may have come from a manager's requirements, but it is your responsibility to indicate the design has a problem and the driver will not conform to what most people expect in Windows. If the driver is for an embedded type of situation where the environment is controlled this needs to communicated so a decision can be made if it is acceptable. If the driver is for general purpose, then it is almost sure to be unacceptable since you are going to at a minimum significantly degrade the performance of the system. Don Burn (MVP, Windows DKD) Windows Filesystem and Driver Consulting Website: http://www.windrvr.com Blog: http://msmvps.com/blogs/WinDrvr > -----Original Message----- > From: mirage2k2 [mailto:mirage2k2(a)discussions.microsoft.com] > Posted At: Friday, March 05, 2010 3:32 PM > Posted To: microsoft.public.development.device.drivers > Conversation: MmAllocateContiguousMemory and non-paged pool size > Subject: Re: MmAllocateContiguousMemory and non-paged pool size > > Firstly, I do not need 100's of MB, I need 32-64MB. Secondly, it is > not my design, I have been provided with a compression/decompression > library which requires me to allocate memory for packet history (1 > history buffer for compressing outbound packets and 1 for decompressing > inbound packets). > > I do not think it is acceptable for my driver to use a large percentage > of available resources ... and this is why I've posted ... I'm looking > for alternatives to non-paged pool. > > What I will probably do is limit my history buffer to about 16MB on xp > 32-bit. I might allocate 1 history buffer from non-paged pool and the > other from paged pool ... one buffer can typically be larger than the > other ... the biggest one will come from paged pool. I will try to > limit total allocation (on xp 32-bit) to about 32MB. > > Mirage2k2. > > "Don Burn" wrote: > > > So you think it is acceptable to take most of the memory that can be > > allocated in the kernel for your driver to keep packet history? > > Remember, in most systems out there you can have 128MB total of > > non-paged pool and 256MB of paged pool. Consider that XP still runs > on a lot of systems with > > 512MB total. Even if the systems have a ton of memory you are > taking a > > large portion on 32-bit of the address space of the kernel since that > > is 2GB. > > > > So no matter how you look at this your design is flawed. > > > > > > Don Burn (MVP, Windows DKD) > > Windows Filesystem and Driver Consulting > > Website: http://www.windrvr.com > > Blog: http://msmvps.com/blogs/WinDrvr > > > > > > > > > -----Original Message----- > > > From: mirage2k2 [mailto:mirage2k2(a)discussions.microsoft.com] > > > Posted At: Friday, March 05, 2010 7:16 AM Posted To: > > > microsoft.public.development.device.drivers > > > Conversation: MmAllocateContiguousMemory and non-paged pool size > > > Subject: Re: MmAllocateContiguousMemory and non-paged pool size > > > > > > the huge megabytes are for packet history, the larger the history > > > the better the compression. Anyway, thanks for all your help. I > > > now do this and it works ok ... > > > > > > PMDL mdl; > > > UCHAR *testPagedMem = ( > > > (UCHAR *)ExAllocatePoolWithTag(PagedPool, MEM_SIZE, 'aaa1') ); if > > > (testPagedMem == 0) { > > > return; > > > } > > > > > > mdl = IoAllocateMdl(testPagedMem, MEM_SIZE, FALSE, FALSE, 0); if > > > (mdl == 0) { > > > return; > > > } > > > > > > MmProbeAndLockPages(mdl, KernelMode, IoWriteAccess); > > > > > > Mirage2k > > > > > > "Maxim S. Shatskih" wrote: > > > > > > > >I need to decompress a compressed inbound packet in my receive > > > packet > > > > >handler > > > > > > > > Then why do you need these huge megabytes? > > > > > > > > Allocate the space enough for, say, 100 packets - i.e. 150KB. > > > > > > > > -- > > > > Maxim S. Shatskih > > > > Windows DDK MVP > > > > maxim(a)storagecraft.com > > > > http://www.storagecraft.com > > > > > > > > . > > > > > > > > > > > > > __________ Information from ESET Smart Security, version of virus > > > signature database 4917 (20100305) __________ > > > > > > The message was checked by ESET Smart Security. > > > > > > http://www.eset.com > > > > > > > . > > > > > __________ Information from ESET Smart Security, version of virus > signature database 4919 (20100305) __________ > > The message was checked by ESET Smart Security. > > http://www.eset.com >
First
|
Prev
|
Next
|
Last
Pages: 1 2 3 4 Prev: driver installation failed in xp with 7600.16385.0 build environme Next: kernel base address |