MemoryError, can I use more?
in [Python]
|
Prev: Configuring apache to execute python scripts using mod_pythonhandler
Next: Helpp I want view page .py on apache WHAT CAN I DO???????????????
From: Ross Ridge on 14 Feb 2010 17:05 "Diez B. Roggisch" <deets(a)nospam.web.de> writes: > No, PAE can be used to access much more memory than 4GB - albeit > through paging. AFAIK up to 2^36 Bytes. Anssi Saari <as(a)sci.fi> wrote: >That too. I admit, after checking, that you can't go above 3 GiB per >process even in server Windows. But for Linux there exists (or >existed, since it seems it hasn't been updated since 2004) a kernel >patch which provides a "4GB/4GB" address split. Kernel is in one >segment, userland in another and hence a process can access full 4GB. Windows has a similar feature that allows 32-bit applications running on 64-bit versions of Windows to have a nearly 4Gb virtual address space. Windows also allows 32-bit applications to use more than 4GB of physical memory through a paging mechanism called "Address Windowing Extensions". Also 32-bit applications can effectively use more than 4GB of RAM through indirect means like multiple processes, the disk cache or video card RAM. Ross Ridge -- l/ // Ross Ridge -- The Great HTMU [oo][oo] rridge(a)csclub.uwaterloo.ca -()-/()/ http://www.csclub.uwaterloo.ca/~rridge/ db //
From: Diez B. Roggisch on 14 Feb 2010 18:20 Am 14.02.10 12:28, schrieb Laszlo Nagy: > 2010.02.13. 17:40 keltez�ssel, Diez B. Roggisch �rta: >> Am 13.02.10 17:18, schrieb Anssi Saari: >>> Nobody<nobody(a)nowhere.com> writes: >>> >>>> A single process can't use much more than 2GiB of RAM without a >>>> 64-bit CPU >>>> and OS. >>> >>> That's not really true. Even Windows XP has the /3GB boot option to >>> allow 3 GiB per process. On PCs, free operating systems and server >>> Windows can use PAE to give access to full 4 GB per process. >> >> No, PAE can be used to access much more memory than 4GB - albeit >> through paging. AFAIK up to 2^36 Bytes. > PAE is for the kernel. Yes, you can use much more memory with 32 bit > kernel + PAE. But the ~3G per 32bit process limit still applies. It is > because the PAE extension allows the kernel to distribute the same > virtual address ranges between different processes, but map them to > different physical memory addresses. However, the process that was > compiled and is running in 32 bit mode, cannot access more than ~3GB > simply because it is not able to address more memory locations with 32 > bit addresses. (minus ~1G is because it needs virtual addresses for I/O > devices as well, and those addresses cannot be mapped to physical memory). No. It can access more, through paging, similar like mmap. Specialized software like databases do that to keep large data in memory. And this has nothing to do with compilation. Of course this doesn't help for python, at least not for python-objects, as these need a flat memory model. But it's not true that only the kernels benefit from the PAE. Diez
From: Dave Angel on 14 Feb 2010 22:16 Echavarria Gregory, Maria Angelica wrote: > Dear Chris, > > One of the machines I tested my app in is 64 bit and happened the same. The RAM consumed by the OS and other processes is already included in the 2.2 I'm telling... my app enters to work when the RAM is already consumed in ~600 MB in the 3- 32 bit machines ... in the 64 bit machine was exactly the same only that it started a little bit higher because it has windows 7... so should I understand that there is nothing I can do for my app to use up more RAM? > > Thanks for your time and answer, > Angelica. > > > M. Angelica Echavarria-Gregory, M.Sc., E.I. > Ph.D Candidate > University of Miami > Phone 305 284-3611 > > ________________________________ > From: Chris Kaynor [ckaynor(a)zindagigames.com] > Sent: Friday, February 12, 2010 7:44 PM > To: Echavarria Gregory, Maria Angelica > Cc: python-list(a)python.org > Subject: Re: MemoryError, can I use more? > > A 32 bit app can only use 4 GB of memory itself (regardless of the amount of system ram), the OS claims some of this for the system, dlls occupy some of it, etc. As such, the app can only really use a smaller subset (generally between 2 to 3 GB, depending upon the app and the OS). > > Chris > > > On Fri, Feb 12, 2010 at 4:21 PM, Echavarria Gregory, Maria Angelica <m.echavarriagregory(a)umiami.edu<mailto:m.echavarriagregory(a)umiami.edu>> wrote: > Dear group: > > I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed. > > Could any of you please help me to figure out how to change that limit? I typed help(MemoryError) and it is a class itself, but that help told me nothing I can use... > > Thanks, > Angelica. > -- > http://mail.python.org/mailman/listinfo/python-list > > > There are three different limits at play here. Since you're still not saying how you're "measuring" usage, we've all been guessing just which one you're hitting. There's physical RAM, virtual address space, and swappable space (swapfile on disk). Each reaches some limit in different ways. And there are probably a dozen different ways to measure "memory use," that get a dozen different answers. If you say which one you're using, that gives "exactly 2.2 GB," maybe someone will be familiar with that particular approach, and its meaning. DaveA
From: Tommy Grav on 14 Feb 2010 22:39
On Feb 14, 2010, at 10:16 PM, Dave Angel wrote: > There are three different limits at play here. Since you're still not saying how you're "measuring" usage, we've all been guessing just which one you're hitting. There's physical RAM, virtual address space, and swappable space (swapfile on disk). Each reaches some limit in different ways. > > And there are probably a dozen different ways to measure "memory use," that get a dozen different answers. If you say which one you're using, that gives "exactly 2.2 GB," maybe someone will be familiar with that particular approach, and its meaning. I ran into the same problem on Mac OS X and investigated it. My problem was that the program reached a memory error after using 2.2GB of memory. Now what caused it specifically I am less sure of, but switching to a 64bit python on a 64bit OS (snow leopard) means that I now frequently move past this boundary (when I have enough memory of course :) Tommy |