MemoryError, can I use more?
in [Python]
|
Prev: Configuring apache to execute python scripts using mod_pythonhandler
Next: Helpp I want view page .py on apache WHAT CAN I DO???????????????
From: Nobody on 13 Feb 2010 02:01 On Fri, 12 Feb 2010 19:21:22 -0500, Echavarria Gregory, Maria Angelica wrote: > I am developing a program using Python 2.5.4 in windows 32 OS. The amount > of data it works with is huge. I have managed to keep memory footprint > low, but have found that, independent of the physical RAM of the machine, > python always gives the MemoryError message when it has occupied exactly > only 2.2 GB. I have tested this in 4 different machines, all with memory > of 3 to 4 GB... I'm amazed. > > Could any of you please help me to figure out how to change that limit? I > typed help(MemoryError) and it is a class itself, but that help told me > nothing I can use... A single process can't use much more than 2GiB of RAM without a 64-bit CPU and OS. If you're stuck with a 32-bit system, you really need to figure out how to limit the amount of data which is kept in memory at any one time.
From: Anssi Saari on 13 Feb 2010 11:18 Nobody <nobody(a)nowhere.com> writes: > A single process can't use much more than 2GiB of RAM without a 64-bit CPU > and OS. That's not really true. Even Windows XP has the /3GB boot option to allow 3 GiB per process. On PCs, free operating systems and server Windows can use PAE to give access to full 4 GB per process.
From: Diez B. Roggisch on 13 Feb 2010 11:40 Am 13.02.10 17:18, schrieb Anssi Saari: > Nobody<nobody(a)nowhere.com> writes: > >> A single process can't use much more than 2GiB of RAM without a 64-bit CPU >> and OS. > > That's not really true. Even Windows XP has the /3GB boot option to > allow 3 GiB per process. On PCs, free operating systems and server > Windows can use PAE to give access to full 4 GB per process. No, PAE can be used to access much more memory than 4GB - albeit through paging. AFAIK up to 2^36 Bytes. Diez
From: Anssi Saari on 13 Feb 2010 13:52 "Diez B. Roggisch" <deets(a)nospam.web.de> writes: > Am 13.02.10 17:18, schrieb Anssi Saari: >> Nobody<nobody(a)nowhere.com> writes: >> >>> A single process can't use much more than 2GiB of RAM without a 64-bit CPU >>> and OS. >> >> That's not really true. Even Windows XP has the /3GB boot option to >> allow 3 GiB per process. On PCs, free operating systems and server >> Windows can use PAE to give access to full 4 GB per process. > > No, PAE can be used to access much more memory than 4GB - albeit > through paging. AFAIK up to 2^36 Bytes. That too. I admit, after checking, that you can't go above 3 GiB per process even in server Windows. But for Linux there exists (or existed, since it seems it hasn't been updated since 2004) a kernel patch which provides a "4GB/4GB" address split. Kernel is in one segment, userland in another and hence a process can access full 4GB.
From: Laszlo Nagy on 14 Feb 2010 06:28
2010.02.13. 17:40 keltez�ssel, Diez B. Roggisch �rta: > Am 13.02.10 17:18, schrieb Anssi Saari: >> Nobody<nobody(a)nowhere.com> writes: >> >>> A single process can't use much more than 2GiB of RAM without a >>> 64-bit CPU >>> and OS. >> >> That's not really true. Even Windows XP has the /3GB boot option to >> allow 3 GiB per process. On PCs, free operating systems and server >> Windows can use PAE to give access to full 4 GB per process. > > No, PAE can be used to access much more memory than 4GB - albeit > through paging. AFAIK up to 2^36 Bytes. PAE is for the kernel. Yes, you can use much more memory with 32 bit kernel + PAE. But the ~3G per 32bit process limit still applies. It is because the PAE extension allows the kernel to distribute the same virtual address ranges between different processes, but map them to different physical memory addresses. However, the process that was compiled and is running in 32 bit mode, cannot access more than ~3GB simply because it is not able to address more memory locations with 32 bit addresses. (minus ~1G is because it needs virtual addresses for I/O devices as well, and those addresses cannot be mapped to physical memory). So with any Python that is running in 32 bit mode, you cannot use more than ~3G memory. But you can start many instances of those programs and use 2G for each process. L |