MemoryError, can I use more?
in [Python]
|
Prev: Configuring apache to execute python scripts using mod_pythonhandler
Next: Helpp I want view page .py on apache WHAT CAN I DO???????????????
From: Echavarria Gregory, Maria Angelica on 12 Feb 2010 19:21 Dear group: I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed. Could any of you please help me to figure out how to change that limit? I typed help(MemoryError) and it is a class itself, but that help told me nothing I can use... Thanks, Angelica.
From: ssteinerX on 12 Feb 2010 19:58 On Feb 12, 2010, at 7:21 PM, Echavarria Gregory, Maria Angelica wrote: > Dear group: > > I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed. > > Could any of you please help me to figure out how to change that limit? I typed help(MemoryError) and it is a class itself, but that help told me nothing I can use... How are you determining that it has occupied "exactly only 2.2GB?" S
From: Mark Lawrence on 12 Feb 2010 20:00 Echavarria Gregory, Maria Angelica wrote: > Dear group: > > I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed. > > Could any of you please help me to figure out how to change that limit? I typed help(MemoryError) and it is a class itself, but that help told me nothing I can use... > > Thanks, > Angelica. Please check the archives for the thread Please help with MemoryError, it was only posted a day or two back. It's a problem specific to Windows. Kindest regards. Mark Lawrence.
From: Benjamin Kaplan on 12 Feb 2010 20:14 On Fri, Feb 12, 2010 at 7:21 PM, Echavarria Gregory, Maria Angelica <m.echavarriagregory(a)umiami.edu> wrote: > Dear group: > > I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2..2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed. > > Could any of you please help me to figure out how to change that limit? I typed help(MemoryError) and it is a class itself, but that help told me nothing I can use... > > Thanks, > Angelica. There's nothing you can do in Python to fix it. A MemoryError means that Python asked the OS to malloc something and it refused to give it any more memory. In this case, it's because Windows will only allocate 2GB of RAM to a single process. There is a way to extend that to 3GB, but I don't quite understand how to do it- something about setting linker flags. Easiest way to handle this would be to switch to a 64-bit Python on a 64-bit platform. That way, you can use up to 8TB of memory. http://msdn.microsoft.com/en-us/library/aa366778%28VS.85%29.aspx#memory_limits > -- > http://mail.python.org/mailman/listinfo/python-list >
From: Christian Heimes on 12 Feb 2010 20:42
Echavarria Gregory, Maria Angelica wrote: > Dear group: > > I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed. The amount of RAM in your boxes is unrelated to your issue. A 32bit process is able to address only 4GB of virtual memory because it has no way to access memory addresses beyond the 32bit border. Large amount of the address space are reserved for multiple purposes like error checking (e.g. the NULL pointer), the entry point of the program, its stack and address space for reallocating dynamic linked libraries. On modern Linux systems the space for the heap is about 2.6 to 2.7 GB large. The heap is the memory segment used by malloc() and friends. It looks like the memory segment for the heap is slightly smaller on Windows. Virtual memory isn't the same as resident memory. A program may only occupy 100 MB in your RAM (resident memory) but it may use 1.5 GB of virtual memory. Virtual memory takes memory fragmentation and unused pages into account. If you malloc() 1 GB memory but just file up the first byte your program requires 1 GB of virtual memory but only a few KB resident memory. At least that's the story on Linux and several other Unix-like OSes. I'm not sure how Windows deals with memory. If your program needs more than 2 GB of RAM you should switch to a 64bit OS and a 64bit version of Python. Windows AMD64 builds for Python are available on python. Christian |