Total maximal size of data
in [Python]
|
Prev: CompIMAGE 2010: Call for Short Communications
Next: how can i know if a python object have a attribute such as'attr1'?
From: Alexander Moibenko on 25 Jan 2010 14:05 I have a simple question to which I could not find an answer. What is the total maximal size of list including size of its elements? I do not like to look into python source. Here is a code example: import struct KB=1024 MB=KB*KB GB=MB*KB buf=[] bs=32*KB n=4*GB/bs print "N",n i=0 size=0L while i < n: data = struct.pack("%ss" % (bs,), "") buf.append(data) size = size+bs if size % (100*MB) == 0: print "SIZE", size/MB, "MB" i=i+1 while 1: # to keep script running while I am looking at the machine status pass Here is what I get on 32-bit architecture: cat /proc/meminfo MemTotal: 8309860 kB MemFree: 5964888 kB Buffers: 84396 kB Cached: 865644 kB SwapCached: 0 kB ....... The program output: N 131072 SIZE 100 MB SIZE 200 MB SIZE 300 MB SIZE 400 MB SIZE 500 MB SIZE 600 MB SIZE 700 MB SIZE 800 MB SIZE 900 MB SIZE 1000 MB SIZE 1100 MB SIZE 1200 MB SIZE 1300 MB SIZE 1400 MB SIZE 1500 MB SIZE 1600 MB SIZE 1700 MB SIZE 1800 MB SIZE 1900 MB SIZE 2000 MB SIZE 2100 MB SIZE 2200 MB SIZE 2300 MB SIZE 2400 MB SIZE 2500 MB SIZE 2600 MB SIZE 2700 MB SIZE 2800 MB SIZE 2900 MB SIZE 3000 MB Traceback (most recent call last): File "bs.py", line 14, in ? data = struct.pack("%ss" % (bs,), "") MemoryError ++++++++++++++++++++ The number of list elements for a given block size is 131072. If I change block size the script traces back at the same total size 3000MB. Somewhere I read that list could have 2147483647 items, on most platforms. Somewhere else that it is *536,870,912 (http://stackoverflow.com/questions/855191/how-big-can-a-python-array-get)** *But what is the maximal size of the whole list including the size of its elements? Thanks.
From: Diez B. Roggisch on 25 Jan 2010 14:23 Am 25.01.10 20:05, schrieb Alexander Moibenko: > I have a simple question to which I could not find an answer. > What is the total maximal size of list including size of its elements? > I do not like to look into python source. But it would answer that question pretty fast. Because then you'd see that all list-object-methods are defined in terms of Py_ssize_t, which is an alias for ssize_t of your platform. 64bit that should be a 64bit long. Diez
From: Diez B. Roggisch on 25 Jan 2010 14:23 Am 25.01.10 20:05, schrieb Alexander Moibenko: > I have a simple question to which I could not find an answer. > What is the total maximal size of list including size of its elements? > I do not like to look into python source. But it would answer that question pretty fast. Because then you'd see that all list-object-methods are defined in terms of Py_ssize_t, which is an alias for ssize_t of your platform. 64bit that should be a 64bit long. Diez
From: AlexM on 25 Jan 2010 14:39 On Jan 25, 1:23 pm, "Diez B. Roggisch" <de...(a)nospam.web.de> wrote: > Am 25.01.10 20:05, schrieb Alexander Moibenko: > > > I have a simple question to which I could not find an answer. > > What is the total maximal size of list including size of its elements? > > I do not like to look into python source. > > But it would answer that question pretty fast. Because then you'd see > that all list-object-methods are defined in terms of Py_ssize_t, which > is an alias for ssize_t of your platform. 64bit that should be a 64bit long. > > Diez Then how do explain the program output? Alex.
From: Diez B. Roggisch on 25 Jan 2010 15:03
Am 25.01.10 20:39, schrieb AlexM: > On Jan 25, 1:23 pm, "Diez B. Roggisch"<de...(a)nospam.web.de> wrote: >> Am 25.01.10 20:05, schrieb Alexander Moibenko: >> >>> I have a simple question to which I could not find an answer. >>> What is the total maximal size of list including size of its elements? >>> I do not like to look into python source. >> >> But it would answer that question pretty fast. Because then you'd see >> that all list-object-methods are defined in terms of Py_ssize_t, which >> is an alias for ssize_t of your platform. 64bit that should be a 64bit long. >> >> Diez > > Then how do explain the program output? What exactly? That after 3GB it ran out of memory? Because you don't have 4GB memory available for processes. Diez |