adjustable arrays
in [Lisp]
|
Prev: Comparing Lisp to Python, what you consider more important: speed or macros.
Next: LispBox - truncated result?
From: Espen Vestre on 12 May 2010 05:07 Peter Keller <psilord(a)merlin.cs.wisc.edu> writes: > So, I got my master/worker codes working and the epilogue to this was I > would run out of heap memory (and get a heap exhaustion error!) because > I was creating and interning too many keyword symbols(!!) which permanently > consumed all of the memory. Yikes! How many keyword symbols did you create before this happened? -- (espen)
From: Peter Keller on 12 May 2010 11:33 Espen Vestre <espen(a)vestre.net> wrote: > Peter Keller <psilord(a)merlin.cs.wisc.edu> writes: > >> So, I got my master/worker codes working and the epilogue to this was I >> would run out of heap memory (and get a heap exhaustion error!) because >> I was creating and interning too many keyword symbols(!!) which permanently >> consumed all of the memory. > > Yikes! How many keyword symbols did you create before this happened? About 1.5 million. :) I am using SBCL 1.0.38.something with the default heap sizes and whatnot. The master codes can generate tasks into the hundreds of millions range with about 1 million in memory at once depending upon size of the forms in the tasks being sent to the workers and the size of the results when they come back. I changed the unique id to be short strings created with format, which can be collected, instead of keyword symbols. Integers might be better, and if there is some more problem in scaling I'll move to that. I wanted something descriptive for the various objects in the system for tracking purposes. The thing about the identifiers is that they have to be valid across process boundaries, so really using keyword symbols for that in the first place was a terrible idea. :) Later, -pete
From: Espen Vestre on 12 May 2010 11:54 Peter Keller <psilord(a)merlin.cs.wisc.edu> writes: > About 1.5 million. :) I am using SBCL 1.0.38.something with the default > heap sizes and whatnot. Hmm, my LispWorks 6.0 (mac 64-bit) didn't think that was too hard - it created 1.5 million keywords in 25 seconds (including some formatting) - but it did grow to 1.4GB ;-) But I agree, it's not a good idea anyway. -- (espen)
From: Antony on 12 May 2010 13:57 Espen Vestre wrote: > Peter Keller <psilord(a)merlin.cs.wisc.edu> writes: > >> About 1.5 million. :) I am using SBCL 1.0.38.something with the default >> heap sizes and whatnot. > > Hmm, my LispWorks 6.0 (mac 64-bit) didn't think that was too hard - it > created 1.5 million keywords in 25 seconds (including some formatting) > - but it did grow to 1.4GB ;-) > 1.5 million (mega) keywords (symbols) require 1.4GB (giga) ? why is the order of magnitude so high -Antony
From: George Neuner on 14 May 2010 00:00
On Wed, 12 May 2010 10:57:24 -0700, Antony <spam+lisp_dot_linux(a)gmail.com> wrote: >Espen Vestre wrote: >> Peter Keller <psilord(a)merlin.cs.wisc.edu> writes: >> >>> About 1.5 million. :) I am using SBCL 1.0.38.something with the default >>> heap sizes and whatnot. >> >> Hmm, my LispWorks 6.0 (mac 64-bit) didn't think that was too hard - it >> created 1.5 million keywords in 25 seconds (including some formatting) >> - but it did grow to 1.4GB ;-) >> >1.5 million (mega) keywords (symbols) require 1.4GB (giga) ? >why is the order of magnitude so high >-Antony Symbols are a heavyweight structure: they contain a name reference, a function slot, a value slot, a property list, and a package reference - at least 40 bytes (64-bit) plus a header if the implementation uses headers. Doesn't add up to 1.4GB though. Lispworks is likely expanding the whole heap so that the symbol space is a constant fraction of it. George |