How to serialize/restore a large data structure?
in [TCL]
|
Prev: freewrap is awesome!
Next: Tcl and .NET
From: drscrypt on 30 Nov 2009 18:56 Georgios Petasis wrote: > Hi all, > > I have a large hash table, whose keys are words, and the values are > dicts, that contain integer pairs. > I am creating this structure in memory, taking care to reuse objects as > much as possible, with the result occupying ~ 1.3GB of memory. Do you really need all of the data in memory at once? If the requirements are flexible and you can work with a relatively large chunk but one at a time (like 500MB), then I would suggest sqlite. DrS
From: Nick Hounsome on 1 Dec 2009 02:32 On 30 Nov, 21:59, Georgios Petasis <peta...(a)iit.demokritos.gr> wrote: > Hi all, > > I have a large hash table, whose keys are words, and the values are > dicts, that contain integer pairs. > I am creating this structure in memory, taking care to reuse objects as > much as possible, with the result occupying ~ 1.3GB of memory. IMHO this is larger than is appropriate for dict. Either create your own C extension with its own read/write extension or use the brilliant Sqlite3 single file RDB extension
From: ddd on 1 Dec 2009 04:43 On Mon, 30 Nov 2009 23:32:45 -0800 (PST), Nick Hounsome <nick.hounsome(a)googlemail.com> wrote: > On 30 Nov, 21:59, Georgios Petasis <peta...(a)iit.demokritos.gr> wrote: >> Hi all, >> >> I have a large hash table, whose keys are words, and the values are >> dicts, that contain integer pairs. >> I am creating this structure in memory, taking care to reuse objects as >> much as possible, with the result occupying ~ 1.3GB of memory. > > IMHO this is larger than is appropriate for dict. > > Either create your own C extension with its own read/write extension > or > use the brilliant Sqlite3 single file RDB extension > Or you could try speedtables from Lehenbauer, see http://wiki.tcl.tk/20026
From: Larry W. Virden on 1 Dec 2009 07:07 On Dec 1, 2:32 am, Nick Hounsome <nick.houns...(a)googlemail.com> wrote: > On 30 Nov, 21:59, Georgios Petasis <peta...(a)iit.demokritos.gr> wrote: > > I have a large hash table, whose keys are words, and the values are > > dicts, that contain integer pairs. > > I am creating this structure in memory, taking care to reuse objects as > > much as possible, with the result occupying ~ 1.3GB of memory. > > IMHO this is larger than is appropriate for dict. > Interesting comment. Has anyone written any guidelines for choosing a Tcl data structure? That seems like it would be a worthwhile tutorial subject.
From: Donald Arseneau on 1 Dec 2009 11:33
On Nov 30, 1:59 pm, Georgios Petasis <peta...(a)iit.demokritos.gr> wrote: > I have a large hash table, whose keys are words, and the values are > dicts, that contain integer pairs. > I am creating this structure in memory, taking care to reuse objects as > much as possible, with the result occupying ~ 1.3GB of memory. > > However, I don't know how to serialise and restore such a large > structure. Just using "array get" needs much more memory, and tcl needs > more than the 2GB a 32-bit application can use. So, I wrote some code > that serialises all elements without requiring conversion to strings. I've never actually had cause to use them, but this sounds like a case for: array startsearch array nextelement array anymore array donesearch |