managing large table of data
in [Lisp]
|
From: Norbert_Paul on 2 Feb 2010 02:30 Tamas K Papp wrote: > Hi, .... > > I don't know much about these things, but I think that the best > solution would be a database of some kind. I am wondering what would > be the simplest and most hassle-free way to do this in CL (if that > matters, I am using SBCL). > > If I think of this tables as a matrix, I will save data along one > dimension (eg rows, each draw), but I will retrieve data along the > other (eg columns, multiple draws for each variable). The second step > will be done more often, so I want that to be fast. Does it matter > for speed which dimension I consider rows or columns? I'd do what Mario S. Mommer calls "overkill": create table Matrix( rowidx integer not null. colidx integer not null, entry doubleOrWhatever, primary key(rowidx,colidx) ); use cl-sql and leave all that memory management stuff, paging, caching, indexing etc. to the DBMS. Extract column 42 by select rowidx,entry from Matrix where colidx=42; If that turns out too slow, an additional index on the colidx attribute might help. Norbert
From: Marco Antoniotti on 5 Feb 2010 04:21
On Feb 2, 8:30 am, Norbert_Paul <norbertpauls_spam...(a)yahoo.com> wrote: > Tamas K Papp wrote: > > Hi, > ... > > > I don't know much about these things, but I think that the best > > solution would be a database of some kind. I am wondering what would > > be the simplest and most hassle-free way to do this in CL (if that > > matters, I am using SBCL). > > > If I think of this tables as a matrix, I will save data along one > > dimension (eg rows, each draw), but I will retrieve data along the > > other (eg columns, multiple draws for each variable). The second step > > will be done more often, so I want that to be fast. Does it matter > > for speed which dimension I consider rows or columns? > > I'd do what Mario S. Mommer calls "overkill": > > create table Matrix( > rowidx integer not null. > colidx integer not null, > entry doubleOrWhatever, > primary key(rowidx,colidx) ); > > use cl-sql and leave all that memory management stuff, > paging, caching, indexing etc. to the DBMS. > > Extract column 42 by > > select rowidx,entry > from Matrix > where colidx=42; > > If that turns out too slow, an additional index on the colidx > attribute might help. > That is also nice. However, as I am very lazy.... is there anybody out there doing HDF5 in CL? Cheers -- Marco |