Elephant experiences?
in [Lisp]
|
Prev: Amazing Video on How Computer Fashion Is Changing, Nice Work !!
Next: macro to expand into function body with declare at top
From: vanekl on 4 Mar 2010 18:03 On Mar 4, 5:32 pm, "Alex Mizrahi" <udode...(a)users.sourceforge.net> wrote: > v> I periodically prune old forms out. The extra time is inconsequential. > v> The time is spent mainly reading disk, not processing it in CL. My > v> data files usually fit within one disk cluster. Reading one byte or 4K > v> takes the exact same time. > > If I'd be dealing with <4K of data, I would probably not even bother with > save/restore functions: just copying printed representation into a file in > Emacs would be fine, I think. It's more complicated than that. Each record has their own file. Even though each record is small, there are numerous records. > v> That's too bad. I never would have implemented such a system in the > v> first place. You seem like an intelligent guy -- I'm surprised you > v> didn't understand the shortcomings of that design earlier. > > We did not consider this use case, and we were under pressure to release a > version ASAP. Oh, one of those types of managers. So much time is wasted when experienced analysts don't sit back and take an hour or two to think the problem through. > Honestly, I don't think we'd be allowed to spend monthes on writing our own > database until we demonstrate that other solution would be *really* slow. There are usually OTS/git solutions already available that can get you 80% of where you need to go. No need to spend months. My tuple database took less time to build than most people spend arguing about what's the best database. It's almost too easy to build new things in CL. > ??>> Well, a feature of a database is that you can update it incrementally, > ??>> avoiding large latencies. > > v> A well-architected system should always take that into account. If you > v> think you need a database to achieve this than we will have to > v> disagree on this point. > > v> Rushing to find the largest hammer in your toolbox may solve your > v> problem, but if you understand how the other tools work you can > v> usually do better. > > Well, by database I meant any structured storage. I don't see a lot of > options here -- either storage is ad-hoc, like a bunch of files, or it is > structured. It can be as simple as key-value store. I agree with your basic premis (structured), but there are a /lot/ of different ways to achieve structure. People reaching for a heavy relational database every time they run into a problem is 20th-century thinking. > Elephant was started as a very simple layer on top of BDB key-value store.. > You can still use it in that mode. > It also has CLOS persistence layer. > > It is very easy to use, you just define some of your classes (if you use > CLOS, that is) as persistent -- just change defclass to defpclass, and > optionall add :index t on fields you want to have indices on. > > Then you open a store with a single command -- > (ele:open-store '(:bdb #p"mydb/")) > > And then it just works -- all objects you create are persisted, > automatically. You don't need to worry about anything, there are just a > handful of functions to use to retrieve your objects back from indices. > > So I'd say using Elephant is even simplier than dealing with those text > files, well, at least when you want to use CLOS anyway. I have no doubt that Elephant is a great solution for a lot of people. When I tried to use it a while ago I couldn't get it to build. Don't remember what the exact problem was. Once it becomes production-ready and runs on all the main lisps I'll probably take another look. It also had some scary documentation that says that it isn't able to store lists, arrays, and hash tables. Maybe I don't understand the documentation, but that sounds like a major limitation to me. Even simple text files can handle this.
From: Tim X on 4 Mar 2010 21:50 vanekl <vanek(a)acd.net> writes: > On Mar 4, 5:32 pm, "Alex Mizrahi" <udode...(a)users.sourceforge.net> > wrote: > > There are usually OTS/git solutions already available that can get you > 80% of where you need to go. No need to spend months. My tuple > database took less time to build than most people spend arguing about > what's the best database. It's almost too easy to build new things in > CL. > I think this is very true. I haven't yet decided if this is a good thing or a bad thing. It is partially why I don't get the constant arguments regarding the need for standard CL libs or updated standards etc - either I can find it already done by someone or I'll write it. Compare this with Java, wehre I want to avoid writing anything as much as possible - in that situation, I want lots of rady to use libs! > >> ??>> Well, a feature of a database is that you can update it incrementally, >> ??>> avoiding large latencies. >> >> v> A well-architected system should always take that into account. If you >> v> think you need a database to achieve this than we will have to >> v> disagree on this point. >> >> v> Rushing to find the largest hammer in your toolbox may solve your >> v> problem, but if you understand how the other tools work you can >> v> usually do better. >> >> Well, by database I meant any structured storage. I don't see a lot of >> options here -- either storage is ad-hoc, like a bunch of files, or it is >> structured. It can be as simple as key-value store. > > > I agree with your basic premis (structured), but there are a /lot/ of > different ways to achieve structure. People reaching for a heavy > relational database every time they run into a problem is 20th-century > thinking. Your environment has a role to play here as well. For example, where I currently work, they have a whole Oracle database infrastructure, including development and testing instances, backups, SAN storage, high availability, dedicated DBAs etc. If I'm writing an application that needs data storage or if I want to implement some object persistency layer etc, I'll almost certainly use Oracle for the storage because its there and maintained. It will almost certainly be too powerful and it may even be a little slower for some things than other solutions, but its there, its stable and I don't have to worry about it. On the other hand, if I am implementing that same application at home for my family to use, there is no way I'm going to use Oracle. I will probably use plain text files or maybe BDB files or possibly postgres, especially if I already have it installed for another application. Maybe even couchdb or existsdb or mungo or .... What will likely be important is to abstract the storage backend so that I can use the same framework for any of the environments I am likely to be working in. This will also enable me to scale things should that be necessary. Unfortunately, it seems that getting the right abstraction and framwework for object persistance is a bit more complex than it wold initially appear - at least, it seems lots of people have done it, but few seem to have done it well and we are still debating the best way to go, so there doesn't seem to be anything definitive! Tim -- tcross (at) rapttech dot com dot au
From: Nicolas Neuss on 5 Mar 2010 08:48 "Alex Mizrahi" <udodenko(a)users.sourceforge.net> writes: > NN> I am looking for a simple persistent storage mechanism for some Common > NN> Lisp data. AllegroCache would probably be the ideal choice, but I > NN> would like to have it available also for other CL implementations. > NN> Does anyone here have experiences with Elephant[*]? > > Yep, I'm one of Elephant's developers (although I worked mostly on a > freaky "postmodern" backend rather than on Elephant's core and main > bdb backend). > > From my experience, Elephant mostly works as advertised. You didn't > mention your requirements, but if you say it's simple, it should work. > I recommend using BDB backend, it is, basically, the only one which is > really solid and it is most supported. > > Right now Elephant version 1.0 is not yet released, and old version is > not supported. But I'd say verion 1.0 alpha is stable enough if you > won't use new fancy features which aren't polished yet. I would like to install it using clbuild, so I guess the following should be the way to go: http://common-lisp.net/project/elephant/faq10a2.html Is this up-to-date? And if yes: Is it possible to use UFFI and CFFI in parallel? I have other libraries depending on CFFI. Nicolas
From: fortunatus on 5 Mar 2010 10:01 On Mar 4, 12:14 pm, vanekl <va...(a)acd.net> wrote: > If your needs are modest, it doesn't get simpler and more portable > than this: > > (defun save (tuple filenm) > (setf *PRINT-READABLY* t) > (with-open-file (stream filenm :direction :output > :if-exists :append :if-does-not-exist :create) > (prin1 tuple stream))) > > (defun restore (filenm) > (with-open-file (stream filenm :direction :input) > (read stream))) > > I get 2,000+ save/restores per second on 7-year-old equipment, faster > than a mysql system I've worked on. Let's just re-iterate something that came out of one of your posts, to make your overall approach more clear: > Each record has their own file. Even though each record is small, > there are numerous records. So I think you are using the filesystem of the host OS as a name index for your tuples. The file names are very much part of your scheme for searching and retrieving records. I like this approach a lot. The limitations are obvious, but a lot of single user apps fit well.
From: Alex Mizrahi on 5 Mar 2010 10:59
??>> Right now Elephant version 1.0 is not yet released, and old version is ??>> not supported. But I'd say verion 1.0 alpha is stable enough if you ??>> won't use new fancy features which aren't polished yet. NN> I would like to install it using clbuild, so I guess the following NN> should be the way to go: NN> http://common-lisp.net/project/elephant/faq10a2.html NN> Is this up-to-date? As far as I understand it is, but I never had this problem myself. NN> And if yes: Is it possible to use UFFI and CFFI in parallel? NN> I have other libraries depending on CFFI. I think there should be no problem with this. You see, there is cffi-uffi-compat system which exposes CFFI functions as if they were UFFI functions. If you don't load this system, it should be OK. |