Elephant experiences?
in [Lisp]
|
Prev: Amazing Video on How Computer Fashion Is Changing, Nice Work !!
Next: macro to expand into function body with declare at top
From: Alex Mizrahi on 8 Mar 2010 15:35 v> I agree with your basic premis (structured), but there are a /lot/ of v> different ways to achieve structure. People reaching for a heavy v> relational database every time they run into a problem is 20th-century v> thinking. Well, you know, mathematicians typically reduce problems to ones they've already solved, no matter how complex that solved ones are. They even try to solve the most general (and most complex) case first, and then reduce all special case to it. So are they dumb or what? :) Ok, there might be performance overhead if you use overly complex tool in programming, but who cares as long as it works and satisfies the requirements? SQL database doesn't not need to be heavy -- SQLite is an example, it is something like 200KB of source code and it supports most features one might want. But using traditional DBMS servers is not a problem in some cases -- e.g. if I have PostgreSQL server running anyway, I might just use it even for a small thing instead of inventing something unique. v> I have no doubt that Elephant is a great solution for a lot of people. v> When I tried to use it a while ago I couldn't get it to build. Usually it is not Elephant's problem, but when you use wrong version of a library, wrong version of BDB, or wrong version of Lisp. v> It also had some scary documentation that says that it isn't able to v> store lists, arrays, and hash tables. Maybe I don't understand the v> documentation, but that sounds like a major limitation to me. Even v> simple text files can handle this. No, it can store lists, arrays and hash tables just well. (And was able to that as long as I remember it.) Maybe it was saying that it cannot automatically track changes in those collections -- obviously it can't do that without hijacking CL implementation. It can only track changes to persistent objects and persistent collections. E.g. if class is defined as persistent, all writes to slots of objects of that class will go into the database, and will be automatically indexed as needed. These slots can have any data structures of any complexity, but if you change that data, you need to call (setf (slot-value ..) ...) for data to be persisted. E.g.: (defpclass thing () ((id :accessor id-of :initarg :id :index t) (value :accessor value-of :initarg :value))) ;; when you create an instance it is automatically persisted ELE> (make-instance 'thing :id 1 :value (list 1 2 3)) #<THING oid:300> ELE> (get-instance-by-value 'thing 'id 1) #<THING oid:300> ;; after you setf a slot ... ELE> (setf (value-of (get-instance-by-value 'thing 'id 1)) (list 3 4 5)) (3 4 5) ;; its new value is automatically persisted ELE> (value-of (get-instance-by-value 'thing 'id 1)) (3 4 5) ELE> (defparameter *thing1* (get-instance-by-value 'thing 'id 1)) *THING1* ;; but if you get slot's value and change data in place without writing it to a slot ELE> (let ((value-of-thing1 (value-of *thing1*))) (print value-of-thing1) (setf (second value-of-thing1) 8) (print value-of-thing1)) (3 4 5) (3 8 5) (3 8 5) ;; it isn't automatically persisted ELE> (value-of *thing1*) (3 4 5) ;; but we can write to goddamn slot ELE> (let ((value-of-thing1 (value-of *thing1*))) (setf (second value-of-thing1) 8) (setf (value-of *thing1*) value-of-thing1)) (3 8 5) ;; to fix it ELE> (value-of *thing1*) (3 8 5) And if you hate objects, you can use "btrees", which are just like hash-tables, but are persisted. Here is an example with btrees: ;; create a btree ELE> (defparameter *mybt* (make-btree)) *MYBT* ;; you need to add to root to be able to get it in later sessions ;; otherwise it will become garbage in database ELE> (add-to-root "mybt" *mybt*) #<BDB-BTREE oid:302> ;; get-value is pretty much like gethash, but is persisted automatically. ELE> (setf (get-value 1 *mybt*) #(1 2 3)) #(1 2 3) ELE> (get-value 1 *mybt*) #(1 2 3) T ELE> (setf (get-value 5 *mybt*) #(5 6 7)) #(5 6 7) ELE> (map-btree (lambda (k v) (format t "~a:~a~%" k v)) *mybt*) 1:#(1 2 3) 5:#(5 6 7) NIL ELE> (setf (get-value 7 *mybt*) #(9 10 11)) #(9 10 11) ;; and it has something that hash-table doesn't have -- ordered traversal. ELE> (map-btree (lambda (k v) (format t "~a:~a~%" k v)) *mybt* :start 5) 5:#(5 6 7) 7:#(9 10 11) NIL
From: Alex Mizrahi on 8 Mar 2010 17:05 v> So is Elephant client/server or is it just a DB abstraction layer? Elephant is a persistence layer. It consists of a core which implements unified API for users and backends which implement persistence itself. Currently there are like four backends: 1. BerkeleyDB (BDB) -- it is a main backend, Elephant started as a thin layer on top of BDB, and Elephant's low-level APIs mimick BDB APIs. BerkeleyDB is a key-value storage, it is not client/server but it is a library which is loaded into a process and it reads/writes some files. However, BDB supports multiple processes to work with same database. 2. CLSQL -- uses SQL database to store stuff, can use MySQL, PostgreSQL and SQLite. However its developer wanted to keep stuff simple, so it loads everything into the memory and it does not work very well. It's deprecated now. 3. Postmodern -- uses PostgreSQL database via a native Lisp connector. It works in truly client/server fashion, so any number of Lisp instances on different machines can work with database simultaneously. But this design has a number of limitations. 4. Native Lisp backend -- exists only as a proof-of-concept by now. Ian Eslick have implemented some based on cl-prevalence in-memory database. Some people have worked on another native lisp backend, but they have nothing to show yet, as I understand. Elephant developers think that native Lisp backend is important, because it's going to be more portable and easy to install, but they don't have time to implement it. Basically, everybody who is interested in persistence solution for himself has no problem using BDB or PostgreSQL or SQLite. Elephant isn't an ORM in any way -- even if it uses SQL database as a backend, its content can't be read by anything except Elephant.
From: Paul Sexton on 8 Mar 2010 20:17 On Mar 9, 11:05 am, "Alex Mizrahi" <udode...(a)users.sourceforge.net> wrote: > v> So is Elephant client/server or is it just a DB abstraction layer? > > Elephant is a persistence layer. It consists of a core which implements > unified API for users and backends which implement persistence itself. > Currently there are like four backends: > > 1. BerkeleyDB (BDB) -- it is a main backend, Elephant started as a thin > layer on top of BDB, and Elephant's low-level APIs mimick BDB APIs. > BerkeleyDB is a key-value storage, it is not client/server but it is a > library which is loaded into a process and it reads/writes some files. > However, BDB supports multiple processes to work with same database. > > 2. CLSQL -- uses SQL database to store stuff, can use MySQL, PostgreSQL and > SQLite. However its developer wanted to keep stuff simple, so it loads > everything into the memory and it does not work very well. It's deprecated > now. > > 3. Postmodern -- uses PostgreSQL database via a native Lisp connector. It > works in truly client/server fashion, so any number of Lisp instances on > different machines can work with database simultaneously. But this design > has a number of limitations. > > 4. Native Lisp backend -- exists only as a proof-of-concept by now. Ian > Eslick have implemented some based on cl-prevalence in-memory database. > Some people have worked on another native lisp backend, but they have > nothing to show yet, as I understand. > Elephant developers think that native Lisp backend is important, because > it's going to be more portable and easy to install, but they don't have time > to implement it. Basically, everybody who is interested in persistence > solution for himself has no problem using BDB or PostgreSQL or SQLite. > > Elephant isn't an ORM in any way -- even if it uses SQL database as a > backend, its content can't be read by anything except Elephant. How about BKNR? (http://bknr.net). Persistence for CLOS classes/instances, implemented entirely in lisp (no separate database backend). The website looks more polished than other persistence libs (elephant excepted), documentation is not bad, and it seems to be used successfully by several websites. Does anyone have any experience with it?
From: fortunatus on 9 Mar 2010 15:20 On Mar 8, 2:24 pm, "Captain Obvious" <udode...(a)users.sourceforge.net> wrote: > f> I assume by "Lisp processes" you are referring to threads in one Lisp > f> image, rather than running multiple Lisp images? > > No, I'm not referring to threads. Pinning threads is very unlikely to have > any effects on race conditions. > > All threads in a Lisp image share same address space (state), and race > conditions happen when more than one thread tries to change same piece of > state, or one thread is reading while another is writing or something like > that. Good point. > The idea is to pin whole Lisp process (image) to one CPU so that whole > process can do only one thing at a time. Meaning all the threads of the Lisp image must then share the same CPU - helping with race conditions since none will really be concurrent. > f> Can you give an example for a particular Lisp implemtation (I'm > f> hoping SBCL or CLISP...) or point a manual page? > > Check your OS manual. On Linux each thread has a processes ID, and you can > use sched_setaffinity() on it. > For example, ... Thanks! That's the key - makes perfect sense. Thanks for example!
From: Stefan Nobis on 10 Mar 2010 03:21
"Captain Obvious" <udodenko(a)users.sourceforge.net> writes: > All threads in a Lisp image share same address space (state), and > race conditions happen when more than one thread tries to change > same piece of state, or one thread is reading while another is > writing or something like that. The idea is to pin whole Lisp > process (image) to one CPU so that whole process can do only one > thing at a time. I don't see how pinning the whole process to one CPU will help here. Even if you don't have real parallel execution you still have to deal with concurrency. Thread A could be interrupted in the middle of its state changing operation and the thread B that tries to read this state could be scheduled. And, by the way, why use multiple threads and then restrict yourself arbitrarily, abandoning the most interesting advantage of multithreading (namely parallel execution)? -- Stefan. |