a potential lisp convert, and interpreting the shootout
in [Lisp]
|
Prev: Pocket Lisp Machine
Next: Next Generation of Language
From: Isaac Gouy on 2 Oct 2006 01:06 Henry Bigelow wrote: > Isaac Gouy wrote: > > Henry Bigelow wrote: > > > > > > > > It's not clear to me what problem your suggestion is supposed to solve. > > > > > > > the problem CVS will solve was mentioned by jon, juho and wade in their > > > description of the "life cycle" of a benchmark entry: > > > > > > jon: "the shootout maintainers claim the program never existed." etc. > > > > As I said, for this go read the shootout mailing-list archive. > > > > > > > juho: > > > > > > >5. The requirements for the benchmark are modified, and the optimized > > > > Lisp implementation gets deleted. There's no sign of it ever > > > > having existed. > > > > > > and wade: > > > > > > >Why the shootout site would have removed the previous faster Lisp > > > >version is beyond me. > > > > The shootout site removed "the previous faster Lisp version" of > > Fannkuch because it no longer had any measured time to display, it no > > longer gave the correct answer, it was broken, it stayed down at the > > bottom of the table showing 'Error' (I don't know how many months > > passed before it was finally removed - maybe we waited until someone > > contributed a working program). > > > > (When Fannkuch was changed back in 2005, I went through the contributed > > Fannkuch programs in the tracker and emailed the people who had > > provided an email address to let them know that the spec had been > > changed. Within a short time we received fixed programs for most of the > > programming languages.) > > > > > > > if several people could all contribute their versions, there might not > > > be any bone of contention as to which implementation of a given > > > benchmark was or is best. they'd all be up there with the results side > > > by side. > > > > > > but, my hope is not just to solve a 'problem' with the shootout, but to > > > make it even more useful. it would be interesting to see what the > > > running time difference would be between a lisp fannkuch imperatively > > > written, and lisp fannkuch functional version would be, for example. > > > > I guess you haven't noticed the 2 Scala programs we show for fannkuch > > http://shootout.alioth.debian.org/gp4/benchmark.php?test=fannkuch&lang=scala&id=2 > > http://shootout.alioth.debian.org/gp4/benchmark.php?test=fannkuch&lang=scala&id=0 > > i didn't notice that. i'm just talking about eliminating the > controversy caused by deleting old submissions, even if they are > non-working. the fact that they are non-working is important > information too. Why do you think the fact that broken programs are important information for the computer language shootout - we deal in working programs, that's why we provide example output files for people to diff against. Did you notice that we accept programs contributed as file attachements to our tracking system - we don't delete from the tracking system. > > > > > > saying this, i realize it might be a lot more work for the maintainers, > > > and possibly create problems. > > > > You seem to be suggesting that we should show every program that has > > ever been contributed, for ever. Do you understand that we continually > > update the language implementations and re-measure the programs? > > i was suggesting keeping the history of edits for each program, but not > necessarily displaying the results for each. and, not to remeasure the > entire history of all versions--that would obviously be impractical. > > so, for each benchmark, a CVS history of edits. and with each leaf on > the version, whatever test results were performed, with whatever > language implementation etc. some versions would be retested with > newer compiler implementations, or against newer algorithm requirements > and assigned 'error', or whatever, just the way you normally do it. > the only difference would be that this information would be recorded, > and you wouldn't have to make a decision whether to delete it or not. I think you are starting to define a project - go do! ;-) > > so, what i propose doesn't require any more computation than you > already do, just more diskspace. > > i don't know, maybe there really isn't any way to remedy this > situation. anyone have any other ideas? > > henry
From: Isaac Gouy on 2 Oct 2006 01:10 Ralph Richard Cook wrote: > Juho Snellman <jsnell(a)iki.fi> wrote: > > >> it's possible that for various reasons, people don't care enough about > >> the shootout > > > ... > > I tried submitting code to the shootout around June 2005, but > apparently I wasn't putting the right code and makes in the right > little dialog boxes, so it didn't go through their automatic build. I > tried to get clarifications but a reply to my e-mails took about a > week to get back, so I just gave up. Was it an implementation for Fasta? http://shootout.alioth.debian.org/gp4/benchmark.php?test=fasta&lang=sbcl&id=0
From: Juho Snellman on 2 Oct 2006 08:29 igouy(a)yahoo.com <igouy(a)yahoo.com> wrote: > That seems to be a more-or-less reasonable summary to me - although I > don't understand what you think the benefit of displaying optimized > Lisp implementations that don't give the expected result would be (#5). The benefit would obviously be that someone who decides to implement that program can start working from the old version, instead of writing it from scratch. I'd expect that it's going to be easier to make the existing optimized program conform to the new specification than re-implement an equally optimized one from scratch. Also, please note that it was absolutely not my intention to accuse Isaac, or anyone else involved with the Shootout, of being deceitful or having an anti-Lisp agenda. Or actually to accuse them of anything. My point was just that people really shouldn't be using the Shootout for making decisions about what languages to use, since it's not unlikely that a program for Blub has been submitted by someone who is still a newbie at programming Blub. -- Juho Snellman
From: Isaac Gouy on 2 Oct 2006 10:17 Juho Snellman wrote: > igouy(a)yahoo.com <igouy(a)yahoo.com> wrote: > > That seems to be a more-or-less reasonable summary to me - although I > > don't understand what you think the benefit of displaying optimized > > Lisp implementations that don't give the expected result would be (#5). > > The benefit would obviously be that someone who decides to implement > that program can start working from the old version, instead of > writing it from scratch. I'd expect that it's going to be easier to > make the existing optimized program conform to the new specification > than re-implement an equally optimized one from scratch. [ #302423 ] Lisp SBCL fannkuch Juho Snellman 2005-10-26 http://alioth.debian.org/tracker/index.php?func=detail&aid=302423&group_id=30402&atid=411646 > Also, please note that it was absolutely not my intention to accuse > Isaac, or anyone else involved with the Shootout, of being deceitful > or having an anti-Lisp agenda. Or actually to accuse them of anything. > > My point was just that people really shouldn't be using the Shootout > for making decisions about what languages to use, since it's not > unlikely that a program for Blub has been submitted by someone who is > still a newbie at programming Blub. Oh it's worse that that, sometimes we get programs that even I as a newbie Blub programmer can write better ;-)
From: Thomas A. Russ on 2 Oct 2006 13:00
"Henry Bigelow" <hrbigelow(a)gmail.com> writes: > it looks like you have a lot of experience. so, benchmark politics > aside, do you find lisp to be fast enough to do heavy-duty computing, > (in my case, a bayesian network with hundreds of nodes and lots of > floating point addition, but also written in a very extensible, > abstract style) Well, I have one anecdote to relate, told to me by a professor who took his sabbatical to work on Bayesian networks. He and another programmer began working on code to do Bayesian netork analysis of genetic factors for genetic counselors. The professor used Lisp and the other programmer C. The Lisp version (no surprise) was completed much more quickly. It also exploited lisp's run-time compiler by taking a particular family tree and generating a program for evaluating the resulting Bayesian network analysis. This program was then compiled and used for the analysis. That alone made it faster than the C program. And with the extra time available, the professor was able to include some additional features (like a nicer user inteface) and some algorithmic improvements that the C programmer never had time to get to, because they were still working on getting the basic code for evaluating the Bayesian network for a network defined at runtime to work. -- Thomas A. Russ, USC/Information Sciences Institute |