a potential lisp convert, and interpreting the shootout
in [Lisp]
|
Prev: Pocket Lisp Machine
Next: Next Generation of Language
From: Henry Bigelow on 1 Oct 2006 21:03 > > It's not clear to me what problem your suggestion is supposed to solve. > the problem CVS will solve was mentioned by jon, juho and wade in their description of the "life cycle" of a benchmark entry: jon: "the shootout maintainers claim the program never existed." etc. juho: >5. The requirements for the benchmark are modified, and the optimized > Lisp implementation gets deleted. There's no sign of it ever > having existed. and wade: >Why the shootout site would have removed the previous faster Lisp >version is beyond me. if several people could all contribute their versions, there might not be any bone of contention as to which implementation of a given benchmark was or is best. they'd all be up there with the results side by side. but, my hope is not just to solve a 'problem' with the shootout, but to make it even more useful. it would be interesting to see what the running time difference would be between a lisp fannkuch imperatively written, and lisp fannkuch functional version would be, for example. saying this, i realize it might be a lot more work for the maintainers, and possibly create problems. but i took a look at the 'shootin' site, and the author seems to promote the idea of a 'wiki but with a community-rated account system'. perhaps commit privileges for CVS could be regulated similarly, and it might then distribute the workload and the accountability. does this answer your question, or was it something else? thanks, henry > > I think the place to understand "what tweaks made something run faster" > is within a particular language community like this: > http://www.haskell.org/hawiki/ShootoutEntry > > The old Doug Bagley benchmarks were mostly replaced by new benchmarks > during autumn/winter 2005 - I haven't been keeping track but I don't > think the benchmarks have been changed this year.
From: Ralph Richard Cook on 1 Oct 2006 22:33 Juho Snellman <jsnell(a)iki.fi> wrote: >> it's possible that for various reasons, people don't care enough about >> the shootout > .... I tried submitting code to the shootout around June 2005, but apparently I wasn't putting the right code and makes in the right little dialog boxes, so it didn't go through their automatic build. I tried to get clarifications but a reply to my e-mails took about a week to get back, so I just gave up.
From: Isaac Gouy on 1 Oct 2006 22:53 Henry Bigelow wrote: > > > > It's not clear to me what problem your suggestion is supposed to solve. > > > the problem CVS will solve was mentioned by jon, juho and wade in their > description of the "life cycle" of a benchmark entry: > > jon: "the shootout maintainers claim the program never existed." etc. As I said, for this go read the shootout mailing-list archive. > juho: > > >5. The requirements for the benchmark are modified, and the optimized > > Lisp implementation gets deleted. There's no sign of it ever > > having existed. > > and wade: > > >Why the shootout site would have removed the previous faster Lisp > >version is beyond me. The shootout site removed "the previous faster Lisp version" of Fannkuch because it no longer had any measured time to display, it no longer gave the correct answer, it was broken, it stayed down at the bottom of the table showing 'Error' (I don't know how many months passed before it was finally removed - maybe we waited until someone contributed a working program). (When Fannkuch was changed back in 2005, I went through the contributed Fannkuch programs in the tracker and emailed the people who had provided an email address to let them know that the spec had been changed. Within a short time we received fixed programs for most of the programming languages.) > if several people could all contribute their versions, there might not > be any bone of contention as to which implementation of a given > benchmark was or is best. they'd all be up there with the results side > by side. > > but, my hope is not just to solve a 'problem' with the shootout, but to > make it even more useful. it would be interesting to see what the > running time difference would be between a lisp fannkuch imperatively > written, and lisp fannkuch functional version would be, for example. I guess you haven't noticed the 2 Scala programs we show for fannkuch http://shootout.alioth.debian.org/gp4/benchmark.php?test=fannkuch&lang=scala&id=2 http://shootout.alioth.debian.org/gp4/benchmark.php?test=fannkuch&lang=scala&id=0 > saying this, i realize it might be a lot more work for the maintainers, > and possibly create problems. You seem to be suggesting that we should show every program that has ever been contributed, for ever. Do you understand that we continually update the language implementations and re-measure the programs? > but i took a look at the 'shootin' site, > and the author seems to promote the idea of a 'wiki but with a > community-rated account system'. perhaps commit privileges for CVS > could be regulated similarly, and it might then distribute the workload > and the accountability. If you think "the shootin" has promise then help make something happen with "the shootin".
From: Henry Bigelow on 1 Oct 2006 22:56 Ralph Richard Cook wrote: > Juho Snellman <jsnell(a)iki.fi> wrote: > > >> it's possible that for various reasons, people don't care enough about > >> the shootout > > > ... > > I tried submitting code to the shootout around June 2005, but > apparently I wasn't putting the right code and makes in the right > little dialog boxes, so it didn't go through their automatic build. I > tried to get clarifications but a reply to my e-mails took about a > week to get back, so I just gave up. hi ralph, i see. well, i suppose it's possible that the maintainers are not very motivated, since it's a free service after all. or, they might be overwhelmed with emails. is there any way you can think to make the submission process more reliable or easier? and, what do you think of the idea of having a community-moderated CVS submission of code? thanks, henry
From: Henry Bigelow on 2 Oct 2006 00:07
Isaac Gouy wrote: > Henry Bigelow wrote: > > > > > > It's not clear to me what problem your suggestion is supposed to solve. > > > > > the problem CVS will solve was mentioned by jon, juho and wade in their > > description of the "life cycle" of a benchmark entry: > > > > jon: "the shootout maintainers claim the program never existed." etc. > > As I said, for this go read the shootout mailing-list archive. > > > > juho: > > > > >5. The requirements for the benchmark are modified, and the optimized > > > Lisp implementation gets deleted. There's no sign of it ever > > > having existed. > > > > and wade: > > > > >Why the shootout site would have removed the previous faster Lisp > > >version is beyond me. > > The shootout site removed "the previous faster Lisp version" of > Fannkuch because it no longer had any measured time to display, it no > longer gave the correct answer, it was broken, it stayed down at the > bottom of the table showing 'Error' (I don't know how many months > passed before it was finally removed - maybe we waited until someone > contributed a working program). > > (When Fannkuch was changed back in 2005, I went through the contributed > Fannkuch programs in the tracker and emailed the people who had > provided an email address to let them know that the spec had been > changed. Within a short time we received fixed programs for most of the > programming languages.) > > > > if several people could all contribute their versions, there might not > > be any bone of contention as to which implementation of a given > > benchmark was or is best. they'd all be up there with the results side > > by side. > > > > but, my hope is not just to solve a 'problem' with the shootout, but to > > make it even more useful. it would be interesting to see what the > > running time difference would be between a lisp fannkuch imperatively > > written, and lisp fannkuch functional version would be, for example. > > I guess you haven't noticed the 2 Scala programs we show for fannkuch > http://shootout.alioth.debian.org/gp4/benchmark.php?test=fannkuch&lang=scala&id=2 > http://shootout.alioth.debian.org/gp4/benchmark.php?test=fannkuch&lang=scala&id=0 i didn't notice that. i'm just talking about eliminating the controversy caused by deleting old submissions, even if they are non-working. the fact that they are non-working is important information too. > > > saying this, i realize it might be a lot more work for the maintainers, > > and possibly create problems. > > You seem to be suggesting that we should show every program that has > ever been contributed, for ever. Do you understand that we continually > update the language implementations and re-measure the programs? i was suggesting keeping the history of edits for each program, but not necessarily displaying the results for each. and, not to remeasure the entire history of all versions--that would obviously be impractical. so, for each benchmark, a CVS history of edits. and with each leaf on the version, whatever test results were performed, with whatever language implementation etc. some versions would be retested with newer compiler implementations, or against newer algorithm requirements and assigned 'error', or whatever, just the way you normally do it. the only difference would be that this information would be recorded, and you wouldn't have to make a decision whether to delete it or not. so, what i propose doesn't require any more computation than you already do, just more diskspace. i don't know, maybe there really isn't any way to remedy this situation. anyone have any other ideas? henry |