Preliminary analysis of existing ASDF practices
in [Lisp]
|
Prev: Is LISP divine?
Next: keywords & macros
From: Juanjo on 13 Apr 2010 17:36 On Apr 13, 10:47 pm, His kennyness <kentil...(a)gmail.com> wrote: > The beauty is Dr McCarthy's main contrib: code as data. Fix ASDF any way > needed and then write one simple parser that can work off existing .asd > files (with more or less hinting, perhaps). Point is not even that can be done 100% safely for all ASDF files out there and for others that people write. It is an illusion to try to bring a new system out of the blue without some cooperation from developers As an example, if you have a look at the code people have added to their ASDF files (*) you will find #. reader evaluated forms and a lot of other toplevel forms that only work when some dependencies have been loaded prior to that or when the value of the current directory is the appropriate one (sic). In the end there is going to be so much ad-hoc in the rules to process all that information that it ends up being quite a lot of manual work. Juanjo (*) See the failures in the html file to see the errors that happen when one tries to parse *.asd files with simple READ.
From: Mario S. Mommer on 13 Apr 2010 18:14 Juanjo <juanjose.garciaripoll(a)googlemail.com> writes: > On Apr 13, 7:32 pm, His kennyness <kentil...(a)gmail.com> wrote: >> There are other bad choices all over ASDF. >> Is it worth saving? Is mk:defsystem in better shape? > I believe we do not need something completely different. I think maybe that would be better. > We may progressively introduce back some rationality into ASDF, making > it closer to mk:defsystem but keeping the extensibility -- it is for > instant interesting to be able to introduce new kinds of components, in > particular for interfacing with other languages (fortran, C, etc). It does not do that at all well, and with mk:defsystem you could do that too. Maybe ASDF does it better than mk:defsystem, but I suspect it just does so differently or even worse. There is a .asd file for matlisp floating around that did take a lot of work but does not really work well for obscure reasons. > What I would like to learn is what people think they have learnt that > ASDF does wrong, What ASDF does wrong is to implement a crippled functional programming language whose domain is loading software in a lazy-evaluation kind of way. And then it presents a pretty baroque interface for "overloading" the operators, so to speak, which you do by defining classes and methods and keyword args and so on and so on. This is bad, because one can spend a lot of time figuring out how to do things that would be trivial when done in an ad-hoc fashion with a proper programming language. ASDF is something of a Turing tar pit. (As to the original authors, well, nobody complained about ASDF for a long time, so no one should judge them.) > things it gets right What it gets right, IMO, is to come with an accepted and more or less adhered-to way of specifying how to load code, and how to put that in one file, and make that information available to the CL implementation. There is a lot of irregularity in the details, but at least that pattern is adhered to. This has made a lot of things possible that were not possible before (asdf-install, etc). You should be aware of the alternatives, which were having ad hoc loaders that did whatever they liked. Like, files loading other files that then loaded other files. There are examples of that around in old repositories. ASDF and mk:defsystem might not be perfect, but they brought some discipline into the issue. > and to what extent the Common > Lisp community is willing to accept any progress along one or another > direction. Right now it seems that progress will encounter little resistance.
From: Juanjo on 13 Apr 2010 18:29 On Apr 13, 9:56 pm, Faré <fah...(a)gmail.com> wrote: > I am the current de facto ASDF maintainer. > I became so shortly after I blogged myself into it.http://fare.livejournal.com/149264.html No intention to make this a campaing for the position. I just want to take the fruitless discussions out of the ASDF developers list and into the public. Decisions on the future of build systems should be based on feedback from developers, not only on trying to get them move to the next big thing. > My objective with ASDF 2 was merely to package all the progress that > happened since various versions of ASDF 1 [...] > Amongst the strong constraints of ASDF 2 was that we should be 100% > backwards compatible, [...] > it is not unconceivable that a future ASDF 3 should introduce > incompatibilities allowing for a better build model. > That said, I am also the author of XCVB, and I believe that > it provides a better starting point for developing > a new, incompatible, build system for Common Lisp. The only way to preserve 100% compatibility is to stay in the past. This is a fact. Moreover when none of ASDF contributors actually understands fully the main function that drives ASDF, TRAVERSE. What matters right now is 1) getting things to work and 2) making the chain as painless as possible. Step 1) can be ensured by gathering the 1000 software components out there and ensuring they build, at least with 99% fidelity. Step 2) is achieved by carefully dropping unneeded features, bloat and complexity. You must know how hard it is to push a new build system through the CL community, specially when you want to introduce different ways of organizing and making dependencies explicit. http://comments.gmane.org/gmane.lisp.cffi.devel/1781?set_blog_all=yes That said, I have read XCVB's manual and I feel it introduces an unneeded level of complexity in it all. Parallel compilation is nice, but forcing the use of makefiles, permitting the developers to scatter the dependencies of their software over different files, and relying on external drivers, plus the size of the software itself -- ASDF could and should be so lean! --, does not really convince me. Right now ASDF contains four components 1- The notion of system definition, components and its implementation using CLOS 2- The infrastracture for locating, parsing and connecting dependencies in *.asd files 3- The logic for deciding how to chain operations based on the dependencies. 4- An interface to the filesystem for storing and locating files. What XCVB does is fixing 3 by restricting the family of operations to 2 and imposing certain restrictions in the dependencies, but it alters 1 and 2 by changing the syntax of dependencies and hiding the process from the user. Furthermore it replaces the interface with Makefiles and makes filesystem dependencies explicit (rules may contain paths). Not having the extension model of ASDF it probably can not handle other components, such as compiled files generated for a FFI, or other stuff. Moreover, it has not faced the problems of deciding a good design for an extension model, so it is not free from making the same mistakes as ASDF. XCVB faces a similar problem as the one I described when integrating with ASDF systems. If I interpret the manual correctly, it can not really handle them, but relies on ASDF http://common-lisp.net/project/xcvb/doc/README.html#xcvb-builds-depending-on-asdf-systems and expects people to gradually convert their ASDF systems to XCVB format using tools XCVB also provides. In other words, it still needs them to fix the problems I pointed out. Now the question is whether it pays of to "fix" ASDF, we jump on the XCVB wagon or we wait for some kind of illumination to write the next big thing. What is wrong with being lazy and * Give people tools to write better *.asd files, side-effect free. * Restrict the set of operations that ASDF supports to a few: compile, load & test * Simplify the way components are written, perhaps adding some syntactic sugar. * Create an API to load and traverse the set of systems and their dependencies. * Replace the routine that drives ASDF, TRAVERSE, with something everybody understands. * Test it works 99% of the cases with the huge set of software we have around. * Write makefile generators in the XCVB fashion, if they are useful for somebody. * Write scripts that can be distributed with libraries so that one may type ./asdf-configure ./asdf-install and get a library built and installed the proper way. The hardest part is changing the core, the logic which chooses what to do and when. Maybe it makes sense to consider the benefits of the old MK:DEFSYSTEM model where compilation was an optional step and thus files relied not on compile-time side effects but on the order in which files were loaded. How many existing systems would break if we reinterpreted ASDF files that way?
From: Pascal J. Bourguignon on 13 Apr 2010 20:14 Juanjo <juanjose.garciaripoll(a)googlemail.com> writes: > What I would like to learn is what people think they have learnt that > ASDF does wrong, things it gets right and to what extent the Common > Lisp community is willing to accept any progress along one or another > direction. My use of ASDF is rather limited. - One thing I find wrong as a developer is that I would have to write the dependencies. I don't. I write programs to sort out the dependencies automatically and to generate the ASD files automatically. If the ASD file is a byproduct, is it really needed? - As a user, ASDF is usable. But obviously, it should provide a simplier API than OPERATE for users, everyone is writing his own asdf-load function... - Another point I had difficulties with, as a user, are the warning that considered as bugs, and preventing compilation/loading of an ASDF system. (It may be relevant only to older versions of ASDF and/or SBCL, I've not tried the newest versions). (For my own projects, I usually stay with the loader.lisp approach). -- __Pascal Bourguignon__
From: Juanjo on 14 Apr 2010 03:20
On Apr 14, 6:14 am, Tim X <t...(a)nospam.dev.null> wrote: > Juanjo <juanjose.garciarip...(a)googlemail.com> writes: > > > What I would like to learn is what people think they have learnt that > > ASDF does wrong, things it gets right and to what extent the Common > > Lisp community is willing to accept any progress along one or another > > direction. > > I think this is the ideal starting point. It may also be worthwhile > asking what people like and dislike about other approaches, though ther > is a danger of information overload. [...] > The main issue I can see with this approach is it does run the > danger of decision by committee and could either result in an overly > complex over designed mess that never gets realised or something that is > so heavy-weight nobody wants to use it or is too limited and nobody > finds it any more useful than what they already have. It depends on how one sets up the path. If the goal is 1) Simplify ASDF till you can see the bare bones 2) Build tools on top of it you can get something that is not too complex by design. > Juanjo, from the little I can tell from your posts, your initial efforts > in this area and your track record with respect to work and > contributions, you may well be the one who could do this (either that or > you are an insane madman!). My only comments are > > 1. Your analysis of ASDF is a good starting point. However, I think you > need to be careful regarding some of your phrasing as it can come across > as an emotional rather than a valid technical criticism. Having experienced this myself in the last weeks. I am not a diplomatic person, just a nerd who does Physics and programming, and I can be rather emotional myself, specially when entering debates which just lead to opinions and not to facts. I do not think I would qualify for the person with leadership that you mentioned before and it was not my intention to put myself forward as such, but rather stimulate a more rational and more open discussion. > 2. I would recommend being very careful about using examples. Yes, I found this myself. If I start talking about ECL and how we build software people just disregard the rest of the discussion. If I start talking about delivering *.deb or *.rpm, just the like. If I say anything about the load & dump development model, people quickly disconnect and forget the rest of the debate. It is very tough and discouraging, and one of the reasons I am pursuing this is just because ECL's progress is impeded by the current status -- hey, I can not even properly automate testing of existing libraries because of it! > 1. Ability to easily define dependencies in systems I am developing. In > general, ASDF does this pretty well provided I don't include other libs > with unusual asdf forms or complex dependency graphs. It would be nice to have even artificially cooked examples, for any effort that aims at fixing / replacing ASDF will encounter them. > 2. I would love a system similar to CPAN. I'd love to be able to just > issue the command CLAN install <some-lib> and hav it identify any > dependencies not available locally, download all necessary sources, > build them, run tests and if all goes well install everything. This can be done if *.asd files at some point become really pure descriptions of the software. |