Preliminary analysis of existing ASDF practices
in [Lisp]
|
Prev: Is LISP divine?
Next: keywords & macros
From: Nathan Froyd on 18 Apr 2010 21:25 On Apr 18, 5:21 pm, Juanjo <juanjose.garciarip...(a)googlemail.com> wrote: > The problem, though, begins when one wants to have things that are not > contemplated by the Common Lisp specification and by ASDF as it stands > now: > * Automatically generated lisp files ASDF already does this, or at least there exist systems using ASDF that do this. See $SBCL/contrib/sb-grovel/ > * Automatically generated documentation I don't see why this is particularly hard, could you explain? > * Binary files for other languages (C++, C, fortran) Very doable. I don't have an example offhand, but earlier incarnations of $SBCL/contrib/sb-bsd-sockets/ were perfectly comfortable compiling C files as part of a system > * Shared libraries in those languages Ditto. > * Dependencies of compiled files on the previous things Ditto. > * ... > > What we need among other things is the possibility to > [...] > * Implement reusable components that we know how to compile and > *LOAD*, but which use other languages Again, sb-grovel. > Note that all this can not be done with arbitary, hand written CLOS > methods: ASDF has to be informed about how to do those tasks > declaratively so that it can incorporate the knowledge in other > contexts, such as when building an executable instead of dumping an > image. I must be missing something here. Could you explain what the procedure for building an executable and/or shared library with ECL is and why that doesn't work with existing ASDF systems? I'm not familiar with ECL; the few times I have used it it worked just fine with my ASDF files. I suppose the problem with shared libraries is that you need a "link together object files" step that's not really explicitly specified in most libraries and it's hard to wedge it in with the current infrastructure--is that correct? I agree that providing reusable components/extensiosn for ASDF is full of problems (need extensions loaded to load .asd file, etc.). But it's not clear to me how waving the "declarative!" flag about is providing solutions; you still need to write code *somewhere* to implement those declarative rules. And declarative systems can be abused too--see glibc's Makefiles for numerous examples. -Nathan
From: Tim X on 19 Apr 2010 06:41 m_mommer(a)yahoo.com (Mario S. Mommer) writes: > Tim X <timx(a)nospam.dev.null> writes: >> Juanjo <juanjose.garciaripoll(a)googlemail.com> writes: >>> An even better version would not need methods at all! What makes >>> TRAVERSE so unpredictable is that class dependencies, method >>> specialization, :around, :after, and other apparent niceties can make >>> a developer blow everything up -- to name a few things, ASDF recently >>> changed the way it specializes some methods and it broke ECL's >>> extensions in various and really misterious ways. >>> >>> OO is nice, but it is a distraction in this camp. > > Oh oh, that's... heretical! :-) > > Seriously though, the problem is not OO but trying to cram everything in > that dependency graph image which, it seems, simply is not a good > metaphor as soon as you are doing something more interesting, at least > not in its current version. I agree that OO is not the base issue. The same problems could as easily exist with any paradigm. >> I agree and think this exposes one of the reasons we have the situation >> we are in. To some extent, ASDF has been so powerful that the individual >> has been able to do things which on one had can make their life easier >> when you are only looking at a system from a single perspective, but >> once you move out into a wider world of possibilities, things get too >> complex to manage. > > (let me just interrupt to say that I think that the words being used in > this discussion are signs of trouble: Control is to be imposed, policies > are to be enforced, the power of the individual curtailed, etc) Let me interrupt to say I think your over stating what people have been saying. I've not seen anywhere a suggestion that the power of the individual is to be curtailed or even that control is to be imposed. Putting everything else aside, there wold be no way to do this anyway. What there has been is a sub-thread within this discussion that has suggested the problem is not with the power that ASDF provides, but rather with a lack of any standard application of that power. It ahs been suggested, through various ways, that some of the issues could be resolved if more conventions were established regarding how ASDF is applied. This is a long way from the totalitarian interpretation you appear to be implying. It strikes me that there are two different camps emerging here. One camp is concerned about anything that would restrict what they can do with ASDF. This could well be a valid concern if the restrictions prevented required or very useful functionality. I do get the impression that many who come form this perspective are focusing on how they use ASDF for their project. The other camp appears to be people who are more focused on how to use ASDF to install and use modules and libraries from multiple different sources. For them, the problem is the weird and often unpredictable interaction that occurs between different ASDF uses/definitions or sometimes, things breaking because of differences in the CL implementations being used and assumptions about an implementation which may not hold for all implementations. Any successful improvement really needs to address the concerns and requirements of both camps (and more/others of course). More consistent and standardised applicaiton of ASDF or more clearly articulated conventions regarding its use does not necessarily mean loss of power or flexibility, but it could reduce unnecessary conflicts and reduce the frequency of inconsistent outcomes. >> Concise and clear definition of dependencies would seem to be >> the most critical base requirement. > > I don't think that that is the problem. The problem is identifying what > kind of thing depend on what, Huh? Isn't that just a concise definition of dependency? I don't gget it. Are you possibly reading more into what I wrote than was there? Note that I've not said anything on how dependencies should be expressed/defined, only that a good starting point is to have a way to clearly and concisely specify them. and I'd like to propose viewing actions > with side-effects as the things, and the dependencies as being that what > everybody calls a dependency anyway. > > For example, consider this very simple rule. > > "the compilation of 'file1.lisp' depends on 'macros.lisp' and > 'packages.lisp' being loaded". > > ASDF and defsystem treat this as dependencies on files, and not > dependencies on the actions themselves. > > Instead of saying 'file1.lisp' depends on 'macros.lisp' and > 'packages.lisp', and then awkwardly saying how, it would be better to > say these things in a straight forward way. ASDF and defsystem tried the > former approach, and maybe it is time to try the latter. Maybe it is time to try an alternative, but I'm not sure we have even got that far yet. Please, using the same example you used above, give an example of how you would express these dependencies using your model so that we can better understand. Tim -- tcross (at) rapttech dot com dot au
From: Tim X on 20 Apr 2010 03:42 m_mommer(a)yahoo.com (Mario S. Mommer) writes: > Tim X <timx(a)nospam.dev.null> writes: >>>> Concise and clear definition of dependencies would seem to be >>>> the most critical base requirement. >>> >>> I don't think that that is the problem. The problem is identifying what >>> kind of thing depend on what, >> >> Huh? Isn't that just a concise definition of dependency? I don't gget >> it. Are you possibly reading more into what I wrote than was there? Note >> that I've not said anything on how dependencies should be >> expressed/defined, only that a good starting point is to have a way to >> clearly and concisely specify them. > > Ok, ok, so you did not mean the arrows, you meant the nodes. Fair > enough. No offense intended. > > What I am saying is that taking a graph whose nodes are "files" and then > manipulating the definition of the arrows to get what one wants is not > the only way to do things, and maybe not the best. It definitely has > some Turing tar pit quality to it; at least that's the way it feels when > I am writing an .asd to do something moderately complex. I can't say > defsystem felt better. I am just questioning the approach. > No offense taken. My point is that all I was stating is that we need a clear and concise way to represent dependencies. I have as yet no preferred approach to how this is done - this is what I feel we still need to identify and I believe it is the basic first step that needs to be nailed down before anything else can be done. Whether this is a declaritive, programatic, graph based or whatever is still I think an open question. >>> Instead of saying 'file1.lisp' depends on 'macros.lisp' and >>> 'packages.lisp', and then awkwardly saying how, it would be better to >>> say these things in a straight forward way. ASDF and defsystem tried the >>> former approach, and maybe it is time to try the latter. >> >> Maybe it is time to try an alternative, but I'm not sure we have even got >> that far yet. >> >> Please, using the same example you used above, give an example of >> how you would express these dependencies using your model so that we can >> better understand. > > As an extension of the defsystem notation: > > (requires (compile "file1.lisp") > (load "macros.lisp") > (load "packages.lisp")) > > I agree that this is not too cool ;-). But how about this: > > (let ((files > "packages" "macros" > "file1" "file2")) ;; etc. A directory listing would be > ;; better, probably > > (load-if-source-is-newer "packages.lisp") > (load-if-source-is-newer "macros.lisp") > > ;; You can even write a helpful comment here, explaining why you need > ;; the macros.lisp file before compiling other stuff. It even feels > ;; natural. > (mapcar #'compile-file-if-source-is-newer files) > (mapcar #'load-file-if-fasl-is-newer files)) > > Small, clean, and easy to understand. With a bit of policy, and a decent > way of handling paths (can be handled by a small lib) it would be > quite usable and explicit. In ASDF you have the :serial option for this, > but this is a hack, and does not really explain what depends on > what. And BTW this hack is needed because otherwise you'd need to write > ("fileN" :depends-on ("packages" "macros")) > for each N, and that is obviously not acceptable. > > I am not advocating the use of ad-hoc loaders, just trying to make a > more general point. I guess that what I am trying to say is that we have > this fantastic programming language that can solve so many problems, but > when it comes to specifying how to build/load software, we insist on > using a very strange DSL (the defsystem form) that is quite limiting and > makes us jump through redundant hoops to do even simple things. > Much of what you say I agree with. However, I'm not convinced that because past attempts of a DSL have had weaknesses that all/any attempt will suffer the same. Perhaps we can learn from the past mistakes and make something better. As you say, this is supposed to be one of the most powerful languages available, so maybe its the way its been applied as a DSL that is the problem rather than the DSL approach per se. I suspect the ultimate solution will be a hybrid that makes the majority of cases very easy, but makes the complex very possible. One thing I do think we need to make any real judgement is a selection of use cases that we can use to evaluate different approaches. It is very easy to fall into a trap where you have a solution that looks good, but in reality only addresses a small sub-set of requirements. Tim -- tcross (at) rapttech dot com dot au
From: Mario S. Mommer on 19 Apr 2010 12:50 Tim X <timx(a)nospam.dev.null> writes: >>> Concise and clear definition of dependencies would seem to be >>> the most critical base requirement. >> >> I don't think that that is the problem. The problem is identifying what >> kind of thing depend on what, > > Huh? Isn't that just a concise definition of dependency? I don't gget > it. Are you possibly reading more into what I wrote than was there? Note > that I've not said anything on how dependencies should be > expressed/defined, only that a good starting point is to have a way to > clearly and concisely specify them. Ok, ok, so you did not mean the arrows, you meant the nodes. Fair enough. No offense intended. What I am saying is that taking a graph whose nodes are "files" and then manipulating the definition of the arrows to get what one wants is not the only way to do things, and maybe not the best. It definitely has some Turing tar pit quality to it; at least that's the way it feels when I am writing an .asd to do something moderately complex. I can't say defsystem felt better. I am just questioning the approach. >> Instead of saying 'file1.lisp' depends on 'macros.lisp' and >> 'packages.lisp', and then awkwardly saying how, it would be better to >> say these things in a straight forward way. ASDF and defsystem tried the >> former approach, and maybe it is time to try the latter. > > Maybe it is time to try an alternative, but I'm not sure we have even got > that far yet. > > Please, using the same example you used above, give an example of > how you would express these dependencies using your model so that we can > better understand. As an extension of the defsystem notation: (requires (compile "file1.lisp") (load "macros.lisp") (load "packages.lisp")) I agree that this is not too cool ;-). But how about this: (let ((files "packages" "macros" "file1" "file2")) ;; etc. A directory listing would be ;; better, probably (load-if-source-is-newer "packages.lisp") (load-if-source-is-newer "macros.lisp") ;; You can even write a helpful comment here, explaining why you need ;; the macros.lisp file before compiling other stuff. It even feels ;; natural. (mapcar #'compile-file-if-source-is-newer files) (mapcar #'load-file-if-fasl-is-newer files)) Small, clean, and easy to understand. With a bit of policy, and a decent way of handling paths (can be handled by a small lib) it would be quite usable and explicit. In ASDF you have the :serial option for this, but this is a hack, and does not really explain what depends on what. And BTW this hack is needed because otherwise you'd need to write ("fileN" :depends-on ("packages" "macros")) for each N, and that is obviously not acceptable. I am not advocating the use of ad-hoc loaders, just trying to make a more general point. I guess that what I am trying to say is that we have this fantastic programming language that can solve so many problems, but when it comes to specifying how to build/load software, we insist on using a very strange DSL (the defsystem form) that is quite limiting and makes us jump through redundant hoops to do even simple things. Regards, Mario.
From: Mario S. Mommer on 19 Apr 2010 14:31
Juanjo <juanjose.garciaripoll(a)googlemail.com> writes: >> ASDF and defsystem treat this as dependencies on files, and not >> dependencies on the actions themselves. > > Yes and no: actions depend on files but actions may also depend on > other actions (internally in the ASDF graph). It is just that this > possibility is not exposed in the DEFSYSTEM grammar. Maybe it is better that way :-) > I am very well aware of this situation and if you peek my previous > emails that is why I introduced the notion of "rules" and makefile- > style programming. Ok. >> Instead of saying 'file1.lisp' depends on 'macros.lisp' and >> 'packages.lisp', and then awkwardly saying how, it would be better to >> say these things in a straight forward way. ASDF and defsystem tried the >> former approach, and maybe it is time to try the latter. > > I do not agree with this. The current situation is ok for a library > with a set of existing lisp files for one simple reason: one may come > with a set of rules for "loading", "compiling" and the like, which may > be ok for a lisp where things are loaded and ready to be dumped, but > if I want to reuse your library for building a set of object files to > be linked together in ECL then it is not going to work just as nicely. > In this case, a simple declarative description with a fixed set of > rules and different backends (SBCL, ECL, CCL, ABCL...) is a powerful > thing. > > The problem, though, begins when one wants to have things that are not > contemplated by the Common Lisp specification and by ASDF as it stands > now: > * Automatically generated lisp files > * Automatically generated documentation > * Binary files for other languages (C++, C, fortran) > * Shared libraries in those languages > * Dependencies of compiled files on the previous things > * ... > > What we need among other things is the possibility to > * Add rules that express how to create some of the elements that > participate in a system > * Create rules that are executed for certain actions > (:install, :uninstall, ...) > * Implement reusable components that we know how to compile and > *LOAD*, but which use other languages > > Note that all this can not be done with arbitary, hand written CLOS > methods: ASDF has to be informed about how to do those tasks > declaratively so that it can incorporate the knowledge in other > contexts, such as when building an executable instead of dumping an > image. Basically, IIUYC, what you have in mind is a rule based DSL where the interpretations of the rules are parametrized over the backend implementation. Sounds good. I'll not miss writing CLOS methods and figuring out how they'll interact by basically trial and error. OTOH, I fear that wether something depends on something else also is dependent on the surrounding semantics, and there might lie some potential for still having to remind people that not all lisps are of the load-load-load-dump type. |