Incremental Java Compile
in [Java Programming]
|
From: Joshua Maurice on 2 May 2010 17:15 On May 1, 12:05 pm, "Mike Schilling" <mscottschill...(a)hotmail.com> wrote: > Lew wrote: > > Lew wrote: > >>> The class that depends on the > >>> compile-time constant, such as 'DEBUG' in your example, compiles the > >>> constant into its class, not the symbol. Without some external > >>> indication of the dependency, there's not any way for a compiler or > >>> build tool to detect that it exists. > > > Mike Schilling wrote: > >> Other than by noting it when the symbol is used during compilation, > >> and storing that bit of information somewhere. But the details of > >> that get messy. > > > I said that there is no way, not that there couldn't be a way. > > > Lew wrote: > >>> With respect to dependencies where the symbol is stored in the class > >>> rather than its value, even 'javac' handles the situation pretty > >>> well. > > > Mike Schilling wrote: > >> There are other difficult cases, like a method being added in class > >> A that results in a method in B (one of A's descendents) becoming > >> overloaded, such that a client of B should now choose the new > >> overload. Java really makes this stuff hard. (C# is no better.) > > > How would any language handle this? > > Languages that store the defintion of a class in a different file than its > implementation (e.g. C++) handle it by simple comparisons of file dates. > It's also possible to do this by having the compiler update a repository of > class definitions (I used to develop a system that did just that.) Actually no. It's not that. It's that the search path lookup is handled in a fundamentally different way. In Java, any piece of code anywhere in the file can result in a file system lookup, and this lookup is "ambiguous", where I mean subject to change, or depends on context. See the paper in my opening post Ghost Dependencies. C++ gets around this by having 2 separate compilation steps, the preprocessor, and the compiler proper. The preprocessor has a very simple well defined lookup process which is not context dependent or "ambiguous" unlike Java's lookup process. The preprocessor produces a single file which the compiler proper takes and produces a single output file. The entire thing is much less black box and much less complex than Java's classpath lookup which makes it much easier to produce a correct incremental build.
From: Joshua Maurice on 2 May 2010 17:16 On May 2, 6:17 am, Andreas Leitgeb <a...(a)gamma.logic.tuwien.ac.at> wrote: > Mike Schilling <mscottschill...(a)hotmail.com> wrote: > >> Now B has a dependency on A. If A changes, for any reason: [...] > >> then B has to recompiled. That's pretty standard stuff I think. There > >> really isn't any need to detect an overloaded method, this > >> simple dependency graph catches it, and many other cases too. > > The overload is a subtle case, because the added method isn;t actually used > > by anyone, so dependencies at the granularity of method usage won't catch > > it. (Unless you conflate all overloads as being "the same method", which is > > probably a good idea.) > > Or, somewhat finer-grained: conflate all overloaded methods that take the > same number of arguments - with some special reasoning about varargs... > > Another approach was: Maintain a database of each .class's API, > and if, after an incremental build, any recompiled class has a > changed API, or any class was removed, or added, then do a clean > build. Insert "non-private" whereever you find it appropriate. This would work, but it's "overkill", and not very incremental. I was hoping for a much more "minimal" rebuild.
From: Lew on 2 May 2010 18:13 Joshua Maurice wrote: >>> So, I post here because I feel better prepared to discuss this >>> subject. I still disagree that "build from clean" is the correct >>> answer. That would make our product's build still around ~25 minutes >>> for just the Java compilation of around ~20,000 source files (and >>> growing). There must / should be something better. Separation >>> translation units make so much sense. I just wish Java had them. Lew wrote: >> What, you never heard of JAR files? >> >> There's no excuse for "build clean" having to touch all 20K files. Joshua Maurice wrote: > And what if all of the code is under active development, aka new > features are being added to each layer on a weekly basis? You have not designed your system in a very modular way. -- Lew
From: Joshua Maurice on 2 May 2010 19:09 On May 2, 3:13 pm, Lew <no...(a)lewscanon.com> wrote: > Joshua Maurice wrote: > >>> So, I post here because I feel better prepared to discuss this > >>> subject. I still disagree that "build from clean" is the correct > >>> answer. That would make our product's build still around ~25 minutes > >>> for just the Java compilation of around ~20,000 source files (and > >>> growing). There must / should be something better. Separation > >>> translation units make so much sense. I just wish Java had them. > Lew wrote: > >> What, you never heard of JAR files? > > >> There's no excuse for "build clean" having to touch all 20K files. > Joshua Maurice wrote: > > And what if all of the code is under active development, aka new > > features are being added to each layer on a weekly basis? > > You have not designed your system in a very modular way. Agreed. Sadly, as I'm a more junior developer, not much I can do about it for such a large codebase, a fair share of which predates C++98 standardization.
From: Arne Vajhøj on 2 May 2010 21:21
On 02-05-2010 17:11, Joshua Maurice wrote: > On May 1, 10:43 am, Lew<no...(a)lewscanon.com> wrote: >> Mike Schilling wrote: >>> "The most obvious example of these limitations is that the task can't tell >>> which classes to recompile when a constant primitive data type exported by >>> other classes is changed. For example, a change in the definition of >>> something like >>> public final class Constants { >>> public final static boolean DEBUG=false; >>> } >> >>> will not be picked up by other classes. " >> >>> That is, it's an incremental (no pun intended) improvement on the usual Ant >>> algorithm of "recompile what onviouslt needs recompilation; if that doesn't >>> seem to work, do a clean build" >> >> You can't blame Ant for that one. The class that depends on the compile-time >> constant, such as 'DEBUG' in your example, compiles the constant into its >> class, not the symbol. Without some external indication of the dependency, >> there's not any way for a compiler or build tool to detect that it exists. >> >> With respect to dependencies where the symbol is stored in the class rather >> than its value, even 'javac' handles the situation pretty well. > > I'm not blaming anyone in particular. I just want to know how to get a > fully correct, aka 100% incremental build under the actions: adding, > removing, modifying java files, and adding, removing, or modifying > build steps of "take these jars, compile them to class files, then jar > them", aka the standard developer actions. To me the entire idea is rather pointless. The tool can not be made to work with binaries. It should be possible to do it working with source code. But it would require a huge effort to create a 100% working tool. You could solve your build problems for much less effort by working on the structure of the project. The project does not provide bang for the buck. Arne |