Incremental Java Compile
in [Java Programming]
|
From: Arne Vajhøj on 2 May 2010 21:22 On 02-05-2010 19:09, Joshua Maurice wrote: > On May 2, 3:13 pm, Lew<no...(a)lewscanon.com> wrote: >> Joshua Maurice wrote: >>>>> So, I post here because I feel better prepared to discuss this >>>>> subject. I still disagree that "build from clean" is the correct >>>>> answer. That would make our product's build still around ~25 minutes >>>>> for just the Java compilation of around ~20,000 source files (and >>>>> growing). There must / should be something better. Separation >>>>> translation units make so much sense. I just wish Java had them. >> Lew wrote: >>>> What, you never heard of JAR files? >> >>>> There's no excuse for "build clean" having to touch all 20K files. >> Joshua Maurice wrote: >>> And what if all of the code is under active development, aka new >>> features are being added to each layer on a weekly basis? >> >> You have not designed your system in a very modular way. > > Agreed. Sadly, as I'm a more junior developer, not much I can do about > it for such a large codebase, a fair share of which predates C++98 > standardization. It would give more benefits for the effort to try and clean up things. Arne
From: Arne Vajhøj on 2 May 2010 21:27 On 02-05-2010 17:08, Joshua Maurice wrote: > On May 1, 7:38 am, Lew<no...(a)lewscanon.com> wrote: >> Joshua Maurice wrote: >>> So, I post here because I feel better prepared to discuss this >>> subject. I still disagree that "build from clean" is the correct >>> answer. That would make our product's build still around ~25 minutes >>> for just the Java compilation of around ~20,000 source files (and >>> growing). There must / should be something better. Separation >>> translation units make so much sense. I just wish Java had them. >> >> What, you never heard of JAR files? >> >> There's no excuse for "build clean" having to touch all 20K files. > > And what if all of the code is under active development, aka new > features are being added to each layer on a weekly basis? Still not a good excuse. Each team/stream/whatever you call it should work on their own component and a stable binary release of all other components. > And what if a large portion of that Java code is generated from a > model file to facilitate serialization between C++ and Java? Thus a > change to a single file would require recompiling a large amount of > generated "interface" files, which theoretically touches a large > portion of the 20,000 Java files. Structure things better. With a good OO model a series of related changes should not require changes in "a large portion of 20000 Java files". Arne
From: Joshua Maurice on 3 May 2010 00:50 On May 2, 6:27 pm, Arne Vajhøj <a...(a)vajhoej.dk> wrote: > Structure things better. > > With a good OO model a series of related changes should not > require changes in "a large portion of 20000 Java files". So, how do you suggest doing that when there's a code generator under active development which generates Java code, and a large portion of the Java code directly or indirectly works with the output of this code generator? We model the object domains in a simple modeling language which is then compiled to C++ and Java code to allow serializing a description of a unit of work from the Java tools to the C++ tools and back. Most of the infrastructure and apps work with the output of this code generator in some form or another. Unfortunately, one cannot fiat interfaces into being stable.
From: Mike Schilling on 3 May 2010 02:54 Arne Vajh�j wrote: > On 02-05-2010 17:11, Joshua Maurice wrote: >> On May 1, 10:43 am, Lew<no...(a)lewscanon.com> wrote: >>> Mike Schilling wrote: >>>> "The most obvious example of these limitations is that the task >>>> can't tell which classes to recompile when a constant primitive >>>> data type exported by other classes is changed. For example, a >>>> change in the definition of something like >>>> public final class Constants { >>>> public final static boolean DEBUG=false; >>>> } >>> >>>> will not be picked up by other classes. " >>> >>>> That is, it's an incremental (no pun intended) improvement on the >>>> usual Ant algorithm of "recompile what onviouslt needs >>>> recompilation; if that doesn't seem to work, do a clean build" >>> >>> You can't blame Ant for that one. The class that depends on the >>> compile-time constant, such as 'DEBUG' in your example, compiles >>> the constant into its class, not the symbol. Without some external >>> indication of the dependency, there's not any way for a compiler or >>> build tool to detect that it exists. With respect to dependencies where >>> the symbol is stored in the >>> class rather than its value, even 'javac' handles the situation >>> pretty well. >> >> I'm not blaming anyone in particular. I just want to know how to get >> a fully correct, aka 100% incremental build under the actions: >> adding, removing, modifying java files, and adding, removing, or >> modifying build steps of "take these jars, compile them to class >> files, then jar them", aka the standard developer actions. > > To me the entire idea is rather pointless. > > The tool can not be made to work with binaries. I think it could. At least, reducing the problem to: * You have a complete correct build of the system * You have a set of changes to the source since that build was done * Do a minimal (more or less) amount of recompliation to arrive at a new complete and correct build is feasible. But I don't see the point, really. It's simpler, cheaper, and more reliable to throw hardware at the problem. Buy a fast machine to do a continuous build and archive the last N days worth of builds. You can now fetch a completely built system at any release level with no compilation required at all.
From: Joshua Maurice on 3 May 2010 04:06
On May 2, 11:54 pm, "Mike Schilling" <mscottschill...(a)hotmail.com> wrote: > Arne Vajhøj wrote: > > On 02-05-2010 17:11, Joshua Maurice wrote: > >> On May 1, 10:43 am, Lew<no...(a)lewscanon.com> wrote: > >>> Mike Schilling wrote: > >>>> "The most obvious example of these limitations is that the task > >>>> can't tell which classes to recompile when a constant primitive > >>>> data type exported by other classes is changed. For example, a > >>>> change in the definition of something like > >>>> public final class Constants { > >>>> public final static boolean DEBUG=false; > >>>> } > > >>>> will not be picked up by other classes. " > > >>>> That is, it's an incremental (no pun intended) improvement on the > >>>> usual Ant algorithm of "recompile what onviouslt needs > >>>> recompilation; if that doesn't seem to work, do a clean build" > > >>> You can't blame Ant for that one. The class that depends on the > >>> compile-time constant, such as 'DEBUG' in your example, compiles > >>> the constant into its class, not the symbol. Without some external > >>> indication of the dependency, there's not any way for a compiler or > >>> build tool to detect that it exists. With respect to dependencies where > >>> the symbol is stored in the > >>> class rather than its value, even 'javac' handles the situation > >>> pretty well. > > >> I'm not blaming anyone in particular. I just want to know how to get > >> a fully correct, aka 100% incremental build under the actions: > >> adding, removing, modifying java files, and adding, removing, or > >> modifying build steps of "take these jars, compile them to class > >> files, then jar them", aka the standard developer actions. > > > To me the entire idea is rather pointless. > > > The tool can not be made to work with binaries. > > I think it could. At least, reducing the problem to: > > * You have a complete correct build of the system > * You have a set of changes to the source since that build was done > * Do a minimal (more or less) amount of recompliation to arrive at a new > complete and correct build > > is feasible. > > But I don't see the point, really. It's simpler, cheaper, and more reliable > to throw hardware at the problem. Buy a fast machine to do a continuous > build and archive the last N days worth of builds. You can now fetch a > completely built system at any release level with no compilation required at > all. My company tried to do that, but I think they missed the important part of the memo: that it only works when the code is decoupled, modular, and relatively stable and well defined interfaces instead of the ~25,000 source file mess we have now. It's made me really hate Maven (~800 poms and counting), though I accept it may have situations in which it's a decent build tool. |