Incremental Java Compile
in [Java Programming]
|
From: Joshua Maurice on 3 May 2010 14:38 On May 3, 11:36 am, Joshua Maurice <joshuamaur...(a)gmail.com> wrote: > As a developer, I would rather > take the extra 10 min - 1.5 hours to do a full clean build to not have > to debug bizarre obscure issues which result from a clean build. Ack. Typo. It should read "[...] which result from a *inconsistent* build."
From: Joshua Maurice on 3 May 2010 14:40 On May 3, 6:26 am, "Mike Schilling" <mscottschill...(a)hotmail.com> wrote: > Joshua Maurice wrote: > > On May 2, 11:54 pm, "Mike Schilling" <mscottschill...(a)hotmail.com> > > wrote: > >> Arne Vajhøj wrote: > >>> On 02-05-2010 17:11, Joshua Maurice wrote: > >>>> On May 1, 10:43 am, Lew<no...(a)lewscanon.com> wrote: > >>>>> Mike Schilling wrote: > >>>>>> "The most obvious example of these limitations is that the task > >>>>>> can't tell which classes to recompile when a constant primitive > >>>>>> data type exported by other classes is changed. For example, a > >>>>>> change in the definition of something like > >>>>>> public final class Constants { > >>>>>> public final static boolean DEBUG=false; > >>>>>> } > > >>>>>> will not be picked up by other classes. " > > >>>>>> That is, it's an incremental (no pun intended) improvement on the > >>>>>> usual Ant algorithm of "recompile what onviouslt needs > >>>>>> recompilation; if that doesn't seem to work, do a clean build" > > >>>>> You can't blame Ant for that one. The class that depends on the > >>>>> compile-time constant, such as 'DEBUG' in your example, compiles > >>>>> the constant into its class, not the symbol. Without some external > >>>>> indication of the dependency, there's not any way for a compiler > >>>>> or build tool to detect that it exists. With respect to > >>>>> dependencies where the symbol is stored in the > >>>>> class rather than its value, even 'javac' handles the situation > >>>>> pretty well. > > >>>> I'm not blaming anyone in particular. I just want to know how to > >>>> get a fully correct, aka 100% incremental build under the actions: > >>>> adding, removing, modifying java files, and adding, removing, or > >>>> modifying build steps of "take these jars, compile them to class > >>>> files, then jar them", aka the standard developer actions. > > >>> To me the entire idea is rather pointless. > > >>> The tool can not be made to work with binaries. > > >> I think it could. At least, reducing the problem to: > > >> * You have a complete correct build of the system > >> * You have a set of changes to the source since that build was done > >> * Do a minimal (more or less) amount of recompliation to arrive at a > >> new complete and correct build > > >> is feasible. > > >> But I don't see the point, really. It's simpler, cheaper, and more > >> reliable to throw hardware at the problem. Buy a fast machine to do > >> a continuous build and archive the last N days worth of builds. You > >> can now fetch a completely built system at any release level with no > >> compilation required at all. > > > My company tried to do that, but I think they missed the important > > part of the memo: that it only works when the code is decoupled, > > modular, and relatively stable and well defined interfaces instead of > > the ~25,000 source file mess we have now. > > Why doesn't it work, even with the mess you have now? ~3-7 hours turnaround time for any change on the build machine. Double that for the common developer machine. That really hurts productivity. The whole thing is a mess, and some degree of componentization is required, and is being done, but that's still no excuse to have a 30 min compile time for developers for a clean build when they could have 5 seconds + minimal rebuild time. Dittos for the automated build machine.
From: Mike Schilling on 3 May 2010 15:39 Joshua Maurice wrote: > On May 3, 6:26 am, "Mike Schilling" <mscottschill...(a)hotmail.com> > wrote: >> Joshua Maurice wrote: >>> On May 2, 11:54 pm, "Mike Schilling" <mscottschill...(a)hotmail.com> >>> wrote: >>>> Arne Vajh�j wrote: >>>>> On 02-05-2010 17:11, Joshua Maurice wrote: >>>>>> On May 1, 10:43 am, Lew<no...(a)lewscanon.com> wrote: >>>>>>> Mike Schilling wrote: >>>>>>>> "The most obvious example of these limitations is that the task >>>>>>>> can't tell which classes to recompile when a constant primitive >>>>>>>> data type exported by other classes is changed. For example, a >>>>>>>> change in the definition of something like >>>>>>>> public final class Constants { >>>>>>>> public final static boolean DEBUG=false; >>>>>>>> } >> >>>>>>>> will not be picked up by other classes. " >> >>>>>>>> That is, it's an incremental (no pun intended) improvement on >>>>>>>> the usual Ant algorithm of "recompile what onviouslt needs >>>>>>>> recompilation; if that doesn't seem to work, do a clean build" >> >>>>>>> You can't blame Ant for that one. The class that depends on the >>>>>>> compile-time constant, such as 'DEBUG' in your example, compiles >>>>>>> the constant into its class, not the symbol. Without some >>>>>>> external indication of the dependency, there's not any way for >>>>>>> a compiler or build tool to detect that it exists. With respect >>>>>>> to dependencies where the symbol is stored in the >>>>>>> class rather than its value, even 'javac' handles the situation >>>>>>> pretty well. >> >>>>>> I'm not blaming anyone in particular. I just want to know how to >>>>>> get a fully correct, aka 100% incremental build under the >>>>>> actions: adding, removing, modifying java files, and adding, >>>>>> removing, or modifying build steps of "take these jars, compile >>>>>> them to class files, then jar them", aka the standard developer >>>>>> actions. >> >>>>> To me the entire idea is rather pointless. >> >>>>> The tool can not be made to work with binaries. >> >>>> I think it could. At least, reducing the problem to: >> >>>> * You have a complete correct build of the system >>>> * You have a set of changes to the source since that build was done >>>> * Do a minimal (more or less) amount of recompliation to arrive at >>>> a new complete and correct build >> >>>> is feasible. >> >>>> But I don't see the point, really. It's simpler, cheaper, and more >>>> reliable to throw hardware at the problem. Buy a fast machine to do >>>> a continuous build and archive the last N days worth of builds. You >>>> can now fetch a completely built system at any release level with >>>> no compilation required at all. >> >>> My company tried to do that, but I think they missed the important >>> part of the memo: that it only works when the code is decoupled, >>> modular, and relatively stable and well defined interfaces instead >>> of the ~25,000 source file mess we have now. >> >> Why doesn't it work, even with the mess you have now? > > ~3-7 hours turnaround time for any change on the build machine. Double > that for the common developer machine. That really hurts > productivity. > > The whole thing is a mess, and some degree of componentization is > required, and is being done, but that's still no excuse to have a 30 > min compile time for developers for a clean build when they could have > 5 seconds + minimal rebuild time. Dittos for the automated build > machine. Sorry, still confused. Is the recompilation time 30 min. or 3-7 hours?
From: Joshua Maurice on 3 May 2010 16:43 On May 3, 12:39 pm, "Mike Schilling" <mscottschill...(a)hotmail.com> wrote: > Joshua Maurice wrote: > > On May 3, 6:26 am, "Mike Schilling" <mscottschill...(a)hotmail.com> > > wrote: > >> Joshua Maurice wrote: > >>> On May 2, 11:54 pm, "Mike Schilling" <mscottschill...(a)hotmail.com> > >>> wrote: > >>>> Arne Vajhøj wrote: > >>>>> On 02-05-2010 17:11, Joshua Maurice wrote: > >>>>>> On May 1, 10:43 am, Lew<no...(a)lewscanon.com> wrote: > >>>>>>> Mike Schilling wrote: > >>>>>>>> "The most obvious example of these limitations is that the task > >>>>>>>> can't tell which classes to recompile when a constant primitive > >>>>>>>> data type exported by other classes is changed. For example, a > >>>>>>>> change in the definition of something like > >>>>>>>> public final class Constants { > >>>>>>>> public final static boolean DEBUG=false; > >>>>>>>> } > > >>>>>>>> will not be picked up by other classes. " > > >>>>>>>> That is, it's an incremental (no pun intended) improvement on > >>>>>>>> the usual Ant algorithm of "recompile what onviouslt needs > >>>>>>>> recompilation; if that doesn't seem to work, do a clean build" > > >>>>>>> You can't blame Ant for that one. The class that depends on the > >>>>>>> compile-time constant, such as 'DEBUG' in your example, compiles > >>>>>>> the constant into its class, not the symbol. Without some > >>>>>>> external indication of the dependency, there's not any way for > >>>>>>> a compiler or build tool to detect that it exists. With respect > >>>>>>> to dependencies where the symbol is stored in the > >>>>>>> class rather than its value, even 'javac' handles the situation > >>>>>>> pretty well. > > >>>>>> I'm not blaming anyone in particular. I just want to know how to > >>>>>> get a fully correct, aka 100% incremental build under the > >>>>>> actions: adding, removing, modifying java files, and adding, > >>>>>> removing, or modifying build steps of "take these jars, compile > >>>>>> them to class files, then jar them", aka the standard developer > >>>>>> actions. > > >>>>> To me the entire idea is rather pointless. > > >>>>> The tool can not be made to work with binaries. > > >>>> I think it could. At least, reducing the problem to: > > >>>> * You have a complete correct build of the system > >>>> * You have a set of changes to the source since that build was done > >>>> * Do a minimal (more or less) amount of recompliation to arrive at > >>>> a new complete and correct build > > >>>> is feasible. > > >>>> But I don't see the point, really. It's simpler, cheaper, and more > >>>> reliable to throw hardware at the problem. Buy a fast machine to do > >>>> a continuous build and archive the last N days worth of builds. You > >>>> can now fetch a completely built system at any release level with > >>>> no compilation required at all. > > >>> My company tried to do that, but I think they missed the important > >>> part of the memo: that it only works when the code is decoupled, > >>> modular, and relatively stable and well defined interfaces instead > >>> of the ~25,000 source file mess we have now. > > >> Why doesn't it work, even with the mess you have now? > > > ~3-7 hours turnaround time for any change on the build machine. Double > > that for the common developer machine. That really hurts > > productivity. > > > The whole thing is a mess, and some degree of componentization is > > required, and is being done, but that's still no excuse to have a 30 > > min compile time for developers for a clean build when they could have > > 5 seconds + minimal rebuild time. Dittos for the automated build > > machine. > > Sorry, still confused. Is the recompilation time 30 min. or 3-7 hours? Well, 30 min compile only for the hypothetical situation after a "realistic" level componentization, and depending on the level of tests run. Currently ~145 min compile and package, no tests, on the automated build machine. 188 min more for the standard regression / acceptance / integration test suite. Some of the tests are currently distributed across several automated build machines, with the longest suite at 87 min. Double those times, or thereabouts, for a lower end developer computer. Any change requires a full clean build as we no incrementally correct build, and it has not been componentized into smaller chunks. For example, the serialization framework implementation changes slightly frequently, which affects a lot of the code, such as the file persistence, database persistence, engine, and GUI "components". Throwing more hardware at the tests is easy for the automated build machine(s). Throwing more hardware at the compile for the automated build is hard. Throwing more hardware at it for the developer is really hard, and really expensive in cash. (I can't imagine a quick solution to giving the developer 5-10 computers each, and the maintenance nightmare to trying to have them all maintain their own build farm.)
From: Mike Schilling on 3 May 2010 18:46
Joshua Maurice wrote: > Currently ~145 min compile and package, no tests, on the automated > build machine. 188 min more for the standard regression / acceptance / > integration test suite. Some of the tests are currently distributed > across several automated build machines, with the longest suite at 87 > min. Double those times, or thereabouts, for a lower end developer > computer. Any change requires a full clean build as we no > incrementally correct build, and it has not been componentized into > smaller chunks. For example, the serialization framework > implementation changes slightly frequently, which affects a lot of the > code, such as the file persistence, database persistence, engine, and > GUI "components". Does the serialization framework change often? That would be horrific, and there's probably nothing to be done to improve the build cost when it does. But I presume that it changes as the result of some feature being added, so that can be mitigated by not checking the change into source control until the feature (or better yet, set of fesatures) is complete. Also, developers are usually good at optimizing their own work. If a developer is adding new classes or changing implementation rather than interface, there's no need to recompile the world. Even when changing interfaces, the developer usually has a good idea of which bits of the system use those interfaces, and can recompile just those parts. Anyway, I'd suggest: 1. Invest in a good SCM system, one that handles multiple branches and shared branches well. 2. Encourage developers to stay isolated, rather than intergating often and updating other developers' changes often. 3. Do a continuous build that allows developers to grab the most recent complete, tested code, so they can recompile only the code they have checked out and the code that depends on it. Throw lots of hardware at this, so that failures are found early. |