Incremental Java Compile
in [Java Programming]
|
From: Mike Schilling on 4 May 2010 10:35 Joshua Maurice wrote: > On May 3, 9:40 pm, "Mike Schilling" <mscottschill...(a)hotmail.com> > wrote: >> Joshua Maurice wrote: >>> On May 3, 3:46 pm, "Mike Schilling" <mscottschill...(a)hotmail.com> >>>> Also, developers are usually good at optimizing their own work. If >>>> a developer is adding new classes or changing implementation rather >>>> than interface, there's no need to recompile the world. Even when >>>> changing interfaces, the developer usually has a good idea of which >>>> bits of the system use those interfaces, and can recompile just >>>> those parts. >> >>> As such a developer, perhaps, but when I mess up, I break the >>> mainline build, and because the build on the automated build >>> machine, or private perforce branch build machine, can take the >>> better part of a day, it's sometimes hard to isolate down who broke >>> it, and especially when ML is broken this leaves people in a bind. >>> Currently we lock ML on such events. Rollback is possible. Devops >>> is floating that idea around at the moment. >> >> You only break the mainline build if you check in code based on >> doing that incorrectly. I'm not suggesting that. > > Interesting idea. "Don't break the mainline build!" says the managers. > Unfortunately, if they're unable to build to a documented interface, > and the whole build takes hours, if not longer, on their own computer, > then it's hard in practice to not break it. Stay isolated longer. Do the full build and test less often.
From: Arne Vajhøj on 4 May 2010 19:44 On 04-05-2010 00:57, Mike Schilling wrote: > Arne Vajh�j wrote: >> On 03-05-2010 02:54, Mike Schilling wrote: >>> Arne Vajh�j wrote: >>>> On 02-05-2010 17:11, Joshua Maurice wrote: >>>>> On May 1, 10:43 am, Lew<no...(a)lewscanon.com> wrote: >>>>>> Mike Schilling wrote: >>>>>>> "The most obvious example of these limitations is that the task >>>>>>> can't tell which classes to recompile when a constant primitive >>>>>>> data type exported by other classes is changed. For example, a >>>>>>> change in the definition of something like >>>>>>> public final class Constants { >>>>>>> public final static boolean DEBUG=false; >>>>>>> } >>>>>> >>>>>>> will not be picked up by other classes. " >>>>>> >>>>>>> That is, it's an incremental (no pun intended) improvement on the >>>>>>> usual Ant algorithm of "recompile what onviouslt needs >>>>>>> recompilation; if that doesn't seem to work, do a clean build" >>>>>> >>>>>> You can't blame Ant for that one. The class that depends on the >>>>>> compile-time constant, such as 'DEBUG' in your example, compiles >>>>>> the constant into its class, not the symbol. Without some >>>>>> external indication of the dependency, there's not any way for a >>>>>> compiler or build tool to detect that it exists. With respect to >>>>>> dependencies where the symbol is stored in the >>>>>> class rather than its value, even 'javac' handles the situation >>>>>> pretty well. >>>>> >>>>> I'm not blaming anyone in particular. I just want to know how to >>>>> get a fully correct, aka 100% incremental build under the actions: >>>>> adding, removing, modifying java files, and adding, removing, or >>>>> modifying build steps of "take these jars, compile them to class >>>>> files, then jar them", aka the standard developer actions. >>>> >>>> To me the entire idea is rather pointless. >>>> >>>> The tool can not be made to work with binaries. >>> >>> I think it could. >> >> As I recall it then the conclusion was that correct handling >> of constants (static final) required source- > > You mean analysis of the source is required, because no traces of the > constant use are left in the class file? You know, I assumed that such > constants would be listed in the constant pool, but (having tried a simple > test) I see that that's wrong (and you're correct.). Bad decision. If I remember correct then const in C# is the same way. Arne
From: Roedy Green on 5 May 2010 20:03 On Mon, 3 May 2010 11:36:43 -0700 (PDT), Joshua Maurice <joshuamaurice(a)gmail.com> wrote, quoted or indirectly quoted someone who said : > >Did you even read any of my other posts in this thread? Ant's >incremental compile is woefully incorrect, so incorrect as to be near >useless on an automated build machine. In the early days the company I worked for a company that had a guy who did nothing but tweak compile scripts using traditional MAKE-like logic. They were INCREDIBLY slow compared with ANT. The errors Javac/ANT makes in deciding what to recompile make little difference compared to the massive speedup of loading Javac.exe only once. They are also insignificant compared with jar and zip time. If you fiddle non-private static finals, remember to do a clean compile of the universe. Other than that, for all practical purposes, ANT works. -- Roedy Green Canadian Mind Products http://mindprod.com What is the point of a surveillance camera with insufficient resolution to identify culprits?
From: Joshua Maurice on 5 May 2010 23:06 On May 5, 5:03 pm, Roedy Green <see_webs...(a)mindprod.com.invalid> wrote: > On Mon, 3 May 2010 11:36:43 -0700 (PDT), Joshua Maurice > <joshuamaur...(a)gmail.com> wrote, > >Did you even read any of my other posts in this thread? Ant's > >incremental compile is woefully incorrect, so incorrect as to be near > >useless on an automated build machine. > > In the early days the company I worked for a company that had a guy > who did nothing but tweak compile scripts using traditional MAKE-like > logic. They were INCREDIBLY slow compared with ANT. The errors > Javac/ANT makes in deciding what to recompile make little difference > compared to the massive speedup of loading Javac.exe only once. They > are also insignificant compared with jar and zip time. > > If you fiddle non-private static finals, remember to do a clean > compile of the universe. Other than that, for all practical purposes, > ANT works. No, no, and no. Perhaps you'll listen this time. First, my solution only loads javac once. It would be silly to do otherwise. Moreover, calling javac once per java file is quite slow, and I was asking for a way around that to get fast, incrementally correct java compiles. However, even with calling javac once per java, I still outperformed Maven for a full clean build. Also, do you have any numbers at all to support your proposition that jar-ing and zip-ing is the time sucker? Any sources, your own or otherwise? For a sizable portion of my company's product, with my new system without incremental dependency analysis, I was able to compile ~3,000 java files in ~3 min. The jar-ing of the resultant class files took ~15 seconds, and ~8 seconds of that was simply my build system overhead. It seems that jar-ing is \much\ faster than java compilation for standard hardware. At least, is it with the -O option to jar, the "do not compress" option, which should be the standard option during development. Finally, no. Ant does not work all of the time for all practical purposes. I can list off numerous times from the last month where our "streaming build" because it uses such poor dependency analysis techniques. It required manual intervention by devops to do a clean to get it to start passing, and I can assure you static finals were not the dominant cause. (Though, admittingly, Ant probably does a better job than Maven.) However, even then, static finals are part of the language, and I want an automated build which can actually do automated builds. The build machine has no (easy) way to determine if a static final was changed or not, or other (potentially obscure) scenarios which Ant would fail on. When I as a developer update to a new revision / changelist using perforce, I do not have an (easy) way to check if there was a change to a static final, so I would have to do a clean build. This is not acceptable if there's a feasible alternative. Also, QA now will never take such a dubiously correct build after having been burned many times by incrementally incorrect builds, wasting days of stress testing because the build was "incorrectly" done. I will not continue if you just repeat these unfounded and inaccurate assessments, especially if you do not cite any sources at all, even your own. For example, I have asked of your own timing numbers for your own builds for java compilation vs jar -0 times, and you have yet to provide any.
From: Lew on 5 May 2010 23:34
Joshua Maurice wrote: > No, no, and no. Perhaps you'll listen this time. And perhaps you won't be so damn rude next time. What the hell? You have consistently rejected every piece of good advice, given complete nonsense excuses for doing so, and thrown mud in the face of people who try to help you. What a piece of work! -- Lew |