Incremental Java Compile
in [Java Programming]
|
From: Mike Schilling on 4 May 2010 00:40 Joshua Maurice wrote: > On May 3, 3:46 pm, "Mike Schilling" <mscottschill...(a)hotmail.com> > wrote: >> Joshua Maurice wrote: >>> Currently ~145 min compile and package, no tests, on the automated >>> build machine. 188 min more for the standard regression / >>> acceptance / integration test suite. Some of the tests are >>> currently distributed across several automated build machines, with >>> the longest suite at 87 min. Double those times, or thereabouts, >>> for a lower end developer computer. Any change requires a full >>> clean build as we no incrementally correct build, and it has not >>> been componentized into smaller chunks. For example, the >>> serialization framework implementation changes slightly frequently, >>> which affects a lot of the code, such as the file persistence, >>> database persistence, engine, and GUI "components". >> >> Does the serialization framework change often? That would be >> horrific, and there's probably nothing to be done to improve the >> build cost when it does. But I presume that it changes as the result >> of some feature being added, so that can be mitigated by not >> checking the change into source control until the feature (or better >> yet, set of fesatures) is complete. > > I wish I knew. I just got an email today from the serialization team > asking "What's with this error?" I "hacked" the C++ Maven plugin to > report "<> has detected visual studios warning <>, deletion of a > pointer to an incomplete type. This is formally undefined behavior > according to the C++ spec. Fix it." Apparently changes are still > ongoing. > >> Also, developers are usually good at optimizing their own work. If a >> developer is adding new classes or changing implementation rather >> than interface, there's no need to recompile the world. Even when >> changing interfaces, the developer usually has a good idea of which >> bits of the system use those interfaces, and can recompile just >> those parts. > > As such a developer, perhaps, but when I mess up, I break the mainline > build, and because the build on the automated build machine, or > private perforce branch build machine, can take the better part of a > day, it's sometimes hard to isolate down who broke it, and especially > when ML is broken this leaves people in a bind. Currently we lock ML > on such events. Rollback is possible. Devops is floating that idea > around at the moment. You only break the mainline build if you check in code based on doing that incorrectly. I'm not suggesting that. > >> Anyway, I'd suggest: >> >> 1. Invest in a good SCM system, one that handles multiple branches >> and shared branches well. > > Done. Perforce is so awesome for the record. It is. They do a hell of a job (and I don't say that just because I know a lot of the folks there.) > >> 2. Encourage developers to stay isolated, rather than intergating >> often and updating other developers' changes often. > > Sounds like integration hell. We do have separate teams working on > their own little view for weeks or a month or two on end, and each > team has their own private branch in perforce which is integrated > roughly weekly with mainline. It shouldn't be hell, especially with a good tool like Perforce helping with any merges that result. Though if there's a lot of churn in the code and everything uses everything else, yeah, it'll be harder than if things were stable and well-organized. > >> 3. Do a continuous build that allows developers to grab the most >> recent complete, tested code, so they can recompile only the code >> they have checked out and the code that depends on it. Throw lots of >> hardware at this, so that failures are found early. > > Also done. > > The problem is that it's not helping. It's way too much code, way too > many tests, taking way too long to build.
From: Mike Schilling on 4 May 2010 00:57 Arne Vajh�j wrote: > On 03-05-2010 02:54, Mike Schilling wrote: >> Arne Vajh�j wrote: >>> On 02-05-2010 17:11, Joshua Maurice wrote: >>>> On May 1, 10:43 am, Lew<no...(a)lewscanon.com> wrote: >>>>> Mike Schilling wrote: >>>>>> "The most obvious example of these limitations is that the task >>>>>> can't tell which classes to recompile when a constant primitive >>>>>> data type exported by other classes is changed. For example, a >>>>>> change in the definition of something like >>>>>> public final class Constants { >>>>>> public final static boolean DEBUG=false; >>>>>> } >>>>> >>>>>> will not be picked up by other classes. " >>>>> >>>>>> That is, it's an incremental (no pun intended) improvement on the >>>>>> usual Ant algorithm of "recompile what onviouslt needs >>>>>> recompilation; if that doesn't seem to work, do a clean build" >>>>> >>>>> You can't blame Ant for that one. The class that depends on the >>>>> compile-time constant, such as 'DEBUG' in your example, compiles >>>>> the constant into its class, not the symbol. Without some >>>>> external indication of the dependency, there's not any way for a >>>>> compiler or build tool to detect that it exists. With respect to >>>>> dependencies where the symbol is stored in the >>>>> class rather than its value, even 'javac' handles the situation >>>>> pretty well. >>>> >>>> I'm not blaming anyone in particular. I just want to know how to >>>> get a fully correct, aka 100% incremental build under the actions: >>>> adding, removing, modifying java files, and adding, removing, or >>>> modifying build steps of "take these jars, compile them to class >>>> files, then jar them", aka the standard developer actions. >>> >>> To me the entire idea is rather pointless. >>> >>> The tool can not be made to work with binaries. >> >> I think it could. > > As I recall it then the conclusion was that correct handling > of constants (static final) required source- You mean analysis of the source is required, because no traces of the constant use are left in the class file? You know, I assumed that such constants would be listed in the constant pool, but (having tried a simple test) I see that that's wrong (and you're correct.). Bad decision.
From: Mike Schilling on 4 May 2010 00:59 Joshua Cranmer wrote: > On 05/03/2010 08:18 PM, Joshua Maurice wrote: >> I am fixing it. I am not whining. I was asking for help on how to do >> it. I have asked for real solutions to the real problems I am facing >> writing it, such as how to get a list of class files per compiled >> java file as if I called javac once per java file in the dir. > > final static int constants (or other constant types that ldc works on) > are directly hardcoded into the class file. And no trace of their origin is written to the class file. > It is therefore impossible > to read a classfile and tell you which Java classes would have to > change for it to need to be recompiled. Because the designers of Java didn't consider that important, or the necessary information would have been written to the class file.
From: Joshua Maurice on 4 May 2010 03:09 On May 3, 9:59 pm, "Mike Schilling" <mscottschill...(a)hotmail.com> wrote: > Joshua Cranmer wrote: > > It is therefore impossible > > to read a classfile and tell you which Java classes would have to > > change for it to need to be recompiled. > > Because the designers of Java didn't consider that important, or the > necessary information would have been written to the class file. And because such information alone is insufficient to do a correct incremental compile. See the paper in my opening post "Ghost Dependencies" or some such. It would go a long way to helping me make it correct though if javac spat out "loading class X when compiling java file Y" for all such pairs.
From: Joshua Maurice on 4 May 2010 03:12
On May 3, 9:40 pm, "Mike Schilling" <mscottschill...(a)hotmail.com> wrote: > Joshua Maurice wrote: > > On May 3, 3:46 pm, "Mike Schilling" <mscottschill...(a)hotmail.com> > >> Also, developers are usually good at optimizing their own work. If a > >> developer is adding new classes or changing implementation rather > >> than interface, there's no need to recompile the world. Even when > >> changing interfaces, the developer usually has a good idea of which > >> bits of the system use those interfaces, and can recompile just > >> those parts. > > > As such a developer, perhaps, but when I mess up, I break the mainline > > build, and because the build on the automated build machine, or > > private perforce branch build machine, can take the better part of a > > day, it's sometimes hard to isolate down who broke it, and especially > > when ML is broken this leaves people in a bind. Currently we lock ML > > on such events. Rollback is possible. Devops is floating that idea > > around at the moment. > > You only break the mainline build if you check in code based on doing that > incorrectly. I'm not suggesting that. Interesting idea. "Don't break the mainline build!" says the managers. Unfortunately, if they're unable to build to a documented interface, and the whole build takes hours, if not longer, on their own computer, then it's hard in practice to not break it. Just saying. |