Automatically Deducing Build Dependencies => Cross ToolRecursive Builds Work

Recursive make sucks. As presented by the Peter's classic paper "Recursive Make Considered Harmful", http://aegis.sourceforge.net/auug97.pdf

Toolssuch as Peter's Aegis and Cons and Scons "automatically" deduce dependencies for an entire project - but only semi-automatically, since they rely on knowledge of the tools you are using. E.g. they know about C++ #include files, but not about languages that are not on their list. Not about things in your shell scripts, unless you tell them, e.g. in the SConstruct file or by writing a special scanner for your language.

I have long been a fan of truly automatically deducing dependencies: run a build, and use a tool like strace to observe the files that get read and written.

Moreover, their whole project dependency analysis only works if all of the project uses the same build tool.

Unfortunately, that is not true for large projects that have subprojects. I am working on asimulator that was originally built using Make. Later it started using Cons,but kept some make around - i.e. the project starts off as a hybrid of Make and Cons.

But now I want to import tools, libraries, I have written. Reuse, eh? But I used SCons.

Having to rewrite all of my SCons into the project-wide Cons is a barrier to reuse.
Similarly, getting the whole project to use SCons is a barrier to reuse.

Basically, any statement of the form
"All of our problems would be solved if everybody did things the same way"
(a) might be true,
but (b) is bigoted, stupid, intolerant of diversity, etc.
Such statements are the root of all evil, and the cause of much ethnic cleansing.
But, ahem, we are talking about programming, aren't we?

I just realized that true automatic generation of build dependencies
would allow projects to be built up out of different build tools.

Best if all of the build tools used "true observation of dependencies". Then tool #1 would just have to know that it should tool#2 to do the build.

But I think that it can also be used to build a true, project wide, dependency system out of
not-fully-automatic build tools. E.g. this meta-builder could call less intelligent tools such as make, or cons, or scons.

It might also be possible for such a meta-build to call another build tool, in a guaranteed-build-from-scratch situation,
and then observe the commands invoked and rerun those.

I.e. such a meta-build might make recursive make correct.


---

John Ousterhout may have created an automatic build dependency tool,
in his company Electric Cloud.
At least the whitepaper
Solving the Dependency Problem in Software Builds
implies that it has been solved.

Unfortunately, I can't click through to be paper,
so I can't really tell.

Past conversations, either with Ousterhout or with EC sales folk,
lead me
(1) unsure as to whether they have done the truly automatic thing for dependencies
but
(2) sure that they are not "diversity tolerant"
- they are yet another tool that requires everyone, the whole project, to use their build tool.

I.e. they may have invented the automatic dependency idea -although I doubt it.

They do not seem tohave realized that it can be used to allow diversity of build tools.

---

My friend Mark Charney grew frustrated with Cons and SCons, because they could not handle truly dynamic systems that create objects. Mark, therefore, wrote his own build tool that was not limited by the phase ordering restrictions of Cons and SCons.

I wonder if the automatic, diversity tolerant, approach I describe above handles such truly dynamic builds. I suspect taht it does.

(With the usual caveat of there being no time dependent behaviour. It is okay to log a timestamp, but it is not okay to create different numbers of files on different days, just because the day is different. Tools must have the property: same inputs, different day => same outputs, at least same output names - although times may be embedded in the output.)