Automatically Deducing Build Dependencies => Cross ToolRecursive Builds Work

Recursive make sucks. As presented by the Peter's classic paper "Recursive Make Considered Harmful", http://aegis.sourceforge.net/auug97.pdf

Toolssuch as Peter's Aegis and Cons and Scons "automatically" deduce dependencies for an entire project - but only semi-automatically, since they rely on knowledge of the tools you are using. E.g. they know about C++ #include files, but not about languages that are not on their list. Not about things in your shell scripts, unless you tell them, e.g. in the SConstruct file or by writing a special scanner for your language.

I have long been a fan of truly automatically deducing dependencies: run a build, and use a tool like strace to observe the files that get read and written.

Moreover, their whole project dependency analysis only works if all of the project uses the same build tool.

Unfortunately, that is not true for large projects that have subprojects. I am working on asimulator that was originally built using Make. Later it started using Cons,but kept some make around - i.e. the project starts off as a hybrid of Make and Cons.

But now I want to import tools, libraries, I have written. Reuse, eh? But I used SCons.

Having to rewrite all of my SCons into the project-wide Cons is a barrier to reuse.
Similarly, getting the whole project to use SCons is a barrier to reuse.

Basically, any statement of the form
"All of our problems would be solved if everybody did things the same way"
(a) might be true,
but (b) is bigoted, stupid, intolerant of diversity, etc.
Such statements are the root of all evil, and the cause of much ethnic cleansing.
But, ahem, we are talking about programming, aren't we?

I just realized that true automatic generation of build dependencies
would allow projects to be built up out of different build tools.

Best if all of the build tools used "true observation of dependencies". Then tool #1 would just have to know that it should tool#2 to do the build.

But I think that it can also be used to build a true, project wide, dependency system out of
not-fully-automatic build tools. E.g. this meta-builder could call less intelligent tools such as make, or cons, or scons.

It might also be possible for such a meta-build to call another build tool, in a guaranteed-build-from-scratch situation,
and then observe the commands invoked and rerun those.

I.e. such a meta-build might make recursive make correct.


---

John Ousterhout may have created an automatic build dependency tool,
in his company Electric Cloud.
At least the whitepaper
Solving the Dependency Problem in Software Builds
implies that it has been solved.

Unfortunately, I can't click through to be paper,
so I can't really tell.

Past conversations, either with Ousterhout or with EC sales folk,
lead me
(1) unsure as to whether they have done the truly automatic thing for dependencies
but
(2) sure that they are not "diversity tolerant"
- they are yet another tool that requires everyone, the whole project, to use their build tool.

I.e. they may have invented the automatic dependency idea -although I doubt it.

They do not seem tohave realized that it can be used to allow diversity of build tools.

---

My friend Mark Charney grew frustrated with Cons and SCons, because they could not handle truly dynamic systems that create objects. Mark, therefore, wrote his own build tool that was not limited by the phase ordering restrictions of Cons and SCons.

I wonder if the automatic, diversity tolerant, approach I describe above handles such truly dynamic builds. I suspect taht it does.

(With the usual caveat of there being no time dependent behaviour. It is okay to log a timestamp, but it is not okay to create different numbers of files on different days, just because the day is different. Tools must have the property: same inputs, different day => same outputs, at least same output names - although times may be embedded in the output.)

I have a new BKM for argument list parsing in Perl, similar to my longstanding BKM in C/C++:

my $argv = {};

for( my @tmp_ARGV = @ARGV; my $arg = shift @tmp_ARGV; ) {
    if(0) {}
    elsif( $arg eq "-short" ) {
 $argv->{test_length} = 'short';
    }
    elsif( $arg eq "-full" ) {
 $argv->{test_length} = 'short';
    }
    else {
 die "unrecognized command line argument: <$arg>";
    }
}

I rather like creating a hash of options,
but if you prefer variables:

my $short;

for( my @tmp_ARGV = @ARGV; my $arg = shift @tmp_ARGV; ) {
    if(0) {}
    elsif( $arg eq "-short" ) {
 $short = 1;
    }
    elsif( $arg eq "-full" ) {
 $short = 0;
    }
    else {
 die "unrecognized command line argument: <$arg>";
    }
}

This can be made extensible, by calling subfunctions to parse, so-called argv-recognizers. These can recognize individual options, or parse a long way.

my $short;

for( my @tmp_ARGV = @ARGV; my $arg = shift @tmp_ARGV; ) {
    if(0) {}
    elsif( $arg eq "-short" ) {
 $short = 1;
    }
    elsif( argv_parser1(\@tmp_ARGV) {}
    elsif( argv_parser2(\@tmp_ARGV) ) {}
    else {
 die "unrecognized command line argument: <$arg>";
    }
}

The following may be more uniform:

for( my @tmp_ARGV = @ARGV; my $arg = @tmp_ARGV[0]; shift @tmp_ARGV ) {
   ...
}

---

Why use this instead of Getopts? Getopts::Lomng? Etc.

Mainly

a) Getopts does not compose modules with different parsers

b) the above is trivial, easy to remember, and can be extended to handle any different argument syntax.

I want an apprentice

I want an apprentice.

I want a junior program who can work with me. Not necessarily pair programming.

An apprentice at my beck and call, who I can call over to my screen when I realize "Here is a little bit of automation that needs to be doing" - who I can explain, verbally, what needs to be done, and whose job it is to make enough notes, and then just do it.

There are always a lot of little programming tasks that need to be done, but which I deliberately neglect in the rush to get urgent, quick and dirty, results. Darn! That may be a career limiting admission: sometimes I do not go and clean up the code or write utility functions when I see the need. Most people would accuse me of not being quick and dirty enough.

It's hard to balance "Getting to the answer as quickly as possible" and "programming well".

part of the problem is that, as a computer architect, my output is not really the simulator. My output is the hardware recommendation, the perf data, the simulation results. The simulator is just a tool, and if it were throw-away, that would be great. But, the simulator is not thrown away, except maybe once a decade. It is an asset that is constaly built upon.

And it is hard to balance getting the answers vers keeping the simulator clean.

Many programmers don't have this problem.

Usually because they simply just aren't as good programmers as me. They don't know how to do things right.

But sometimes because they don't know some of the really fancy quick and dirty programming tricks that I know.

I.e. because I am a pretty good programmer, better than most, I can write really good code -or really quick and dirty code.

The balance is hard. Sometimes I am rushed.

Hence the apprentice. If I had two apprentices, one could be devoted to pair programming, and the other to "doing the right thing". Basically dedicating 50% of apprentice resources to making the code better.

Plus, the apprentice making the code better does not have to thrash. His job is to make it better. Period.

Why RAII is convenient

RAII is convenient in C++.

Essentially, it is the trick of using the destructor to do something like arrange flr printing of a data structure.

Essentially, it is a trivial form of introspection.Practically the only way in C++ than you can automatically traverse a type, and al of its members.

Today, I would like to automatically create a print function. I am tempted to use RAII, so that it gets printed at destruction. That seems silly, however; but it is the only way I know to autromatically call a method on all members.

Change

I fear this is the start of one of those days.

I'm working on Saturday, instead of getting a Christmas tree, putting up Christmas lights, etc. Big review meeting on the 22nd or 23rd.

Walk the dogs at 5am. That was okay.

Can't find hummus in the only grocery store open at 6am.

The UsualPC Startup Hassles:

At work at 6:30. The usual PC startup hassles, which I am sure that anyone who has ever read my blog is bored with, but which venting about keeps me sane. Bottom line: I'm just starting to work at 7:35.

Laptop PC comes out of standby. The usual circa 5-10 minutes of inability to type. I go and get tabouli from the fridge.

Laptop responsive, but slow. The Connected DataProtector backup is running. This backupprogram has a dialog box with a cancel button, but experience has shown that it doesn't respond quickly to the cancel button - 10 minutes being not unusual. Experience has shown that the fastest way to cancel Connected DataProtector is to restart the PC. (Killing the process damages some database that the backup program uses.) So, I hit the cancel button, and also try to go Start/Shutdown/Restart, which takes about 6 minutes because the machine is so slow. But, as usual, restart happens before Connected DataProtector has noticed that I wanted to cancel.

(My laptop was plugged in at home all night - but the backup program was apparently unable to detect idleness and thus did not start until after aI reopened my PC at work.)

Along the way, noticed a new IT patch, probably a security patch, is also slowing things down.

Restart eventually happens. I log in, and then walk away - I am learning that I might as well go and get a tea while waiting for all of the startup and login stuff to settle down.

Company now has free drinks (coffee, tea, and soda) as well as free fruit. One of the nice things that happened since the new guy became CEO. But, no free drinks on the weekend, only when the cafeteria is open.

Disincentives to Working Extra Hours:

As I walk off to get my tea, I think to myself "Why should I bother coming in on the weekend?"

a) My computer is so slow - I will have wasted half an hour (that was before I realized it was turning into a full hour wasted) just waiting for my PC to become usable.

During the work week, say, if I come in at 9am, I can usually find somebody to talk to while waiting for my computer. In fact, such informal, drop-by-their-cubicle, conversations are one of the best ways of staying in touch. Not just socialization, but work related conversations, asking for help, advice. But, if you come in on the weekend, or even early on a workday (say, 6am, or even 8am), this way of overlapping the time wasting of PC startup is lost.

b) I can't even get free tea when the cafeteria isn't open.

Two disincentives for working extra hours on the weekend.

Change:

But, more: I find a vending machine to buy a back of almonds and tea.

The almonds cost 1.10$. I use a 20$ bill. Usually the vending machine provides change mainly in Sacagawea dollar coins, but today:

20$ - 1.10$ = 18.90$
= 4.25$ in quarters + 10$ in dimes + 2.65$ in nickels
= 16.90$ => I was shortchanged 2$

Yes: I sat on the floor and counted all 170 coins. Do you know how hard it is to make 20 nickels into a stack?

When I was young, I used to obsessively count my change. I do so less often now, in part because I use credit card so often. I wonder how often I have been short-changed or long-totalled?

...

But even after all of this, my PC is not quite usable.

Are Slow PCs good for the Environment?:

I probably should just not get stressed about my PC startup times. I probably should just plan on starting my PC, and going and getting tea first thing in the morning.

Currently, I try to save time by parking such that I walk by the cafeteria to get tea on the way between my car and my desk. Unfortunately, I do not carry a reusable mug with me,so I end up usimng a disposable paper mug. I.e. just adapting to my slow laptop might be good for the environment: park, go to my desk, start my PC, pick up my mug, walk to the cafeteria, get tea, walk back to my desk, and, one might hope, find that my PC is up and ready. Although probably it is asking for me to click on some dialog box.

Getting off to a Quick Start

Why does slowness in PC startup bother me so much?

I am a morning person. I wake up raring to go, wanting to start work RIGHT NOW!

I have often thought about a solution to a problem overnight - often while drowsing in bed. Should I admit that I often dream about work problems, computer designs?

When I get up, I want to start work right away. If I don't, sometimes the solution drifts away.

I resent something so basic as having to feed and walk my dogs in the morning. I resent wasting time driving to work. Actually, I more resent time spent in busy work such as feeding the dogs; I can think about work in activities such as walking the dogs that I can do fairly automatically, although there is always a risk of forgetting.

I used to wake up regularly early in the morning or in the middle of the night and hope ontomy computer. Now, since it takes a randomly long time to start up, I no longer do so.

I resent the fact that I get so frustrated by my PC's slowness that I am impelled to blog about this. Writing this blog has taken 45 minutes. I would rather not have this to bitch about.

I would rather that I had speech recognition, so that I could speak this blog instead of typing it. I speak much faster than I type.

Resuable Mugs:

Recently, a few months after my company started the free drinks policy, signs started appearing saying "Please use your reusable mug instead of paper."

That's fair enough - although, as I mentioned above, one does not always carry the mug around. I already carry enough crap around that I roll my bag.

Methinks it would be better for the cafeteria just to provide washable mugs: I go to the cafeteria, pick up a mug. Probably with a lid. Drink it. Eventually, leave it in the vending area along with the food trays. Eventually it gets taken back to the cafeteria and gets washed.

Downside: more cafeteria staff effort required to pick up the mugs. Some possibility of theft, lossage, breakage. I suppose that giving people their own mugs moves responsibility to the end user.

But then don't bitch if you don't get as much reuse as you might like.

Change people's habits, or adapt to them. I am people centered: I prefer to adapt the system to the people, rather than vice versa.

I bitch a lot, but what can be done about it?

I bitch a lot about how slow my computer is, but what can be done about it.

First, my job is to make computers faster. I am working on one aspect of speeding this up - but I am frustrated that executives do not perceive that this is a problem. Probably because they have faster, newer, computers than me.

Second, I wish that my IT department would give me a desktop PC rather than a laptop. The point of my bitching is that my laptop is becoming less and less useful with these slownesses. My productivity at work would be sped up having a desktop, that did not need to go slow as often.

Actually, I would rather have a Linux PC than a Windows PC. All I really need is an X Windows terminal on my desk that I can open up on some interactive compute server. Such a configuration probably saves money. Or, if you have desktop PCs, the desktops can be used for "cloud" computing.

I wish that my IT department would allow me to "dial in" to work using my own PC. Anything to avoid the slowness of a Windows PC with an IT load of software coming out of standby.

Third, I would like my IT department to seriously consider giving me a device like a BlackBerry. Many of my supposedly quick attempts to access my computer are to check my calendar, with checking email a little bit less frequent. Something like a BlackBerry would help (at least until BlackBerry gets lots of virus scanners).

Since my IT department does not want to spend the money for me to have a BackBerry, I wish that my IT department would allow me to purchase a BlackBerry, and allow me to spend the money, circa 50$/month, to have it connected to the corporate network.

Finally, again on a personal level, since my employer's IT department doesn't seem to be willing to do anything to resolve these slownesses, at some point I am going to have to follow the adage "If you can't change your organization, change your organization." I want to see my current project to an end, but after this (which may be 5 years from now) I will go looking for a new employer unless this IT stupidity is resolved. Criterion for such a new employer, apart from the usual interesting work, and sufficient money, will definitely include IT support:
* "instant-on" mobile PCs
* dedicated desktop screen access to compute servers
* ability to dial in from a personal computer, not a corporate laptop
* remote access to calendar (and email) from a SmartPhone
* willingness to upgrade my PC more than once every 4 years

This economic downturn is probably not the time to be ranting, in public, threatening to leave my job. But this IT situation is driving me crazy!!!!!

I don't want to leave my job before the end of my current project. I like seeing my projects through to completion. But it is almost ready to roll on its own - it no longer needs the original inventor and architect (me), to drive it along.

So, if a recruiter and headhunter wants to contact me about some other company --- well, if you can promise me a better IT situation, I will seriously consider it.

However, although I get a headhunter circa once a week, I doubt very much that any of them will google me this much. And I will probably scare off any mediocre hiring exec, who will read this public bitching. So be it! I do not want to work at such a company anyway.

---

But, overall, I hope that my project, and others, will improve this crappy situation.

More slowness coming out of standby?

Meetings all morning.

At 12:08 start trying to use laptop - bring it out of standby.

I can't read Outlook email until 12:36.

@8 minutes. No rebooting, no patches (that I know about). Just slowness.

So much for "quickly checking my email".