Recognizing errors in build scripts

A very important thing, when writing or some other form of build recipe, is recognizing when an error has occurred that might prevent you going on the the next step (of the build DAG).

This is easy when commands are well behaved.  When they indicate errors tidily. (E.g. in the UNIX convention, 0 for success, nonzero for error.)

It is less easy for tools that are not so well behaved.  When the tool regularly produces errors, both in the UNIX exit code, and in whatever output it sends to stdout or stderr.  When past users seem to have nearly always been running the tool by hand, and there is folklore about what errors can be ignored, and which errors matter.

Better if the folklore is recorded, e.g. on a webpage or wiki. Bad if the folklore is passed by word of mouth.  Worse, if no single person knows all the folklore.

Not so bad if the tool produces a well characterized set of outputs, e.g. output files, and if the file only exists if it was correctly built.  Bad if the output file can be created, but malformed, not completed, etc.

I am tempted to say "BKM: make all output go tgo output.tmp, and then mv to output.final when okay."  But in this same effort I have found several examples of where the output file is perfectly fine, but where an error occured later in the tool producing it.

Not so bad if the output file contains error markers, such as "When processing this record, there was an error".  No matter if this occurs at the very end, or in the middle.  Bad if the output file contains no error markers, so you are left wondering, in a partially created output file, why it stopped.  Really bad if the output file format ignores missing data, such as assuming it is all zeroes.  really, Really, REALLY bad.  I wasted lots of time on that, umm, "feature".

Here are some things that I have done to ciope with such error-ful output:

First, wrapperize the tools that produce unreliable error indications.

Second, in the wrapper or in the build tool, place code to look at the output. Let this "validation" code decide if it is safe to go on to the next step or not.

At first, perhaps unconditionally go on to the next step, since any particular error may be ignorable.

But as you debug stuff, start recognizing error conditions.  When you are confident that an error is non-ignorable, have the wrapper return an error.  When you are confident that the error is ignorable, or that there was no error, have the wrapper return success.

Print messages when you are in-between.  Probably continue anyway, but at least flag when your validation code - typically pattern matching on error messages - thinks that something suspicious has happened, but is not sure.

Be schizophrenic.  Don't try to give a single answer.  Have your validation code do

GOOD: I saw a passed message

BAD:!!: the log file was much tiner than it has ever been on a successful run 

SUSPICIOUS:?: manycompilation errors, but they were ignored in the past.

BAD:!!: the expected output files were not created

Use any heuristic that you can imagine, that you are willing to take the time to code.




Probably most important: save examples of good and bad runs, so that you can start looking for patterns.

 

A good example of how make is needlessly repetitive is how I have been building makefile lines today:

I copy a command line that I pasted into the shell, e.g.

command < file1 -other_input file2 > file3

I paste it into my makefile, twice

command < file1 -other_input file2 > file3
command < file1 -other_input file2 > file3

and then edit

.PHONY: verb-to-run-command
verb-to-run-command: file3
file3: command file1 file2
        command < file1 -other_input file2 > file3

Using namespace - fixing the problem

We just had the "using namespace std, versus std:: all over the place" conversation at work.  The usual: "using namespace" can lead to had to find bugs.

Behavior can change when functions, methods, and operators are imported when a new "using namespace" line is added possibly leading to an import being bound instead of a preexisting function, because of the prioritization implicit in overloading / polymorphism.

(Some of the toughest bugs to find in this area have involved templates. Templates in libray Lib1 using a naked name such as foo(T1,T2), that are used in Lib2, and whose naked name foo gets bound to something in Lib3... up until the time Lib4 is imported via "using namespace", at which time it gets changed.)

I spouted the standard reason not to allow "using namespace".  But I hate myself for doing so, since I find code that is cluttered up by std:: ugly to read.

So of course I try to think about what the real problem is.

The real problem is that some names, functions, operators, methods match multiple prototypes in different libraries.  And the match chosen may change in major ways with minor edits (or, possibly even worse, in minor ways with minor edits).

So, let's look at these one at a time:

First, some names, functions, operators, methods match multiple prototypes in different libraries.

(1) Surely the compiler could compare namespaces, and warn the user about names that might possibly conflict - that might possibly be coerced to different instances in different modules/libraries/namespaces. (Heck, I think that such an ability might be useful even inside the same library/module/namespace.)

(2) Specifically, perhaps an given use of a name that is remapped via overloading/polymorphism could warn if there was more than one possible mapping?  Especially if in separate libraries/modules/namespaces?

Second, the match may change:

(3) Perhaps the editing/build system could record which overloading is applied at every point in the source code.  And warn when it changes.

Like so much source annotation, this would have to be resilient across edits.  But, that's what patch, and so many VCS, do.  Heck, it would even be useful if only inexactly resilient across edits - if there was just a table of source module/name -> dest module/instance mappings, and that was diffed.

Version controlling VCS metadata

My project .hg/hgrc is growing pretty long.

It needs to be under version control.

Unfortunately, hg does not, refuses to, version metadat such as .hg/hgrc.  Security.  Since .hg/hgrc may contain executable hooks...

Seems to me there should be a better way. E.g. disabling it's executability, although allowing it to be versioned.

Or - since some (but by no means all) such metadata is per repo, and is meaningless in a clone, perhaps there should be a meta-VCS system.  This is fairly natural for me, since i have had my repos nested within each other for decades.

Unfortunately, not so natural for Mercurial.

Symlinks to URLs

http://superuser.com/questions/277186/symlink-to-a-url/384182#384182

As people have explained, although you can create a symlink that links to an URL, it does not accomplish much.  UNIX is not smart enough to look at the link content, recognize it as an URL, and then do something appropriate to it.

Nevertheless, I occasionally do create symlinks to URLs.  Mainly just as documentation - I am used to going to looking at ~/links/* when I forget where, on the filesystem, I have stored something.  It's not much of a stretch to do the same thing for URLs.  However, doing so tends to break consistency checks and other stuff that assumess or requires that symlinks point to valid filesystem locations.

Furthermore, I have made some of my own tools smart enough to recognize that, if they see a symlink where an URL is needed, that they should dereference the symlink and look at its contents.  More and more of my tools treat URLs and paths in the filesystem as interchangeable.  Assuming, of course, that the URL protocol has appropriate methods for reading and writing.

[[Write coalescing]] is the term some GPUs, notably AMD/ATI and Nvidia, use to describe how they, umm, combine or coalesce writes from different N different SIMD threads into a single, or at least fewer than N, accesses. There is also [[read coalescing]], and one can imagine other forms of coalescing, such as atomic fetch-and-op coalescing.

At AFDS11 I (Glew) asked an AMD/ATI GPU architect
"What is the difference between [[write coalescing]] and [[write combining]]?"

He replied that [[write combining]] was an x86 CPU feature that used a [[write combining buffer]],
whereas [[write coalescing]] was a GPU feature that performed the optimization between multiple writes that were occurring simultaneously, not in a buffer.

Hmmm...

Since I (Glew) had a lot to do with x86 write combining
- arguably I invented it on P6, although I was inspired by a long line of work in this area,
most notably the [[NYU Ultracomputer]] [[fetch-and-op]] [[combining network]]
- I am not sure that this distinction is fundamental.

Or, rather, it _is_ useful to distinguish between buffer based implementations and implementations that look at simultaneous accesses.

However, in the original NYU terminology, [[combining]] referred to both:
operations received at the same time by a switch in the [[combining network]],
and operations received at a later time that match an operation buffered in the switch,
awaiting either to be forwarded on,
or a reply.
(I'm not sure which was in the Ultracomputer.)

A single P6 processor only did one store per cycle, so a buffer based implementation that performed [[write combining]] between stores
at different times was the only possibility. Or at least the most useful.
Combining stores from different processors was not done (at least, not inside the processor, and could not legally be done to all UC stores).

The NYU Ultracomputer performed this optimization in a switch for multiple processors,
so combining both simultaneous operations and operations performed at different times
was a possibility.

GPUs do many, many, stores at the same time, in a [[data memory coherent]] manner.
This creates a great opportunity for optimizing simultaneous stores.
Although I would be surprised and disappointed to learn that
GPUs did not combine or coalesce
(a) stores from different cycles in the typically 4 cycle wavefront or warp,
and
(b) stores from different SIMD engines, if they encounter each other on the way to memory.

I conclude therefore that the difference between [[write combining]] and [[write coalescing]] is really one of emphasis.
Indeed, this may be yet another example where my
(Glew's) predilection is to [[create new terms by using adjectives]],
e.g. [[write combining buffer]] or [[buffer-based write combining]]
versus [[simultaneous write combining]] (or the [[AFAIK]] hypiothetical special case [[snoop based write combining]]),
rather than creating gratuitous new terminology,
such as [[write combining]] (implicitly restricted to buffer based)
versus [[write coalescing]] (simultaneous, + ...).

= See Also =

This discussion prompts me to create

* [[a vocabulary of terms for memory operation combining]]

Project structure: code, tests, external depndencies

Say you have a project foo, which I will call .../foo, to emphasize that there is probably a project directory somewhere in the namespace.

.../foo

Where do you put the tests?  It's nice to put them in .../foo/tests, so that when you checkout the project, you get the tests as well:

.../foo/tests

by which I mean


.../foo
    .../foo/tests

or


...
   foo
      tests

but  emphasizing the full context via .../ and .../foo/

It's also good to minimize the external dependencies of .../foo.

But what if the tests have more external dependencies than the non-test part of the project.  Should you increase the external dependencies of the non-test part just to have the tests?

Conversely, should you make it harder to write the tests by forbidding external dependencies in them?  Tests are hard enough to write, and they often depend on extra libraries that the source code per se does not?

More specifically, should you get stuff that has increased external dependencies if you check out .../foo and .../foo/tests comes along for the ride?

If you don't want the extra dependencies in .../foo/tests to come along with .../foo, you might structure them as separate modules, possibly separate in the file space:

.../foo+tests
    .../foo
    .../tests


This works, but it creates extra levels of such "meta-modules": .../foo+tests, .../foo+interactive_tests, .../foo+debugging_tools+tests, etc.

You might structure it as a series of optional modules, all within a single metamodule:

.../foo+stuff
    .../foo
    .../tests
    .../stress-tests

or, rather


.../foo+stuff
    .../foo+stuff/foo
    .../foo+stuff/tests
    .../foo+stuff/stress-tests

and so on.

It's still annoying to have an extra level of indirection, but the purity of the bodily fluids of .../foo is maintained.

Now, of course, it is highly likely that you will have some tests within foo and some that are assoviated with foo, but that foo won't let in:

.../foo+stuff
    .../foo+stuff/foo
        .../foo+stuff/foo/tests
    .../foo+stuff/tests
    .../foo+stuff/stress-tests

This can be confusing, but apart from changing names it may be unavoidable: if a country has immigration control country, you often get refugee camps on the border.  The country may ignore them, but the UNHCR does not.

Some systems allow this "stuff" to be overlays within foo:

.../foo

    .../foo/tests -- optional

    .../foo/stress-tests  -- optional