Disclaimer

The content of this blog is my personal opinion only. Although I am an employee - currently of Nvidia, in the past of other companies such as Iagination Technologies, MIPS, Intellectual Ventures, Intel, AMD, Motorola, and Gould - I reveal this only so that the reader may account for any possible bias I may have towards my employer's products. The statements I make here in no way represent my employer's position, nor am I authorized to speak on behalf of my employer. In fact, this posting may not even represent my personal opinion, since occasionally I play devil's advocate.

See http://docs.google.com/View?id=dcxddbtr_23cg5thdfj for photo credits.

Monday, February 06, 2012

Why I like XML

In XML, or pseudo XML, you can see the matching brackets

<matching bracket> stuff
</matching >

whereas in simply parenthesized notations like JSON

{n1: {n2: n3: "do you have enough matching brackets?" }}}

?


You can usefully define semantics for mismatching constructs. XML also allows unquoted text, because of the very verbosity of its punctuation.

Using = and : binding operators to delineate structure

I have messed around with "pseudo-XML" a lot. Oh, heck, whop cares about the XML part, except that it is a standard?  Structured, simple, data.  Trying to make stuff human readable, but also parseable.

Here's a new-to-me trick:

I often use key=value pairs.

Sometimes I use name: value.

I.e. both = and : are nice ways of binding name and value.

How about:
n1: nA=vA, nB=vB, nC=vC
=>  nA=vA, nB=vB, nC=vC
=>    
or whatever way you want to express name/value in XML.

I.e. : as a binding operator that can bind to a list of name=value pairs

Or non-XML, if that's what tickles your fancy.  JSON?:
{ "n1": {"nA": "vA", "nB": "vB", "nC": "vC" } }
(BTW, if I ever declare that I am doing pseudo-JSON, it will be to have fewer quotes.)

{ n1: {nA: vA, nB: vB, nC: vC } }

The biggest difference between shell script languages and regular languages is quoting. "Barewords" are a big characteristic of scripting languages.



:s can bind until the next colon - i.e, all colons have the same strength
n1: nA=vA, nB=vB, n2: v2, n3: nX=vX, nY=vY
{ n1: { nA:vA, nB:vB}, n2:v2, n3: {nX:vX, nY: vY} }
To get more levels deep, would need brackets, or indentation. more operators: ::==



Heck, I'm not trying to define a full language here.  There are at least 3 perfectly good data languages - XML, JSON, S-expressions.

I just want to define some notation that is a bit more user friendly, but can fall back to the other notations if necessary.

Saturday, February 04, 2012

Recognizing errors in build scripts

A very important thing, when writing or some other form of build recipe, is recognizing when an error has occurred that might prevent you going on the the next step (of the build DAG).

This is easy when commands are well behaved.  When they indicate errors tidily. (E.g. in the UNIX convention, 0 for success, nonzero for error.)

It is less easy for tools that are not so well behaved.  When the tool regularly produces errors, both in the UNIX exit code, and in whatever output it sends to stdout or stderr.  When past users seem to have nearly always been running the tool by hand, and there is folklore about what errors can be ignored, and which errors matter.

Better if the folklore is recorded, e.g. on a webpage or wiki. Bad if the folklore is passed by word of mouth.  Worse, if no single person knows all the folklore.

Not so bad if the tool produces a well characterized set of outputs, e.g. output files, and if the file only exists if it was correctly built.  Bad if the output file can be created, but malformed, not completed, etc.

I am tempted to say "BKM: make all output go tgo output.tmp, and then mv to output.final when okay."  But in this same effort I have found several examples of where the output file is perfectly fine, but where an error occured later in the tool producing it.

Not so bad if the output file contains error markers, such as "When processing this record, there was an error".  No matter if this occurs at the very end, or in the middle.  Bad if the output file contains no error markers, so you are left wondering, in a partially created output file, why it stopped.  Really bad if the output file format ignores missing data, such as assuming it is all zeroes.  really, Really, REALLY bad.  I wasted lots of time on that, umm, "feature".

Here are some things that I have done to ciope with such error-ful output:

First, wrapperize the tools that produce unreliable error indications.

Second, in the wrapper or in the build tool, place code to look at the output. Let this "validation" code decide if it is safe to go on to the next step or not.

At first, perhaps unconditionally go on to the next step, since any particular error may be ignorable.

But as you debug stuff, start recognizing error conditions.  When you are confident that an error is non-ignorable, have the wrapper return an error.  When you are confident that the error is ignorable, or that there was no error, have the wrapper return success.

Print messages when you are in-between.  Probably continue anyway, but at least flag when your validation code - typically pattern matching on error messages - thinks that something suspicious has happened, but is not sure.

Be schizophrenic.  Don't try to give a single answer.  Have your validation code do

GOOD: I saw a passed message
BAD:!!: the log file was much tiner than it has ever been on a successful run 
SUSPICIOUS:?: manycompilation errors, but they were ignored in the past.
BAD:!!: the expected output files were not created
Use any heuristic that you can imagine, that you are willing to take the time to code.




Probably most important: save examples of good and bad runs, so that you can start looking for patterns.
 



A good example of how make is needlessly repetitive is how I have been building makefile lines today:

I copy a command line that I pasted into the shell, e.g.

command < file1 -other_input file2 > file3

I paste it into my makefile, twice

command < file1 -other_input file2 > file3
command < file1 -other_input file2 > file3

and then edit

.PHONY: verb-to-run-command
verb-to-run-command: file3
file3: command file1 file2
        command < file1 -other_input file2 > file3

Wednesday, February 01, 2012

Using namespace - fixing the problem

We just had the "using namespace std, versus std:: all over the place" conversation at work.  The usual: "using namespace" can lead to had to find bugs.

Behavior can change when functions, methods, and operators are imported when a new "using namespace" line is added possibly leading to an import being bound instead of a preexisting function, because of the prioritization implicit in overloading / polymorphism.

(Some of the toughest bugs to find in this area have involved templates. Templates in libray Lib1 using a naked name such as foo(T1,T2), that are used in Lib2, and whose naked name foo gets bound to something in Lib3... up until the time Lib4 is imported via "using namespace", at which time it gets changed.)

I spouted the standard reason not to allow "using namespace".  But I hate myself for doing so, since I find code that is cluttered up by std:: ugly to read.

So of course I try to think about what the real problem is.

The real problem is that some names, functions, operators, methods match multiple prototypes in different libraries.  And the match chosen may change in major ways with minor edits (or, possibly even worse, in minor ways with minor edits).

So, let's look at these one at a time:

First, some names, functions, operators, methods match multiple prototypes in different libraries.

(1) Surely the compiler could compare namespaces, and warn the user about names that might possibly conflict - that might possibly be coerced to different instances in different modules/libraries/namespaces. (Heck, I think that such an ability might be useful even inside the same library/module/namespace.)

(2) Specifically, perhaps an given use of a name that is remapped via overloading/polymorphism could warn if there was more than one possible mapping?  Especially if in separate libraries/modules/namespaces?

Second, the match may change:

(3) Perhaps the editing/build system could record which overloading is applied at every point in the source code.  And warn when it changes.

Like so much source annotation, this would have to be resilient across edits.  But, that's what patch, and so many VCS, do.  Heck, it would even be useful if only inexactly resilient across edits - if there was just a table of source module/name -> dest module/instance mappings, and that was diffed.


Monday, January 30, 2012

Version controlling VCS metadata


My project .hg/hgrc is growing pretty long.

It needs to be under version control.

Unfortunately, hg does not, refuses to, version metadat such as .hg/hgrc.  Security.  Since .hg/hgrc may contain executable hooks...

Seems to me there should be a better way. E.g. disabling it's executability, although allowing it to be versioned.

Or - since some (but by no means all) such metadata is per repo, and is meaningless in a clone, perhaps there should be a meta-VCS system.  This is fairly natural for me, since i have had my repos nested within each other for decades.

Unfortunately, not so natural for Mercurial.

Symlinks to URLs


http://superuser.com/questions/277186/symlink-to-a-url/384182#384182

As people have explained, although you can create a symlink that links to an URL, it does not accomplish much.  UNIX is not smart enough to look at the link content, recognize it as an URL, and then do something appropriate to it.

Nevertheless, I occasionally do create symlinks to URLs.  Mainly just as documentation - I am used to going to looking at ~/links/* when I forget where, on the filesystem, I have stored something.  It's not much of a stretch to do the same thing for URLs.  However, doing so tends to break consistency checks and other stuff that assumess or requires that symlinks point to valid filesystem locations.

Furthermore, I have made some of my own tools smart enough to recognize that, if they see a symlink where an URL is needed, that they should dereference the symlink and look at its contents.  More and more of my tools treat URLs and paths in the filesystem as interchangeable.  Assuming, of course, that the URL protocol has appropriate methods for reading and writing.