< tags with words breaking XML rules >

<tag some attributes>
...
...
</tag>
 e.g.

<test expected-time="10min" start="">



</test elapsed-time=5min>

--
<tags that occupy a line by themselves are fairly clear/>

and, if so restricted, allowing < and > inside text is simple.



<tag>on same line</tag> and <more/>
no so bad.  But tyhe morte allowed, the more possibilities for confusion.

Serialization Formats

There are so many: http://en.wikipedia.org/wiki/Comparison_of_data_serialization_formats And yet, I want some more, with my pseudo-XML (or pseudo-JSON, or whatever)

ASN/1: an XML subset, that defines how values are represented (as text, not attributes).

Netstring: length:strings,

JSON: hashes ansd arrays; keys are strings, just like values. 

OGDL: trees, indentation based, with commas and parenrtheses for when you don't want new lines. #comments and references. Guaranteed round tripping, except for comments.

Property lists; NeXT to Apple. 

YAML: outline indentation, name:, little quoting, {hash: value}, [arrays, ...]; &id anchors and *id references, name: value, name: !!type value, !!binary 

XML: <tag>barewords</tag>, <>tag/>, attribute metadata. Basically simple, and then a whole lot of crap descended from SGML. Schemas and DTDs :-( Namespaces, references, XQuery, XSLT. An occasionally uncomfortable distinction between metadata (atributes) and data (text).

Why can't we just gedt along? Why can't they all interconvert?

Why I dislike XML

I want to be abkle to write <matching-brackert>
text, maybe some math, like a <b and c> d </matching-brackert>
withiout having to write ugly \&\l\t\; Heck, this WYSIWYG blog ceditor is an example - I can;t figure out how to type thye escape code for & lt ;

Why I like XML

In XML, or pseudo XML, you can see the matching brackets

<matching bracket> stuff
</matching >

whereas in simply parenthesized notations like JSON

{n1: {n2: n3: "do you have enough matching brackets?" }}}

?


You can usefully define semantics for mismatching constructs. XML also allows unquoted text, because of the very verbosity of its punctuation.

Using = and : binding operators to delineate structure

I have messed around with "pseudo-XML" a lot. Oh, heck, whop cares about the XML part, except that it is a standard?  Structured, simple, data.  Trying to make stuff human readable, but also parseable.

Here's a new-to-me trick:

I often use key=value pairs.

Sometimes I use name: value.

I.e. both = and : are nice ways of binding name and value.

How about:

n1: nA=vA, nB=vB, nC=vC
=>  nA=vA, nB=vB, nC=vC
=>    

or whatever way you want to express name/value in XML.

I.e. : as a binding operator that can bind to a list of name=value pairs

Or non-XML, if that's what tickles your fancy.  JSON?:

{ "n1": {"nA": "vA", "nB": "vB", "nC": "vC" } }

(BTW, if I ever declare that I am doing pseudo-JSON, it will be to have fewer quotes.)

{ n1: {nA: vA, nB: vB, nC: vC } }

The biggest difference between shell script languages and regular languages is quoting. "Barewords" are a big characteristic of scripting languages.



:s can bind until the next colon - i.e, all colons have the same strength

n1: nA=vA, nB=vB, n2: v2, n3: nX=vX, nY=vY

{ n1: { nA:vA, nB:vB}, n2:v2, n3: {nX:vX, nY: vY} }

To get more levels deep, would need brackets, or indentation. more operators: ::==



Heck, I'm not trying to define a full language here.  There are at least 3 perfectly good data languages - XML, JSON, S-expressions.

I just want to define some notation that is a bit more user friendly, but can fall back to the other notations if necessary.

Recognizing errors in build scripts

A very important thing, when writing or some other form of build recipe, is recognizing when an error has occurred that might prevent you going on the the next step (of the build DAG).

This is easy when commands are well behaved.  When they indicate errors tidily. (E.g. in the UNIX convention, 0 for success, nonzero for error.)

It is less easy for tools that are not so well behaved.  When the tool regularly produces errors, both in the UNIX exit code, and in whatever output it sends to stdout or stderr.  When past users seem to have nearly always been running the tool by hand, and there is folklore about what errors can be ignored, and which errors matter.

Better if the folklore is recorded, e.g. on a webpage or wiki. Bad if the folklore is passed by word of mouth.  Worse, if no single person knows all the folklore.

Not so bad if the tool produces a well characterized set of outputs, e.g. output files, and if the file only exists if it was correctly built.  Bad if the output file can be created, but malformed, not completed, etc.

I am tempted to say "BKM: make all output go tgo output.tmp, and then mv to output.final when okay."  But in this same effort I have found several examples of where the output file is perfectly fine, but where an error occured later in the tool producing it.

Not so bad if the output file contains error markers, such as "When processing this record, there was an error".  No matter if this occurs at the very end, or in the middle.  Bad if the output file contains no error markers, so you are left wondering, in a partially created output file, why it stopped.  Really bad if the output file format ignores missing data, such as assuming it is all zeroes.  really, Really, REALLY bad.  I wasted lots of time on that, umm, "feature".

Here are some things that I have done to ciope with such error-ful output:

First, wrapperize the tools that produce unreliable error indications.

Second, in the wrapper or in the build tool, place code to look at the output. Let this "validation" code decide if it is safe to go on to the next step or not.

At first, perhaps unconditionally go on to the next step, since any particular error may be ignorable.

But as you debug stuff, start recognizing error conditions.  When you are confident that an error is non-ignorable, have the wrapper return an error.  When you are confident that the error is ignorable, or that there was no error, have the wrapper return success.

Print messages when you are in-between.  Probably continue anyway, but at least flag when your validation code - typically pattern matching on error messages - thinks that something suspicious has happened, but is not sure.

Be schizophrenic.  Don't try to give a single answer.  Have your validation code do

GOOD: I saw a passed message

BAD:!!: the log file was much tiner than it has ever been on a successful run 

SUSPICIOUS:?: manycompilation errors, but they were ignored in the past.

BAD:!!: the expected output files were not created

Use any heuristic that you can imagine, that you are willing to take the time to code.




Probably most important: save examples of good and bad runs, so that you can start looking for patterns.

 

A good example of how make is needlessly repetitive is how I have been building makefile lines today:

I copy a command line that I pasted into the shell, e.g.

command < file1 -other_input file2 > file3

I paste it into my makefile, twice

command < file1 -other_input file2 > file3
command < file1 -other_input file2 > file3

and then edit

.PHONY: verb-to-run-command
verb-to-run-command: file3
file3: command file1 file2
        command < file1 -other_input file2 > file3