Krazy Glew's Blog

Disclaimer

The content of this blog is my personal opinion only. Although I am an employee - currently of Nvidia, in the past of other companies such as Iagination Technologies, MIPS, Intellectual Ventures, Intel, AMD, Motorola, and Gould - I reveal this only so that the reader may account for any possible bias I may have towards my employer's products. The statements I make here in no way represent my employer's position, nor am I authorized to speak on behalf of my employer. In fact, this posting may not even represent my personal opinion, since occasionally I play devil's advocate.

See http://docs.google.com/View?id=dcxddbtr_23cg5thdfj for photo credits.

Monday, June 25, 2012

abspath and simple diff tests

I like tests.

Lacking a full fledged test infrastructure, simply diffing outputfiles is often sufficient.

Except for stuff like

a) using absolute paths related to where the source tree is

b) other time and position variant stuff.

I'd almost like a diff

1) that was smart enough to recognize and skip timestamps

2) that was smart enough to recognize absolurte/relative file patterns.

I have added relative/absolute file options to many tools - so often that I suspect it is not the tools fault, but really should be fixed in the test infrastructure.

I have preprocessed expected output files - but that can be fragile, if the preprocess tool breaks and, e.g., gives an empty string.

Saturday, June 23, 2012

Ubiquitous tracking (and versioning)

Sometimes I wish that the whole bloody filesystem was capable of the sort of tracking that a version control system does.

E.g. today I am looking at some saved output files, that I copied from a path I got in email a while back.

But I forgot to record the provenance. I.e. I forgot to record where I got them from.

It sure would be nice if the fgilsystem remembered that, a few weeks back, I did

cp -R jffggghggh here

to get this data.

--

Just recording the commands might be useful.

Full versioning may not be necessary.

--

Ultimately: log everything?

Apart from the privacy and big brother implications, the real problem with logging everything is the query system. Just recording all of your commands is NOT very useful, is hard to query.

Want to record, e.g. that command

cp -R jffggghggh here

executing in directyory a/aa/h/g

produced file here/b/h

...

Hmm... this is the sort of thing that the automatic build systems do.

One-liners considered dangerous

It is nice to come up with one-liners. E.g. today I was looking for a one-liner hg post-clone hook.

But... even when you can find a one-liner, oftentimes you give up error checking or good error messages to get it. Or, you might create something fraguke, that only works on the input you happened to have seen today, when generalizing it even a little bit is easy, but would make it more than one line.

I wonder if one liners are a source of security bugs?

Thursday, June 21, 2012

recursive make can break parallel make -j

Recursive make is dangerously stupid. There are better tools, but not everyone has realized.

In a system using recursive make, two rules invoke recursive make for different rulesof the same directory.
Unfortunately, these rukes are not parallelizable. The nonrecursive makefile knows this, but the separate recursive make invocations do not.

I.e. recursive make can break parallel make -j.

how-can-mercurial-or-any-other-dvcs-recognize-partially-overlapped-histories

http://stackoverflow.com/questions/11146096/how-can-mercurial-or-any-other-dvcs-recognize-partially-overlapped-histories

Q: is there any way in Mercurial to usefully merge two repositories
whrrer the lines of history are similar, but not identical?

E.g. where one rep has coarse grain revisions 0,1,2
and the other has fine graun revisions 0, 0.1, 0.2, 1, 1.1, 1.2, 2,
and come up with a single history?

Rather than a mess of branches and heads, which is what I get when I try using what I know of Mercurial?

Or the even fancier
Repo1: 0, 1, 1.1, 1.2, 2
Repo 2: 0, 0.1, 0.2, 1, 2, 3

Merge: 0, 0.1, 0.2, 1, 1.1, 1.2 2, 3

In more detail:

What I want is a merge that can recognize when file contents are the same,
or which can recognize that two lines of history are similar, although not all versions in one line
are in the other,
and give something like:

o=o changesets with same file contents on different historical lines
o | (line1)
| | changeset: 2:2a02e67e7b5d
| | user: Andy Glew
| | date: Thu Jun 21 12:40:15 2012 -0700
| | summary: 2
| o (line2)
| | changeset: 8:089179dde80a
| | user: Andy Glew
| | date: Thu Jun 21 12:40:15 2012 -0700
| | summary: 2
| |
| o changeset: 7:615416921e33
| | user: Andy Glew
| | date: Thu Jun 21 12:40:14 2012 -0700
| | summary: 1.2
| |
| o changeset: 6:a43a88065141
| | user: Andy Glew
| | date: Thu Jun 21 12:40:14 2012 -0700
| | summary: 1.1
| |
| |
o=o changesets with same file contents on different historical lines
o | (line1)
| | changeset: 1:93cbae111269
| | user: Andy Glew
| | date: Thu Jun 21 12:40:13 2012 -0700
| | summary: 1
| o (line2)
| | changeset: 5:fef4050e0162
| | user: Andy Glew
| | date: Thu Jun 21 12:40:12 2012 -0700
| | summary: 1
| |
| o changeset: 4:b51fbedc72e5
| | user: Andy Glew
| | date: Thu Jun 21 12:40:12 2012 -0700
| | summary: 0.2
| |
| o changeset: 3:45b7f64b2a23
| | parent: 0:c80bc10826be
| | user: Andy Glew
| | date: Thu Jun 21 12:40:12 2012 -0700
| | summary: 0.1
| |
| |
|/
|
o changeset: 0:c80bc10826be
user: Andy Glew
date: Thu Jun 21 12:40:11 2012 -0700
summary: 0

I can imagine that possibly a merge changeset would be necessary at the o=o points.

But I would like to have it recognized automatically.

Here's an example of how such a history would be created.
Contrived in this example, but something siomilar is happening to me in real-life,
when a project wants coarse grain commits, but where I want to preserve the fine grain ciommits
(as well as the coarse grain stuff released to the project).

[glew@mipscs587 ~/hack/hg-granularity] 900$ bash 12:39:54>. ./eg

% set verbose

% mkdir hg-repo
% cd hg-repo
% ./hg-repo
% hg init
% echo 0 > a
% hg add a
% hg ci -m0 a

% cd ..
% hg clone hg-repo fine
updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
% hg clone hg-repo coarse
updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved

% cd fine
./fine
% echo 0.1 > a; hg ci -m0.1
% echo 0.2 > a; hg ci -m0.2
% echo 1 > a; hg ci -m1
% cat a
1
% hg push default
pushing to /home/glew/hack/hg-granularity/hg-repo
searching for changes
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 1 files
% hg glog
@ changeset: 3:fef4050e0162
| tag: tip
| user: Andy Glew
| date: Thu Jun 21 12:40:12 2012 -0700
| summary: 1
|
o changeset: 2:b51fbedc72e5
| user: Andy Glew
| date: Thu Jun 21 12:40:12 2012 -0700
| summary: 0.2
|
o changeset: 1:45b7f64b2a23
| user: Andy Glew
| date: Thu Jun 21 12:40:12 2012 -0700
| summary: 0.1
|
o changeset: 0:c80bc10826be
user: Andy Glew
date: Thu Jun 21 12:40:11 2012 -0700
summary: 0

% cd ../coarse
% cp ../fine/a .
% cat a
1
% hg ci -m1
% hg glog
@ changeset: 1:93cbae111269
| tag: tip
| user: Andy Glew
| date: Thu Jun 21 12:40:13 2012 -0700
| summary: 1
|
o changeset: 0:c80bc10826be
user: Andy Glew
date: Thu Jun 21 12:40:11 2012 -0700
summary: 0

% cd ../fine
% echo 1.1 > a; hg ci -m1.1
% echo 1.2 > a; hg ci -m1.2
% echo 2 > a; hg ci -m2
% cat a
2
% hg push default
pushing to /home/glew/hack/hg-granularity/hg-repo
searching for changes
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 1 files
% hg glog
@ changeset: 6:089179dde80a
| tag: tip
| user: Andy Glew
| date: Thu Jun 21 12:40:15 2012 -0700
| summary: 2
|
o changeset: 5:615416921e33
| user: Andy Glew
| date: Thu Jun 21 12:40:14 2012 -0700
| summary: 1.2
|
o changeset: 4:a43a88065141
| user: Andy Glew
| date: Thu Jun 21 12:40:14 2012 -0700
| summary: 1.1
|
o changeset: 3:fef4050e0162
| user: Andy Glew
| date: Thu Jun 21 12:40:12 2012 -0700
| summary: 1
|
o changeset: 2:b51fbedc72e5
| user: Andy Glew
| date: Thu Jun 21 12:40:12 2012 -0700
| summary: 0.2
|
o changeset: 1:45b7f64b2a23
| user: Andy Glew
| date: Thu Jun 21 12:40:12 2012 -0700
| summary: 0.1
|
o changeset: 0:c80bc10826be
user: Andy Glew
date: Thu Jun 21 12:40:11 2012 -0700
summary: 0

% cd ../coarse
% cp ../fine/a .
% cat a
2
% hg ci -m2
% hg glog
@ changeset: 2:2a02e67e7b5d
| tag: tip
| user: Andy Glew
| date: Thu Jun 21 12:40:15 2012 -0700
| summary: 2
|
o changeset: 1:93cbae111269
| user: Andy Glew
| date: Thu Jun 21 12:40:13 2012 -0700
| summary: 1
|
o changeset: 0:c80bc10826be
user: Andy Glew
date: Thu Jun 21 12:40:11 2012 -0700
summary: 0

OK, so now I have a fine grain history in the fine repo,
and a coarse grain history in the coarse repo. I wouldf like to merge them.
(Firget that the coarse is a subset of the fine: I can easily contrive exanples where they are not).

Simply pushing the coarse graon history gives a warning.
I will opush it later,
but first I will try merging in a separate clone.

% hg push default
pushing to /home/glew/hack/hg-granularity/hg-repo
searching for changes
abort: push creates new remote head 2a02e67e7b5d!
(you should pull and merge or use push -f to force)

% cd ..

% hg clone coarse merge-fine-and-coarse
updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
% cd merge-fine-and-coarse/
./merge-fine-and-coarse/

% hg glog
@ changeset: 2:2a02e67e7b5d
| tag: tip
| user: Andy Glew
| date: Thu Jun 21 12:40:15 2012 -0700
| summary: 2
|
o changeset: 1:93cbae111269
| user: Andy Glew
| date: Thu Jun 21 12:40:13 2012 -0700
| summary: 1
|
o changeset: 0:c80bc10826be
user: Andy Glew
date: Thu Jun 21 12:40:11 2012 -0700
summary: 0

% hg pull ../hg-repo
pulling from ../hg-repo
searching for changes
adding changesets
adding manifests
adding file changes
added 6 changesets with 6 changes to 1 files (+1 heads)
(run 'hg heads' to see heads, 'hg merge' to merge)

% hg heads
changeset: 8:089179dde80a
tag: tip
user: Andy Glew
date: Thu Jun 21 12:40:15 2012 -0700
summary: 2

changeset: 2:2a02e67e7b5d
user: Andy Glew
date: Thu Jun 21 12:40:15 2012 -0700
summary: 2

Here is the merge.

Notice that the pairs
o changeset: 8:089179dde80a
| @ changeset: 2:2a02e67e7b5d
and
o changeset: 5:fef4050e0162
| o changeset: 1:93cbae111269
have the same file contents,
one from the coartse and the other from the fine repo.
But the Mercurial history graph does not reflect this.

% hg glog
o changeset: 8:089179dde80a
| tag: tip
| user: Andy Glew
| date: Thu Jun 21 12:40:15 2012 -0700
| summary: 2
|
o changeset: 7:615416921e33
| user: Andy Glew
| date: Thu Jun 21 12:40:14 2012 -0700
| summary: 1.2
|
o changeset: 6:a43a88065141
| user: Andy Glew
| date: Thu Jun 21 12:40:14 2012 -0700
| summary: 1.1
|
o changeset: 5:fef4050e0162
| user: Andy Glew
| date: Thu Jun 21 12:40:12 2012 -0700
| summary: 1
|
o changeset: 4:b51fbedc72e5
| user: Andy Glew
| date: Thu Jun 21 12:40:12 2012 -0700
| summary: 0.2
|
o changeset: 3:45b7f64b2a23
| parent: 0:c80bc10826be
| user: Andy Glew
| date: Thu Jun 21 12:40:12 2012 -0700
| summary: 0.1
|
| @ changeset: 2:2a02e67e7b5d
| | user: Andy Glew
| | date: Thu Jun 21 12:40:15 2012 -0700
| | summary: 2
| |
| o changeset: 1:93cbae111269
|/ user: Andy Glew
| date: Thu Jun 21 12:40:13 2012 -0700
| summary: 1
|
o changeset: 0:c80bc10826be
user: Andy Glew
date: Thu Jun 21 12:40:11 2012 -0700
summary: 0

% hg diff -r 2a02e67e7b5d -r 089179dde80a

So I'll try a merge

% hg merge -r 8
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)

% hg ci -m'merge of fine and coarse at 2'

Better -
this shows that
o changeset: 8:089179dde80a
| @ changeset: 2:2a02e67e7b5d
are a convergence point,
although an extra dummy changesrt was necessary.

But it does not show the commonality between
o changeset: 5:fef4050e0162
| o changeset: 1:93cbae111269

Here's the merged graph

% hg glog
@ changeset: 9:328db8187d31
|\ tag: tip
| | parent: 2:2a02e67e7b5d
| | parent: 8:089179dde80a
| | user: Andy Glew
| | date: Thu Jun 21 12:43:51 2012 -0700
| | summary: merge of fine and coarse at 2
| |
| o changeset: 8:089179dde80a
| | user: Andy Glew
| | date: Thu Jun 21 12:40:15 2012 -0700
| | summary: 2
| |
| o changeset: 7:615416921e33
| | user: Andy Glew
| | date: Thu Jun 21 12:40:14 2012 -0700
| | summary: 1.2
| |
| o changeset: 6:a43a88065141
| | user: Andy Glew
| | date: Thu Jun 21 12:40:14 2012 -0700
| | summary: 1.1
| |
| o changeset: 5:fef4050e0162
| | user: Andy Glew
| | date: Thu Jun 21 12:40:12 2012 -0700
| | summary: 1
| |
| o changeset: 4:b51fbedc72e5
| | user: Andy Glew
| | date: Thu Jun 21 12:40:12 2012 -0700
| | summary: 0.2
| |
| o changeset: 3:45b7f64b2a23
| | parent: 0:c80bc10826be
| | user: Andy Glew
| | date: Thu Jun 21 12:40:12 2012 -0700
| | summary: 0.1
| |
o | changeset: 2:2a02e67e7b5d
| | user: Andy Glew
| | date: Thu Jun 21 12:40:15 2012 -0700
| | summary: 2
| |
o | changeset: 1:93cbae111269
|/ user: Andy Glew
| date: Thu Jun 21 12:40:13 2012 -0700
| summary: 1
|
o changeset: 0:c80bc10826be
user: Andy Glew
date: Thu Jun 21 12:40:11 2012 -0700
summary: 0

How about another merge?

% hg update -r 1
1 files updated, 0 files merged, 0 files removed, 0 files unresolved

% hg merge -r 5
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)

% hg ci -m'merge of fine and coarse at 1'
created new head

% hg glog
@ changeset: 10:cca7fec90d3f
|\ tag: tip
| | parent: 1:93cbae111269
| | parent: 5:fef4050e0162
| | user: Andy Glew
| | date: Thu Jun 21 12:45:03 2012 -0700
| | summary: merge of fine and coarse at 1
| |
| | o changeset: 9:328db8187d31
| | |\ parent: 2:2a02e67e7b5d
| | | | parent: 8:089179dde80a
| | | | user: Andy Glew
| | | | date: Thu Jun 21 12:43:51 2012 -0700
| | | | summary: merge of fine and coarse at 2
| | | |
| | | o changeset: 8:089179dde80a
| | | | user: Andy Glew
| | | | date: Thu Jun 21 12:40:15 2012 -0700
| | | | summary: 2
| | | |
| | | o changeset: 7:615416921e33
| | | | user: Andy Glew
| | | | date: Thu Jun 21 12:40:14 2012 -0700
| | | | summary: 1.2
| | | |
| +---o changeset: 6:a43a88065141
| | | user: Andy Glew
| | | date: Thu Jun 21 12:40:14 2012 -0700
| | | summary: 1.1
| | |
| o | changeset: 5:fef4050e0162
| | | user: Andy Glew
| | | date: Thu Jun 21 12:40:12 2012 -0700
| | | summary: 1
| | |
| o | changeset: 4:b51fbedc72e5
| | | user: Andy Glew
| | | date: Thu Jun 21 12:40:12 2012 -0700
| | | summary: 0.2
| | |
| o | changeset: 3:45b7f64b2a23
| | | parent: 0:c80bc10826be
| | | user: Andy Glew
| | | date: Thu Jun 21 12:40:12 2012 -0700
| | | summary: 0.1
| | |
+---o changeset: 2:2a02e67e7b5d
| | user: Andy Glew
| | date: Thu Jun 21 12:40:15 2012 -0700
| | summary: 2
| |
o | changeset: 1:93cbae111269
|/ user: Andy Glew
| date: Thu Jun 21 12:40:13 2012 -0700
| summary: 1
|
o changeset: 0:c80bc10826be
user: Andy Glew
date: Thu Jun 21 12:40:11 2012 -0700
summary: 0

This is not right. It has established a new head, whereas
what we wanted was some way of indicating that
o changeset: 5:fef4050e0162
| o changeset: 1:93cbae111269
are the same.

OK, switch back to the original coarse

% cd ../coarse

% hg push default
pushing to /home/glew/hack/hg-granularity/hg-repo
searching for changes
abort: push creates new remote head 2a02e67e7b5d!
(you should pull and merge or use push -f to force)

% hg push -f default
pushing to /home/glew/hack/hg-granularity/hg-repo
searching for changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 2 changes to 1 files (+1 heads)

% hg glog
@ changeset: 2:2a02e67e7b5d
| tag: tip
| user: Andy Glew
| date: Thu Jun 21 12:40:15 2012 -0700
| summary: 2
|
o changeset: 1:93cbae111269
| user: Andy Glew
| date: Thu Jun 21 12:40:13 2012 -0700
| summary: 1
|
o changeset: 0:c80bc10826be
user: Andy Glew
date: Thu Jun 21 12:40:11 2012 -0700
summary: 0

% cd ../hg-repo/

% hg glog
o changeset: 8:2a02e67e7b5d
| tag: tip
| user: Andy Glew
| date: Thu Jun 21 12:40:15 2012 -0700
| summary: 2
|
o changeset: 7:93cbae111269
| parent: 0:c80bc10826be
| user: Andy Glew
| date: Thu Jun 21 12:40:13 2012 -0700
| summary: 1
|
| o changeset: 6:089179dde80a
| | user: Andy Glew
| | date: Thu Jun 21 12:40:15 2012 -0700
| | summary: 2
| |
| o changeset: 5:615416921e33
| | user: Andy Glew
| | date: Thu Jun 21 12:40:14 2012 -0700
| | summary: 1.2
| |
| o changeset: 4:a43a88065141
| | user: Andy Glew
| | date: Thu Jun 21 12:40:14 2012 -0700
| | summary: 1.1
| |
| o changeset: 3:fef4050e0162
| | user: Andy Glew
| | date: Thu Jun 21 12:40:12 2012 -0700
| | summary: 1
| |
| o changeset: 2:b51fbedc72e5
| | user: Andy Glew
| | date: Thu Jun 21 12:40:12 2012 -0700
| | summary: 0.2
| |
| o changeset: 1:45b7f64b2a23
|/ user: Andy Glew
| date: Thu Jun 21 12:40:12 2012 -0700
| summary: 0.1
|
@ changeset: 0:c80bc10826be
user: Andy Glew
date: Thu Jun 21 12:40:11 2012 -0700
summary: 0

% echo This is not right
This is not right

What I want is a merge that can recognize when file contents are the same,
or which can recognize that two lines of history are similar, although not all versions in one line
are in the other,
and give something like:

o=o changesets with same file contents on different historical lines
o | (line1)
| | changeset: 2:2a02e67e7b5d
| | user: Andy Glew
| | date: Thu Jun 21 12:40:15 2012 -0700
| | summary: 2
| o (line2)
| | changeset: 8:089179dde80a
| | user: Andy Glew
| | date: Thu Jun 21 12:40:15 2012 -0700
| | summary: 2
| |
| o changeset: 7:615416921e33
| | user: Andy Glew
| | date: Thu Jun 21 12:40:14 2012 -0700
| | summary: 1.2
| |
| o changeset: 6:a43a88065141
| | user: Andy Glew
| | date: Thu Jun 21 12:40:14 2012 -0700
| | summary: 1.1
| |
| |
o=o changesets with same file contents on different historical lines
o | (line1)
| | changeset: 1:93cbae111269
| | user: Andy Glew
| | date: Thu Jun 21 12:40:13 2012 -0700
| | summary: 1
| o (line2)
| | changeset: 5:fef4050e0162
| | user: Andy Glew
| | date: Thu Jun 21 12:40:12 2012 -0700
| | summary: 1
| |
| o changeset: 4:b51fbedc72e5
| | user: Andy Glew
| | date: Thu Jun 21 12:40:12 2012 -0700
| | summary: 0.2
| |
| o changeset: 3:45b7f64b2a23
| | parent: 0:c80bc10826be
| | user: Andy Glew
| | date: Thu Jun 21 12:40:12 2012 -0700
| | summary: 0.1
| |
| |
|/
|
o changeset: 0:c80bc10826be
user: Andy Glew
date: Thu Jun 21 12:40:11 2012 -0700
summary: 0

I can imagine that possibly a merge changeset would be necessary at the o=o points.

But I would like to have it recognized automatically.

Heck, forget "recognized". I would like to have a way that I can recognize it manually, but have it represented in Mercurial.

Wednesday, June 20, 2012

Knobs should be left to right - why (some) getopts are broken

There are two main styles of command line argument or options parsers - which my subculture calls "knobs".

My preference is to parse strictly left to right.

Another common approach is to partse all the knobs, possibly out of order. Often the last value written for a knob overrides all previous values.

This can cause problems - hence my preference for left to right. Here's an example:

Imagine that you have a "big" knob, -big A or -big B, that has a big effect - e.g. swapping in a table of multiple settings.

And imagine that you have a small knob, -small x or -small y, that changesd a subsetting.

E.g.
command -big A => table[0] = A0, table[1] = A1, table[2] = A2...

command -big B => table[0] = B0, table[1] = B1, table[2] = B2...

command -big A -big B
=> the later B overrides the earlier A
=> table[0] = B0, table[1] = B1, table[2]=B2...

so far so good.

But imagine that -small changes only one entry.

command -big A -small x => table[0] = A0, table[1] = x, table[2] = A2...

command -big B -small y => table[0] = B0, table[1] = y, table[2] = B2...

Later -small may override earlier -small

command -big A -small x -small y => table[0] = A0, table[1] = y, table[2] = A2...

But... should a later -big override an earlier -small? I think so:

command -big A -small x -big B
=> My preference
=> table[0] = B0, table[1] = B1, table[2] = B2...

But many getopts parsers instead do:

command -big A -small x -big B
=> if -small is processed after -big
=> table[0] = B0, table[1] = x, table[2] = B2...

or, worse, have -big override even earlier -smalls.

List valued knobs don't help much here. Short-sight. Although I admit that I have occasionally use list valued knobs as a kluge when dealing with a non left to right argumernt parser.

---

Some knobs have side effects. Some even create other knobs. There needs to be a defined order of evaluation. Left to right is as goos as any.

---

The problem with left to right is that it tends to imply that you are building a data structure as you parse the knobs. Which sometimes means that a later knob may cause you to want to destroy a datastructure that was being built.

Tuesday, June 19, 2012

Monolithics and Makefiles

I think that one of the reasons why some people write big monolithic programs with too many classes and functions and just plain stuff in the same headers and .c/.cpp files is that they do not have goopd dynamic Make tools. They get the Makefile working, and then don't touch it, since touching it would require editing by hand.