Disclaimer

The content of this blog is my personal opinion only. Although I am an employee - currently of Nvidia, in the past of other companies such as Iagination Technologies, MIPS, Intellectual Ventures, Intel, AMD, Motorola, and Gould - I reveal this only so that the reader may account for any possible bias I may have towards my employer's products. The statements I make here in no way represent my employer's position, nor am I authorized to speak on behalf of my employer. In fact, this posting may not even represent my personal opinion, since occasionally I play devil's advocate.

See http://docs.google.com/View?id=dcxddbtr_23cg5thdfj for photo credits.

Thursday, June 21, 2012

recursive make can break parallel make -j

Recursive make is dangerously stupid.  There are better tools, but not everyone has realized.

In a system using recursive make, two rules invoke recursive make for different rulesof the same directory.
Unfortunately, these rukes are not parallelizable.  The nonrecursive makefile knows this, but the separate recursive make invocations do not.

I.e. recursive make can break parallel make -j.


how-can-mercurial-or-any-other-dvcs-recognize-partially-overlapped-histories


http://stackoverflow.com/questions/11146096/how-can-mercurial-or-any-other-dvcs-recognize-partially-overlapped-histories

Q: is there any way in Mercurial to usefully merge two repositories
whrrer the lines of history are similar, but not identical?

E.g. where one rep has coarse grain revisions 0,1,2
and the other has fine graun revisions 0, 0.1, 0.2, 1, 1.1, 1.2, 2,
and come up with a single history?

Rather than a mess of branches and heads, which is what I get when I try using what I know of Mercurial?

Or the even fancier
Repo1: 0, 1, 1.1, 1.2, 2
Repo 2: 0, 0.1, 0.2, 1, 2, 3

Merge: 0, 0.1, 0.2, 1, 1.1, 1.2 2, 3


In more detail:


What I want is a merge that can recognize when file contents are the same,
or which can recognize that two lines of history are similar, although not all versions in one line
are in the other,
and give something like:



    o=o changesets with same file contents on different historical lines
    o |  (line1)
    | |  changeset:   2:2a02e67e7b5d
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:15 2012 -0700
    | |  summary:     2
    | o  (line2)
    | |  changeset:   8:089179dde80a
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:15 2012 -0700
    | |  summary:     2
    | |
    | o  changeset:   7:615416921e33
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:14 2012 -0700
    | |  summary:     1.2
    | |
    | o  changeset:   6:a43a88065141
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:14 2012 -0700
    | |  summary:     1.1
    | |
    | |
    o=o changesets with same file contents on different historical lines
    o |  (line1)
    | |  changeset:   1:93cbae111269
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:13 2012 -0700
    | |  summary:     1
    | o  (line2)
    | |  changeset:   5:fef4050e0162
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:12 2012 -0700
    | |  summary:     1
    | |
    | o  changeset:   4:b51fbedc72e5
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:12 2012 -0700
    | |  summary:     0.2
    | |
    | o  changeset:   3:45b7f64b2a23
    | |  parent:      0:c80bc10826be
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:12 2012 -0700
    | |  summary:     0.1
    | |
    | |
    |/
    |
    o  changeset:   0:c80bc10826be
       user:        Andy Glew
       date:        Thu Jun 21 12:40:11 2012 -0700
       summary:     0

I can imagine that possibly a merge changeset would be necessary at the o=o points.

But I would like to have it recognized automatically.




Here's an example of how such a history would be created.
Contrived in this example, but something siomilar is happening to me in real-life,
when a project wants coarse grain commits, but where I want to preserve the fine grain ciommits
(as well as the coarse grain stuff released to the project).


    [glew@mipscs587 ~/hack/hg-granularity] 900$ bash 12:39:54>. ./eg

    % set verbose

    % mkdir hg-repo
    % cd hg-repo
    % ./hg-repo
    % hg init
    % echo 0 > a
    % hg add a
    % hg ci -m0 a

    % cd ..
    % hg clone hg-repo fine
    updating to branch default
    1 files updated, 0 files merged, 0 files removed, 0 files unresolved
    % hg clone hg-repo coarse
    updating to branch default
    1 files updated, 0 files merged, 0 files removed, 0 files unresolved

    % cd fine
    ./fine
    % echo 0.1 > a; hg ci -m0.1
    % echo 0.2 > a; hg ci -m0.2
    % echo 1 > a; hg ci -m1
    % cat a
    1
    % hg push default
    pushing to /home/glew/hack/hg-granularity/hg-repo
    searching for changes
    adding changesets
    adding manifests
    adding file changes
    added 3 changesets with 3 changes to 1 files
    % hg glog
    @  changeset:   3:fef4050e0162
    |  tag:         tip
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:12 2012 -0700
    |  summary:     1
    |
    o  changeset:   2:b51fbedc72e5
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:12 2012 -0700
    |  summary:     0.2
    |
    o  changeset:   1:45b7f64b2a23
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:12 2012 -0700
    |  summary:     0.1
    |
    o  changeset:   0:c80bc10826be
       user:        Andy Glew
       date:        Thu Jun 21 12:40:11 2012 -0700
       summary:     0


    % cd ../coarse
    % cp ../fine/a .
    % cat a
    1
    % hg ci -m1
    % hg glog
    @  changeset:   1:93cbae111269
    |  tag:         tip
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:13 2012 -0700
    |  summary:     1
    |
    o  changeset:   0:c80bc10826be
       user:        Andy Glew
       date:        Thu Jun 21 12:40:11 2012 -0700
       summary:     0


    % cd ../fine
    % echo 1.1 > a; hg ci -m1.1
    % echo 1.2 > a; hg ci -m1.2
    % echo 2 > a; hg ci -m2
    % cat a
    2
    % hg push default
    pushing to /home/glew/hack/hg-granularity/hg-repo
    searching for changes
    adding changesets
    adding manifests
    adding file changes
    added 3 changesets with 3 changes to 1 files
    % hg glog
    @  changeset:   6:089179dde80a
    |  tag:         tip
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:15 2012 -0700
    |  summary:     2
    |
    o  changeset:   5:615416921e33
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:14 2012 -0700
    |  summary:     1.2
    |
    o  changeset:   4:a43a88065141
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:14 2012 -0700
    |  summary:     1.1
    |
    o  changeset:   3:fef4050e0162
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:12 2012 -0700
    |  summary:     1
    |
    o  changeset:   2:b51fbedc72e5
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:12 2012 -0700
    |  summary:     0.2
    |
    o  changeset:   1:45b7f64b2a23
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:12 2012 -0700
    |  summary:     0.1
    |
    o  changeset:   0:c80bc10826be
       user:        Andy Glew
       date:        Thu Jun 21 12:40:11 2012 -0700
       summary:     0


    % cd ../coarse
    % cp ../fine/a .
    % cat a
    2
    % hg ci -m2
    % hg glog
    @  changeset:   2:2a02e67e7b5d
    |  tag:         tip
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:15 2012 -0700
    |  summary:     2
    |
    o  changeset:   1:93cbae111269
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:13 2012 -0700
    |  summary:     1
    |
    o  changeset:   0:c80bc10826be
       user:        Andy Glew
       date:        Thu Jun 21 12:40:11 2012 -0700
       summary:     0


OK, so now I have a fine grain history in the fine repo,
and a coarse grain history in the coarse repo.  I wouldf like to merge them.
(Firget that the coarse is a subset of the fine: I can easily contrive exanples where they are not).


Simply pushing the coarse graon history gives a warning.
I will opush it later,
but first I will try merging in a separate clone.

    % hg push default
    pushing to /home/glew/hack/hg-granularity/hg-repo
    searching for changes
    abort: push creates new remote head 2a02e67e7b5d!
    (you should pull and merge or use push -f to force)

    % cd ..

    % hg clone coarse merge-fine-and-coarse
    updating to branch default
    1 files updated, 0 files merged, 0 files removed, 0 files unresolved
    % cd merge-fine-and-coarse/
    ./merge-fine-and-coarse/


    % hg glog
    @  changeset:   2:2a02e67e7b5d
    |  tag:         tip
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:15 2012 -0700
    |  summary:     2
    |
    o  changeset:   1:93cbae111269
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:13 2012 -0700
    |  summary:     1
    |
    o  changeset:   0:c80bc10826be
       user:        Andy Glew
       date:        Thu Jun 21 12:40:11 2012 -0700
       summary:     0



    % hg pull ../hg-repo
    pulling from ../hg-repo
    searching for changes
    adding changesets
    adding manifests
    adding file changes
    added 6 changesets with 6 changes to 1 files (+1 heads)
    (run 'hg heads' to see heads, 'hg merge' to merge)


    % hg heads
    changeset:   8:089179dde80a
    tag:         tip
    user:        Andy Glew
    date:        Thu Jun 21 12:40:15 2012 -0700
    summary:     2

    changeset:   2:2a02e67e7b5d
    user:        Andy Glew
    date:        Thu Jun 21 12:40:15 2012 -0700
    summary:     2


Here is the merge.

Notice that the pairs
    o  changeset:   8:089179dde80a
    | @  changeset:   2:2a02e67e7b5d
and
    o  changeset:   5:fef4050e0162
    | o  changeset:   1:93cbae111269
have the same file contents,
one from the coartse and the other from the fine repo.
But the Mercurial history graph does not reflect this.

    % hg glog
    o  changeset:   8:089179dde80a
    |  tag:         tip
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:15 2012 -0700
    |  summary:     2
    |
    o  changeset:   7:615416921e33
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:14 2012 -0700
    |  summary:     1.2
    |
    o  changeset:   6:a43a88065141
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:14 2012 -0700
    |  summary:     1.1
    |
    o  changeset:   5:fef4050e0162
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:12 2012 -0700
    |  summary:     1
    |
    o  changeset:   4:b51fbedc72e5
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:12 2012 -0700
    |  summary:     0.2
    |
    o  changeset:   3:45b7f64b2a23
    |  parent:      0:c80bc10826be
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:12 2012 -0700
    |  summary:     0.1
    |
    | @  changeset:   2:2a02e67e7b5d
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:15 2012 -0700
    | |  summary:     2
    | |
    | o  changeset:   1:93cbae111269
    |/   user:        Andy Glew
    |    date:        Thu Jun 21 12:40:13 2012 -0700
    |    summary:     1
    |
    o  changeset:   0:c80bc10826be
       user:        Andy Glew
       date:        Thu Jun 21 12:40:11 2012 -0700
       summary:     0


    % hg diff -r 2a02e67e7b5d -r 089179dde80a



So I'll try a merge

    % hg merge -r 8
    1 files updated, 0 files merged, 0 files removed, 0 files unresolved
    (branch merge, don't forget to commit)


    % hg ci -m'merge of fine and coarse at 2'

Better -
this shows that
    o  changeset:   8:089179dde80a
    | @  changeset:   2:2a02e67e7b5d
are a convergence point,
although an extra dummy changesrt was necessary.


But it does not show the commonality between
    o  changeset:   5:fef4050e0162
    | o  changeset:   1:93cbae111269

Here's the merged graph

    % hg glog
    @    changeset:   9:328db8187d31
    |\   tag:         tip
    | |  parent:      2:2a02e67e7b5d
    | |  parent:      8:089179dde80a
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:43:51 2012 -0700
    | |  summary:     merge of fine and coarse at 2
    | |
    | o  changeset:   8:089179dde80a
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:15 2012 -0700
    | |  summary:     2
    | |
    | o  changeset:   7:615416921e33
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:14 2012 -0700
    | |  summary:     1.2
    | |
    | o  changeset:   6:a43a88065141
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:14 2012 -0700
    | |  summary:     1.1
    | |
    | o  changeset:   5:fef4050e0162
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:12 2012 -0700
    | |  summary:     1
    | |
    | o  changeset:   4:b51fbedc72e5
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:12 2012 -0700
    | |  summary:     0.2
    | |
    | o  changeset:   3:45b7f64b2a23
    | |  parent:      0:c80bc10826be
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:12 2012 -0700
    | |  summary:     0.1
    | |
    o |  changeset:   2:2a02e67e7b5d
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:15 2012 -0700
    | |  summary:     2
    | |
    o |  changeset:   1:93cbae111269
    |/   user:        Andy Glew
    |    date:        Thu Jun 21 12:40:13 2012 -0700
    |    summary:     1
    |
    o  changeset:   0:c80bc10826be
       user:        Andy Glew
       date:        Thu Jun 21 12:40:11 2012 -0700
       summary:     0


How about another merge?

    % hg update -r 1
    1 files updated, 0 files merged, 0 files removed, 0 files unresolved


    % hg merge -r 5
    1 files updated, 0 files merged, 0 files removed, 0 files unresolved
    (branch merge, don't forget to commit)


    % hg ci -m'merge of fine and coarse at 1'
    created new head


    % hg glog
    @    changeset:   10:cca7fec90d3f
    |\   tag:         tip
    | |  parent:      1:93cbae111269
    | |  parent:      5:fef4050e0162
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:45:03 2012 -0700
    | |  summary:     merge of fine and coarse at 1
    | |
    | | o    changeset:   9:328db8187d31
    | | |\   parent:      2:2a02e67e7b5d
    | | | |  parent:      8:089179dde80a
    | | | |  user:        Andy Glew
    | | | |  date:        Thu Jun 21 12:43:51 2012 -0700
    | | | |  summary:     merge of fine and coarse at 2
    | | | |
    | | | o  changeset:   8:089179dde80a
    | | | |  user:        Andy Glew
    | | | |  date:        Thu Jun 21 12:40:15 2012 -0700
    | | | |  summary:     2
    | | | |
    | | | o  changeset:   7:615416921e33
    | | | |  user:        Andy Glew
    | | | |  date:        Thu Jun 21 12:40:14 2012 -0700
    | | | |  summary:     1.2
    | | | |
    | +---o  changeset:   6:a43a88065141
    | | |    user:        Andy Glew
    | | |    date:        Thu Jun 21 12:40:14 2012 -0700
    | | |    summary:     1.1
    | | |
    | o |  changeset:   5:fef4050e0162
    | | |  user:        Andy Glew
    | | |  date:        Thu Jun 21 12:40:12 2012 -0700
    | | |  summary:     1
    | | |
    | o |  changeset:   4:b51fbedc72e5
    | | |  user:        Andy Glew
    | | |  date:        Thu Jun 21 12:40:12 2012 -0700
    | | |  summary:     0.2
    | | |
    | o |  changeset:   3:45b7f64b2a23
    | | |  parent:      0:c80bc10826be
    | | |  user:        Andy Glew
    | | |  date:        Thu Jun 21 12:40:12 2012 -0700
    | | |  summary:     0.1
    | | |
    +---o  changeset:   2:2a02e67e7b5d
    | |    user:        Andy Glew
    | |    date:        Thu Jun 21 12:40:15 2012 -0700
    | |    summary:     2
    | |
    o |  changeset:   1:93cbae111269
    |/   user:        Andy Glew
    |    date:        Thu Jun 21 12:40:13 2012 -0700
    |    summary:     1
    |
    o  changeset:   0:c80bc10826be
       user:        Andy Glew
       date:        Thu Jun 21 12:40:11 2012 -0700
       summary:     0


This is not right.  It has established a new head, whereas
what we wanted was some way of indicating that
    o  changeset:   5:fef4050e0162
    | o  changeset:   1:93cbae111269
are the same.




OK, switch back to the original coarse

    % cd ../coarse


    % hg push default
    pushing to /home/glew/hack/hg-granularity/hg-repo
    searching for changes
    abort: push creates new remote head 2a02e67e7b5d!
    (you should pull and merge or use push -f to force)


    % hg push -f default
    pushing to /home/glew/hack/hg-granularity/hg-repo
    searching for changes
    adding changesets
    adding manifests
    adding file changes
    added 2 changesets with 2 changes to 1 files (+1 heads)


    % hg glog
    @  changeset:   2:2a02e67e7b5d
    |  tag:         tip
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:15 2012 -0700
    |  summary:     2
    |
    o  changeset:   1:93cbae111269
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:13 2012 -0700
    |  summary:     1
    |
    o  changeset:   0:c80bc10826be
       user:        Andy Glew
       date:        Thu Jun 21 12:40:11 2012 -0700
       summary:     0



    % cd ../hg-repo/


    % hg glog
    o  changeset:   8:2a02e67e7b5d
    |  tag:         tip
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:15 2012 -0700
    |  summary:     2
    |
    o  changeset:   7:93cbae111269
    |  parent:      0:c80bc10826be
    |  user:        Andy Glew
    |  date:        Thu Jun 21 12:40:13 2012 -0700
    |  summary:     1
    |
    | o  changeset:   6:089179dde80a
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:15 2012 -0700
    | |  summary:     2
    | |
    | o  changeset:   5:615416921e33
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:14 2012 -0700
    | |  summary:     1.2
    | |
    | o  changeset:   4:a43a88065141
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:14 2012 -0700
    | |  summary:     1.1
    | |
    | o  changeset:   3:fef4050e0162
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:12 2012 -0700
    | |  summary:     1
    | |
    | o  changeset:   2:b51fbedc72e5
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:12 2012 -0700
    | |  summary:     0.2
    | |
    | o  changeset:   1:45b7f64b2a23
    |/   user:        Andy Glew
    |    date:        Thu Jun 21 12:40:12 2012 -0700
    |    summary:     0.1
    |
    @  changeset:   0:c80bc10826be
       user:        Andy Glew
       date:        Thu Jun 21 12:40:11 2012 -0700
       summary:     0



    % echo This is not right
    This is not right





What I want is a merge that can recognize when file contents are the same,
or which can recognize that two lines of history are similar, although not all versions in one line
are in the other,
and give something like:



    o=o changesets with same file contents on different historical lines
    o |  (line1)
    | |  changeset:   2:2a02e67e7b5d
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:15 2012 -0700
    | |  summary:     2
    | o  (line2)
    | |  changeset:   8:089179dde80a
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:15 2012 -0700
    | |  summary:     2
    | |
    | o  changeset:   7:615416921e33
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:14 2012 -0700
    | |  summary:     1.2
    | |
    | o  changeset:   6:a43a88065141
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:14 2012 -0700
    | |  summary:     1.1
    | |
    | |
    o=o changesets with same file contents on different historical lines
    o |  (line1)
    | |  changeset:   1:93cbae111269
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:13 2012 -0700
    | |  summary:     1
    | o  (line2)
    | |  changeset:   5:fef4050e0162
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:12 2012 -0700
    | |  summary:     1
    | |
    | o  changeset:   4:b51fbedc72e5
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:12 2012 -0700
    | |  summary:     0.2
    | |
    | o  changeset:   3:45b7f64b2a23
    | |  parent:      0:c80bc10826be
    | |  user:        Andy Glew
    | |  date:        Thu Jun 21 12:40:12 2012 -0700
    | |  summary:     0.1
    | |
    | |
    |/
    |
    o  changeset:   0:c80bc10826be
       user:        Andy Glew
       date:        Thu Jun 21 12:40:11 2012 -0700
       summary:     0

I can imagine that possibly a merge changeset would be necessary at the o=o points.

But I would like to have it recognized automatically.

Heck, forget "recognized".  I would like to have a way that I can recognize it manually, but have it represented in Mercurial.