git directories not first class?

git apparently does not treat directories as first class citizens.

E.g. there apparently is no way to checkin an empty directory. Or, at least, no way to provide a log entry when creating a directory.

SomeSecretHostname /users/glew/hack/git-hacking/ 441 : mkdir git-dir-example
SomeSecretHostname /users/glew/hack/git-hacking/ 442 : cd git-dir-example
Directory: /users/glew/hack/git-hacking/git-dir-example
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 443 : got init
got: Command not found.
     # my usual typo: got instead of git
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 444 : git init
Initialized empty Git repository in /fs30/home.directory.11/glew/hack/git-hacking/git-dir-example/.git/
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 445 : git log
fatal: bad default revision 'HEAD'
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 446 : echo hi > there
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 447 : git add there
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 448 : git commit -m'there'
[master (root-commit) 2849a27] there
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 there
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 449 : git log
commit 2849a27602d10ec8192e931c5410491521f3fe73
Author: Andy Glew Linux 
Date:   Tue May 26 11:48:49 2009 -0700

    there
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 450 : git mkdir foo
git: 'mkdir' is not a git-command. See 'git --help'.
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 451 : mkdir foo
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 452 : git commit
# On branch master
nothing to commit (working directory clean)
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 453 : git add foo
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 454 : git commit
# On branch master
nothing to commit (working directory clean)
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 455 : echo hi > foo/there
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 456 : git add foo/there
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 457 : git commit
Waiting for Emacs...
[master aa4fd7d] foo/there
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 foo/there
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 458 : git log
commit aa4fd7d713584ca2a3781f264fc9d48b1169f032
Author: Andy Glew Linux 
Date:   Tue May 26 11:50:01 2009 -0700

    foo/there

commit 2849a27602d10ec8192e931c5410491521f3fe73
Author: Andy Glew Linux 
Date:   Tue May 26 11:48:49 2009 -0700

    there
SomeSecretHostname /users/glew/hack/git-hacking/git-dir-example/ 459 : 

Merging unrelated repositories

As I start up with git, I am also starting up work with several codebases that I am unfamiliar with.

The codebases are mostly under SVN, but not all; some are under CVS, some Perforce. I have decided to just suck everything into git for purposes of tracking any changes I am making myself. I am not linking to svn repos or anything else; I am just sucking in the SVN and CVS metadata, with the expectation that I can work under git, and then do a svn checkin.

The codebases are structured as overlapping projects, subprojects, and libraries. I was not aware of the relationship between the subprojects and libraries. E.g. I checked out a library, placed it under git, made changes - and now have checked out another tool that has this library as a subcomponent. I want the new tool to lie in git as well, and I want to checkin the vanilla source code - but I also, eventually, want to use my edits to the library with the new tool.

Basically, I am doing a lot of work with subprojects. I am doing a lot of work with initially unrelated repositories.


Here's a link to a procedure on "Merging two unrelated repositories":
http://www.simplicidade.org/notes/archives/2009/04/merging_two_unr.html

What does it mean to temporarily converge branches?

In previous blog posts I have talked about how I would like to merge two diverged version control branches of my glew-home personal directory version control tree. Incrementally, however, not all in one go. Others have responded that git's data model does not support such file at a time merger.

True enough. But it inspires me to think about how such branch merger should behave.

I have these two branches. They are separate and distinct. I want them to converge, a file at a time. I don't really want to create a merged copy of a file that lives ob two different branches. What I really want is for certain files, on a file by file basis, to belong to two different branches.

What does it mean to "belong to two different branches?"

If the branches remain independent, and just happen to have the same file version, then checkins on one branch do NOT automatically affect the other branch. The first such checkin will cause the branches to diverge. A manual merger may ensure that the two branches stay converged - but it is a manual action. If three or more branches are "converged" but not eliminated, the merging work increases correspondingly.

That's the key: merging branches, but not intending to eliminate the merged branches. At least not immediately. "Converging" branches, with the intention that changes made to one branch be made to the other branch(es), as automatically as possible.

When converging branches that contain multiple files, some files may be converged as described above, whereas others may remain distinct. Thus, the multifile branches tahat are to be merged, even if the intent is to eliminate one or both of them eventually, may pass through this phase where certain files are unmerged, but others are merged or converged - as I am trying to describe here.

OK, so what does this "converged" state mean? I think what it means is that the intent is that the branches shall remain identically. It changes the default from a new checkin causing the beranches to diverge, with a manual re-merger required, to where a new checkin by default is made to both branches simultaneously. As correctly as possible.

Let's talk about a single file first.

File1:
  v0 -> branch1.v1 -> branch1.v2
  v0 -> branch2.v1' -> branch2.v2
where   
  branch1.v1 != branch2.v1'
  branch1.v2 == branch2.v2
which we might indicate as
  converged(branch1,branch2).v2

I would like to draw a diagram of diverging branches reconverging at this point, but I hate ascii art. Perhaps I can do a real drawing and attach it.

Anyway, if we have separate branches that have temporarily converged
v0 -> branch1.v1 -> branch1.v2
v0 -> branch2.v1' -> branch2.v2
then new checkins will cause the branches to diverge:
v0 -> branch1.v1 -> branch1.v2 -> branch1.v3
v0 -> branch2.v1' -> branch2.v2-> branch2.v3'

whereas if they are intended to be converged we might see
v0 -> branch1.v1 -> converged(branch1,branch2).v2
v0 -> branch2.v1' -> converged(branch1,branch2).v2
after the first checkin of v3 on branch1,
v0 -> branch1.v1 -> converged(branch1,branch2).v2 -> converged(branch1,?branch2).v2
v0 -> branch2.v1' -> converged(branch1,branch2).v2 -> converged(branch1,?branch2).v2

A new checkin v4 on branch1 may cause it to look like

v0 -> branch1.v1 -> converged(branch1,branch2).v2 -> converged(branch1,?branch2).v3 -> converged(branch1,?branch2).v4
v0 -> branch2.v1' -> converged(branch1,branch2).v2 -> converged(branch1,?branch2).v3 -> converged(branch1,?branch2).v4

which I prefer to indicate as a single history subsequent to the merge point:
-> converged(branch1,branch2).v2 -> converged(branch1,?branch2).v3 -> converged(branch1,?branch2).v4

The point of the ?branch2 notation is that, although the checkins of v3 and v4 were actually made ion branch1, they are intended to also be made on branch2. Underlying this is an assumption that there needs to be some testing or validation that these checkins are also good for branch2. If we did not require such testing, we would mark them as converged(branch1,branch2).v4.

But I have drunk deep of the koolaid of testing and agile methodologies, so I will not make such marking the default. I will assume the "tested on branch1, but not tested om branch2" state. And I will assume that there is a way of eventually removing the ? marking.

So, what does it mean for a file object to be in the converged, but tested only on one branch, state?
-> converged(branch1,branch2).v2 -> converged(branch1,?branch2).v3 -> converged(branch1,?branch2).v4

What it means is that branch1 can continue to check in, but branch2 must merge the branch1 changes marked as ?branch2, before advancing. E.g. if somebody wants to do a checkin of vx on branch2, they will have to remove the ?branch2 marking on v4.

-> converged(branch1,branch2).v2 -> converged(branch1,?branch2).v3 -> converged(branch1,branch2).v4 -> converged(?branch1,branch2).vx

If necessary, it may be required to merge v4 and vx

-> converged(branch1,branch2).v2 -> converged(branch1,?branch2).v3 -> converged(branch1,branch2).v4 -> converged(?branch1,branch2).v4x

and since I believe in checking everything in, vx would have to be checked in on sort of a merge/task branch. This time a real, temporary, branch.

I.e. "converged" file object branches just affect the defaults. They require that the ?branch2 branch be updated before advancing the merged branch.

In some ways this is like a CVS workspace versus a shared repository that has been updated elsewhere. The CVS workspace is in some ways a temporary branch, although not always recorded in permanent history. (Sometimes regrettably so - who has not update merged a working workspace, to have had it break - and not be able to recover the unmerged files?)

It should always be possible to have branch2 diverge away earlier - say right after v2. But the intention is for branch1 and branch2 to stay converged.

As for multifile branches:

Back to my earlier post: to incrementally merge two multifile branches, we try to converge file objects a few at a time, until the whole branches are converged. At that point we can choose to eliminate or combine the branches, or leave them independent but converged.

Actually, I really should not be talking about converged branches. These are better called streams, or lines of development, that occasionally converge, in whole or in part, but which remain independent and which may subsequently diverge.

Note also that multifile branches or streams may be converged in some files but not others. This could be a way of using streams to basically conditionalize code, at file granularity - e.g. distinct .logins used at different systems. I've tried to use conventional branches to track this sort of thing, and it is a real pain. Perhaps this convergence idea would make it easier.

Finally, it is interesting that this is really quite relevantr to my recent hardware interest in SIMT processors. Threads may drift into or out of coherence, aka convergence. Version control systems continue to be a good way to work out ideas for microarchitecture such as the maintenance of speculative state.

Git submodules

Appear to be like CVS's modules.

Long ago I gave up on CVS's modules. I found that checking out subtrees worked reasonably well.

I want to figure out how to check out ad in just particular objects or directory subtrees from a git repo. Checking out should be easy enough, particularly if just exporting: scan whatever head of branch or contour you want, and check out the objects that match a criterion such as "under this tree".

Checking out the history is a bit more challenging. If file objects have always been at the same position in the tree. no worries. If file objects have moved ... moving within the sub-tree that is being checked out, agaiun no worries. But if they have moved into or out of the sub-tree being checked out, what should you do?

I lean towards checking out the history objects, but somehow preventing checking out of file version objects that are outside the current sub-tree. You could read about them in the log, diff against them, do the equivalent of cvs update -p on them to look at the contents ... but the nice default of checking out a version of such a file object into its historical position in the filesystem would not apply to such out-of-bounds file objects.

Checking in is more of a challenge, mainly because of assumptions wrt atomicity of multi-file, whole-project, checkins. Clearly this atomicity cannot be supported at all times if my goal of checking in/out individual file objects and subtrees is to be supported.

However, I think that we should support such atomicity as much as possible, since so many people like it. The lack was one of the biggest complaints about CVS.

I think that it boils down to questions such as "What is a branch?" and "What is the head of a branch?" Whole project commits correspond a set of file objects, with a guarantee that the next checkin on the branch will be linked backwards to the current set.

E.g. imagine a smal multi-file system, all on the same branch:

     F1/v1 -> F1/v2 -> F1/v3
     F2/v1 -> F2/v2 -> F2/v3
     F3/v1 -> F3/v2 -> F3/v3

Checking in/out the whole project conceptually advances the versions of all file objects in lockstep, or, equivalently

Project/v1{F1,F2,F3} -> Project/v2{F1,F2,F3} -> Project/v3{F1,F2,F3}

Checking in some, but not all, of the files corresponds to something like this:

     F1/v1 -> F1/v2 -> F1/v3
     F2/v1 -> F2/v2 -> F2/v3 -> F2/v4
     F3/v1 -> F3/v2 -> F3/v3 -> F3/v4

or almost equivalently:

     Project/v1{F1,F2,F3} -> Project/v2{F1,F2,F3} -> Project/v3{F1,F2,F3}
            -> IncompleteProject/v4{F2,F3}

On this branch, you could ask for the most recent whole-project atomic commit, or the latest version of all files.

Part of the problem is that the "filesystem" consists of the logical equivalent of directory files that list all filenames in a given version of the whole project, and point to the version (content) of the corresponding file objects. The filesystem does not really have a representation for a version of an individual file, except its content. Having per-file-object version tracking is conceptually easy, but would be more work, and would be liable to inconsistencies.

This is not a conceptual problem, just an implementation artifact. If we had database views...

We could keep the whole project viewpoint by doing something like

     Project/v1{F1,F2,F3} -> Project/v2{F1,F2,F3} -> Project/v3{F1,F2,F3}
            -> Project/v4{F1/v3,IncompleteProject/v4{F2,F3}}

git usage model - individual files, subdirs

My attempts o use git are running into usage model problems.

As mentioned before, I have checked two diverged CVS trees into different branches of the same git repository. I want to merge the branches - but not all at once. I want to merge on a file by file, directory by directory, subprobject by subproject, basis, testing as I go.

Unfortunately, apparently git-merge is all or nothing. It merges all of the files in a branch, but does not have the option of merging one at a time.

I.e. it does not seem to have the ability to do what in CVS might look like

   cvs update -rbranch1
   cvs update -jbranch2 foo.c
   cvs ci

merging only foo.c

This is the rub. Git seems only capable of treating the entire repo, the entire branch, as an object, whereas I look on files, directories, etc. as potential subobjects.

I am falling back to checking out two different git repos for the branches, and merging "manually" by copying files from one to the other. But this sucks.

Must have FireFox

Every time I get a new PC I try to live just within the standard Windows environment.

But I just can't take Internet Explorer. Specifically, IE6: company IT does not allow IE7 deployment. In any case, I don't think that IE7 fixes this problem.

I have lots of nice pixels on my screen. But this makes standard fontsize hard to read.

I tried increasing Windows display fonysize. But many, many, webpages break.

Moreover, even with the larger fonts, some webpages are still too hard to read.

Gotta have FireFix, with the ability to easily change fonts on a page by page basis. With more font sizes than MS IE supports. Just can't stand this laptop otherwise.

Implications of paying for data transfers at home

More and more broadband providers are starting to set limits on bandwidth available to home broadband users, and/or to charge for "excessive" bandwidth use.

This will impacting folks who download movies.

But it may also impact folks like me,who don't download movies, but who do work at home. My biggest bandwidth usages are:?

a) downloading software to install on my work PC

This includes both open source software like Open Office or Cygwin, as well as company provided software. I often do such downloads overnight, at home, instead of at work, so as not tio impact work time.

b) backing up my lapop PC - company provided backup software that mainly, again, runs overnight, typically when I am at home.

c) installing software updates, bug patches, etc.

Again, stuff pushed to me by IT.

If I run into such bandwidth limitations at home, how will it impact productivity at work?

This is another nail in the coffin

a) of the "free ride" companies have been getting on home broadband

b) of laptop PCs - if my work PC were a desktop at work, it could use company bandwidth

c) of PCs overall - keep the high bandwidth stuff on a centralized server.