Error reporting in libraries

It's a perennial problem: You have written a library dfunction, perhaps a header-only class. Usually it is silent. However, occasionally it suffers errors or warnings. The interface does not provide a nice way to return an error code - e.g. you are trying to return a string, not an int where 0 or -1 indicates failure. You want to print error messages or otherwise report errors for these hopefully rare cases.

Special case: assertion failures. If you are writing something like a simulator, with simulator stdout and stderr separate from program-under-simulation stdout or stderr, you probaby want to ensure your assertion failures go the the simulator log. This is usually easy, but in Pin simulator and program file stdout and stderr are shared.

Note: we are not just talking about error messages that are followed by death. Sometikes we are talking about warning or informational messages, after which the program should keep running.

if everything belongs to a single class, you might give the class daa members for the streams to be used. e.g.

class Foo {
private:
    std::ostream m_cout;
    std::ostream m_cerr;
public:
    std::ostream& my_cout();
    std::ostream& my_cerr();
};

with the usual methods wrapping the data member ostreams.
E.g. to allow you to print to std::cout and std::cerr if not initialized.
But not everything lies in the same object/class.

Passing around ostreams to every possible function is ugly.

Adding ostreams to every possible class is ugly.

I've occasionally passed generic "env" objects to lots of functions, where the env carries things like ostreams, and other good stuff like exit functions.
But this is ugly.

My BKM is beginning to look like ... Cpp (C preprocessor) macros COUT and CERR used everywhere. With COUT defined tio be std::cout, or my_cout(), or whatever.


Formalized as a header file COUT_CERR_ostream_redirection.hpp:

// Interface:
// 
// To use std::cout and std::cerr
//   #define USE_STD_COUT 1
//   #include "COUT_CERR_ostream_redirection.hpp"
// This is also the default - if neither USE_STD_COUT and USE_MY_COUT_FUNCTIONS are defined
// 
// To use my_cout() and my_cerr()
//   #define USE_MY_COUT_FUNCTIONS 1
//   #include "COUT_CERR_ostream_redirection.hpp"
// 
// To use neither
//   #define USE_STD_COUT 0
//   #define USE_MY_COUT_FUNCTIONS 0
//   #include "COUT_CERR_ostream_redirection.hpp"
// 
// If you want to #include this header multiple times
// caller can #define INCLUDE_COUT_CERR_OSTREAM_REDIRECTION
// - equivalent to doing a #undef COUT_CERR_OSTREAM_REDIRECTION_already_included

#ifdef INCLUDE_COUT_CERR_OSTREAM_REDIRECTION
#undef COUT_CERR_OSTREAM_REDIRECTION_already_included
#undef INCLUDE_COUT_CERR_OSTREAM_REDIRECTION
#endif

#ifndef COUT_CERR_OSTREAM_REDIRECTION_already_included
#define COUT_CERR_OSTREAM_REDIRECTION_already_included

#if !defined(USE_STD_COUT) && !defined(USE_MY_COUT_FUNCTIONS)
#define USE_STD_COUT 1
#endif

#if USE_STD_COUT
#define COUT std::cout
#define CERR std::cerr
#elif USE_MY_COUT_FUNCTIONS
extern std::ostream& my_cout();
extern std::ostream& my_cerr();
#define COUT my_cerr()
#define CERR my_cout()
#else
// let user define his or her own COUT and CERR macros
// e.g. #define COUT foo_cout() in module foo, and bar_cout() in module_bar()
#endif

#endif //COUT_CERR_OSTREAM_REDIRECTION_already_included

Implicit vs. Explicit Filenames

In an earlier post I talked about creating a UNIX command line tool for user level filesystems.

I described primitives such as

FSinF get fileSrc fileDst
get contents of fileSrc (in FSinF) and write to fileDst (in native filesystem)

FSinF put fileSrc fileDst
put contents of fileSrc (in native filesystem) to fileDst (in FSinF)

I must admit that I was thinking in terms of UNIX like filesystems, where the user specifies the filename. However, there is a more basic sort of filesystem: one where the storage system assigns the name.

My first real encounter with such a system was the MH Mail Handling system. Other mail systems are similar: when you save or file an email, you don't have to give it a name. MH gave it a number, sequentially incrementing. Outlook doesn't even do that, although there is probably a message ID somewhere. Apple Itunes' music manager is similar. The user typically looks at listing of the files in a folder, using metadata that may be extracted from the file, or otherwise associated. In other words, there is a browse/search/select interface. It is harder to automate than a simple filename interface, because uniqueness is harder to guarantee for a query selection.

Content based filesystems are similar. E.g. git, where the name is the hash.

Some content based filesystems cannot handle conflicts - two different files with the same hash. Git seems to have little provision for conflicts, although I have been told by Linus that it handls them. My own work in content based filesystems handles hash conflicts - the filename is the hash, with a version number to handle conflicts. The comparison to handle conflicts can be avoided, at the risk of data loss (such as occurs in other content based filesystes).

In any case: for a strictly hash based filesystem that does not handle conflicts, the user could calculate the hash outside and install it using a filename interface. However, for a content based filesystem such as mine, only the filesystem can return the filename.

This suggests an interface:

fileDst := FSinF put fileSrc
put contents of fileSrc (in native filesystem)
into fileDst (in FSinF). FSinF specifies the filename.

where the storage system (FSinF) returns the filename.

Call these two types of filesystem implicitly versus explicitly name filesystem.

Obviously, an implicit filesystem can be built on top of an explicit filesystem. The wrapper calculates the names.

An explicit filesystem can be built on top of an implicit filesystem, but it needs bootstrapping: one file that can be located without knowing the implicit name, that contains the mappings of explicit to implicit names.

Version Control - File Branches and Directory Tree Branches

You have file version objects. These are the contents of a file at some particular point in time.

You have file objects.

I know that file objects are not necessarily the right thing. E.g. Linus' objection, that is is better to have tools that can track pieces of code around, and automatically figure out, e.g. when a file has been renamed or a function has been moved between files. E.g. the old Cray and IBM version control systems that treated entire projects as card decks, and could version particular regions or intervals of cards. But these just say that there should be provision for subobjects at finer granularity than file objects - line objects, function objects. expression objects? Character objects? When I start thinking of XML as source code, such things become natural. Nevertheless, for now, file objects are a natural unit.

There are file name objects.

File name objects are grouped into directory tree objects. (We'll probably just say tree objects.) There obviously should be no conflicts - no two file objects should have the same pathname within a tree object. If a given file object appears at two dfferent places in a sdirectory tree, they are logicaly linked. We may want these links to be hard links a change to one fileame is reflected in a change in the others), or copy-on-write links (chanes break the links). Soft links, such as UNIX symlinks, are different sorts of file objects.

It s possibly to traverse from file version objects to file objects, and hece to filename objects and directory tree objects. And vice versa.

File objects may be versioned. Obviously.

File objects may have branches or lines. Lists of file version objects. Wth description of intent: if "on" branch1 for fileA, a new checkin of fileA is made to branchA, extending branch1.

It is possible to go from any particular file version object to file objects, and to all file object branches or lines associated with the file object.

File object branches or lines are associated with individual file objects. (Often, file name objects.) Usually they are not visible to users, since users normally care about multiple files, i.e. directory tree objects.

Directory tree objects may have branches or lines.

Directory tree branches or lines may refer to particular file object branches or lines. This is an indication of *intent*. It says that any new checkins on those file object branches or lines should be included in the directory tree branch or line. After appropriate testing.

Directory tree branches or lines may refer to particular directory tree file version sets. Which are, of course, objects themselves. A directory tree version set is a set of filename objects, file objects, and file version objects. I.e. a directory tree file version set object is a particular checkpoint of file versions.

Directory tree branches refer to sets or sequences of directory tree file version set objects. Typically aranged in a directed graph. However, where in git branches are really just these tree objects, which point to each other, in my ideal version control system directory tree branch objecs are separate from and conain additional information beyond the directory tree file version set objects.

The directory tree file version set objects are just particular points or contours of file objects. The file objects themselves have their own lines or branches. File object versions that belong to file object branches that are part of a directory tree object, but which have not yet been incorporated into a directory tree file version set object for that branch, can be said to be EXTRAPOLATIONS of the branch. Similarly, file versions (say file.v2) that belong to file branches that are between other versions file.v1 and file.v3, where file.v1 and file.v3 belong to file version set objects on the branch, but where file.v2 does not, can be said to be INTRAPOLATIONS of the branch. Logically, such intrapolated file versions belong to the directory tree's history, but they are not first class. It may be useful to know abot them, so that you can go back to them while trying to figure out where a bug was introduced. But they do not necessarily belong to a consistent set or contour, as reflected in the file version set object.

Directory tree branches or lines may be bound to different file object branches or lines at different times. Such varying tree/file branch or line bindings should be recorded historically. Notice that there is a difference beween current bindings, the bindings at any particular point in time, and the set of all such.

Branches or lines are necessarily history objects. There is a distinction between the branch, and the branch head.

In git, branches seem mainly to be poiners between objects or tgree versions. I want branches to be first class. git gives examples of rebasing, followed by garbage collection. I want the excursions and backtracking to be recorded in the tree history. Of course, it should be possible to change history - to go v1 -> v2 -> v3 -> v4 / backtrack, and get a pruned history v1 -> v2 -> v3 -> v5. But I want also to include

• v1 (time t1)
• v1->v2 (time t2)
• v2->v3 (time t3)
• v3 was backed out, and reverted to v2 (time t4)
• v3->v4 (time t5)

Now that if the file object branch or directory tree object branch is itself a file, then the slightest change in history will create a new version. (A new file object branch version file.) We can expect that they will pack nicely.

What does it mean to have a subtree branch? The subtree branch is just a directry ree branch, that does not contan all of the (current) file objects. Because checkins to subtree branches automatically are made to individual file object braches, they automatically become candidates for inclusion in other directory tree branches that refer to that file object branch.

Branching is hard or soft. Hard branches actually break the default, implicit, connection with other directory tree objects sharing a file object branch. (Paradoxically, this is amost exactly the opposite meaning as hard links.) Directory tree branches may share file object branches in a soft manner: changes to soft shared branches are candidates for inclusion , i.e. they are extrapolated, in other directory tree branches sharing that file object branch.

Directory tree branches or lines will probably be named.

File object branches or lines probably usually do not have names. They are made as side effects of the user visible directory tre branches or lines. E.g. if a directory tree branch declines to accet a file version object that was checked in on a soft shaed file object branch, then, implicitly, a new branch is probaby going to be created for that file iobject. We don't want the user to have to create such names. Creating names is hard work. We probably want to annotate the file version objecs and file branch objects with the directory tree branch or lines that led to their creation - also, for that matter, which diectory tree branches incorporate tyhem - we do nolt want to use this in the naming of the file object branch, since it would incorrectly imply that the file object branch belongs exclusively to that directory tree iobject branch. Which is not the case. A file object branch belongs to many directory tree branches or lines, especially when there is soft sharing (extrapolation or intrapolation).

Directory tree objects may be logically assembled out of other drectory tree objects; and/or directory tree object branches may be logically assembled out of other directory tree object branches. Probaby with simple assembly rules: no conflicts, or, override on a per-file-object basis, or, override on a per subtree basis, or ...

Although directory trees may be assembled this way, it may be wisest to flatten this informaion for recording into a directory tree file version set. This way we avoid revising history on one branch affecting the history recorded on others that include that branch. However, we probaby want to record the overlaying history.

This flattening is rather like AMD's build-list.

Tags are normally applied to directory tree file version set objects.

Tags may also be applied to subtree objects or file version objects. e should distinguih "whole" and "partial" tags.

Tags applied to branch objects are, naturally enough, good candiates for branch names.

Tags themselves are versioned. E.g. "This tag XYZZY applied to ... at time t0, and ... at time t1".

--

What data structure? Files? I keep coming back to relational tables - albeit ones with many conraints. But my usual objects to RDBMSes apply.

5 seconds to file mail and move to next in Outlook

Reading a lot of email, and immediately filing and moving on to next.

The command is accepted quickly enough, but it is taking about 5 seconds to move on to the next email.

During that 5 seconds, I have the subject line of the new email, but the contents of the old. Several times I have wrongly handled mail because I thought the contents were for the old subject line, not the new.

Outlook. Specifically Outlook 2007.

It would nt be so bad if I were locked out for 5 seconds while filing the old and fetching the new - if, for example, there was some visual indication such as graying the old and new messages in the subject line summary, displaying the old message greyed in the preview pane, etc.

And if new interactive commands were locked out until the display was consistent.

And if the time delay were consistently, say, 5 seconds. But with it randomly varying between 3 and 10 seconds, it is hard to know when to trst what I see on my screen. I am finding myself counting secnds every time I go to a new screen, to give it a chance to become consistent.

More Git

http://hans.fugal.net/blog/2008/11/10/git-push-is-worse-than-worthless

git slowness

Lots of coding in my checkin early and often style - I'm actually coding, rather than spending most of my time trying to figure out how somebody else's code works.

Unfortunately, git is becoming a real performance bottleneck. Especially compared to CVS.

Reason: I'm trying to use a single big repository. Logically there are many separate subprojects, but I found that using srparate git repos was a real hassle.

In CVS you can restrict the repository scans to a subtree. Apparently not so in git. Or, at least, I don't know how.

Full repo scans are great, but have a perf cost.

Towards my own git-like VC tool - user level filesystem in a directory tree, filesystem in a file.

I am not happy with git. I want to improve on it. It's depressing to think of how much good stuff would need to be duplicated, but anyway...

I want to explore my ideas on how branches, lines of development, tags, and subprojects work.

I think the filesystem oriented design is the way to go. Strictly speaking, I first saw this in svn, but Linus certainly took it further.

Think of it as an abstract interface to a filesystem - a filesystem embedded in a directory tree. Or possibly in a file. E.g. an XML file. I'll concentrate on the filesystem in a directory tree idea first, but want to always keep filesystem n a file, filesystem-in-XML, filesystem-in-tar-archive, etc., around at the back of my mind.

The API to such a user level filesystem is not UNIX kernel level open/read/write. In particular, we cannot assume file handles. No state apart from the filesystem itself.

The API must interface to an existing filesystem.

The API should be (a) in a scripting language, and (b) at the *IX command level. The mapping between them should be 1:1 as much as possible - i.e. I am lazy, and do not want to have to define both. I want to be able to automatically generate the command line tools from the scripting modules.

The filesystem should be layerable. E.g. for version control, I want a layer that handles my VC concepts - branches, tags, lines, etc. I want this layered on top of a contenht based, deduplicated, filesystem. And I want that layered on top of a compression layer, i.e. a packing layer. I want to be able to design these layers separately, and mix and match.

I want this layering so that, ideally, I can steal other people's implementations.
E.g. I might want to steal Linus' git packfiles, or bzr's packfiles.
E.g. I will probably write my own content based layer, deliberately using a hash that has lots of collisions, to prove that I can handle collisions. But I would like be able to use any other existying implementation.
E.g. I may want to implement things in a directory, or in some single file. If it is properly layered, that should just involve switching a layer. Although the performance considerations might be extreme.

I want the UNIX command line interface to support such layering. E.g. I may want to look at a filesystem in something like a .git directory tree as a VC level - or at the content based level, or at the raw pack level.

I want the API in the scripting language so that I can have the layering in the script language, somewhat efficiently. But if layering modules are written in different scripting languages, say Perl or Python or Ruby, or even in C/C++ like git, I want to be able to layer through tyhe command line as well.
I.e. command line layering required. Inside-script-language layering nice.

Given my hisory, I will probably start coding in object oriented perl. Hope to leverage lots of code out of CPAN. Call it PerlFS? No, that name is taken. How about FSinF or FSinD - FileSystem in a File and FileSystem in a Directory (Tree). Perl_FSinF, Perl_FSinD. FSinF being the generic name, since directories are just filesystems.

I'm not scared of using a scripting language. Bzr argues they have acheived good performance in Python. I like portability.
As usual, ultimately I would like the system to install as a single file. Likes scons.py. Or at least a directory.

Thoughts about the FSinF (FilesSystem in File/Directory) Interface

Can't have primitives like UNIX open/read/write/close. Thesed assume state - file descriptors.

Interface must specify both files on the FSinF filesystem, as well as data on the native filesystem.

Operations such as:

Copying

FSinF copy fileSrc fileDst
Copy fileSrc (in FSinF) to fileDst (in FSinF)

FSinF get fileSrc fileDst
get contents of fileSrc (in FSinF) and write to fileDst (in native filesystem)

FSinF put fileSrc fileDst
put contenhts of fioleSrc (in native filesystem) to fileDst (in FSinF)

If we have syntax to distinguish native and FSinF filesystem names, this might be a single command "copy", although get/put is nicely documenting.

Moving and Renaming

FSinF move fileSrc fileDst
move or rename fileSrc (in FSinF) to fileDst (in FSinF)

Again, syntax could allow moving and renaming between FSinF and the native filesystem.

Inter-filesystem operations

get and put are inter-filesystem operations. We don't need to limit ourselves to getting from the FSinF to the native filessytem. We could transfer between two different FSinFs, FSinF1 and FSinF2. Possibly might stage through the native filesystem, but not necessarily always.

Parts of Files

Don't want always to operate on whole files. Although that is natural.

May want to specify parts of files:

• byte offset regions
• line regions
• XML clauses
• record numbers

I don't think that we need to define these at the level of the FSinF API. They could be FSinF filesystem specific.

At the command line leve, these would be optional parameters.

E.g.

FSinF get -range "line 40:line 100" file


Hints and Requirements

Different filesystems have different semantics. E.g. "emove" may mean "remove completely", or "remove from current version, but keep earlier versions around".

Such semantics are FSinF specific. Optional arguments.

May want to indicate if the semantics are mandatory or optional. E.g. "remove from current version, but keep earlier versions around" motivated by desire to remove copyright infringing stuff is mandatory - and must be done. Or must receive an error if cannot be done.

E.g.

FSinF remove -require hard-removal file ...

FSinF remove -optional soft-removal file ...

Other Filesystem Operations

FSinF remove files ...

FSinF create files ....
Empty files created.
Possibler specify types.

FSinF mkdir dirname

FSinF mkfs fsinf-file-or-dirname

FSinF fsck fsinf-file-or-dirname

FSinF chperm files...
Changes permisions
Also used to change owner, group name, etc.
Expect to extend:
E.g.
May start off with UNIX permissions, or ACLs
May create my own extra permission classes and ACLs, more flexble than native OS.

FSinF metadata add file metadata...
FSinF metadata remove file metadata...
Manipulae extra metdata associated with files
Synchronization

FSinF get file1 file2 ...
atomically get consistent file versions

FSinF atomic compare-and-swap old-file oldcmpdata newfile

FSinF atomic compare-and-put old-file1 cmpdata1 newfile2

Have the "server" do the comparison, and then put/swap.

Possibly

FSinF lock file ...
FSinF unlock file ...
Although I dislike the stae implied by locking.

Layering, Access to

E.g. in a directory ~/myproject
with a subdirectory .git

I would want to be able t say

FSinF -type git move file1 file2

But, I might also want tio say, at the level of the conent based

FSinF -type git-sha-content-based-fs remove 0x6416fec17...

I might also have a .svn link, so I may want to say

FSinF -type svn get "-r4 file1 "
note how "-r4 file1" almost looks like a filename / specification to the svn filesystem.

I.e. I may want to have several different filesystems under a tree, such as .git / .cvn. Or even my old CVS / CVS.othr-reo stuff.

But I also want to be able to look at the layers in any of thenm.

Layers I want

Raw filesystem. I.e. basically a NOP.

FSinF pack:
Compacts the above.
Probably needs some form of delta specification, for delta compression.

FSinF content-based
Uses hashes to deduplicate.
Me, I want to be able to handle hash collisions. Want both stupid hashes, like UNIX compress, and fancier hashes almost guaranteed to be collsions free.

FSinF checksummed
Adds checksums, verifies integrity
Maybe even ECC

How about:
FSinF mirrored?

FSinF encrypted.

FSinF signed
bit not encrypted

FSinF version-conrol
version controlled
in my dreams, with my extended semantics.

FSinF dirtree, file, xml, tar ...