Calendars and Phones

A friend just got an iPhone. Primary uses, apart from as a cell phone and music player: PDA stuff like calendaring.

Now she keeps her calendar on her iPhone. Both personal/family, and work related.

Her employer uses Outlook Exchange Calendaring. But they do not allow calendars to be synchronized with personal calendars outside of work. So my friend simply does not use the company calendar. She says that co-workers complain, when they arrange meetings with her that appear to be free on her company calendar, which is empty, but which she cannot make because of conflicts. But she says that is so much more important to her to be able to manage her personal and family life in the same place as her work meetings, that she is willing to put up with the loss of the shared work calendar.

Perhaps I should mention that she works, not exactly part-time, but very flexible hours. She is constantly managing family commitments, getting kids to events, as well as weekend and evening work assignments. Her day is not neatly partitioned into work and non-work.

MORAL:

• People want both personal and work calendars on their devices like phones.
• Corporate security rules get in the way.

Restart Time Matters

This morning, after a 6-7am meeting, my work laptop PC hung up. Restarting it highlights another performance issue.

The original hang was not bad, but annoying. The Nvidia nView Desktop Manager, that I use to change my screen settings - e.g. to go into and out of reduced resolution, so that the folks in Israel could see my slides - left a "turd" on my screen. A "just wait" box was thrown up, and stayed around for 15 minutes, after all Nvidia tool was closed down, after I left to do something else for a while. Past experience has shown that occasionally just waiting a few minutes leads to such turds being cleaned up - but not this time.

(I suspect that I could have killed rundll32.exe from the process (not task) manager, but I am always nervous about killing processes that I do not know exactly what they do. Especially sometrhing like rundll32.exe, that gets used by may facilities.)

I could have left this turd around and kept going. But it is annoying to have a small rectangular area on your screen that won't redraw. So I decided to restart my PC, in the hopes that (a) that would clean up the turd, and (b) I could go and have breakfast while waiting for the restart.

So, I hit restart. Waited a bit. Walked away.

After my beans and tomatos on toast (there, that proves I'm a Brit, even if my accent is more American than Canadian after 24 years in the US), I went back to check. I expected that I might have to type in my hard disk password (for PGP Full Disk Encryption).

But, surprise! Actually, no surprise, just disappointment. Windows' restart had hung, waiting for me to say "OK" to killing some application. There was a flurry of such kill dialog boxes, some with timeouts, but enough without timeouts that I had to sit and wait. The rundll32.exe that I suspect was used by Nvidia, that I suspect was causing the problem, was one of them.

OK, answer all of these dialog boxes. Now, surely, I can walk away...

Nope. Come back another 15 minutes later, to see a blank screen. Not dead - when I moved the mouse, and typed ctl-alt-del, suddenly things started proceeding to shutdown again. But, it certainly was hung waiting for something to be rejiggered or unblocked.

Finally, it reboots. Watch the reboot. Type in the PGP full disk encryption password.

What a loss! What a lost opportunity! If I could have overlapped the reboot with eating breakfast, I would have had half an hour more for work.

MORAL: even the time to resatrt matters. Automation, to allow a completely automatic reboot without having to type in passwords, or anticipation - having the reboot subsystem ask, in advance "Do you want to end all processes that aren't responsive", rather than waiting to ask me about them one by one. Automation and anticipation, even of something like the reboot process, would be a good thing.

At work we often say things like "Surely people don't care about the speed of rebooting after a Machine Check". Wrong! Maybe we don't care about is as *much* as we care about normal operation. But the performance (and automation) of anything we do a lot matters. And, on Wintel PCs, rebooting happens quite a lot.

--

There may be another moral here. I like the idea of microrebooting. Perhaps I couold microreboot the display manager, and/or otherwise tell the display manager that the non-responsive process's window should not be redrawn.

Moving to the Net: Encrypted Execution for User Code on a Hosting Site

I'm gravitating towards establishing at least 2 net.presences - 1 on a shared hosting site, the other on something like Amazon ECC where I get to administer the (virtual) machine. Working on the assumption that it is unlikely that both will go down simultaneously. (The sysadmins of the shared hosting site are more likey than I am to respond quickly to some spreading net.malware that requires immediate patching.)

Plus, of course, the usual personally owned computers/servers.

The prospect of moving my "home" off computers I own to computers that hosting service provides is a bit worrisome. E.g. would I/should I trust storing my TurboTax on a hosting site? Running Gnu Cash?

Put that way, it is more attractive to use net.storage like Amazon S3 for encrypted storage, but to always try to do the secure manipulation on computers I own.

However, just think of how already all my financial information is spread across various provider websites.

Anyway... Thinking about virtual hosting with Amazon S3... How could you have enough trust to do sensitive work on such a system? Or, rather, could we build systems where the hosting sysadmins would not be able to look at the data their hosting clients are working with?

It's a lot like DRM, Digital Rights Management, the sort of thing that motivated LT. However, instead of trying to prevent the user and owner of a PC from accessing unencrypted data on his own PC, what I would like to do in the hosting space is prevent the provider of the hosting service from accessing unencrypted data that belongs to the person renting the hosting service.

Let's imagine that this could be done. In some ways it might prevent one of the IMHO big advantages of centralized hosting, which is that you could have a sysadmin scanning for problems. The sysadmin might still be able to look for usage patterns that indicate a malware breakin, but the sysadmin would certainly have an easier time of it if he could look at the data.

It would also get in the way of another of the IMHO big advantages of centralized hosting: data de-duplication. (That seems to be the new term for what I have mainly been calling content based filesystems, inspired by rsync, Monotone, git, etc.)

E.g. I might reasonably want to store a copy of every piece of software I have ever bought on my net.storage-system. E.g. I might want to store every blamed copy of Windows I have ever had a license too. So that I can restore to the PCs that they run (ran) on, whenever... Subject to licensing, of course. Now say that the host is a big site, like Google or Yahoo or Amazon. Millions of Windows users might be doing the same thing. The opportunities for data de-duplication are large. But if each user's data is separately encrypted, deduplication would not work.

These arguments lead me to suspect that completely encrypting all information, in a way that prevents the host from looking at it, is undesirable. At the very least, optionally encrypting on a file by file basis would be desirable. E.g. I might encryt my TurboTax files, but not my saved copy of Office 95.

Okay, okay... now, can we think of ways of preventing the sysadmins managing a hosting service from having full access to at least some user data?

Start off with encrypting data on disk. Encrypt it in main memory. Encrypt it into the cache hierarchy.

Decrypt only when moving between cache and registers. (Or possibly in the innermost levels of the cache hierarchy (L0, L1).)

How do you prevent the OS from looking at the unencrypted register state? On an interrupt, re-encrypt the registers as they are saved to memory.

Note that this means that you probably won't do "efficient" interrupts, that leave as many registers as possible live. You have to save them.... or, at least, you might encrypt them in-situ. Or have hardware "lock" them in-situ, and then do a lazy save to the encrypted area. But, you can't let the host look at the unencrypted registers.

Debuggers would have to be disabled while manipulating encrypted content. Or... you could allow debugging, but "lock" registers containing encrypted content. The debugger would either not be allowed to look at such registers, or might be allowed to look at "obfuscated" contents. I say "obfuscated" because, while it might be challenging to encrypt a 64B cache line, it will be almost impossible to encrypt 64bit (32bit, or 8bit) registers. Despite these attempts, the debugger can probably infer register contents (for all bits do: jmp if set). So, while I can imagine ways of making it harder for a debugger to observe sensitive data, at the very least debugging would be a very high bandwidth covert channel. Just like reading the time or reading a performance counter, debugging should be a facility that requires special permission.

Besides, why would a hosting sysadmin want to debug a user's application? The only legitimate reasons I can imagine involve trying to figure out if a user has been compromised by malware. Some intrusion detection systems singlestep execute in a debugger.

So, I am formulating a vision where the OS may run user tasks, but the user tasks may establish encryption based security that prevents the OS from looking at the user task data. There is some sort of key inside the processor that the OS cannot look at. This key is used to encrypt and decrypt data as it flows between processor registers and cache/memory. Registers are saved/restored encryted. Hardware probably defines a save area for all register state. Registers may be locked against the OS, and/or for lazy encrypted save/restore.

System calls? The classic UNIX system calls - read, write, etc. - do not require the OS to interpret the memory contents. Thedy can handle encrypted as well as unencrypted data.

Bootstrapping? How do you bootstrap this? How do you start a user process, and give it a private key, in a manner that the OS cannot see? Basically, you will have to have a small trusted kernel, trusted bootstrap. It doesn't have to be done once and only once: I believe there are already many patents on late secure bootstrap (with me as one of the inventors). So, the untrusted OS can be running, and can invoke a small trusted kernel. This small trusted kernel must have its own private key(s), and can lookup tables of user private keys it maintains on disk, and/or obtain such keys across the network, after the appropriate secure handshake and communication. For my usage model - wanting to execute my stuff at a hosting site - I would probably prefer the "access across the network" approach. However, if I wanted to run cron jobs on the host, host-side storage of user keys might be necessary. It would be necessary for the user to trust this kernel, even though the user did not trust the OS. But this is the same issue as for DRM.

This trusted kernel could be microcode. But it could just as well be user code, with the trusted key hardwired in.

This is all very much like DRM. But it differs in emphasis: in DRM, you typically only want one or a few applications - the audio or video player - to be protected from everything else running on a system. Whereas here I have described how to run user applications on a hosting site, protected from the host OS (and any other software). In this usage model it is desirable to be able to protect almost any application from the OS. The DRM audio and video subsystem can be a special case; but this hosting application wants to truly be part of the OS. Or, rather - it wants to NOT be poart of the OS, but wants to be a facility available to most processes and users under the OS.

This works for generic user processes on a generic OS, e.g. a UNIX user account. It is not just limited to virtual machines.

Can the OS intercede, e.g. by munging with the interrupt return IP or virtual memory? While we probably should not prevent the OS from such manipulations, we can prevent the OS from gaining access to sensitive data. The state and/or code being returned to can be authenticated. Untrusted code can be prevented from ever using the trusted keys.

It would be straightforward to similarly protect the code from the OS. But, while DRM may want that, I think that is much less necessary for this hosting usage model. Generic code, manipulating sensitive data. E.g. I want "cat" and "grep" to be securable. I envisage something like

sensitive-data gnu-cach dump account | sensitive-data grep my-hotel-bill > file-whose-sensitive-stuff-I-can-only-view-on-my-client

or

sensitive-data gnu-cach dump account | sensitive-data grep my-hotel-bill | sensitive-data -stdout unencrypted more

Yahoo Geocities closing - Don't Trust Free Websites

Yahoo Geocities is closing down.

For several years this was the home of my resume, and other stuff: http://www.geocities.com/andrew_f_glew.

Yahoo gives you the option of paying 5$/month for webhosting, or saving your files. Yahoo requires you to save the files one at a time, clicking Save As on a menu. There is no ability to download all of your files as a tarball or other archive. There is no WebDav access, or other filesystem-like access. I.e. there is no easy path to automatically transferring a large website.

Fortunately, I did not have many files on Yahoo.

Even more fortunately, I abandoned Yahoo Geocities a few months ago, and moved all of my stuff to Google Docs. http://docs.google.com/Doc?id=dcxddbtr_6dvpxg2cj&hl=en

However, there is no guarantee that Google Docs will remain free, or even available, forever.

Google Docs doesn't appear to have an easy way of downloading all files. Who was it that provided such a facility? One of the wiki sites that closed down?

Coincidentally, I have been shopping for a hopefully more permanent home for me and my files on the web. I've been looking at Amazon S3 and ECC. Yes, I am willing to pay.

Amazon ECC is good for virtual hosting. However, while I like being root, I would also like an ordinary user account on some UNIX system. I would like somebody else to be responsible for keeping up to date on security patches. Ideally I would like to run a wiki on such an account, but I want the wiki to run, not even as me, but as a Mini-Me, with even less privilege.

I guess I want it all: the convenience of having someone else sysadmin, but with the ability to run a small number of personal web services in deprivileged accounts.

For that matter, I'd like the hosting system to run a non-standard OS like FreeBSD, and a non-standard instruction set. ARM? Power? Is it security through obscurity?

As for my free websites:

* I learned that I still hasd a Yahoo mail account from long ago. Once again, no bulk download, unless I upgrade to pay, at which point I get POP.

* Google Docs - no automated access

* Google Mail - I have IMAP and POP access. I think I better start backing it up better.

Backing it up to S3? Wherever.

Just like there is a period of recent history that was lost, because information stopped being recorded on paper and started being recorded on quickly obsolete digital formats such as tape, floppies, etc., now we are traversing the stage where history is lost because of service providers closing down. One can only hope that we will come out the other end with storage as a commodity, with standard procedures for migration.

Gilding the Lily

By the way, I recognize that in the preceding posts I risk the sort of gilding the lily that so many software teams do when they define many, many, different comment keywords.

E.g. http://www.cs.clemson.edu/~mark/330/colwell/p6-coding-standards.html.
This is relatively small - only 4 primary keywords, with 3 secondary.
Other example standardsare much larger.

TBD

- This "flag" is used to indicate items that require later definition. It stands for To Be Defined (or Determined). The ensuing comment should provide more particulars.

NYI

- This "flag" is used to indicate items that have been defined and are now awaiting implementation. It stands for Not Yet Implemented. The ensuing comment should provide more particulars.

MACHDEP

- This "flag" is used to indicate the existence of an explicit machine dependency in the code. Again, the ensuing comment should provide more particulars.

BUG:

- This "flag" is used to indicate the existence of a bug of some form. It should be followed immediately in the comment (on the same line) with one of the keywords incomplete, untested, or wrong, to indicate its type along with a more descriptive comment if appropriate. If none of the keywords is applicable, some other descriptive word should be used along with a more extensive comment.

The most comonly used such keyword is probably

• TBD or TODO

and other common keywords include

• NYI: not yet implemented
• BUG:
• NOTE:
  

But frequency of use rapidly goes down. E,g. although I approve of the concept of MACHDEP, I hardly ever used it during the P6 project, and I had totally forgotten about it.


Comments such as these work best when automated.

E.g., although I hate doxygen, doxygen comments get automatically extracted, and are hence more likely to be seen and corrected.

E.g. ditto Perldoc. Again, I find perldoc makes code harder to read.

Preparing this blog, I found
http://www.icsharpcode.net/TechNotes/Commenting20020413.pdf
"The Fine Art of Commenting",
Bernhard Spuida, 2002.

Spuida mentions several such systems, including Microsoft's C# .NET system,
where XML tags are embedded in comments.
As indicated by Spuida's title "Senior Word Wrangler",
this motivation seems to be mainly about preparing documentation.

It was fun to re-find the Indian Hill Coding Standards
http://www.psgd.org/paul/docs/cstyle/cstyle.htm

Every few years I seem to go through a spate of reading of coding standards.
I think every programmer should be familiar with several, many, different coding standards.
So that he or she can pick parts that work well,
for projects that need those parts.

Final words:
Brian Marick's lynx.
Hypertext embedded within the code itself.
My own hope of wiki files in the source trees,
and/or linked to from the code itself.
Programming languages using extensible notations, such as XML,
allowing arbitrary structured annotations.

Test passed/failed

More wrt test monitoring.

I concluded the last post with (slightly extended):

TEST RUN: test1
TEST START: test2
TEST CHECK OKAY: test2 check1
TEST STARTED: test2.1
TEST PASSED: test2.1
TEST FINISHED: test2.1
TEST END: test2

Implying 2 top level tests, test1 and test2. Test1 is a "monad", reported by TEST RUN ooutside of START/END. Test2 is bracketed by START/END, and contains subtest 2.1.

When I started testing seriously, I thought that all tests could be classified passed/failed. That is always a worthwhile goal. If it could be accomplished automatically, it might suggest what we might express in pseudo-XML as:

<test name="test1" result="passed"/>
<test name="test2">
<test-check result="ok" test_name="test1" check_name="check1"/>
...
</test name="test2" result="passed">

My pseudo-XML allows attributes on the close. Without this, one might just expeft a TEST PASSED message immediately before the close.

However, over the years I have learned that things are not always so clear cut. While it is always best to write completely automated tests that clearly pass or fail ...

Sometimes you write tests and just run them, but do not automatically determine pass or fail.

Sometimes manual inspection of the output is required.

Sometimes you just want to say that you have run the test, but you have not yet automated the checking of the results... and sometimes, given real-world schedule pressure, you never get around to automating the checking. In such cases, IMHO it is better to say

TEST STARTED: foo
TEST TBD: have not yet automated results checking yet
TEST ENDED: foo

than it would be to just omit the test.

Oftentimes, the fact that a test has compiled and run tells you something. Or, rather: if the test fails to compile or run it tells you that you definitely have a problem.

Sometimes you can automate part of a test, but needmanual inspection for other parts. In this case, I think reporting "TEST PASSED" is dangerously misleading:

TEST STARTED: foo
TEST PASSED
TEST ENDED: foo

or, better

TEST STARTED: foo
TEST PASSED
TEST TBD: foo: need manual inspection of rest of test output
TEST ENDED: foo

I think that "TEST PASSED" tends to imply that the entire test has passed. If you say "TEST PASSED" without a label test name, it tends to imply that the enclosing test has passed.

Better to say

TEST STARTED: foo
TEST PASSED: sub-test bar of test foo
TEST TBD: foo: need manual inspection of rest of test output
TEST ENDED: foo

I have recently started using other phrases, such as "TEST CHECK"

TEST STARTED: foo
TEST CHECK OKAY: foo check1
TEST PASSED: sub-test bar of test foo
TEST TBD: foo: need manual inspection of rest of test output
TEST ENDED: foo

Q: what is the difference between a TEST PASSED: subtest and a TEST CHECK OKAY (or TEST CHECK PASSED)? Not much: mainly, the name tends to imply something about importance. Saying that a test or subtest passed seems to imply that something freestanding has passed. A check within a test seems naturally less important.

This is along the lines of assertions. Some XUnit tests count all assertions passed. While this can be useful - particularly if some edit accidentally removes thousands of assertions - I myself have found that the number of assertions gives a false measure of test effort.

It may be that I am conflating "test" with "test scenario". A "test scenario" or "test case" may be subject to thousands of assertions. Particularly if the asserts are in the infrastructure. But I really want to count test cases and scenarios.

Here's one reason why I try to distinguish #tests passed from #checks performed:

• my test monitor performs consistency checks such as tests_passed = test_cases, tests_started = tests_ended, etc.
  

What I really want is things like

• Number of tests that had positive indication of complete success - tests passed. (Or, at least, success as complete as any test can indicate.)
  
• Number of tests that had postive indication of a failure or error.
• Similarly, warnings.
• Number of tests that had no positive indication - a monad "TEST RUN" message was sen, or perhaps a TEST START/END pair, but no positive indication.
• Number of tests where failure can be inferred - e.g. TEST START without a corresponding test end.
  

Test monitoring keywords

Playing with my text-test-summarizer-monitor, a little Perl/Tcl script that looks at the output of tests, filtering things likes tests started/ended, passed/failed, looking for various common indications of problems. Throws up a Tcl widget. The closest thing I have to a green bar.

Here's an annoyance: my test may look like

TEST END: test1
TEST START: test2
TEST CHECK OKAY: test2 check1
TEST END: test2

Oftentimes I like to announce "TEST STARTED" and "TEST ENDED". (I just had to extend my scripts to handle both START and STARTED. This is useful in case the test crashes in the middle, and you never get to the test end.

However, occasionally the test infrastructure does not allow this. Occasionally I just say, once, at the end "I ran this test, and it passed". That's what I mean above by the "TEST END: test1" without the corresponding TEST START.

In XML, this would be simple:

<test name="test1"/>
<test name="test2">
<test-check result="ok" test_name="test1" check_name="check1"/>
</test name="test2">

Note:

1. I added an attribute to the closing, </test name="test2">. Although not part of standard XML, occasionally this has helped me. I call this, therefore, pseudo-XML
2. Note that test is available in both and ... form

Note that this is pretty verbose. Although human readable, I would not call it human friendly.
Because the pseudo-XML is not so human friendly, I often prefer to print messages such as

TEST END: test1
TEST START: test2
TEST CHECK OKAY: test2 check1
TEST END: test2

But here I run into terminology: I don't have a natural way of having a message, that is not confused with START/END.

First: I am reasonably happy with brackets {BEGIN,START}/{END,FINISH},
and {STARTED}/{ENDED,FINISHED}. English grammar, how inconsistent.
I want to tolerate all of these, since I have used all from time to time,
and regularly find a test suite falsely broken if I assume that I have consistently used START and not STARTED.

(I'm going to arbitrarily reject TEST COMPLETED. Too verbose. Especially the dual TEST INITIATED. At least, I'll eject until it seems I encounter it a lot.)

But a TEST END without a TEST START is too confusing. I need an English phrase that doesn't need a corresponding start. Let's try a few:

• TEST PASSED: test name.
  
  with the corresponding
  • TEST FAILED: test name
  
  However, there might be some confusion because I definitely want to use TEST PASSED/FAILED withing TEST START/END. See below.
  
  
• TEST RESULT: test name
  
  Similarly, there might be some confusion because I might want to use TEST RESULT within TEST START/END.
  
  
• TEST RUN: test name
  
  Nice because of the possible dual TEST NOT RUN
  
  

I think that I am arriving at the conclusion that any of the above, outside a TEST START/END, make sense, and should be considered equivalent to &lttest .../>

I am not currently checking for the proper nesting of tests, but I could be.

I think that it would be good to have a count of top level tests, either TEST START/END brackets or TEST outside such brackets, but ignoring stuff within the brackets.


Giving me

TEST RUN: test1
TEST START: test2
TEST CHECK OKAY: test2 check1
TEST END: test2