Hi guys!
First - about file format(s) - I really don't think it is a key issue but I personally would prefer something very simple. But not hardwiring a specific format into the code seems a smart move.
And also - I am not close to fileout/in yet anyway. ;)
[SNIP]
DeltaStreams, on disk, could be large. Especially if used as a kind of finer grained .changes file. Quickly scanning the file to find 'interesting' entries, without deserializing the whole file into memory, would be useful.
Again, I was thinking KISS. A stream is a directory of deltas. A delta is a gzipped file of a readable format with a name that has a number prefix so that the files have an order in the directory.
Lots of small files? Sure. A problem? I presume not with modern filesystems.
We can also just zip up such a directory if we like.
regards, Göran