On Thu, Jun 30, 2016 at 7:07 AM, David T. Lewis lewis@mail.msen.com wrote:
On Wed, Jun 29, 2016 at 02:00:19PM -0400, David T. Lewis wrote:
Let's not solve the wrong problem folks. I only looked at this for 10 minutes this morning, and I think (but I am not sure) that the issue affects the case of saving the image, and that the normal writing of changes is fine.
I am wrong.
I spent some more time with this, and it is clear that two images saving chunks to the same changes file will result in corrupted change records in the changes file. It is not just an issue related to the image save as I suggested above.
In practice, this is not an issue that either Chris or I have noticed, probably because we are not doing software development (saving method changes) at the same time that we are running RemoteTask and similar. But I can certainly see how it might be a problem if, for example, I had a bunch of students running the same image from a network shared folder.
Maybe its time to consider a fundamental change in how method-sources are referred to. Taking inspiration from git... A content addressable key-value file store might solve concurrent access. Each CompiledMethod gets written to a file named for the hash of its contents, which is the only reference the Image getsto a method's source. Each such file would *only* need be written once and thereafter could be read simultaneously by multiple Images. Anyone on the network wanting store the same source would see the file already exists and have nothing to do. Perhaps having many individual files implies abysmal performance,
Or maybe something similar to Mecurial's reflog format [1] could be used, one file per class.
The thing about the Image *only* referring to a method's source by its content hash would seem to great flexibility in backends to locate/store that source. Possibly... * stored as individual files as above * bundled in a zip file in random order * a school could configure a database server in Image provided to students * hashes could be thrown at a service on the Internet * cached locally with a key-value database like LMDB [2] * remote replication to multiple internet backup locations * in an emergency you could throw bundle of hashes as a query to the mail list and get an adhoc response of individual files. * Inter-Smalltalk image communication
Pharo has a stated goal to get rid of the changes file. Changing to content-hash-addressable method-source seems a logicial step along that road. Even if the Squeak community doesn't want to go so far as eliminating the .changes file, can they see value in changing method source references to be content-hashes rather than indexes into a particular file?
[1] http://blog.prasoonshukla.com/mercurial-vs-git-scaling [2] https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database
Just having a poke at this, it seems a new form of CompiledMethodTrailer may need to be defined, being invoked from CompiledMethod>>sourceCode. CompiledMethodTrailer>>sourceCode would find the source code based on a content-hash held by the CompiledMethod. If found, the call to #getSourceFromFile that accesses the .changes file will be bypassed, and could remain as a backup.
cheers -ben
Dave
Max was running on Pharo, which may or may not be handling changes the same way. I think he may be seeing a different problem from the one I confirmed.
So a bit more testing and verification would be in order. I can't look at it now though.
Dave
On 29-06-2016, at 10:35 AM, Eliot Miranda eliot.miranda@gmail.com wrote:
{snip much rant}
The most obvious place where this is an issue is where two images are using the same changes file and think they???re appending. Image A seeks to the end of the file, ???writes??? stuff. Image B near-simultaneously does the same. Eventually each process gets around to pushing data to hardware. Oops! And let???s not dwell too much on the problems possible if either process causes a truncation of the file. Oh, wait, I think we actually had a problem with that some years ago.
The thing is that this problem bites even if we have a unitary primitive that both positions and writes if that primitive is written above a substrate that, as unix and stdio streams do, separates positioning from writing. The primitive is neat but it simply drives the problem further underground.
Oh absolutely - we only have real control over a small part of it. It would probably be worth making use of that where we can.
A more robust solution might be to position, write, reposition, read, and compare, shortening on corruption, and retrying, using exponential back-off like ethernet packet transmission. Most of the time this adds only the overhead of reading what's written.
Yes, for anything we want reliable that???s probably a good way. A limit on the number of retries would probably be smart to stop infinite recursion. Imagine the fun of an error causing infinite retries of writing an error log about an infinite recursion. On an infinitely large Beowulf cluster!
It???s all yet another example of where software meeting reality leads to nightmares.
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim If it was easy, the hardware people would take care of it.