<div dir="ltr">Hi Tim,<br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 29, 2016 at 10:24 AM, tim Rowledge <span dir="ltr"><<a href="mailto:tim@rowledge.org" target="_blank">tim@rowledge.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
> On 29-06-2016, at 7:08 AM, Levente Uzonyi <<a href="mailto:leves@caesar.elte.hu">leves@caesar.elte.hu</a>> wrote:<br>
><br>
> This seems to be a missing #flush after changes are written to the file.<br>
> Without #flush both processes (unix) will maintain their own version of the file in memory.<br>
<br>
Pretty much exactly what I was about to type. We just had part of this discussion wrt Scratch project files on the Pi - adding flush/sync etc.<br>
<br>
In many cases letting an OS buffer the buffering of the buffer’s buffer buffer is tolerable - though insane, and wasteful, and a symptom of the lack of careful analysis that seems to pervade the world of software these days - because nothing goes horribly wrong in most cases. Everything eventually gets pushed to actual hardware, the system doesn’t crash, evaporate, get zapped by Zargon DeathRay(™) emissions, the power doesn’t get ripped out etc. Evidently, on a Pi in a classroom we can’t assign quite such a low probability to the Zargon problem.<br>
<br>
However, the changes file is supposed to be a transaction log and as such I claim the data ought to hit hardware as soon as possible and in a way that as near as dammit guarantees correct results. So the mega-layer buffering is An Issue so far as I’m concerned.<br>
<br>
We also, still, and for decades now, have the behaviour I consider stupid beyond all reason whereby a file write is done by<br>
a) tell the file pointer to move to a certain location<br>
b) think about it<br>
c) oh, finally write some text to the file.<br>
With the obvious possibility that the file pointer can be changed in b)<br>
Then if you can open-for-write a file multiple times, how much confusion can that actually cause? What about a forked process with nominally the same file objects? Are we at all sure any OS properly deals with it? Are we sure that what is purportedly ‘proper’ makes any sense for our requirements?<br>
<br>
The most obvious place where this is an issue is where two images are using the same changes file and think they’re appending. Image A seeks to the end of the file, ‘writes’ stuff. Image B near-simultaneously does the same. Eventually each process gets around to pushing data to hardware. Oops! And let’s not dwell too much on the problems possible if either process causes a truncation of the file. Oh, wait, I think we actually had a problem with that some years ago.<br></blockquote><div><br></div><div>The thing is that this problem bites even if we have a unitary primitive that both positions and writes if that primitive is written above a substrate that, as unix and stdio streams do, separates positioning from writing. The primitive is neat but it simply drives the problem further underground.</div><div><br></div><div>A more robust solution might be to position, write, reposition, read, and compare, shortening on corruption, and retrying, using exponential back-off like ethernet packet transmission. Most of the time this adds only the overhead of reading what's written. </div></div><div><br></div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><span style="font-size:small;border-collapse:separate"><div>_,,,^..^,,,_<br></div><div>best, Eliot</div></span></div></div></div>
</div></div>