On Sat, Jan 03, 2004 at 11:07:30AM +0100, Cees de Groot wrote:
> Julian Fitzell  <squeak-dev at lists.squeakfoundation.org> said:
> >This has come up before but we need the file locking primitive in Squeak 
> >to support our work in OmniBase.  There has been enough interest in this 
> >in the past that I think it's worthwhile coming up with a primitive 
> >interface that is implementable in all the common VMs.
> >
> This week, I spent some work on the latest version of OmniBase which now
> has an FFI-based file class for Win32 that implements locking, share
> modes, and some other things (Win32 atomic writes). 
> 
> I'm using that class as a template to write an equivalent Posix class so
> that file locking will work on the two main platforms (Win32 and Posix,
> with the remark that my code uses both lockf() and flock() so it expects
> some BSD extensions to be present as well). On all other platforms,
> OmniBase will fall back to a 'lock-less' version but will still do
> internal locking, so OmniBase should be safe to use on any platform
> unless you want to access a single database from multiple Squeak
> instances on a platform where locking is not supported.
> 
> When this works, I was planning to rewrite the FFI stuff for Posix into
> a file I/O plugin; this work is (IMO) necessary anyway on Posix hosts
> because Squeak currently uses the stdio library, which is an unnecessary
> layer. This will leave Squeak with two methods for I/O: a 'basic one' in
> the VM core, and an 'enhanced one' in a plug-in that can be used by
> people wanting to write application servers, database servers, etcetera.

With respect to file locking, it's not clear that there are platform-independent
semantics that would make sense to Squeak, but it would certainly be interesting
and helpful if someone can think of a reasonable subset of functions that has
some hope of being supported across VMs.

It would be straightforward to include file locking calls in OSProcessPlugin, at
least for Unix and Windows, because OSPP already has quite a few file control
methods. But it was never very clear to me what functions were really useful
or necessary (and writing good test cases is going to be a real pain!), so I have
not done anything about it. Feel free to borrow code out of OSPP to make a
standalone file locking plugin if that helps any, or just add the new calls to
OSPP if that's easier.

With respect to direct file IO rather than stdio, this is going to be a bit messy.
My own opinion is that the Right Thing To Do is to get all the low level IO stuff
out of FileStream and Socket, and factor it into an IOHandle (see Squeak Map for
my earlier example of this, but not updated since the network rewrite). Then
let the IOHandle worry about the specifics of IO channels, Win32 handles, Unix
file numbers, SQFile and SQSocket structures, ansychronous event notification,
and so forth. This may be too big a step for the problem at hand, but I would be
interested in whether folks think that this is a good approach or not. Frankly,
I think we are eventually going to have to break some eggs if we want to clean
up file streams and file names and such.


I reviewed some previous email (on and off list), and here are some possibly
helpful clippings:

>From Lex Spoon:
  Note, by the way, that some sort of way to do an exclusive creating open
  is all that is needed for mail files--some way to make O_EXCL|O_CREAT
  available from Squeak.  Possibly, the answer for mail locking is just to
  have a darned lockMailFileNamed: call, which does whatever is
  appropriate for the particular system.

>From Todd Blanchard:
  On Friday, March 14, 2003, at 11:10  AM, Cees de Groot wrote:
  > Now, I want to enhance the file plugin with a locking primitive. Are 
  > the posix lockf() semantics available on all the platforms?
 
  Definitely not on Windows.  You will need to come up with a different
  lock strategy not dependent on the file system.  I ran into this while
  trying to use file locks to sync separate java processes.  There is no
  atomic createAndLockFile on Windows.
 
  All unix based systems should be fine though.

>From Tim Rowledge:
  On Friday, March 14, 2003, at 10:10  AM, Cees de Groot wrote:
  > Now, I want to enhance the file plugin with a locking primitive. Are 
  > the posix lockf() semantics available on all the platforms?
  Nope. Make sure any plugin you create is either factored so they can be
  faked out or can be ignored.

>From Dave Lewis:
  On Fri, Mar 14, 2003 at 04:02:55PM +0100, Cees de Groot wrote:
  > 
  > Err, am I right in concluding that all file I/O goes through the
  > (notoriously slow) C stdio library? Ouch. That'll kill most database
  > efforts ;-). And it's probably a bit hard to go from a FILE * to a unix
  > file handle inside Squeak :-(
  
  Load OSProcess, then IOHandle, from SM. OSPP does a lot of what you want
  with files, and IOHandle makes it nicely browsable in Squeak. The quickest
  path to get file locking primitives would be to clone some of the methods
  in OSPP. See the primitives in category "primitives - files" in class
  UnixOSProcessPlugin.
 
  Most Squeak VM's use stdio (FILE *), but on Win32 you are talking directly to
  the Win32 HANDLEs.
 
  On Unix, to see how to get at a low level file handle, try something like:
    OSProcess accessor unixFileNumber: SourceFiles second ioHandle
 
  Try opening an explorer on a FileStream and looking at its ioHandle. This
  is an image-side view of the SQFile data structure in the VM.
 
  I guess I should stick some file locking prims in OSPP one of these days.