[Vm-dev] File I/O performance (Re: [Pharo-dev] [VM] Windows VM SurfacePlugin.dll LoadLibrary failure)

[David T. Lewis]

On Sat, Apr 01, 2017 at 10:06:34AM +0200, Nicolas Cellier wrote:
> AFAIR, flushing after each write was a workaround against some linux
> problem.

This was specifically for the case of writing a chunk to e.g. the changes
file. The extra flush for that case is protecting for two processes writing
to the same stream. This would not affect overall read/write performance
though.

> I think that flush is not exagerately expensive on linux, because there's a
> difference between flush (transfer data from user space buffers to kernel
> space buffers) and sync (transfer data to the disc).

The Windows VM uses a different FilePlugin. The I/O calls are made at the
HANDLE level, not at the (FILE *) level as in the Unix plugin. A HANDLE on
Windows would be loosely equivalent to a file descriptor number on Unix.

If #primitiveFileFlush is intended to mean "flush everything from the VM out
to the operating system", and #primitiveFileSync is intended to mean "tell the
operating system to flush it all way to disk", then flush would be a no-op
on Windows, and you would call the sync primitive only in cases where you
really need to guarantee a flush to disk.

An interesting project would be to adapt the Unix plugin to run on Windows
(maybe it already just works, I never tried), or conversely to write a Unix
plugin that reads and writes to the file descriptor level as in done with
the Windows plugin.

My guess is that both approaches are valid, and they probably will have
differing performance advantages based on the I/O workload. But someone
would have to try it to find out :-)

Dave


> 
> In Win32 VM, the fileFlush and fileSync primitives do the same call to
> FlushFileBuffers.
> The msdn tells it's a bad thing to repeatedly call FlushFileBuffers
> https://msdn.microsoft.com/en-us/library/windows/desktop/aa364439(v=vs.85).aspx
> I don't see such thing as a "cheap" flush on windows.
> 
> But do we really need to flush?
> Couldn't we rather implement a linuxWorkaroundFlush: aFile method that
> would flush on linux and do nothing anywhere else?
> 
> 2017-04-01 6:40 GMT+02:00 Martin Dias <tinchodias at gmail.com>:
> 
> > Hi all. I rarely use Windows but this night entered again to repeat the
> > load test in Pharo.
> >
> > Cyril: I uninstalled the antivirus and excluded .changes and .images files
> > in the "Windows Defender" (didn't know it exists). I may note some speed up
> > but not much, so it wasn't THE solution.
> >
> > Peter: I didn't check firewall but "Compiling methods..." is what stays on
> > screen so it shouldn't be the network.
> >
> > Eliot: this time I profiled it and can highlight from report that Leaves
> > shows 91% in StandardFileStream>>primFlush:. I guess it isn't too
> > surprising since file writing is expensive but it's a lot. However I didn't
> > compare it yet with the respective report in linux. For the record, I
> > attach the report.
> >
> > Code evaluated:
> > AndreasSystemProfiler spyOn: [
> > Gofer new
> >         url: 'http://www.smalltalkhub.com/mc/ObjectProfile/Roassal2/main';
> >         package: 'ConfigurationOfRoassal2';
> >         load.
> > #ConfigurationOfRoassal2 asClass load.]
> >
> > Regards
> > Mart??n
> >
> >
> > On Thu, Mar 16, 2017 at 8:03 PM, Cyril Ferlicot D. <
> > cyril.ferlicot at gmail.com> wrote:
> >
> >> Le 16/03/2017 ?? 05:11, Martin Dias a ??crit :
> >> > And it takes >30 minutes to load Roassal2, while takes <1 minyte in
> >> > Linux (same notebook). Script:
> >> >
> >>
> >> Do you have an anti-virus? If yes did you excluded the .changes and ombu
> >> files from the files to check? If not the anti-virus might analyze your
> >> file at each compilation.
> >>
> >> --
> >> Cyril Ferlicot
> >>
> >> http://www.synectique.eu
> >>
> >> 2 rue Jacques Pr??vert 01,
> >> 59650 Villeneuve d'ascq France
> >>
> >>
> >