[Box-Admins] [Vm-dev] Could someone add a VMMaker Inbox project to source.squeak.org?

[Chris Muller]

Agree, given Levente's measurements seemingly so short for diffing
VMMaker, I'd like to know what's taking our server so long...  My
guess is it was busy indexing a previously-saved version of VMMaker at
the time.  Even running at a lower priority, there is some
Mutex control in there, so there still could be some blocking on
the indexing.  I should look into spawning that as a separate VM
process when I upgrade it...   Also, that feature can be turned off on
a per-Project basis if that pain is worse than the benefit of the mc
revision history.

It's high on my list.  Thanks for your patience.

 - Chris

PS -- In case anyone is interested in looking at the code and possibly
getting involved, it installs easily into a trunk image:

    Scanner allowUnderscoreAsAssignment: true.
    Installer new merge: #squeaksource

On Tue, Oct 2, 2018 at 6:23 PM Bert Freudenberg <bert at freudenbergs.de> wrote:
>
> [Let's move that part of the discussion here]
>
> Our virtual server might be significantly slower than your machine. But it still seems suspiciously slow, agreed.
>
> - Bert -
>
>
> On Tue, Oct 2, 2018 at 3:54 PM Levente Uzonyi <leves at caesar.elte.hu> wrote:
>>
>>
>> On Tue, 2 Oct 2018, Chris Muller wrote:
>>
>> >
>> > There are a number of factors that combine to lead to occasional
>> > timeouts depending on the package.
>> >
>> > In this case, VMMaker is a huge package, and it takes a long time for
>> > the server to open up the .mcz and materialize its contents.  That's
>> > why, for example, the diffing operation, which doesn't use Magma at
>> > all, times out.
>>
>> I measured diff creation on my machine using various VMMaker versions:
>> - creating a diff of 2 versions takes less than 2 seconds
>> - from this 2 seconds the actuall diff creation takes about 150ms, which
>> can easily be halved with some basic optimizations
>> - more than a second is spent by DataStream reading and mangling strings
>> for no real reason. We should be able to significantly reduce that as
>> well.
>> - the remaining few hundred milliseconds is spent on parsing the ancestry,
>> which can only be solved by using a new non-recursive ancestry format
>> which doesn't require you to parse everything to read a single entry.
>>
>> But 2 seconds is not enough to get a 504 response, so there must be some
>> other reason for those timeouts.
>>
>> Levente
>>
>> >
>> > When a new version of VMMaker is saved, once again, it must be opened
>> > up by the server and then, all of its MCDefinitions are enumerated and
>> > indexed into huge Dictionary's stored by Magma.  This takes a long
>> > time and even though it occurs in a background local Smalltalk
>> > Process, it can affect local image performance.
>> >
>> > Another cause are the many thousands of files which the server must
>> > constantly refresh its directory cache, again and again, as it grows
>> > ever larger.
>> >
>> > Yet another issue is the dual persistence -- currently since we use
>> > both File-based AND Magma, we have to save a "data.obj" file, and so
>> > we're forced to keep the entire model into memory instead of taking
>> > advantage of Magma's ability to work efficiently with subgraphs of
>> > models.  But this is the only way we can have the MC history and
>> > origin functions.
>> >
>> > As I run the exact same code repository on my laptops to manage my own
>> > code as runs source.squeak.org, after Squeak 5.2, I plan to refresh
>> > and test my own codebase, bring it up to 64-bit, latest Magma, and
>> > make some optimizations.  I'll test it by running my own development
>> > with it for a while and then apply the same upgrades to
>> > source.squeak.org.
>> >
>> > - Chris
>> > On Tue, Oct 2, 2018 at 12:08 PM Levente Uzonyi <leves at caesar.elte.hu> wrote:
>> >>
>> >>
>> >> My impression is that these slowdowns are present since the update to the
>> >> Magma backend.
>> >> I haven't seen the code yet, but I presume the code accessing data from
>> >> Magma is responsible for this, and someone familiar with Magma can easily
>> >> fix it.
>> >>
>> >> Levente