[Vm-dev] Could someone add a VMMaker Inbox project to source.squeak.org?

[Levente Uzonyi]

On Tue, 2 Oct 2018, Chris Muller wrote:

> 
> There are a number of factors that combine to lead to occasional
> timeouts depending on the package.
>
> In this case, VMMaker is a huge package, and it takes a long time for
> the server to open up the .mcz and materialize its contents.  That's
> why, for example, the diffing operation, which doesn't use Magma at
> all, times out.

I measured diff creation on my machine using various VMMaker versions:
- creating a diff of 2 versions takes less than 2 seconds
- from this 2 seconds the actuall diff creation takes about 150ms, which 
can easily be halved with some basic optimizations
- more than a second is spent by DataStream reading and mangling strings 
for no real reason. We should be able to significantly reduce that as 
well.
- the remaining few hundred milliseconds is spent on parsing the ancestry,
which can only be solved by using a new non-recursive ancestry format 
which doesn't require you to parse everything to read a single entry.

But 2 seconds is not enough to get a 504 response, so there must be some 
other reason for those timeouts.

Levente

>
> When a new version of VMMaker is saved, once again, it must be opened
> up by the server and then, all of its MCDefinitions are enumerated and
> indexed into huge Dictionary's stored by Magma.  This takes a long
> time and even though it occurs in a background local Smalltalk
> Process, it can affect local image performance.
>
> Another cause are the many thousands of files which the server must
> constantly refresh its directory cache, again and again, as it grows
> ever larger.
>
> Yet another issue is the dual persistence -- currently since we use
> both File-based AND Magma, we have to save a "data.obj" file, and so
> we're forced to keep the entire model into memory instead of taking
> advantage of Magma's ability to work efficiently with subgraphs of
> models.  But this is the only way we can have the MC history and
> origin functions.
>
> As I run the exact same code repository on my laptops to manage my own
> code as runs source.squeak.org, after Squeak 5.2, I plan to refresh
> and test my own codebase, bring it up to 64-bit, latest Magma, and
> make some optimizations.  I'll test it by running my own development
> with it for a while and then apply the same upgrades to
> source.squeak.org.
>
> - Chris
> On Tue, Oct 2, 2018 at 12:08 PM Levente Uzonyi <leves at caesar.elte.hu> wrote:
>>
>>
>> My impression is that these slowdowns are present since the update to the
>> Magma backend.
>> I haven't seen the code yet, but I presume the code accessing data from
>> Magma is responsible for this, and someone familiar with Magma can easily
>> fix it.
>>
>> Levente