Agree, given Levente's measurements seemingly so short for diffing VMMaker, I'd like to know what's taking our server so long... My guess is it was busy indexing a previously-saved version of VMMaker at the time. Even running at a lower priority, there is some Mutex control in there, so there still could be some blocking on the indexing. I should look into spawning that as a separate VM process when I upgrade it... Also, that feature can be turned off on a per-Project basis if that pain is worse than the benefit of the mc revision history.
It's high on my list. Thanks for your patience.
- Chris
PS -- In case anyone is interested in looking at the code and possibly getting involved, it installs easily into a trunk image:
Scanner allowUnderscoreAsAssignment: true. Installer new merge: #squeaksource
On Tue, Oct 2, 2018 at 6:23 PM Bert Freudenberg bert@freudenbergs.de wrote:
[Let's move that part of the discussion here]
Our virtual server might be significantly slower than your machine. But it still seems suspiciously slow, agreed.
- Bert -
On Tue, Oct 2, 2018 at 3:54 PM Levente Uzonyi leves@caesar.elte.hu wrote:
On Tue, 2 Oct 2018, Chris Muller wrote:
There are a number of factors that combine to lead to occasional timeouts depending on the package.
In this case, VMMaker is a huge package, and it takes a long time for the server to open up the .mcz and materialize its contents. That's why, for example, the diffing operation, which doesn't use Magma at all, times out.
I measured diff creation on my machine using various VMMaker versions:
- creating a diff of 2 versions takes less than 2 seconds
- from this 2 seconds the actuall diff creation takes about 150ms, which
can easily be halved with some basic optimizations
- more than a second is spent by DataStream reading and mangling strings
for no real reason. We should be able to significantly reduce that as well.
- the remaining few hundred milliseconds is spent on parsing the ancestry,
which can only be solved by using a new non-recursive ancestry format which doesn't require you to parse everything to read a single entry.
But 2 seconds is not enough to get a 504 response, so there must be some other reason for those timeouts.
Levente
When a new version of VMMaker is saved, once again, it must be opened up by the server and then, all of its MCDefinitions are enumerated and indexed into huge Dictionary's stored by Magma. This takes a long time and even though it occurs in a background local Smalltalk Process, it can affect local image performance.
Another cause are the many thousands of files which the server must constantly refresh its directory cache, again and again, as it grows ever larger.
Yet another issue is the dual persistence -- currently since we use both File-based AND Magma, we have to save a "data.obj" file, and so we're forced to keep the entire model into memory instead of taking advantage of Magma's ability to work efficiently with subgraphs of models. But this is the only way we can have the MC history and origin functions.
As I run the exact same code repository on my laptops to manage my own code as runs source.squeak.org, after Squeak 5.2, I plan to refresh and test my own codebase, bring it up to 64-bit, latest Magma, and make some optimizations. I'll test it by running my own development with it for a while and then apply the same upgrades to source.squeak.org.
- Chris
On Tue, Oct 2, 2018 at 12:08 PM Levente Uzonyi leves@caesar.elte.hu wrote:
My impression is that these slowdowns are present since the update to the Magma backend. I haven't seen the code yet, but I presume the code accessing data from Magma is responsible for this, and someone familiar with Magma can easily fix it.
Levente