WriteBarrier etc

[Chris Muller]

Hi!

> I have made some further discoveries and actually see the degradation on the
> Magma server (vs in my client image). I am doing some tests with SSD disk,
> commit sizes and will update this thread more about this as I progress and
> write it up on a blog.
> I think this is key to the performance:
>
> 68.9% {11294ms} [] MaHashIndex>>add:at:
>   25.1% {4112ms} MaHashIndexRecord>>canAdd:
> ...
>   24.2% {3968ms} MaHashIndex>>selectRecord:
>    24.0% {3931ms} MaHashIndex>>readRecord:at:
> ...
> 17.2% {2814ms} MaHashIndex>>writeRecord:
>   17.1% {2799ms} MaFileRecordBroker>>writeFrom:at:

Some background:

  - MaHashIndex is a file-structure that provides access to Integer
key->value pairs.

  - MaHashIndex is the back-end representation of a single
MagmaCollection object.  It is named with the oid of the
MagmaCollection object to ensure filename uniqueness.  The keys are
the oids of the objects added to the collection, the values are
unused.

  - Each index added to that MagmaCollection causes TWO additional
MaHashIndex files to be created on the backend server.  They are named
with the oid+[index attribute name].   The keys are an integer
representation of the indexed "value" object, while the values are the
oids at that key.  The additional MaHashIndexFile simply reverses the
key->value pairs; necessary to support complex queries.

So, for example, if there are two indexes on a MagmaCollection, each
object added to that collection cause MaHashIndex>>#add:at: to be
called FIVE times (once for MC, plus twice for each index).

On top of this, MaHashIndexes are inefficient at adding duplicate keys
because, even though it is quick to locate the first of any key in the
file, it has to read sequentially the remainder to find the end where
to append the new duplicates.  It may be possible to design a fix to
this.

But before that, one can see how MaHashIndex[Record]Tester runs every
combination of key and record sizes, and use that to determine that
one that performs best for the keySize you need and value distribution
of those keys (you'd have to feed it a sampling of your input data).
Also important is to reconsider the model itself.  It's easy to lean
on MagmaCollections when designing if their costs are not fully
understood, even when other design options could meet the actual
requirements without employing MagmaCollections..

You may have already seen it, but it is still a quite accurate
description of the MaHashIndex file format and algorithm (starting on
page 4).

  http://wiki.squeak.org/squeak/uploads/2665/magma_file_format.pdf

> But this has to wait since I have run into something very annoying (and I
> cannot understand how my image became corrupted):
> Basically, I have a MagmaSession disconnect hanging on (this time I tried
> primDisconnect, the stack is almost the same on disconnect):
>
> Mutex>>critical:
> MaNetworkServerLink>>submit:
> [] in [] in [] in MagmaSession>>submit:to:
> [] in [] in BlockClosure>>maOn:do:on:do:on:do:
> BlockClosure>>on:do:

If you have another debugger open inside the Mutex then, yes, the next
one will wait.

You should be able to close the other debugger, then proceed the one
doing the disconnect.

> [] in BlockClosure>>maOn:do:on:do:on:do:
> [] in [] in BlockClosure>>maOn:do:on:do:
> BlockClosure>>on:do:
> [] in BlockClosure>>maOn:do:on:do:
> BlockClosure>>on:do:
> BlockClosure>>maOn:do:on:do:
> BlockClosure>>maOn:do:on:do:on:do:
> [] in [] in MagmaSession>>submit:to:
> BlockClosure>>ensure:
> CursorWithMask(Cursor)>>showWhile:
> [] in MagmaSession>>submit:to:
> Mutex>>critical:
> MagmaSession>>submit:to:
> [] in [] in MagmaSession>>primDisconnect
> MagmaSession>>linksDo:
>
> The mutex is held by a suspended Process in Debugger
> class>>morphicOpenOn:context:label:contents:fullView:
>
> I've tried to resume the process but MethodContext>>cannotReturn: is coming
> back to me  and some poking around tells me that the title of the debugger
> was: "'MaObjectSerializationUserError: No createProxyBlock specified.  See
> MaObjectSerializer>>toCreateProxies:'"

This can happen if an object changed while it was being serialized.
This is not supported.  Are you serializing in a separate Process?

> Now, how can I break this Mutex to clean up the MagmaSession?
>
> If I try to terminate the process - it simply goes into suspend.
> Since I am doing changes, testing and ALT-. to break long commits or other
> vicious cycles - it would be good to know how to drop all Magma tracked
> objects from memory (i.e. all "Magma state", oids etc) and reset the
> database?
> What is the best strategy to do this?

When my image gets this wrecked and I can't abort it (e.g., exit image
without saving), then I use clean up utility methods:

    MagmaSession cleanUp.   "print it.    It closes and cleans up all
in memory MagmaSessions and their references, reports the number of
intstances before->after."
    MaInstaller chasePointerToAnyMagmaSession     "Pick any
MagmaSession instance and show the trail references to facilitate
getting rid of it with another run of MagmaSession cleanUp."

Best,
  Chris