Hi!
I have made some further discoveries and actually see the degradation on the Magma server (vs in my client image). I am doing some tests with SSD disk, commit sizes and will update this thread more about this as I progress and write it up on a blog. I think this is key to the performance:
68.9% {11294ms} [] MaHashIndex>>add:at: 25.1% {4112ms} MaHashIndexRecord>>canAdd: ... 24.2% {3968ms} MaHashIndex>>selectRecord: 24.0% {3931ms} MaHashIndex>>readRecord:at: ... 17.2% {2814ms} MaHashIndex>>writeRecord: 17.1% {2799ms} MaFileRecordBroker>>writeFrom:at:
Some background:
- MaHashIndex is a file-structure that provides access to Integer key->value pairs.
- MaHashIndex is the back-end representation of a single MagmaCollection object. It is named with the oid of the MagmaCollection object to ensure filename uniqueness. The keys are the oids of the objects added to the collection, the values are unused.
- Each index added to that MagmaCollection causes TWO additional MaHashIndex files to be created on the backend server. They are named with the oid+[index attribute name]. The keys are an integer representation of the indexed "value" object, while the values are the oids at that key. The additional MaHashIndexFile simply reverses the key->value pairs; necessary to support complex queries.
So, for example, if there are two indexes on a MagmaCollection, each object added to that collection cause MaHashIndex>>#add:at: to be called FIVE times (once for MC, plus twice for each index).
On top of this, MaHashIndexes are inefficient at adding duplicate keys because, even though it is quick to locate the first of any key in the file, it has to read sequentially the remainder to find the end where to append the new duplicates. It may be possible to design a fix to this.
But before that, one can see how MaHashIndex[Record]Tester runs every combination of key and record sizes, and use that to determine that one that performs best for the keySize you need and value distribution of those keys (you'd have to feed it a sampling of your input data). Also important is to reconsider the model itself. It's easy to lean on MagmaCollections when designing if their costs are not fully understood, even when other design options could meet the actual requirements without employing MagmaCollections..
You may have already seen it, but it is still a quite accurate description of the MaHashIndex file format and algorithm (starting on page 4).
http://wiki.squeak.org/squeak/uploads/2665/magma_file_format.pdf
But this has to wait since I have run into something very annoying (and I cannot understand how my image became corrupted): Basically, I have a MagmaSession disconnect hanging on (this time I tried primDisconnect, the stack is almost the same on disconnect):
Mutex>>critical: MaNetworkServerLink>>submit: [] in [] in [] in MagmaSession>>submit:to: [] in [] in BlockClosure>>maOn:do:on:do:on:do: BlockClosure>>on:do:
If you have another debugger open inside the Mutex then, yes, the next one will wait.
You should be able to close the other debugger, then proceed the one doing the disconnect.
[] in BlockClosure>>maOn:do:on:do:on:do: [] in [] in BlockClosure>>maOn:do:on:do: BlockClosure>>on:do: [] in BlockClosure>>maOn:do:on:do: BlockClosure>>on:do: BlockClosure>>maOn:do:on:do: BlockClosure>>maOn:do:on:do:on:do: [] in [] in MagmaSession>>submit:to: BlockClosure>>ensure: CursorWithMask(Cursor)>>showWhile: [] in MagmaSession>>submit:to: Mutex>>critical: MagmaSession>>submit:to: [] in [] in MagmaSession>>primDisconnect MagmaSession>>linksDo:
The mutex is held by a suspended Process in Debugger class>>morphicOpenOn:context:label:contents:fullView:
I've tried to resume the process but MethodContext>>cannotReturn: is coming back to me and some poking around tells me that the title of the debugger was: "'MaObjectSerializationUserError: No createProxyBlock specified. See MaObjectSerializer>>toCreateProxies:'"
This can happen if an object changed while it was being serialized. This is not supported. Are you serializing in a separate Process?
Now, how can I break this Mutex to clean up the MagmaSession?
If I try to terminate the process - it simply goes into suspend. Since I am doing changes, testing and ALT-. to break long commits or other vicious cycles - it would be good to know how to drop all Magma tracked objects from memory (i.e. all "Magma state", oids etc) and reset the database? What is the best strategy to do this?
When my image gets this wrecked and I can't abort it (e.g., exit image without saving), then I use clean up utility methods:
MagmaSession cleanUp. "print it. It closes and cleans up all in memory MagmaSessions and their references, reports the number of intstances before->after." MaInstaller chasePointerToAnyMagmaSession "Pick any MagmaSession instance and show the trail references to facilitate getting rid of it with another run of MagmaSession cleanUp."
Best, Chris