Hi there, I've looked through the mailing list but didn't find help... :(
We've been trying to get magma to work in a useful state the whole day now. ;) We have a clean 3.9 image, got a server and a connection to it. however batch inserting 15000 entries into a magmacollection seems to be very slow. We couldn't get the MagmaBenchmark MagmaBenchmarker runRemoteBaseline: false to work. It displays a "cannot connect to 127.0.0.1 5109" error message.
We tried to load www.squeaksource.com/MagmaTester, but this gaves an "Outage" error as reported earlier on this list...
Help is appreciated, regards, Martin
Hi Martin, thank you for at least trying to run the benchmarker so we could have a common understanding of "slow". For the error you got, "cannot connect to 127.0.0.1 5109", is it possible that loopback address is wrong or maybe the computer was not allowed to open a port and therefore Squeak's Socket class was unable to establish a connection? I did try
Depending on several factors, you may expect adding 15000 objects to an unindexed MagmaCollection to take about two minutes on a medium-speed computer.
Bulk loads aren't fast, but there are a few settings that can ensure they maintain optimum speed, otherwise it can degenerate badly.
Would you share your load script and MessageTally that I might see something obvious?
"Magma tester" off of *SqueakMap* is a stable version you may wish to start with.
Let me know how I may help, Chris
--- Martin Beck martin.beck@hpi.uni-potsdam.de wrote:
Hi there, I've looked through the mailing list but didn't find help... :(
We've been trying to get magma to work in a useful state the whole day now. ;) We have a clean 3.9 image, got a server and a connection to it. however batch inserting 15000 entries into a magmacollection seems to be very slow. We couldn't get the MagmaBenchmark MagmaBenchmarker runRemoteBaseline: false to work. It displays a "cannot connect to 127.0.0.1 5109" error message.
We tried to load www.squeaksource.com/MagmaTester, but this gaves an "Outage" error as reported earlier on this list...
Help is appreciated, regards, Martin _______________________________________________ Magma mailing list Magma@lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma
Hi Chris, our first try was indeed with the SqueakMap release. I'll give it a try now with some ip-changes. However our source code is somewhat silly because we're trying to switch from GOODS to Magma in our project. It is very far implemented so we tried to implement a Wrapper for a MagmaCollection to the interface of a TSTree. Perhaps this is the speed problem.
However, as we cannot get the benchmark to run we're not really convinced by now, because even MagmaTest kickoff doesn't work. It looks for a Magma*.1.st file, which we don't have.
But we'll keep trying... ;)
Greets, Martin
Chris Muller schrieb:
Hi Martin, thank you for at least trying to run the benchmarker so we could have a common understanding of "slow". For the error you got, "cannot connect to 127.0.0.1 5109", is it possible that loopback address is wrong or maybe the computer was not allowed to open a port and therefore Squeak's Socket class was unable to establish a connection? I did try
Depending on several factors, you may expect adding 15000 objects to an unindexed MagmaCollection to take about two minutes on a medium-speed computer.
Bulk loads aren't fast, but there are a few settings that can ensure they maintain optimum speed, otherwise it can degenerate badly.
Would you share your load script and MessageTally that I might see something obvious?
"Magma tester" off of *SqueakMap* is a stable version you may wish to start with.
Let me know how I may help, Chris
Hi Martin,
our first try was indeed with the SqueakMap release. I'll give it a try now with some ip-changes.
Yeah, SqueakMap is the one to use.
However our source code is somewhat silly because we're trying to switch from GOODS to Magma in our project. It is very far implemented so we tried to implement a Wrapper for a MagmaCollection to the interface of a TSTree. Perhaps this is the speed problem.
What is a TSTree? Perhaps it would be simpler to just use the TSTree directly with Magma to get it working and then worry about the wrapper or upgrading the code to MagmaCollections later..?
However, as we cannot get the benchmark to run we're not really convinced by now,
The benchmark listens on a port and sends to that same port from the same image. It sounds like there may be something with your OS or configuration that is preventing this..? What OS are you running?
You could also runLocalBaseline: as it would not be appreciably slower with regard to adding to MagmaCollections than the runRemoteBaseline.
because even MagmaTest kickoff doesn't work. It looks for a Magma*.1.st file, which we don't have.
You may find these needed files on the Magma documentation page, "About the MagmaTester". Here is the link:
http://wiki.squeak.org/squeak/2661
Oops, I do see the files are named without the ".1." in the middle on the Swiki. I am having trouble getting them renamed here (getting a "Proxy Error") but you can just put these two files into your image directory and rename MagmaPerson.st to MagmaPerson.1.st and the same for MagmaContact and then the tests will work.
You may wish also to install OSProcess so you don't have to launch the images manually. The tests do take about an hour or so to run all the way through.
But we'll keep trying... ;)
Ok great! It does work, so I'm sure you're really close.
I saw your other email with the profile, I'll look at that and reply again.
Regards, Chris
Just as a side note, I had trouble in the past when trying images that had GOODS and Magma packages loaded. I don't have a scientific proof about it but everything worked fine for me with exclusively either one of them loaded, and since then, I have not mixed them anymore. Besides I have only been using Magma.
That's interesting to know, because Magma does not make any changes to the base image, as interoperability is a high priority. If Goods has base changes I imagine Magma could be affected.
But loading Magma should not affect Goods or anything else.
Thanks..
--- Ramiro Diaz Trepat ramirodt@gmail.com wrote:
Just as a side note, I had trouble in the past when trying images that had GOODS and Magma packages loaded. I don't have a scientific proof about it but everything worked fine for me with exclusively either one of them loaded, and since then, I have not mixed them anymore. Besides I have only been using Magma. _______________________________________________ Magma mailing list Magma@lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma
Chris Muller schrieb:
What is a TSTree? Perhaps it would be simpler to just use the TSTree directly with Magma to get it working and then worry about the wrapper or upgrading the code to MagmaCollections later..?
A TSTree is a tree structure for GOODS, which allows a bag of objects to be loaded from the database one at a time. They can be accessed via a string. However, as we only want to have a collection of objects index with a string (possibly having no problems with concurrency), we chose the MagmaCollection.
You could also runLocalBaseline: as it would not be appreciably slower with regard to adding to MagmaCollections than the runRemoteBaseline.
Yes, this one works. Our insert seems to be that slow, because we use a string index (MaSearchStringIndex). It took 15 Minutes for our 14,000 objects. We already sent the message tally for the first 1000 ones.
But now we got another problem: When committing changes to the server, there are some cases in which we always get the following error message: "MagmaUserError: Invalid reference, an object may not change during serialization". Some hints to this error message would be appreciated...
Regards, Martin
Martin Beck schrieb:
But now we got another problem: When committing changes to the server, there are some cases in which we always get the following error message: "MagmaUserError: Invalid reference, an object may not change during serialization". Some hints to this error message would be appreciated...
Got this one. We had a link to the database session somewhere in our data structure. Obviously, without trying to serialize this one, too, it works...
Thanks, Martin
A TSTree is a tree structure for GOODS, which allows a bag of objects to be loaded from the database one at a time. They can be accessed via a string. However, as we only want to have a collection of objects index with a string (possibly having no problems with concurrency), we chose the MagmaCollection.
Ah, ok. Yes, a MagmaCollection permits concurrent adds and removes.
You could also runLocalBaseline: as it would not be appreciably
slower
with regard to adding to MagmaCollections than the
runRemoteBaseline. Yes, this one works. Our insert seems to be that slow, because we use a string index (MaSearchStringIndex). It took 15 Minutes for our 14,000 objects. We already sent the message tally for the first 1000 ones.
Wow, 15 minutes?! Depending on what the issue is (hard to know without script and MessageTally) it may be
Got this one. We had a link to the database session somewhere in our data structure. Obviously, without trying to serialize this one, too, it works...
Ok, you are not the first to try this. So I have made an enhancement where any reference to a MagmaSession will resolve, upon materialization, to whatever session is doing the materialization. It'll be regarded as an "immutable" object in terms of change-detection, and all references to any MagmaSession will reconcile to the same oid (per repository).
I'll test and include this fix with the next release which will (hopefully) also include these asynchronous index updates.
- Chris
Ok, we now tried to insert 1000 entries and got a MessageTally of it. It seems, that we are waiting for the TcpSocket very long, meaning that the server is too slow?
- 32310 tallies, 32366 msec.
**Tree** 98.8% {31978ms} CWDatabase class>>initialSetup 80.3% {25990ms} CWDatabase class>>initialSetupCityDBOn: |80.3% {25990ms} CWDatabase class>>loadCityDBFromFile:on: | 74.2% {24016ms} MagmaSession>>commit | |74.2% {24016ms} MagmaSession>>commitAndBegin: | | 48.7% {15762ms} MagmaSession>>submit: | | |48.7% {15762ms} MaTcpRequestServerLink>>submit: | | | 45.1% {14597ms} MaTcpRequestServerLink>>getByteArrayResponseFor: | | | |42.7% {13820ms} MaClientSocket>>sendData:startingAt:count:waitForReplyIn: | | | | |42.6% {13788ms} MaClientSocket>>receiveInto: | | | | | 42.6% {13788ms} Socket>>waitForDataFor:ifClosed:ifTimedOut: | | | | | 42.6% {13788ms} Semaphore>>waitTimeoutMSecs: | | | | | 42.5% {13756ms} primitives | | | |2.4% {777ms} MaObjectSerializer>>serializeGraph: | | | | 2.4% {777ms} MaObjectSerializer>>serializeGraph:do: | | | | 2.4% {777ms} MaObjectSerializer>>appendGraph:do: | | | | 2.0% {647ms} MaObjectSerializer>>append: | | | 3.6% {1165ms} MaObjectSerializer>>materializeGraph: | | | 3.6% {1165ms} MaObjectSerializer>>materializeGraph:do: | | | 3.6% {1165ms} MaObjectSerializer>>materializeGraphDo: | | 11.5% {3722ms} MagmaSession>>newCommitPackageFor: | | |9.0% {2913ms} MaTransaction>>changedObjects | | | |8.9% {2881ms} MaTransaction>>addChangesFromReadSet | | | | 8.7% {2816ms} MaTransaction>>didChange:from: | | | | 6.2% {2007ms} MaFixedObjectBuffer>>isDifferent:using: | | | | 2.2% {712ms} MaObjectSerializer>>oidFor: | | | | 2.0% {647ms} MagmaOidManager(MaOidManager)>>oidFor: | | |2.3% {744ms} MaCommitPackage>>serializeObjectsUsing: | | | 2.1% {680ms} MaObjectSerializer>>serializeGraph:do: | | | 2.1% {680ms} MaObjectSerializer>>appendGraph:do: | | 10.8% {3496ms} MagmaSession>>processNewAndRemovedIndexesIn:using: | | |10.7% {3463ms} MagmaCollection>>buildIndexes:ignoring: | | | 10.2% {3301ms} MagmaSession>>commitAndBegin | | | 10.2% {3301ms} MagmaSession>>commitAndBegin: | | | 8.6% {2783ms} MagmaSession>>newCommitPackageFor: | | | 8.5% {2751ms} MaTransaction>>changedObjects | | | 8.5% {2751ms} MaTransaction>>addChangesFromReadSet | | | 8.3% {2686ms} MaTransaction>>didChange:from: | | | 5.2% {1683ms} MaFixedObjectBuffer>>isDifferent:using: | | | 2.1% {680ms} MaByteObjectBuffer(MaVariableBuffer)>>isDifferent:using: | | 3.1% {1003ms} MagmaSession>>refreshViewUsing: | | 3.1% {1003ms} MaCommitResult>>refresh: | | 2.1% {680ms} MagmaSession>>assignPermanentOidsFrom: | 3.8% {1230ms} MultiByteFileStream(FileStream)>>fileInObjectAndCode | 3.8% {1230ms} MultiByteFileStream(ReadWriteStream)>>fileInObjectAndCode | 3.8% {1230ms} MultiByteFileStream>>fileIn | 3.8% {1230ms} MultiByteFileStream(FileStream)>>fileIn | 3.8% {1230ms} MultiByteFileStream(PositionableStream)>>fileInAnnouncing: 12.9% {4175ms} CWDatabase>>databaseSession |7.8% {2525ms} MagmaSession>>connectAs: | |7.8% {2525ms} MagmaSession>>connect: | | 7.8% {2525ms} MagmaSession>>primConnect | | 4.0% {1295ms} MaTcpRequestServerLink>>connect | | |3.2% {1036ms} MaObjectSerializer>>classDefinitionsByteArray: | | | 3.2% {1036ms} MaObjectSerializer>>materializeGraph: | | | 3.2% {1036ms} MaObjectSerializer>>materializeGraph:do: | | | 3.2% {1036ms} MaObjectSerializer>>materializeGraphDo: | | | 2.5% {809ms} MaObjectSerializer>>postMaterialize: | | | 2.5% {809ms} MessageSend>>valueWithArguments: | | | 2.5% {809ms} MaClassIdManager>>addClassDefinition: | | 3.1% {1003ms} MagmaSession>>loadClassDefinitionsFrom: | | 2.8% {906ms} MagmaSession>>materializeObject: | | 2.7% {874ms} MaObjectSerializer>>materializeGraph:do: | | 2.7% {874ms} MaObjectSerializer>>materializeGraphDo: | | 2.5% {809ms} MaObjectSerializer>>postMaterialize: | | 2.5% {809ms} MessageSend>>valueWithArguments: | | 2.5% {809ms} MaClassIdManager>>addClassDefinition: |5.1% {1651ms} MagmaSession class>>hostAddress:port: | 3.4% {1100ms} MagmaSession class>>link: | 3.4% {1100ms} MagmaSession class(Behavior)>>new | 3.4% {1100ms} MagmaSession>>initialize | 3.4% {1100ms} MagmaSession>>initializeSerializer 4.2% {1359ms} MagmaSession>>commit 4.2% {1359ms} MagmaSession>>commitAndBegin: 3.2% {1036ms} MagmaSession>>submit: 3.2% {1036ms} MaTcpRequestServerLink>>submit: 2.9% {939ms} MaTcpRequestServerLink>>getByteArrayResponseFor: 2.8% {906ms} MaClientSocket>>sendData:startingAt:count:waitForReplyIn: 2.8% {906ms} MaClientSocket>>receiveInto: 2.8% {906ms} Socket>>waitForDataFor:ifClosed:ifTimedOut: 2.8% {906ms} Semaphore>>waitTimeoutMSecs: **Leaves** 49.7% {16086ms} Semaphore>>waitTimeoutMSecs: 3.2% {1036ms} Dictionary>>scanFor:
**Memory** old -821,448 bytes young -196,264 bytes used -1,017,712 bytes free -2,558,176 bytes
**GCs** full 2 totalling 394ms (1.0% uptime), avg 197.0ms incr 3072 totalling 921ms (3.0% uptime), avg 0.0ms tenures 89 (avg 34 GCs/tenure) root table 0 overflows
Ok, we now tried to insert 1000 entries and got a MessageTally of it. It seems, that we are waiting for the TcpSocket very long, meaning that the server is too slow?
- 32310 tallies, 32366 msec.
**Tree** 98.8% {31978ms} CWDatabase class>>initialSetup 80.3% {25990ms} CWDatabase class>>initialSetupCityDBOn: |80.3% {25990ms} CWDatabase class>>loadCityDBFromFile:on: ...
Hi Martin, thanks for the MessageTally, it reveals a lot and helps communication.
Although there were 24 seconds spent doing the first commit of 1000, after eliminating "one-time" items that only occur on this first commit (see my breakdown below for details), it looks like subsequent batches of 1000 should take around 10-11 seconds each (total roundtrip), which is about what I would expect.
Of this 10 seconds each, probably about half is spent updating the back-end index files. For a long time I have considered making these updates asynchronous; perhaps now is the time to do it. It just doesn't make sense to hold everyone up while all possible keywords are inserted into a keyword index.
So, with bulk-load best-practices on the client (i.e., stubOut:, commitAndBegin:, etc.), combined with asynchronous index updates on the server, this would probably get you down to about 5 seconds per 1000 objects, and maybe even faster if you could use multiple images connected to one server.
Would that meet your performance requirements or still too slow?
Regards, Chris
"breakdown of the MessageTally" 2.5 seconds to make the one-time initial connection 1.1 seconds to do the one-time initialize of the serializer(s) ~4 seconds to do the serialization of the model loaded from the file. ~4 seconds to do a one-time extra serialize due to adding a new MagmaCollection and new class-definitions to the repository-definition. 3.5 seconds to do the one-time, enumeration of the collection (this is actually not necessary for a new collection, so this was two unnecessary requests; I will fix this) 13 seconds in the server, but this includes the extra enumeration, so its probably less.
magma@lists.squeakfoundation.org