Hi again. Well, as usual the MessageTally is very revealing. It should have hit me before, but I finally realize now you have all of what you want to commit to the database in memory (right?) and that is why tremendous amounts of time are spent to determine which objects changed. As more and more objects become persisted in the database, Magma must scan the ever-larger Dictionary of objects on subsequent commits to see if they changed.
WriteBarrier can virtually eliminate this. It detects when instance variables change and marks those objects dirty as the changes happen, allowing the readSet to remain very small.
But really, databases are used to have object models larger than memory, and Magma wants to be a database. It is designed for OLTP-style programs that work with a relatively small chunk of the model at a time, throw it away and retrieve work with another chunk. i.e., it should perform well working with a "screen at a time". Having a large portion of the model in memory will cause changed-detection to slow down significantly (unless WriteBarrier is used), as well as slowing down refreshes of the model (changes by other users) when transaction boundaries are crossed.
For your problem, though, you have a large inter-connected object graph in memory that you want to get into the database. You can't do it piecemeal because everything is linked together, right? Therefore, the fastest way to do that is with a single commit. Did you try changing that method (MaObjectBuffer>>#ensureSpaceFor:) to allow 50-meg and letting it run all night?
Regards, Chris
On 6/14/07, Florian Minjat florian.minjat@emn.fr wrote:
Hi again Chris,
Chris Muller wrote:
Hi Florian,
Yesterday I tried again to save all the Letters in a dictionary of Players, add all the Players in Magma, then put back the Letters inside the Players. But the last part was so slow I had to stop the whole process after about one hour. I don't understand this slowliness because thoses Letters are only some references to Players which should already be inside Magma.
The resulting code gives something like this (a mailbox is a collection of Letters):
mailboxes := Dictionary new. players do: [:p | mailboxes at: (p login) put: (p mailbox). p mailbox: nil.]. session begin. players do: [:p | session root players add: p. session commitAndBegin]. players do: [:p | |magmaPlayer| magmaPlayer := session players where: [:r | r login equals: (p login)]. magmaPlayer mailbox: (mailboxes at: (p login)). session commitAndBegin]
Ok, thanks for the code, it looks real good. Except one thing that should be done to speed it up is add more than one player per commit. Add 100 or 1000 per commit. That should speed up the first part a lot.
Same comment for the second part. Instead of commitAndBegin after each one, do it every 100 or 1000. You could also choose to commit "every five seconds". I know it makes the code less clean, sorry, but this is one-time "bulk load" code anyway.
You should also send #finalizeOids to the session after each commit.
If it is still slow after doing all that, if you post a MessageTally spy it will help a lot.
I fact there are only 59 Players ^^. And the whole dump takes more than one hour so I don't think it would speed up so much. I put 5 players per commit to test.
Here is the new code (sorry for the length) :
mailboxes := Dictionary new. players do: [:p | mailboxes at: (p login) put: (p mailbox). p mailbox: nil.]. commitCount := 0. players do: [ :p | session root players add: p. (commitCount = 5) ifTrue: [ self commitAndBegin. self magmaSession finalizeOids. commitCount := 0] ifFalse: [commitCount := commitCount +1]. ].
commitCount := 0. players do: [ :p | |magmaPlayer| magmaPlayer := (session players where: [:r | r login equals: (p login)]) first. magmaPlayer mailbox: (mailboxes at: (p login)). (commitCount = 5) ifTrue: [ self commitAndBegin. self magmaSession finalizeOids. commitCount := 0] ifFalse: [commitCount := commitCount +1]. ].
You can find the MessageTally at the end. It is much quicker (36min) with the finalizeOids and five adds by commit.
Perhaps the players references by the Letters in the last block are detected in Magma as not yet inserted and it does it again ?
No, the object is available in the collection as soon as it returns from the commit.
I tried to change my method to replace the Players by their logins before the first save, then iterate through the Letters to put back the references instead of the login. I am sure the references of the Letters comes from persistent players. But it was slower because there are much more magma queries. With your answer it is useless so I came back to the first version I showed you.
Do you have an example of use of a transient variable ? I way I understand it is that I make the Letters transients so they are not serialized inside magma, then I remove the transient property to load the Letters in a second pass ?
Yes, the idea is to implement #maTransientVariables on the referencing class:
maTransientVariables ^ super maTransientVariables, #('letters')
My idea was to add a new variable temporarily, 'letterIds' or whatever. Populate this with some uniquely-identifying string or number for the letters. Commit it.
THEN, remove the #maTransientVariables method and enumerate them again, initializing each from the letterIds. Finally, remove the letterIds variable.
This is a cumbersome approach, hopefully the first one will work.
This method is a little more complex than the first one. As I need only to do it once and for all to transfer the database, I'll use the first approach.
Regards, Chris
Thanks again !
Florian
- 2077529 tallies, 2205004 msec.
**Tree** 98.4% {2169724ms} DOLBDD>>populateCollectionsFromBDDInSqueak 96.3% {2123419ms} DOLBDD>>commitAndBegin 96.3% {2123419ms} MagmaSession>>commitAndBegin 96.3% {2123419ms} MagmaSession>>commitAndBegin: 71.2% {1569963ms} MagmaSession>>newCommitPackageFor: |64.4% {1420023ms} MaTransaction>>changedObjects | |64.4% {1420023ms} MaTransaction>>addChangesFromReadSet | | 64.2% {1415613ms} MaTransaction>>didChange:from: | | 42.0% {926102ms} MaVariableObjectBuffer(MaVariableBuffer)>>isDifferent:using: | | |38.5% {848927ms} Array>>maIsChangedFrom:using: | | | |38.0% {837902ms} Array(Object)>>maIsChangedFrom:using: | | | | 21.9% {482896ms} MaVariableObjectBuffer(MaObjectBuffer)>>maInstVarAt: | | | | |20.4% {449821ms} MaVariableObjectBuffer(MaObjectBuffer)>>uint:at: | | | | | 20.2% {445411ms} ByteArray>>maUint:at: | | | | | 20.0% {441001ms} ByteArray>>maUnsigned48At: | | | | | 11.1% {244755ms} LargePositiveInteger>>+ | | | | | |8.4% {185220ms} LargePositiveInteger(Integer)>>+ | | | | | | |8.3% {183015ms} primitives | | | | | |2.7% {59535ms} primitives | | | | | 8.7% {191835ms} SmallInteger>>bitShift: | | | | | 8.1% {178605ms} SmallInteger(Integer)>>bitShift: | | | | | 7.9% {174195ms} primitives | | | | 13.6% {299881ms} MaObjectSerializer>>oidFor: | | | | 13.2% {291061ms} MagmaOidManager(MaOidManager)>>oidFor: | | | | 13.1% {288856ms} MagmaOidManager>>oidFor:ifAbsent: | | | | 9.9% {218295ms} MagmaOidManager(MaOidManager)>>oidFor:ifAbsent: | | | | |8.4% {185220ms} SmallInteger(Integer)>>maOid | | | | | 8.3% {183015ms} MaOidCalculator class>>oidForInteger: | | | | | 8.0% {176400ms} SmallInteger>>+ | | | | | 7.8% {171990ms} SmallInteger(Integer)>>+ | | | | | 5.9% {130095ms} primitives | | | | 2.6% {57330ms} primitives | | |3.2% {70560ms} Dictionary>>maIsChangedFrom:using: | | | 2.1% {46305ms} MaObjectSerializer>>oidFor: | | | 2.0% {44100ms} MagmaOidManager(MaOidManager)>>oidFor: | | | 2.0% {44100ms} MagmaOidManager>>oidFor:ifAbsent: | | 19.9% {438796ms} MaFixedObjectBuffer>>isDifferent:using: | | |9.7% {213885ms} MaObjectSerializer>>oidFor: | | | |9.6% {211680ms} MagmaOidManager(MaOidManager)>>oidFor: | | | | 9.6% {211680ms} MagmaOidManager>>oidFor:ifAbsent: | | | | 6.4% {141120ms} MagmaOidManager(MaOidManager)>>oidFor:ifAbsent: | | | | |2.7% {59535ms} WeakIdentityKeyDictionary(Dictionary)>>maAt:ifPresent:ifAbsent: | | | | | |2.3% {50715ms} WeakIdentityKeyDictionary(Dictionary)>>at:ifAbsent: | | | | |2.3% {50715ms} SmallInteger(Integer)>>maOid | | | | | 2.2% {48510ms} MaOidCalculator class>>oidForInteger: | | | | 2.2% {48510ms} WeakIdentityKeyDictionary(Dictionary)>>maAt:ifPresent:ifAbsent: | | |6.5% {143325ms} MaFixedObjectBuffer(MaObjectBuffer)>>maInstVarAt: | | | 6.0% {132300ms} MaFixedObjectBuffer(MaObjectBuffer)>>uint:at: | | | 5.9% {130095ms} ByteArray>>maUint:at: | | | 5.9% {130095ms} ByteArray>>maUnsigned48At: | | | 3.3% {72765ms} LargePositiveInteger>>+ | | | |2.5% {55125ms} LargePositiveInteger(Integer)>>+ | | | | 2.4% {52920ms} primitives | | | 2.5% {55125ms} SmallInteger>>bitShift: | | | 2.3% {50715ms} SmallInteger(Integer)>>bitShift: | | | 2.2% {48510ms} primitives | | 2.2% {48510ms} MaTransaction>>useWriteBarrierOn: |4.5% {99225ms} MaCommitPackage>>serializeObjectsUsing: | |4.4% {97020ms} MaObjectSerializer>>serializeGraph:do: | | 4.4% {97020ms} MaObjectSerializer>>appendGraph:do: | | 3.8% {83790ms} MaObjectSerializer>>append: | | 3.7% {81585ms} MaObjectSerializer>>bufferFor:storageObject:startingAt: |2.4% {52920ms} MagmaSession>>newCommitPackageFor: 17.5% {385876ms} MagmaSession>>refreshViewUsing: |17.5% {385876ms} MaCommitResult>>refresh: | 16.0% {352801ms} MagmaSession>>assignPermanentOidsFrom: | 13.9% {306496ms} MaObjectSerializer>>oidOf:is: | |13.9% {306496ms} MagmaOidManager>>oidOf:is: | | 13.9% {306496ms} MagmaOidManager(MaOidManager)>>oidOf:is: | | 13.4% {295471ms} MaWeakValueDictionary(MaDictionary)>>at:put: | | 12.7% {280036ms} WeakValueDictionary(Dictionary)>>includesKey: | | 12.6% {277831ms} WeakValueDictionary(Dictionary)>>at:ifAbsent: | | 12.2% {269010ms} WeakValueDictionary(Set)>>findElementOrNil: | | 11.8% {260190ms} WeakValueDictionary(Dictionary)>>scanFor: | 2.0% {44100ms} MaObjectSerializer>>objectWithOid:ifFound:ifAbsent: | 2.0% {44100ms} MagmaOidManager>>objectWithOid:ifFound:ifAbsent: 7.6% {167580ms} MagmaSession>>submit: 7.6% {167580ms} MaLocalServerLink>>submit: 7.6% {167580ms} MaLocalRequestServer(MaRequestServer)>>processRequest: 7.6% {167580ms} MagmaRepositoryController>>value: 7.6% {167580ms} MagmaRepositoryController>>processRequest: 7.6% {167580ms} MaWriteRequest>>process 7.6% {167580ms} MaObjectRepository>>submitAll:for:beginAnother: 7.5% {165375ms} MaObjectRepository>>write: 4.5% {99225ms} MaRecoveryManager>>log:flush: 4.0% {88200ms} MaObjectSerializer>>serializeGraph: 4.0% {88200ms} MaObjectSerializer>>serializeGraph:do: 4.0% {88200ms} MaObjectSerializer>>appendGraph:do: 3.1% {68355ms} MaObjectSerializer>>append: 3.0% {66150ms} MaObjectSerializer>>bufferFor:storageObject:startingAt: 2.2% {48510ms} MaVariableObjectBuffer>>populateBodyFor:using: 2.2% {48510ms} Dictionary>>maStreamVariablyInto:for: 2.0% {44100ms} MaObjectSerializer>>oidFor: 2.0% {44100ms} MaOidManager>>oidFor:
**Leaves** 19.3% {425566ms} SmallInteger(Integer)>>+ 18.1% {399106ms} Dictionary>>scanFor: 10.5% {231525ms} SmallInteger(Integer)>>bitShift: 3.9% {85995ms} MagmaOidManager>>oidFor:ifAbsent: 3.7% {81585ms} LargePositiveInteger>>+ 3.5% {77175ms} Dictionary>>at:ifAbsent: 3.3% {72765ms} SmallInteger(Number)>>negative 2.4% {52920ms} WeakKeyAssociation>>key
**Memory** old +44,200,708 bytes young +2,281,748 bytes used +46,482,456 bytes free -2,281,036 bytes
**GCs** full 66 totalling 38,913ms (2.0% uptime), avg 590.0ms incr 144172 totalling 1,192,826ms (54.0% uptime), avg 8.0ms tenures 1,426 (avg 101 GCs/tenure) root table 0 overflows
Magma mailing list Magma@lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma