Hi again Chris,
Chris Muller wrote:
Hi Florian,
Yesterday I tried again to save all the Letters in a dictionary of Players, add all the Players in Magma, then put back the Letters inside the Players. But the last part was so slow I had to stop the whole process after about one hour. I don't understand this slowliness because thoses Letters are only some references to Players which should already be inside Magma.
The resulting code gives something like this (a mailbox is a collection of Letters):
mailboxes := Dictionary new. players do: [:p | mailboxes at: (p login) put: (p mailbox). p mailbox: nil.]. session begin. players do: [:p | session root players add: p. session commitAndBegin]. players do: [:p | |magmaPlayer| magmaPlayer := session players where: [:r | r login equals: (p login)]. magmaPlayer mailbox: (mailboxes at: (p login)). session commitAndBegin]
Ok, thanks for the code, it looks real good. Except one thing that should be done to speed it up is add more than one player per commit. Add 100 or 1000 per commit. That should speed up the first part a lot.
Same comment for the second part. Instead of commitAndBegin after each one, do it every 100 or 1000. You could also choose to commit "every five seconds". I know it makes the code less clean, sorry, but this is one-time "bulk load" code anyway.
You should also send #finalizeOids to the session after each commit.
If it is still slow after doing all that, if you post a MessageTally spy it will help a lot.
I fact there are only 59 Players ^^. And the whole dump takes more than one hour so I don't think it would speed up so much. I put 5 players per commit to test.
Here is the new code (sorry for the length) :
mailboxes := Dictionary new. players do: [:p | mailboxes at: (p login) put: (p mailbox). p mailbox: nil.]. commitCount := 0. players do: [ :p | session root players add: p. (commitCount = 5) ifTrue: [ self commitAndBegin. self magmaSession finalizeOids. commitCount := 0] ifFalse: [commitCount := commitCount +1]. ].
commitCount := 0. players do: [ :p | |magmaPlayer| magmaPlayer := (session players where: [:r | r login equals: (p login)]) first. magmaPlayer mailbox: (mailboxes at: (p login)). (commitCount = 5) ifTrue: [ self commitAndBegin. self magmaSession finalizeOids. commitCount := 0] ifFalse: [commitCount := commitCount +1]. ].
You can find the MessageTally at the end. It is much quicker (36min) with the finalizeOids and five adds by commit.
Perhaps the players references by the Letters in the last block are detected in Magma as not yet inserted and it does it again ?
No, the object is available in the collection as soon as it returns from the commit.
I tried to change my method to replace the Players by their logins before the first save, then iterate through the Letters to put back the references instead of the login. I am sure the references of the Letters comes from persistent players. But it was slower because there are much more magma queries. With your answer it is useless so I came back to the first version I showed you.
Do you have an example of use of a transient variable ? I way I understand it is that I make the Letters transients so they are not serialized inside magma, then I remove the transient property to load the Letters in a second pass ?
Yes, the idea is to implement #maTransientVariables on the referencing class:
maTransientVariables ^ super maTransientVariables, #('letters')
My idea was to add a new variable temporarily, 'letterIds' or whatever. Populate this with some uniquely-identifying string or number for the letters. Commit it.
THEN, remove the #maTransientVariables method and enumerate them again, initializing each from the letterIds. Finally, remove the letterIds variable.
This is a cumbersome approach, hopefully the first one will work.
This method is a little more complex than the first one. As I need only to do it once and for all to transfer the database, I'll use the first approach.
Regards, Chris
Thanks again !
Florian
------------------------------------------------------------ - 2077529 tallies, 2205004 msec.
**Tree** 98.4% {2169724ms} DOLBDD>>populateCollectionsFromBDDInSqueak 96.3% {2123419ms} DOLBDD>>commitAndBegin 96.3% {2123419ms} MagmaSession>>commitAndBegin 96.3% {2123419ms} MagmaSession>>commitAndBegin: 71.2% {1569963ms} MagmaSession>>newCommitPackageFor: |64.4% {1420023ms} MaTransaction>>changedObjects | |64.4% {1420023ms} MaTransaction>>addChangesFromReadSet | | 64.2% {1415613ms} MaTransaction>>didChange:from: | | 42.0% {926102ms} MaVariableObjectBuffer(MaVariableBuffer)>>isDifferent:using: | | |38.5% {848927ms} Array>>maIsChangedFrom:using: | | | |38.0% {837902ms} Array(Object)>>maIsChangedFrom:using: | | | | 21.9% {482896ms} MaVariableObjectBuffer(MaObjectBuffer)>>maInstVarAt: | | | | |20.4% {449821ms} MaVariableObjectBuffer(MaObjectBuffer)>>uint:at: | | | | | 20.2% {445411ms} ByteArray>>maUint:at: | | | | | 20.0% {441001ms} ByteArray>>maUnsigned48At: | | | | | 11.1% {244755ms} LargePositiveInteger>>+ | | | | | |8.4% {185220ms} LargePositiveInteger(Integer)>>+ | | | | | | |8.3% {183015ms} primitives | | | | | |2.7% {59535ms} primitives | | | | | 8.7% {191835ms} SmallInteger>>bitShift: | | | | | 8.1% {178605ms} SmallInteger(Integer)>>bitShift: | | | | | 7.9% {174195ms} primitives | | | | 13.6% {299881ms} MaObjectSerializer>>oidFor: | | | | 13.2% {291061ms} MagmaOidManager(MaOidManager)>>oidFor: | | | | 13.1% {288856ms} MagmaOidManager>>oidFor:ifAbsent: | | | | 9.9% {218295ms} MagmaOidManager(MaOidManager)>>oidFor:ifAbsent: | | | | |8.4% {185220ms} SmallInteger(Integer)>>maOid | | | | | 8.3% {183015ms} MaOidCalculator class>>oidForInteger: | | | | | 8.0% {176400ms} SmallInteger>>+ | | | | | 7.8% {171990ms} SmallInteger(Integer)>>+ | | | | | 5.9% {130095ms} primitives | | | | 2.6% {57330ms} primitives | | |3.2% {70560ms} Dictionary>>maIsChangedFrom:using: | | | 2.1% {46305ms} MaObjectSerializer>>oidFor: | | | 2.0% {44100ms} MagmaOidManager(MaOidManager)>>oidFor: | | | 2.0% {44100ms} MagmaOidManager>>oidFor:ifAbsent: | | 19.9% {438796ms} MaFixedObjectBuffer>>isDifferent:using: | | |9.7% {213885ms} MaObjectSerializer>>oidFor: | | | |9.6% {211680ms} MagmaOidManager(MaOidManager)>>oidFor: | | | | 9.6% {211680ms} MagmaOidManager>>oidFor:ifAbsent: | | | | 6.4% {141120ms} MagmaOidManager(MaOidManager)>>oidFor:ifAbsent: | | | | |2.7% {59535ms} WeakIdentityKeyDictionary(Dictionary)>>maAt:ifPresent:ifAbsent: | | | | | |2.3% {50715ms} WeakIdentityKeyDictionary(Dictionary)>>at:ifAbsent: | | | | |2.3% {50715ms} SmallInteger(Integer)>>maOid | | | | | 2.2% {48510ms} MaOidCalculator class>>oidForInteger: | | | | 2.2% {48510ms} WeakIdentityKeyDictionary(Dictionary)>>maAt:ifPresent:ifAbsent: | | |6.5% {143325ms} MaFixedObjectBuffer(MaObjectBuffer)>>maInstVarAt: | | | 6.0% {132300ms} MaFixedObjectBuffer(MaObjectBuffer)>>uint:at: | | | 5.9% {130095ms} ByteArray>>maUint:at: | | | 5.9% {130095ms} ByteArray>>maUnsigned48At: | | | 3.3% {72765ms} LargePositiveInteger>>+ | | | |2.5% {55125ms} LargePositiveInteger(Integer)>>+ | | | | 2.4% {52920ms} primitives | | | 2.5% {55125ms} SmallInteger>>bitShift: | | | 2.3% {50715ms} SmallInteger(Integer)>>bitShift: | | | 2.2% {48510ms} primitives | | 2.2% {48510ms} MaTransaction>>useWriteBarrierOn: |4.5% {99225ms} MaCommitPackage>>serializeObjectsUsing: | |4.4% {97020ms} MaObjectSerializer>>serializeGraph:do: | | 4.4% {97020ms} MaObjectSerializer>>appendGraph:do: | | 3.8% {83790ms} MaObjectSerializer>>append: | | 3.7% {81585ms} MaObjectSerializer>>bufferFor:storageObject:startingAt: |2.4% {52920ms} MagmaSession>>newCommitPackageFor: 17.5% {385876ms} MagmaSession>>refreshViewUsing: |17.5% {385876ms} MaCommitResult>>refresh: | 16.0% {352801ms} MagmaSession>>assignPermanentOidsFrom: | 13.9% {306496ms} MaObjectSerializer>>oidOf:is: | |13.9% {306496ms} MagmaOidManager>>oidOf:is: | | 13.9% {306496ms} MagmaOidManager(MaOidManager)>>oidOf:is: | | 13.4% {295471ms} MaWeakValueDictionary(MaDictionary)>>at:put: | | 12.7% {280036ms} WeakValueDictionary(Dictionary)>>includesKey: | | 12.6% {277831ms} WeakValueDictionary(Dictionary)>>at:ifAbsent: | | 12.2% {269010ms} WeakValueDictionary(Set)>>findElementOrNil: | | 11.8% {260190ms} WeakValueDictionary(Dictionary)>>scanFor: | 2.0% {44100ms} MaObjectSerializer>>objectWithOid:ifFound:ifAbsent: | 2.0% {44100ms} MagmaOidManager>>objectWithOid:ifFound:ifAbsent: 7.6% {167580ms} MagmaSession>>submit: 7.6% {167580ms} MaLocalServerLink>>submit: 7.6% {167580ms} MaLocalRequestServer(MaRequestServer)>>processRequest: 7.6% {167580ms} MagmaRepositoryController>>value: 7.6% {167580ms} MagmaRepositoryController>>processRequest: 7.6% {167580ms} MaWriteRequest>>process 7.6% {167580ms} MaObjectRepository>>submitAll:for:beginAnother: 7.5% {165375ms} MaObjectRepository>>write: 4.5% {99225ms} MaRecoveryManager>>log:flush: 4.0% {88200ms} MaObjectSerializer>>serializeGraph: 4.0% {88200ms} MaObjectSerializer>>serializeGraph:do: 4.0% {88200ms} MaObjectSerializer>>appendGraph:do: 3.1% {68355ms} MaObjectSerializer>>append: 3.0% {66150ms} MaObjectSerializer>>bufferFor:storageObject:startingAt: 2.2% {48510ms} MaVariableObjectBuffer>>populateBodyFor:using: 2.2% {48510ms} Dictionary>>maStreamVariablyInto:for: 2.0% {44100ms} MaObjectSerializer>>oidFor: 2.0% {44100ms} MaOidManager>>oidFor: **Leaves** 19.3% {425566ms} SmallInteger(Integer)>>+ 18.1% {399106ms} Dictionary>>scanFor: 10.5% {231525ms} SmallInteger(Integer)>>bitShift: 3.9% {85995ms} MagmaOidManager>>oidFor:ifAbsent: 3.7% {81585ms} LargePositiveInteger>>+ 3.5% {77175ms} Dictionary>>at:ifAbsent: 3.3% {72765ms} SmallInteger(Number)>>negative 2.4% {52920ms} WeakKeyAssociation>>key
**Memory** old +44,200,708 bytes young +2,281,748 bytes used +46,482,456 bytes free -2,281,036 bytes
**GCs** full 66 totalling 38,913ms (2.0% uptime), avg 590.0ms incr 144172 totalling 1,192,826ms (54.0% uptime), avg 8.0ms tenures 1,426 (avg 101 GCs/tenure) root table 0 overflows