Object identity
Florian Minjat
florian.minjat at emn.fr
Thu Jun 14 11:21:45 UTC 2007
Hi again Chris,
Chris Muller wrote:
> Hi Florian,
>
>> Yesterday I tried again to save all the Letters in a dictionary of
>> Players, add all the Players in Magma, then put back the Letters
>> inside the Players. But the last part was so slow I had to stop the
>> whole process after about one hour. I don't understand this slowliness
>> because thoses Letters are only some references to Players which
>> should already be inside Magma.
>>
>> The resulting code gives something like this (a mailbox is a
>> collection of Letters):
>>
>> mailboxes := Dictionary new.
>> players do: [:p | mailboxes at: (p login) put: (p mailbox). p mailbox:
>> nil.].
>> session begin.
>> players do: [:p | session root players add: p. session commitAndBegin].
>> players do: [:p | |magmaPlayer| magmaPlayer := session players where:
>> [:r | r login equals: (p login)]. magmaPlayer mailbox: (mailboxes at:
>> (p login)). session commitAndBegin]
>
> Ok, thanks for the code, it looks real good. Except one thing that
> should be done to speed it up is add more than one player per commit.
> Add 100 or 1000 per commit. That should speed up the first part a
> lot.
>
> Same comment for the second part. Instead of commitAndBegin after
> each one, do it every 100 or 1000. You could also choose to commit
> "every five seconds". I know it makes the code less clean, sorry, but
> this is one-time "bulk load" code anyway.
>
> You should also send #finalizeOids to the session after each commit.
>
> If it is still slow after doing all that, if you post a MessageTally
> spy it will help a lot.
I fact there are only 59 Players ^^. And the whole dump takes more
than one hour so I don't think it would speed up so much.
I put 5 players per commit to test.
Here is the new code (sorry for the length) :
mailboxes := Dictionary new.
players do: [:p | mailboxes at: (p login) put: (p mailbox). p mailbox:
nil.].
commitCount := 0.
players do: [ :p | session root players add: p.
(commitCount = 5)
ifTrue: [ self commitAndBegin. self magmaSession finalizeOids.
commitCount := 0]
ifFalse: [commitCount := commitCount +1].
].
commitCount := 0.
players do: [ :p | |magmaPlayer| magmaPlayer := (session players
where: [:r | r login equals: (p login)]) first.
magmaPlayer mailbox: (mailboxes at: (p login)).
(commitCount = 5)
ifTrue: [ self commitAndBegin. self magmaSession finalizeOids.
commitCount := 0]
ifFalse: [commitCount := commitCount +1].
].
You can find the MessageTally at the end. It is much quicker (36min)
with the finalizeOids and five adds by commit.
>> Perhaps the players references by the Letters in the last block are
>> detected in Magma as not yet inserted and it does it again ?
>
> No, the object is available in the collection as soon as it returns
> from the commit.
I tried to change my method to replace the Players by their logins
before the first save, then iterate through the Letters to put back
the references instead of the login. I am sure the references of the
Letters comes from persistent players. But it was slower because there
are much more magma queries. With your answer it is useless so I came
back to the first version I showed you.
>> Do you have an example of use of a transient variable ? I way I
>> understand it is that I make the Letters transients so they are not
>> serialized inside magma, then I remove the transient property to load
>> the Letters in a second pass ?
>
> Yes, the idea is to implement #maTransientVariables on the referencing
> class:
>
> maTransientVariables
> ^ super maTransientVariables, #('letters')
>
> My idea was to add a new variable temporarily, 'letterIds' or
> whatever. Populate this with some uniquely-identifying string or
> number for the letters. Commit it.
>
> THEN, remove the #maTransientVariables method and enumerate them
> again, initializing each from the letterIds. Finally, remove the
> letterIds variable.
>
> This is a cumbersome approach, hopefully the first one will work.
This method is a little more complex than the first one. As I need
only to do it once and for all to transfer the database, I'll use the
first approach.
> Regards,
> Chris
Thanks again !
Florian
------------------------------------------------------------
- 2077529 tallies, 2205004 msec.
**Tree**
98.4% {2169724ms} DOLBDD>>populateCollectionsFromBDDInSqueak
96.3% {2123419ms} DOLBDD>>commitAndBegin
96.3% {2123419ms} MagmaSession>>commitAndBegin
96.3% {2123419ms} MagmaSession>>commitAndBegin:
71.2% {1569963ms} MagmaSession>>newCommitPackageFor:
|64.4% {1420023ms} MaTransaction>>changedObjects
| |64.4% {1420023ms} MaTransaction>>addChangesFromReadSet
| | 64.2% {1415613ms} MaTransaction>>didChange:from:
| | 42.0% {926102ms}
MaVariableObjectBuffer(MaVariableBuffer)>>isDifferent:using:
| | |38.5% {848927ms} Array>>maIsChangedFrom:using:
| | | |38.0% {837902ms}
Array(Object)>>maIsChangedFrom:using:
| | | | 21.9% {482896ms}
MaVariableObjectBuffer(MaObjectBuffer)>>maInstVarAt:
| | | | |20.4% {449821ms}
MaVariableObjectBuffer(MaObjectBuffer)>>uint:at:
| | | | | 20.2% {445411ms} ByteArray>>maUint:at:
| | | | | 20.0% {441001ms}
ByteArray>>maUnsigned48At:
| | | | | 11.1% {244755ms}
LargePositiveInteger>>+
| | | | | |8.4% {185220ms}
LargePositiveInteger(Integer)>>+
| | | | | | |8.3% {183015ms} primitives
| | | | | |2.7% {59535ms} primitives
| | | | | 8.7% {191835ms}
SmallInteger>>bitShift:
| | | | | 8.1% {178605ms}
SmallInteger(Integer)>>bitShift:
| | | | | 7.9% {174195ms} primitives
| | | | 13.6% {299881ms} MaObjectSerializer>>oidFor:
| | | | 13.2% {291061ms}
MagmaOidManager(MaOidManager)>>oidFor:
| | | | 13.1% {288856ms}
MagmaOidManager>>oidFor:ifAbsent:
| | | | 9.9% {218295ms}
MagmaOidManager(MaOidManager)>>oidFor:ifAbsent:
| | | | |8.4% {185220ms}
SmallInteger(Integer)>>maOid
| | | | | 8.3% {183015ms} MaOidCalculator
class>>oidForInteger:
| | | | | 8.0% {176400ms} SmallInteger>>+
| | | | | 7.8% {171990ms}
SmallInteger(Integer)>>+
| | | | | 5.9% {130095ms} primitives
| | | | 2.6% {57330ms} primitives
| | |3.2% {70560ms} Dictionary>>maIsChangedFrom:using:
| | | 2.1% {46305ms} MaObjectSerializer>>oidFor:
| | | 2.0% {44100ms}
MagmaOidManager(MaOidManager)>>oidFor:
| | | 2.0% {44100ms}
MagmaOidManager>>oidFor:ifAbsent:
| | 19.9% {438796ms}
MaFixedObjectBuffer>>isDifferent:using:
| | |9.7% {213885ms} MaObjectSerializer>>oidFor:
| | | |9.6% {211680ms}
MagmaOidManager(MaOidManager)>>oidFor:
| | | | 9.6% {211680ms}
MagmaOidManager>>oidFor:ifAbsent:
| | | | 6.4% {141120ms}
MagmaOidManager(MaOidManager)>>oidFor:ifAbsent:
| | | | |2.7% {59535ms}
WeakIdentityKeyDictionary(Dictionary)>>maAt:ifPresent:ifAbsent:
| | | | | |2.3% {50715ms}
WeakIdentityKeyDictionary(Dictionary)>>at:ifAbsent:
| | | | |2.3% {50715ms}
SmallInteger(Integer)>>maOid
| | | | | 2.2% {48510ms} MaOidCalculator
class>>oidForInteger:
| | | | 2.2% {48510ms}
WeakIdentityKeyDictionary(Dictionary)>>maAt:ifPresent:ifAbsent:
| | |6.5% {143325ms}
MaFixedObjectBuffer(MaObjectBuffer)>>maInstVarAt:
| | | 6.0% {132300ms}
MaFixedObjectBuffer(MaObjectBuffer)>>uint:at:
| | | 5.9% {130095ms} ByteArray>>maUint:at:
| | | 5.9% {130095ms} ByteArray>>maUnsigned48At:
| | | 3.3% {72765ms} LargePositiveInteger>>+
| | | |2.5% {55125ms}
LargePositiveInteger(Integer)>>+
| | | | 2.4% {52920ms} primitives
| | | 2.5% {55125ms} SmallInteger>>bitShift:
| | | 2.3% {50715ms}
SmallInteger(Integer)>>bitShift:
| | | 2.2% {48510ms} primitives
| | 2.2% {48510ms} MaTransaction>>useWriteBarrierOn:
|4.5% {99225ms} MaCommitPackage>>serializeObjectsUsing:
| |4.4% {97020ms} MaObjectSerializer>>serializeGraph:do:
| | 4.4% {97020ms} MaObjectSerializer>>appendGraph:do:
| | 3.8% {83790ms} MaObjectSerializer>>append:
| | 3.7% {81585ms}
MaObjectSerializer>>bufferFor:storageObject:startingAt:
|2.4% {52920ms} MagmaSession>>newCommitPackageFor:
17.5% {385876ms} MagmaSession>>refreshViewUsing:
|17.5% {385876ms} MaCommitResult>>refresh:
| 16.0% {352801ms} MagmaSession>>assignPermanentOidsFrom:
| 13.9% {306496ms} MaObjectSerializer>>oidOf:is:
| |13.9% {306496ms} MagmaOidManager>>oidOf:is:
| | 13.9% {306496ms}
MagmaOidManager(MaOidManager)>>oidOf:is:
| | 13.4% {295471ms}
MaWeakValueDictionary(MaDictionary)>>at:put:
| | 12.7% {280036ms}
WeakValueDictionary(Dictionary)>>includesKey:
| | 12.6% {277831ms}
WeakValueDictionary(Dictionary)>>at:ifAbsent:
| | 12.2% {269010ms}
WeakValueDictionary(Set)>>findElementOrNil:
| | 11.8% {260190ms}
WeakValueDictionary(Dictionary)>>scanFor:
| 2.0% {44100ms}
MaObjectSerializer>>objectWithOid:ifFound:ifAbsent:
| 2.0% {44100ms}
MagmaOidManager>>objectWithOid:ifFound:ifAbsent:
7.6% {167580ms} MagmaSession>>submit:
7.6% {167580ms} MaLocalServerLink>>submit:
7.6% {167580ms}
MaLocalRequestServer(MaRequestServer)>>processRequest:
7.6% {167580ms} MagmaRepositoryController>>value:
7.6% {167580ms}
MagmaRepositoryController>>processRequest:
7.6% {167580ms} MaWriteRequest>>process
7.6% {167580ms}
MaObjectRepository>>submitAll:for:beginAnother:
7.5% {165375ms} MaObjectRepository>>write:
4.5% {99225ms} MaRecoveryManager>>log:flush:
4.0% {88200ms}
MaObjectSerializer>>serializeGraph:
4.0% {88200ms}
MaObjectSerializer>>serializeGraph:do:
4.0% {88200ms}
MaObjectSerializer>>appendGraph:do:
3.1% {68355ms}
MaObjectSerializer>>append:
3.0% {66150ms}
MaObjectSerializer>>bufferFor:storageObject:startingAt:
2.2% {48510ms}
MaVariableObjectBuffer>>populateBodyFor:using:
2.2% {48510ms}
Dictionary>>maStreamVariablyInto:for:
2.0% {44100ms}
MaObjectSerializer>>oidFor:
2.0% {44100ms}
MaOidManager>>oidFor:
**Leaves**
19.3% {425566ms} SmallInteger(Integer)>>+
18.1% {399106ms} Dictionary>>scanFor:
10.5% {231525ms} SmallInteger(Integer)>>bitShift:
3.9% {85995ms} MagmaOidManager>>oidFor:ifAbsent:
3.7% {81585ms} LargePositiveInteger>>+
3.5% {77175ms} Dictionary>>at:ifAbsent:
3.3% {72765ms} SmallInteger(Number)>>negative
2.4% {52920ms} WeakKeyAssociation>>key
**Memory**
old +44,200,708 bytes
young +2,281,748 bytes
used +46,482,456 bytes
free -2,281,036 bytes
**GCs**
full 66 totalling 38,913ms (2.0% uptime), avg 590.0ms
incr 144172 totalling 1,192,826ms (54.0% uptime), avg 8.0ms
tenures 1,426 (avg 101 GCs/tenure)
root table 0 overflows
More information about the Magma
mailing list