Object identity
Chris Muller
asqueaker at gmail.com
Fri Jun 15 03:48:02 UTC 2007
Hi again. Well, as usual the MessageTally is very revealing. It
should have hit me before, but I finally realize now you have all of
what you want to commit to the database in memory (right?) and that is
why tremendous amounts of time are spent to determine which objects
changed. As more and more objects become persisted in the database,
Magma must scan the ever-larger Dictionary of objects on subsequent
commits to see if they changed.
WriteBarrier can virtually eliminate this. It detects when instance
variables change and marks those objects dirty as the changes happen,
allowing the readSet to remain very small.
But really, databases are used to have object models larger than
memory, and Magma wants to be a database. It is designed for
OLTP-style programs that work with a relatively small chunk of the
model at a time, throw it away and retrieve work with another chunk.
i.e., it should perform well working with a "screen at a time".
Having a large portion of the model in memory will cause
changed-detection to slow down significantly (unless WriteBarrier is
used), as well as slowing down refreshes of the model (changes by
other users) when transaction boundaries are crossed.
For your problem, though, you have a large inter-connected object
graph in memory that you want to get into the database. You can't do
it piecemeal because everything is linked together, right? Therefore,
the fastest way to do that is with a single commit. Did you try
changing that method (MaObjectBuffer>>#ensureSpaceFor:) to allow
50-meg and letting it run all night?
Regards,
Chris
On 6/14/07, Florian Minjat <florian.minjat at emn.fr> wrote:
> Hi again Chris,
>
> Chris Muller wrote:
> > Hi Florian,
> >
> >> Yesterday I tried again to save all the Letters in a dictionary of
> >> Players, add all the Players in Magma, then put back the Letters
> >> inside the Players. But the last part was so slow I had to stop the
> >> whole process after about one hour. I don't understand this slowliness
> >> because thoses Letters are only some references to Players which
> >> should already be inside Magma.
> >>
> >> The resulting code gives something like this (a mailbox is a
> >> collection of Letters):
> >>
> >> mailboxes := Dictionary new.
> >> players do: [:p | mailboxes at: (p login) put: (p mailbox). p mailbox:
> >> nil.].
> >> session begin.
> >> players do: [:p | session root players add: p. session commitAndBegin].
> >> players do: [:p | |magmaPlayer| magmaPlayer := session players where:
> >> [:r | r login equals: (p login)]. magmaPlayer mailbox: (mailboxes at:
> >> (p login)). session commitAndBegin]
> >
> > Ok, thanks for the code, it looks real good. Except one thing that
> > should be done to speed it up is add more than one player per commit.
> > Add 100 or 1000 per commit. That should speed up the first part a
> > lot.
> >
> > Same comment for the second part. Instead of commitAndBegin after
> > each one, do it every 100 or 1000. You could also choose to commit
> > "every five seconds". I know it makes the code less clean, sorry, but
> > this is one-time "bulk load" code anyway.
> >
> > You should also send #finalizeOids to the session after each commit.
> >
> > If it is still slow after doing all that, if you post a MessageTally
> > spy it will help a lot.
>
> I fact there are only 59 Players ^^. And the whole dump takes more
> than one hour so I don't think it would speed up so much.
> I put 5 players per commit to test.
>
> Here is the new code (sorry for the length) :
>
> mailboxes := Dictionary new.
> players do: [:p | mailboxes at: (p login) put: (p mailbox). p mailbox:
> nil.].
> commitCount := 0.
> players do: [ :p | session root players add: p.
> (commitCount = 5)
> ifTrue: [ self commitAndBegin. self magmaSession finalizeOids.
> commitCount := 0]
> ifFalse: [commitCount := commitCount +1].
> ].
>
> commitCount := 0.
> players do: [ :p | |magmaPlayer| magmaPlayer := (session players
> where: [:r | r login equals: (p login)]) first.
> magmaPlayer mailbox: (mailboxes at: (p login)).
> (commitCount = 5)
> ifTrue: [ self commitAndBegin. self magmaSession finalizeOids.
> commitCount := 0]
> ifFalse: [commitCount := commitCount +1].
> ].
>
>
> You can find the MessageTally at the end. It is much quicker (36min)
> with the finalizeOids and five adds by commit.
>
>
> >> Perhaps the players references by the Letters in the last block are
> >> detected in Magma as not yet inserted and it does it again ?
> >
> > No, the object is available in the collection as soon as it returns
> > from the commit.
>
> I tried to change my method to replace the Players by their logins
> before the first save, then iterate through the Letters to put back
> the references instead of the login. I am sure the references of the
> Letters comes from persistent players. But it was slower because there
> are much more magma queries. With your answer it is useless so I came
> back to the first version I showed you.
>
> >> Do you have an example of use of a transient variable ? I way I
> >> understand it is that I make the Letters transients so they are not
> >> serialized inside magma, then I remove the transient property to load
> >> the Letters in a second pass ?
> >
> > Yes, the idea is to implement #maTransientVariables on the referencing
> > class:
> >
> > maTransientVariables
> > ^ super maTransientVariables, #('letters')
> >
> > My idea was to add a new variable temporarily, 'letterIds' or
> > whatever. Populate this with some uniquely-identifying string or
> > number for the letters. Commit it.
> >
> > THEN, remove the #maTransientVariables method and enumerate them
> > again, initializing each from the letterIds. Finally, remove the
> > letterIds variable.
> >
> > This is a cumbersome approach, hopefully the first one will work.
>
> This method is a little more complex than the first one. As I need
> only to do it once and for all to transfer the database, I'll use the
> first approach.
>
> > Regards,
> > Chris
>
> Thanks again !
>
> Florian
>
>
> ------------------------------------------------------------
> - 2077529 tallies, 2205004 msec.
>
> **Tree**
> 98.4% {2169724ms} DOLBDD>>populateCollectionsFromBDDInSqueak
> 96.3% {2123419ms} DOLBDD>>commitAndBegin
> 96.3% {2123419ms} MagmaSession>>commitAndBegin
> 96.3% {2123419ms} MagmaSession>>commitAndBegin:
> 71.2% {1569963ms} MagmaSession>>newCommitPackageFor:
> |64.4% {1420023ms} MaTransaction>>changedObjects
> | |64.4% {1420023ms} MaTransaction>>addChangesFromReadSet
> | | 64.2% {1415613ms} MaTransaction>>didChange:from:
> | | 42.0% {926102ms}
> MaVariableObjectBuffer(MaVariableBuffer)>>isDifferent:using:
> | | |38.5% {848927ms} Array>>maIsChangedFrom:using:
> | | | |38.0% {837902ms}
> Array(Object)>>maIsChangedFrom:using:
> | | | | 21.9% {482896ms}
> MaVariableObjectBuffer(MaObjectBuffer)>>maInstVarAt:
> | | | | |20.4% {449821ms}
> MaVariableObjectBuffer(MaObjectBuffer)>>uint:at:
> | | | | | 20.2% {445411ms} ByteArray>>maUint:at:
> | | | | | 20.0% {441001ms}
> ByteArray>>maUnsigned48At:
> | | | | | 11.1% {244755ms}
> LargePositiveInteger>>+
> | | | | | |8.4% {185220ms}
> LargePositiveInteger(Integer)>>+
> | | | | | | |8.3% {183015ms} primitives
> | | | | | |2.7% {59535ms} primitives
> | | | | | 8.7% {191835ms}
> SmallInteger>>bitShift:
> | | | | | 8.1% {178605ms}
> SmallInteger(Integer)>>bitShift:
> | | | | | 7.9% {174195ms} primitives
> | | | | 13.6% {299881ms} MaObjectSerializer>>oidFor:
> | | | | 13.2% {291061ms}
> MagmaOidManager(MaOidManager)>>oidFor:
> | | | | 13.1% {288856ms}
> MagmaOidManager>>oidFor:ifAbsent:
> | | | | 9.9% {218295ms}
> MagmaOidManager(MaOidManager)>>oidFor:ifAbsent:
> | | | | |8.4% {185220ms}
> SmallInteger(Integer)>>maOid
> | | | | | 8.3% {183015ms} MaOidCalculator
> class>>oidForInteger:
> | | | | | 8.0% {176400ms} SmallInteger>>+
> | | | | | 7.8% {171990ms}
> SmallInteger(Integer)>>+
> | | | | | 5.9% {130095ms} primitives
> | | | | 2.6% {57330ms} primitives
> | | |3.2% {70560ms} Dictionary>>maIsChangedFrom:using:
> | | | 2.1% {46305ms} MaObjectSerializer>>oidFor:
> | | | 2.0% {44100ms}
> MagmaOidManager(MaOidManager)>>oidFor:
> | | | 2.0% {44100ms}
> MagmaOidManager>>oidFor:ifAbsent:
> | | 19.9% {438796ms}
> MaFixedObjectBuffer>>isDifferent:using:
> | | |9.7% {213885ms} MaObjectSerializer>>oidFor:
> | | | |9.6% {211680ms}
> MagmaOidManager(MaOidManager)>>oidFor:
> | | | | 9.6% {211680ms}
> MagmaOidManager>>oidFor:ifAbsent:
> | | | | 6.4% {141120ms}
> MagmaOidManager(MaOidManager)>>oidFor:ifAbsent:
> | | | | |2.7% {59535ms}
> WeakIdentityKeyDictionary(Dictionary)>>maAt:ifPresent:ifAbsent:
> | | | | | |2.3% {50715ms}
> WeakIdentityKeyDictionary(Dictionary)>>at:ifAbsent:
> | | | | |2.3% {50715ms}
> SmallInteger(Integer)>>maOid
> | | | | | 2.2% {48510ms} MaOidCalculator
> class>>oidForInteger:
> | | | | 2.2% {48510ms}
> WeakIdentityKeyDictionary(Dictionary)>>maAt:ifPresent:ifAbsent:
> | | |6.5% {143325ms}
> MaFixedObjectBuffer(MaObjectBuffer)>>maInstVarAt:
> | | | 6.0% {132300ms}
> MaFixedObjectBuffer(MaObjectBuffer)>>uint:at:
> | | | 5.9% {130095ms} ByteArray>>maUint:at:
> | | | 5.9% {130095ms} ByteArray>>maUnsigned48At:
> | | | 3.3% {72765ms} LargePositiveInteger>>+
> | | | |2.5% {55125ms}
> LargePositiveInteger(Integer)>>+
> | | | | 2.4% {52920ms} primitives
> | | | 2.5% {55125ms} SmallInteger>>bitShift:
> | | | 2.3% {50715ms}
> SmallInteger(Integer)>>bitShift:
> | | | 2.2% {48510ms} primitives
> | | 2.2% {48510ms} MaTransaction>>useWriteBarrierOn:
> |4.5% {99225ms} MaCommitPackage>>serializeObjectsUsing:
> | |4.4% {97020ms} MaObjectSerializer>>serializeGraph:do:
> | | 4.4% {97020ms} MaObjectSerializer>>appendGraph:do:
> | | 3.8% {83790ms} MaObjectSerializer>>append:
> | | 3.7% {81585ms}
> MaObjectSerializer>>bufferFor:storageObject:startingAt:
> |2.4% {52920ms} MagmaSession>>newCommitPackageFor:
> 17.5% {385876ms} MagmaSession>>refreshViewUsing:
> |17.5% {385876ms} MaCommitResult>>refresh:
> | 16.0% {352801ms} MagmaSession>>assignPermanentOidsFrom:
> | 13.9% {306496ms} MaObjectSerializer>>oidOf:is:
> | |13.9% {306496ms} MagmaOidManager>>oidOf:is:
> | | 13.9% {306496ms}
> MagmaOidManager(MaOidManager)>>oidOf:is:
> | | 13.4% {295471ms}
> MaWeakValueDictionary(MaDictionary)>>at:put:
> | | 12.7% {280036ms}
> WeakValueDictionary(Dictionary)>>includesKey:
> | | 12.6% {277831ms}
> WeakValueDictionary(Dictionary)>>at:ifAbsent:
> | | 12.2% {269010ms}
> WeakValueDictionary(Set)>>findElementOrNil:
> | | 11.8% {260190ms}
> WeakValueDictionary(Dictionary)>>scanFor:
> | 2.0% {44100ms}
> MaObjectSerializer>>objectWithOid:ifFound:ifAbsent:
> | 2.0% {44100ms}
> MagmaOidManager>>objectWithOid:ifFound:ifAbsent:
> 7.6% {167580ms} MagmaSession>>submit:
> 7.6% {167580ms} MaLocalServerLink>>submit:
> 7.6% {167580ms}
> MaLocalRequestServer(MaRequestServer)>>processRequest:
> 7.6% {167580ms} MagmaRepositoryController>>value:
> 7.6% {167580ms}
> MagmaRepositoryController>>processRequest:
> 7.6% {167580ms} MaWriteRequest>>process
> 7.6% {167580ms}
> MaObjectRepository>>submitAll:for:beginAnother:
> 7.5% {165375ms} MaObjectRepository>>write:
> 4.5% {99225ms} MaRecoveryManager>>log:flush:
> 4.0% {88200ms}
> MaObjectSerializer>>serializeGraph:
> 4.0% {88200ms}
> MaObjectSerializer>>serializeGraph:do:
> 4.0% {88200ms}
> MaObjectSerializer>>appendGraph:do:
> 3.1% {68355ms}
> MaObjectSerializer>>append:
> 3.0% {66150ms}
> MaObjectSerializer>>bufferFor:storageObject:startingAt:
> 2.2% {48510ms}
> MaVariableObjectBuffer>>populateBodyFor:using:
> 2.2% {48510ms}
> Dictionary>>maStreamVariablyInto:for:
> 2.0% {44100ms}
> MaObjectSerializer>>oidFor:
> 2.0% {44100ms}
> MaOidManager>>oidFor:
>
> **Leaves**
> 19.3% {425566ms} SmallInteger(Integer)>>+
> 18.1% {399106ms} Dictionary>>scanFor:
> 10.5% {231525ms} SmallInteger(Integer)>>bitShift:
> 3.9% {85995ms} MagmaOidManager>>oidFor:ifAbsent:
> 3.7% {81585ms} LargePositiveInteger>>+
> 3.5% {77175ms} Dictionary>>at:ifAbsent:
> 3.3% {72765ms} SmallInteger(Number)>>negative
> 2.4% {52920ms} WeakKeyAssociation>>key
>
> **Memory**
> old +44,200,708 bytes
> young +2,281,748 bytes
> used +46,482,456 bytes
> free -2,281,036 bytes
>
> **GCs**
> full 66 totalling 38,913ms (2.0% uptime), avg 590.0ms
> incr 144172 totalling 1,192,826ms (54.0% uptime), avg 8.0ms
> tenures 1,426 (avg 101 GCs/tenure)
> root table 0 overflows
>
>
> _______________________________________________
> Magma mailing list
> Magma at lists.squeakfoundation.org
> http://lists.squeakfoundation.org/mailman/listinfo/magma
>
More information about the Magma
mailing list