Object identity
Florian Minjat
florian.minjat at emn.fr
Fri Jun 15 17:21:23 UTC 2007
Chris Muller wrote:
> Hi again. Well, as usual the MessageTally is very revealing. It
> should have hit me before, but I finally realize now you have all of
> what you want to commit to the database in memory (right?) and that is
> why tremendous amounts of time are spent to determine which objects
> changed. As more and more objects become persisted in the database,
> Magma must scan the ever-larger Dictionary of objects on subsequent
> commits to see if they changed.
Yes you are right. I used GOODS which was not reliable at that time.
Then I used a database inside the Squeak image and made saves of the
db with fileOuts. But it is not stable at all with the over 2Go memory
problem : the saves randomly hanged the image. So now I struggle to
use Magma which seems far much reliable but need more tuning :).
> WriteBarrier can virtually eliminate this. It detects when instance
> variables change and marks those objects dirty as the changes happen,
> allowing the readSet to remain very small.
>
> But really, databases are used to have object models larger than
> memory, and Magma wants to be a database. It is designed for
> OLTP-style programs that work with a relatively small chunk of the
> model at a time, throw it away and retrieve work with another chunk.
> i.e., it should perform well working with a "screen at a time".
> Having a large portion of the model in memory will cause
> changed-detection to slow down significantly (unless WriteBarrier is
> used), as well as slowing down refreshes of the model (changes by
> other users) when transaction boundaries are crossed.
>
> For your problem, though, you have a large inter-connected object
> graph in memory that you want to get into the database. You can't do
> it piecemeal because everything is linked together, right? Therefore,
> the fastest way to do that is with a single commit. Did you try
> changing that method (MaObjectBuffer>>#ensureSpaceFor:) to allow
> 50-meg and letting it run all night?
I just tried it. It took at least 40 min. But I am not sure because I
paused the process to use the computer and it disturbed MessageTally
which gave me an error at the end. I will do it again when I have some
more time.
But as it is almost the same length of time, I think I won't use the
system with removal/insertion of the Letters anymore.
One thing that I am worried about though : each 12h my app runs a
process to update the whole players, which changes a lot of things.
Will it takes so much time or it is just insertion which is very long ?
Florian
> Regards,
> Chris
>
>
>
>
>
> On 6/14/07, Florian Minjat <florian.minjat at emn.fr> wrote:
>> Hi again Chris,
>>
>> Chris Muller wrote:
>> > Hi Florian,
>> >
>> >> Yesterday I tried again to save all the Letters in a dictionary of
>> >> Players, add all the Players in Magma, then put back the Letters
>> >> inside the Players. But the last part was so slow I had to stop the
>> >> whole process after about one hour. I don't understand this slowliness
>> >> because thoses Letters are only some references to Players which
>> >> should already be inside Magma.
>> >>
>> >> The resulting code gives something like this (a mailbox is a
>> >> collection of Letters):
>> >>
>> >> mailboxes := Dictionary new.
>> >> players do: [:p | mailboxes at: (p login) put: (p mailbox). p mailbox:
>> >> nil.].
>> >> session begin.
>> >> players do: [:p | session root players add: p. session commitAndBegin].
>> >> players do: [:p | |magmaPlayer| magmaPlayer := session players where:
>> >> [:r | r login equals: (p login)]. magmaPlayer mailbox: (mailboxes at:
>> >> (p login)). session commitAndBegin]
>> >
>> > Ok, thanks for the code, it looks real good. Except one thing that
>> > should be done to speed it up is add more than one player per commit.
>> > Add 100 or 1000 per commit. That should speed up the first part a
>> > lot.
>> >
>> > Same comment for the second part. Instead of commitAndBegin after
>> > each one, do it every 100 or 1000. You could also choose to commit
>> > "every five seconds". I know it makes the code less clean, sorry, but
>> > this is one-time "bulk load" code anyway.
>> >
>> > You should also send #finalizeOids to the session after each commit.
>> >
>> > If it is still slow after doing all that, if you post a MessageTally
>> > spy it will help a lot.
>>
>> I fact there are only 59 Players ^^. And the whole dump takes more
>> than one hour so I don't think it would speed up so much.
>> I put 5 players per commit to test.
>>
>> Here is the new code (sorry for the length) :
>>
>> mailboxes := Dictionary new.
>> players do: [:p | mailboxes at: (p login) put: (p mailbox). p mailbox:
>> nil.].
>> commitCount := 0.
>> players do: [ :p | session root players add: p.
>> (commitCount = 5)
>> ifTrue: [ self commitAndBegin. self magmaSession finalizeOids.
>> commitCount := 0]
>> ifFalse: [commitCount := commitCount +1].
>> ].
>>
>> commitCount := 0.
>> players do: [ :p | |magmaPlayer| magmaPlayer := (session players
>> where: [:r | r login equals: (p login)]) first.
>> magmaPlayer mailbox: (mailboxes at: (p login)).
>> (commitCount = 5)
>> ifTrue: [ self commitAndBegin. self magmaSession finalizeOids.
>> commitCount := 0]
>> ifFalse: [commitCount := commitCount +1].
>> ].
>>
>>
>> You can find the MessageTally at the end. It is much quicker (36min)
>> with the finalizeOids and five adds by commit.
>>
>>
>> >> Perhaps the players references by the Letters in the last block are
>> >> detected in Magma as not yet inserted and it does it again ?
>> >
>> > No, the object is available in the collection as soon as it returns
>> > from the commit.
>>
>> I tried to change my method to replace the Players by their logins
>> before the first save, then iterate through the Letters to put back
>> the references instead of the login. I am sure the references of the
>> Letters comes from persistent players. But it was slower because there
>> are much more magma queries. With your answer it is useless so I came
>> back to the first version I showed you.
>>
>> >> Do you have an example of use of a transient variable ? I way I
>> >> understand it is that I make the Letters transients so they are not
>> >> serialized inside magma, then I remove the transient property to load
>> >> the Letters in a second pass ?
>> >
>> > Yes, the idea is to implement #maTransientVariables on the referencing
>> > class:
>> >
>> > maTransientVariables
>> > ^ super maTransientVariables, #('letters')
>> >
>> > My idea was to add a new variable temporarily, 'letterIds' or
>> > whatever. Populate this with some uniquely-identifying string or
>> > number for the letters. Commit it.
>> >
>> > THEN, remove the #maTransientVariables method and enumerate them
>> > again, initializing each from the letterIds. Finally, remove the
>> > letterIds variable.
>> >
>> > This is a cumbersome approach, hopefully the first one will work.
>>
>> This method is a little more complex than the first one. As I need
>> only to do it once and for all to transfer the database, I'll use the
>> first approach.
>>
>> > Regards,
>> > Chris
>>
>> Thanks again !
>>
>> Florian
>>
>>
>> ------------------------------------------------------------
>> - 2077529 tallies, 2205004 msec.
>>
>> **Tree**
>> 98.4% {2169724ms} DOLBDD>>populateCollectionsFromBDDInSqueak
>> 96.3% {2123419ms} DOLBDD>>commitAndBegin
>> 96.3% {2123419ms} MagmaSession>>commitAndBegin
>> 96.3% {2123419ms} MagmaSession>>commitAndBegin:
>> 71.2% {1569963ms} MagmaSession>>newCommitPackageFor:
>> |64.4% {1420023ms} MaTransaction>>changedObjects
>> | |64.4% {1420023ms} MaTransaction>>addChangesFromReadSet
>> | | 64.2% {1415613ms} MaTransaction>>didChange:from:
>> | | 42.0% {926102ms}
>> MaVariableObjectBuffer(MaVariableBuffer)>>isDifferent:using:
>> | | |38.5% {848927ms} Array>>maIsChangedFrom:using:
>> | | | |38.0% {837902ms}
>> Array(Object)>>maIsChangedFrom:using:
>> | | | | 21.9% {482896ms}
>> MaVariableObjectBuffer(MaObjectBuffer)>>maInstVarAt:
>> | | | | |20.4% {449821ms}
>> MaVariableObjectBuffer(MaObjectBuffer)>>uint:at:
>> | | | | | 20.2% {445411ms} ByteArray>>maUint:at:
>> | | | | | 20.0% {441001ms}
>> ByteArray>>maUnsigned48At:
>> | | | | | 11.1% {244755ms}
>> LargePositiveInteger>>+
>> | | | | | |8.4% {185220ms}
>> LargePositiveInteger(Integer)>>+
>> | | | | | | |8.3% {183015ms} primitives
>> | | | | | |2.7% {59535ms} primitives
>> | | | | | 8.7% {191835ms}
>> SmallInteger>>bitShift:
>> | | | | | 8.1% {178605ms}
>> SmallInteger(Integer)>>bitShift:
>> | | | | | 7.9% {174195ms} primitives
>> | | | | 13.6% {299881ms} MaObjectSerializer>>oidFor:
>> | | | | 13.2% {291061ms}
>> MagmaOidManager(MaOidManager)>>oidFor:
>> | | | | 13.1% {288856ms}
>> MagmaOidManager>>oidFor:ifAbsent:
>> | | | | 9.9% {218295ms}
>> MagmaOidManager(MaOidManager)>>oidFor:ifAbsent:
>> | | | | |8.4% {185220ms}
>> SmallInteger(Integer)>>maOid
>> | | | | | 8.3% {183015ms} MaOidCalculator
>> class>>oidForInteger:
>> | | | | | 8.0% {176400ms} SmallInteger>>+
>> | | | | | 7.8% {171990ms}
>> SmallInteger(Integer)>>+
>> | | | | | 5.9% {130095ms} primitives
>> | | | | 2.6% {57330ms} primitives
>> | | |3.2% {70560ms} Dictionary>>maIsChangedFrom:using:
>> | | | 2.1% {46305ms} MaObjectSerializer>>oidFor:
>> | | | 2.0% {44100ms}
>> MagmaOidManager(MaOidManager)>>oidFor:
>> | | | 2.0% {44100ms}
>> MagmaOidManager>>oidFor:ifAbsent:
>> | | 19.9% {438796ms}
>> MaFixedObjectBuffer>>isDifferent:using:
>> | | |9.7% {213885ms} MaObjectSerializer>>oidFor:
>> | | | |9.6% {211680ms}
>> MagmaOidManager(MaOidManager)>>oidFor:
>> | | | | 9.6% {211680ms}
>> MagmaOidManager>>oidFor:ifAbsent:
>> | | | | 6.4% {141120ms}
>> MagmaOidManager(MaOidManager)>>oidFor:ifAbsent:
>> | | | | |2.7% {59535ms}
>> WeakIdentityKeyDictionary(Dictionary)>>maAt:ifPresent:ifAbsent:
>> | | | | | |2.3% {50715ms}
>> WeakIdentityKeyDictionary(Dictionary)>>at:ifAbsent:
>> | | | | |2.3% {50715ms}
>> SmallInteger(Integer)>>maOid
>> | | | | | 2.2% {48510ms} MaOidCalculator
>> class>>oidForInteger:
>> | | | | 2.2% {48510ms}
>> WeakIdentityKeyDictionary(Dictionary)>>maAt:ifPresent:ifAbsent:
>> | | |6.5% {143325ms}
>> MaFixedObjectBuffer(MaObjectBuffer)>>maInstVarAt:
>> | | | 6.0% {132300ms}
>> MaFixedObjectBuffer(MaObjectBuffer)>>uint:at:
>> | | | 5.9% {130095ms} ByteArray>>maUint:at:
>> | | | 5.9% {130095ms} ByteArray>>maUnsigned48At:
>> | | | 3.3% {72765ms} LargePositiveInteger>>+
>> | | | |2.5% {55125ms}
>> LargePositiveInteger(Integer)>>+
>> | | | | 2.4% {52920ms} primitives
>> | | | 2.5% {55125ms} SmallInteger>>bitShift:
>> | | | 2.3% {50715ms}
>> SmallInteger(Integer)>>bitShift:
>> | | | 2.2% {48510ms} primitives
>> | | 2.2% {48510ms} MaTransaction>>useWriteBarrierOn:
>> |4.5% {99225ms} MaCommitPackage>>serializeObjectsUsing:
>> | |4.4% {97020ms} MaObjectSerializer>>serializeGraph:do:
>> | | 4.4% {97020ms} MaObjectSerializer>>appendGraph:do:
>> | | 3.8% {83790ms} MaObjectSerializer>>append:
>> | | 3.7% {81585ms}
>> MaObjectSerializer>>bufferFor:storageObject:startingAt:
>> |2.4% {52920ms} MagmaSession>>newCommitPackageFor:
>> 17.5% {385876ms} MagmaSession>>refreshViewUsing:
>> |17.5% {385876ms} MaCommitResult>>refresh:
>> | 16.0% {352801ms} MagmaSession>>assignPermanentOidsFrom:
>> | 13.9% {306496ms} MaObjectSerializer>>oidOf:is:
>> | |13.9% {306496ms} MagmaOidManager>>oidOf:is:
>> | | 13.9% {306496ms}
>> MagmaOidManager(MaOidManager)>>oidOf:is:
>> | | 13.4% {295471ms}
>> MaWeakValueDictionary(MaDictionary)>>at:put:
>> | | 12.7% {280036ms}
>> WeakValueDictionary(Dictionary)>>includesKey:
>> | | 12.6% {277831ms}
>> WeakValueDictionary(Dictionary)>>at:ifAbsent:
>> | | 12.2% {269010ms}
>> WeakValueDictionary(Set)>>findElementOrNil:
>> | | 11.8% {260190ms}
>> WeakValueDictionary(Dictionary)>>scanFor:
>> | 2.0% {44100ms}
>> MaObjectSerializer>>objectWithOid:ifFound:ifAbsent:
>> | 2.0% {44100ms}
>> MagmaOidManager>>objectWithOid:ifFound:ifAbsent:
>> 7.6% {167580ms} MagmaSession>>submit:
>> 7.6% {167580ms} MaLocalServerLink>>submit:
>> 7.6% {167580ms}
>> MaLocalRequestServer(MaRequestServer)>>processRequest:
>> 7.6% {167580ms} MagmaRepositoryController>>value:
>> 7.6% {167580ms}
>> MagmaRepositoryController>>processRequest:
>> 7.6% {167580ms} MaWriteRequest>>process
>> 7.6% {167580ms}
>> MaObjectRepository>>submitAll:for:beginAnother:
>> 7.5% {165375ms} MaObjectRepository>>write:
>> 4.5% {99225ms} MaRecoveryManager>>log:flush:
>> 4.0% {88200ms}
>> MaObjectSerializer>>serializeGraph:
>> 4.0% {88200ms}
>> MaObjectSerializer>>serializeGraph:do:
>> 4.0% {88200ms}
>> MaObjectSerializer>>appendGraph:do:
>> 3.1% {68355ms}
>> MaObjectSerializer>>append:
>> 3.0% {66150ms}
>> MaObjectSerializer>>bufferFor:storageObject:startingAt:
>> 2.2% {48510ms}
>> MaVariableObjectBuffer>>populateBodyFor:using:
>> 2.2% {48510ms}
>> Dictionary>>maStreamVariablyInto:for:
>> 2.0% {44100ms}
>> MaObjectSerializer>>oidFor:
>> 2.0% {44100ms}
>> MaOidManager>>oidFor:
>>
>> **Leaves**
>> 19.3% {425566ms} SmallInteger(Integer)>>+
>> 18.1% {399106ms} Dictionary>>scanFor:
>> 10.5% {231525ms} SmallInteger(Integer)>>bitShift:
>> 3.9% {85995ms} MagmaOidManager>>oidFor:ifAbsent:
>> 3.7% {81585ms} LargePositiveInteger>>+
>> 3.5% {77175ms} Dictionary>>at:ifAbsent:
>> 3.3% {72765ms} SmallInteger(Number)>>negative
>> 2.4% {52920ms} WeakKeyAssociation>>key
>>
>> **Memory**
>> old +44,200,708 bytes
>> young +2,281,748 bytes
>> used +46,482,456 bytes
>> free -2,281,036 bytes
>>
>> **GCs**
>> full 66 totalling 38,913ms (2.0% uptime),
>> avg 590.0ms
>> incr 144172 totalling 1,192,826ms (54.0% uptime),
>> avg 8.0ms
>> tenures 1,426 (avg 101 GCs/tenure)
>> root table 0 overflows
>>
>>
>> _______________________________________________
>> Magma mailing list
>> Magma at lists.squeakfoundation.org
>> http://lists.squeakfoundation.org/mailman/listinfo/magma
>>
>
More information about the Magma
mailing list