Sorry for bother you with these elementary questions.
I am playing with differentes keySizes in indexes, but everytime I change the keySize of an index and rebuild the database Squeak hungs (or it takes too much time to do the task). I'm saveving only 200 objects in a one single entry with two indexes. The first time I save them, its very fast, but when I begin playing with differents keySizes, Squeak loops. Normally I close the session (a single user sesssion) and reopen it, before to rebuild the database and the indexes, but I have tried even deleting the db files and recreate them. I thought its must be something about references that remains in my image, so I try to find how to perform a garbageCollect of Magma. I found a comment in MagmaCompressor that says "yourRepositoryController garbageColllect". But this object does not understand #garbageCollect, nor any other magma object. Would you give me any hint? Or something to read about that?
Thank you in advance. Norberto
I would never figure I would receive such amount of answers. I can hardly read them. Obviously, I won't use Magma, consider my second question in a very simple matter that denounce an unstable behaviour has no answer in two weeks. Bye
On 3/29/06, Norberto Manzanos nmanzanos@gmail.com wrote:
Sorry for bother you with these elementary questions.
I am playing with differentes keySizes in indexes, but everytime I change the keySize of an index and rebuild the database Squeak hungs (or it takes too much time to do the task). I'm saveving only 200 objects in a one single entry with two indexes. The first time I save them, its very fast, but when I begin playing with differents keySizes, Squeak loops. Normally I close the session (a single user sesssion) and reopen it, before to rebuild the database and the indexes, but I have tried even deleting the db files and recreate them. I thought its must be something about references that remains in my image, so I try to find how to perform a garbageCollect of Magma. I found a comment in MagmaCompressor that says "yourRepositoryController garbageColllect". But this object does not understand #garbageCollect, nor any other magma object. Would you give me any hint? Or something to read about that?
Thank you in advance. Norberto
Norberto, I did not mean to ignore your post, I just had no idea how to respond. I really wanted to help but it was like trying to answer, "My car won't start, what is the problem?" Still, I could have at least asked you for more information; I actually thought you were going to do further investigation and post again.
I will try to do better to answer questions the best I can. Although I am very busy (who isn't), I am pleased to provide free support each weeknight, after work. Of course I would be very happy to offer "daytime" professional consultation to anyone willing to assist with my bills. :) I guarantee I can greatly assist any Magma project.
In any case, detailed descriptions, objectives, and tangible examples such as code-snippets, stack-traces, message-tally's, etc, are essential that any software question, Magma or otherwise, can be properly addressed.
Best Regards, Chris
--- Norberto Manzanos nmanzanos@gmail.com wrote:
I would never figure I would receive such amount of answers. I can hardly read them. Obviously, I won't use Magma, consider my second question in a very simple matter that denounce an unstable behaviour has no answer in two weeks. Bye
On 3/29/06, Norberto Manzanos nmanzanos@gmail.com wrote:
Sorry for bother you with these elementary questions.
I am playing with differentes keySizes in indexes, but everytime I change the keySize of an index and rebuild the database Squeak
hungs
(or it takes too much time to do the task). I'm saveving only 200 objects in a one single entry with two
indexes.
The first time I save them, its very fast, but when I begin playing with differents keySizes, Squeak loops. Normally I close the session (a single user sesssion) and reopen
it,
before to rebuild the database and the indexes, but I have tried
even
deleting the db files and recreate them. I thought its must be something about references that remains in my image, so I try to
find
how to perform a garbageCollect of Magma. I found a comment in MagmaCompressor that says "yourRepositoryController
garbageColllect".
But this object does not understand #garbageCollect, nor any other magma object. Would you give me any hint? Or something to read about that?
Thank you in advance. Norberto
Magma mailing list Magma@lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma
Hi Noberto,
I have been using Magma for a couple of years and agressively for the last year. I can assure you that Chris's willingess to assist is unequalled.
I have had serious problems, usually misunderstanding on my part, cleared up by the time I check my mail again.
I do however recommend capturing your problem in a well commented workspace of code which reproduces your problem. The rest of us can then reproduce your problem as you experienced it.
Regards
Brent
Sorry for bother you with these elementary questions.
No bother!
I am playing with differentes keySizes in indexes, but everytime I change the keySize of an index and rebuild the database Squeak hungs (or it takes too much time to do the task).
#keySize: is an 'initializing' method. You cannot change the keySize of an index, but you may remove the old index and create a new index with a different keySize.
I'm saveving only 200 objects in a one single entry with two indexes.
By "one single entry with two indexes" do you mean, one single MagmaCollection with two indexes?
The first time I save them, its very fast, but when I begin playing with differents keySizes, Squeak loops.
How are you "playing with different key sizes? Its true that larger keys are significantly slower than smaller keys, but I don't think its looping endlessly..
Normally I close the session (a single user sesssion) and reopen it, before to rebuild the database and the indexes, but I have tried even deleting the db files and recreate them. I thought its must be something about references that remains in my image, so I try to find how to perform a garbageCollect of Magma. I found a comment in MagmaCompressor that says "yourRepositoryController garbageColllect". But this object does not understand #garbageCollect, nor any other magma object.
To "rebuild the database" I presume you mean to load it from some source? If so, this advice will help you go faster:
http://minnow.cc.gatech.edu/squeak/2985
If you decide to keep going, I will help the best I can.
- Chris
I'm very sorry. I was confused. The problem was not the keySize at all. Evidently, when I thought I was adding objects, sometimes I wasn't, so it was faster and I believed something was wrong when actually it was ok. Thas the code
MagmaRepositoryController create: Pathname default,'magma\CaicytDB.magma' root: Dictionary new.
magmaSession:= (MagmaSession openLocal:Pathname default,'magma\CaicytDB.magma'). magmaSession connectAs: 'nor'.
mCollection:= magmaSession commit: [ magmaSession root at: 'Country' put: MagmaCollection new ]. mCollection addIndex: ((MaSearchStringIndex attribute: #stringName) keySize: 128).
"thats my collection of countries, in a sort of pseudo db" Time millisecondsToRun:[ (1 to: 10) do:[:idx | magmaSession commit: [ (magmaSession root at: 'Country') add: (countries at: idx) object]]]. No matter what is the size of the index, the time to add ten objects is about one minut. Even without index, the time is almost the same. So the problem is not the indexes, but the time consumed on adding new objects. I noticed that the first country has an orderedCollection with 295 cities. So I save these cities first (540652 ms for the 295 cities) and then, when I save the first ten countries, the elapsed times descended to 14 seconds. Then I tried saving a new country, without any reference. It took 3 seconds (sometimes 2 or 1.5). It's slow... But I really like Magma. Thanks for your time And sorry again. I will study more on magma before make another question. Norberto
On 4/17/06, Chris Muller chris@funkyobjects.org wrote:
Sorry for bother you with these elementary questions.
No bother!
I am playing with differentes keySizes in indexes, but everytime I change the keySize of an index and rebuild the database Squeak hungs (or it takes too much time to do the task).
#keySize: is an 'initializing' method. You cannot change the keySize of an index, but you may remove the old index and create a new index with a different keySize.
I'm saveving only 200 objects in a one single entry with two indexes.
By "one single entry with two indexes" do you mean, one single MagmaCollection with two indexes?
The first time I save them, its very fast, but when I begin playing with differents keySizes, Squeak loops.
How are you "playing with different key sizes? Its true that larger keys are significantly slower than smaller keys, but I don't think its looping endlessly..
Normally I close the session (a single user sesssion) and reopen it, before to rebuild the database and the indexes, but I have tried even deleting the db files and recreate them. I thought its must be something about references that remains in my image, so I try to find how to perform a garbageCollect of Magma. I found a comment in MagmaCompressor that says "yourRepositoryController garbageColllect". But this object does not understand #garbageCollect, nor any other magma object.
To "rebuild the database" I presume you mean to load it from some source? If so, this advice will help you go faster:
http://minnow.cc.gatech.edu/squeak/2985
If you decide to keep going, I will help the best I can.
- Chris
Magma mailing list Magma@lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma
So the problem is not the indexes, but the time consumed on adding new objects. I noticed that the first country has an orderedCollection with 295 cities. So I save these cities first (540652 ms for the 295 cities) and then, when I save the first ten countries, the elapsed times descended to 14 seconds. Then I tried saving a new country, without any reference. It took 3 seconds (sometimes 2 or 1.5). It's slow... But I really like Magma. Thanks for your time And sorry again. I will study more on magma before make another question.
Hi Norberto, would you post a MessageTally here to help see what's going on?
I don't know how complex your City objects are, but 540 seconds to add 295 cities seems awfully slow to me, what kind of computer you running, a PDA?? :)
Also, when "loading" a database, it is good to not let the cachedObjectCount grow and grow. I'm not sure where you are getting the objects (importing from a disk file? creating in memory?), but if they stay in memory then loading will get slower and slower.
Please post a MessageTally of the slow operation.
- Chris
Hi Chris. The messageTally is at the end of the message. I tried with only 10 cities, in order to make the list easier to read.
My City objects are of a medium complexity level. But the cities I'm persisting now have no many things. They have a collection of names (most of the cases, just one, until now), and a city code. They know their country. And each country knows their cities. Though this double reference problem, I have some objects to avoid recursion and duplication. I have many double-referenced objects in my model, and all them has some of this objects, wich are very simple ones (some symbols v.i). There are other v.i. that have nil values until now. I think that the double-reference could not be a problem for the moment, because I'm not changing the depth. The objects are created in memory. I'm testing a migration from and old data base to Magma, but, for some specific reasons my plan is to upload them from this DB to memory, then persist the objects in Magma, and throw away the rest. I'm not worried with time consumed on the migration. These could be a problem when the application will be running. The machine is a Pentium 4 with 512 MB RAM. It's not bad. But my image is too big. (100Mb) Could be this huge image a problem?
Other topic Is there any reason for not including this method: ?
MagmaCollectionReader>> union: aCollection "Answer the set theoretic union of two collections"
^ self asSet addAll: aCollection; yourself
I think that if you have #intersection: and #difference:, #union: might have its place. I tested it and works fine.
Regards Norberto
- 78841 tallies, 80737 msec.
**Tree** 95.8% {77346ms} MagmaSession>>commit: |95.8% {77346ms} MagmaSession>>commit | 95.8% {77346ms} MagmaSession>>commitAndBegin: | 54.3% {43840ms} MagmaSession>>newCommitPackageFor: | |37.9% {30599ms} MaTransaction>>changedObjects | | |37.9% {30599ms} MaTransaction>>addChangesFromReadSet | | | 37.4% {30196ms} MaTransaction>>didChange:from: | | | 22.1% {17843ms} MaFixedObjectBuffer>>isDifferent:using: | | | |9.5% {7670ms} MaObjectSerializer>>oidFor: | | | | |9.1% {7347ms} MagmaOidManager(MaOidManager)>>oidFor: | | | | | 9.0% {7266ms} MagmaOidManager>>oidFor:ifAbsent: | | | | | 6.3% {5086ms} MagmaOidManager(MaOidManager)>>oidFor:ifAbsent: | | | | | 3.9% {3149ms} WeakIdentityKeyDictionary(Dictionary)>>maAt:ifPresent:ifAbsent: | | | | | 3.3% {2664ms} WeakIdentityKeyDictionary(Dictionary)>>at:ifAbsent: | | | | | 2.7% {2180ms} WeakIdentityKeyDictionary(Set)>>findElementOrNil: | | | | | 2.5% {2018ms} WeakIdentityKeyDictionary>>scanFor: | | | |5.3% {4279ms} MaFixedObjectBuffer(MaObjectBuffer)>>maInstVarAt: | | | | 3.4% {2745ms} MaFixedObjectBuffer(MaObjectBuffer)>>uint:at: | | | | 3.2% {2584ms} ByteArray>>maUint:at: | | | | 2.8% {2261ms} ByteArray>>maUnsigned48At: | | | 10.3% {8316ms} MaVariableObjectBuffer(MaVariableBuffer)>>isDifferent:using: | | | |9.6% {7751ms} Array>>maIsChangedFrom:using: | | | | 8.6% {6943ms} Array(Object)>>maIsChangedFrom:using: | | | | 4.4% {3552ms} MaObjectSerializer>>oidFor: | | | | 3.7% {2987ms} MagmaOidManager(MaOidManager)>>oidFor: | | | | 3.6% {2907ms} MagmaOidManager>>oidFor:ifAbsent: | | | | 2.2% {1776ms} MagmaOidManager(MaOidManager)>>oidFor:ifAbsent: | | | 4.6% {3714ms} MaTransaction>>useWriteBarrierOn: | | | 2.4% {1938ms} MagmaPreferences>>allowWriteBarrier | | | 2.2% {1776ms} Array(Object)>>maAllowsWriteBarrier | |12.9% {10415ms} MaCommitPackage>>serializeObjectsUsing: | | |12.7% {10254ms} MaObjectSerializer>>serializeGraph:do: | | | 12.7% {10254ms} MaObjectSerializer>>appendGraph:do: | | | 11.7% {9446ms} MaObjectSerializer>>append: | | | 11.2% {9043ms} MaObjectSerializer>>bufferFor:storageObject:startingAt: | | | 5.7% {4602ms} MaFixedObjectBuffer>>populateBodyFor:using: | | | |5.4% {4360ms} MaObjectSerializer>>oidFor: | | | | 5.4% {4360ms} MagmaOidManager(MaOidManager)>>oidFor: | | | | 4.1% {3310ms} MagmaOidManager>>oidOf:is: | | | | 3.0% {2422ms} MaWeakValueDictionary(MaDictionary)>>at:put: | | | | 2.5% {2018ms} WeakValueDictionary(Dictionary)>>includesKey: | | | | 2.5% {2018ms} WeakValueDictionary(Dictionary)>>at:ifAbsent: | | | | 2.4% {1938ms} WeakValueDictionary(Set)>>findElementOrNil: [2.4% {1938ms} WeakValueDictionary(Dictionary)>>scanFor: | | | 2.6% {2099ms} MaVariableObjectBuffer>>populateBodyFor:using: | | | 2.6% {2099ms} Array(Object)>>maStreamVariablyInto:for: | | | 2.3% {1857ms} MaObjectSerializer>>oidFor: | | | 2.2% {1776ms} MagmaOidManager(MaOidManager)>>oidFor: | |3.6% {2907ms} MagmaSession>>newCommitPackageFor: | | 3.5% {2826ms} MaCommitPackage>>serializeObjectsUsing: | | 3.4% {2745ms} MaObjectSerializer>>serializeGraph:do: | | 3.4% {2745ms} MaObjectSerializer>>appendGraph:do: | | 2.3% {1857ms} MaObjectSerializer>>append: | | 2.0% {1615ms} MaObjectSerializer>>bufferFor:storageObject:startingAt: | 21.2% {17116ms} MagmaSession>>submit: | |21.2% {17116ms} MaLocalServerLink>>submit: | | 21.2% {17116ms} MaLocalRequestServer(MaRequestServer)>>processRequest: | | 21.2% {17116ms} MagmaRepositoryController>>value: | | 21.2% {17116ms} MagmaRepositoryController>>processRequest: | | 21.2% {17116ms} MaWriteRequest>>process | | 21.2% {17116ms} MaObjectRepository>>submitAll:for:beginAnother: | | 20.9% {16874ms} MaObjectRepository>>write: | | 13.0% {10496ms} MaRecoveryManager>>log:flush: | | |11.3% {9123ms} MaObjectSerializer>>serializeGraph: | | | 11.3% {9123ms} MaObjectSerializer>>serializeGraph:do: | | | 11.3% {9123ms} MaObjectSerializer>>appendGraph:do: | | | 9.3% {7509ms} MaObjectSerializer>>append: | | | 9.0% {7266ms} MaObjectSerializer>>bufferFor:storageObject:startingAt: [6.7% {5409ms} MaVariableObjectBuffer>>populateBodyFor:using: [ 6.7% {5409ms} Dictionary>>maStreamVariablyInto:for: [ 6.4% {5167ms} MaObjectSerializer>>oidFor: [ 6.3% {5086ms} MaOidManager>>oidFor: [ 4.8% {3875ms} MaOidManager>>oidOf:is: [ 3.6% {2907ms} MaWeakValueDictionary(MaDictionary)>>at:put: [ 3.3% {2664ms} WeakValueDictionary(Dictionary)>>includesKey: [ 3.3% {2664ms} WeakValueDictionary(Dictionary)>>at:ifAbsent: [ 3.2% {2584ms} WeakValueDictionary(Set)>>findElementOrNil: [ 3.2% {2584ms} WeakValueDictionary(Dictionary)>>scanFor: | | 6.0% {4844ms} MaObjectRepository>>applyToCache: | | 4.0% {3229ms} MaObjectFiler>>write:index: | | 2.2% {1776ms} MaObjectFiler>>write:at: | | 2.1% {1695ms} MaTransactionalFileStream>>maWrite:bytesFro...:atFilePosition: | 20.2% {16309ms} MagmaSession>>processNewAndRemovedIndexesIn:using: | 12.7% {10254ms} MagmaSession>>refreshViewUsing: | |12.7% {10254ms} MaCommitResult>>refresh: | | 9.3% {7509ms} MagmaSession>>assignPermanentOidsFrom: | | |5.7% {4602ms} MaObjectSerializer>>oidOf:is: | | | |5.7% {4602ms} MagmaOidManager>>oidOf:is: | | | | 5.7% {4602ms} MagmaOidManager(MaOidManager)>>oidOf:is: | | | | 3.4% {2745ms} MagmaOidManager>>removeGarbageCollectedObjectEntries | | | | 3.4% {2745ms} MaWeakValueDictionary(MaDictionary)>>reject: | | | | 3.4% {2745ms} MaWeakValueDictionary(MaDictionary)>>add: | | | | 2.3% {1857ms} MaWeakValueDictionary(MaDictionary)>>removeKey:ifAbsent: | | | | 2.2% {1776ms} WeakValueDictionary(Dictionary)>>removeKey:ifAbsent: | | | | 2.2% {1776ms} WeakValueDictionary(Set)>>findElementOrNil: | | | | 2.1% {1695ms} WeakValueDictionary(Dictionary)>>scanFor: | | |3.6% {2907ms} MaObjectSerializer>>objectWithOid:ifFound:ifAbsent: | | | 3.5% {2826ms} MagmaOidManager>>objectWithOid:ifFound:ifAbsent: | | | 3.2% {2584ms} MaWeakValueDictionary(MaDictionary)>>maAt:ifPresent:ifAbsent: | | | 3.2% {2584ms} MaWeakValueDictionary(MaDictionary)>>at:ifAbsent: | | | 3.1% {2503ms} WeakValueDictionary(Dictionary)>>at:ifAbsent: | | | 3.0% {2422ms} WeakValueDictionary(Set)>>findElementOrNil: | | | 3.0% {2422ms} WeakValueDictionary(Dictionary)>>scanFor: | | 3.4% {2745ms} MagmaSession>>recordObjects: | 5.9% {4763ms} MagmaCollection>>buildIndexes:ignoring: | 5.9% {4763ms} MagmaCollection>>commitDo: | 2.0% {1615ms} MagmaCollection>>commitUnlock | 2.0% {1615ms} MagmaSession>>commit: | 2.0% {1615ms} MagmaSession>>commit | 2.0% {1615ms} MagmaSession>>commitAndBegin: | 2.0% {1615ms} MagmaSession>>newCommitPackageFor: | 2.0% {1615ms} MaTransaction>>changedObjects | 2.0% {1615ms} MaTransaction>>addChangesFromReadSet 3.8% {3068ms} MagmaSession>>initializeSerializer 2.9% {2341ms} MagmaSession>>oidFor: 2.8% {2261ms} MaObjectSerializer>>oidFor: 2.8% {2261ms} MagmaOidManager(MaOidManager)>>oidFor: 2.8% {2261ms} MagmaOidManager>>oidFor:ifAbsent: 2.5% {2018ms} WeakIdentityKeyDictionary(Dictionary)>>maAt:ifPresent:ifAbsent: 2.4% {1938ms} WeakIdentityKeyDictionary(Dictionary)>>at:ifAbsent: 2.3% {1857ms} WeakIdentityKeyDictionary(Set)>>findElementOrNil: 2.2% {1776ms} WeakIdentityKeyDictionary>>scanFor: **Leaves** 14.6% {11788ms} Dictionary>>scanFor: 8.5% {6863ms} WeakIdentityKeyDictionary>>scanFor: 6.7% {5409ms} WeakKeyAssociation>>key 3.5% {2826ms} WeakValueDictionary(Dictionary)>>at:ifAbsent: 2.4% {1938ms} Set>>findElementOrNil: 2.4% {1938ms} Dictionary>>maAt:ifPresent:ifAbsent: 2.3% {1857ms} LargePositiveInteger(Integer)>>+ 2.2% {1776ms} IdentitySet>>scanFor: 2.1% {1695ms} MagmaOidManager(MaOidManager)>>oidFor:ifAbsent: 2.0% {1615ms} MaVariableObjectBuffer(MaObjectBuffer)>>maInstVarAt:
**Memory** old +11,548,552 bytes young -162,240 bytes used +11,386,312 bytes free -612,792 bytes
**GCs** full 6 totalling 3,593ms (4.0% uptime), avg 599.0ms incr 10963 totalling 9,990ms (12.0% uptime), avg 1.0ms tenures 173 (avg 63 GCs/tenure) root table 0 overflows
Hi,
My City objects are of a medium complexity level. But the cities I'm persisting now have no many things. They have a collection of names (most of the cases, just one, until now), and a city code. They know their country. And each country knows their cities. Though this double reference problem, I have some objects to avoid recursion and duplication.
I hope this is not for Magma's sake. Magma can handle recursive structures, no problem at all. In fact, using a rich model of shared objects is encouraged.
The objects are created in memory. I'm testing a migration from and old data base to Magma, but, for some specific reasons my plan is to upload them from this DB to memory, then persist the objects in Magma, and throw away the rest.
I can only guess that you do have some volume of data here.. It is taking 12 seconds just to serialize your commit package and 16 seconds to write the buffers? On a Pentium 4 that's got to be a fair amount of data.
I don't see that this portion of the migration can be reduced, but are you loading them all in one commit or multiple smaller commits? If multiple small commits then it is important to keep your #cachedObjectCount as low possible, but this means that you cannot have them sitting around in memory.
I have tested WriteBarrier to see it speed up commit by a factor of 4x, this would probably help but I can't promise I can support it.
I'm not worried with time consumed on the migration. These could be a problem when the application will be running. The machine is a Pentium 4 with 512 MB RAM. It's not bad. But my image is too big. (100Mb) Could be this huge image a problem?
It could be a problem especially if the objects consuming memory are the ones being committed to Magma.
Other topic Is there any reason for not including this method: ?
MagmaCollectionReader>> union: aCollection "Answer the set theoretic union of two collections"
^ self asSet addAll: aCollection; yourself
Just an oversight. Thanks for bringing to my attention.
If you can share a bit more on your migration program it might help. How many objects are you committing at a time and is there any way you can get them out of memory? Out of curiousity, what is your #cachedObjectCount after loading the 10 cities?
It's also possible Magma may not be able to (quickly) handle the volume of objects you are throwing at it..
Regards, Chris
magma@lists.squeakfoundation.org