Magma August 2007

magma@lists.squeakfoundation.org

10 participants
19 discussions

Re: MagmaCollectionReader>>size takea a long time
by Burella Juan M. 01 Sep '07

01 Sep '07

Thanks Igor for your answer. If calculating a MagmaCollectionReader size is costly, then why not forward #size to its MagmaCollection? My problem is I'm sending #size to a MagmaCollectionReader because I need to paginate the collection for showing its contents in the UI. Any suggestions? Thanks Juan M. On 8/31/07, Igor Stasenko <siguctua(a)gmail.com> wrote: > > use collection. > a #size for MagmaCollectionReader is not something which can be fast, > because collectionreader designed for reading/filtering collections > with provided filter query. It's impossible to determine the number of > items in filtered collection in other way than reading and testing > full collection, thats why #size is bad practice for using with > collectionreader. It designed for sequential reading, not for random > access. > > > On 31/08/2007, Burella Juan M. <juan.burella(a)gmail.com> wrote: > > Hi list, > > > > I have a indexed MagmaCollection ( it has about 11,000 objects). I > > have noticed that requesting the size to a MagmaCollectionReader takes a > > long time whereas the same requesting to a MagmaCollection doesn´t > > > > For example: > > > > "MagmaCollection>>size - fiirst evaluation" > > [ (magmaSession root at: #Serial) size ] timeToRun "answer about 525 > > milliseconds" > > "Note: magmaSession root at: #Serial answer a MagmaCollection" > > > > > > "second evaluation" > > [ (magmaSession root at: #Serial) size ] timeToRun "answer about > > 1milliseconds" > > > > "MagmaCollectionReader>>size first evaluation" > > [ ((magmaSession root at: #Serial) read: #issnString) size ] timeToRun > > "answer about 9798 milliseconds" > > > > "second evaluation" > > [ ((magmaSession root at: #Serial) read: #issnString) size ] timeToRun > > "answer about 8992 milliseconds" > > > > Any suggestions for reducing this time? > > > > Thanks in advance. > > Juan Matias. > > > > _______________________________________________ > > Magma mailing list > > Magma(a)lists.squeakfoundation.org > > http://lists.squeakfoundation.org/mailman/listinfo/magma > > > > > > > -- > Best regards, > Igor Stasenko AKA sig. >

2 2

MagmaCollectionReader>>size takea a long time
by Burella Juan M. 01 Sep '07

01 Sep '07

Hi list, I have a indexed MagmaCollection ( it has about 11,000 objects). I have noticed that requesting the size to a MagmaCollectionReader takes a long time whereas the same requesting to a MagmaCollection doesn´t For example: "MagmaCollection>>size - fiirst evaluation" [ (magmaSession root at: #Serial) size ] timeToRun "answer about 525 milliseconds" "Note: magmaSession root at: #Serial answer a MagmaCollection" "second evaluation" [ (magmaSession root at: #Serial) size ] timeToRun "answer about 1milliseconds" "MagmaCollectionReader>>size first evaluation" [ ((magmaSession root at: #Serial) read: #issnString) size ] timeToRun "answer about 9798 milliseconds" "second evaluation" [ ((magmaSession root at: #Serial) read: #issnString) size ] timeToRun "answer about 8992 milliseconds" Any suggestions for reducing this time? Thanks in advance. Juan Matias.

2 1

Re: coordinating Magma development - configurations
by Igor Stasenko 24 Aug '07

24 Aug '07

Thank you for reply. I was bit worried you didn't replied to message about dictionaries, where i described what causing problems with my dicts. A reason is quite surprising: because they too fast :) On 24/08/07, Chris Muller <ma.chris.m(a)gmail.com> wrote: > Hi Sig, I guess I did not see your email about the Dictionary's, but > here is my reply to Keith after he asked. > > I won't have time to get too much into benchmarking in the near future > (plate is overflowing) but am grateful for your work (as I'm sure all > Magma users are). > > I want to establish an appropriate benchmark (maybe add an additional > to MagmaBenchmarker) and try all of the below-mentioned solutions; > Martin Loewies', yours, HashTable and any other possibilities. > > Best obviously is if the MagmaBenchmarker can identify which ones are > most suitable but I think the existing tests in there are a bit too > short (not enough entires in the Dictionary's, they don't test that) > to really reveal what would be the best dictionary, so we need to add > an additional test to the output. > > Thanks. > Chris > > > ---------- Forwarded message ---------- > From: Chris Muller <ma.chris.m(a)gmail.com> > Date: Aug 2, 2007 10:42 PM > Subject: Re: coordinating Magma development - configurations > To: Keith Hodges <keith_hodges(a)yahoo.co.uk> > > > Yes. I am keenly interested in the work Sig has done on the > WeakKeyIdentityDictionary's. Magma spends a considerable portion of > its time spinning through these Dictionary's, so I believe speeding > them up will have a direct, impactful benefit to Magma. > > Release 40 is about to go out the door so it is way too late to try to > evaluate his new "MaDictionary" upgrade against the other Dictionary > options like the one from Martin Loewis at > > http://bugs.squeak.org/view.php?id=6348 > > Also, there was mention of a HashTable (?) and others that need investigating. > > I have, however, included his enhancement that lets one plug whatever > kind of Dictionary one wants to use, which is a perfect segway to > "Release 41", so we'll have time to plug in the various options and > see how they all perform compared to each other. Oh, this enhancement is not complete , i added a choice to select WeakKeyIdentityDictionary class, but not WeakValueDictionary. While testing Db, which Florian gave me i noticed, that most speed impact lies on replacing WeakValueDictionary. > > Thanks, > Chris > > PS - Sig is welcome to contact me anytime, better yet, this discussion > would be great for the mailing list. > > > On 8/2/07, Keith Hodges <keith_hodges(a)yahoo.co.uk> wrote: > > Sig asked me whether you were going to look at his replacement dictionaries. > > > > I told him that you were probably being methodical and would get around > > to it. > > > > best regards > > > > Keith > > > -- Best regards, Igor Stasenko AKA sig.

1 0

Re: r41Alpha1 (was: Re: Counter in Gjallar)
by Göran Krampe 24 Aug '07

24 Aug '07

You're Da Man Chris! :) regards, Göran

2 1

Ma special collections-sig.87.mcz
by Igor Stasenko 24 Aug '07

24 Aug '07

Hi Cris (and everyone), lately i had conversation with Florian on IRC , he says that my dictionaries is about 2 times faster than current magma ones (he gave me read/commit times of his data crunching 12s - my vs 25s - old ). The only thing which darkening existence is that they seem having bug and need to be tested. Florian gave me his image, so i can try find the bug, but since i'm don't have deep knowledge in magma, maybe someone have time to test it with different DBs too? P.S. Remember, you need to _load_ this package, not merge with existing. Its located in magma tester squeaksource repository -- Best regards, Igor Stasenko AKA sig.

2 5

Bug in MagmaCollectionReader size ?
by Florian Minjat 21 Aug '07

21 Aug '07

Hi, I was toying around with the last beta version of magma in order to have all my tests green. And I think I found a bug in MagmaCollectionReader : if you add an element in a MagmaCollection without commit, sending the size message to the collection will count this non-committed element. But if you send the size message to a MagmaCollectionReader, it will not. Or it's a feature ? Here is the code of the test : session begin session root news add: un. session root news add: deux. session commitAndBegin. coll := (session root news read: #id descending: false). self assert: (coll size = coll collection asArray size). '=> assertion ok' "add without commit" session root add: trois inCollection: #news. coll := (session root news read: #id descending: false). self assert: (coll size = coll collection asArray size). '=> assertion fails' Florian

2 1

call site read strategy learning
by Igor Stasenko 16 Aug '07

16 Aug '07

I wonder, is it possible to implement a something which will learn till code runs? Lets say i have in some method a block, which invokes code for accessing some db objects and/or does some manipulations with them. What if at first invocation startup magma will learn from this process, so the next time it enters this block of code, it will already know how deep particular object types must be read to be optimal for speed/space? something as easy as: handle := session captureReadStrategy: [ some code ]. and then: session withReadStrategy: handle do: [ some code ]. or ever: session useAndCapture: handle do: [some code]. I think it can be clever enough to use as default approach for creating read strategies, so developers will be freed from making tons of tests and supposes for making static read strategies work fine with their code. Even, if its not or hard to adopt, i think some analysis can be done while running user code (like messagetally), so developer can then make decisions based on it. -- Best regards, Igor Stasenko AKA sig.

2 1

Re: magma handle memory
by Burella Juan M. 15 Aug '07

15 Aug '07

Hi Chris, there´s no problem at all with adding the package code anywhere you want. Feel free to contact me for any suggestion. >From what I´ve seen from the sample code I sent to you, although the image size grows up, when I closed the repository and evaluated MagmaSession cleanUp, the image size goes back to the state before the evaluation. In the "real migration" what I told in the first mail still the size is growing at very high rates, we cannot detect yet what objects aren´t collected by the GC. Regards, Juan Matías. On 8/10/07, Chris Muller <ma.chris.m(a)gmail.com> wrote: > > Hi Juan, THANKS for the best test case code anyone has ever sent me > for asking a question about Magma..! > > I have been too busy to get to it but am finally spending some time > with this tonight. > > Do you mind if I harvest this for the Magma tester package? It might > be a really great addition for benchmarking / profiling memory as well > overall performance. > > Let me explore this; I'll write again in the next few days.. > > Regards, > Chris > > On 8/7/07, Burella Juan M. <juan.burella(a)gmail.com> wrote: > > Hi Chris, Thanks for your answer. > > I've attached a monticello package with the tests, open a > Transcript > > window and see class comments for instructions of how to run them. > > > > This is the Transcript output for 50 "trees" with 5 of depth: > > > > Creating magma repository > > Opening magma session > > Image size before creating magma collection: 69851252 > > Creating magma collection at MagmaMemoryTest > > --Iteration: 1..5: > > Image size BEFORE creating test object: 69851252 > > Creating test objects... > > Image size AFTER creating test object: 69851252 > > Saving 5 next objects from collection to magma collection... > > Image size BEFORE Magma cleaning: 85098444 > > Image size AFTER Magma cleaning: 82623944 > > --Iteration: 6..10: > > Image size BEFORE creating test object: 82623944 > > Creating test objects... > > Image size AFTER creating test object: 82623944 > > Saving 5 next objects from collection to magma collection... > > Image size BEFORE Magma cleaning: 106407888 > > Image size AFTER Magma cleaning: 83066628 > > --Iteration: 11..15: > > Image size BEFORE creating test object: 83066628 > > Creating test objects... > > Image size AFTER creating test object: 83066628 > > Saving 5 next objects from collection to magma collection... > > Image size BEFORE Magma cleaning: 108570520 > > Image size AFTER Magma cleaning: 85184212 > > --Iteration: 16..20: > > Image size BEFORE creating test object: 85184212 > > Creating test objects... > > Image size AFTER creating test object: 85184212 > > Saving 5 next objects from collection to magma collection... > > Image size BEFORE Magma cleaning: 108627824 > > Image size AFTER Magma cleaning: 85212848 > > --Iteration: 21..25: > > Image size BEFORE creating test object: 85212848 > > Creating test objects... > > Image size AFTER creating test object: 85212848 > > Saving 5 next objects from collection to magma collection... > > Image size BEFORE Magma cleaning: 108615528 > > Image size AFTER Magma cleaning: 85216916 > > --Iteration: 26..30: > > Image size BEFORE creating test object: 85216916 > > Creating test objects... > > Image size AFTER creating test object: 85216916 > > Saving 5 next objects from collection to magma collection... > > Image size BEFORE Magma cleaning: 108787536 > > Image size AFTER Magma cleaning: 85257868 > > --Iteration: 31..35: > > Image size BEFORE creating test object: 85257868 > > Creating test objects... > > Image size AFTER creating test object: 85257868 > > Saving 5 next objects from collection to magma collection... > > Image size BEFORE Magma cleaning: 108857164 > > Image size AFTER Magma cleaning: 85307012 > > --Iteration: 36..40: > > Image size BEFORE creating test object: 85307012 > > Creating test objects... > > Image size AFTER creating test object: 85307012 > > Saving 5 next objects from collection to magma collection... > > Image size BEFORE Magma cleaning: 108623676 > > Image size AFTER Magma cleaning: 85257812 > > --Iteration: 41..45: > > Image size BEFORE creating test object: 85257812 > > Creating test objects... > > Image size AFTER creating test object: 85257812 > > Saving 5 next objects from collection to magma collection... > > Image size BEFORE Magma cleaning: 108390116 > > Image size AFTER Magma cleaning: 85159312 > > --Iteration: 46..50: > > Image size BEFORE creating test object: 85159312 > > Creating test objects... > > Image size AFTER creating test object: 85159312 > > Saving 5 next objects from collection to magma collection... > > Image size BEFORE Magma cleaning: 108594588 > > Image size AFTER Magma cleaning: 85183668 > > > > > > On 8/7/07, Chris Muller <asqueaker(a)gmail.com> wrote: > > > HI Juan, > > > > > > I really need to add a "How to do a Bulk Load" page to the > documentation, > > sorry. > > > > > > In the meantime, you weren't clear what you were adding the 19K trees > > > of objects *TO*. My guess is you are adding to a Smalltlak > > > collection; a Dictionary or OrderedCollection perhaps? It's good you > > > are doing the stubbing and finalizieOids, but they won't help much if > > > the root collection is a Smalltalk collection because the entire > > > persistent graph is in memory then (except for the stubbed, maybe). > > > > > > You need to add them to a MagmaCollection instead, so that transparent > > > paging of objects in and out of memory can occur. > > > > > > To verify what kinds of persistents you have in memory, check out your > > > #cachedObjectCountByClass. You should have no more than one 19K tree > > > worth of objects given that you are doing a garbageCollect after each > > > one. > > > > > > If you would share the skeleton of your load script it also might shed > > > some light. > > > > > > Thanks, > > > Chris > > > > > > > > > On 8/6/07, Burella Juan M. <juan.burella(a)gmail.com> wrote: > > > > I need to do a massive migration to Magma. My approach is currently > like > > > > this: I'm loading the data from external files, then build a complex > > object > > > > and commit it to a magma repository, and over again until the file > ends. > > > > > > > > While doing this, my image was growing at very high rates, approx. > 200Mb > > and > > > > taking several days, between 5 or 6. So, to see what's happening, > I've > > done > > > > some tests with toy objects, each one of them has 5 iVars. and each > one > > has > > > > another 5 iVars. This generates a tree of 19.556 objects > > > > (5^0+5^1+5^2+5^3+5^4+5^5). Then I started to adding 50 of these > "tree" > > (or > > > > smock?) objects, by groups of 5. I tried to maintain the complexity > of > > these > > > > objects by adding "random" variable sized strings to the leafs and > other > > > > miscellaneous objets. > > > > When this 50 are finalized, I've done: > > > > > > > > self magmaSession stubOut: aCollection. "aCollection has 50 smock > > objects" > > > > aCollection := nil. > > > > Smalltalk garbageCollect. > > > > self magmaSession finalizeOids. > > > > > > > > But still the image is increasing its size. At beggining of the test > the > > > > image size (as reported from #vmParameterAt: 3) was approx. 60Mb and > at > > the > > > > end was approx. 89MB. > > > > > > > > Any suggestions for reducing the growing of the image size? > > > > > > > > Thanks in advance. > > > > Juan Matias. > > > > > > > > _______________________________________________ > > > > Magma mailing list > > > > Magma(a)lists.squeakfoundation.org > > > > > > http://lists.squeakfoundation.org/mailman/listinfo/magma > > > > > > > > > > > > > > > > > >

2 2

Re: Using remote magma server problem
by Igor Stasenko 12 Aug '07

12 Aug '07

On 10/08/07, Chris Muller <asqueaker(a)gmail.com> wrote: > Put a halt in that method before it checks MaClientServerPreferences > debug and tell me what "exc" is. > MaObjectSerializationSoftwareError: class-definition not found it tries to find a class definition with id 207 while MaClassIdManager nextClassId is 206 at same time, on client side i have a two instances of MaClassIdManager both have classes with id 207. I'm not sure which one is used in anchor request, here their classDefinitionsById dictionaries contents: .... few hours later... i finally found the cause! It was because of assigning new read strategy to session just after connecting to it. I disabled assignment, and after this all works just fine (because serializing anchor request graph not causes of serializing readstrategy because its nil ). Can you give an example in what sequence we must set read strategy , does it needs to be inside transaction and/or only after reifying root object? And why it works with local magma repository, but fails with remote? > Also look at the the entire stack to see what led to the cause of the > error. If you paste me the stack I might be able to make an educated > guess. > > Is there some modification you've made to Magma? Anchor requests are > made hundreds of times across the remote connection in the test cases. > I believe Hilaire also has tested his program across the Internet. > Have you tried it in a standard Magma image (without Gjaller) to > observe it works there before adding the additional Gjaller components > in? > As far as i know, no modifications was made to magma (Keith - correct me if i'm wrong) -- Best regards, Igor Stasenko AKA sig.

2 1

Using remote magma server problem
by Igor Stasenko 10 Aug '07

10 Aug '07

We have problems with using magma via remote link in gjallar project. I debugged code and now have some extra details. There are 3 packets sent to server: MagmaSession>>connectAs: - ok MagmaSession>>abort - ok MagmaSession>>anchor - fails with error on server side (mailformed packet). Take a look at MaTcpRequestServer>>materializeRequest: at error (when debug is off) it creates MaMalformedRequestError and return its as result. Then code, which follows #materializeRequest: trying to use MaMalformedRequestError object as request which of course causes MNU, opening debugger and prevents server to respond with 'mailformed request' packet. I think that to find the cause of problem, first we need to fix this behaviour on server side. Have any ideas why anchor request packet formed on client side cannot be materialized on server side? -- Best regards, Igor Stasenko AKA sig.

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Magma August 2007