magma handle memory

Fri Aug 10 16:36:40 UTC 2007

Hi Chris, there´s no problem at all with adding the package code anywhere
you want. Feel free to contact me for any suggestion.
>From what I´ve seen from the sample code I sent to you, although the image
size grows up, when I closed the repository and evaluated MagmaSession
cleanUp, the image size goes back to the state before the evaluation.
In the "real migration" what I told in the first
mail still the size is growing at very high rates, we cannot detect
yet what objects aren´t collected by the GC.

Regards,
Juan Matías.

On 8/10/07, Chris Muller <ma.chris.m at gmail.com> wrote:
>
> Hi Juan, THANKS for the best test case code anyone has ever sent me
> for asking a question about Magma..!
>
> I have been too busy to get to it but am finally spending some time
> with this tonight.
>
> Do you mind if I harvest this for the Magma tester package?  It might
> be a really great addition for benchmarking / profiling memory as well
> overall performance.
>
> Let me explore this; I'll write again in the next few days..
>
> Regards,
>   Chris
>
> On 8/7/07, Burella Juan M. <juan.burella at gmail.com> wrote:
> > Hi Chris, Thanks for your answer.
> >      I've attached a monticello package with the tests, open a
> Transcript
> > window and see class comments for instructions of how to run them.
> >
> > This is the Transcript output for 50 "trees" with 5 of depth:
> >
> > Creating magma repository
> > Opening magma session
> > Image size before creating magma collection: 69851252
> > Creating magma collection at MagmaMemoryTest
> > --Iteration: 1..5:
> > Image size BEFORE creating test object: 69851252
> > Creating test objects...
> > Image size AFTER creating test object: 69851252
> > Saving 5 next objects from collection to magma collection...
> > Image size BEFORE Magma cleaning: 85098444
> > Image size AFTER Magma cleaning: 82623944
> > --Iteration: 6..10:
> > Image size BEFORE creating test object: 82623944
> > Creating test objects...
> > Image size AFTER creating test object: 82623944
> > Saving 5 next objects from collection to magma collection...
> >  Image size BEFORE Magma cleaning: 106407888
> > Image size AFTER Magma cleaning: 83066628
> > --Iteration: 11..15:
> > Image size BEFORE creating test object: 83066628
> > Creating test objects...
> > Image size AFTER creating test object: 83066628
> > Saving 5 next objects from collection to magma collection...
> > Image size BEFORE Magma cleaning: 108570520
> > Image size AFTER Magma cleaning: 85184212
> > --Iteration: 16..20:
> > Image size BEFORE creating test object: 85184212
> > Creating test objects...
> > Image size AFTER creating test object: 85184212
> > Saving 5 next objects from collection to magma collection...
> > Image size BEFORE Magma cleaning: 108627824
> > Image size AFTER Magma cleaning: 85212848
> > --Iteration: 21..25:
> > Image size BEFORE creating test object: 85212848
> > Creating test objects...
> > Image size AFTER creating test object: 85212848
> > Saving 5 next objects from collection to magma collection...
> >  Image size BEFORE Magma cleaning: 108615528
> > Image size AFTER Magma cleaning: 85216916
> > --Iteration: 26..30:
> > Image size BEFORE creating test object: 85216916
> > Creating test objects...
> > Image size AFTER creating test object: 85216916
> > Saving 5 next objects from collection to magma collection...
> > Image size BEFORE Magma cleaning: 108787536
> > Image size AFTER Magma cleaning: 85257868
> > --Iteration: 31..35:
> > Image size BEFORE creating test object: 85257868
> > Creating test objects...
> > Image size AFTER creating test object: 85257868
> > Saving 5 next objects from collection to magma collection...
> > Image size BEFORE Magma cleaning: 108857164
> > Image size AFTER Magma cleaning: 85307012
> > --Iteration: 36..40:
> > Image size BEFORE creating test object: 85307012
> > Creating test objects...
> > Image size AFTER creating test object: 85307012
> > Saving 5 next objects from collection to magma collection...
> >  Image size BEFORE Magma cleaning: 108623676
> > Image size AFTER Magma cleaning: 85257812
> > --Iteration: 41..45:
> > Image size BEFORE creating test object: 85257812
> > Creating test objects...
> > Image size AFTER creating test object: 85257812
> > Saving 5 next objects from collection to magma collection...
> > Image size BEFORE Magma cleaning: 108390116
> > Image size AFTER Magma cleaning: 85159312
> > --Iteration: 46..50:
> > Image size BEFORE creating test object: 85159312
> > Creating test objects...
> > Image size AFTER creating test object: 85159312
> > Saving 5 next objects from collection to magma collection...
> > Image size BEFORE Magma cleaning: 108594588
> > Image size AFTER Magma cleaning: 85183668
> >
> >
> > On 8/7/07, Chris Muller <asqueaker at gmail.com> wrote:
> > > HI Juan,
> > >
> > > I really need to add a "How to do a Bulk Load" page to the
> documentation,
> > sorry.
> > >
> > > In the meantime, you weren't clear what you were adding the 19K trees
> > > of objects *TO*.  My guess is you are adding to a Smalltlak
> > > collection; a Dictionary or OrderedCollection perhaps?  It's good you
> > > are doing the stubbing and finalizieOids, but they won't help much if
> > > the root collection is a Smalltalk collection because the entire
> > > persistent graph is in memory then (except for the stubbed, maybe).
> > >
> > > You need to add them to a MagmaCollection instead, so that transparent
> > > paging of objects in and out of memory can occur.
> > >
> > > To verify what kinds of persistents you have in memory, check out your
> > > #cachedObjectCountByClass.  You should have no more than one 19K tree
> > > worth of objects given that you are doing a garbageCollect after each
> > > one.
> > >
> > > If you would share the skeleton of your load script it also might shed
> > > some light.
> > >
> > > Thanks,
> > >   Chris
> > >
> > >
> > > On 8/6/07, Burella Juan M. <juan.burella at gmail.com> wrote:
> > > > I need to do a massive migration to Magma. My approach is currently
> like
> > > > this: I'm loading the data from external files, then build a complex
> > object
> > > > and commit it to a magma repository, and over again until the file
> ends.
> > > >
> > > > While doing this, my image was growing at very high rates, approx.
> 200Mb
> > and
> > > > taking several days, between 5 or 6. So, to see what's happening,
> I've
> > done
> > > > some tests with toy objects, each one of them has 5 iVars. and each
> one
> > has
> > > > another 5 iVars. This generates a tree of 19.556 objects
> > > > (5^0+5^1+5^2+5^3+5^4+5^5). Then I started to adding 50 of these
> "tree"
> > (or
> > > > smock?) objects, by groups of 5. I tried to maintain the complexity
> of
> > these
> > > > objects by adding "random" variable sized strings to the leafs and
> other
> > > > miscellaneous objets.
> > > > When this 50 are finalized, I've done:
> > > >
> > > > self magmaSession stubOut: aCollection. "aCollection has 50 smock
> > objects"
> > > > aCollection := nil.
> > > > Smalltalk garbageCollect.
> > > > self magmaSession finalizeOids.
> > > >
> > > > But still the image is increasing its size. At beggining of the test
> the
> > > > image size (as reported from #vmParameterAt: 3) was approx. 60Mb and
> at
> > the
> > > > end was approx. 89MB.
> > > >
> > > > Any suggestions for reducing the growing of the image size?
> > > >
> > > > Thanks in advance.
> > > > Juan Matias.
> > > >
> > > > _______________________________________________
> > > > Magma mailing list
> > > > Magma at lists.squeakfoundation.org
> > > >
> > http://lists.squeakfoundation.org/mailman/listinfo/magma
> > > >
> > > >
> > >
> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/magma/attachments/20070810/48e1b271/attachment.htm