magma handle memory

Sat Aug 11 00:50:11 UTC 2007

Juan, I spent some time running your neat test case but it didn't show
me any unreasonable growth.  So I then went back to the output you
posted and looked at it in detail.

Regarding the output you pasted in your previous message:

=======================================
Creating magma repository
Opening magma session
Image size before creating magma collection: 69851252
Creating magma collection at MagmaMemoryTest
--Iteration: 1..5:
Image size BEFORE creating test object: 69851252
Creating test objects...
Image size AFTER creating test object: 69851252
Saving 5 next objects from collection to magma collection...
Image size BEFORE Magma cleaning: 85098444
Image size AFTER Magma cleaning: 82623944
--Iteration: 6..10:
Image size BEFORE creating test object: 82623944
Creating test objects...
Image size AFTER creating test object: 82623944
Saving 5 next objects from collection to magma collection...
Image size BEFORE Magma cleaning: 106407888
Image size AFTER Magma cleaning: 83066628
    ...
--Iteration: 46..50:
Image size BEFORE creating test object: 85159312
Creating test objects...
Image size AFTER creating test object: 85159312
Saving 5 next objects from collection to magma collection...
Image size BEFORE Magma cleaning: 108594588
Image size AFTER Magma cleaning: 85183668
==================================

End of memory at the beginning shows:

--Iteration: 6..10:
Image size BEFORE creating test object: 82623944

at the end:

Image size AFTER Magma cleaning: 85183668

85183668 -82623944      =  2559724

So, end-of-memory, through all of the bulk-load operations, needed to
advance only 2.5MB through dynamic allocation, etc.  Is this
unreasonable?

Further, your output indicates there was only initial mild growth from
the first three iterations.  Then, beginning with --Iteration:
16..20:, it stayed relatively static at roughly 85MB.

Am I missing something?

Regards,
  Chris

On 8/10/07, Burella Juan M. <juan.burella at gmail.com> wrote:
> Hi Chris, there´s no problem at all with adding the package code anywhere
> you want. Feel free to contact me for any suggestion.
> From what I´ve seen from the sample code I sent to you, although the image
> size grows up, when I closed the repository and evaluated MagmaSession
> cleanUp, the image size goes back to the state before the evaluation.
> In the "real migration" what I told in the first mail still the size is
> growing at very high rates, we cannot
> detect yet what objects aren´t collected by the GC.
>
> Regards,
> Juan Matías.
>
>  On 8/10/07, Chris Muller <ma.chris.m at gmail.com> wrote:
> > Hi Juan, THANKS for the best test case code anyone has ever sent me
> > for asking a question about Magma..!
> >
> > I have been too busy to get to it but am finally spending some time
> > with this tonight.
> >
> > Do you mind if I harvest this for the Magma tester package?  It might
> > be a really great addition for benchmarking / profiling memory as well
> > overall performance.
> >
> > Let me explore this; I'll write again in the next few days..
> >
> > Regards,
> >   Chris
> >
> >
> > On 8/7/07, Burella Juan M. < juan.burella at gmail.com> wrote:
> > > Hi Chris, Thanks for your answer.
> > >      I've attached a monticello package with the tests, open a
> Transcript
> > > window and see class comments for instructions of how to run them.
> > >
> > > This is the Transcript output for 50 "trees" with 5 of depth:
> > >
> > > Creating magma repository
> > > Opening magma session
> > > Image size before creating magma collection: 69851252
> > > Creating magma collection at MagmaMemoryTest
> > > --Iteration: 1..5:
> > > Image size BEFORE creating test object: 69851252
> > > Creating test objects...
> > > Image size AFTER creating test object: 69851252
> > > Saving 5 next objects from collection to magma collection...
> > > Image size BEFORE Magma cleaning: 85098444
> > > Image size AFTER Magma cleaning: 82623944
> > > --Iteration: 6..10:
> > > Image size BEFORE creating test object: 82623944
> > > Creating test objects...
> > > Image size AFTER creating test object: 82623944
> > > Saving 5 next objects from collection to magma collection...
> > >  Image size BEFORE Magma cleaning: 106407888
> > > Image size AFTER Magma cleaning: 83066628
> > > --Iteration: 11..15:
> > > Image size BEFORE creating test object: 83066628
> > > Creating test objects...
> > > Image size AFTER creating test object: 83066628
> > > Saving 5 next objects from collection to magma collection...
> > > Image size BEFORE Magma cleaning: 108570520
> > > Image size AFTER Magma cleaning: 85184212
> > > --Iteration: 16..20:
> > > Image size BEFORE creating test object: 85184212
> > > Creating test objects...
> > > Image size AFTER creating test object: 85184212
> > > Saving 5 next objects from collection to magma collection...
> > > Image size BEFORE Magma cleaning: 108627824
> > > Image size AFTER Magma cleaning: 85212848
> > > --Iteration: 21..25:
> > > Image size BEFORE creating test object: 85212848
> > > Creating test objects...
> > > Image size AFTER creating test object: 85212848
> > > Saving 5 next objects from collection to magma collection...
> > >  Image size BEFORE Magma cleaning: 108615528
> > > Image size AFTER Magma cleaning: 85216916
> > > --Iteration: 26..30:
> > > Image size BEFORE creating test object: 85216916
> > > Creating test objects...
> > > Image size AFTER creating test object: 85216916
> > > Saving 5 next objects from collection to magma collection...
> > > Image size BEFORE Magma cleaning: 108787536
> > > Image size AFTER Magma cleaning: 85257868
> > > --Iteration: 31..35:
> > > Image size BEFORE creating test object: 85257868
> > > Creating test objects...
> > > Image size AFTER creating test object: 85257868
> > > Saving 5 next objects from collection to magma collection...
> > > Image size BEFORE Magma cleaning: 108857164
> > > Image size AFTER Magma cleaning: 85307012
> > > --Iteration: 36..40:
> > > Image size BEFORE creating test object: 85307012
> > > Creating test objects...
> > > Image size AFTER creating test object: 85307012
> > > Saving 5 next objects from collection to magma collection...
> > >  Image size BEFORE Magma cleaning: 108623676
> > > Image size AFTER Magma cleaning: 85257812
> > > --Iteration: 41..45:
> > > Image size BEFORE creating test object: 85257812
> > > Creating test objects...
> > > Image size AFTER creating test object: 85257812
> > > Saving 5 next objects from collection to magma collection...
> > > Image size BEFORE Magma cleaning: 108390116
> > > Image size AFTER Magma cleaning: 85159312
> > > --Iteration: 46..50:
> > > Image size BEFORE creating test object: 85159312
> > > Creating test objects...
> > > Image size AFTER creating test object: 85159312
> > > Saving 5 next objects from collection to magma collection...
> > > Image size BEFORE Magma cleaning: 108594588
> > > Image size AFTER Magma cleaning: 85183668
> > >
> > >
> > > On 8/7/07, Chris Muller <asqueaker at gmail.com> wrote:
> >
> > > > HI Juan,
> > > >
> > > > I really need to add a "How to do a Bulk Load" page to the
> documentation,
> > > sorry.
> > > >
> > > > In the meantime, you weren't clear what you were adding the 19K trees
> > > > of objects *TO*.  My guess is you are adding to a Smalltlak
> > > > collection; a Dictionary or OrderedCollection perhaps?  It's good you
> > > > are doing the stubbing and finalizieOids, but they won't help much if
> > > > the root collection is a Smalltalk collection because the entire
> > > > persistent graph is in memory then (except for the stubbed, maybe).
> > > >
> > > > You need to add them to a MagmaCollection instead, so that transparent
> > > > paging of objects in and out of memory can occur.
> > > >
> > > > To verify what kinds of persistents you have in memory, check out your
> > > > #cachedObjectCountByClass.  You should have no more than one 19K tree
> > > > worth of objects given that you are doing a garbageCollect after each
> > > > one.
> > > >
> > > > If you would share the skeleton of your load script it also might shed
> > > > some light.
> > > >
> > > > Thanks,
> > > >   Chris
> > > >
> > > >
> > > > On 8/6/07, Burella Juan M. <juan.burella at gmail.com> wrote:
> > > > > I need to do a massive migration to Magma. My approach is currently
> like
> > > > > this: I'm loading the data from external files, then build a complex
> > > object
> > > > > and commit it to a magma repository, and over again until the file
> ends.
> > > > >
> > > > > While doing this, my image was growing at very high rates, approx.
> 200Mb
> > > and
> > > > > taking several days, between 5 or 6. So, to see what's happening,
> I've
> > > done
> > > > > some tests with toy objects, each one of them has 5 iVars. and each
> one
> > > has
> > > > > another 5 iVars. This generates a tree of 19.556 objects
> > > > > (5^0+5^1+5^2+5^3+5^4+5^5). Then I started to adding 50 of these
> "tree"
> > > (or
> > > > > smock?) objects, by groups of 5. I tried to maintain the complexity
> of
> > > these
> > > > > objects by adding "random" variable sized strings to the leafs and
> other
> > > > > miscellaneous objets.
> > > > > When this 50 are finalized, I've done:
> > > > >
> > > > > self magmaSession stubOut: aCollection. "aCollection has 50 smock
> > > objects"
> > > > > aCollection := nil.
> > > > > Smalltalk garbageCollect.
> > > > > self magmaSession finalizeOids.
> > > > >
> > > > > But still the image is increasing its size. At beggining of the test
> the
> > > > > image size (as reported from #vmParameterAt: 3) was approx. 60Mb and
> at
> > > the
> > > > > end was approx. 89MB.
> > > > >
> > > > > Any suggestions for reducing the growing of the image size?
> > > > >
> > > > > Thanks in advance.
> > > > > Juan Matias.
> > > > >
> > > > > _______________________________________________
> > > > > Magma mailing list
> > > > > Magma at lists.squeakfoundation.org
> > > > >
> > >
> http://lists.squeakfoundation.org/mailman/listinfo/magma
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> >
>
>
> _______________________________________________
> Magma mailing list
> Magma at lists.squeakfoundation.org
> http://lists.squeakfoundation.org/mailman/listinfo/magma
>
>