Hi Chris, there´s no problem at all with adding the package code anywhere you want. Feel free to contact me for any suggestion.<br>From what I´ve seen from the sample code I sent to you, although the image size grows up, when I closed the repository and evaluated MagmaSession cleanUp, the image size goes back to the state before the evaluation.
<br>In the "real migration" what I told in the first mail still the size is growing at very high rates, we cannot detect yet what objects aren´t collected by the GC.<br> <br>Regards,<br>Juan Matías.<br><br><div>
<span class="gmail_quote">On 8/10/07, <b class="gmail_sendername">Chris Muller</b> <<a href="mailto:ma.chris.m@gmail.com">ma.chris.m@gmail.com</a>> wrote:</span><blockquote class="gmail_quote" style="margin-top: 0; margin-right: 0; margin-bottom: 0; margin-left: 0; margin-left: 0.80ex; border-left-color: #cccccc; border-left-width: 1px; border-left-style: solid; padding-left: 1ex">
Hi Juan, THANKS for the best test case code anyone has ever sent me<br>for asking a question about Magma..!<br><br>I have been too busy to get to it but am finally spending some time<br>with this tonight.<br><br>Do you mind if I harvest this for the Magma tester package? It might
<br>be a really great addition for benchmarking / profiling memory as well<br>overall performance.<br><br>Let me explore this; I'll write again in the next few days..<br><br>Regards,<br> Chris<br><br>On 8/7/07, Burella Juan M. <
<a href="mailto:juan.burella@gmail.com">juan.burella@gmail.com</a>> wrote:<br>> Hi Chris, Thanks for your answer.<br>> I've attached a monticello package with the tests, open a Transcript<br>> window and see class comments for instructions of how to run them.
<br>><br>> This is the Transcript output for 50 "trees" with 5 of depth:<br>><br>> Creating magma repository<br>> Opening magma session<br>> Image size before creating magma collection: 69851252
<br>> Creating magma collection at MagmaMemoryTest<br>> --Iteration: 1..5:<br>> Image size BEFORE creating test object: 69851252<br>> Creating test objects...<br>> Image size AFTER creating test object: 69851252
<br>> Saving 5 next objects from collection to magma collection...<br>> Image size BEFORE Magma cleaning: 85098444<br>> Image size AFTER Magma cleaning: 82623944<br>> --Iteration: 6..10:<br>> Image size BEFORE creating test object: 82623944
<br>> Creating test objects...<br>> Image size AFTER creating test object: 82623944<br>> Saving 5 next objects from collection to magma collection...<br>> Image size BEFORE Magma cleaning: 106407888<br>> Image size AFTER Magma cleaning: 83066628
<br>> --Iteration: 11..15:<br>> Image size BEFORE creating test object: 83066628<br>> Creating test objects...<br>> Image size AFTER creating test object: 83066628<br>> Saving 5 next objects from collection to magma collection...
<br>> Image size BEFORE Magma cleaning: 108570520<br>> Image size AFTER Magma cleaning: 85184212<br>> --Iteration: 16..20:<br>> Image size BEFORE creating test object: 85184212<br>> Creating test objects...
<br>> Image size AFTER creating test object: 85184212<br>> Saving 5 next objects from collection to magma collection...<br>> Image size BEFORE Magma cleaning: 108627824<br>> Image size AFTER Magma cleaning: 85212848
<br>> --Iteration: 21..25:<br>> Image size BEFORE creating test object: 85212848<br>> Creating test objects...<br>> Image size AFTER creating test object: 85212848<br>> Saving 5 next objects from collection to magma collection...
<br>> Image size BEFORE Magma cleaning: 108615528<br>> Image size AFTER Magma cleaning: 85216916<br>> --Iteration: 26..30:<br>> Image size BEFORE creating test object: 85216916<br>> Creating test objects...
<br>> Image size AFTER creating test object: 85216916<br>> Saving 5 next objects from collection to magma collection...<br>> Image size BEFORE Magma cleaning: 108787536<br>> Image size AFTER Magma cleaning: 85257868
<br>> --Iteration: 31..35:<br>> Image size BEFORE creating test object: 85257868<br>> Creating test objects...<br>> Image size AFTER creating test object: 85257868<br>> Saving 5 next objects from collection to magma collection...
<br>> Image size BEFORE Magma cleaning: 108857164<br>> Image size AFTER Magma cleaning: 85307012<br>> --Iteration: 36..40:<br>> Image size BEFORE creating test object: 85307012<br>> Creating test objects...
<br>> Image size AFTER creating test object: 85307012<br>> Saving 5 next objects from collection to magma collection...<br>> Image size BEFORE Magma cleaning: 108623676<br>> Image size AFTER Magma cleaning: 85257812
<br>> --Iteration: 41..45:<br>> Image size BEFORE creating test object: 85257812<br>> Creating test objects...<br>> Image size AFTER creating test object: 85257812<br>> Saving 5 next objects from collection to magma collection...
<br>> Image size BEFORE Magma cleaning: 108390116<br>> Image size AFTER Magma cleaning: 85159312<br>> --Iteration: 46..50:<br>> Image size BEFORE creating test object: 85159312<br>> Creating test objects...
<br>> Image size AFTER creating test object: 85159312<br>> Saving 5 next objects from collection to magma collection...<br>> Image size BEFORE Magma cleaning: 108594588<br>> Image size AFTER Magma cleaning: 85183668
<br>><br>><br>> On 8/7/07, Chris Muller <<a href="mailto:asqueaker@gmail.com">asqueaker@gmail.com</a>> wrote:<br>> > HI Juan,<br>> ><br>> > I really need to add a "How to do a Bulk Load" page to the documentation,
<br>> sorry.<br>> ><br>> > In the meantime, you weren't clear what you were adding the 19K trees<br>> > of objects *TO*. My guess is you are adding to a Smalltlak<br>> > collection; a Dictionary or OrderedCollection perhaps? It's good you
<br>> > are doing the stubbing and finalizieOids, but they won't help much if<br>> > the root collection is a Smalltalk collection because the entire<br>> > persistent graph is in memory then (except for the stubbed, maybe).
<br>> ><br>> > You need to add them to a MagmaCollection instead, so that transparent<br>> > paging of objects in and out of memory can occur.<br>> ><br>> > To verify what kinds of persistents you have in memory, check out your
<br>> > #cachedObjectCountByClass. You should have no more than one 19K tree<br>> > worth of objects given that you are doing a garbageCollect after each<br>> > one.<br>> ><br>> > If you would share the skeleton of your load script it also might shed
<br>> > some light.<br>> ><br>> > Thanks,<br>> > Chris<br>> ><br>> ><br>> > On 8/6/07, Burella Juan M. <<a href="mailto:juan.burella@gmail.com">juan.burella@gmail.com</a>> wrote:
<br>> > > I need to do a massive migration to Magma. My approach is currently like<br>> > > this: I'm loading the data from external files, then build a complex<br>> object<br>> > > and commit it to a magma repository, and over again until the file ends.
<br>> > ><br>> > > While doing this, my image was growing at very high rates, approx. 200Mb<br>> and<br>> > > taking several days, between 5 or 6. So, to see what's happening, I've<br>
> done<br>> > > some tests with toy objects, each one of them has 5 iVars. and each one<br>> has<br>> > > another 5 iVars. This generates a tree of 19.556 objects<br>> > > (5^0+5^1+5^2+5^3+5^4+5^5). Then I started to adding 50 of these "tree"
<br>> (or<br>> > > smock?) objects, by groups of 5. I tried to maintain the complexity of<br>> these<br>> > > objects by adding "random" variable sized strings to the leafs and other<br>> > > miscellaneous objets.
<br>> > > When this 50 are finalized, I've done:<br>> > ><br>> > > self magmaSession stubOut: aCollection. "aCollection has 50 smock<br>> objects"<br>> > > aCollection := nil.
<br>> > > Smalltalk garbageCollect.<br>> > > self magmaSession finalizeOids.<br>> > ><br>> > > But still the image is increasing its size. At beggining of the test the<br>> > > image size (as reported from #vmParameterAt: 3) was approx. 60Mb and at
<br>> the<br>> > > end was approx. 89MB.<br>> > ><br>> > > Any suggestions for reducing the growing of the image size?<br>> > ><br>> > > Thanks in advance.<br>> > > Juan Matias.
<br>> > ><br>> > > _______________________________________________<br>> > > Magma mailing list<br>> > > <a href="mailto:Magma@lists.squeakfoundation.org">Magma@lists.squeakfoundation.org
</a><br>> > ><br>> <a href="http://lists.squeakfoundation.org/mailman/listinfo/magma">http://lists.squeakfoundation.org/mailman/listinfo/magma</a><br>> > ><br>> > ><br>> ><br>><br>>
<br>><br></blockquote></div><br>