Saving big objects?

Wed Oct 24 01:51:40 UTC 2007

Hi Sebastian, #stubOut: can be useful for chopping off large branches
of objects.

The only objects you should keep in memory are ones you actively need
to send messages to.  For a bulk load, this is nothing more than the
next batch about to be committed.  You should never have a 100meg
image in that scenario; not even 50 meg for that matter.

The MagmaCollection you're loading will only keep a "page" (125
objects, I think) in memory at a time.  So it's ok to reference a
large MagmaCollection in memory.

Hope this helps.  If you need help with an example, please feel free
to post it here.

 - Chris

On 10/22/07, Sebastian Sastre <ssastre at seaswork.com> wrote:
> Thanks for patiently educating me about this details Chris. Indeed making
> transactions to commit in that medium-small size is the way to go (a very
> reasonable tip for any transactional system by the way).
>
> The part I have to understood better is how I will make that discard of
> objects you mention to keep image not that hungry of RAM. I looked Gjallar
> use of #stubOut: to see an usage reference of it but hasn't senders. What
> are the best practices of using #stubOut:? I mean, I imagine what to do when
> one has to traversal a big model already loaded in magma discarding parts as
> the process goes ahead but what happen if one are creating that model like
> in this load test? Of course this test are not realistic usability because
> the real use will make lots of small changes making the model grow at ant
> step. But supose you have an application that instantiates initially (on
> first user hit for instance) making a magma repository to have a
> significatively big standard repository and once created that small changes
> will be made by the users. What if that instantiation of the model in magma
> makes squeak go beyond 200 megabytes? As I saw it crash systematically about
> 250 - 300       megas I wonder how to keep things below 200.
>
>         thanks again,
>
> Sebastian Sastre
>
>
> > -----Mensaje original-----
> > De: Chris Muller [mailto:asqueaker at gmail.com]
> > Enviado el: Sábado, 20 de Octubre de 2007 12:45
> > Para: Sebastian Sastre
> > CC: magma at lists.squeakfoundation.org
> > Asunto: Re: Saving big objects?
> >
> > Hi Sebastian,
> >
> > Magma can go plenty big, but the individual transactions
> > which build it up must, themselves, remain under 10meg each.
> > This is an arbitrary limit that you could change, but it is
> > really more polite to other users not to because it ensures
> > the servers time is sliced into smaller pieces so other users
> > can have a chance to get their work done too.
> >
> > What you should do is commit every 100 Persons with 20 Acts
> > each instead of all 10000 at once.
> >
> > Now, when you do this, be sure to discard the Persons you've
> > already committed from memory, otherwise you will be
> > accumulating a larger and larger portion of the
> > now-persistent model in your local client, which will bog
> > down subsequent commits.  See the performance tuning tips on
> > the Magma documentation pages for more information.
> >
> >   http://wiki.squeak.org/squeak/2985
> >
> > >     I've read about starting small with Magma but I need a
> > glimpse on
> > > how the model I'm thinking on support with Magma will
> > scale. From this
> > > test it seems that Magma collections supports lots of
> > objects but any
> > > common object suporting even a low number of magma collections that
> > > will be growing will easily reach more that 10 megabytes when
> > > serialized. So I'm missing something?
> >
> > Yes.  MagmaCollections are tiny for the serializer, even if
> > they have a billion objects in them.  The serializer only
> > serializes new or changed objects, so you do need to do them
> > a smaller-than-10-meg piece at a time.
> >
> > >     Same question in other scenario: the main object of the domain
> > > model the root of the application domain from where all is
> > accessed by
> > > reachability will also suffer this limitation. What one can
> > do at that point?
> >
> > The serializer only serializes what is new or changed which
> > is completely independent of the overall size of the persistent model.
> >
> > Regards,
> >   Chris
> >
> > On 10/20/07, Sebastian Sastre <ssastre at seaswork.com> wrote:
> > >
> > >
> > > Hi Chris, all,
> > >
> > >     I've been reading some Gjallar code to see how they use
> > Magma. I
> > > saw the a main object which is tha application model and it
> > has 4 or 5
> > > MagmaCollections in it's instVars.
> > >
> > >     So I in a load test I've made a model object with a magma
> > > collection with 10000 person objects. It performed nicely.
> > >
> > >     Then I pushed that by making this person object to
> > store in one of
> > > it's instVars #acts and set up there a magma collection and
> > added 20
> > > acts objects to every person. So it should end up with 10K persons
> > > having one magma collection with 20 act objects each.
> > >
> > >     What I've got is a serializer security exception
> > > MaObjectSerializationSecurityViolation saying that one
> > can't serialize
> > > objects of more than 10 megabytes.
> > >
> > >     I've read about starting small with Magma but I need a
> > glimpse on
> > > how the model I'm thinking on support with Magma will
> > scale. From this
> > > test it seems that Magma collections supports lots of
> > objects but any
> > > common object suporting even a low number of magma collections that
> > > will be growing will easily reach more that 10 megabytes when
> > > serialized. So I'm missing something? There is a more
> > appropiate use of Magma for a model like that?
> > >
> > >     Same question in other scenario: the main object of the domain
> > > model the root of the application domain from where all is
> > accessed by
> > > reachability will also suffer this limitation. What one can
> > do at that point?
> > >
> > >     thanks,
> > >
> > >
> > >
> > > Sebastian Sastre
> > >
> > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Magma mailing list
> > > Magma at lists.squeakfoundation.org
> > > http://lists.squeakfoundation.org/mailman/listinfo/magma
> > >
> > >
>
> _______________________________________________
> Magma mailing list
> Magma at lists.squeakfoundation.org
> http://lists.squeakfoundation.org/mailman/listinfo/magma
>