Hi Chris, all,
I've been reading some Gjallar code to see how they use Magma. I saw the a main object which is tha application model and it has 4 or 5 MagmaCollections in it's instVars.
So I in a load test I've made a model object with a magma collection with 10000 person objects. It performed nicely.
Then I pushed that by making this person object to store in one of it's instVars #acts and set up there a magma collection and added 20 acts objects to every person. So it should end up with 10K persons having one magma collection with 20 act objects each.
What I've got is a serializer security exception MaObjectSerializationSecurityViolation saying that one can't serialize objects of more than 10 megabytes.
I've read about starting small with Magma but I need a glimpse on how the model I'm thinking on support with Magma will scale. From this test it seems that Magma collections supports lots of objects but any common object suporting even a low number of magma collections that will be growing will easily reach more that 10 megabytes when serialized. So I'm missing something? There is a more appropiate use of Magma for a model like that?
Same question in other scenario: the main object of the domain model the root of the application domain from where all is accessed by reachability will also suffer this limitation. What one can do at that point?
thanks,
Sebastian Sastre
Hi Sebastian,
Magma can go plenty big, but the individual transactions which build it up must, themselves, remain under 10meg each. This is an arbitrary limit that you could change, but it is really more polite to other users not to because it ensures the servers time is sliced into smaller pieces so other users can have a chance to get their work done too.
What you should do is commit every 100 Persons with 20 Acts each instead of all 10000 at once.
Now, when you do this, be sure to discard the Persons you've already committed from memory, otherwise you will be accumulating a larger and larger portion of the now-persistent model in your local client, which will bog down subsequent commits. See the performance tuning tips on the Magma documentation pages for more information.
http://wiki.squeak.org/squeak/2985
I've read about starting small with Magma but I need a glimpse on how
the model I'm thinking on support with Magma will scale. From this test it seems that Magma collections supports lots of objects but any common object suporting even a low number of magma collections that will be growing will easily reach more that 10 megabytes when serialized. So I'm missing something?
Yes. MagmaCollections are tiny for the serializer, even if they have a billion objects in them. The serializer only serializes new or changed objects, so you do need to do them a smaller-than-10-meg piece at a time.
Same question in other scenario: the main object of the domain model the
root of the application domain from where all is accessed by reachability will also suffer this limitation. What one can do at that point?
The serializer only serializes what is new or changed which is completely independent of the overall size of the persistent model.
Regards, Chris
On 10/20/07, Sebastian Sastre ssastre@seaswork.com wrote:
Hi Chris, all,
I've been reading some Gjallar code to see how they use Magma. I saw the
a main object which is tha application model and it has 4 or 5 MagmaCollections in it's instVars.
So I in a load test I've made a model object with a magma collection
with 10000 person objects. It performed nicely.
Then I pushed that by making this person object to store in one of it's
instVars #acts and set up there a magma collection and added 20 acts objects to every person. So it should end up with 10K persons having one magma collection with 20 act objects each.
What I've got is a serializer security exception
MaObjectSerializationSecurityViolation saying that one can't serialize objects of more than 10 megabytes.
I've read about starting small with Magma but I need a glimpse on how
the model I'm thinking on support with Magma will scale. From this test it seems that Magma collections supports lots of objects but any common object suporting even a low number of magma collections that will be growing will easily reach more that 10 megabytes when serialized. So I'm missing something? There is a more appropiate use of Magma for a model like that?
Same question in other scenario: the main object of the domain model the
root of the application domain from where all is accessed by reachability will also suffer this limitation. What one can do at that point?
thanks,
Sebastian Sastre
Magma mailing list Magma@lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma
Thanks for patiently educating me about this details Chris. Indeed making transactions to commit in that medium-small size is the way to go (a very reasonable tip for any transactional system by the way).
The part I have to understood better is how I will make that discard of objects you mention to keep image not that hungry of RAM. I looked Gjallar use of #stubOut: to see an usage reference of it but hasn't senders. What are the best practices of using #stubOut:? I mean, I imagine what to do when one has to traversal a big model already loaded in magma discarding parts as the process goes ahead but what happen if one are creating that model like in this load test? Of course this test are not realistic usability because the real use will make lots of small changes making the model grow at ant step. But supose you have an application that instantiates initially (on first user hit for instance) making a magma repository to have a significatively big standard repository and once created that small changes will be made by the users. What if that instantiation of the model in magma makes squeak go beyond 200 megabytes? As I saw it crash systematically about 250 - 300 megas I wonder how to keep things below 200.
thanks again,
Sebastian Sastre
-----Mensaje original----- De: Chris Muller [mailto:asqueaker@gmail.com] Enviado el: Sábado, 20 de Octubre de 2007 12:45 Para: Sebastian Sastre CC: magma@lists.squeakfoundation.org Asunto: Re: Saving big objects?
Hi Sebastian,
Magma can go plenty big, but the individual transactions which build it up must, themselves, remain under 10meg each. This is an arbitrary limit that you could change, but it is really more polite to other users not to because it ensures the servers time is sliced into smaller pieces so other users can have a chance to get their work done too.
What you should do is commit every 100 Persons with 20 Acts each instead of all 10000 at once.
Now, when you do this, be sure to discard the Persons you've already committed from memory, otherwise you will be accumulating a larger and larger portion of the now-persistent model in your local client, which will bog down subsequent commits. See the performance tuning tips on the Magma documentation pages for more information.
http://wiki.squeak.org/squeak/2985
I've read about starting small with Magma but I need a
glimpse on
how the model I'm thinking on support with Magma will
scale. From this
test it seems that Magma collections supports lots of
objects but any
common object suporting even a low number of magma collections that will be growing will easily reach more that 10 megabytes when serialized. So I'm missing something?
Yes. MagmaCollections are tiny for the serializer, even if they have a billion objects in them. The serializer only serializes new or changed objects, so you do need to do them a smaller-than-10-meg piece at a time.
Same question in other scenario: the main object of the domain
model the root of the application domain from where all is
accessed by
reachability will also suffer this limitation. What one can
do at that point?
The serializer only serializes what is new or changed which is completely independent of the overall size of the persistent model.
Regards, Chris
On 10/20/07, Sebastian Sastre ssastre@seaswork.com wrote:
Hi Chris, all,
I've been reading some Gjallar code to see how they use
Magma. I
saw the a main object which is tha application model and it
has 4 or 5
MagmaCollections in it's instVars.
So I in a load test I've made a model object with a magma
collection with 10000 person objects. It performed nicely.
Then I pushed that by making this person object to
store in one of
it's instVars #acts and set up there a magma collection and
added 20
acts objects to every person. So it should end up with 10K persons having one magma collection with 20 act objects each.
What I've got is a serializer security exception
MaObjectSerializationSecurityViolation saying that one
can't serialize
objects of more than 10 megabytes.
I've read about starting small with Magma but I need a
glimpse on
how the model I'm thinking on support with Magma will
scale. From this
test it seems that Magma collections supports lots of
objects but any
common object suporting even a low number of magma collections that will be growing will easily reach more that 10 megabytes when serialized. So I'm missing something? There is a more
appropiate use of Magma for a model like that?
Same question in other scenario: the main object of the domain
model the root of the application domain from where all is
accessed by
reachability will also suffer this limitation. What one can
do at that point?
thanks,
Sebastian Sastre
Magma mailing list Magma@lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma
Hi Sebastian, #stubOut: can be useful for chopping off large branches of objects.
The only objects you should keep in memory are ones you actively need to send messages to. For a bulk load, this is nothing more than the next batch about to be committed. You should never have a 100meg image in that scenario; not even 50 meg for that matter.
The MagmaCollection you're loading will only keep a "page" (125 objects, I think) in memory at a time. So it's ok to reference a large MagmaCollection in memory.
Hope this helps. If you need help with an example, please feel free to post it here.
- Chris
On 10/22/07, Sebastian Sastre ssastre@seaswork.com wrote:
Thanks for patiently educating me about this details Chris. Indeed making transactions to commit in that medium-small size is the way to go (a very reasonable tip for any transactional system by the way).
The part I have to understood better is how I will make that discard of objects you mention to keep image not that hungry of RAM. I looked Gjallar use of #stubOut: to see an usage reference of it but hasn't senders. What are the best practices of using #stubOut:? I mean, I imagine what to do when one has to traversal a big model already loaded in magma discarding parts as the process goes ahead but what happen if one are creating that model like in this load test? Of course this test are not realistic usability because the real use will make lots of small changes making the model grow at ant step. But supose you have an application that instantiates initially (on first user hit for instance) making a magma repository to have a significatively big standard repository and once created that small changes will be made by the users. What if that instantiation of the model in magma makes squeak go beyond 200 megabytes? As I saw it crash systematically about 250 - 300 megas I wonder how to keep things below 200.
thanks again,
Sebastian Sastre
-----Mensaje original----- De: Chris Muller [mailto:asqueaker@gmail.com] Enviado el: Sábado, 20 de Octubre de 2007 12:45 Para: Sebastian Sastre CC: magma@lists.squeakfoundation.org Asunto: Re: Saving big objects?
Hi Sebastian,
Magma can go plenty big, but the individual transactions which build it up must, themselves, remain under 10meg each. This is an arbitrary limit that you could change, but it is really more polite to other users not to because it ensures the servers time is sliced into smaller pieces so other users can have a chance to get their work done too.
What you should do is commit every 100 Persons with 20 Acts each instead of all 10000 at once.
Now, when you do this, be sure to discard the Persons you've already committed from memory, otherwise you will be accumulating a larger and larger portion of the now-persistent model in your local client, which will bog down subsequent commits. See the performance tuning tips on the Magma documentation pages for more information.
http://wiki.squeak.org/squeak/2985
I've read about starting small with Magma but I need a
glimpse on
how the model I'm thinking on support with Magma will
scale. From this
test it seems that Magma collections supports lots of
objects but any
common object suporting even a low number of magma collections that will be growing will easily reach more that 10 megabytes when serialized. So I'm missing something?
Yes. MagmaCollections are tiny for the serializer, even if they have a billion objects in them. The serializer only serializes new or changed objects, so you do need to do them a smaller-than-10-meg piece at a time.
Same question in other scenario: the main object of the domain
model the root of the application domain from where all is
accessed by
reachability will also suffer this limitation. What one can
do at that point?
The serializer only serializes what is new or changed which is completely independent of the overall size of the persistent model.
Regards, Chris
On 10/20/07, Sebastian Sastre ssastre@seaswork.com wrote:
Hi Chris, all,
I've been reading some Gjallar code to see how they use
Magma. I
saw the a main object which is tha application model and it
has 4 or 5
MagmaCollections in it's instVars.
So I in a load test I've made a model object with a magma
collection with 10000 person objects. It performed nicely.
Then I pushed that by making this person object to
store in one of
it's instVars #acts and set up there a magma collection and
added 20
acts objects to every person. So it should end up with 10K persons having one magma collection with 20 act objects each.
What I've got is a serializer security exception
MaObjectSerializationSecurityViolation saying that one
can't serialize
objects of more than 10 megabytes.
I've read about starting small with Magma but I need a
glimpse on
how the model I'm thinking on support with Magma will
scale. From this
test it seems that Magma collections supports lots of
objects but any
common object suporting even a low number of magma collections that will be growing will easily reach more that 10 megabytes when serialized. So I'm missing something? There is a more
appropiate use of Magma for a model like that?
Same question in other scenario: the main object of the domain
model the root of the application domain from where all is
accessed by
reachability will also suffer this limitation. What one can
do at that point?
thanks,
Sebastian Sastre
Magma mailing list Magma@lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma
Magma mailing list Magma@lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/magma
magma@lists.squeakfoundation.org