Hi Ramiro,
I hope you don't mind my copying the Magma list since this is a useful discussion for everyone.
== Scalability ==
In the context of web applications there are two tiers of scaling to more users. First, use multiple client sessions across different images all accessing a server in its own image. Second, serve the domain from multiple Magma servers. Let me explain in order.
-- Tier 1, Multiple clients --
First, you configure to run multiple MagmaClient sessions. What you do is have each web session have its own MagmaSession view of the repository, and use ReadStrategy's and #stubOut: to keep only enough in memory for the current working web page. This was proposed years ago here:
http://wiki.squeak.org/squeak/5817
and I think Brent may have successfully implemented the multi-image configuration described on that page under "Remote Connections". Goran and I have discussed it in the past, but I don't know whether he ever tried it in earnest. If so, I hope he will comment about this.
That is the first tier. Before describing the second tier, let me first remind that the design of Magma is such that the client does 80% of the heavy-lifting; a remote server will mostly sit idle, starving for requests to process until you get around 10 (or more, depending on lots of factors), busy clients.
Until then, it doesn't do any good to do move to the second tier (multiple servers) until your clients can swamp the remote server, causing it to begin to queue the requests it receives from the multiple client sessions, making them wait.
-- Tier 2, multiple servers --
When you get to this point, you can then begin to break up the server duties across multiple repositories in several ways.
1) Partition your domain by its major categories and use MagmaForwardingProxy's to link to objects across repositories. For more information:
http://wiki.squeak.org/squeak/5604
I think this is a smart thing to do from the beginning simply for organizational purposes, but it also helps performance by dividing up the server burden across multiple images/computers.
This feature works and has been around for years, although I don't know anyone who has used it (because no one has ever asked about it).
2) Many domains have information that becomes immutable after it is created. For example, transactions in an accounting system, messages in a communication system, digitized external artifacts, etc.
Making one or more separate "warm backups" provides a mostly up-to-date copy of your repository that are on-line for, for example, user queries and reporiting. This can alleviate considerable burden from the main repository, which is concerned with the still-dynamic portion of the persistent model. For more information:
http://wiki.squeak.org/squeak/5724
== Faster? ==
Where did you hear the new version is much faster? Perhaps you are referring to Sig's new Dictionary's that sped things up? That may be the case but I haven't had time to look at them yet. Thanks to Sig, Magma now includes the ability to specify whatever WeakIdentityKeyDictionary you want (see MagmaPreferences), so you can try out some other Dictionary's but I can't promise I can support them (or a Magma implementation that uses them).
They need to be tested. The first step would be to run the entire test suite with the replacement Dictionary(s). If those pass then you can be pretty confident, but I would burn them in in a heavily-used test environment for a good month first.
Beyond this, Magma r40 is not any faster than 39, except for single-clause query's, which are 100X faster due to a bug fix.
== 3.10 ==
The official 3.10 is supported, but I have not tried the 3.10 "Development image" yet. Unless Damien has changed low-level stuff there shouldn't be any problem. Again, the way to answer this question for yourself is to do what I would do to give you the answer; run the MagmaTester test suite. Documentation on how to do this is spelled out on the Magma home page on the Swiki.
http://wiki.squeak.org/squeak/2661
I'll bet it will be fine, but if you encounter a problem, please let me know because I do intend to support 3.10.
- Chris - Hide quoted text -
On 8/29/07, Ramiro Diaz Trepat ramirodt@gmail.com wrote:
---------- Forwarded message ---------- From: Ramiro Diaz Trepat ramirodt@gmail.com Date: Aug 28, 2007 6:24 PM Subject: Magma To: Chris Müller chris@funkyobjects.org
Hello Chris, I am happy you are about to release a new Magma. I wanted to ask you a couple of questions. Is there a way to make Magma scale, on a cluster with different servers? In case of course one had a lot of queries from a very active web site for instance. Is there a way to have several servers running on the same database files or with replication? Will it run on Damien Cassous' version of Squeak 3.10? I've heared that the new version is much faster, do you have some figures? Thanks
== Faster? ==
Where did you hear the new version is much faster? Perhaps you are referring to Sig's new Dictionary's that sped things up? That may be the case but I haven't had time to look at them yet. Thanks to Sig, Magma now includes the ability to specify whatever WeakIdentityKeyDictionary you want (see MagmaPreferences), so you can try out some other Dictionary's but I can't promise I can support them (or a Magma implementation that uses them).
About preferences, I missed another type of dictionary which used as well: WeakValueDictionary. I think we need to put an option for choosing its class to MagmaPreferences too. Or remove both. Want me to send you patch? And btw, can you add me as dev to magma-tester repository, so i'll not ask Keith every time i need to upload new snapshot. I registered as 'sig' on squeaksource.
They need to be tested. The first step would be to run the entire test suite with the replacement Dictionary(s). If those pass then you can be pretty confident, but I would burn them in in a heavily-used test environment for a good month first.
I am totally agree with you that dictionaries need to be heavily tested before put in use. Sadly i didn't have a time yet to test them with Magma tester suite. But instead, i tested them on Florian's real-world application , and as you maybe know there was only single issue - they are too fast :) . Something, running in parrallel going out of sync. After placing a delay in #reject: all works fine. I'm still curious what you thinking about it, what can interfere with dictionary or in what ways this can be avoided (putting delay is not a solution of course).
I designed them mostly for use with magma (even though they can be a good replacement of default dictionaries when you using #removeKey: frequently).
Beyond this, Magma r40 is not any faster than 39, except for single-clause query's, which are 100X faster due to a bug fix.
Hi Sig,
Want me to send you patch? And btw, can you add me as dev to magma-tester repository, so i'll not ask Keith every time i need to upload new snapshot. I registered as 'sig' on squeaksource.
Done. Please post your patches there and send me an e-mail, I'll integrate them as appropriate.
Something, running in parrallel going out of sync. After placing a delay in #reject: all works fine. I'm still curious what you thinking about it, what can interfere with dictionary or in what ways this can be avoided (putting delay is not a solution of course).
I don't think about it. :) I have not had time to get into Dictionary implementations, so I have appreciated your help looking into it. In the end, I just want to plug in whatever fastest Dictionary library that proves workable for Magma.
- Chris
I don't think about it. :) I have not had time to get into Dictionary implementations, so I have appreciated your help looking into it. In the end, I just want to plug in whatever fastest Dictionary library that proves workable for Magma.
If someone is looking into dictionaries, please also talk to the list about the MagmaDictionary class for VERY large dictionaries.
Chris had my implementation in sometime back, but it needs looking at again.
Brent
Hi!
"Chris Muller" ma.chris.m@gmail.com wrote: [SNIP]
First, you configure to run multiple MagmaClient sessions. What you do is have each web session have its own MagmaSession view of the repository, and use ReadStrategy's and #stubOut: to keep only enough in memory for the current working web page. This was proposed years ago here:
http://wiki.squeak.org/squeak/5817
and I think Brent may have successfully implemented the multi-image configuration described on that page under "Remote Connections". Goran and I have discussed it in the past, but I don't know whether he ever tried it in earnest. If so, I hope he will comment about this.
Before summer we moved back to the above "default" approach and now given Keith's work on getting proper readstrategies and other smart tricks in place - we have approximately the same speed that we earlier achieved by cheating through a shared session (for parts of Gjallar).
So the answer is that yes, Gjallar has always been running one MagmaSession per Seaside session except that we did a detour through a "shared session" for one particular UI (the one that shows a table of lots of cases). But now we are back to the "clean" 1-per-1 approach.
[SNIP]
- Partition your domain by its major categories and use
MagmaForwardingProxy's to link to objects across repositories. For more information:
http://wiki.squeak.org/squeak/5604
I think this is a smart thing to do from the beginning simply for organizational purposes, but it also helps performance by dividing up the server burden across multiple images/computers.
This feature works and has been around for years, although I don't know anyone who has used it (because no one has ever asked about it).
We haven't considered it. We are busy getting Gjallar running in a multi-Seaside + one Magma server image configuration at the moment.
regards, Göran
Hi,
and I think Brent may have successfully implemented the multi-image configuration described on that page under "Remote Connections". Goran and I have discussed it in the past, but I don't know whether he ever tried it in earnest. If so, I hope he will comment about this.
We used this successfully with the following notes:
1. We have one MagmaSession per Seaside session in each Seaside image 2. We have one Magmer Server image open.
This worked well as we have a simple mechansin to pre-allocate one MagmaSession for speed.
The ony problem we ever have was running the Magma Server image on *NIX (linux/solaris) in headles or on a vncserver. In this case, we had real problem with TCP connections in the magma server just hanging.
I would like to hear other peoplese experience on *NIX.
magma@lists.squeakfoundation.org
-
Brent Pinkney -
Chris Muller -
Chris Muller -
goran＠krampe.se -
Igor Stasenko