[Seaside] saving an image while serving

Sebastian Sastre sebastian at flowingconcept.com
Mon Apr 18 13:59:05 UTC 2011 

"But you cannot scale horizontally just simply"

Not working the way Nevin does. But that's not the Nevin's case. He's saying this "yes is one image, yes if it crashes we need to rollback, yes that architecture doesn't scale horizontally so what? is enough for the business it get us there just fine." (Nevin please correct me if I'm wrong)

Our case is different. Image persistence didn't fit our needs. We really needed to be able to scale horizontally. The way we found is to treat images like servers. Any user is be able to be attended by any image. That's how you do it.

Sticky sessions is a non issue once the httpd is configured (and properly balanced). About being forced to use a central data storage, that sounds plain wrong. I mean at design level. To be able to scale horizontally any time you have to design in a way that doesn't generate points of contention. How you design evading points of contention is an art. You need to find your most convenient way for your application. In our case we used one object database per group of people. Lots of groups, plenty of room for everybody's activity.

About memory size in images, we monitor the images and restart them as needed. The worst thing that can happen in an image restart is that you see your last request failed and in your next request, the http balancer sends you to another image. This is ACID so things gets done or do not. Nothing is lost.

That could happen with any image in any server (hardware).

About oversimplification and *most* scenarios: I'm sharing with you guys some details on how we got this thing done. What I'm not doing is trying to make people decide on "this" or "that" like some people suggested. Who the f* will invest or employ a guy deciding architecture like that? I'm making my case because is working and I think some could find it interesting or useful.

I thought it was clear but... hey, I'm counting with the intelligence and critical thinking of the reader. Always. 

Seriously, who decides the architecture of a startup in what he/she reads on a f* list?

Do your homework. If it f* works, then it does. If it doesn't, then it doesn't. 

with love

sebastian

o/



On Apr 18, 2011, at 3:19 AM, Norbert Hartl wrote:

> 
> Am 17.04.2011 um 20:13 schrieb Sebastian Sastre:
> 
>> The only realistic way we see for that is by scaling horizontally.
>> 
>> Many many worker images will do. And if people needs more, well... you simply add more.
>> 
> I would really like to discuss this with Nevin and you. Most of the time you cannot "...simply add more" images. I would like to know what setups you have. In a web scenario where data is created by web clients in the image you can have it in one image, right. But you cannot scale horizontally just simply. You need to take special action like sticky sessions/domain partitioning if the domain model is not highly inter-connected or you forced to use a central data storage. 
> If you can solve it this way than you are lucky guys. Btw. is there a memory size in pharo from which performance will degrade faster? 
> 
> I agree with you guys that you can achieve a lot with a simple image. But it sounded to me like an over-simplification of deployments which won't work in many scenarios.
> 
> Norbert
> 
>> Of course that caution should be taken in the income/spending on validable* equations (monetization, customer adquisition, workers per server costs, etc) so things can grow smoothly.
>> 
>> *by validable equations I mean the math that comes out from the experience (and not math based on assumptions)
>> 
>> 
>> 
>> On Apr 17, 2011, at 6:29 AM, Norbert Hartl wrote:
>> 
>>> 
>>> Am 17.04.2011 um 01:57 schrieb Igor Stasenko:
>>> 
>>>> This is really good to hear that.
>>>> 
>>>> I leaned something new. Usual way of design is to prevent failure(s).
>>>> But going that way we tend to forget that failures are inevitable, and
>>>> paying little attention towards what can be done
>>>> to restart/reload data from backups quickly and easily.
>>>> Indeed, embracing failure is the way we should design our systems.
>>>> 
>>> Well, I can tell if you plan big systems than the failure scenarios planning takes a big part of the overall planning. There you embrace failures very very much. You can't go anywhere with something that is not redundant in at least one dimension. And the software has to fit in the hardware planning. So, not embracing failure is a pleasure that only hobbyists and smaller companies can have.
>>> 
>>> Norbert
>>> 
>>>> On 17 April 2011 01:37, Nevin Pratt <nevin at bountifulbaby.com> wrote:
>>>>> Define "handful".  You'd be surprised.
>>>>> 
>>>>> We are a large enough company with enough orders, to support more than a
>>>>> dozen employees, and a half a dozen contractors.  Our warehouse/office is
>>>>> now about 26,000 square feet.  And, we're growing, and profitable, with 2011
>>>>> looking to be our best year ever.
>>>>> 
>>>>> The exact numbers are confidential, but I am confident that we are the
>>>>> largest company on the planet in our field (i.e., "Reborn Doll Supplies").
>>>>> 
>>>>> Besides, the orders aren't exactly "hand reentered".  I just didn't give
>>>>> enough detail, because I didn't feel it was relevant.  We have an admin page
>>>>> that can easily slurp the data in rapidly, if needed.  It's not hard to do.
>>>>> 
>>>>> And Squeak/Seaside has served us well.  And we just use the image for
>>>>> persistence, as previously mentioned.  And we are very happy with it.
>>>>> 
>>>>> Nevin
>>>>> 
>>>>> 
>>>>> Of course that only works when you have a handful of orders to reenter,
>>>>> which isn't the case for most systems.
>>>>> 
>>>>> Sent from my iPhone
>>>>> On 2011-04-16, at 12:41, "Sebastian Sastre" <sebastian at flowingconcept.com>
>>>>> wrote:
>>>>> 
>>>>> So, 15 minutes times 4 makes a whole hour per year.
>>>>> Beautiful :)
>>>>> Doesn't even justify to implement automation of the manual rollback (that
>>>>> sounds feasible)
>>>>> 
>>>>> 
>>>>> 
>>>>> On Apr 16, 2011, at 1:29 PM, Nevin Pratt wrote:
>>>>> 
>>>>> Sebastion, that is *exactly* what I initially did at Bountiful Baby.
>>>>> However, it has been several *years* since I've had to do your #4, so things
>>>>> have changed slightly since.
>>>>> 
>>>>> Bountiful Baby is an eCommerce site, and the critical "object graph" (your
>>>>> #2, below) information consists of inventory data (the website keeps track
>>>>> of our inventory), and gift certificate data (for gift certificates that
>>>>> have been issued).  Also, whenever either of those datums change, the
>>>>> website sends an email-- for example, emails are (obviously) sent for each
>>>>> order accepted, and, it sends emails when it issues a gift certificate.  So,
>>>>> it is easy to discover (via the emails) what data was lost since the last
>>>>> image crash.
>>>>> 
>>>>> Consequently, currently this is what happens:
>>>>> 
>>>>> 1. image is saved from time to time (usually daily), and copied to a
>>>>> separate "backup" machine.
>>>>> 2. if anything bad happens, the last image is grabbed, and the orders and/or
>>>>> gift certificates that were issued since the last image save are simply
>>>>> re-entered.
>>>>> 
>>>>> And, #2 has been *very* rarely done-- maybe a two or three times a year, and
>>>>> then it turns out it is usually because I did something stupid.
>>>>> 
>>>>> For us, it's a whole lot easier to do persistence this way than bothering
>>>>> with any persistence mechanism.  And, it turns out, it is *very* reliable,
>>>>> with exactly one easily fixable glitch:
>>>>> 
>>>>> The glitch is: occasionally the production image UI will freeze, for no
>>>>> known reason.  It doesn't effect Seaside, though, so the website keeps going
>>>>> just fine.  And, if we run the "screenshot" app (in the configuration
>>>>> screen), there is a link at the top for "Suspend UI Process", and "Resume UI
>>>>> Process".  Just suspend and resume, and the UI becomes unstuck.
>>>>> 
>>>>> We've been doing persistence this way for years now, and I've been
>>>>> *extremely* impressed with the reliability.
>>>>> 
>>>>> Before that, we used PostgreSQL and GLORP for persistence.  But I yanked
>>>>> that code out years ago.  It wasn't worth the bother maintaining it.
>>>>> 
>>>>> If you have a daily image save, then on average there will be 12 hours of
>>>>> lost data on a "random" crash.  Re-entering 12 hours of orders and/or gift
>>>>> certificates (discovered from the emails, as mentioned above) might take 10
>>>>> to 15 minutes.  Not a big deal at all for us.
>>>>> 
>>>>> Ten years ago I wouldn't have even considered doing persistence this way,
>>>>> but I've changed.  Squeak has changed.  Seaside has changed.  And all for
>>>>> the better.
>>>>> 
>>>>> Nevin
>>>>> 
>>>>> 
>>>>> 
>>>>> Given that you don't need transactions, to make that style work, I suggest
>>>>> this:
>>>>> 1. save the image from time to time (as best suits your needs) like you're
>>>>> doing now
>>>>> 2. use a secondary way to dump the object graph
>>>>> 3. use the normal image for as long as things are good
>>>>> 4. when shit happens you "transplant" the object graph into a new
>>>>> "reincarnation" of your app in a fresh image
>>>>> 5. repeat
>>>>> For dumping the ODB you have options: image segments, SIXX comes to mind now
>>>>> If you make for yourself some "rescue" kit (script, tools, preloaded code in
>>>>> fresh image), you can make 4 quite painless (or, why not, monitored and
>>>>> automated)
>>>>> sebastian
>>>>> o/
>>>>> 
>>>>> 
>>>>> On Apr 16, 2011, at 12:23 PM, Michal wrote:
>>>>> 
>>>>> hi -
>>>>> 
>>>>> Despite the warnings, I am really interested in sticking to the
>>>>> simplest way of saving my seaside application data, ie periodically
>>>>> saving and backuping the image. The seaside book states that
>>>>> 
>>>>> "saving [the image] while processing http requests is a risk you
>>>>> want to avoid."
>>>>> 
>>>>> What is the status on that? Is that something we can fix? I have been
>>>>> running an image in this mode for a few weeks, with no ill effect so
>>>>> far, but I have had major problems with old image/vm combinations. So
>>>>> is this something that might be fixed already?
>>>>> 
>>>>> Also, I recall that Avi had made a number of attempts at having an
>>>>> image saved in a forked background process, eg
>>>>> 
>>>>> http://lists.squeakfoundation.org/pipermail/squeak-dev/2005-October/095547.html
>>>>> 
>>>>> did anybody pick up on this, or did anything come out of it?
>>>>> 
>>>>> thanks,
>>>>> Michal
>>>>> _______________________________________________
>>>>> seaside mailing list
>>>>> seaside at lists.squeakfoundation.org
>>>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> seaside mailing list
>>>>> seaside at lists.squeakfoundation.org
>>>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>>> 
>>>>> _______________________________________________
>>>>> seaside mailing list
>>>>> seaside at lists.squeakfoundation.org
>>>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>>> 
>>>>> _______________________________________________
>>>>> seaside mailing list
>>>>> seaside at lists.squeakfoundation.org
>>>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>>> 
>>>>> _______________________________________________
>>>>> seaside mailing list
>>>>> seaside at lists.squeakfoundation.org
>>>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> seaside mailing list
>>>>> seaside at lists.squeakfoundation.org
>>>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> Best regards,
>>>> Igor Stasenko AKA sig.
>>>> _______________________________________________
>>>> seaside mailing list
>>>> seaside at lists.squeakfoundation.org
>>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>> 
>>> _______________________________________________
>>> seaside mailing list
>>> seaside at lists.squeakfoundation.org
>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>> 
>> _______________________________________________
>> seaside mailing list
>> seaside at lists.squeakfoundation.org
>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
> 
> _______________________________________________
> seaside mailing list
> seaside at lists.squeakfoundation.org
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/seaside/attachments/20110418/4daa6c1e/attachment-0001.htm

More information about the seaside mailing list