[Seaside] I need success stories

[Michael Lucas-Smith]

>
> So, does anyone on this list have experience using Seaside in a high 
> traffic scenario.  I'm talking 200+ hits per second at least.  How did 
> it handle the load?  What sort of hardware did you choose?  What load 
> balancing?  What database?
>
>
I've seen this question pop up from time to time and unfortunately there 
is no easy answer to it. However, there are some things you can take 
away that should make you feel more confident about any web framework 
you choose to use.

The 200+ requests per second is probably the thing I like to analyze the 
most, because I always wondered when this sort of traffic would happen. 
A prime example of an unexpectedly high traffic effect is the digg 
effect / slashdot effect / various other names. Usually this effect will 
hit a site that is generally low on traffic, say, 2000+ hits a day. 
They'll suddenly get boosted to 40k hits in a single hour.

http://www.geology.smu.edu/~dpa-www/attention_span/

The important thing to see here is that this 'extreme' where suddenly 
you're getting the attention of a page like google or yahoo for a few 
short minutes is exactly how many hits per second you actually have to 
handle. In the above page they witnessed 78,000 hits per hour on their 
first slashdotting and 1930 hits per minute.

The don't give us a higher resolution than that to know exactly how many 
hits per second they were seeing, but if we take a nieve approach, the 
average hits per second was 32. If say that all of that traffic hit 
within the first 15 seconds, then we get up to 96 hits per second.

Can Seaside handle that? James Robertson did some rather coarse 
statistics to see how Seaside currently compares across Smalltalks and 
between Seaside 2.8 and Seaside 2.7. Admittedly, he could have done more 
but it serves as a nice rough guide:

http://www.cincomsmalltalk.com/blog/blogView?showComments=true&printTitle=Scaling_Seaside_in_Cincom_Smalltalk&entry=3372841275
His results show 79.4 hits per second.

So without real hard data to know exactly what would happen, it appears 
on the surface that you would actually survive a slashdotting without 
any extra hardware or any extra caching strategies.

With that out of the way, we can now talk about what you would do to 
scale beyond. You can of course, put caching strategies in front of your 
application, or distribute it across machines, serve it up on Amazon's 
cloud, use one of the distributed web cache engines, etc... though 
applications that lend themselves to this sort of distribution are 
rarely transaction based, so you can almost certainly rule out a 
requirement for writing to a database as part of that 'massive hit'.

Even if we say the high hit rate is the norm for your application - and 
that would require you to really hit something big like Twitter (and 
Twitter maxed out at a mere 5 hits per second until they started 
optimizing themselves) - then running two servers at once immediately 
doubles your capacity.. it is near linear capacity growth when you add 
more servers.

I wish I had stats on having two Seaside images running simulatenously 
on the same machine with dual cores for you, but I don't just yet.. 
sorry about that.

So we know that Seaside itself isn't really a major bottleneck to 
performance, but there are other obvious factors that you've listed 
above. The big one is Database whether it be an Object database or a 
Relational database, you end up with the same sorts of bottlenecks.. 
number of connections, speed of connection to the database, 
unintentional callback blocking, accidental 'read it all in to memory' 
instead of streaming... that sort of stuff is quite commonly missed.

Luckily all those things have been solved when working with Seaside and 
various databases. Gemstone is multithreaded, Glorps interfaces can 
connection pooling, VWs VM is non-blocking on callbacks.. most database 
drivers are streaming based.

This is in contrast to the way Ruby on Rails is initially setup. It 
comes with one database connection with a global lock on it. In fact, it 
was this design flaw in RoR that caused Twitter to faulter so badly. The 
biggest change they made on their site was not to stop producing pages 
dynamically (although they did in the end) but to have multiple database 
connections at once in a pool.

Another thing that can drastically cripple your application is excessive 
memory use by your application or other applications on your server such 
that memory is forced to be paged out to disk. This is an absolute show 
stopper and can be caused by excessive state storage in your 
application. This is rarely caused by Seasides continuations and is more 
often caused by large object domains read from a database and cached in 
memory for each user or something similar.

Conversely, creating too many objects and throwing them away immediately 
(or worse, putting them in a weak set and throwing them away 
immediately) can tax the server by putting too much pressure on the 
garbage collector. This can easily halve the number of hits per second 
you can support. But again, this is a rather rare scenario to be in and 
it's easy to diagnose with profiling tools.

A few years back the company I worked for did an online federal election 
polling site. It was surprisingly popular and experienced regular 
'slashdotting' effects. The machine and network were over specced.. quad 
core, four servers, load balanced, huge raided multicore database server 
on 1 gb network ... none of the servers batted an eyelid. We could 
easily have taken a hundred fold traffic increase and we'd still 
probably have plenty of CPU time left to play civ4 on a huge map.

Sorry, this has been a long email - but I think my point was - Seaside 
is unlikely to be a major cause of bottleneck if you do start getting a 
lot of traffic. The usual strategies for load balancing and removing 
bottlenecks on the back end work the same as in any other web 
development environment.

Cheers,
Michael