[Seaside] Re: Scaling Seaside apps (was: About SToR)

[Jeremy Shute]

On 8/2/06, Yanni Chiu <yanni at rogers.com> wrote:

> Jeremy Shute wrote:
> > GLORP is a dream to work with.  It almost makes those spurious
> > object-access patterns look free.  :-)  But, if you don't want to store
> a
> > whole table in memory and you don't want to go twiddling down the whole
> > B-tree every time you do an object access, you want a cursor, and I
> > haven't quite figured out how to get that to work...
> >
> > On a side note, I achieved 10-12x the throughput with my prototype
> program
> > (written in a different language and dumping the serialized
> representation
> > to disk), and I have moved on to yet another language to finish the job.
> > *Sigh*  One day I'll be able to use Squeak.
>
> Do you have to have GLORP in the critical path?
> If you only have a few tables, maybe coding the SQL directly
> is possible. Or, use GLORP for the bulk of your model, but
> isolate the performance critical portion of the model in a
> separate subsystem, and use custom SQL for that portion.


Yes, it's very much in the critical path.  I'm sorry but I'm still amazed
that it can't assemble and ship queries to Postgres as fast as I can get
data from a cable modem.  That was a huge shocker -- I've got a weak cable
connection on one side and a disk on the other, and rifling through strings
and objects in RAM is the issue???

I agree that there are solutions which involve direct-SQL access, and making
a mess of otherwise clean code (but a well isolated mess, of course).  I
could also simply contribute to GLORP in order to make it better.  I would
do this in a second if I thought it would get me from point A to point B,
GLORP is great software!

As an off-topic side-note, in order to GET from point A to point B, I
addressed the problem by developing a "new to me" paradigm for dealing with
data of this type.  So far, I think I did the right thing.  Like SQL (and
unlike serialization or the Prevayler approach), multiple programs can get
access to the same objects from out-of-core datastructures.  But unlike SQL,
the indices require ~1 disk seeks to get at objects after a cache-miss ( i.e.
hash based), columns can be in a much more structured format (think of
something similar to memcpy for a DOM tree, for instance), etc.



Having said that, unless you use cursors, the postgres
> driver will pull the entire result set into memory.


Sigh.  I know.  The options seem to be:

* Get the whole result set if it fits in memory.
* Seek the same B-tree nodes over and over again if it doesn't (the root
should be cached by the RDBMS, of course).

Cursors would definitely be the answer to this, but I recognize that I am in
the minority in my need for them.  Really, I don't think the cursor would
fix my 10-12x problem.  For me, it's a matter of bypassing caches and using
prepared statements.  But, I wanted to deal in objects, and found a fine way
to continue to do that without the overhead of OR mapping.



Assuming you can get cursors working, I'd be surprised if
> you couldn't match the 10-12x increase you got using another
> language. Basically, the postgres driver just pulls bytes off
> the socket and makes arrays of strings.
>

I'm betting that Squeak is capable of that 10-12x with proper massage.  But
the 10-12x will simply match the prototype implementation, which in turn has
not been massaged.  (In fact, both implementations are really stupid in that
they are SERIAL.)  I have figured out how to use a proxy object to get the
GLORP sessions to be thread-safe, but the next barrier will be lock
contention as the serial implementation becomes a simple producer/consumer
queue.

I would say that Squeak is currently state-of-the-art in terms of programmer
interface -- Seaside and GLORP are basically unrivalled in their design and
terseness.  Avi didn't need any of the stuff I mentioned to make a great
piece of software.  But, the subject was "scalability", so I wanted to offer
myself as a data point.

Jeremy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/seaside/attachments/20060803/b9e0690d/attachment-0001.htm