[Seaside] Re: Scaling Seaside apps (was: About SToR)

[Yanni Chiu]

Jeremy Shute wrote:
> GLORP is a dream to work with.  It almost makes those spurious
> object-access patterns look free.  :-)  But, if you don't want to store a
> whole table in memory and you don't want to go twiddling down the whole
> B-tree every time you do an object access, you want a cursor, and I
> haven't quite figured out how to get that to work...
> 
> On a side note, I achieved 10-12x the throughput with my prototype program
> (written in a different language and dumping the serialized representation
> to disk), and I have moved on to yet another language to finish the job. 
> *Sigh*  One day I'll be able to use Squeak.

Do you have to have GLORP in the critical path?
If you only have a few tables, maybe coding the SQL directly
is possible. Or, use GLORP for the bulk of your model, but
isolate the performance critical portion of the model in a
separate subsystem, and use custom SQL for that portion.

Maybe GLORP is not appropriate for your data set.
Your use case does not sound ideal for any O-R framework.
Even in Java using Hibernate O-R, the recommend you NOT
use it for bulk data processing. But they do suggest a
workaround suitable for some cases, which is to use a
"report" query. What that does is bypass all the object
instantiation and caching framework needed for O-R,
(i.e. you don't need to create an actual object,
you just want the data values to push out a report).

Having said that, unless you use cursors, the postgres
driver will pull the entire result set into memory.
This behaviour is an artifact of the communication protocol
between the postgres server and a client process. However,
the newer version 3 of this protocol does not pull in the
entire data set. I'd be interested to know whether you can
in fact avoid pulling in the entire data set by using cursors,
with the current postgres driver (which implements version 2
protocol).

Assuming you can get cursors working, I'd be surprised if
you couldn't match the 10-12x increase you got using another
language. Basically, the postgres driver just pulls bytes off
the socket and makes arrays of strings.