Hi!

Willem van den Ende <squeak at willemvandenende.com> wrote:
> Hi Göran,
> 
> cool that someone is working on this! Maybe the new FastSocketStream can solve 
> a problem I've been having...

Hope so too. :)
 
> I'm writing a webspider in squeak, and as far as I could find out (it happens 
> occasionaly... haven't managed to write a test for it. I wrote some logging 
> code to pinpoint it) the old SocketStream locks up the image sometimes on 
> slow sites (cpu usage goes from between 3 and 10 percent to 99 when it locks 
> up). Frustratingly enough, lockups don't happen when I query the same page 
> 4000 times...
> 
> I'm using SocketStream under the SWHTTPClient package (it is less automagic 
> than the httpclient that comes with the image). Differences with 
> FastSocketStream seems that it (hopefully) does not lock up the imate.(I have 
> to run more tests to be confident about this- it takes ages...).   On the 
> downside, it does generate more 'a primitive has failed' errors and 
> 'ConnectionClosed: Connection closed while waiting for data.' (the old 
> SocketStream didn't generate those).

Right, the old SocketStream (as I painstakingly describe in the very
long class comment) swallows exceptions - which IMHO was bad design. You
can get the same behavior with FSS if you send it "shouldSignal: false"
but then lots of methods will not honour the timeout (they are carefully
marked in the method comments, and the class comment describes that
too). Which is actually the same behavior as the old SocketStream has -
which also SocketStreamTest verified.

So it is not the Right Thing to turn off the signals. The Right Thing
would be to catch the ConnectionClosed and handle it. The only method
that actually catches ConnectionClosed is #upToEnd because it
intrinsically must catch it - otherwise it makes very little sense. :)

The failed primitives I have no idea about. Do you have any more info on
that?

> I'm using a 3.7 image with a /3.7b-5 VM from ian pumarta's site. I used an 
> older vm as well, with the same problems.

Ok. Well, I use that VM too, haven't seen any real problems - but I am
not running a spider so... :)

> The good news is, the FastSocketStream does seem to be faster on sites that 
> are not slow, and I hope it doesn't lock anymore...

Well, that is why I call it FastSocketStream :). It IS faster. The
implementation is carefully designed to use faster primitives and much
less copying.

> I hope this helps. Keep up the good work!

Thanks, Göran