spiders and docs (was: [Seaside] REST and Seaside)

[Michal]

> Ok, so clearly this is a problem we need to deal with (I've never seen
> it because all the apps I've deployed have login pages :).  Does
> anyone have any suggestions?  Cees already has code in Janus for
> detecting spiders; what do we do with them once we have them?

When it happened, as a stop-gap I patched WAKom>>process: to return a
hardcoded deny-all string to /robots.txt requests, just like Brian
suggests (and temporarily kicked out Jeeves/Toema). Worked for me.

> > ii. WADocumentHandler never expiring (isActive always returns
> > true) and thus never getting collected, again inflating the count
> > of seaside entities sticking around.

> Yeah, that's a bit of a problem, but I'm not really sure what to do
> about it.  You should have a finite number of static documents being
> used anyway, shouldn't you?

I didn't spend time looking into it, but I did note that I had about
130 handlers for the same document...