[Seaside] RSS and Seaside

[Cees de Groot]

On Mon, 2003-12-29 at 19:29, Nevin Pratt wrote:
> Why does Janus bother generating a 'robots.txt' response?
>
Well, this might be the result of experimenting. The idea was that if
Janus didn't think a spider was hitting the site but robots.txt was
fetched nevertheless, it would generate a deny-all version to make sure
that nothing automated would ever go into the Seaside-dynamic-url
jungle. I don't think that happens anymore - forgive me for not knowing
exactly what Janus does at the moment, I fiddled with it until it worked
and tried a lot of alternatives on the way to it, one of them being that
anything that would fetch robots.txt would get an allow-all and would
subsequently be treated as a spider for some time.