[Seaside] RSS and Seaside
Nevin Pratt
nevin at smalltalkpro.com
Mon Dec 29 19:29:46 CET 2003
Cees de Groot wrote:
>On Thu, 2003-12-25 at 20:52, Nevin Pratt wrote:
>
>
>>Could an RSS feed potentially be a solution for Seaside's unfriendliness towards spiders?
>>
>>
>>
>Janus is my shot at a solution. It seems to works fine, my Seaside site
>has shot up in traffic and search engine presence over the last months
>(from basically nothing to currently 160 visitors/day).
>
>
>
Cees,
Why does Janus bother generating a 'robots.txt' response?
As far as I can see, your generated 'robots.txt' file either denies all,
or allows all. And, if I'm not mistaken, if a spider does not find a
'robots.txt', it assumes "allows all". And, why would you ever "deny
all" to a spider?
If you always want to "allow all" to a spider, and if that is the
default anyway if the spider does not find a 'robots.txt', then just
don't bother generating one.
Or am I missing something?
Nevin
--
Nevin Pratt
Bountiful Baby
http://www.bountifulbaby.com
(801) 992-3137
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/seaside/attachments/20031229/d7bcaacb/attachment.htm
More information about the Seaside
mailing list