[Seaside] RSS and Seaside

Nevin Pratt nevin at smalltalkpro.com
Mon Dec 29 19:29:46 CET 2003


Cees de Groot wrote:

>On Thu, 2003-12-25 at 20:52, Nevin Pratt wrote:
>  
>
>>Could an RSS feed potentially be a solution for Seaside's unfriendliness towards spiders?
>>
>>    
>>
>Janus is my shot at a solution. It seems to works fine, my Seaside site
>has shot up in traffic and search engine presence over the last months
>(from basically nothing to currently 160 visitors/day). 
>
>  
>

Cees,

Why does Janus bother generating a 'robots.txt' response?

As far as I can see, your generated 'robots.txt' file either denies all, 
or allows all.  And, if I'm not mistaken, if a spider does not find a 
'robots.txt', it assumes "allows all".  And, why would you ever "deny 
all" to a spider?

If you always want to "allow all" to a spider, and if that is the 
default anyway if the spider does not find a 'robots.txt', then just 
don't bother generating one.

Or am I missing something?

Nevin

-- 
Nevin Pratt
Bountiful Baby
http://www.bountifulbaby.com
(801) 992-3137

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/seaside/attachments/20031229/d7bcaacb/attachment.htm


More information about the Seaside mailing list