[squeak-dev] SqueakSource indexability (aka should we just ask
crawlers to desist?)
Bert Freudenberg
bert at freudenbergs.de
Wed Apr 28 19:59:18 UTC 2010
On 28.04.2010, at 21:07, Ken Causey wrote:
>
> At times access to source.squeak.org becomes slower, as has been the
> case today. I can see in the logs that various web-crawlers are the
> likely culprit. Having the information there accessible via search
> engines is a wornderful thing but I have to suspect that the Seaside
> session IDs eliminate this option. (Of course when URLs like
> http://source.squeak.org/trunk.html are found on other sites they then
> become indexed.)
Which URLs are the bots accessing?
> Unless I'm mistaken about this, and I would appreciate any guidance, it
> seems like we need to add a robots.txt to the site which guides or
> simply asks crawlers to stay away. Thoughts? I'm no SEO export.
We do have a robots.txt:
http://source.squeak.org/robots.txt
- Bert -
More information about the Squeak-dev
mailing list
|