[Seaside] Static sites and spider handling

[Colin Putney]

On Tuesday, August 26, 2003, at 01:40  PM, Avi Bryant wrote:

>
> On Tue, 26 Aug 2003, Colin Putney wrote:
>
>> We were advised not to do this by a search engine consultant, as she
>> had a few clients that were just about blacklisted by the search
>> engines. We ended up getting by on the strength of the domain name, 
>> the
>> fact that the site had been around forever, and all the genuinely
>> useful information we put on the site.
>
> I did a little research just now; my impression is that this kind of
> blacklisting is not done automatically, but based on human observation.
> Bear in mind that what we're talking about does not present the search
> engine with actual content that's any different from what a human would
> see - we're simply tweaking the *structure* of the site to make it more
> robot friendly.  Unlike most "cloaked" sites, for example, we wouldn't
> want to disable the google cache using this approach (it might be a 
> little
> odd that the cached page had no links on it, but the content and layout
> would all be normal).

You may be right about that. I vaguely remember something about human 
involvement, but I couldn't remember if it was only Yahoo that did that 
or what.

> I would be surprised if this ended up genuinely annoying Google, unless
> they're trying to make some kind of a political point about cloaking.  
> If
> a human did end up auditing the site, it ought to be pretty obvious 
> that
> the aim is to help search engines, not to trick them.

Yes, this is a crucial distinction, which might even hold for the 
robots. They are becoming ever more sophisticated. My intent with this 
anecdote wasn't dire predictions of doom, but just to point out that 
there's a larger context for this issue. It's not *just* a technical 
problem, it has business, social and even political dimensions as well.

Depending on what your goals (or more directly, Cees' and possibly 
Nevin's goals) are, it might useful to find out what the rest of the 
internet marketing community has learned about this. Heck, it might 
even be worthwhile to contact Google about it. They have an interest in 
making sure that dynamic sites are meaningfully indexed too.

Colin