[Seaside] Static sites and spider handling

[Colin Putney]

On Tuesday, August 26, 2003, at 11:22 AM, Avi Bryant wrote:

>
> On Tue, 26 Aug 2003, Colin Putney wrote:
>
>> One thing I learned while working at Whistler.com is that search 
>> engine
>> robots are incredibly paranoid and cynical. It would seem that the
>> primary design goal of a search engine is to detect and defeat
>> spamdexing in order to present an accurate view of the web to its
>> users. As a result, the bots absolutely *hate* two-faced web sites, 
>> and
>> will penalize them in the rankings accordingly.
>
> I'm curious - how do they *know*?  Do they occasionally use fake UA
> strings to keep you honest?

Good question. I don't know the answer.

I remember a few years ago, the hip thing in the SEO community was 
"feeding" the search engines. They'd build dummy web sites at separate 
domains that linked into the real web site. Both the dummy sites and 
the real site would detect bots and dynamically generate really 
optimized versions of their pages - high keyword densities, urls with 
keywords in them etc. That would throw the relevancy metrics off in the 
index, and the main site would get a boost in the rankings for the 
target keywords.

We were advised not to do this by a search engine consultant, as she 
had a few clients that were just about blacklisted by the search 
engines. We ended up getting by on the strength of the domain name, the 
fact that the site had been around forever, and all the genuinely 
useful information we put on the site.

In those days, there was a sort of "Top 10" of search engines that you 
had to do well in, and they all had different quirks. Now that Google 
has eclipsed the others, it might be more feasible to tailor content 
for Google and just ignore the others. Google might even be more 
forgiving about that kind of thing, since it values links more than 
keyword density, and links are harder to fake. Maybe this is why Amazon 
does so well.

Anyway, since we didn't go that route, I never found out the details of 
how to feed the search engines successfully.


Colin