The Absurdity of the Robots.txt Plan
Posted on | November 24, 2009 | No Comments
A lot of people are talking about blocking their content from Google’s spiders lately.
Rupert Murdoch kicked up this dust storm with his digital impression of a grumpy old man telling the neighboor’s kids to get off his lawn. Apparently he’s trying to rope Microsoft in as well. Now, Silicon Valley Watcher makes the case search should be non-profit and site owners should block all commercial search engines.
Now, all this makes for interesting theater, but all this noise makes me worry that these people are serious and I wonder when they are going to wake up and realize that blocking Google is not the way to making the web better, or monetizing content.
The problem, as far as I can see it, is that the people who think they can robots.txt their way out of Google’s fundamentally misunderstand searchers and search’s place on the web. I seriously doubt any of these people have spent any meaningful time talking to searchers.
If they did, they’d understand a key underpinning of search’s success – that many searchers and many people who go online don’t distinguish between the Internet, their browser, and Google. They all blend into one single mass called the “web”, and like it or not, for many of these people, Google is the face of the web.
Think about it for a second – someone doesn’t care enough about your site enough to type its name in the address bar, you really think they’re going to give up their search engine for it? That would be like a store complaining about Ford, Chevy and the other car manufacturers because its customers drive to it.
What people like Murdoch aren’t realizing is that if they pull their content from Google, the likely reality is that they won’t be missed as much as Murdoch et al think they would. More than likely, they’ll create a de facto gold rush by other sites to fill the vacuum left by their departure.
Comments
Leave a Reply