Rupert Murdoch, the Australian who owns the Wall Street Journal, sparked some controversy recently when he announced that the WSJ would block Google’s search bots from indexing its site. It’s been speculated that up to 25% of WSJ’s traffic comes from Google, so this could have far reaching consequences as newspapers are a dying breed these days. Everyone is familiar with WSJ’s pay-to-read subscription model but if users find an article from Google News, then they can actually bypass the pay-to-read screen. WSJ subscribers who number in the hundreds of thousands who pay about $99 annually are actually a shining example of how web sites can get users to pay for content.
The beauty of Google is that it arranges and sorts the convoluted mess we call the Internet and allowing us to find what we need very quickly. But who asked Google to search your web site? You spent a lot of time and effort developing your site and countless hours creating fresh content for it — shouldn’t you be allowed to profit from that? What if everyone started to ban Google from indexing their site and started asking Google to charge for indexing rights? Sort of like a ContentSense for web site publishers.
Going one step further, what if we charged each search engine to index our site? Let’s say Bing was willing to bid $X for each page it indexes while Google would pay $Y. Perhaps there could even be a clearing house to bring content generators and search engines alike to bid for content. I firmly believe that the reason the Internet adoption grew so rapidly was because monetization wasn’t the prime concern, but rather the whole ideology of freely sharing knowledge. Once we all turtle up and begin demanding payment for content, things can deteriorate very rapidly. For example, what if spammers started to create fake content, or how would one evaluate the quality of content against others?
I highly doubt WSJ will bar Google from indexing its site, and besides if they did, users would simply switch to other competitors like Bloomberg, BNET, or Market Watch. It’s not like the WSJ has exclusive content rights similar to Entourage on HBO.