Search Quality, Content Farms, and Conspiracy Theories

The blogosphere and media outlets have been abuzz lately with reports of low-quality search results from the major search engines. It started in December 2010 with a New York Times story about an unscrupulous merchant whose bad treatment of customers and negative reviews pushed the site to prominence in Google searches. In response to the flap that arose, Google quickly addressed this with changes to its search algorithms. Since then, there have been a New York Times expose of J.C. Penney’s SEO practices (“The Dirty Little Secrets of Search”) and Google’s subsequent changes to its algorithm, Google’s assertion that Microsoft Bing was copying Google search results, Google’s banning of spam from low-quality sites and so-called “content farms,” and even a conspiracy theory about which company was launching a negative publicity campaign against Google. This has indeed been a wild and crazy time.

Clearly, battling spammers is a never-ending task for the search engines. There will always be those who try to game the system to their advantage. But for searchers who put their trust in the results delivered by search engines, these kinds of stories give us little hope for retrieving reliable information.

Matt Cutts, principal engineer at Google, recently blogged about the problem. He says that, “according to the evaluation metrics that we’ve refined over more than a decade, Google’s search quality is better than it has ever been in terms of relevance, freshness, and comprehensiveness. Today, English-language spam in Google’s results is less than half what it was five years ago, and spam in most other languages is even lower than in English.”

On Feb. 24, Cutts and Amit Singhal, a Google Fellow, blogged that the company had “launched a pretty big algorithmic improvement to our ranking—a change that noticeably impacts 11.8% of our queries—and we wanted to let people know what’s going on. This update is designed to reduce rankings for low-quality sites—sites which are low-value add for users, copy content from other websites or sites that are just not very useful. At the same time, it will provide better rankings for high-quality sites—sites with original content and information such as research, in-depth reports, thoughtful analysis and so on.”

Since then the bloggers have been busy analyzing the numbers and results stemming from what has been dubbed the “Farmer” algorithm update. At this point, some low-quality sites do seem to have been blocked or at least impacted by the changes. But, the jury is still out. A recent post on SearchEngineWatch points out that many good “innocent” sites have also been penalized and swept from Google’s ranking. Google says it is working on this problem.

Other Developments

One search engine that says it is not trying to beat Google but it is trying to beat web spam is Blekko. Its “slashtags” let users search only the sites they want and cut out the spam sites. In her NewsBreak, Marydee Ojala calls Blekko “an excellent alternative to Google—not a replacement for Google, but a search engine that provides different answers and varying perspectives. It allows for vertical, opinion, and customized searches. It encourages information sharing.” Blekko is noteworthy for introducing “the world’s first Spam Clock. This clock is going to record in real time the amount of web spam that is being spewed out. The clock is designed to bring greater attention to this growing problem.” The company says, “Every hour 1 million new spam pages are created.” Now that’s a scary thought.

Blekko recently took the bold step of banning the top 20 offending sites its users have marked as sources of spam. The list includes Demand Media’s eHow and Answerbag, as well as allexperts.com, experts-exchange.com, and even encyclopedia.com (owned by HighBeam, which is owned by Gale).

Another start-up search engine with the goal of eliminating spam from search results has the funny name of DuckDuckGo. Search expert Chris Sherman says it’s “actually a very good search engine.” It also has a focus on privacy and, says Sherman, “has built very powerful tools into the search engine that go to great lengths to keep your queries (and your identity) anonymous.” Blekko and DuckDuckGo are partnering on search results. Rich Skrenta of Blekko says, “It’s because we can kill spam a lot faster working together than we can working against each other.” Both of these search engines are worth trying.

Users of Google’s free Chrome web browser can now try an experimental extension to block sites from their web search results. When you block a site with the extension, you won’t see results from that domain again in your Google search results. You can always revoke a blocked site and you can edit your list of sites. If installed, the extension also sends blocked site information to Google, which says it will study the resulting feedback and explore using it as a potential ranking signal for its search results.

Other Developments

More Links