There has never been a search engine that accurately reflects the Internet.
In the 1990s and 2000s, the limitation was technical. The so-called "deep web" and "dark Internet" — which sound shady and mysterious, but simply refer to web sites inaccessible by conventional means — have always existed.
Many parts of the Internet are hard to index, or are blocked from being indexed by their owners.
Companies like Google have worked hard to surface and bring light to the "deep, dark" recesses of the global web on a technical level.
But in the past few years, a disturbing trend has emerged where governments — either through law or technical means or by the control of the companies that provide access — have forced inaccuracy, omissions and misleading results on the world's major search engines.
Until recently, search engine censorship was not on the list of first-world problems. But in the last few years, governments in the United States, Europe and elsewhere in the industrialized world have discovered that, although they're prevented by free-speech laws from actually blocking or banning content where it lives, censoring search engine results is a kind of "loophole" they can get away with. In an increasingly digitized, search-engine discoverable world of content, censoring search results is a way to censor without technically violating free speech protections.
Starting in 2011, companies like Google started reporting a disturbing rise in government requests for search engine results to lie — to essentially tell users that existing pages and content on the Internet do not exist when in fact they do. Requests for such removals by the U.S. government, for example, rose 718% from the first half of 2011 to the last half. And they've continued to rise since.
And such requests weren't just coming from the U.S., but from "Western democracies not typically associated with censorship," according to the Google policy analyst who reported the trend on behalf of the company and talked about Google's Transparency Report.
The reasons for these requests vary, and often sound reasonable — national security, law and order, national pride, religious sensitivity, social order, suppression of hate speech, privacy, protection of children — you name it. But when you add them up and allow them to grow in number over time, the cumulative effect is that increasingly, search results don't reflect the real Internet.
Many of these cases start out with the best intentions but result in serious problems. Let's start with a disturbing recent case in Canada.
A Supreme Court of British Columbia ruling on an intellectual property dispute between two small industrial equipment companies ordered Google to not only delete all search results referring to one of the companies, but all future such results as well -- not only in Canada, but worldwide. (Yet another unsavory dimension to the case was that the ruling applied only to Google. Bing and other search engines were not required to comply.)
Sign up for Computerworld eNewsletters.