The
company maintains at least nine different blacklists that impact our
lives, generally without input or authority from any outside advisory
group, industry association or government agency. Google is not the only
company suppressing content on the internet. Reddit has frequently been
accused of banning postings on specific topics, and a recent report
suggests that Facebook has been deleting conservative news stories from
its newsfeed, a practice that might have a significant effect on public
opinion – even on voting. Google, though, is currently the biggest
bully on the block.
When
Google's employees or algorithms decide to block our access to
information about a news item, political candidate or business, opinions
and votes can shift, reputations can be ruined and businesses can crash
and burn. Because online censorship is entirely unregulated at the
moment, victims have little or no recourse when they have been harmed.
Eventually, authorities will almost certainly have to step in, just as
they did when credit bureaus
were regulated in 1970. The alternative would be to allow a large
corporation to wield an especially destructive kind of power that should
be exercised with great restraint and should belong only to the public:
the power to shame or exclude.
If
Google were just another mom-and-pop shop with a sign saying "we
reserve the right to refuse service to anyone," that would be one thing.
But as the golden gateway to all knowledge, Google has rapidly become
an essential in people's lives – nearly as essential as air or water. We
don't let public utilities make arbitrary and secretive decisions about
denying people services; we shouldn't let Google do so either.
Let's
start with the most trivial blacklist and work our way up. I'll save
the biggest and baddest – one the public knows virtually nothing about
but that gives Google an almost obscene amount of power over our
economic well-being – until last.
1. The autocomplete blacklist. This is a list of words and phrases that are excluded from the autocomplete feature
in Google's search bar. The search bar instantly suggests multiple
search options when you type words such as "democracy" or "watermelon,"
but it freezes when you type profanities, and, at times, it has frozen
when people typed words like "torrent," "bisexual" and "penis." At this
writing, it's freezing when I type "clitoris." The autocomplete
blacklist can also be used to protect or discredit political candidates.
As recently reported,
at the moment autocomplete shows you "Ted" (for former GOP presidential
candidate Ted Cruz) when you type "lying," but it will not
show you "Hillary" when you type "crooked" – not even, on my computer,
anyway, when you type "crooked hill." (The nicknames for Clinton and
Cruz coined by Donald Trump, of course.) If you add the "a," so you've
got "crooked hilla," you get the very odd suggestion "crooked Hillary
Bernie." When you type "crooked" on Bing, "crooked Hillary" pops up
instantly. Google's list of forbidden terms varies by region and
individual, so "clitoris" might work for you. (Can you resist checking?)
2. The Google Maps blacklist. This
list is a little more creepy, and if you are concerned about your
privacy, it might be a good list to be on. The cameras of Google Earth
and Google Maps have photographed your home for all to see. If you don't
like that, "just move," Google's former CEO Eric Schmidt said. Google also maintains a list of properties
it either blacks out or blurs out in its images. Some are probably
military installations, some the residences of wealthy people, and some –
well, who knows? Martian pre-invasion enclaves? Google doesn't say.
3. The YouTube blacklist. YouTube,
which is owned by Google, allows users to flag inappropriate videos, at
which point Google censors weigh in and sometimes remove them, but not,
according to a recent report by Gizmodo, with any great consistency – except perhaps when it comes to politics. Consistent with the company's strong and open support
for liberal political candidates, Google employees seem far more apt to
ban politically conservative videos than liberal ones. In December
2015, singer Joyce Bartholomew sued YouTube
for removing her openly pro-life music video, but I can find no
instances of pro-choice music being removed. YouTube also sometimes
acquiesces to the censorship demands of foreign governments. Most
recently, in return for overturning a three-year ban on YouTube in
Pakistan, it agreed to allow Pakistan's government to determine which videos it can and cannot post.
4. The Google account blacklist. A
couple of years ago, Google consolidated a number of its products –
Gmail, Google Docs, Google+, YouTube, Google Wallet and others – so you
can access all of them through your one Google account. If you somehow
violate Google's vague and intimidating terms of service agreement, you will join the ever-growing list of people who are shut out of their accounts, which means you'll lose access to all
of these interconnected products. Because virtually no one has ever
read this lengthy, legalistic agreement, however, people are shocked when they're shut out, in part because Google reserves the right
to "stop providing Services to you … at any time." And because Google,
one of the largest and richest companies in the world, has no customer
service department, getting reinstated can be difficult. (Given,
however, that all of these services gather personal information about
you to sell to advertisers, losing one's Google account has been judged
by some to be a blessing in disguise.)
5. The Google News blacklist.
If a librarian were caught trashing all the liberal newspapers before
people could read them, he or she might get in a heap o' trouble. What
happens when most of the librarians in the world have been replaced by a
single company? Google is now the largest news aggregator in the world,
tracking tens of thousands of news sources in more than thirty languages and recently adding thousands of small, local news sources to its inventory. It also selectively bans news sources as it pleases. In 2006,
Google was accused of excluding conservative news sources that
generated stories critical of Islam, and the company has also been
accused of banning individual columnists and competing companies from its news feed. In
December 2014, facing a new law in Spain that would have charged Google
for scraping content from Spanish news sources (which, after all, have
to pay to prepare their news), Google suddenly withdrew its news service from Spain, which led to an immediate drop
in traffic to Spanish new stories. That drop in traffic is the problem:
When a large aggregator bans you from its service, fewer people find
your news stories, which means opinions will shift away from those you
support. Selective blacklisting of news sources is a powerful way of
promoting a political, religious or moral agenda, with no one the wiser.
6. The Google AdWords blacklist. Now things get creepier. More than 70 percent of Google's $80 billion in annual revenue comes from its AdWords advertising service, which it implemented in 2000 by infringing
on a similar system already patented by Overture Services. The way it
works is simple: Businesses worldwide bid on the right to use certain
keywords in short text ads that link to their websites (those text ads
are the AdWords); when people click on the links, those businesses pay
Google. These ads appear on Google.com and other Google websites and are
also interwoven into the content of more than a million non-Google
websites – Google's "Display Network." The problem here is that if a
Google executive decides your business or industry doesn't meet its
moral standards, it bans you from AdWords; these days, with Google's
reach so large, that can quickly put you out of business. In 2011,
Google blacklisted an Irish political group that defended sex workers
but which did not provide them; after a protest, the company eventually backed down.
In May 2016, Google blacklisted an entire industry – companies providing high-interest "payday" loans. As always, the company billed this dramatic move as an exercise in social responsibility, failing to note that it is a major investor
in LendUp.com, which is in the same industry; if Google fails to
blacklist LendUp (it's too early to tell), the industry ban might turn
out to have been more of an anticompetitive move than one of conscience.
That kind of hypocrisy has turned up before in AdWords activities.
Whereas Google takes a moral stand,
for example, in banning ads from companies promising quick weight loss,
in 2011, Google forfeited a whopping $500 million to the U.S. Justice
Department for having knowingly allowed Canadian drug companies to sell
drugs illegally in the U.S. for years through the AdWords system, and
several state attorneys general believe that Google has continued to engage in similar practices since 2011; investigations are ongoing.
7. The Google AdSense blacklist.
If your website has been approved by AdWords, you are eligible to sign
up for Google AdSense, a system in which Google places ads for various
products and services on your website. When people click on those ads,
Google pays you. If you are good at
driving traffic to your website, you can make millions of dollars a year
running AdSense ads – all without having any products or services of
your own. Meanwhile, Google makes a net profit by charging the companies
behind the ads for bringing them customers; this accounts for about 18 percent
of Google's income. Here, too, there is scandal: In April 2014, in two
posts on PasteBin.com, someone claiming to be a former Google employee
working in their AdSense department alleged
the department engaged in a regular practice of dumping AdSense
customers just before Google was scheduled to pay them. To this day, no
one knows whether the person behind the posts was legit, but one thing
is clear: Since that time, real lawsuits filed by real companies have,
according to WebProNews,
been "piling up" against Google, alleging the companies were
unaccountably dumped at the last minute by AdSense just before large
payments were due, in some cases payments as high as $500,000.
8. The search engine blacklist.
Google's ubiquitous search engine has indeed become the gateway to
virtually all information, handling 90 percent of search in most
countries. It dominates search because its index is so large: Google
indexes more than 45 billion
web pages; its next-biggest competitor, Microsoft's Bing, indexes a
mere 14 billion, which helps to explain the poor quality of Bing's
search results.
Google's dominance in search is why businesses large and small live in constant "fear of Google,"
as Mathias Dopfner, CEO of Axel Springer, the largest publishing
conglomerate in Europe, put it in an open letter to Eric Schmidt in
2014. According to Dopfner, when Google made one of its frequent adjustments
to its search algorithm, one of his company's subsidiaries dropped
dramatically in the search rankings and lost 70 percent of its traffic
within a few days. Even worse than the vagaries of the adjustments,
however, are the dire consequences that follow when Google employees
somehow conclude you have violated their "guidelines": You either get
banished to the rarely visited Netherlands of search pages beyond the
first page (90 percent of all clicks go to links on that first page) or
completely removed from the index. In 2011, Google took a "manual action" of a "corrective" nature
against retailer J.C. Penney – punishment for Penney's alleged use of a
legal SEO technique called "link building" that many companies employ
to try to boost their rankings in Google's search results. Penney was
demoted 60 positions or more in the rankings.
Search
ranking manipulations of this sort don't just ruin businesses; they
also affect people's opinions, attitudes, beliefs and behavior, as my
research on the Search Engine Manipulation Effect
has demonstrated. Fortunately, definitive information about Google's
punishment programs is likely to turn up over the next year or two
thanks to legal challenges the company is facing. In 2014, a Florida
company called e-Ventures Worldwide filed a lawsuit
against Google for "completely removing almost every website"
associated with the company from its search rankings. When the company's
lawyers tried to get internal documents relevant to Google's actions
though typical litigation discovery procedures, Google refused to
comply. In July 2015, a judge ruled
that Google had to honor e-Ventures' discovery requests, and that case
is now moving forward. More significantly, in April 2016, the Fifth
Circuit Court of Appeals ruled
that the attorney general of Mississippi – supported in his efforts by
the attorneys general of 40 other states – has the right to proceed with
broad discovery requests in his own investigations into Google's
secretive and often arbitrary practices.
This
brings me, at last, to the biggest and potentially most dangerous of
Google's blacklists – which Google's calls its "quarantine" list.
9. The quarantine list. To
get a sense of the scale of this list, I find it helpful to think about
an old movie – the classic 1951 film "The Day the Earth Stood Still," which
starred a huge metal robot named Gort. He had laser-weapon eyes, zapped
terrified humans into oblivion and had the power to destroy the world.
Klaatu, Gort's alien master, was trying to deliver an important message
to earthlings, but they kept shooting him before he could. Finally, to
get the world's attention, Klaatu demonstrated the enormous power of the
alien races he represented by shutting down – at noon New York time –
all of the electricity on earth for exactly 30 minutes. The earth stood still.
Substitute
"ogle" for "rt," and you get "Google," which is every bit as powerful
as Gort but with a much better public relations department – so good, in
fact, that you are probably unaware that on Jan. 31, 2009, Google blocked access to virtually the entire internet. And, as if not to be outdone by a 1951 science fiction move, it did so for 40 minutes.
Impossible, you say. Why would do-no-evil Google do such an apocalyptic thing, and, for that matter, how, technically, could a single company block access to more than 100 million websites?
The answer has to do with the dark and murky world of website blacklists – ever-changing
lists of websites that contain malicious software that might infect or
damage people's computers. There are many such lists – even tools, such
as blacklistalert.org, that scan
multiple blacklists to see if your IP address is on any of them. Some
lists are kind of mickey-mouse – repositories where people submit the
names or IP addresses of suspect sites. Others, usually maintained by
security companies that help protect other companies, are more
high-tech, relying on "crawlers" – computer programs that continuously
comb the internet.
But the best and longest list of suspect websites is Google's, launched in May 2007. Because Google is crawling the web more extensively than anyone else, it is also in the best position to find malicious websites. In 2012, Google acknowledged
that each and every day it adds about 9,500 new websites to its
quarantine list and displays malware warnings on the answers it gives to
between 12 and 14 million search queries. It won't reveal the exact
number of websites on the list, but it is certainly in the millions on
any given day.
In 2011, Google blocked an entire subdomain,
co.cc, which alone contained 11 million websites, justifying its action
by claiming that most of the websites in that domain appeared to be
"spammy." According to Matt Cutts,
still the leader of Google's web spam team, the company "reserves the
right" to take such action when it deems it necessary. (The right? Who
gave Google that right?)
And that's nothing: According to The Guardian, on Saturday, Jan. 31, 2009, at 2:40 pm GMT, Google blocked the entire internet
for those impressive 40 minutes, supposedly, said the company, because
of "human error" by its employees. It would have been 6:40 am in
Mountain View, California, where Google is headquartered. Was this time
chosen because it is one of the few hours of the week when all of the
world's stock markets are closed? Could this have been another of the many pranks for which Google employees are so famous? In 2008, Google invited the public to submit applications to join the "first permanent human colony on Mars." Sorry, Marsophiles; it was just a prank.
When
Google's search engine shows you a search result for a site it has
quarantined, you see warnings such as, "The site ahead contains malware"
or "This site may harm your computer" on the search result. That's
useful information if that website actually contains malware, either
because the website was set up by bad guys or because a legitimate site
was infected with malware by hackers. But Google's crawlers often make mistakes,
blacklisting websites that have merely been "hijacked," which means the
website itself isn't dangerous but merely that accessing it through the
search engine will forward you to a malicious site. My own website, http://drrobertepstein.com,
was hijacked in this way in early 2012. Accessing the website directly
wasn't dangerous, but trying to access it through the Google search
engine forwarded users to a malicious website in Nigeria. When this
happens, Google not only warns you about the infected website on its
search engine (which makes sense), it also blocks you from accessing the
website directly through multiple browsers – even non-Google browsers. (Hmm. Now that's odd. I'll get back to that point shortly.)
The
mistakes are just one problem. The bigger problem is that even though
it takes only a fraction of a second for a crawler to list you, after
your site has been cleaned up Google's crawlers sometimes take days or
even weeks to delist you – long enough to threaten the existence of some
businesses. This is quite bizarre considering how rapidly automated
online systems operate these days. Within seconds after you pay for a
plane ticket online, your seat is booked, your credit card is charged,
your receipt is displayed and a confirmation email shows up in your
inbox – a complex series of events involving multiple computers
controlled by at least three or four separate companies. But when you
inform Google's automated blacklist system that your website is now
clean, you are simply advised to check back occasionally to see if any
action has been taken. To get delisted after your website has been
repaired, you either have to struggle with the company's online
Webmaster tools, which are far from friendly, or you have to hire a
security expert to do so – typically for a fee ranging between $1,000
and $10,000. No expert, however, can speed up the mysterious delisting
process; the best he or she can do is set it in motion.
So
far, all I've told you is that Google's crawlers scan the internet,
sometimes find what appear to be suspect websites and put those websites
on a quarantine list. That information is then conveyed to users
through the search engine. So far so good, except of course for the
mistakes and the delisting problem; one might even say that Google is
performing a public service, which is how some people who are familiar
with the quarantine list defend it. But I also mentioned that Google
somehow blocks people from accessing websites directly through multiple
browsers. How on earth could it do that? How could Google block you when
you are trying to access a website using Safari, an Apple product, or
Firefox, a browser maintained by Mozilla, the self-proclaimed "nonprofit defender of the free and open internet"?
The
key here is browsers. No browser maker wants to send you to a malicious
website, and because Google has the best blacklist, major browsers such
as Safari and Firefox – and Chrome, of course, Google's own browser, as
well as browsers that load through Android, Google's mobile operating
system – check Google's quarantine list before they send you to a
website. (In November 2014, Mozilla announced it will no longer list
Google as its default search engine, but it also disclosed that it will continue to rely on Google's quarantine list to screen users' search requests.)
If
the site has been quarantined by Google, you see one of those big,
scary images that say things like "Get me out of here!" or "Reported
attack site!" At this point, given the default security settings on most
browsers, most people will find it impossible to visit the site – but
who would want to? If the site is not on Google's quarantine list, you
are sent on your way.
OK, that explains how Google blocks you even when you're using a non-Google browser, but why
do they block you? In other words, how does blocking you feed the
ravenous advertising machine – the sine qua non of Google's existence?
Have
you figured it out yet? The scam is as simple as it is brilliant: When a
browser queries Google's quarantine list, it has just shared
information with Google. With Chrome and Android, you are always giving
up information to Google, but you are also doing so even if you are
using non-Google browsers. That is
where the money is – more information about search activity kindly
provided by competing browser companies. How much information is shared
will depend on the particular deal the browser company has with Google.
In a maximum information deal, Google will learn the identity of the
user; in a minimum information deal, Google will still learn which
websites people want to visit – valuable data when one is in the
business of ranking websites. Google can also charge fees for access to
its quarantine list, of course, but that's not where the real gold is.
Chrome, Android, Firefox and Safari currently carry about 92 percent of all browser traffic
in the U.S. – 74 percent worldwide – and these numbers are increasing.
As of this writing, that means Google is regularly collecting
information through its quarantine list from more than 2.5 billion
people. Given the recent pact
between Microsoft and Google, in coming months we might learn that
Microsoft – both to save money and to improve its services – has also
started using Google's quarantine list in place of its own much smaller
list; this would further increase the volume of information Google is
receiving.
To
put this another way, Google has grown, and is still growing, on the
backs of some of its competitors, with end users oblivious to Google's
antics – as usual. It is yet another example of what I have called "Google's Dance"
– the remarkable way in which Google puts a false and friendly public
face on activities that serve only one purpose for the company:
increasing profit. On the surface, Google's quarantine list is yet
another way Google helps us, free of charge, breeze through our day safe
and well-informed. Beneath the surface, that list is yet another way
Google accumulates more information about us to sell to advertisers.
You
may disagree, but in my view Google's blacklisting practices put the
company into the role of thuggish internet cop – a role that was never
authorized by any government, nonprofit organization or industry
association. It is as if the biggest bully in town suddenly put on a
badge and started patrolling, shuttering businesses as it pleased, while
also secretly peeping into windows, taking photos and selling them to
the highest bidder.
Consider: Heading into the holiday season in late 2013, an online handbag business suffered a 50 percent drop in business because of blacklisting. In 2009, it took an eco-friendly pest control company
60 days to leap the hurdles required to remove Google's warnings, long
enough to nearly go broke. And sometimes the blacklisting process
appears to be personal: In May 2013, the highly opinionated PC Magazine
columnist John Dvorak wondered "When Did Google Become the Internet Police?"
after both his website and podcast site were blacklisted. He also ran
into the delisting problem: "It's funny," he wrote, "how the site can be
blacklisted in a millisecond by an analysis but I have to wait forever
to get cleared by the same analysis doing the same scan. Why is that?"
Could Google really be arrogant enough to mess with a prominent journalist? According to CNN,
in 2005 Google "blacklisted all CNET reporters for a year after the
popular technology news website published personal information about one
of Google's founders" – Eric Schmidt – "in a story about growing
privacy concerns." The company declined to comment on CNN's story.
Google's
mysterious and self-serving practice of blacklisting is one of many
reasons Google should be regulated, just as phone companies and credit
bureaus are. The E.U.'s recent antitrust actions against Google, the recently leaked FTC staff report
about Google's biased search rankings, President Obama's call for
regulating internet service providers – all have merit, but they
overlook another danger. No one company, which is accountable to its
shareholders but not to the general public, should have the power to
instantly put another company out of business or block access to any website in the world. How frequently Google acts irresponsibly is beside the point; it has the ability
to do so, which means that in a matter of seconds any of Google's
37,000 employees with the right passwords or skills could laser a
business or political candidate into oblivion or even freeze much of the
world's economy.
Some
degree of censorship and blacklisting is probably necessary; I am not
disputing that. But the suppression of information on the internet needs
to be managed by, or at least subject to the regulations of,
responsible public officials, with every aspect of their operations
transparent to all.