The dirty little secrets of search

Pretend for a moment that you are Google’s search engine. Someone types the word “dresses” and hits enter. What will be the very first result?

There are  of course a lot of possibilities. Macy’s comes to mind. Maybe a specialty chain, like J Crew or the Gap. Perhaps a Wikipedia entry on the history of hemlines.

OK, how about the word “bedding”? Bed Bath & Beyond seems a candidate or Walmart, or perhaps the bedding section of Amazon.com.

“Area rugs”? Crate & Barrel is a possibility. Home Depot, too, and Sears, Pier 1 or any of those websites with “area rug” in the name, like arearugs.com.

You could imagine a dozen contenders for each of these searches. But in the past several months, one name turned up, with uncanny regularity, in the No 1 spot for each and every term: J C  Penney

The company bested millions of sites – and not just in searches for dresses, bedding and area rugs. For months, it was consistently at or near the top in searches for “skinny jeans,” “home decor,” “comforter sets,” “furniture” and dozens of other words and phrases, from the blandly generic (“tablecloths”) to the strangely specific (“grommet top curtains”).

This striking performance lasted for months, most crucially through the holiday season, when there is a huge spike in online shopping.

J C Penney even beat out the sites of manufacturers in search for the products of those manufacturers. Type in “Samsonite carry on luggage,” for instance, and Penney for months was first on the list, ahead of Samsonite.com.

With more than 1,100 stores and $17.8 billion in total revenue in 2010, Penney is certainly a major player in American retailing.

But Google’s stated goal is to sift through every corner of the Internet and find the most important, relevant websites.

Does the collective wisdom of the Web really say that Penney has the most essential site when it comes to dresses? And bedding? And area rugs? And dozens of other words and phrases?

The New York Times asked an expert in online search, Doug Pierce of Blue Fountain Media in New York, to study this question, as well as Penney’s astoundingly strong search-term performance in recent months. What he found suggests that the digital age’s most mundane act, the Google search, often represents layer upon layer of intrigue.

And the intrigue starts in the sprawling, subterranean world of “black-hat” optimisation, the dark art of raising the profile of a website with methods that Google considers tantamount to cheating.

Despite the cowboy outlaw connotations, black-hat services are not illegal, but trafficking in them risks the wrath of Google. The company draws a pretty thick line between techniques it considers deceptive and “white-hat” approaches, which are offered by hundreds of consulting firms and are legitimate ways to increase a site’s visibility.

Penney’s results were derived from methods on the wrong side of that line, says Pierce. He described the optimisation as the most ambitious attempt to game Google’s search results that he has ever seen.

“Actually, it’s the most ambitious attempt I’ve ever heard of,” he said.  “This whole thing just blew me away. Especially for such a major brand. You’d think they would have people around them that would know better.” To understand the strategy that kept J C Penney in the pole position for so many searches, you need to know how websites rise to the top of Google’s results. We’re talking to be clear about the “organic” results – in other words, the ones that are not paid advertisements. In deriving organic results, Google’s algorithm takes into account dozens of criteria, many of which the company will not discuss.

But it has described one crucial factor in detail: links from one website to another.
If you own a website, for instance, about Chinese cooking, your site’s Google ranking will improve as other sites link to it. In a way, what Google is measuring is your site’s popularity by polling the best-informed online fans of Chinese cooking and counting their links to your site as votes of approval.

But even links that have nothing to do with Chinese cooking can bolster your profile if your site is barnacled with enough of them. And here’s where the strategy that aided Penney comes in. Someone paid to have thousands of links placed on hundreds of sites scattered around the Web, all of which lead directly to JCPenney.com.

Who is that someone? A spokeswoman for J C Penney, Darcie Brossart, says it was not Penney. “J C Penney did not authorise, and we were not involved with or aware of, the posting of the links that you sent to us, as it is against our natural search policies,” Brossart wrote in an e-mail.

The links do not bear any fingerprints, but nothing else about them was particularly subtle. Using an online tool called Open Site Explorer, Pierce found 2,015 pages with phrases like “casual dresses,” “evening dresses,” “little black dress” or “cocktail dress.” Click on any of these phrases on any of these 2,015 pages, and you are bounced directly to the main page for dresses on JCPenney.com.

Some of the 2,015 pages are on sites related, at least nominally, to clothing. But most are not. The phrase “black dresses” and a Penney link were tacked to the bottom of a site called nuclear.engineeringaddict.com. “Evening dresses” appeared on a site called casino-focus.com. “Cocktail dresses” showed up on bulgariapropertyportal.com.

 “Casual dresses” was on a site called elistofbanks.com. “Semi-formal dresses” was pasted, rather incongruously, on usclettermen.org.

There are links to JCPenney.com’s dresses page on sites about diseases, cameras, cars, dogs, aluminum sheets, travel, snoring, diamond drills, bathroom tiles, hotel furniture, online games, commodities, fishing, Adobe Flash, glass shower doors, jokes and dentists – and the list goes on.

When you read the enormous list of sites with Penney links, the landscape of the Internet acquires a whole new topography. It starts to seem like a city with a few familiar, well-kept buildings, surrounded by millions of hovels kept upright for no purpose.
Exploiting those hovels for links is a Google no-no. The company’s guidelines warn against using tricks to improve search engine rankings.

In 2006, Google announced that it had caught BMW using a black-hat strategy to bolster the company’s German website, BMW.de. That site was temporarily given what the BBC at the time called “the death penalty,” stating that it was “removed from search results.”
BMW acknowledged that it had set up “doorway pages,” which exist just to attract search engines and then redirect traffic to a different site.

 The company at the time said it had no intention of deceiving users, adding “if Google says all doorway pages are illegal, we have to take this into consideration.”

J C Penney, it seems, will not suffer the same fate. But starting on Wednesday, it was the subject of what Google calls “corrective action.”

Last week, The Times sent Google the evidence it had collected about the links to JCPenney.com. Google promptly set up an interview with Matt Cutts, the head of the webspam team at Google, and a man whose every speech, blog post and Twitter update is parsed like papal encyclicals by players in the search engine world.

“I can confirm that this violates our guidelines,” Cutts said during an interview on Wednesday, after looking at a list of paid links to JCPenney.com.

He said Google had detected previous guidelines violations related to JCPenney.com on three occasions. Each time, steps were taken that reduced Penney’s search results – Cutts avoids the word “punished” – but Google did not later “circle back” to the company to see if it was still breaking the rules, he said.

Cutts sounded remarkably upbeat and unperturbed during this conversation, which was a surprise given that we were discussing a large, sustained effort to snooker his employer.

“Is Google going to take strong corrective action?  “We absolutely will.”

And the company did. On Wednesday evening, Google began what it calls a “manual action” against Penney, essentially demotions specifically aimed at the company.
At 7 pm on Wednesday, J C Penney was still the No 1 result for “Samsonite carry on luggage.”Two hours later, it was at No 71 and it was No 1 in searches for “living room furniture.” By 9 pm, it had sunk to No 68.

Penney reacted to this instant reversal of fortune by firing its search engine consulting firm, SearchDex. Executives there did not return e-mail or phone calls.

Penney also issued a statement: “We are disappointed that Google has reduced our rankings due to this matter,” Search experts say Penney likely reaped substantial rewards from the paid links. If you think of Google as the entrance to the planet’s largest shopping center, the links helped Penney appear as though it was the first and most inviting spot in the mall, to millions of online shoppers.

How valuable was that? A study in May by Daniel Ruby of Chitika, an online advertising network of 100,000 sites, found that, on average, 34 per cent of Google’s traffic went to the No 1 result, about twice the percentage that went to No 2.

The Keyword Estimator at Google puts the number of searches for “dresses” in the United States at 11.1 million a month, an average based on 12 months of data.
So for “dresses” alone, Penney may have been attracting roughly 3.8 million visits every month it showed up as No. 1. Exactly how many of those visits translate into sales, and the size of each sale, only Penney would know.

Last year, Advertising Age obtained a Google document that listed some of its largest advertisers, including AT&T, eBay and yes, J C Penney. The company spent $2.46 million a month on paid Google search ads.

Is it possible that Google was willing to countenance an extensive black-hat campaign because it helped one of its largest advertisers? It’s the sort of question that European Union officials are now studying in an investigation of possible antitrust abuses by Google.

Before The Times presented evidence of the paid links to JCPenney.com, Google had just begun to roll out an algorithm change that had a negative effect on Penney’s search results. (The tweak affected “how we trust links,” Cutts said, declining to elaborate.)