Internet's well-guarded secret: It's a power-guzzler & polluter

Worldwide, the digital warehouses use electricity equivalent to the output of 30 nuclear power plants

Jeff Rothschild’s machines at Facebook had a problem he knew he had to solve immediately. They were about to melt. The company had been packing a 40-by-60-foot rental space at Santa Clara, Calif., with racks of computer servers that were needed to store and process information from members’ accounts.

The electricity pouring into the computers was overheating Ethernet sockets and other crucial components.

Thinking fast, Rothschild, the company’s engineering chief, took some employees on an expedition to buy every fan they could find – “We cleaned out all of the Walgreens in the area,” he said – to blast cool air at the equipment and prevent the website from going down.

That was in early 2006, when Facebook had a quaint 10 million or so users and the one main server site. Today, the information generated by nearly 1 billion people requires outsize versions of these facilities, called data centres, with rows and rows of servers spread over hundreds of thousands of square feet, and all with industrial cooling systems.

They are a mere fraction of the tens of thousands of data centres that now exist to support the overall explosion of digital information. Stupendous amounts of data are set in motion each day as, with an innocuous click or tap, people download movies on iTunes, check credit card balances on Visa’s website, send Yahoo email with files attached, buy products on Amazon, post on Twitter or read newspapers online.

A yearlong examination by The New York Times has revealed that this foundation of the information industry is sharply at odds with its image of sleek efficiency and environmental friendliness.

Most data centres, by design, consume vast amounts of energy in an incongruously wasteful manner, interviews and documents show. Online companies typically run their facilities at maximum capacity around the clock, whatever the demand. As a result, data centres can waste 90 per cent or more of the electricity they pull off the grid, The Times found.

To guard against a power failure, they further rely on banks of generators that emit diesel exhaust. The pollution from data centres has increasingly been cited by the authorities for violating clean air regulations, documents show. In Silicon Valley, many data centres appear on the state government’s Toxic Air Contaminant Inventory, a roster of the area’s top stationary diesel polluters.

Worldwide, the digital warehouses use about 30 billion watts of electricity, roughly equivalent to the output of 30 nuclear power plants, according to estimates industry experts compiled for The Times. Data centres in the United States account for one-quarter to one-third of that load, the estimates show.

“It’s staggering for most people, even people in the industry, to understand the numbers, the sheer size of these systems,” said Peter Gross, who helped design hundreds of data centres. “A single data centre can take more power than a medium-size town.”

Energy efficiency varies widely from company to company. But at the request of The Times, the consulting firm McKinsey & Co. analysed energy use by data centres and found that, on average, they were using only 6 to 12 per cent of the electricity powering their servers to perform computations. The rest was essentially used to keep servers idling and ready in case of a surge in activity that could slow or crash their operations.

A server is a sort of bulked-up desktop computer, minus a screen and keyboard, that contains chips to process data. The study sampled some 20,000 servers in about 70 large data centres spanning the commercial gamut: drug companies, military contractors, banks, media companies and government agencies. “This is an industry dirty secret, and no one wants to be the first to say mea culpa,” said a senior industry executive who asked not to be identified to protect his company’s reputation. “If we were a manufacturing industry, we’d be out of business straightaway.”

These physical realities of data are far from the mythology of the Internet: where lives are lived in the “virtual” world and all manner of memory is stored in “the cloud.” The inefficient use of power is largely driven by a symbiotic relationship between users who demand an instantaneous response to the click of a mouse and companies that put their business at risk if they fail to meet that expectation.

Use of generators

Even running electricity at full throttle has not been enough to satisfy the industry. In addition to generators, most large data centres contain banks of huge, spinning flywheels or thousands of lead-acid batteries – many of them similar to automobile batteries – to power the computers in case of a grid failure as brief as a few hundredths of a second, an interruption that could crash the servers.

“It’s a waste,” said Dennis P Symanski, a senior researcher at the Electric Power Research Institute, a nonprofit industry group. “It’s too many insurance policies.” At least a dozen major data centres have been cited for violations of air quality regulations in Virginia and Illinois alone, according to state records. Amazon was cited with more than 24 violations over a three-year period in Northern Virginia, including running some of its generators without a basic environmental permit.
A few companies say they are using extensively re-engineered software and cooling systems to decrease wasted power. Among them are Facebook and Google, which also have redesigned their hardware. Still, according to recent disclosures, Google’s data centres consume nearly 300 million watts and Facebook’s about 60 million watts.

Many of these solutions are readily available, but in a risk-averse industry, most companies have been reluctant to make wholesale change, according to industry experts. Improving or even assessing the field is complicated by the secretive nature of an industry that is largely built around accessing other people’s personal data.
For security reasons, companies typically do not even reveal the locations of their data centres, which are housed in anonymous buildings and vigilantly protected. Companies also guard their technology for competitive reasons, said Michael Manos, a longtime industry executive. “All of those things play into each other to foster this closed, members-only kind of group,” he said.

That secrecy often extends to energy use. To further complicate any assessment, no single government agency has the authority to track the industry. In fact, the federal government was unable to determine how much energy its own data centres consume, according to officials involved in a survey completed last year. The survey did discover that the number of federal data centres grew from 432 in 1998 to 2,094 in 2010.

To investigate the industry, The Times obtained thousands of pages of local, state and federal records, some through freedom of information laws, that are kept on industrial facilities that use large amounts of energy.

Copies of permits for generators and information about their emissions were obtained from environmental agencies, which helped pinpoint some data centre locations and details of their operations.

In addition to reviewing records from electrical utilities, The Times also visited data centres across the country and conducted hundreds of interviews with current and former employees and contractors. Some analysts warn that as the amount of data and energy use continue to rise, companies that do not alter their practices could eventually face a shake-up in an industry that has been prone to major upheavals, including the bursting of the first Internet bubble in the late 1990s.

“It’s just not sustainable,” said Mark Bramfitt, a former utility executive who now consults for the power and information technology industries. “They’re going to hit a brick wall.”

The high cost of internet

Set in the dry hills and irrigated farmland of central Washington, Grant County in the Quincy province, is known for its robust harvest of apples, potatoes, cherries and beans. But for Microsoft, a prime lure was the region’s other valuable resource: cheap electrical power.

The technology giant created a stir here in 2006 when it bought about 75 acres of bean fields to build a giant data centre, a digital warehouse to support various Internet services. Its voracious appetite for electricity would be fed by hydroelectric generators that work off the flow of the nearby Columbia River, and Microsoft officials pledged to operate their new enterprise with a focus on energy efficiency and environmental sensitivity.

“You’re talking about one of the largest corporations,” said Tim Culbertson, who was the general manager of the local utility at the time. “You’re talking Microsoft and Bill Gates. Wow!”

But for some in Quincy, the gee-whiz factor of such a prominent high-tech neighbour wore off quickly. First, a citizens’ group initiated a legal challenge over pollution from some of nearly 40 giant diesel generators that Microsoft’s facility – near an elementary school – is allowed to use for backup power. Then came a showdown late last year between the utility and Microsoft, whose hardball tactics shocked some local officials.

In an attempt to erase a $210,000 penalty the utility said the company owed for underestimating its power use, Microsoft proceeded to simply waste millions of watts of electricity, records show. Then it threatened to continue burning power in what it acknowledged was an “unnecessarily wasteful” way until the fine was substantially cut, according to documents obtained by The New York Times.

“For a company of that size and that nature, and with all the ‘green’ things they advertised to me, that was an insult,” said Randall Allred, a utility commissioner and local farmer. A Microsoft spokeswoman said the episode was “a one-time event that was quickly resolved.”

Internet-based industries have honed a reputation for sleek, clean convenience based on the magic they deliver to screens everywhere. At the heart of every Internet enterprise are data centres, which have become more sprawling and ubiquitous as the amount of stored information explodes, sprouting in community after community.

But the Microsoft experience in Quincy shows that when these Internet factories come to town, they can feel a bit more like old-time manufacturing than modern magic. In Santa Clara, Calif., a hub of technology facilities in Silicon Valley, diesel emissions from generators at a Microsoft data centre landed the company on a list of polluters for potentially threatening the health of workers at nearby businesses. Microsoft, which was notified by state regulators last year, says it has reduced its emissions.

Unlikely technology outpost

Over the last few years, Quincy has become an unlikely technology outpost, with five data centres and a sixth under construction. Far from the software meccas of Northern California or Seattle, Quincy has barely 6,900 residents, two hardware stores, two supermarkets, no movie theatre and a main drag, State Route 28, whose largest buildings are mostly food packers and processors. Its tallest building is a grain elevator.

“A farming community in the middle of a desert,” said Warren Morgan, the president of Double Diamond Fruit. The remarkable scale of the Quincy data centres, and their power demands, have made this town something of a test tube for studying the planet’s exploding need to house and process digital information.

Microsoft’s operation has now spread to four buildings and is the largest of Quincy’s data centres. Taken together, Microsoft and Yahoo’s operations overwhelm all nonindustrial electric usage, utility figures show. All residential and small commercial accounts in Quincy consumed an average of 9.5 million watts last year, while Microsoft and Yahoo used 41.8 million watts, the utility said.

The loads are growing so fast that some local residents and business owners – particularly irrigation farmers, who also depend on low-cost electricity – are concerned. With other industries also chasing low electricity prices, the increases could lead to higher prices or even a shortage of available power from the dams.
Not long after Microsoft arrived in 2006, Robert Koster, an environmental engineer in the Spokane office of the Washington state Department of Ecology, was assigned to review the company’s request for permits for 24 diesel generators.

Such huge backup generators, which can weigh thousands of pounds and stand over 10 feet tall, produce thousands of horsepower – enough to generate 2 million to 3 million watts each.

Although emissions containing diesel particulates are an environmental threat, they were not yet classified as toxic pollutants in Washington. The original permit did not impose stringent limits, allowing Microsoft to operate its generators for a combined total of more than 6,000 hours a year for “emergency backup electrical power” or unspecified “maintenance purposes.”

Even so, as Quincy’s data centres grow, so do its diesel generators. The state Ecology Department says enough permits for generating power have been issued in Quincy to eventually pump out 337 million watts – roughly a third of the output of a major nuclear power plant. Martin and her group are now challenging generators for an expansion of Yahoo’s data centre as well as for facilities run by Dell and Sabey.
Those cases are pending, according to the state attorney general’s office.

Meanwhile, some Ecology Department officials involved in approving the backup generators now express doubts about the soaring number of them in Quincy.
“I find it hard to believe that this is the best way to store data,” Koster said. “Something’s flawed in that thought process.”