Putting your datacentre on an energy diet
- 22 August, 2007 22:00
Sal Azzaro, director of facilities for Time Warner Cable, is trying to cram additional power into prime real estate at the company's 22 facilities in New York.
"Its gone wild," says Azzaro "Where we had 20-amp circuits before, we now have 60-amp circuits." And, he says, "there is a much greater need now for a higher level of redundancy and a higher level of fail-safe than ever before."
If Time Warner Cable's network loses power, not only do televisions go black, but businesses can't operate and customers can't communicate over the company's VoIP and broadband connections.
When it comes to the power crunch, Time Warner Cable is in good company. In February, Jonathan Koomey, a staff scientist at Lawrence Berkeley National Laboratory and a consulting professor at Stanford University, published a study showing that in 2005, organisations worldwide spent $7.2 billion to provide their servers and associated cooling and auxiliary equipment with 120 billion kilowatt-hours of electricity. This was double the power used in 2001.
According to Koomey, the growth is occurring among volume servers (those that cost less than US$25,000 per unit), with the aggregate power consumption of midrange ($25,000 to $500,000 per unit) and high-end (over $500,000) servers remaining relatively constant.
One way Time Warner Cable is working on this problem is by installing more modular power gear that scales as its needs grow. Oversized power supplies, power distribution units (PDU) and uninterruptible power supplies (UPS) tie up capital funds, are inefficient and generate excess heat. Time Warner Cable has started using Liebert Corp.'s new NX modular UPS system, which scales in 20-kilowatt increments, to replace some of its older units.
"The question was how to go forward and rebuild your infrastructures when you have a limited amount of space," Azzaro says.
With the NX units, instead of setting up two large UPSs, he set up five modules — three live and the other two on hot standby. That way, any two of the five modules could fail or be shut down for service and the system would still operate at 100 percent load.
Some users are trying innovative approaches. One new way of approaching the power issue is a technique called combined heat and power, or cogeneration, which combines a generator with a specialized chiller that turns the exhausted waste heat into a source of chilled water. (See related story.)
Another new approach is to build data centres that operate off DC rather than AC power. In a typical data centre, the UPSs convert the AC power coming from the mains into DC power and then back into AC again. Then the server power supplies again convert the power to DC for use within the server.
Each time the electricity is switched between AC and DC, some of that power is converted into heat. Converting the AC power to DC power just once, as it comes into the data centre, eliminates that waste. Rackable Systems Inc. in Fremont, Calif. has a rack-mounted power supply that converts power from 220 VAC to -48VDC in the cabinet and then distributes the power via a bus bar to the servers.
On a larger scale, last summer the Lawrence Berkeley lab set up an experimental data centre, hosted by Sun Microsystems Inc., that converted incoming 480 VAC power to 380 VDC power for distribution to the racks, eliminating the use of PDUs altogether. Overall, the test system used 10 percent to 20 percent less power than a comparable AC data centre.
For Rick Simpson, president of Belize Communication and Security in Belmopan, Belize, power management means using wind and solar energy.
Simpson's company supports wireless data and communications relays in the Central American wilderness for customers including the U.K. Ministry of Defence and the U.S. embassy in Belize. He builds in enough battery power -- 10,000 amp hours -- to run for two weeks before even firing up the generators at the admittedly small facility.
"We have enough power redundancy at hand to make sure that nothing goes down -- ever," Simpson says. So even though the country was hit by Category 4 hurricanes in 2000 and 2001, "we haven't been down in 15 years," he says.
Belize Communications equipment all runs directly off the batteries and UPSs from Falcon Electric in Irwindale, California Mains power is only used to charge the batteries.
While there is a lot of talk lately about building green data centres, and many hardware vendors are touting the efficiency of their products, the primary concern is still just ensuring you have a reliable source of adequate power.
Even though each core on a multicore processor uses less power than it would if it was on its own motherboard, a rack filled with quad-core blades consumes more power than a rack of single-core blades, according to Intel.
"It used to be you would have one power cord coming into the cabinet, then there were dual power cords," says Bob Sullivan, senior consultant at The Uptime Institute in Santa Fe, New Mexico "Now with over 10 kilowatts being dissipated in a cabinet, it is not unusual to have four power cords, two A's and two B's."
With electricity consumption rising, data centres are running out of power before they run out of raised floor space. A Gartner survey last year showed that half of data centres will not have sufficient power for expansion by 2008.
"Power is becoming more of a concern," says Dan Agronow, chief technology officer at The Weather Channel Interactive in Atlanta. "We could put way more servers physically in a cabinet than we have power for those servers."
The real cost, however, is not just in the power being used but in the costs of the infrastructure equipment -- generators, UPSs, PDUs, cabling and cooling systems. For the highest level of redundancy and reliability -- a Tier 4 data centre -- for every kilowatt used for processing, some $22,000 is spent on power and cooling infrastructure.
There are strategies for cutting power in each area Yellin outlined above. For example, multicore processors with lower clock speeds reduce power at the processor level. And server virtualisation, better fans and high-efficiency power supplies -- such as those certified by the 80 Plus program -- cut power utilization at the server level.
Five years ago, the average power supply was operating at 60 percent to 70 percent efficiency, says Kent Dunn, partnerships director at PC power-management firm Verdiem Corp. in Seattle, Washington and program manager for 80 Plus. He says that each 80 Plus power supply will save data centre operators about 130 to 140 kilowatt-hours of power per year.
Rack-mounted cooling and power supplies such as Liebert's XD CoolFrame and American Power Conversion's InfraStruXure cut waste at the rack level. And at the data centre level, there are more efficient ways of distributing air flow, using outside air or liquid cooling, and doing computational fluid dynamics modelling of the data centre for optimum placement of servers and air ducts.
"We've deployed a strategy within our facility that has hot and cold aisles, so the cold air is where it needs to be and we are not wasting it," says Fred Duball, director of the service management organization for the Virginia state government's IT agency, which just opened a 192,000-square-foot data centre in July and will be ramping up the facility over the next year or so. "We are also using automation to control components and keep lights off in areas that don't need lights on."
Finding a fit
There is no single answer that meets the needs of every data centre operator.
When Elbert Shaw, a project manager at Science Applications International in San Diego, consolidated U.S. Army IT operations in Europe for several dozen locations into four data centres, he had to come up with a unique solution for each location. At a new facility, he was able to put in 48-inch floors and run the power and cooling underneath. But one data centre being renovated only had room for a 12-inch floor and two feet of space above the ceiling. So instead of bundling the cables, which could have eaten up eight of those 12 inches, blocking most of the airflow, he got permission to unbundle and flatten out the cables. In other instances he used 2-inch underfloor channels, rather than the typical 4-inch variety, and turned to overhead cabling at one location.
"Little tricks that are OK in the 48-inch floor cause problems with the 12-inch floor when you renovate a site," says Shaw.
"These facilities are unique, and each has its own little quirks," Koomey says.
Power management tips
Various experts suggest the following ways of getting all you can from your existing power setup:
• Don't oversize. Adopt a modular strategy for power and cooling that grows with your needs instead of buying a monolithic system that will meet your needs years down the road.
• Plan for expansion. Although you don't want to buy the extra equipment yet, install conduits that are large enough to accommodate additional cables to meet future power needs.
• Look at each component. Power-efficient CPUs, power supplies and fans reduce the amount electricity used by a server. But be sure to look at their impact on other components. For example, quad-core chips use less power than four single chips but may require additional memory.
• Widen racks. Use wider racks and run the cables to the side, rather than down the back where they block the air flow. Air flows from the front of a server, through the box and out the back. There are no inlet and outlet vents on the sides. It is similar to a PC, and you can put a piece of plywood along the side of server without affecting airflow through the machine. But, if you put the wood along the back, it will overheat.
• Install a UPS bypass. This is a power cable that goes around the UPS rather than through it. That way, if the UPS is taken offline, there's still a route available for the electricity to flow through and you have power redundancy when you bring a UPS device down for maintenance.