Everyone is looking at their data center efficiency and trying to quantify it and improve it. Of course, there are actually two separate groups addressing it from different positions, the facilities team and the IT group.
The facilities team is responsible for the power and cooling of the overall enclosed space. The IT group is in charge of the servers, storage and networking hardware. Typically, each side speaks to each other as little as possible-except when they have reached the limits of power or cooling (or both) in the server room or data center.
One group, The Green Grid, was created by both the IT equipment manufacturers and the power and cooling equipment manufacturers. The Green Grid has created, and has been promoting, the Power Usage Effectiveness (PUE) and Data Center Infrastructure Efficiency (DCiE) methods of measuring data center efficiency.
While I won’t delve into all the details of PUE calculations here, the basic premise is that it represents the ratio of the total power consumed (including uninterruptible power supply and cooling) by the data center, including the IT load itself, divided by the IT load. A simple example is that if the total load is 200 kilowatts (kW) and the IT load is 100 kW, a PUE of 2.0 would result. While the PUE can vary from 1.x to 3.x, a PUE of 2.0 is a fairly common operating ratio for many data centers.
However, even these new measurement standards, oddly enough, do not directly address any IT equipment efficiency-only the power and cooling equipment efficiency. And, while this article is not focused on data center infrastructure efficiency, it is important for the IT department to understand and consider that for every watt of IT equipment, the data center infrastructure requires additional energy to support it.
Even the United States government, after spending a significant amount of time and resources, has not been able to fully define and regulate the power efficiency of the data center and the servers and IT equipment (according to an EPA report to Congress in August 2007). It is still in the process and, according to the EPA ENERGY STAR Computer Server Stakeholder Meeting of July 9, 2008 in Redmond, WA, the first-tier rules are expected to become finalized in 2009.
In the rush to optimize, virtualize and consolidate in the name of making our computing-related operations more effective and efficient (and, of course, green), we have heard many server manufacturers profess that their products provide the most computing power for the least amount of energy. Only recently have the server manufacturers even begun to discuss or disclose the efficiency of their servers. Currently, there are no real standards for the overall energy efficiency of servers.
There are several key components which impact the total energy a typical server utilizes: these components are power supplies, fans, CPUs, memory, hard drives, I/O card and ports, and other motherboard components and supporting chip sets. These main components exist in both conventional servers and in blade servers.
However, in the case of the blade servers, some items-such as power supplies, fans and I/O ports-are shared on a common chassis, while the CPU and other related motherboard items are located on the individual blades. Depending on the design of the blade server, the hard drives can be located on either the chassis or the blades. In addition to the aforementioned list, the operating system and virtualization software will also impact the overall usable computing throughput of the hardware platform.
Every manufacturer likes to claim that their product or platform is the most energy-efficient. However, while each one may have a particular sweet spot (for example, the chip set may be more efficient with a particular operating system), overall they all utilize the same basic components and are in the same boat when it comes to the power being used by these components.
Of course, we all want the fastest, most powerful servers for our data center. So, although energy efficiency (for example, green) is the watchword, historically it would seem we only think about energy usage when our power and/or cooling systems are maxed out and may need to be upgraded. Normally, when we need to know how much power the server requires, we turn to the name plate. However, it just represents the maximum amount of power the unit could draw, not the actual power draw in reality. Let’s now examine what it really costs to operate a server.
What’s a Watt Worth: The Cost of Energy
What’s a watt worth: The cost of energy
We don’t always stop to think what it costs to operate a “small” server that typically can consume 500 watts of power. That server also needs to cool 500 watts of heat load (approximately 1,700 British thermal units or BTUs). So, for the typical data center that has a PUE of 2.0, that means that it uses one watt to support (power losses and cooling power) each watt of “plug power” to the IT load itself. This means that it takes 1,000 watts, or 1 kW, of power into the data center to run the 500-watt small server.
A single kW does not sound like much in a data center-until you begin to calculate that since it is consumed continuously (the proverbial 7x24x365), that it equals 8,760 kilowatt hours (kWh) per year! At 11.5 cents per kWh, one kW costs $1,000 per year (of course, 11.5 cents is just an average; in many areas the cost is much higher).
So, over a 3-year period, that one small 500-watt server can cost $3,000 or more just in energy costs. In fact, since many of these small servers cost less than $3,000, you can see why some analysts have predicted that the power to operate the server will exceed the cost of the server-especially as the cost of energy rises. Let’s now examine where the power goes and what we can do to optimize it.
Power supplies
The power supply is, of course, where the power enters the server and is converted from 120 to 240V AC to 3.3, 5 and 12V DC. Until just recently, the efficiency numbers were unpublished (and still only listed by some manufacturers). In fact, the EPA Energy Star Program, which mandated that all PCs have power supplies that were 80 percent efficient or greater, specifically exempted servers! This is one area where a few extra dollars spent to purchase a server with an 80 percent or greater efficiency rating can pay back large returns via the energy cost saving over the operational 3 to 5-year life of the server.
The difference between a 70 percent and an 87 percent efficient power supply will result in a 20 percent overall energy savings for the server power usage (assuming that same internal server load). This would also mean a similar range of overall energy reduction for the data center.
Moreover, these efficiency ratings are usually only provided at the power supply maximum-rated load. That does not reflect the actual operating loads that the server will be operating at in production. Typically, a server will be only drawing 30 to 50 percent of the maximum power supply rating (name plate), which means that the fixed losses in the power supply will result in less than the rated power supply efficiency value at full load.
Moreover, since we also want redundancy to improve uptime, we typically also order servers with redundant power supplies. These redundant power supplies normally load share the internal load, resulting in each power supply only supplying half of the actual load-which means that each power supply is only running at 13 to 25 percent of rated load. This means that the fixed losses are a greater percentage of the actual internal power being drawn by the internal server components.
When buying new servers, the lowest-cost unit may not be the best choice, even if the computing performance specifications are the same. So when specifying a new server, this is one of the best places to start saving energy. If the server vendor does not publish or cannot provide the efficiency of the power supply, think twice if the server really represents a good value.
In fact, if you are shopping for a large amount of servers, it would pay to invest in testing the actual total power drawn by the different manufacturers and models you are considering-specifically when loaded with your operating system and applications, both at idle and at full computing load. By spending an extra $50 to $100 on a more efficient server now, you may save several hundreds of dollars in total energy costs over the typical life of the server. Moreover, you may avoid having to upgrade your power and cooling infrastructure.
Another area that can save about two to three percent in energy usage is to operate the servers at 208V or 240V instead of 120V, since the power supplies are more efficient at the higher voltage (as well as the power distribution system).
Server Fans: How to Optimize the Power Used
Server fans: How to optimize the power used
Second only to the power supply, server fans have become a large user of power (other than the actual computing-related components themselves). As servers have become smaller and smaller-and now commonly pack several multi-core CPUs in a 1U-high server-the challenge of moving a sufficient amount of air through the server requires multiple small, high-velocity fans. They need to push air through very small restrictive airflow areas within the server and the very small intake and exhaust areas at the front and rear of the server chassis.
These fans can consume 10 to 15 percent or more of the total power drawn by the server. And since the fans are DC, they draw power from the power supply, thus increasing the input power to the server, again multiplied by the inefficiency of the power supply. In addition, in 1U servers, most or all of the airflow is routed through the power supply fans since there is virtually little or no free area on the rear panel to exhaust the hot air.
To improve efficiency, many new servers have thermostatically-controlled fans which raise the fan speed as more airflow is needed to cool the server. This is an improvement over the old method of fixed-speed server fans that run maximum speed all the time. However, these variable speed fans still require a lot of energy as internal heat loads rise and/or input air temperature rises.
For example, if the server’s internal CPUs and other computing-related components draw 250 to 350 watts from the power supply, the fans may require 30 to 75 watts to keep enough air moving through the server. This results in the overall increase in server power draw as heat density and air temperature rises in the data center. In fact, studies that have measured and plotted fan energy usage versus server power and inlet air temperatures show some very steep, fan-related power curves in temperature-controlled fans of small servers.
CPUs
The CPU is the heart of every server and the largest computing-related power draw. While both Intel and AMD offer many different families of CPUs, all with the goal of providing more computing power per watt, the overall power requirement of servers has continued to rise (since the demand for computing power has also risen).
For example, the power requirement for the Intel CPU varies from 40 to 80 watts for a Dual-Core Intel Xeon Processor to 50 to 120 watts for a Quad-Core Processor, depending on version and clock speed. As mentioned previously, many servers are configured with two, four or even eight dual or quad-core CPUs. And naturally, we all want the fastest servers we can buy today in the hope of having a 3-year usable life, before the next wave of software or applications overwhelms them.
It has been well documented that the average CPU is at idle over 90 percent of the time and only hits peak demand for very short periods, yet continuously draws a substantial portion of its maximum power requirement 24 hours a day. Moreover, even when servers are equipped with power-saving features in the hardware and software (as most servers are), these features are usually disabled by the server administrators.
One of the primary goals of virtualization is to decrease the number of servers that are mostly running at idle, and consolidate their function/application onto fewer, more powerful servers that run a higher average utilization rate.
Ultimately, the performance requirements and the types of computing loads your applications face will be the determining factor in your choice in the number and type of CPUs. Hopefully, by trying to match the computing load with the performance and number of CPUs, you will optimize the efficiency of each server.
Memory: How to Optimize the Power Used
Memory: How to optimize the power used
Memory is often overlooked as a factor in determining the overall actual power usage when specifying the configuration of a server. Memory chips vary widely from vendor to vendor and are usually not particularly well documented when it comes to power consumption. Generally speaking, the more memory per chip set module, the lower the power per GB of memory. Also, the faster the memory, the more power it draws (this is tied into the speed of the memory bus of the server and CPUs).
Ideally, try to get as much memory as your application will need, but do not maximize the memory on all the servers just based on the old adage that you can never have too much memory. Over-specified, unused memory increases initial costs and draws unnecessary power over the life of the server. Even though some larger memory chips cost somewhat more per GB, a larger, more power-efficient memory chip can lower the power used over the life of the server. In addition, if you do need to add more memory in the future, it will leave more sockets open.
Hard drives
The capacity, physical density and energy efficiency of hard drives have outpaced the performance increases of many other computing components. We seem to have an insatiable appetite for data storage, which means that it is almost a zero sum gain. However, the power required by the newer, small-form factor 2.5-inch drives is fairly low when compared to “full-size” 3.5-inch drives of only a generation ago. Also, since the magnetic density of the media continues to increase per platter, larger-capacity hard drives use the same energy as smaller-capacity drives (assuming the same drive type).
Spindle speed has a direct effect on power efficiency in the same class of drive, a 10K-RPM version, and either a 146GB or 300GB drive uses seven watts in use, and only 3.5 watts when idle. Unless you have a specialized application requirement requiring faster disk response, the 10K-RPM drive offers far more storage per watt for general purpose storage. Consider using the lower-power drives wherever possible as the power savings add up.
Recently, solid-state drives (SSD) for notebooks have increased in capacity to as much as 512 GB and also begun to come down in price. They will soon be making inroads into the server market and would result in even more energy saving-especially when compared to the 15K-RPM drives. Of course, check with your server vendor to see what your OEM drive options are.
I/O cards and ports
While most IT people do not think in terms of how much power the network interface card (NIC) or I/O cards draw, it is an opportunity to save several watts per server. Some servers come with embedded cards; others use add-on cards or a combination of both. When selecting a NIC card, we want the fastest throughput, usually without any consideration of power usage. For example, Intel makes several NIC cards ranging in power from the INTEL PRO/1000 PT (which draws only 3.3 watts) to a 10-Gigabit Dual Fiber 10GB XF card (which draws 14 watts).
In the case of OEM server NICs, a major manufacturer’s power estimator tool indicates 22 watts for their OEM PCI Gigabit Ethernet card. Since many servers have embedded NIC cards, they may or may not draw power even if they are disabled. If you intend to use multiple NICs for redundancy or throughput, a careful comparison of internal or OEM cards can save several watts per card.
Other motherboard components and supporting chip sets
Obviously, each server requires it own supporting chip sets which are required to form the complete system. It is beyond the scope of this article to try to compare the wide variety of systems on the market. This is where the different vendors can each tout their claims that their server is the most energy-efficient system on the market. If the system motherboard already is equipped with the majority of on-board NICs, RAID controller or other I/O devices to meet your requirements, then you may not need to add those additional cards.
Each major manufacturer now seems to have a power estimating tool for their servers. It is not meant to be an absolute indicator of the actual power that the server will draw, but it will provide a good estimate and a way to compare different components and configurations.
The bottom line
All these factors add up in determining the power your data center uses. By carefully comparing and selecting more efficient components and configurations options, you can potentially save significant power over time. Remember, by carefully specifying and configuring your servers to adequately meet-but not exceed-your computing requirements, each watt that you save per server can save you a significant amount of money per year. Or it could mean the difference between needing to upgrade the power and cooling in your data center or server room, or being able to continue to operate with the existing capacity of your infrastructure.
The last recommendation, and perhaps the most simple and effective method to save energy, is to review the status and purpose of every IT device in the data center. Many studies have shown that there are a significant number of servers and other IT devices that are no longer in production but are still powered up. No one seems to know what application or function they support but no one wants the responsibility of switching them off. So take a total device inventory regularly. You may find several servers, routers and switches that are unused and powered up. Once you find them, just turn them off.