Sri Donthi had a big surprise waiting for him when, just over two years ago, he showed up at Motorola Inc.s Personal Communication business unit to get started on a major systems consolidation and standardization project. Namely, nobody could tell him how many storage devices the business had, how many people were assigned to keep storage devices running, or even utilization levels of the units disk subsystems and other storage devices.
Talk about operating in the dark.
“It was bad, but thats the way it was,” said Donthi, who today is director of global computer operations at Motorola, in Schaumburg, Ill. “The first thing we needed to do was to figure out what was out there, utilization levels, resources applied to managing everything, and backup and business risk levels.”
So Donthi brought in consultants to help him conduct an audit of the business units storage environment. Not surprisingly, what they found wasnt pretty. On the 96 storage devices deployed from five vendors, only about 30 percent of available capacity was used, on average. Support staff was more than double what it needed to be.
From that information, Donthi and his team put together a plan to consolidate the sectors storage onto two Symmetrics disk subsystems from EMC Corp., of Hopkinton, Mass. The project more than doubled usage rates while significantly cutting support overhead, Donthi said.
Unfortunately, Donthi isnt the only cost-conscious IT manager who will have an ugly surprise in store when he or she finally gets serious about controlling escalating enterprise data storage costs. Experts estimate that only about 10 percent of enterprises in the United States have detailed, up-to-date information on how many storage devices they have and where theyre located, storage growth trends, usage rates, support costs, availability, and other performance metrics. Without such base-line information, its difficult, if not impossible, to put together a serious storage cost containment program. So, like Motorola, many organizations—with the help of service providers—are beginning to conduct TCO (total-cost-of-ownership) audits of their storage environments in much the same way that they conducted TCO studies in the late 1990s to understand and get a handle on runaway desktop support costs.
Theres a right way and a wrong way to conduct such audits, however. First, experts say, be sure to establish the objectivity of the service providers you call on to help. Second, be sure storage TCO audits tote up not just hardware and software costs but also support spending—which, in many organizations, represents by far the fastest-growing component of storage costs. Third, experts say, be sure you look at the business impacts of the current storage environment and anything that might replace it.
So why are enterprises getting serious now about cutting storage TCO, and why dont they have good information on storage costs to start with? The answer to the first question is easy. Even though storage hardware costs continue to decline on a per-gigabyte basis 25 percent to 35 percent per year, overall spending on storage is still accelerating. Gartner Inc. estimates that, by 2004, buying and managing storage will represent the majority of total server-oriented costs for the average enterprise.
Driving those rising costs are the sheer volume of data being generated by e-business and transaction-oriented applications, plus increasing support personnel costs. Evaluator Group Inc., an Englewood, Colo., storage consultancy, estimates storage support costs on average are rising 5 percent to 10 percent per year. In light of the ongoing economic slowdown, experts say, most enterprises are pushing to understand and cut storage costs.
“Storage is now the pain point for many organizations, and theyre casting about for relief,” said Nick Allen, an analyst at Gartner, in Stamford, Conn. “For many, a place to start is a storage audit.” Gartner, Allen said, came up with a storage TCO methodology and tool kit four years ago but shelved it when enterprise acceptance was lukewarm. The company has now revived the idea, however, and is about to publish a new storage TCO methodology.
The reason so many enterprises seem to know so little about their storage-related costs, experts say, has a lot to do with how storage has historically been purchased: in an ad hoc manner, usually to support individual applications and rarely with a view to improving enterprisewide use or cutting support costs. Rare is the IT organization that has designated anyone to be responsible for companywide storage planning, use and procurement.
“Very often, the CIO will be busy writing a check to purchase more storage for one system while, down the hall, another group has enough unused capacity to satisfy his needs for the next 12 months,” said Tom Sweeney, president and CEO of ManagedStorage International Inc., a storage services provider in Broomfield, Colo., that is conducting TCO audits.
While tools do exist to help IT managers track storage subsystem performance and use levels, many are too platform-specific and too expensive, experts say. One organization using such tools is VW Credit Inc., a subsidiary of Volkswagen of America Inc., in Libertyville, Ill. Since the company began using Sun Microsystems Inc.s HighGround Storage Resource Manager software four years ago, it has been able to recover the equivalent of 30GB of storage space, mainly by improving usage and capping individual user file sizes on its Microsoft Corp. Exchange and SQL Server applications running on Windows NT servers. The company, however, must also continue to use tools, such as Hewlett-Packard Co.s OpenView and EcoTools from Compuware Corp., to monitor storage resources on other platforms, said Windows server systems team leader Bill Wheeler.
To help IT managers establish base-line storage cost and performance information, storage hardware, software, service and consulting providers have begun to roll out storage TCO auditing methodologies and services. However, given the level of manual processes involved in simply tracking existing storage resources, usage levels and sometimes hidden support personnel, such auditing services are neither quick nor inexpensive. Storage hardware vendor EMC, which has been conducting storage TCO audits for enterprise customers for the past 18 months, spends three to four weeks on each audit and charges an average of $60,000.
Touch all the bases
Touch all the bases
The most important factor in conducting a storage TCO audit, many experts say, is to make sure youre taking a comprehensive look at all of the factors that influence storage costs as well as storage-related impacts on the business. Besides simply tracking all the storage devices and usage rates, for example, its important to look at the financial facts that go with the storage resources. Find out which part of the organization purchased the device and which business unit is benefiting from its use so you can put in place an accurate chargeback system to cover support costs. Compile information on when each piece of gear will be fully depreciated so you can plan a transition to new technology that makes financial sense.
Its also important, experts say, to detail all support costs, although doing so can be the most challenging part of putting together a storage TCO assessment. It can be difficult to track all administrators who have storage management as part of their responsibilities. In addition, attempting to do so can stir up a hornets nest of political conflict as support workers often become concerned about losing their jobs.
The answer at Motorola was for Donthi and his team to reassure potentially affected IT support workers that, even if the storage-related portion of their jobs was consolidated, there were plenty of opportunities for them—such as working on new customer relationship management projects at Motorola.
The focus on support costs has more than paid off for Motorola. Donthis team was able to reduce the number of employees supporting storage significantly. In the companys Midwest region alone, it has cut use of storage support staff by more than half, to about one person for every 20 terabytes of storage. So successful have the consolidation efforts been that the infrastructure consolidation effort that began with the companys Personal Communication sector has spread across the company. Donthis global computer operations is now responsible for providing shared data center services to all of Motorolas business units. In the last six months, the global computer operations unit has grown from 400 employees to 1,600.
Besides including support costs in storage TCO audits, its important to look at the business impact of the existing and future storage infrastructure, experts say. If, for example, key business processes such as order processing or supply chain execution could be made significantly more efficient by improving response times or improving data availability, what would be the savings to the business? Also, what improvements in IT costs—software development, for example—could be realized by consolidating storage in a SAN (storage area network) or by making it more efficient?
“We try to make [IT managers] aware of how storage costs relate to other costs,” said Roger Schwanhausser, director of storage and storage networking services at IBM Global Services, in Southbury, Conn. “[Storage] infrastructure may be where significant savings come from, but weve seen situations where infrastructure improvements would have only captured about half the potential cost savings. If you dont look at all the business issues, you run the risk of missing half the value.”
IBM isnt the only service provider advocating the inclusion of business impact issues in storage TCO assessments. EMC, for example, typically includes analysis of business continuance, disaster recovery, application development and user productivity in its assessments, said EMCs Bradford. Similarly, the assessments prepared by the Evaluator Group focus on impacts on user productivity, business continuity, and performance from current and future storage environments. (For a white paper on Evaluator Groups TCO model, see www.evaluatorgroup.com.)
While storage TCO audits can provide the critical base-line information needed to begin improving IT and business processes, experts caution that IT managers should consider the objectivity of service providers when seeking outside help. With the increased focus on storage costs and TCO, some storage vendors have embraced the concept mainly as a tool for justifying the purchase of their SANs or other products. “Theres a lot of marketing around TCO right now,” said Gartners Allen.
One answer, said Allen, is to get multiple assessments from different vendors. Another is to rely on outside providers for only pieces of the total TCO assessment. At Motorola, for example, Donthi uses EMC only to evaluate the hardware and software pieces of each sectors storage environment. Motorola does the business impact analysis itself. After the infrastructure and business impact assessments are completed, a separate Motorola architecture team makes the call on which storage technologies to deploy.
“We used EMC to help us understand our environment, trends in the industry and some of our exposures,” said Donthi. “We werent looking for a product endorsement.”
Experts have a final word of advice for IT managers considering conducting a storage TCO audit: Since such studies often kick up political controversy and often lead to IT consolidation, make sure youve got top-management support before you set out. Otherwise, you may find yourself with a ton of valuable new information about storage costs but no way to use it.
“Who cares if you can count everything if you dont have the clout?” said Allen.