SAN Saga

Washington group international operations manager recalls lessons learned on path to SAN.

Enterprise data storage may create more tension than any other element of IT architecture—facing the pressure of constant growth and presenting a spectrum of options that range from costly obsolescence to unacceptable bleeding-edge risk. At Washington Group International Inc., IT Enterprise Operations Manager Gary Bronson met the storage challenge with a SAN deployment, now in its final stages, that offers useful lessons to those in similar positions.

Readers of eWeek know Bronson as a member of our Corporate Partner advisory panel, through which he shares his hard-won insights into managing heterogeneous, widely dispersed systems supporting worldwide engineering, construction, environmental, mining and program management efforts. In a 15-year IT career thats included managing operations through three corporate mergers, Bronson has learned to run along the knife-edge boundary that lies somewhere between expensive-risk avoidance and reckless enthusiasm for novel technology.

Bronson is already mitigating such common storage and operations headaches as e-mail and Web site hosting through outsource arrangements with, respectively, United Messaging Inc. and Sprint eSolutions, a division of Sprint Corp.

Internally, one of WGIs key architectural elements had been direct-attached storage, on a Sun Microsystems Inc. Enterprise 10K platform, for the companys Oracle Corp. Oracle Financials production environment. Despite natural conservatism with this critical system, "we had to address our growth in storage requirements," Bronson said. "We had numerous budget pressures that prevented us from continuing in our previous architecture."

Specifically, said Bronson, "that architecture required us to keep buying expensive system boards to accommodate additional I/O." Crucial considerations, he added, were achievement of cost reduction targets while avoiding any reduction of application performance.

With cost and performance criteria in mind, said Bronson, in Boise, Idaho, WGI initially considered both a SAN (storage area network) and NAS (network-attached storage). Although he was not able to undertake large-scale lab testing of trial installations, his research suggested that a dedicated storage network, rather than the sharing of network bandwidth between storage and other traffic, would best meet WGIs needs. It appeared, he said, that a SAN would perform better in high-end, large-database applications. "We wanted to minimize the risk of latency," he said.

Wanting to limit commercial as well as technical risk, said Bronson, "we limited our SAN vendor choices to those that we already had in-house. We have limited resources and wanted to avoid bringing in another vendor if possible."

That gave him a choice between proposals from Sun (with Hitachi Ltd.) and Compaq Computer Corp. After requesting quotations and visiting reference sites, WGI proceeded with Compaq, primarily for reasons of cost. Confidentiality agreements with his vendors limit his ability to disclose details, but, "overall, we saw approximately 20 percent delta," Bronson said.

Like any builder testing a new approach to a structural design, Bronson sought validation on a small scale before making a major commitment.

"Our initial plan was to take a smaller application and migrate that application to the SAN environment," he said. "This would enable us to establish internal expertise and work out any wrinkles. Based on that success, we would then look to port our Oracle Financials environment over to the SAN."

Establishing comfort with both the technology and its supplier should be part of any leap as significant as the choice of a new storage platform. And buyers today are entitled to expect accommodation of heterogeneous environments from any enterprise systems vendor: "The first step," Bronson said, "was a proof of concept at the Compaq lab. We took a copy of our application and loaded it at the Compaq lab in Colorado Springs. Compaq prepared a Sun environment similar to our existing environment."

Get Buy-In

by involving line-of-business staff at the outset, rather than presenting a fait accompli after an opaque selection process, IT departments enhance their credibility and accelerate acceptance by user organizations. "We flew the application admin and one of the key stakeholders to the lab, along with our [database administrator] and SAN man, to do initial testing," Bronson said. "This process lasted about one week, and the proof of concept was successful. We then began the initial implementation."

By shifting responsibilities among his four Unix system administrators, Bronson was able to make one of them his SAN specialist while also providing cross-training to assure continuity of operations. Other measures provided similar safety for the data. "Planning always includes rollback procedures and proper staging to minimize risk of data loss," Bronson said. "We maintained an old copy on tape and the original system at the point of conversion. We brought in another box and connected it to the SAN."

Never Too Safe

bronson favors high levels of protection for critical data: "The architecture we use in our environment is to have three images of the given environment. We take one image offline and perform backups off that image; when we complete the offline backup, we reassociate that image. We always have two active images available."

Bronson acknowledged that this model creates an added cost factor for every megabyte of storage but said it minimizes the impact to the user during backup and reduces risk in WGI operations. That conservatism, he added, was one of the drivers in moving to a SAN: Bronson wanted to spend money on storage capacity, not on additional I/O ports.

No amount of planning can eliminate surprises, and its part of any seasoned planning process to provide some flexibility for problems that do arise.

"In particular," Bronson said, "the Sun box that we planned to run the application on, a Sun 3800, was not supported yet in Compaqs SAN environment. Time frames continued to be missed while we waited (and still wait) for a host bus adapter that fulfills our requirements; we finally had Compaq lease a 4500 machine from Sun, until the problem is resolved satisfactorily. This [delayed] our schedule approximately four weeks."

In addition, a certain degree of future shock affected the technology transition: "The pre-sales team tends to sell the next new function/feature/configuration to differentiate themselves from the competition," said Bronson. "The problem is that the new function/ feature/configuration hasnt quite made it through the support model. Nothing is more frustrating than having the support team tell you that you have an unsupported configuration when it was the vendor that specified and installed it."

Bronson offered this emphatic warning to other IT buyers: "It will happen again if you are not having your support team sign off on what the pre-sales team is prescribing."

Even so, Bronson was able to put his adoption process on the fast track in pursuit of the earliest-possible cost reductions.

"The investigating and initial planning happened over two to three months," Bronson said. The time from purchase requisition to production, he estimated, was just over eight weeks.

Bronson said his targets for the new technologys adoption have been met: "We are addressing our growth requirements within the budget we set forth. We are also noticing improved performance. It is also enabling us to simplify some of our backup operations."

Lessons Learned

with this success came lessons learned. "Overall, I think SAN/NAS options are worth investigating," said Bronson. He recommended that those who want to make a similar evolution "follow a similar pattern: Investigate the alternatives, go see the solutions working in production at real sites, prototype in a lab setting."

Bronson said, "Understand the support model and escalation process, implement with a small application, train internal resources prior to implementation, and gain experience prior to moving the critical application into the environment."

We asked Bronson how he now stacks up SANs against other storage alternatives.

"Storage on IP? Too new for my blood," Bronson said. Network-attached storage devices have earned a role in many shops, he said, but "based on the information we had available, it appears that the SAN environment is better suited for the multiterabyte [database] application."

Bronson said outsourced storage is not yet an option for his critical applications. "I still think of this offering as immature. I see too many risks at this juncture. We are presently cutting our teeth with outsourcing e-mail; were not ready to do it with our financials," he said.

Regardless of the specific technology, Bronson said, "Do not underestimate the value of solid change management during this process: testing, establishing base lines, establishing appropriate go/no-go milestones, training, monitoring and establishing the vendor support model."

Technology Editor Peter Coffee can be reached at