A Recovery Solution with Heart

 
 
By eweek  |  Posted 2006-01-03
 
 
 
In the wake of hurricanes Rita and Katrina last year, CompuCom Systems found that many of its customers were struggling, in some cases, to stay in business. One new customer, however, was struggling to keep thousands of other people alive and realized it needed high-powered technology to help make that possible.

That customer was Houston-based LifeGift Organ Donation Center, which turned to CompuCom to design and implement a disaster recovery system to keep its systems up and running, and the delicate flow of life-saving organs on track, even in the event of a natural disaster or other catastrophe. CompuCom, based in Dallas, took stock of LifeGifts infrastructure, as well as currently available products, and concluded it could exceed the donation centers uptime and recovery-time goals for only a fraction of its original budget.

"As hurricanes Rita and Katrina became a big topic here in Houston, LifeGift realized they werent as prepared as they thought they were from an IT perspective," said Charley Ballmer, solutions architect at CompuCom. "It came to the forefront during Rita, when the expectation was that they could shut down the system in Houston and transport tapes and restore data at their Fort Worth location. They found that was not the case."

Rather than design a new system from the ground up, CompuCom worked to use existing IT assets, including Symantec backup software, more wisely. The solution provider also reduced the systems complexity to make management and maintenance easier and to save money, trimming nearly $190,000 from the original projects budget. LifeGift was so pleased with the results that it decided to concentrate on the business of saving lives and outsource all its IT activities to CompuCom.

Making a difference

LifeGift transforms life-threatening situations into saved lives each day through its organ donation program. One such case involved a 14-year-old boy from Austin, Texas, whose severe cystic fibrosis left him unable to even laugh without risking a collapsed lung. By teaming with surgeons at Texas Childrens Hospital, in Houston, LifeGift gave the teenager a second chance at life with a historic double lung and liver transplant in January 2004.

"My transplant did not just save my life, but gave me a quality of life I never thought I would ever have," the teenager wrote in the 2005 LifeGift Annual Report. "I am now back in school, I can sleep an entire night, and I can laugh. These are all things most people take for granted. Not me!"

Nationwide, more than 92,000 people currently are on the transplant list, waiting for a call that tells them a heart, a kidney, a liver, eyes, or any of eight other types of tissue or organs has become available. LifeGift is one of 58 organ-procurement organizations licensed by the federal government and serves 109 counties in Texas and such cities as Houston, Fort Worth, Lubbock and Amarillo.

The 120-person organization leads families who have lost a loved one through the process of authorizing organ donation and draws recipients from the waiting list maintained by the United Network for Organ Sharing, a nonprofit tasked by Congress to supervise organ distribution and allocation in the United States.

For each organ, LifeGift gathers and manages up to 30 pages of medical information to ensure that all potentially harmful issues are caught. The center documents hundreds of medical items for each transplant, including lab tests, X-rays, biopsies and more. "When a patient receives an organ, his or her immune system is purposely suppressed so the body wont reject it," said LifeGift President and CEO Samuel Holtzman. "Any disease in that organ can kill the recipient."

The medical information is stored in the centers DonorTrack system, said Rudy Taylor, a CompuCom network specialist who works on-site at LifeGifts headquarters.

The information must be collected and shared quickly, since a donated organ or tissue becomes unusable in a matter of hours. "Because so many people are waiting," Holtzman said, "an organ lost is a life lost."

Help for the helper

In August 2005, just weeks after avoiding the effects of Hurricane Katrina, LifeGift couldnt fend off Hurricane Rita and suffered a network outage. "The IT director found that the only recourse was to disconnect the servers and transport them to Houston," said CompuComs Ballmer. "IT put them down for a day and a half, and when they were up, they werent at full capacity."

In the aftermath of the hurricane, LifeGift decided it needed a better way to ensure uptime so the organization put an IT upgrade at the top of its to-do list. The stated objective was to ensure business continuity and to enhance data protection in case of disaster, while minimizing IT costs and overhead. "Many organizations can afford to go down for a couple hours," Holtzman said. "But for us, that means someones going to die."

To find a solution partner, Holtzman polled business colleagues. Based on the companys reputation with medical community colleagues, he selected CompuCom. "We do a lot of work in a medical center in Houston, and they have strong relationships with other area hospital facilities and recommended us," said Ballmer.

LifeGift initially engaged CompuCom, a Platinum Partner of security vendor Symantec, to review its disaster preparedness plans and provide recommendations on best-practices solutions.

"LifeGift came to us with the stated goal of having us review their existing hardware and software infrastructure and discover how their existing infrastructure could fit into an updated disaster recovery plan," said Ballmer. "In the event of a disaster, they wanted to be in a failover scenario rather than a recovery scenario. They wanted us to help them discover a path to get to that with minimal disruption within their environment."

Symantecs Backup Exec 11d shifts the focus to disaster recovery. Click here to read more.

To achieve this, CompuCom staff spent six weeks talking with LifeGift workers to get a feel for how they leveraged data and technology, said Ballmer. "We talked to people in business accounting, as well as clinical personnel, administrative staff and people performing work at the hospital," he said. "We wanted to figure out the path of what is constantly needed; what recovery times needed to be; and to create a matrix of which data was absolutely critical, somewhat critical and low-priority."

Putting it in practice

I n January 2006, the donation center asked CompuCom to put its recommendations into practice. First, the solution provider upgraded LifeGift from Symantec Backup Exec 9, which LifeGift was using in all three of its offices, to Version 10d of the backup suite and added Symantecs Backup Exec System Recovery for data recovery.

CompuCom had determined that four LifeGift servers in Houston needed to be recoverable within a matter of hours: the Microsoft Exchange-based server, the file server, the Microsoft SQL server and a server that runs a LifeGift in-house application. With the new system, these four key functions could be quickly switched to any of the organizations remote sites in the event of a disaster.

CompuCom also consolidated the eight servers in LifeGifts Houston headquarters down to six. "We sent the extra equipment to the Fort Worth office to serve as standby equipment," said Ballmer.

Backup Exec System Recovery creates hourly recovery points, which are backed up to tape by Backup Exec, on each server. CompuCom then set up the system to replicate the Houston offices Backup Exec 10d and Backup Exec System Recovery files every hour over an IP connection to the organizations Fort Worth satellite office. The files sit there ready for use when needed, and no transportation of tapes or relocation of hardware is necessary.

"We demonstrated to them that they had the ability to access data across any one of the locations as if it is sitting in their physical office," Ballmer said. "Once a system goes down, they need zero interaction with the system for the recovery. They werent even aware that that could be done."

Raising the bar

With the approach CompuCom recommended, LifeGift achieved significant savings while reducing the threshold recovery times and recovery thresholds, Ballmer said. Although LifeGift had budgeted $320,000 for consulting services, software and hardware, CompuCom created a plan that would cost only $130,000 by leveraging the organizations existing technology to meet its goals. Previously, LifeGift had been considering a plan that would have cost $80,000 to buy four new standby servers for its Fort Worth location, in addition to $60,000 in salary benefits for a new IT staffer to administer them.

"We determined that they had a lot of the physical hardware infrastructure in place, and they had purchased excess hardware in many areas," said Ballmer. "In addition, they had already made a software investment in Symantec for backup and recovery. That allowed us to come up with a creative approach."

CompuCom also reduced administrative staff time required for daily backup activities by 92 percent, using automated reporting capabilities.

Better still, CompuCom dramatically exceeded LifeGifts objectives in terms of recovery time. "In LifeGifts former disaster recovery plan, the recovery time objective [or RTO] to switch the four key servers from the primary site in Houston to the satellite office in Fort Worth was one day," Ballmer said. "With the new CompuCom-Symantec solution, theyre switched and running in 26 minutes, a 98 percent improvement."

In addition, the recovery point objective was decreased by 97 percent, from 48 hours to 1 hour, Ballmer said.

"CompuCom brought a level of expertise that is really quite incomparable to anything else Ive seen," said LifeGifts Holtzman. "It certainly is light-years ahead of our previous system."

Building a reputation

The disaster recovery project proved so successful that LifeGift put CompuCom in charge of all its IT functions. "We implemented the service desk on Oct. 5," CompuComs Taylor said. "End users can call in from any of the four sites and receive assistance with their IT needs."

LifeGift has contracted for the service desk to handle up to 250 incidents for the monthly fee, with an option for more services as needed, Taylor said. "Now, users can call 24/7 and get the attention they need," he said. "LifeGift wants someone to pick up the phone in 60 seconds, and we are averaging 6 seconds per call."

In addition, CompuCom is helping plan the security and network technology and infrastructure for a new facility that LifeGift will open in Houston in 2007.

The relationship has grown in other creative directions. For example, CompuCom has become a Workplace for Life partner company for LifeGift. "Weve made a commitment to evangelize what LifeGift and other organ procurement organizations do," Ballmer said.

To that end, the solution provider hosted an organ donor education day at its offices, at which 800 of its 1,200 employees registered to be a donor. "In this project, we became familiar with what LifeGift really does," Ballmer said. "Our work allows LifeGift to do what they do without having to worry about the technology behind it. It allows us to be part of something bigger than ourselves."

Hailey Lynne McKeefry, a partner in Professional Ink, has been writing about technology and channel topics for more than 17 years. She can be contacted at hailey@cyberdeacon.com.

Solution Provider Quick Facts

  • Company CompuCom Systems
  • Headquarters Dallas
  • Number of employees 7,100
  • Years in business 19
  • Web site www.compucom.com

    Case Data

  • The problem: A disaster or outage could bring LifeGift Organ Donation Centers systems down for a day or more, leaving it unable to match those waiting for tissue and organ transplants with available donors. It was literally a life-and-death situation.

  • The solution: Leveraging existing IT assets, CompuCom installed Symantecs Backup Exec 10d and Backup Exec System Recovery and added on-site IT support.

  • The result: The CompuCom solution improved LifeGifts key recovery metrics by more than 90 percent, reducing in-house staff in the process and saving $140,000 over an alternative disaster recovery plan LifeGift had considered.
  • Planned budget: $320,000
  • Actual spending: $130,000
  • Duration: September 2005 to present

    Editors Note: This story was updated to correct the number of CompuCom employees.

    Check out eWEEK.coms for the latest news, views and analysis on servers, switches and networking protocols for the enterprise and small businesses.

  • Rocket Fuel