eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Bob Czeizinger, information resources manager at the Delaware Department of Education, knows that keeping his pupil accounting system up and running is his top priority.
After experiencing a catastrophic database server failure several years ago that brought the departments most crucial student account management software to a screeching halt for more than half a day, Czeizinger knew he had to find a way to keep the system running, regardless of hardware or software issues.
Although the department didnt lose any data, Czeizinger also knew he had to do something to improve recovery time, and he learned about disaster recovery software while having a conversation with his Dell account manager, Annette Perry.
Czeizinger says he and Perry have a great relationship and that he consults with Perry and Dell systems consultants regularly to discuss issues and problems he is having. It was at one such meeting that he brought up disaster recovery.
“When we were discussing how we had recovered from [the catastrophic database server failure], we came to the realization—especially with our pupil accounting system—that 3 to 4 hours was too much time to take to recover from a database failure.
“At [about the same time], I had a talk with our Dell sales representative. I remember saying, Its nice we do backup and we have redundant servers, power supplies and stuff on servers, but the one thing we dont have is an easy way to recover from server failure,” Czeizinger said.
According to Czeizinger, Perry suggested that he try Double-Take, a product that other Dell customers were using successfully to keep critical systems in operation.
In fact, Double-Take Software, formerly NSI Software, of Southboro, Mass., and Dell have had a successful partnership for five years, according to Rob Dutkiewicz, inside technical sales representative for Dell at Double-Take. Dutkiewicz said Dell called Double-Take in this instance to act as subject matter experts around disaster recovery for the Delaware Department of Education sale.
“In this particular situation, with the Delaware Department of Education, [Dell] called us and got us on the phone with the customer to be the subject matter experts for high-availability disaster recovery,” Dutkiewicz said, explaining that Dell asked Double-Take to provide guidance for the customer.
“They said, Were the hardware experts, but you guys are the high-availability [and disaster recovery] experts, so tell us what fits best in this type of environment,” Dutkiewicz said.
At that point, according to Dutkiewicz, Perry invited the Double-Take sales team to make a presentation to the Delaware Department of Education. “Annette and Dell came to us and said, Weve got this opportunity, and it seems like a perfect fit. Go in there and talk to them.”
Dutkiewicz and the sales team met with the Department of Education and explained how the technology works and how it would best fit the departments requirements.
The Delaware Department of Education is based in Dover. The 30-person Department of Technology Management and Design—seven of whom deal with networking issues—is responsible for maintaining data on 116,000 students across 19 school districts encompassing 120 schools.
Approximately 8,500 teachers and administrators access the eSchoolPlus pupil accounting system on a daily basis, according to Czeizinger.
The system tracks such information as statewide student identification numbers, school profiles, school report cards, statewide student testing data, teacher schedules, attendance, report cards and more. Any time this system goes down, it brings school administration to a virtual standstill, and the pressure to get it back and running can be intense, Czeizinger said.
“If the system goes down, there is a lot of pressure from the school districts and the higher-level officials in the Department of Education, and thats besides the loss of reputation [we face],” Czeizinger said. “School districts in Delaware expect a pupil accounting system thats up and running, and they dont want excuses that we had a hardware failure. They expect it to work consistently, and thats our goal.”
To achieve this high level of service, Czeizinger began shopping for disaster recovery software, concentrating on two applications: CAs BrightStor High Availability and the Double-Take product.
Czeizinger said he visited both vendors for a product demonstration, then actually bought a copy of each one and ran the two packages on separate servers to get a feel for how they worked (although actual systemwide implementation would prove more complex and require the help of an installation consultant).
After putting each package through its paces, and working with each companys help desk, Czeizinger and his colleagues decided that the Double-Take software suited their needs better, mostly because the company offered better support and documentation.
“[Both products] did what they were supposed to do,” Czeizinger said. But with CAs product, Czeizinger said if the software didnt work according to plan, he needed to call the support desk so staffers could walk him through the recovery. It was an unsatisfactory approach, he said.
Next Page: Implementing Double-Takes system.
Implementing Double
-Takes System”>
When it came time to implement the system, Double-Take assigned Senior Implementation Engineer Scorey Dubose to help the Delaware Department of Education install and learn to use the Double-Take software.
Dubose also helped the department set up a cluster and installed Double-Takes GeoCluster product.
In addition, Dubose helped transfer four SQL database servers to a single server, something that Dubose said he sees as part of the total solution for a customer.
“Each environment is different, and in the Delaware education environment, they didnt have a lot of exposure to clustering, so it was better for them to wait for my help,” Dubose said.
Dubose worked with Delaware Department of Education personnel to make sure they used Microsoft best practices for building the cluster. He also wanted to be sure that the department knew how to maintain the cluster after the initial installation was completed. As part of the process, Dubose taught Delaware Department of Education personnel how to install, configure and maintain the cluster and how to install future service packs. This meant making sure they understood clustering and how it worked, as well as, if one node had a problem, how it rolled over to the backup node.
In addition, the Delaware Department of Education presented a somewhat unusual situation for Dubose because it required that the nodes of the cluster be in separate buildings. He said the usual way of setting up a cluster is with nodes in the same room.
“In a typical environment with clusters, the two servers are sitting in the same room, and they are side by side,” Dubose said. “But in their particular environment, they had one node in one building and the other node in another building, so they had to make sure they had the proper bandwidth between the two locations so there wouldnt be any type of network latency.”
Paul Pond, a network engineer at the Delaware Department of Education who worked with Dubose during the installation, said Double-Take gave him everything he needed to understand and maintain the system in the future.
“When [Double-Take] came down, they actually set up the first implementation and walked us through what the product does and how it does it. So the understanding of the product helped us out when we had to start setting it up on our own,” Pond said. “Once we got through that first setup, the rest was a piece of cake.”
Dubose agreed that the installation team was very open to learning the new system, and it went very smoothly.
“They were very receptive about the technology and the documentation that we provided for them,” Dubose said. “The key—what really made this successful—was that they allotted the time to this implementation so we could be successful.”
Dubose said the Delaware Department of Education blocked out five uninterrupted days, so they could understand how the products work.
Since the Delaware Department of Education has had the Double-Take software in place, Pond said there have been instances where—due to hardware failure—the secondary server has had to take over. Pond said the process went so smoothly that he would not have known about it had he not received an automatic e-mail notification that the failover had occurred.
In one case, Pond decided to take the system down because he knew he had a bad switch, and he had to do a manual failover to the backup server. When he shut down the server, Pond said he received an e-mail notification, went into the Double-Take software and clicked one button—and the manual failover began.
The Delaware Department of Education does not measure the return on this investment in strict dollar amounts, said Czeizinger. It also weighs staffs peace of mind that the system will be up and running, no matter what.
“The reality is we are a state government agency, so its hard to put a [hard-dollar] figure on return on investment,” Czeizinger said. “When we were here in the past with a database failure, we were talking from 4 hours to an all-day event, and it tied up a lot of personnel time. I know, for example, in the past, its actually tied up four or five people trying to recover from a database failure.”
When you add in the pressure the department feels in these situations to get this critical system up and running, Czeizinger said the software more than pays for itself.
Pond and Czeizinger recommend that people evaluating disaster recovery software purchase and test the packages as they did to get a feel for the technical support, the documentation and how easy it is to recover.
“The [vendor] support of the hardware or software is, to me, among the most important things. We are a relatively small shop, and I have some really capable technical people here, but nobody knows it all. Frankly, Double-Take and Dell have bailed us out more than a couple of times,” Czeizinger said.
While the competitive product was capable, Czeizinger said the difference came down to support. “Double-Takes support was better, which made the product better.”
Ron Miller is a freelance writer based in Amherst, Mass. Contact him at ronsmiller@ronsmiller.com.
Case File
- Customer: Delaware Department of Education
- Location: Dover
- Organizational snapshot: State department responsible for all public schools and related educational facilities
- Business problem: Disaster recovery methodology was in place with backups and redundancies, but systems took too long to recover from a failure and the department required that a critical pupil accounting system run without interruption
- Technology provider: Double-Take Software, in Southboro, Mass.
- Recommended solution: Install Double-Take disaster recovery software; should primary server fail, the secondary backup server takes over automatically without any operator intervention or loss of service
- Return on investment: The new system provided peace of mind that a crucial pupil accounting system would stay up and running, no matter what
- Lesson learned: Test the software and make sure your vendor provides good technical support and solid documentation
Check out eWEEK.coms for the latest news, views and analysis on servers, switches and networking protocols for the enterprise and small businesses.