eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
“Rather than building a traditional hospital and then trying to retrofit it, we thought wed just bite the bullet and do it from the beginning,” said Dr. Robert Kipperman, one of the Oklahoma City hospitals founders.
With the help of a consultant, the founders chose an electronic medical records system that best fit the hospitals needs. They realized they were breaking new ground when they discovered that they had to hire consultants to write the interfaces to all the electronics equipment in the hospital. Officials said the hospital spent about $5 million more than if it had not taken the all-digital route.
“I think we spent more on IS than any hospital had, and we had to write a lot of the content. It just wasnt available,” said Kipperman. “We had a vision of what we wanted to do, and we learned as we went along.”
Another lesson was that downtime of applications and systems is not an option. But maintaining high availability for a small IT staff of seven was not easy, and it came at a high cost for IT managers. Many alarms from the BMC Software Inc. Patrol application monitoring system in the middle of the night as well as during the day left managers more focused on maintaining existing systems rather than improving services to patients, doctors and medical staff.
As with most enterprise management systems that monitor events, Patrol generates a large number of alarms and alerts that dont necessarily indicate trouble but that cant be completely ignored or suppressed because some do indicate outages.
“With Patrol and those types of monitoring tools, you get a lot of alarms. Im on call, too. And my beeper goes off all the time. But I cant turn it off, even if its considered normal—like an alarm that keeps happening at 3 a.m. every Wednesday,” said Dave Stinson, IT manager at the 3-year-old facility.
The hospital, designed from the ground up to be all digital and completely paperless when it opened its doors in August 2002, runs on Cerner Corp.s Millennium Health Information System. Components of the Millennium suite implemented at the specialty hospital include the PowerOrders computerized physician order entry system; the PowerChart electronic health record system; the PowerNote physician documentation system; the PharmNet pharmacy system; and the CareNet nursing system, which provides access to clinical information.
Every room in the hospital has a bedside computer, and caregivers have access to information in the Millennium system from multiple locations, including computers located at each nurses station, in the emergency rooms, day-patient areas, operating rooms and catheterization labs. Caregivers can pull up any clinical information on patients in digital form, including medical images, medication and allergy reaction records, past procedures, and consultations with other doctors.
“We have hundreds of computers. You want to access the [patients] chart when you need to. You live and breathe by your chart. You have to have it. It has all the information you have to have to take care of your patient,” said Peggy Tipton, the hospitals chief operating officer.
But without an understanding of the full power at their fingertips, doctors could inadvertently bring the system to its knees. By scanning heart rhythms into the system using uncompressed file formats, for example, doctors could fill up all available storage and lock up the system, making it unusable.
Stinson, in fact, prevented such a scenario before any downtime occurred using a new tool that BMC brought to his attention. The BMC Patrol Analytics tool, OEMed early this year from Netuitive Inc., works in conjunction with Patrol to watch and learn the behavior of systems and then automatically generate dynamic, real-time thresholds. It cuts down on the number of false alarms and generates Netuitives Trusted Alarms, which provide more accurate and detailed trouble reporting.
Because it can also spot and highlight trends that forecast potential outages, Patrol Analytics allows users to head off issues such as the storage incident. “Weve caught two problems that were prevented because of the forecasting in [Patrol] Analytics,” said Stinson.
Next page: Possible disasters adverted.
Possible Disasters Averted
File sizes of 15MB for uncompressed TIFFs, which were supposed to be compressed JPEG images, quickly filled up a terabyte of storage—thanks to doctors who overrode the compressed JPEG default. “The alarm we got was [that] this disk was running its usage trend, and [Patrol Analytics] saw it was beyond normal usage. Another two days and we probably would have lost the system,” Stinson said.
In the other instance, a very busy Abbott Laboratories Inc. i-Stat lab computer analyzing blood tests and other lab work was flagged when normally heavy usage increased even further. “It kept getting busier and busier to where jobs started to get queued. I saw that on the trending report, even though the CPU was always at 100 percent, and that was considered normal. Before the system became unusable, we had time to upgrade it,” said Stinson.
Patrol Analytics also dynamically flags underutilization, allowing IT operators to catch issues such as certain jobs not running as scheduled. “If the CPU is normally loaded at 80 percent on Wednesday, and now its 40 percent, you see that,” said Stinson.
The tool uses a simple Web interface for reporting, and it provides charts on the metrics it gathers. “The charts that you can bring up for the systems [it monitors] give you a great feel for what the systems are doing—how much available memory they have, how many processes are running, what the hardware components are doing. It gives you a good feel for how busy that system is and when its going to die,” said Stinson.
Patrol Analytics also includes a Performance Assistant, which allows users to fine-tune it. “It does a great job of creating these bands of normalcy. The longer the product runs, the smaller the band gets,” said Stinson.
Patrol Analytics uses existing Patrol Knowledge Modules running on the Patrol agent for each system its monitoring. In this case, those include a pair of RS/6000s running IBMs AIX and the Millennium suite, along with an Oracle Corp. database.
“We poll it using BMCs API to contact the agent and gather the parameters we monitor, and we store that data in a database,” said James Wood, technical support manager at Netuitive, in Reston, Va. The data is analyzed in the tools analytics engine, which populates the database with its results. Any alarms generated are exported to the Patrol console.
“The analytics engine is the heart of it. It has a base line engine that analyzes how specific metrics are running over time and builds a base line of whats normal for them during times of day and days of week. It also automatically learns correlation between different metrics, such as if this goes down, thats expected to go up. And it does trend analysis—if things continue, this is where this will be at,” Wood said.
Netuitive created the custom integration with the Cerner Millennium suite for the hospital at its own expense. That investment paid for itself almost immediately in several ways.
In addition to freeing up Stinson and other IT staffers to devote more time to helping users—a primary concern—Patrol Analytics showed where resources were underutilized and presented opportunities for consolidation.
“Ive consolidated SQL servers, and Ive probably moved six different business applications down to one server. Ive also consolidated my six Citrix [Systems Inc.] servers. We saw Citrix usage was so minor, we consolidated the boxes, and were still well within capacity. Those are $35,000 boxes,” said Stinson, who said the hospital paid about $35,000 for the Patrol Analytics integration with Cerner.
“But still most important is my time. I can prevent problems from happening, and I have more time allocated to patient care,” Stinson added.
Stinson plans to use some of the time Patrol Analytics has given back to developing Internet access for patients from within their rooms. “Patients bring in their laptops and expect to get Internet access,” he said.
But the ultimate goal is to boost the efficiency of the IT staff to make them more available to help end users with any issues they have, according to Tipton.
“The constant monitoring you have to visually do will drain you from helping end users deal with day-to-day activities. People always need help for different things,” she said. “If we can make life easier for that nurse, doctor or pharmacist, ultimately the patients get better care. Our patient satisfaction is unbelievable. Weve been in the top 1 percent in the nation for patient satisfaction.”
Check out eWEEK.coms for the latest news, views and analysis of technologys impact on health care.