Possible Disasters Averted
File sizes of 15MB for uncompressed TIFFs, which were supposed to be compressed JPEG images, quickly filled up a terabyte of storagethanks to doctors who overrode the compressed JPEG default. "The alarm we got was [that] this disk was running its usage trend, and [Patrol Analytics] saw it was beyond normal usage. Another two days and we probably would have lost the system," Stinson said. In the other instance, a very busy Abbott Laboratories Inc. i-Stat lab computer analyzing blood tests and other lab work was flagged when normally heavy usage increased even further. "It kept getting busier and busier to where jobs started to get queued. I saw that on the trending report, even though the CPU was always at 100 percent, and that was considered normal. Before the system became unusable, we had time to upgrade it," said Stinson.The tool uses a simple Web interface for reporting, and it provides charts on the metrics it gathers. "The charts that you can bring up for the systems [it monitors] give you a great feel for what the systems are doinghow much available memory they have, how many processes are running, what the hardware components are doing. It gives you a good feel for how busy that system is and when its going to die," said Stinson. Patrol Analytics also includes a Performance Assistant, which allows users to fine-tune it. "It does a great job of creating these bands of normalcy. The longer the product runs, the smaller the band gets," said Stinson. Patrol Analytics uses existing Patrol Knowledge Modules running on the Patrol agent for each system its monitoring. In this case, those include a pair of RS/6000s running IBMs AIX and the Millennium suite, along with an Oracle Corp. database. "We poll it using BMCs API to contact the agent and gather the parameters we monitor, and we store that data in a database," said James Wood, technical support manager at Netuitive, in Reston, Va. The data is analyzed in the tools analytics engine, which populates the database with its results. Any alarms generated are exported to the Patrol console. "The analytics engine is the heart of it. It has a base line engine that analyzes how specific metrics are running over time and builds a base line of whats normal for them during times of day and days of week. It also automatically learns correlation between different metrics, such as if this goes down, thats expected to go up. And it does trend analysisif things continue, this is where this will be at," Wood said. Netuitive created the custom integration with the Cerner Millennium suite for the hospital at its own expense. That investment paid for itself almost immediately in several ways. In addition to freeing up Stinson and other IT staffers to devote more time to helping usersa primary concernPatrol Analytics showed where resources were underutilized and presented opportunities for consolidation. "Ive consolidated SQL servers, and Ive probably moved six different business applications down to one server. Ive also consolidated my six Citrix [Systems Inc.] servers. We saw Citrix usage was so minor, we consolidated the boxes, and were still well within capacity. Those are $35,000 boxes," said Stinson, who said the hospital paid about $35,000 for the Patrol Analytics integration with Cerner. "But still most important is my time. I can prevent problems from happening, and I have more time allocated to patient care," Stinson added. Stinson plans to use some of the time Patrol Analytics has given back to developing Internet access for patients from within their rooms. "Patients bring in their laptops and expect to get Internet access," he said. But the ultimate goal is to boost the efficiency of the IT staff to make them more available to help end users with any issues they have, according to Tipton. "The constant monitoring you have to visually do will drain you from helping end users deal with day-to-day activities. People always need help for different things," she said. "If we can make life easier for that nurse, doctor or pharmacist, ultimately the patients get better care. Our patient satisfaction is unbelievable. Weve been in the top 1 percent in the nation for patient satisfaction." Check out eWEEK.coms for the latest news, views and analysis of technologys impact on health care.
Patrol Analytics also dynamically flags underutilization, allowing IT operators to catch issues such as certain jobs not running as scheduled. "If the CPU is normally loaded at 80 percent on Wednesday, and now its 40 percent, you see that," said Stinson.