eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
When it comes to lean it operations, the it operations group at Rockwell Automations Power Systems division just may give new meaning to the term.
Rockwell Automation Inc. markets industrial automation power, control and information systems designed to help customers boost their manufacturing productivity.
While most businesses of Rockwell Automations type fund IT operations at approximately 3 percent or 4 percent of total sales, Rockwells Power Systems IT shop is funded at between 1.5 percent and 1.6 percent of sales.
Thats a tough target to hit, and, in fact, Rockwells IT division wasnt hitting it, with expenditures coming in at 1.9 percent of its sales.
But with a strong corporate culture of lean and efficient operations, Clyde Fowler, vice president of IT, in Greenville, S.C., “wanted to see how good we could get,” he said.
“I view what were doing as part of the cost of doing business. I dont want my cost to the business to be 3 or 4 percent,” Fowler said. “That might mean our products are not cost-competitive. This is one method in which we continue to maintain a lean, low-cost environment.”
To get to that 1.5 percent cost target, the IT shop in the Power Systems division looked to trim costs in its NOC (network operations center) and in the maintenance it paid on its Hewlett-Packard Co. OpenView Network Node Manager and OpenView Operations software.
In addition to cost reduction, the IT shop wanted to improve its monitoring capability.
This set the stage for the decision to get rid of OpenView maintenance as a way to cut costs.
With an IT staff of 15 supporting 2,800 users and a global frame relay network, operations personnel had to wear multiple hats.
The divisions earlier OpenView administrator also supported PCs and was a Microsoft Systems Management Server administrator.
But as staff shifted around and with the OpenView software sitting idle for months at a time, “it didnt make sense to run OpenView if we didnt have someone full-time dedicated to it,” said PJ Goodwin, infrastructure manager.
“There was no way I was going to get a full-time [OpenView] administrator,” Goodwin added.
In addition to cutting costs, the IT group wanted to continue to improve monitoring of approximately 250 critical network devices, applications and production servers.
However, its OpenView software was never fully implemented and didnt do much more than act as “an expensive ping-monitoring tool,” said Goodwin.
“We didnt have anyone qualified enough or couldnt assign a resource to install it [fully]. It never progressed to the full vision of what it was supposed to be,” Goodwin said.
So Rockwell set out to find a service provider that could deliver the monitoring capability more cost-efficiently and at the same time act as an adjunct to its own IT staff.
Next Page: A streamlined model.
A Streamlined Model
Rather than hire outside experts to put the procedures in place to fully exploit OpenView, Rockwell instead found a service provider that already had a streamlined monitoring and service delivery model in place that could act as an adjunct to the Rockwell IT staff.
Local MSP (management services provider) Digital-DNS Inc. had been pitching to the company for a while, Goodwin said, and came in with the most compelling proposal out of those submitted by larger vendors including Hewlett-Packard Co., Dell Inc. and IBM.
“We had different vendors propose to monitor our systems and be a NOC for us,” Goodwin said. “It just didnt feel right. I was looking for a business partner rather than someone that would just take my money.”
Digital-DNS StressFree I/T Enterprise Monitoring Services provides 24-by-7 monitoring of customers critical network devices, servers and applications. It is intended to allow a clients IT staff to focus on more mission-critical functions.
Digital-DNS got the go-ahead to provide a simple proof-of-concept demonstration, which Goodwin said made it easy to see how well the service could perform.
After installing a SilverBack Technologies Inc. SilverStreak monitoring appliance on-site at Rockwell, Digital-DNS monitored six or seven devices for a month, “and we helped them pinpoint a major problem during that proof of concept. And that sold them,” explained Al Gossett, president and CEO of Digital-DNS, also in Greenville.
The StressFree I/T monitoring service uses the SilverBack appliance as a service delivery platform for providing round-the-clock monitoring of critical devices chosen by the client.
The service also includes remediation of problems and a monthly report on the health of the clients systems, which can include potential problems spotted that require attention, according to Gossett.
The types of devices that Digital-DNS monitors for Rockwell range from Windows NT 4.0 and Windows 2000 servers to Linux systems; VMware Inc.s VMware; Sun Microsystems Inc. Solaris servers; Cisco Systems Inc. Pix firewalls and Catalyst 6500 switches; VPN concentrators; Web proxy servers; and site-to-site VPN tunnels.
Digital-DNS Chief Technology Officer Bill Brown said the monthly report “highlights what we see in their entire network, performance-related issues, security vulnerabilities, and we have a recommendations section.”
Each of Rockwells networking, Lotus Notes, Windows NT Server, Unix, security and database groups receives a specific report on devices Digital-DNS oversees.
Once the SilverBack monitoring appliance is installed and devices and applications to be monitored are added, both Digital-DNS and its clients create notification rules for the client and for Digital-DNS help desk, according to Gossett, and then escalation procedures are established.
“We know exactly what we do with certain situations, who to notify and what to notify,” said Gossett.
In Rockwells case, its IT staff receives alerts and handles all break/fix work.
“We watch, too, and if we see something they miss, we notify them,” said Brown. “After 5 [p.m.], we pick it up, and if there is a frame relay outage, we put the ticket in with their carrier. If its a server or device outage, we contact their on-call personnel. Were watching [the Rockwell network] from 5 p.m. to 8 a.m. and on weekends.”
Rockwell also shifted management of its Citrix Systems Inc. environment to the MSP, and Digital-DNS handles remediation for Rockwells frame relay network, which includes “hundreds of circuits” and extends from the United States to China and Mexico, according to Brown.
Goodwin said Digital-DNS has been a trusted partner and has become the first resource that Rockwell IT calls on when it needs help fixing a problem it cant handle alone.
“We had a virus that hit. I was able to talk to Bill Brown throughout the night on technical issues, and [about] pushing out patches. I was bouncing ideas off him as I dealt with my staff cleaning up the virus. Thats invaluable service right there,” said Goodwin.
Although calculating the cost savings that Digital-DNS has enabled is difficult, Fowler estimated that it saved about $150,000 in the first year by being able to move staff around and by eliminating annual maintenance on its OpenView license.
It has also increased overall system uptime from somewhere in the 80 percent range to the “high 90s,” said Fowler.
“We removed a third-shift operator position dedicated to watching systems, and weve begun looking at other ways to be able to move people around. Having Digital-DNS there gives me confidence that Ive got a qualified set of eyes always looking at my environment,” said Goodwin.
But the benefits reportedly go far beyond that.
“Weve increased our uptime, and theres a huge cost of downtime for some systems,” Fowler said. “When we can look at trending information and do something to prevent a problem, Im saving a lot of money there. We have a better and more cost-effective environment, and it contributes to our ability to operate at a low cost. And our environment is better. We have a higher level of uptime, and our people are less overworked. One of our teams is working on converting a mission-critical application to Linux. In the past, they would have spent a lot of time on maintaining OpenView. Now were about to complete this major transition to Linux.”
Case file
- Customer Rockwell Automation Power Systems division
- Location Greenville, S.C.
- Organizational snapshot IT staff of 15 supporting 2,800 users on some 250 nodes; staff is divided across operations, network (LAN and WAN), Lotus Notes, Unix and Windows groups
- Business need Dramatically reduce the cost of IT operations while improving network, system and application monitoring
- Technology partner Digital-DNS, a technology services provider also based in Greenville
- Recommended solution StressFree I/T Enterprise Monitoring Services to provide 24-by-7 monitoring of critical network devices, applications and servers
- Lessons learned Recognize internal resistance to change and go with the most efficient mode of operation as quickly as possible