As one of the nations top oil refining companies, Valero Energy Corp. is no stranger to acquisition fever. A Fortune 500 company with approximately 20,000 employees and expected revenues this year of $55 billion, it has an extensive refining system that generates more than 2.5 million barrels of fuel per day. Valero has approximately 4,700 branded retail sites such as service stations. It also has 4,800 miles of refined-product and crude-oil petroleum pipelines.
But bigger is never enough in the oil refinery business. Tough competition for more capacity and pipelines drives San Antonio-based Valero to act like a corporate Pacman by acquiring more oil refining companies.
And each time Valero makes an acquisition, IT managers at the company are faced with the arduous task of merging two independent IT infrastructures into one.
In January 2001, Valero merged with Ultramar Diamond Shamrock—another oil refinery, also in San Antonio. The companies were nearly equal in size, so Valero grew from a $15 billion business to $32 billion with the Ultramar merger—meaning that Valeros IT infrastructure doubled in size with just one acquisition.
With more recent mergers, Valero has expanded to 15 refineries (from just one in 1997)—with a network that stretches from Canada to the United States Gulf Coast and West Coast to the Caribbean.
“This is a company that has grown with acquisitions,” said Greg Jordan, a project manager for IT at Valero. “So there are more in our future.”
One of the areas that pose a mammoth post-merger problem for IT is backups. In general, the lack of backup reporting software has been a maddening problem for IT managers, according to industry experts. They are often in the dark about which nightly and weekly backups have been successful. Information on capacity utilization is usually estimated, experts say.
Many managers monitor the success rate manually, culling data from server logs and assimilating it within a Microsoft Corp. Excel spreadsheet. What can be even more problematic is when a backup application reports that there was a problem but does not specify what caused it.
With the Ultramar merger, Valero went overnight from managing backup on fewer than 300 servers to about 600 servers. The total storage needs jumped from 20TB to 40TB. Valeros IT staff also faced the additional—but not uncommon—issue of integrating the incoming companys backup application. Both companies had 12 backup servers, but they were using different backup applications: Ultramar used IBMs Tivoli Storage Manager, while Valero used Hewlett-Packard Co.s OpenView Storage Data Protector.
To compound the issue, to get additional resources, Jordon needed to show upper management concrete statistics that proved backups were taking longer.
“One of the things that is getting harder and harder to do is to get management to approve upgrades,” Jordan said. “We needed more people, more resources such as tape drives, backup servers and backup software client licenses—which all meant a bigger budget. They wanted a report showing actual data, not just some guys gut feeling that more resources were needed.”
In spring 2003, Jordan turned to Bocada Inc., in Bellevue, Wash., to solve his business problem. Bocada is an enterprise software company that specializes in data protection management, and it produces one of the few tools on the storage market that has an agentless architecture and is vendor-agnostic for backup applications.
Bocadas Bocada Enterprise automates some key tasks that often are done manually. It gathers, organizes, analyzes and presents reports on backup performance and system utilization. Moreover, Bocada Enterprise provides root-cause analysis of service failures and pinpoints troubleshooting on chronic and critical errors that disrupt backups. It supports any backup application, including Veritas Software Corp.s software, IBMs Tivoli, HPs OpenView Storage Data Protector and CAs ARCserve.
Bocada Simplifies Success, Failure
Valero installed Bocada software and conducted a pilot test for about four months. Jordan said one of the primary features was Bocadas ability to report on the success and failures of nightly and weekly backups.
Previous to installing the automated software, Jordan said he had an IT administrator spending six hours per day verifying the success of backups and troubleshooting the previous nights sessions.
That person checked each servers log to find which jobs reported an error. With the Bocada system, the task has been simplified.
“We run a success and failure report first thing each morning,” Jordan said. “Now, instead of I hope to find the problem, we are told what the problem is.”
The problems that Jordan faced are universally common for IT organizations involved with corporate mergers. Drake Pruitt, vice president of marketing for Bocada, said there are several standard business issues these companies face.
For one, the larger company involved will want to standardize operational practices—which are never the same at two companies—such as backup schedules and how backup report statistics are disseminated through the companys chain of command.
“Every company has their own way of doing this,” Pruitt said.
Second, companies also want to eliminate the costs and challenges of training staff and support for new infrastructures—in the case of Valero, training staff on a new backup application.
Most important, the IT organization that is integrated into the larger company will have to conform to new data protection policies that include the frequency of backups, length of data retention, and whether it wants a full or incremental backup.
Bocada takes these management jobs and streamlines them through automated software, Pruitt said. For example, Valeros policies are ported into the Bocada software. The policy is reflected in the Bocada software, the information is captured from all the backup environments and performance statistics are measured against the company policy.
Overall, Valero most often uses the consolidated reporting tool of Bocada software. This centralized view means that, in seconds, IT staff can see what backup job has failed in the past 24 hours, Jordan said.
“At most, we spend an hour a day on [monitoring and troubleshooting],” Jordan said. “Typically, its less than 30 minutes, which, for 1,000 backup jobs, is pretty good. Next time we go through an acquisition, it will be a more seamless transition.”