Public Utility Finds the Power of Virtual Backups

By Michael Hickins  |  Posted 2009-04-01

Public Utility Finds the Power of Virtual Backups

The utility company for Snohomish County District 1, Snohomish PUD No. 1, knows the value of power since it supplies electricity for 300,000 residences and businesses, including Boeing.

But just because the utility produces electricity doesn't mean it gets the power for free, so Snohomish PUD No. 1 found itself in the same situation as many other organizations-wanting to conserve power and reduce data center growth.

Also like many organizations, it decided that virtualization would help address those issues, reducing the number of physical servers employed by having each one host a large number of virtual servers running concurrently and independently of one another.

Since late 2007, Snohomish PUD No. 1 has purchased 28 ProLiant 460 and 480 blade servers from Hewlett-Packard, installed VMware's ESX Server virtualization software, and reduced the number of traditional physical servers and server racks.

Melinda Cooper, senior infrastructure system analyst and domain administrator for Snohomish PUD No. 1, noted that instead of buying 30 new servers over the past year, the utility is now running 30 virtual machines on four ESX Server-based systems.

Cooper's organization is responsible for providing services to internal customers, including the creation of development and testing environments for in-house staff. The servers also run applications such as "cashiering, keycard access, payroll, PeopleSoft [enterprise resource planning], [Microsoft] Exchange and many SQL databases," Cooper told eWEEK.

Clearly, these kinds of mission-critical data and applications need to be backed up frequently, but unless managed carefully, that process can eat up an inordinate amount of time and resources. Cooper has implemented a virtual storage backup system that conserves that time and energy in proportion to the energy and cost savings of virtualizing physical servers.

Technological trade-offs

There are several approaches to backing up VMs, and each of them involves a trade-off in terms of complexity and reliability.

One approach is to install a backup agent as if the VM were a physical machine. Snohomish PUD No. 1 was using IBM's Tivoli Storage Manager and began using the application to back up virtualized servers. However, this presented a number of problems.

Administrators have to use a command-line interface to configure TSM, which is not a trivial exercise, Cooper said. Moreover, TSM makes only file-level backups, while Cooper also wanted full-machine backups.

Another drawback to this approach is that when there are a number of VMs on a single ESX Server host, administrators have to perform the laborious chore of scheduling backups to ensure that the VMs don't compete for resources during the backup period, slowing overall system performance.

Moreover, machines have to be shut down prior to the backup because if the VMs are working when the backup is under way there's a chance that transactions will be missed. Alternatively, administrators can take a snapshot of the machine's VMDK (Virtual Machine Disk Format) file, which represents the whole VM as a single file. In either case, manual scripting of pre- and post-backup processing is required.

While a full-machine snapshot provides for fast restoration of the VMDK files-and thus quick recovery of the entire VM-it doesn't allow individual files or folders to be restored.

This means, "You don't get the configuration files, so if a [virtual] server is damaged, you can't restore that," Cooper told eWEEK. And unless the admin specifically adds backup of the configuration files to the full machine backup of the VMDK files, he or she won't be able to restore the virtual server if it is damaged.

STORServers Solution


VMware tried to reduce this complexity by providing VCB (VMware Consolidated Backup)-a set of scripts and drivers that allows TSM to centralize backups of VMs on a single proxy server, freeing administrators from having to put a backup agent on each client and server.

The script takes a snapshot of the VM server and allows file-level backup as well as full-machine backup.

However, VCB isn't always enough, as it's hard for administrators to manage if they're not familiar with scripting in languages like Perl, Cooper said.

"While we had virtual consolidated backup with VMware and Tivoli, the virtual consolidated backup on a proxy server required a lot of hand-holding, syntaxes and other background work I didn't feel comfortable with," she said.

To solve this issue and reduce labor costs at the same time, Cooper tried the STORServer Agent for VCB, essentially a front end for VCB and TSM. The tool provides a graphical interface on the proxy server that discovers all the VMs on a network and allows administrators to schedule both file-level and full backups without having to use the command line and scripting.

The STORServer Agent allows IT administrators to provide fast disaster recovery services and makes it possible for end users to restore their own files so that a TSM administrator doesn't have to restore a whole VM just to retrieve one file.

There are different flavors of STORServer available, from on-premises appliances to bare-bones software that integrates with TSM. Cooper selected the latter for Snohomish PUD No. 1; the package includes reporting capabilities and free online support.

According to Cooper, once she downloaded the software to a proxy server, it picked up all the VMs on her network and allowed her to schedule backups by just pointing and clicking. Individual files are backed up for incremental changes every night, and Cooper schedules a full-machine backup once a month for security patches and software updates.

Cooper said the STORserver user interface is easy to use. "I just go in every morning and make sure everything is backed up," she said. "It has event logs so if I'm unsure why something wasn't backed up, I can copy the log and send a ticket online" and take advantage of the online support.

Cooper is currently backing up 25 VM servers in a production environment. Cooper couldn't pin a specific dollar amount on the power saved by using virtualized storage, but said, "We do know we're saving on power."

Snohomish PUD No. 1 has also been able to gain server space by testing new software on VMs and then storing it virtually.

"We're saving a lot of time by not having to use a full-sized server and building a whole OS-it's a godsend as far as being able to do things like that for our internal customers," Cooper said.

Cooper added that it was vital for her to keep copious notes during the testing phase, especially as she was unfamiliar with the new environment.

"[STORserver] might take a little longer to install, but you need to take those notes. Did you keep all the defaults, or did you change anything? What port numbers did you use? Because then when you get to the real production environment, you're scratching your head trying to remember all those details."

Michael Hickins is a freelance writer based in New York.  

Rocket Fuel