Few companies know better than Premier Bankcard Inc. that sometimes you have to spend money to make money. Case in point: The credit card issuer has made back—and then some—the roughly $1.9 million it has devoted to a storage area network and business analytics project.
With rivals crowding consumers mailboxes with the latest credit card offers, Premier Bankcard needed to find a way to pinpoint and acquire high-value customers. In 2004—building on a project that started in 2001—the Sioux Falls, S.D., company put into play an ambitious, multivendor SAN strategy to support a highly available and centralized data warehouse.
By providing an analytics foundation that was stable and transparently scalable, Premier Bankcards Dave Geiver, vice president of technology, and Scott Erkonen, director of technology, hoped to fundamentally change the way credit cards were issued and to whom.
The results have been impressive so far. The SAN deployment played a big role in allowing Premier Bankcard to become the 14th-largest MasterCard credit card issuer and the 16th-largest Visa card issuer in the nation, according to Geiver. Using new analysis tools, Premier Bankcard employees are able to generate new and profitable business that today translates into an 8 percent yearly growth rate.
The data warehouse expansion and SAN project has proved so successful that Premier Bankcard saw a $5.6 million return within a year after its June 2004 deployment and has a projected five-year cumulative benefit of nearly $63 million—or an annual ROI (return on investment) of 740 percent. The organization also estimates that it will realize a total five-year benefit per credit card analyst of almost $1.5 million.
“As we matured and grew to hit about 2 million credit accounts in our portfolio, we realized that we were going to have to do more analytics to sustain [our goal of an annual] 8 percent growth rate,” Geiver said. “We werent talking about just doing mailings or Web sites, either. We wanted to really start to home in on our customers and on how to get to our customers with good, intelligent marketing. From an executive and organizational level, we knew wed have to invest heavily in people and in a platform to get us there.”
Premier Bankcard offers secured credit cards to customers who are trying to establish—or sometimes re-establish—credit. The company was founded in 1989 with one manager and one employee. Today, the company has 2,500 employees and about 3.7 million active credit cards. The company adds roughly 160,000 new accounts each month, according to Geiver.
Next Page: The customer is always right.
2
In an effort to capture a growing percentage of customers, Geiver and Erkonen decided in 2001 to use SAN technology to develop an analytics effort that would help analysts determine whether to offer credit to new customers.
After an ROI survey conducted by Hewlett-Packard Co. reported an estimated $3.5 million return—or about 592 percent—on the deployment of a SAN, Premier Bankcard implemented a 30GB data warehouse using an HP ProLiant 3000 server that drew data stored on an HP StorageWorks EMA (Enterprise Modular Array) 12000 storage array. By using SAN technologies, Premier Bankcard could centralize its storage, allowing the company to speed access to information and improve the productivity of its data warehouse.
“We knew we were doing things right because we had to write a business case to prove we could get huge returns from a centralized storage strategy,” Geiver said. “The potential ROI was just phenomenal and actually ended up exceeding our expectations.”
As employees increasingly relied on the data warehouse to generate reports on customers, Erkonen and Geiver began a data warehouse expansion in 2002. Premier Bankcard customers regularly use their credit cards for such day-to-day purchases as groceries, which meant the amount of data generated and stored was growing exponentially.
At the same time, Premier Bankcard was expanding internally and adding more analysts. The organization began seeing file loads several hundred megabytes in size per day. To support this growth and level of activity, the company added a second HP StorageWorks EMA array in March 2003.
At the time, users pulled information from the data warehouse into Microsoft Corp.s Excel and Access, as well as Business Objects S.A.s Crystal Reports; users worked on the files using file shares on the ProLiant server.
Over time, users began to extract massive amounts of data—some users were pulling as many as 1 million rows of data off the server and into an Excel spreadsheet. A bottleneck developed, with some queries taking as long as 22 hours, Erkonen said.
In an attempt to resolve performance issues, Premier Bankcard deployed Microsofts SQL Server 2000 database, but greater data growth followed. When the SQL Server database reindexed, for example, it would increase by as much as 30GB overnight. Finding their queries backlogged, Premier Bankcard analysts tried to work around the problem by loading queries at night and over the weekend, which had a negative impact on scheduled backup processes.
Next Page: The price of success.
3
Erkonen and Geiver said they knew after five months that the installation of the second array was more of a band-aid than a solution. Throwing more hardware at the problem wasnt going to make it go away, they reasoned at the time—Premier Bankcard needed a new infrastructure if it was going to continue to gain market share.
“There was a fundamental shift in what the business was asking IT to do as analytics began to play a very large role in growing the business,” Erkonen said. “We quickly identified that wed need to rethink everything to support rapid growth for a sustained period of time.”
Geiver and Erkonen decided that the only way to sustain Premier Bankcards high rates of data growth was to deploy a centralized, unified business intelligence platform to support the 70 employees who have access to the data warehouse. In 2004, they developed a strategy that took advantage of 64-bit computing technologies to ensure the maintenance of a highly available and centralized data warehouse, speedy access to information in the data warehouse, improved application performance, and up-to-date data loads.
In June 2004, working with HP consultants, Premier Bankcards IT staff began to overhaul its infrastructure and installed an HP Integrity rx8620-32 Server running Microsofts 64-bit SQL Server 2000 for mission-critical business intelligence. An HP Integrity rx5670 Server was brought in to run SAS Enterprise Miner r8, while an HP Integrity rx4640-8 Server was deployed to run Microsoft Analysis Services. Premier Bankcard also decided to move to the 64-bit version of Windows Server 2003 in hopes of increasing performance and throughput.
Erkonen and Geiver took advantage of an existing HP Enterprise Virtual Array, adding 25.5TB to serve as the cornerstone of the companys SAN and provide centralized data access.
With a single data warehouse and a unified set of data, multiple employees can now simultaneously load files and query the data warehouse without contention. The queries that used to take as long as 22 hours now take roughly 45 minutes. A profitability query (a query that determines the profit derived from a particular customer) that used to take 13 minutes now takes as little as 36 seconds, Geiver said.
“One of our goals has always been to determine the profitability of individual credit card accounts in our portfolio, but that meant digging through millions of accounts and transactions,” Geiver said. “We now have the data modeling, mining and analytical tools we need to [efficiently] determine profitability.”
Erkonen said the time required to load data into the warehouse has decreased by 30 percent and that performance has improved by as much as 100 percent. Analysts now spend from 80 to 90 percent of their time analyzing data, up from 50 percent.
4
With the success of customer analytics, however, came another explosion in data growth. In 2004, Geiver and Erkonen estimated that the data warehouse would grow at a rate of about 12GB a day; today, the warehouse is growing at a rate of 18GB per day.
To handle the backup and storage management of all this data, Premier Bankcard decided to use Veritas CommandCentral from Symantec Corp. Erkonen said CommandCentral has enabled the automation of many manual functions, including capacity utilization and trending, and has also provided alerting capabilities. CommandCentral has been deployed to 70 percent of the companys SAN-based systems and will be deployed to the remaining systems by next month.
Looking ahead, Premier Bankcard is evaluating several technologies to make operations even more efficient, including HSM (hierarchical storage management) and ILM (information lifecycle management), as well as disk-based backup.
Geiver and Erkonen also are applying lessons learned from their success with SANs on the analytics side to their production networks. Theyre combining two SAN fabrics at the companys primary location in Sioux Falls using Brocade Silkworm Director from Brocade Communications Systems Inc. By using Director instead of a mesh design with multiple smaller switches, Geiver and Erkonen estimate that costs can be driven down by 20 percent per port.
As part of this SAN consolidation, Geiver and Erkonen are also considering replacing four EMA storage arrays with one EVA 8000 storage array, a move that will save the company an estimated $850,000 in maintenance and support, Erkonen said. At the same time, the EVA storage array at Premier Bankcards backup site in Watertown, S.D., will be expanded by 125TB of capacity.
Premier Bankcard is also exploring options for messaging storage. The company is currently using Novell Inc.s GroupWise; Geiver and Erkonen are considering a move to Microsofts Exchange 2003, which would enable them to replicate the Exchange environment to the Watertown backup data center. Such a move would provide high availability, using either HSM or ILM technologies to ensure that the system is being archived in a manageable way.
“We have been working with our compliance group to determine retention requirements, and what we have determined will be the best course of action is to put user size and time limitations into place while providing our users a better file share methodology for document retention,” Geiver said. “We will utilize policy-based systems to ensure the messaging system remains manageable and to decrease our file share backup sizes through the use of HSM technologies.”
One of the most challenging storage projects that Geiver and Erkonen are working on is finding the right encryption technologies for their environment. In an industry plagued by consumer privacy concerns and stolen data, the two are constantly searching for ways to ensure that data remains safe.
As a credit card institution, the company is held to industry data security standards that require—at minimum—account numbers to be encrypted on disk. Rather than relying on host-based encryption or selective database encryption that would incur high overhead, Premier Bankcard is planning to address encryption in the storage fabric itself. Geiver and Erkonen hope to develop an encryption strategy within the next six months to a year.
“Identity theft has brought an awareness to many businesses that you really have a responsibility to take care of your assets and that the business has to change with the environment if you want to continue being successful,” Geiver said. “There shouldnt even have to be a law—its just common sense. Were ahead of the curve here. We realize this is where the industry needs to move.”
Senior Writer Anne Chen can be reached at anne_chen@ziffdavis.com.
