Charlie Crystle, CEO of Mission Research, a developer of fund-raising software for nonprofit organizations, based in Lancaster, Pa., dropped on-demand software vendor Salesforce.com Inc. over recent outages. Cliff Bell, CIO of Phoenix Technologies Ltd., a developer of embedded software in Milpitas, Calif., is forgiving about outages. And Melissa Caylor, director of IT at mortgage broker First New England Mortgage Corp., of Newton, Mass., will change her internal practices to cope with any pain caused by downtime.
Add it up and Salesforce.com CEO Marc Benioff may have some explaining to do at the companys Jan. 17 AppExchange Day and Winter 06 release conference in San Francisco.
On Dec. 20, Salesforce.com, which provides CRM (customer relationship management) software, suffered an outage for nearly 5 hours.
On Jan. 6, the company confirmed “a minor interruption” in the Europe, Middle East and Africa regions.
In a dozen interviews with Salesforce.com customers and partners, eWEEK has found sporadic outages over the last year.
Bruce Francis, the companys vice president of corporate strategy, said the Dec. 20 outage was caused by an “extremely rare, undocumented bug” in its databases provided by Oracle Corp. Francis said the Jan. 6 outage was resolved quickly but didnt elaborate further.
In addition, Benioff said some customers saw “performance degradation” in the last 90 days because the company was prepping a new data center, which was turned on in November and can support 1 million subscribers, up from 350,000. “We, of course, had to work out a few kinks,” Benioff said, adding that the Dec. 20 outage wasnt related to the data center startup.
Salesforce.coms recent problems are far from becoming an epidemic, but they do raise key questions about on-demand software. Should customers expect 99.999 percent reliability from a hosted application service? Is Salesforce.com the proverbial canary in the coal mine as other hosted services grow? How should customers deal with these outages when they occur?
Salesforce.Com: Is software as a service reliable?
Any problems at Salesforce.com may be a product of its own success, customers say. As of Oct. 31, Salesforce.com had 18,700 customers, up 115 percent from the 8,700 it had on Jan. 31, 2004, and up 434 percent from the 3,500 it had on Jan. 31, 2002. That growth has made Salesforce.com a Wall Street darling, with shares gaining 248 percent through Jan. 11, up from an $11 IPO (initial public offering) price on June 23, 2004.
Too much, too fast?
While Salesforce.coms growth has been stellar, it worries customers such as Don Devost, director of finance and administration in the worldwide sales division at Analog Devices Inc., a Norwood, Mass., electronics manufacturer.
“I am concerned about Salesforce managing their growth and being able to deliver high uptime and good service,” Devost said.
Analog Devices, which uses Salesforce.com to support 800 salespeople, has seen an increase in alerts from Salesforce.com reporting that its “experiencing problems with the server.”
“Generally, what we see is periodic slowdowns, and weve not had very many user complaints,” Devost said. These problems are appearing “at the same time their customer base is growing rapidly, so I would have to say there is a correlation,” he said. “I wouldnt want to keep on this trajectory for another 12 to 18 months. That would be problematic.”
Phoenix Technologies Bell said Salesforce.coms outages are getting attention largely because its system is “out in the open” on the Internet and outages are high-profile, compared with downtime at internal technology departments. He compared Salesforce.coms problems to the high-profile outages eBay Inc. had to contend with in the late 1990s. “[eBay] had to clean that up, and they did,” Bell said.
Customer choices
The big question for customers of Salesforce.com, or any other hosted application service, is, What are your choices in an outage?
Benioff maintains that outages are not commonplace. He said Salesforce.coms system availability is still more than 99 percent and “has never been below 99 percent in our entire history.”
However, some customers interviewed by eWEEK said Benioffs benchmark may not be enough. These customers take three approaches: Find a new provider, live with outages or come up with a workaround to get access to applications offline.
Mission Researchs Crystle, who signed up with Salesforce.com in March, is moving on. He said the performance of Salesforce.coms CRM applications started declining in July and August and became a big issue in September.
Meanwhile, slowdowns seemed to occur during the last 10 days of the month—a time when Mission Research closes the bulk of its sales. “We are very much based on that monthly close. And that last week is always just terrible, with a lot of click and wait, click and wait,” Crystle said.
On Dec. 20, privately held Mission Research booked near “zero sales.” “That was a huge hit to us,” Crystle said, adding that the outage cost his company 5 percent of monthly sales. He declined to estimate his cash loss.
By that last outage, Mission Research had already made plans to bail out on Salesforce.com. Crystle decided in early November that the company needed to build its own internal application based on its fund-raising software for nonprofits. Crystle acknowledged that his internal system “isnt without its glitches,” but he said they are his and he can fix them. Crystle will retain one Salesforce.com account to compile historical reports for 2005.
Next Page: Other solutions to the downtime problem.
Other Solutions to the
Downtime Problem”>
Bell said hes staying put with Salesforce.com, that the companys new data center will resolve performance issues and that recent issues are “part of their growing up.” Bell, however, can be forgiving because outages havent affected his sales.
It also has helped that Bell has been in touch with Salesforce.com frequently. “Theyve been pretty good about that,” he said.
For First New Englands Caylor, the Dec. 20 outage was a serious matter.
The mortgage broker uses Salesforce.com to distribute and track leads for 90 employees covering 26 states, as well as to receive all the companys Internet leads.
First New England also integrates Salesforce.com with its loan-origination system and with its credit vendor.
“A lot of our world is in Salesforce, so when it goes out, its bad,” Caylor said. “People just basically panicked. They couldnt get to their leads, and thats our main business.”
Caylors solution: Keep a local copy of Salesforce.coms applications. “If there are outages of a day or less, we probably wont do anything about it [in terms of looking to another vendor], but well have access to our data,” Caylor said.
Rebecca Wettemann, vice president of research at Nucleus Research Inc., of Wellesley, Mass., said Caylors approach makes sense. Salesforce.com customers also can buy offline editions of the companys applications that provide some capabilities if Internet service is down.
“If youre using an on-demand application, uptime is an issue, but its more important to consider how you are going to provide offline access if Salesforce.coms servers are down,” Wettemann said. “The Holy Grail is not uptime but how you minimize the impact.”
Great expectations (for the money)
Whether Salesforce.coms growing pains hurt the rest of the hosted application service industry remains to be seen, but peers are watching closely to see if customer confidence is shaken.
John Webster, executive director of the Center for Remote Enterprise Systems Hosting Inc., of Sioux Falls, S.D., said he is worried that the Salesforce.com outages will hurt his business. Websters organization hosts Oracle PeopleSoft and Hyperion Solutions Corp. applications, primarily for universities and government agencies.
The outage “puts a negative connotation on hosted applications,” Webster said. “In the hosting arena, you put forth an image of reliability, and it doesnt help when a prominent company has an outage.”
Nevertheless, Webster bets Salesforce.com will keep its customers due to the prices it offers—as low as $65 per month per user. “The trade-off is dollars for service,” Webster said. “Maybe youre willing to accept an outage at the Salesforce price points.”
Webster said he tries to control growth by adding two big customers each quarter and making sure he has the capacity to support them. Even if providers of hosted services have uptime that exceeds most internal technology departments, a perception issue remains.
“Uptime of 99.999 percent means you have about 5 minutes a year of downtime,” Webster said. “As a hosting company, thats too much, but at some point things fail.”
Zach Nelson, CEO of Salesforce.com rival NetSuite Inc., a San Mateo, Calif., software-as-a-service provider, agrees that uptime is a key issue. NetSuite announced 18 months ago a money-back guarantee on application availability that promises an uptime of 99.5 percent per month.
The challenge for vendors is balancing uptime and costs. Nelson added that when vendors offer 99.999 percent availability, every nine becomes “exponentially more expensive to deliver” because of the infrastructure needed to build redundancy.
Another issue that the on-demand software industry may face is stricter SLAs (service-level agreements). In a November regulatory filing with the Securities and Exchange Commission, Salesforce.com said that it has entered into agreements with “a small number of its customers warranting certain levels of uptime reliability and performance.” If Salesforce.com falls short of requirements, it offers credits, but it has set aside only $500,000 to cover credits and, as of November, had paid only one customer.
Wettemann said its possible Salesforce.com and its peers will see more SLAs in the future. “SLAs are going to be considered more and more with on-demand software,” she said. “Its just another layer of protection.”
Executive News Editor Larry Dignan contributed to this article.