Questions for the Provider
Since a SAAS provider's infrastructure is-from the perspective of its customers-largely a black box system, one of the most important reliability-related SAAS characteristics to consider is the provider's service health dashboard, the system through which a SAAS vendor provides its users with information regarding service health and interruptions. Since most SAAS providers make their status pages available to the public on their Web sites, this is a good place to begin your SAAS provider reliability evaluation.
Make sure that the SAAS vendor you're evaluating provides information about the uptime and performance state of each of the discrete services it offers, and that the information is accurate, up-to-date and easy to locate. It's also important that the SAAS provider offers information related not only to current status, but also to historical status. What's more, the SAAS provider should offer ways to notify you of outages or service degradations, such as through e-mail alerts or RSS feeds that administrators can integrate into their own infrastructure-monitoring dashboards.
When it's time to discuss reliability with the SAAS provider, SLAs (service-level agreements)-which lay out both the uptime and performance characteristics that customers can expect from their providers, as well as the compensation customers will receive when these commitments are not met-are a natural starting point. However, many SAAS providers, including well-established outfits such as Salesforce.com, do not offer SLAs, due perhaps to challenges of defining SAAS reliability with enough rigor to suit a formal agreement.
If your prospective SAAS vendor will accept an SLA, the agreement can serve as a useful tool for matching up your organization's expectations with the level of service the vendor is prepared to offer, particularly if your SLA drills down beyond basic uptime to include specific performance metrics.
However, the burden of monitoring uptime and claiming compensation for service outages and degradations typically falls on the customer, and partial credits on the SAAS usage bill won't make up for significant application downtime. Whether or not the SAAS provider you're evaluating offers an SLA for its service, the most important thing is that the vendor proves capable of meeting your uptime and performance needs.
You should discuss with the SAAS provider how its architecture deals with surges in load, and how it is prepared to scale to additional customers without affecting your application reliability. Also, you should ask about the geographical distribution of the vendor's data centers-the closer the data centers are to your users, the better the performance you can expect.
In addition, you should talk with the SAAS provider about customizations it plans to make to its applications and consider the potential performance implications of those customizations. Along similar lines, you should find out about software update schedules and discuss what sort of impact these updates have on performance. If the updates typically occur at a time when you require full uptime and performance from your application, discuss whether you can control the time when the updates occur.
Regarding the SAAS application itself, find out to what extent the vendor's application takes advantage of client-side execution or RIA (rich Internet application) technologies, which can reduce the impact of Internet connectivity issues-such as poor latency-by reducing the server-to-browser round trips required to execute certain operations, such as item sorting.
Along similar lines, make sure to ask the SAAS vendor whether its application makes any provisions for offline data access. For example, SAAS e-mail services tend to have a built-in offline access story, as both the IMAP and Exchange MAPI protocol work with local mail clients to keep mail available locally as well as in the cloud. With other applications that have fewer standardized data protocols, the offline story might be less attractive. However, technologies such as Google's Gears, which Google and Zoho employ for some of their SAAS offerings, allow for some offline data access. Even SAAS vendors without a current offline strategy should at least be able to discuss where such capabilities fit into their product road maps.