Fit To Be Tied

Bad co-location deals can drive your customers crazy. But good service-level agreements can keep the peace.

As outsourcing of complex Internet functions becomes the norm, its becoming increasingly important for service providers and their customers to come up with effective service-level agreements (SLAs).

With the recent downturn in the Internet economy, there is an abundance of rack and floor space in co-location facilities. Thus, while prospective customers formerly might have accepted contract terms at face value, we now see their requirements and expectations gaining the upper hand. Furthermore, as SLAs mature, negotiation is more commonplace; the result is both increased competition between service providers and increasingly customer-focused SLAs. What were focusing on here are co-location providers that house client-owned servers as opposed to hosting providers that furnish servers and manage Internet sites.

Coming to Terms To begin with, there are several reasons why companies sign up for co-location. Typically, attractive offerings include high bandwidth and access to multiple backbone providers, among other features. But these days relationships are becoming more complex, as providers offer additional services and customer requirements reach new levels. While bandwidth is still a prime consideration, co-location companies now earn upwards of 75 percent of their income from other services such as hardware/software procurement, data storage, backup and site security.

The difficult task of writing an SLA begins with defining and negotiating measurable requirements and remedies for both standard and client-specific service items. The goal is to transition from implied terms to expressed terms; otherwise, there are too many nuances to be overlooked or misunderstood. Ambiguous agreements lead to problems at a later date when interpretation becomes difficult. Good SLAs, on the other hand, are objective, agreed upon and enforceable.

While customers might assume that basics are covered, savvy clients make sure adequate service is provided both for regular operations and for the inevitable emergency or disaster. Following is more information on a few of the standard features. Before signing up, pay attention to the following:

Obtaining Services The time to install a customer varies, depending on the provider. When asked how long it takes to provision services, Verio indicates that, depending on customer requirements, "standard" cabinet or cage installations normally take from five to seven business days. Custom builds at Verio can take up to several weeks. At, customer circuits will be delivered in "45 business days or less" from the day their provisioning department accepts an order. When this timeframe is not met, Broadwing offers credit for a months free service.

The importance of getting specifics on the basics is demonstrated when technical glitches force clients to withdraw from a contract before the ink has dried on the dotted line. One resource at a large co-lo provider revealed that a top 10 Fortune 500 company pulled out before 30 days had passed.

Specifically, the client had spent several weeks installing servers and working on security with a third-party firewall vendor. Apparently, in the end it all boiled down to the client not being able to access their site externally via PC-Anywhere using a laptop connection through AOL. The co-lo provider could not resolve the problem and wouldnt claim responsibility for possible technical issues with the third-party vendor.

While an isolated example, fortunately the contract permitted termination within 30 days when specific conditions could not be met. Regardless, a lot of time and energy was spent.

Power and Backup Power While commercial power often is taken for granted, paying for service at a co-location facility requires knowing what youre getting and what happens in the event of an outage.

For example, it is important that your equipment have connections to multiple power-distribution units at the co-lo facility. Without redundancy, changes to your setup may require power-downs and service interruptions. One large customer we spoke to needed to increase their power requirements to accommodate more servers. After submitting their request, they found that a lack of overlapping power required their equipment to be shut off while changes were made. When they asked the provider for an explanation, they were told that to place new connections into existing live circuits, other customers would be affected. This complication increased the turnaround time involved because the other clients had to be notified. Finally, to perform the work, outside contractors had to be brought in.

Had the customer known and understood the upgrade capabilities of their power setup and that outside qualified electrical service personnel would be required, they could have planned for and requested their changes sooner and avoided downtime and lost revenue.

Backup power systems are also complex, and battery power typically lasts for less than half an hour, so youll need to be familiar with how the uninterrupted power supply (UPS) system transitions to generator power. Considering the rolling blackouts now taking place in California, it is also imperative to know about fuel storage on-site at the co-lo facility, for how long the supply can power the generators, and how readily the supply can be replenished. Also, some generators run on multiple types of fuel, and knowing this provides further options and security.

Co-location providers may advertise that their fuel supplies permit generators to run for days at a time. However, with intermittent blackouts a real possibility this summer, it becomes increasingly important to know that fuel tanks will be topped off at regular intervals. Without a full tank, occasional generator use can result in an empty tank.

The Heat Is On Environmental controls have always been central to operations. Therefore, when considering a co-location provider, its important to know as much as possible about their heating, ventilation and air conditioning (HVAC) capabilities.

According to Kevin Facinelli, director or operations at co-location portal, with computer density per square foot increasing due to thinner systems and more efficient rack space, increased heat output has a greater impact on room temperature buildup in the event of a power loss. Even though a UPS may keep computers running, the temperature inside a data center can quickly rise due to built-in delays of 5 minutes or so before HVAC systems restart and generators are up and running.

It is during this "ride-through" period that the mercury may reach upwards of 100* F and cause computer hardware reliability to decrease. Should temperatures continue to increase unchecked, it becomes difficult for personnel to work on equipment. To prevent such a scenario, customers need to know, for example, how their co-lo provider manages heat during a crisis. Do their chillers produce enough chilled water to compensate for such a gap?

Research Network Redundancy With multiple wideband network connections a primary reason to go co-lo, it becomes imperative to know up front exactly what level of network redundancy is physically built in to the facility.

Provider sales personnel may say that youll have access to several carriers. This may sound appealing during a presales walk-through, but people tend to hear what they want and neglect probing the surface. One co-lo Web site, for example, advertises "several fiber entrances." Yet, when we reached a building manager at one of their facilities, he stated that, at his location, all the fiber comes into the building at one point. While he further assured us that the connection was encased underground to prevent a disturbance, such an oversight becomes a single point of failure when construction digs take place. Ideally, your co-location facility will offer multiple carriers with more than one physical entry into the facility. Get it in writing.

Bandwidth Considerations Depending on the type of business, it can be difficult to anticipate bandwidth needs. Nonetheless, knowing what is available from your co-lo and being able to make changes from time to time is part of an effective SLA. High-end Web sites, for instance, must be able to handle spikes in traffic that come during the holiday season, advertising campaigns and special promotions.

A case in point involved an entertainment company due to release a new version of an online game. To be able to handle increased traffic, it was essential for bandwidth to be augmented by a specific time. The customer made the request and specified the product would not successfully hit the market unless needed amendments were made by a certain time. As it turned out, lengthy policies and procedures combined with overburdened staff at the co-lo made the deadline impossible to meet.

Further probing revealed that when their service contract came up for renewal, they instead went ahead with plans to build a site closer to their head office. This option is possible for clients that are willing to make the investment and that are near backbone fiber connections and ample power supplies.

Perhaps better planning on the part of the customer could have helped them to anticipate their needs ahead of time, but the bottom line is that their requirements were not met and their product launch had to be pushed back. As with our other examples, had both parties better managed expectations, the situation might have been avoided.

Negotiating Additional Services Chances are, your data is worth considerably more to your business than your servers. With this in mind, youll want to do everything possible to keep both your equipment and applications up and running. After all, from the customers perspective, "its all about uptime." On the co-location side, enhanced services are what your provider wants you to buy. This is the one area where you can maximize the benefits of your service agreement.

In order to get what you really want in writing, the following are necessary: performance standards, knowing whos responsible for specific items, and—to cover Murphys Law—explicit terms for accountability and resolution. To minimize risk, hardware and software monitoring are necessary to decrease maintenance and prevent breakdowns. Here are two important items to consider:

Claiming Support The most common support items handled by support staff at the co-lo are backups (primarily tape changes) and system reboots. Ideally, customers want to call the hotline as infrequently as possible and to seldom have technicians work on their machines.

Even so, in the event service is needed, you want your contract to include the highest level of support required by your business needs. To ensure quality service, find out who is on staff, and what technical training and certifications are held by support personnel. For example, is a Cisco certified expert on hand at all times? It will be frustrating if your highly skilled technician is on the phone with someone who needs guidance to change an IP address. Ask a lot of questions beforehand.

When negotiating complex installations, you may require highly customized support options. This is when you will need to work with the provider to establish and follow stringent procedures to adequately address service needs. For example, you may require that certain technicians be dedicated to your site. In addition, you may need the ability to escalate trouble calls when necessary.

For many customers, structured service levels are an acceptable option. In such a scenario, you may be able to sign up for basic support with the ability to upgrade to a higher level of service as needs change. This type of arrangement can benefit both providers and customers as the ground rules are laid out in explicit, tested terms and require less negotiating. When your requirements change, you will know ahead of time what is included.

Monitoring, Measurement and Client Notification To be successful, a co-location provider must know why customers engage their services. Once needs are known, the provider can determine service standards and which items to monitor, measure and share with clients. Several providers advertise site-monitoring tools available to clients through a Web browser. However, none of the co-location companies we contacted offered much more than connectivity and bandwidth monitoring. As long as a site is up, little else is taken into consideration.

In terms of monitoring client systems and sharing related statistics, the prime factors given by providers center on restoring service in the event of equipment breakdowns.

For example, monitoring tools at the co-lo might include temperature sensors at multiple points in a room. Centralized monitoring equipment is akin to having a smoke detector in a hallway when it would be more suitable to have one in each room. Rather, the best plan for both parties is to have a comprehensive risk-management strategy.

Fortunately for customers, the limitations of provider-based server monitoring are in tune with customers goal to limit service requests. Instead, customers are incorporating hardware- and software-monitoring tools into their systems. Machine diagnostics can provide useful data regarding the operating system, hardware and applications.

Looking outward from within the system enables a greater variety of items to be monitored. Hardware-based tools in this category range from Compaqs Insight Manager to platform-independent products such as Crystal Groups DarkSite Remote Management Appliance. Beyond the standard ISA card-based hardware-monitoring capabilities are further enhancements such as software that may be installed to monitor applications.

Items to be monitored range from system temperature, fan rotation, CPU, RAM, disk arrays and network connectivity. As expected, remote reboots are possible and access is via a Web interface or remote connections.

Planning for remote management or your off-site system can help to close the gap in service interruptions. Not only will this save your site, but it also will save you time. Kevin Facinelli provides a real-world example, which really drives the point home. His scenario involves a down system at a distant co-lo facility. Without effective remote tools or a standby server on hand, an immediate site visit can be necessary. Rather than having to rush into such a situation, wouldnt you rather be able to plan your trip in advance and make better use of your time?

As co-location infrastructure gains in complexity, automation will become more common and monitoring tools increasingly will be tied to reports measured against SLAs. Products will either notify site personnel of needed technical adjustments or handle changes automatically. As with ERP software that ties into back-end systems, monitoring products will connect into client billing systems and beyond.

Until more advanced system-management tools are in place, perhaps the best way to see your SLA meet your objectives is to aim to be your own support center.