Like any company bringing new products or services to market, Snap Appliances Inc. caps development of its server appliance products with rigorous beta testing. However, the periodic nature of beta tests, along with the need for accurate input from a products users, can make in-house beta testing an unwieldy task for IT departments to manage effectively.
After conducting several beta tests using Microsoft Corp.s Microsoft Access database-driven system and labor diverted from engineering and technical support staffs, Snap official went looking for a better way to beta. This subsidiary of Quantum Corp., which holds a significant share of the network-attached storage system market, called on BetaSphere Inc. to help design and administer its beta testing programs. By using BetaSpheres services, the company aimed to streamline the process, get more accurate results, and save money and time.
“[Beta testing] doesnt happen constantly, so it doesnt make sense to hire someone to always do it [in-house],” said Jeff Hill, Snaps senior director of product marketing.
eWeek Labs traveled to Snaps San Jose, Calif., headquarters to check out the BetaSphere service. Like Hill and his staff, we were impressed during our Labs On-site visit with the ease and completeness with which BetaSphere enabled Snap to conduct tests.
BetaSpheres FMS (Feedback Management Server) is a hosted, Web-based application through which testers submit product feedback that Snap developers then address and incorporate into the products.
“Were getting three to four times more information now than when we did it ourselves,” Hill said.
Snap, which has conducted four beta tests with BetaSphere and is planning a fifth, negotiates the terms of its tests with BetaSphere on a case-by-case basis. BetaSphere also offers its services on a retainer-based fee structure. (Prices vary based on a companys needs and the size of the beta test.)
Snaps test of Version 3.0 of its server appliance software, which spanned 30 test sites and lasted three months, cost less than $50,000. Although Snap officials said they hadnt done a cost-benefit analysis of outsourced vs. in-house beta testing, the major in-house costs were for the time that employees were diverted to run the tests.
“Theyve saved us a lot of money, and theyve saved us some launch disasters,” Hill said. During tests of the Snap Server 1000 appliance, for example, participants from a BetaSphere-managed beta group identified a problem with the appliances power switch—testers were confused by the design of the switch, and they conveyed their concerns to Snap.
“Instead of making the decision to retool in one week, we would have made it in three with our own beta,” Hill said. The extended retooling delay would have pushed back the product launch, costing Snap money and frustrating its marketing efforts.
Although outsourcing management of the beta testing stage of their product development process made sense for Snap, this may not be the best path for all companies. For enterprises that conduct continuous beta tests, it may make more sense to bring the test administration efforts in-house, under the care of employees who can devote all of their time to the task.
For companies that wish to follow this path but still wish to use BetaSpheres FMS, the company allows customers to license its software for in-house hosting. The FMS runs on Linux 2.x or Solaris 2.5 servers and stores its data on an Oracle8.x database.
A Few Good Testers
Before each beta test begins, snap meets with BetaSphere to lay out its test requirements, including the number of testers needed and the geographic locations, company sizes, and supported platforms of the testers it seeks to use.
Under the parameters laid out by Snap, BetaSphere handled recruitment and management of beta testers. BetaSphere finds participants through advertising, trade shows and its Web site.
“Without help from BetaSphere, itd be hard for us to come up with the group of users we need,” Hill said. “In the past, sales would present a group of users.” Snap found beta groups drawn from sales leads were a generally unenthusiastic crowd whose feedback would tend to trail off once a purchase decision had been made.
“We look for people who like to test. These people do a better, more comprehensive job than sales evaluators,” Hill said.
Drawing on its database of volunteer testers and a list of potential testers provided by Snap, BetaSphere screens testers to represent a variety of the products likely audience and selects those with the time and interest to provide valuable feedback. “It seems like its their goal in life to make it crash,” said Kevin Osborn, Snaps director of software development.
Spies Neednt Apply
Security is a potential concern for customers in any beta test, and BetaSphere screens its potential testers with an eye toward eliminating employees of competing companies.
For Snap, the FMS was configured to route feedback via e-mail to the appropriate members of the development team as the feedback came in. “Before, youd get a big block at once—now its real time,” Osborn said.
In addition to the e-mail updates, Snap can generate reports from the FMS and can export data for use with other database or analysis applications.
During the testing cycle, representatives from BetaSphere periodically contact a programs testers to ensure that the product being tested has been properly installed, as well as ask additional questions provided by Snap.
The BetaSphere representative then sits in on Snaps weekly beta meetings—either by phone or in person—and provides a sense of how testers are receiving the product, as well as collecting questions for the next round of tester contact.
For Snap, the close screening and high level of tester interaction from BetaSphere resulted in much higher rates of tester participation than theyd experienced when beta testing on their own. “At least 66 [percent] to 90 percent of testers give good feedback throughout [the tests],” Hill said.
