Close
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Logo
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Development
    • Development

    Better Beta Testing

    By
    Jason Brooks
    -
    May 28, 2001
    Share
    Facebook
    Twitter
    Linkedin

      Like any company bringing new products or services to market, Snap Appliances Inc. caps development of its server appliance products with rigorous beta testing. However, the periodic nature of beta tests, along with the need for accurate input from a products users, can make in-house beta testing an unwieldy task for IT departments to manage effectively.

      After conducting several beta tests using Microsoft Corp.s Microsoft Access database-driven system and labor diverted from engineering and technical support staffs, Snap official went looking for a better way to beta. This subsidiary of Quantum Corp., which holds a significant share of the network-attached storage system market, called on BetaSphere Inc. to help design and administer its beta testing programs. By using BetaSpheres services, the company aimed to streamline the process, get more accurate results, and save money and time.

      “[Beta testing] doesnt happen constantly, so it doesnt make sense to hire someone to always do it [in-house],” said Jeff Hill, Snaps senior director of product marketing.

      eWeek Labs traveled to Snaps San Jose, Calif., headquarters to check out the BetaSphere service. Like Hill and his staff, we were impressed during our Labs On-site visit with the ease and completeness with which BetaSphere enabled Snap to conduct tests.

      BetaSpheres FMS (Feedback Management Server) is a hosted, Web-based application through which testers submit product feedback that Snap developers then address and incorporate into the products.

      “Were getting three to four times more information now than when we did it ourselves,” Hill said.

      Snap, which has conducted four beta tests with BetaSphere and is planning a fifth, negotiates the terms of its tests with BetaSphere on a case-by-case basis. BetaSphere also offers its services on a retainer-based fee structure. (Prices vary based on a companys needs and the size of the beta test.)

      Snaps test of Version 3.0 of its server appliance software, which spanned 30 test sites and lasted three months, cost less than $50,000. Although Snap officials said they hadnt done a cost-benefit analysis of outsourced vs. in-house beta testing, the major in-house costs were for the time that employees were diverted to run the tests.

      “Theyve saved us a lot of money, and theyve saved us some launch disasters,” Hill said. During tests of the Snap Server 1000 appliance, for example, participants from a BetaSphere-managed beta group identified a problem with the appliances power switch—testers were confused by the design of the switch, and they conveyed their concerns to Snap.

      “Instead of making the decision to retool in one week, we would have made it in three with our own beta,” Hill said. The extended retooling delay would have pushed back the product launch, costing Snap money and frustrating its marketing efforts.

      Although outsourcing management of the beta testing stage of their product development process made sense for Snap, this may not be the best path for all companies. For enterprises that conduct continuous beta tests, it may make more sense to bring the test administration efforts in-house, under the care of employees who can devote all of their time to the task.

      For companies that wish to follow this path but still wish to use BetaSpheres FMS, the company allows customers to license its software for in-house hosting. The FMS runs on Linux 2.x or Solaris 2.5 servers and stores its data on an Oracle8.x database.

      A Few Good Testers

      Before each beta test begins, snap meets with BetaSphere to lay out its test requirements, including the number of testers needed and the geographic locations, company sizes, and supported platforms of the testers it seeks to use.

      Under the parameters laid out by Snap, BetaSphere handled recruitment and management of beta testers. BetaSphere finds participants through advertising, trade shows and its Web site.

      “Without help from BetaSphere, itd be hard for us to come up with the group of users we need,” Hill said. “In the past, sales would present a group of users.” Snap found beta groups drawn from sales leads were a generally unenthusiastic crowd whose feedback would tend to trail off once a purchase decision had been made.

      “We look for people who like to test. These people do a better, more comprehensive job than sales evaluators,” Hill said.

      Drawing on its database of volunteer testers and a list of potential testers provided by Snap, BetaSphere screens testers to represent a variety of the products likely audience and selects those with the time and interest to provide valuable feedback. “It seems like its their goal in life to make it crash,” said Kevin Osborn, Snaps director of software development.

      Spies Neednt Apply

      Security is a potential concern for customers in any beta test, and BetaSphere screens its potential testers with an eye toward eliminating employees of competing companies.

      For Snap, the FMS was configured to route feedback via e-mail to the appropriate members of the development team as the feedback came in. “Before, youd get a big block at once—now its real time,” Osborn said.

      In addition to the e-mail updates, Snap can generate reports from the FMS and can export data for use with other database or analysis applications.

      During the testing cycle, representatives from BetaSphere periodically contact a programs testers to ensure that the product being tested has been properly installed, as well as ask additional questions provided by Snap.

      The BetaSphere representative then sits in on Snaps weekly beta meetings—either by phone or in person—and provides a sense of how testers are receiving the product, as well as collecting questions for the next round of tester contact.

      For Snap, the close screening and high level of tester interaction from BetaSphere resulted in much higher rates of tester participation than theyd experienced when beta testing on their own. “At least 66 [percent] to 90 percent of testers give good feedback throughout [the tests],” Hill said.

      Jason Brooks
      As Editor in Chief of eWEEK Labs, Jason Brooks manages the Labs team and is responsible for eWEEK's print edition. Brooks joined eWEEK in 1999, and has covered wireless networking, office productivity suites, mobile devices, Windows, virtualization, and desktops and notebooks. Jason's coverage is currently focused on Linux and Unix operating systems, open-source software and licensing, cloud computing and Software as a Service. Follow Jason on Twitter at jasonbrooks, or reach him by email at jbrooks@eweek.com.
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.

      MOST POPULAR ARTICLES

      Latest News

      Zeus Kerravala on Networking: Multicloud, 5G, and...

      James Maguire - December 16, 2022 0
      I spoke with Zeus Kerravala, industry analyst at ZK Research, about the rapid changes in enterprise networking, as tech advances and digital transformation prompt...
      Read more
      Applications

      Datadog President Amit Agarwal on Trends in...

      James Maguire - November 11, 2022 0
      I spoke with Amit Agarwal, President of Datadog, about infrastructure observability, from current trends to key challenges to the future of this rapidly growing...
      Read more
      IT Management

      Intuit’s Nhung Ho on AI for the...

      James Maguire - May 13, 2022 0
      I spoke with Nhung Ho, Vice President of AI at Intuit, about adoption of AI in the small and medium-sized business market, and how...
      Read more
      Applications

      Kyndryl’s Nicolas Sekkaki on Handling AI and...

      James Maguire - November 9, 2022 0
      I spoke with Nicolas Sekkaki, Group Practice Leader for Applications, Data and AI at Kyndryl, about how companies can boost both their AI and...
      Read more
      Cloud

      IGEL CEO Jed Ayres on Edge and...

      James Maguire - June 14, 2022 0
      I spoke with Jed Ayres, CEO of IGEL, about the endpoint sector, and an open source OS for the cloud; we also spoke about...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2022 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×