Putting Open Source to the Test

During the weeks leading up to our IT stack testing, eWEEK Labs tried out several performance testing tools. We considered several commercial (and often pricey) products, and decided ultimately to use Borlands SilkPerformer.

One tool that didnt make the cut for our tests but is worthy of consideration for internal application and product testing is OpenSTA (www.opensta.org).

This product isnt among the best weve seen in any area—it doesnt have the best interface for configuring tests, and its built-in reports options are fairly weak, for example.

Click here to read find out the results of eWEEK Labs performance tests of open-source and .Net IT stacks.

But OpenSTA has one clear advantage over its expensive commercial competitors: Its a free, open-source product that, despite its limitations, lets businesses carry out high-level tests of enterprise Web applications.

Commercial products usually offer some form of trial version that can be freely downloaded, but they typically come with strict limitations on the number of virtual users that can be created (often less than 10). With OpenSTA, we were able to easily create thousands of virtual users running on several linked agent systems.

Another possible drawback to OpenSTA is that it runs only on Microsoft Windows systems. Nonetheless, it installed easily on our Windows XP test systems, and, using name server configurations, we were able to link agent-only systems to a master system that managed the actual performance tests.

We ran OpenSTA through several tests of Web-based applications. Like most Web application performance testing tools, OpenSTA includes a recorder application that launches a browser and then creates a test script from the actions recorded in the browser.

Configuration of the actual test scenario was a bit time-consuming—we had to go through many more steps than we did using most of the commercial options. Still, once we finished setting up the scenario, it provided the type of deep-level testing that we required.

During tests, OpenSTA provided basic real-time feedback, though much less so than competing products do. Once the testing was completed, we could see a few basic results, but the built-in reporting proved to be the weakest part of OpenSTA (and one of the main reasons we didnt use it for our IT stack tests). There is an option for exporting data for analysis in Microsoft Excel, which expands the reporting options somewhat.

While not stagnant, OpenSTA doesnt seem to be one of the most active open-source projects weve ever seen. A community site at portal.opensta.org offers forums, documentation and FAQs on using OpenSTA.

Despite some of OpenSTAs limitations, we still recommend that companies take a look at it. With the high price of many commercial performance testing tools, OpenSTA provides a real option for those who want a way to test the performance and scalability of applications and infrastructure but cant get the budget approved for a commercial tool.

Check out eWEEK.coms for the latest news, reviews and analysis in Web services.