Evaluation of Load/Stress Tools for Web Applications Testing

April 24, 2008

With the continued tightening of expenses, IT organizations are forced to do more with less. The emergence of open source software has given IT organizations the opportunity to reduce their development cost without sacrificing quality. Software applications such as Apache web server and Open Office productivity package are examples of open source products used by businesses to reduce the total cost of ownership (TCO) within their organizations.

The high cost of performance test automation software is requiring QA professionals to begin to evaluate open source alternatives. The goal of this article is to explain how to select open source tools, implement a sound performance test strategy, and gain the same benefits as the commercial performance test tools at a fraction of the commercial tools cost.

Although there are several open source performance test tools available, the detials in following document is as a result of Evaluation Study. Document explains the most popular open source performance test tools OpenSTA, TestMaker, JMeter and Grinder. Each of these tools provides the functionality necessary to complete a performance, load or stress test.

Key Parameters Used For The Evaluation

Tool Name: JMeter

Organization: APACHE SOFTWARE FOUNDATION https://jakarta.apache.org/jmeter/

Requirement: Java 1.4 or higher virtual machine on Windows and Unix (Solaris, Linux etc)

Description: Java desktop application from the Apache Software Foundation designed to load test functional behavior and measure performance. Originally designed for testing Web Applications but has since expanded to other test functions; may be used to test performance both on static and dynamic resources (files, Servlets, P-erl scripts, Java Objects, Data Bases and Queries, FTP Servers and more).

Scripting Language: It has support for Java requests and BeanShell scripts. Feature pro/cons:

Scripting Language Feature pro/cons:

Tool Name: TestMaker

Organization: PUSHTOTEST https://www.pushtotest.com /

Requirement: Java 1.4 or higher virtual machine on Windows, Linux, Solaris, and Macintosh.

Description: TestMaker from PushToTest.com delivers a rich environment for building and running intelligent test agents that test Web-enabled applications for scalability, functionality, and performance. It comes with a friendly graphical user environment, an object-oriented scripting language (Jython – simplified version of Python) to build intelligent test agents, an extensible library of protocol handlers (HTTP, HTTPS, SOAP, XML-RPC, SMTP, POP3, IMAP), a new agent wizard featuring an Agent Recorder to write scripts for you, a library of fully-functional sample test agents, and shell scripts to run test agents from the command line and from unit test utilities.

Scripting Language: It uses Jython – an open source object-oriented scripting language with a friendly graphical user environment

Scripting Language Feature pro/cons:

Tool Name: GRINDER

Organization: A DEVELOPER GROUP https://grinder.sourceforge.net /

Requirement: Java 1.4 or higher virtual machine on any Operating System

Description: Grinder is a GUI desktop application designed to load test functional behavior and measure performance. Grinder uses Jython for defining test. The Grinder is a Java load-testing framework making it easy to orchestrate the activities of a test script in many processes across many machines, using a graphical console application. Three types of processes make up The Grinder:

Agent processes: A single agent process runs on each test-client machine and is responsible for managing the worker processes on that machine.
Worker processes:Created by The Grinder agent processes,they are responsiblefor performing the tests.
The console: Coordinates the other processes and collates statistics.

Scripting Language: Latest version, Grinder 3, uses the powerful scripting language Jython, and allows any Java code to be tested without the need to write a plug-in.

Scripting Language: Feature pro/cons:

Tool Name: OpenSTA

Organization: OPEN SYSTEM TESTING ARCHITECTURE https://OpenSTA.org/ https://portal.opensta.org/

Requirement: Java 1.3 or higher virtual machine on Windows NT4 only

Description: A distributed software testing architecture based on CORBA. Using OpenSTA (Open System Testing Architecture) a user can generate realistic heavy loads simulating the activity of hundreds to thousands of virtual users. OpenSTA graphs both virtual user response times and resource utilization information from all Web Servers, Application Servers, Database Servers and Operating Platforms under test, so that precise performance measurements can be gathered during load tests and analysis on these measurements can be performed.

Scripting Language: Script Control Language (SCL), uses a proprietary, “BASIC” like language that has special automation scripting facilities.

Scripting Language: Script Control Language (SCL)

Evaluation of Identified Tools w.r.t Key Parameters vs. Commercial Tools (Load Runner)

Different open source tools were considered for evaluation and based on Key parameters criteria, many tools were omitted from the list. Only most suitable tools were considered for the detailed study. Different sections of sheet include evaluator’s description for the ratings for the tools.

Evaluation of Identified Tools w.r.t Key Parameters vs. Commercial Tools (Load Runner)

Conclusion

Jmeter offers some good features and can perform tests on several systems, but it is a little more difficult to use than the other software. It is possible to realize a wide variety of load tests, even to distribute the scenario. However, the application is weaken by the management of Java objects and memory concerns which affect global performance. The major advantage is the possibility to add new features very simply in the application; it is also a good example of modularity and customization.

TestMaker is a powerful tool since the scenario is created with scripts in Jython which allow to use all the Java features. This framework permits to build a customized scenario and very precise test plan, with specific classes. However, its average on the monitoring part and distributed testing is not possible. But it seems interesting to have a look at the classes structure and hierarchy.

Grinder is similar to TestMaker, except that it allows distributed tests, so it can simulate heavy loads. OpenSTA provides some good performances for HTTP testing, with simple charts and distributed tests. It is a good application for simple and reliable HTTP tests.

To summarize the evaluation study, JMeter and Grinder are of Programmer’s tool category and scores more over OpenSTA. TestMaker is an equivalently good tool for Tester’s use.

We will get back to you!