Automation Tool Selection Criteria

August 24, 2016

Factors Contributing to the Selection of the Right Automation Tool

Need for Selection of Tools

Today's world of software is all about delivering large number of quality product with fewer resources and limited time. With this paradigm shift, Test cycle needs to repeat numerous times to achieve the following:

  • Improved test effectiveness
  • Improved quality
  • Increased test coverage
  • Reduced time to run repetitive tests
  • Cost Saving

Choosing a right tool can help an Organization in the following ways:

  • Reducing time to automate
  • Increased reusability of code
  • Reduction in maintenance cost
  • Higher ROI

Factors considered for Tool Selection

Compatibility with AUT (Application Under Test)

Test tool selection largely depends on the technology that the AUT is built on. A specific test tool might be pretty popular in its own space but might fail to address the testing needs for certain types of applications. For ex. QTP is a famous test automation tool but it cannot support Informatica so it cannot be used to test applications built on Informatica. Every tool has its own set of pros and cons. Before starting to invest in a particular test tool it is important to do a fine print reading to find out ìWhat a tool canít doî. Generally, itís a common trend for all tools, open source or commercial, to highlight how mighty a certain tool is. What is often hidden under the hood are the things it canít do. Before selecting an automation tool, it is highly advised to understand the technology that the application is built on and the test requirements. For instance, one may want to find out whether the application testing requires only functional or both functional and performance testing. There are some applications that also require Image testing/verification etc. Answers to these questions will help to gauge whether those requirements are fulfilled by a particular test tool. Few common points to ponder upon are:

  • Understand the application technology and automation requirements
  • Study feature set of all the tools in the relevant area, both open source and commercial
  • Set the expectations from the automation tool
  • Choose the tool that can fulfil the most automation needs for the AUT

Platform supported and tested

This requirement is necessitated by the growing number of platforms day by day. Application developers have the inherent implicit requirement to support the application on all possible platforms which thereby imposes the requirement on testing such an application deployments on all supported platforms. Even in one variant of the platform there are various versions that need to be supported. For example, if a desktop application claims to support Windows, then it has to run on Windows 7 (32 bit/64 bit), Windows XP etc., or on Linux, then various flavors of Linux like RHEL, Ubuntu etc. Similarly, a mobile application could be supported on Android, iOS, Windows Mobile etc. For an automation tool this is an implicit requirement that once a test script or test case is written, it should be able to test the application on all supported platforms with a minimal change say in a configuration file. While setting the goal for an automation tool, therefore, it becomes imperative to carry out an extensive study to ensure that it has support for doing cross-platform testing for all required platforms with maximum reuse of the test scripts.

Ease of Use

This factor becomes very important once the tool has been selected and deployed or implemented. Before a tool is finalized, it should be considered how the end users will perceive it. This can be achieved by creating variations, gauging the impact of the tool and learning about its end users. These steps have to be considered before arriving at an automation tool that is targeted to address the needs of the organization and a more generic testing group. For instance, based on the complexity of the tool, the usage stage could either be painful or an appreciative phase. Few questions that should be asked from usage perspective are as follows:

As a starting step it is important to clear these questions. With the passage of time, as the experience and comfort level of teams increases, one can deploy a tool of varying complexities to service any type of test. But while starting out, it is important to achieve a basic threshold of test velocity. This helps to streamline the learning process.


Popularity of the tool directly relates to the chances of finding the relevant testing engineers for maintaining and using the testing tool. Less popular tools will have fewer engineers who would have worked on it earlier and thus new skills have to be acquired to understand and implement the tool. On the other hand, if the testing tool is popular enough there are high chances that several engineers have already worked on it or at least have some exposure to the tool during their experience. For instance, let us consider Selenium vs Sahi for web applications automation domain. While Selenium is more popular and widely used in the industry, Sahi is not so common. Popularity of a tool is directly proportional to the availability of support, technical forums and documentation quality.

Documentation and Support

Support and documentation have been an integral part of any technology product. This factor adds up exponentially if the tool is costly or complex. For commercial tools, there are various packages available under different license umbrellas, which come along with different levels and types of support. In-house trainings, developer services, consultation of course at some additional costs are some of the support activities associated with the above. For organizations that are low on IT or in-house development support, it is always wise to spend on acquiring extensive, hands-on support.

On the other hand, few open source programs also have these training sessions for an extra cost, but more often than not, you are on your own when choosing to go for an open source tool. In such a scenario, it becomes important that you carefully examine the tool support. Following is a list of few key items though not limited to, to consider before choosing an open-source:

  • Is the new tool easy enough to be used even with the person new to testing group, say a trainee or an entry level engineer?
  • Is there in-depth technology knowledge required before starting to use the tool?
  • Is the setup too easy or cryptic?
  • Is the tool intuitive enough?

    License Cost

    An important concern while choosing any product or service is the cost. It is also the case when choosing a testing tool. The important thing to take care on this aspect is to balance between the needs and the budget. It might not be as simple as comparing the posted rates on the website. Every license cost is different in terms of the pricing of their products. Some offer monthly rates while some others offer annual. Some charge by the number of simultaneous users while others by the number of tests run. What fits your budget depends on the targets and situation as much it does on the amount of money to spend. Some tools offer a bare minimum service for a nominal license cost, however, it becomes important to read the fine print and find out the extent of feature set provided. Each upgrade to the next set of features could be a costly affair which may also end up in becoming much more than the initial license cost. Other than the cost of the add-ons, there could be additional price like the support fee, training fee and upgrade fee.

    Open source license is another lucrative option for many organizations; while commercial licenses guarantee better support open source license if admissible for its no license fee weighs out lack of support in many cases. However, while choosing an open source tool, the license agreement associated with the tool needs to be carefully assessed since there are too many open source licenses each with its own caveat. For ex. GPL ensure open source but any changes done to the software have to be returned to the community and hence it can become an IPR issue for many organizations. Generally, before choosing an open source testing tool and making any changes to the base software, it is advisable to carefully examine the use of the tool, whether it will be totally for internal use or needs to be redistributed.

    Test Management and Reporting

    There is a good chance that the organizations are already using a test case management or bug management tool. Itís an obvious need that any new automation testing tool should be integrated with their existing test management tool so that the whole testing cycle can be managed alongside other available tools. This aspect is also worth noting before a new test tool selection is made. For ex. QTP supports QLM, TestComplete support QAComplete.

    Reporting is another critical factor while choosing an automation test tool. One important aspect is that whether the tool allows customized reporting, what form the data can be exported/imported from the tool. If the organizations already have reporting system itís important to check whether same reporting can be derived from the new testing tool. In many tools these options are built in while in others there are ways to make the report more comprehensive. If a tool gives comprehensive report on failure then itís the best for the organization.

    Integration with other Tools

    While this factor is closely linked with the above point, it nevertheless becomes an important consideration to know while making a test tool selection as to whether a testing tool can work with already existing platforms. Some testing tools have ready plugins for common bug reporting and project management tools like Jira, Fogbugz, RedMine etc. It is important to carefully analyze the support of the integration with these tools available in the new automation tool. While in some cases some API implementation may be required but that requires development skills to be integrated. It could be a simple integration with the e-mail service in an organization, or an SMS based reporting in case of failure, but these requirements could be essential for a testing program in an organization so should be carefully considered.

    Steps for Selection of Tool

    Define Requirement

    This helps to define the category of Test Automation Requirements like Test Cases that are to be automated and segregating them as per the relevant category along with the target audience from the usage part

    • Community support - how well is the tool supported in technical blogs, forums
    • Reliability- ability of the tool to give reliable test results time and again. Typically popular open source tools are peer reviewed software, hence more reliable
    • Scalability  ability of the tool to expand as per the future testing needs
    • Security - well supported tools mean more critically reviewed software hence less of security loop holes and defects
    • Return on Investment (ROI) - this factor translates to the test velocity and efforts required of test teams picking up and converting manual to automated test cases
    • Flexibility - this aspect covers if the same test tool can cater to various compatibility testing. For ex. Selenium webdriver cross browser and cross platform support
    • Standards compliance

  • Based on the above, all the requirements of this tool shall be listed down in one place:
      • Essential features: These are the ones, which are extremely necessary to meet requirements
      • Desirable features: There are the ones, which will make a particular tool standout among many of its competitors
      • Irrelevant features: These comprise of the ones, which are not of any great significance & may not be able to provide some tangible benefit to the specified requirements

      Shortlist Available Tools

      Preliminary study shall be carried out in order to shortlist a tool and at this stage one should evaluate around 5-6 tools any one of which could qualify to be the final tool by satisfying ultimate choice.

      As a next step, download an evaluation or trial version of the tool for refining the decision. At this stage this should clearly explain all your requirements of testing along with set of constraints to the tool supplier

      Create DAR (Decision Analysis Report)

      This is a process followed by organizations to define a formal method of evaluating with respect to factors considered for selection

      As an outcome of the above DAR Sheet, one can select 2-3 tools for Proof of concept

      Proof of concept with 2 Tools (if feasible)

      The scope of the POC is defined by identifying the subset of test scenarios and the needs that have to be evaluated for the tool to be considered suitable. Once the scope is decided, POC can be commenced.

      Results of the POC phase are compared and the most fitting tool can be selected.

No Comments

Add Comment


We use cookies (including third party cookies) to ensure you get the best experience while visiting our website. Click "Accept All Cookies" to accept the cookie usage. Click "Cookie Settings" to adjust cookie settings.

Mandatory Cookies

These cookies cannot be disabled

These cookies are necessary for the website to function and cannot be switched off.

  • __RequestVerificationToken
  • authentication
  • dnn_IsMobile
  • language
  • LastPageId
  • NADevGDPRCookieConsent_portal_0
  • userBrowsingCookie

Analytics Cookies

These cookies allow us to monitor traffic to our website so we can improve the performance and content of our site. They help us to know which pages are the most and least popular and see how visitors move around the site. All information these cookies collect is aggregated and therefore anonymous. If you do not allow these cookies we will not know when you have visited or how you navigated around our website.

  • _ga
  • _gat
  • _gid

Functional Cookies

These cookies enable the website to provide enhanced functionality and content. They may be set by the website or by third party providers whose services we have added to our pages. If you do not allow these cookies then some or all of these services may not function properly.

  • __atuvc
  • euconsent

Targeting Cookies

These cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.


Not used.