Factors to be Considered for Tool Selection
Compatibility with AUT (Application Under Test)
Test tool selection largely depends on the technology that the AUT is built on. A specific test tool might be pretty popular in its own space but might fail to address the testing needs for certain types of applications.
For example, QTP is a famous test automation tool but it cannot support Informatica. So, it cannot be used to test applications built on Informatica. Every tool has its own set of pros and cons. Before starting to invest in a particular test tool, it is crucial to do a fine print reading to find out What a tool can’t do.
Generally, it is a common trend for all tools, open source or commercial, to highlight how mighty a particular tool is. What is often hidden under the hood are the things it cannot do. Before selecting an automation tool, you must understand the technology that the application is built on and the test requirements.
For instance, you may want to determine whether the application testing requires only functional or both functional and performance testing. Some applications also require Image testing/verification etc. Answers to these questions will help to gauge whether those requirements are fulfilled by a particular test tool. Some common points to ponder upon are:
- Understand the application technology and automation requirements
- Study the features set of all the tools in the relevant area, both open source and commercial
- Set the expectations from the automation tool
- Choose the tool that can fulfil the most automation needs for the AUT
Platform supported and tested
This requirement is necessitated by the growing number of platforms day by day. Application developers have the inherent, implicit need to support the application on all possible platforms, which thereby imposes the requirement to test such application deployments on all supported platforms.
Even in one variant of the platform, various versions need to be supported. For example, if a desktop application claims to support Windows, it has to run on Windows 7 (32-bit/64-bit), Windows XP, etc., or on Linux, then various flavors of Linux like RHEL or Ubuntu. Similarly, a mobile application could be supported on Android, iOS, Windows Mobile etc. For an automation tool this is an implicit requirement that once a test script or test case is written, it should be able to test the application on all supported platforms with a minimal change, say in a configuration file. While setting the goal for an automation tool, therefore, it becomes imperative to carry out an extensive study to ensure that it has support for doing cross-platform testing for all required platforms with maximum reuse of the test scripts.
Ease of Use
This factor becomes very important once the tool is selected, deployed, or implemented. Before a tool is finalized, you must consider how the end users will perceive it. You can achieve this by creating variations, gauging the impact of the tool, and learning about its end users. Consider these steps before arriving at an automation tool that addresses the needs of the organization and a more generic testing group.
For instance, based on the complexity of the tool, the usage stage could either be a painful or an appreciative phase. A few questions you should ask from a usage perspective are as follows:
- Is the new tool easy enough to be used even with the person new to the testing group, say a trainee or an entry-level engineer?
- Is there in-depth technology knowledge required before starting to use the tool?
- Is the setup too easy or cryptic?
- Is the tool intuitive enough?
As a starting step, it is essential to clear these questions. With the passage of time, as the experience and comfort level of teams increase, one can deploy a tool of varying complexities to service any type of test. But while starting, achieving a basic threshold of test velocity is essential. This will help you streamline the learning process.
Popularity
The popularity of the tool directly relates to the chances of finding the relevant testing engineers for maintaining and using the testing tool. Less popular tools will have fewer engineers who would have worked on it earlier. Thus, new skills have to be acquired to understand and implement the tool.
On the other hand, if the testing tool is popular enough there are high chances that several engineers have already worked on it or at least have some exposure to the tool during their experience.
For instance, let us consider Selenium vs. Sahi for the web applications automation domain. While Selenium is more popular and widely used in the industry, Sahi is not so common. The popularity of a tool is directly proportional to the availability of support, technical forums, and documentation quality.
Documentation and Support
Support and documentation have been an integral part of any technology product. This factor adds up exponentially if the tool is costly or complex.
For commercial tools, various packages are available under different license umbrellas, which come with different levels and types of support. In-house training, developer services and consultation at some additional costs are some of the support activities associated with the above. It is always wise to acquire extensive, hands-on support for organizations that are low on IT or in-house development support.
On the other hand, few open-source programs also have these training sessions for an extra cost, but more often than not, you are on your own when choosing to go for an open-source tool. In such a scenario, it becomes important that you carefully examine the tool support.
Following is a list of a few key items, though not limited to, to consider before choosing an open-source tool:
- Community support – How well is the tool supported in technical blogs and forums.
- Reliability- The ability of the tool to give reliable test results time and again. Typically, the popular open-source tools are peer-reviewed software; hence more reliable.
- Scalability – The ability of the tool to expand as per the future testing needs.
- Security – Well-supported tools mean more critically reviewed software; hence less security loopholes and defects.
- Return on Investment (ROI) – This factor translates to the test velocity and efforts required of test teams picking up and converting manual to automated test cases.
- Flexibility – This aspect covers whether the same test tool can cater to various compatibility testing. For e.g., Selenium WebDriver cross-browser and cross-platform support.
- Compliance to standards
License Cost
A major concern while choosing any product or service is the cost. It is also the case when choosing a testing tool. The critical thing to take care of in this aspect is to strike a balance between the needs and the budget.
It might not be as simple as comparing the posted rates on the website. Every license cost is different in terms of the pricing of their products. Some offer monthly rates while others offer annual. Some charge by the number of simultaneous users, while others by the number of tests run. What fits your budget depends on the targets and situation as much as it does on the amount of money to spend. Some tools offer a bare minimum service for a nominal license cost.
However, it becomes essential to read the fine print and find out the extent of the feature set provided. Each upgrade to the next set of features could be a costly affair, which may end up becoming much more than the initial license cost. Other than the cost of the add-ons, there could be additional prices like the support fee, training fee, and upgrade fee.
Open source license is another lucrative option for many organizations. Although commercial licenses guarantee better support than open source licenses, its no license fee weighs out the lack of support in many cases. However, when choosing an open-source tool, the license agreement associated with the tool must be carefully assessed since there are too many open-source licenses, each with its own caveat.
For example. GPL ensures open source, but any changes done to the software have to be returned to the community, and hence it can become an IPR issue for many organizations. Generally, before choosing an open-source testing tool and making any changes to the base software, you should carefully examine the use of the tool, whether it will be totally for internal use or needs to be redistributed.
Test Management and Reporting
There is a good chance that you are already using a test case management or bug management tool. It is an obvious need that any new automation testing tool should be integrated with their existing test management tool so that the whole testing cycle can be managed alongside other available tools.
This aspect is also worth noting before you select a new test automation tool. For example, QTP supports QLM, TestComplete supports QAComplete etc.
Reporting is another critical factor when choosing an automation test tool. One crucial aspect is whether the tool allows customized reporting and what form the data can be exported/imported from the tool. If the organizations already have a reporting system, it is essential to check whether the same reporting can be derived from the new testing tool. In many tools, these options are built-in, while in others, there are ways to make the report more comprehensive. If a tool gives a comprehensive report on failure, it is the best for the organization.
Integration with Other Tools
While this factor is closely linked with the above point, it nevertheless becomes an important consideration to know while selecting a test automation tool whether a testing tool can work with already existing platforms.
Some testing tools have ready plugins for standard bug reporting and project management tools like Jira, FogBugz, RedMine, etc. It is important to carefully analyze the integration support with these tools available in the new automation tool.
While in some cases, some API implementation may be required, which requires development skills to be integrated. It could be a simple integration with the e-mail service in an organization or an SMS-based reporting in case of failure. However, these requirements could be essential for a testing program in an organization, so you must consider them carefully.
Product Engineering Services Customized software development services for diverse domains
