Product Engineering Services Customized software development services for diverse domains
Quality Assurance End-to-end quality assurance and testing services
Managed Services Achieve scalability, operational efficiency and business continuity
Technology Consulting & Architecture Leverage the extensive knowledge of our Domain Experts
August 24, 2016
Today’s world of software is all about delivering a large number of quality products with fewer resources and limited time. With this paradigm shift, the Test cycle needs to repeat numerous times to achieve the following:
Choosing the right tool can help an Organization in the following ways:
Test tool selection largely depends on the technology that the AUT is built on. A specific test tool might be pretty popular in its own space but might fail to address the testing needs for certain types of applications.
For example, QTP is a famous test automation tool but it cannot support Informatica. So, it cannot be used to test applications built on Informatica. Every tool has its own set of pros and cons. Before starting to invest in a particular test tool, it is crucial to do a fine print reading to find out What a tool can’t do.
Generally, it is a common trend for all tools, open source or commercial, to highlight how mighty a particular tool is. What is often hidden under the hood are the things it cannot do. Before selecting an automation tool, you must understand the technology that the application is built on and the test requirements.
For instance, you may want to determine whether the application testing requires only functional or both functional and performance testing. Some applications also require Image testing/verification etc. Answers to these questions will help to gauge whether those requirements are fulfilled by a particular test tool. Some common points to ponder upon are:
This requirement is necessitated by the growing number of platforms day by day. Application developers have the inherent, implicit need to support the application on all possible platforms, which thereby imposes the requirement to test such application deployments on all supported platforms.
Even in one variant of the platform, various versions need to be supported. For example, if a desktop application claims to support Windows, it has to run on Windows 7 (32-bit/64-bit), Windows XP, etc., or on Linux, then various flavors of Linux like RHEL or Ubuntu. Similarly, a mobile application could be supported on Android, iOS, Windows Mobile etc. For an automation tool this is an implicit requirement that once a test script or test case is written, it should be able to test the application on all supported platforms with a minimal change, say in a configuration file. While setting the goal for an automation tool, therefore, it becomes imperative to carry out an extensive study to ensure that it has support for doing cross-platform testing for all required platforms with maximum reuse of the test scripts.
This factor becomes very important once the tool is selected, deployed, or implemented. Before a tool is finalized, you must consider how the end users will perceive it. You can achieve this by creating variations, gauging the impact of the tool, and learning about its end users. Consider these steps before arriving at an automation tool that addresses the needs of the organization and a more generic testing group.
For instance, based on the complexity of the tool, the usage stage could either be a painful or an appreciative phase. A few questions you should ask from a usage perspective are as follows:
As a starting step, it is essential to clear these questions. With the passage of time, as the experience and comfort level of teams increase, one can deploy a tool of varying complexities to service any type of test. But while starting, achieving a basic threshold of test velocity is essential. This will help you streamline the learning process.
The popularity of the tool directly relates to the chances of finding the relevant testing engineers for maintaining and using the testing tool. Less popular tools will have fewer engineers who would have worked on it earlier. Thus, new skills have to be acquired to understand and implement the tool.
On the other hand, if the testing tool is popular enough there are high chances that several engineers have already worked on it or at least have some exposure to the tool during their experience.
For instance, let us consider Selenium vs. Sahi for the web applications automation domain. While Selenium is more popular and widely used in the industry, Sahi is not so common. The popularity of a tool is directly proportional to the availability of support, technical forums, and documentation quality.
Support and documentation have been an integral part of any technology product. This factor adds up exponentially if the tool is costly or complex.
For commercial tools, various packages are available under different license umbrellas, which come with different levels and types of support. In-house training, developer services and consultation at some additional costs are some of the support activities associated with the above. It is always wise to acquire extensive, hands-on support for organizations that are low on IT or in-house development support.
On the other hand, few open-source programs also have these training sessions for an extra cost, but more often than not, you are on your own when choosing to go for an open-source tool. In such a scenario, it becomes important that you carefully examine the tool support.
Following is a list of a few key items, though not limited to, to consider before choosing an open-source tool:
A major concern while choosing any product or service is the cost. It is also the case when choosing a testing tool. The critical thing to take care of in this aspect is to strike a balance between the needs and the budget.
It might not be as simple as comparing the posted rates on the website. Every license cost is different in terms of the pricing of their products. Some offer monthly rates while others offer annual. Some charge by the number of simultaneous users, while others by the number of tests run. What fits your budget depends on the targets and situation as much as it does on the amount of money to spend. Some tools offer a bare minimum service for a nominal license cost.
However, it becomes essential to read the fine print and find out the extent of the feature set provided. Each upgrade to the next set of features could be a costly affair, which may end up becoming much more than the initial license cost. Other than the cost of the add-ons, there could be additional prices like the support fee, training fee, and upgrade fee.
Open source license is another lucrative option for many organizations. Although commercial licenses guarantee better support than open source licenses, its no license fee weighs out the lack of support in many cases. However, when choosing an open-source tool, the license agreement associated with the tool must be carefully assessed since there are too many open-source licenses, each with its own caveat.
For example. GPL ensures open source, but any changes done to the software have to be returned to the community, and hence it can become an IPR issue for many organizations. Generally, before choosing an open-source testing tool and making any changes to the base software, you should carefully examine the use of the tool, whether it will be totally for internal use or needs to be redistributed.
There is a good chance that you are already using a test case management or bug management tool. It is an obvious need that any new automation testing tool should be integrated with their existing test management tool so that the whole testing cycle can be managed alongside other available tools.
This aspect is also worth noting before you select a new test automation tool. For example, QTP supports QLM, TestComplete supports QAComplete etc.
Reporting is another critical factor when choosing an automation test tool. One crucial aspect is whether the tool allows customized reporting and what form the data can be exported/imported from the tool. If the organizations already have a reporting system, it is essential to check whether the same reporting can be derived from the new testing tool. In many tools, these options are built-in, while in others, there are ways to make the report more comprehensive. If a tool gives a comprehensive report on failure, it is the best for the organization.
While this factor is closely linked with the above point, it nevertheless becomes an important consideration to know while selecting a test automation tool whether a testing tool can work with already existing platforms.
Some testing tools have ready plugins for standard bug reporting and project management tools like Jira, FogBugz, RedMine, etc. It is important to carefully analyze the integration support with these tools available in the new automation tool.
While in some cases, some API implementation may be required, which requires development skills to be integrated. It could be a simple integration with the e-mail service in an organization or an SMS-based reporting in case of failure. However, these requirements could be essential for a testing program in an organization, so you must consider them carefully.
This helps to define the category of Test Automation Requirements like Test Cases that are to be automated and segregate them as per the relevant category along with the target audience from the usage part
Based on the above, all the requirements of this tool shall be listed in one place:
A preliminary study shall be carried out to shortlist a tool. At this stage, you should evaluate around 5-6 tools, any one of which could qualify to be the final tool by satisfying the ultimate choice.
As a next step, download an evaluation or trial version of the tool for refining the decision. At this stage, this should clearly explain all your requirements for testing along with a set of constraints to the tool provider.
Organizations follow this process to define a formal method of evaluating with respect to factors considered for selection
As an outcome of the above DAR Sheet, one can select 2-3 tools for Proof of concept.
The scope of the POC is defined by identifying the subset of test scenarios and the needs that must be evaluated for the tool to be considered suitable. Once the scope is decided, POC can be commenced.
Results of the POC phase are compared, and the most fitting tool can be selected.
Selecting the right automation testing tool is essential for your organization. However, before doing so, you need to be clear about the testing tool selection criteria. These points will help you reduce time, save costs, and increase operational efficiency.
If you want to enhance your test automation process, please contact us.
