While UI test automation is well understood for the desktop market, with a plethora of good tools available, the mobile market is different. Due to platform fragmentation and restrictions imposed by mobile device OSs, UI automation is a harder problem to solve. Over time, several tools have evolved that offer different mechanisms for UI automation for mobile devices, each with certain advantages and disadvantages.
The goal of this document is to compare and contrast some of the popular tools against various dimensions to help developers assess suitability of such tools for their need. The document limits itself only to provide information about the features / capabilities available in different tools and does not describe in detail on how to use that particular tool. Further, this document will only focus on Android and iOS when comparing such tools.
Test automation is the use of software to automate and control the setting up of test preconditions, execution of tests, test control and test reporting functions, with minimal user intervention. It is not practical to try to automate everything, especially for mobile devices. However, leveraging Common Off The Shelf (COTS) tools can greatly benefit the automation process.
The following are some of the reasons that make effective testing automation of mobile applications challenging:
However, mobile test automation provides many advantages, including:
Figure 1: Mobile Application Development
Figure 1 describes the typical phases in mobile application testing and feature enhancement. Most mobile applications today are developed in tandem with an Agile software development methodology which involves very frequent interim releases. In such cases, implementing test automation can greatly help reduce iterative time to market. To achieve test automation it is worthwhile to discuss various tools available and the nature of testing they enable that could be aligned to the an Agile software delivery process. This is discussed next.
There are quite a few GUI test automation tools available in the market. We have broadly categorized these tools as below:
Subsequent sections will describe the applicability of some of these tools in more detail.
Sikuli is a visual technology tool to automate testing of graphical user interfaces (GUI) using image recognition. Sikuli scripts can be used to take a snapshot of a GUI element and compare it with some reference snapshot. Since Sikuli uses a visually driven approach (i.e.,drag/drop images to test against), a test script writer can write test cases without any knowledge of the target application code. Sikuli provides an IDE that allows for rapid development of test scripts with application screen shots. It allows automation of anything that is visible on a screen. Sikuli can be used to control a web page, a desktop application, an android application or even an iPhone application running in a Simulator.
An application that satisfies the following criteria can be tested with Sikuli:
Sikuli scripts are Jython scripts and can be extended to use other available Jython modules.
Sikuli uses images patterns to deliver mouse/keyboard events to appropriate GUI elements. Sikuli consists of two main parts
Sikuli runs on Windows/Linux/Mac platforms. To be able to test mobile applications with Sikuli, the mobile phone/tablet screen should be transferred to Desktops where Sikuli is installed. Following are some of the ways to test mobile applications with Sikuli:
One of the possible way to test an iOS application with Sikuli is to run the iOS application in an iPhone/iPad simulator within Xcode. The other possibility is to run a VNC server on the iOS device and run a VNC client on the desktop to get the iPhone/iPad screen on the Desktop. This approach has a disadvantage that it can only be used with a jail broken iOS device because VNC servers are not available for non-jail broken devices.
For testing Android applications there are two possibilities to transfer the mobile phone/tablet screen to the desktop.
The Screen cast application can transfer an Android phone/tablet screen to the desktop. However, it requires the phone to be rooted to be able to pass keyboard/touch events to the phone. Most of the VNC servers also require rooted Android phones. One VNC server, “VNCLite” available at Google Play does not require a rooted phone and works well with Sikuli.
Note: For ICS and later releases, Screen cast application cannot pass type/touch event to phone even for rooted phones.
Below is a sample Sikuli test script. This test script automates the test case to add a news category and RSS news feed for Hughes Systique Pace application. Following is the list of activities performed by the test script
Monkeyrunner tool is part of AOSP (Android Open Source Project) which provides APIs for writing programs that can control an Android device remotely. With MonkeyRunner, we can write python programs that install an Android application, send keystrokes to it, take screenshots of its user interface, and store screenshots on the Test Driver. The screen shots can then be compared with sample baseline screenshots and checked for accuracy.
MonkeyRunner, in general, provides the facilities for functional testing, regression testing as well as ways to extend the tool and write frameworks on top of it. MonkeyRunner provides access to three basic classes MonkeyRunner, MonkeyDevice and MonkeyImage. These classes provide a set of APIs which could be invoked from custom Python programs to create and execute test cases.
The Plug-in architecture of MonkeyRunner also allows users to add their own classes to the API.
It does not need to be installed separately as it is part of Android SDK.
The sample script below demonstrates a simple test case for the following actions:
Note: Please note that com.android. monkeyrunner.easy class of monkeyrunner used in the code snippet above is still (Android Jellybean) under development and may change without notice.
‘Instruments’ is a performance, analysis, and testing tool for dynamically tracing and profiling OS X and iOS code.
Instruments not only provides GUI Test automation tool but it has various other automation tools, which help a developers perform various kind of tests related to memory leaks, graphics, animation, time profiling and other areas as shown below:
It is part of the Xcode developer tool suite and it come as a pre-bundled tool with Xcode 4.0+.
There are however some limitations to the UI Automation instrument. If running against a device, that device must be running iOS 4.0+ and it must have hardware capable of multitasking (this won't work on iPhone 3G or the second generation iPod touch).
Below is a very simple GUI test script targeted for an iPad application. This script will generate random touch events for the application.
One can simulate user interaction with the application interface in a number of ways. If we have buttons or other touch-responding elements, we can simulate a variety of touches and gestures. First, we need to find the correct element. To do that, one can use manual logging and traversal of the UI elements as described in the code fragment above, or one can do a more generic reference based on element type and name.
For example, the following will tap the button named “Equation library”, which is accessible via the root view in the application:
What's happening here is that we're accessing an array containing only the buttons within this view, then finding the one with the name Equation library. Once we have that item, we use the tap() function to simulate a user tapping on the button.
Instruments supports the following events/gestures:.
Froglogic provides wide range of cross-platform GUI test automation tools, which specifically support the automation of tests based on QT. Froglogic has three main products:
Squish runs a small server (squishserver) that handles the communication between the Application Under Test (AUT) and the test script. The “squishserver” starts the AUT and injects the Squish hook into it. The hook is a small library that makes the AUT's lives running objects accessible and that can communicate with the squish server. With the hook in place, the squish server can query AUT objects regarding their state and can execute commands—all on behalf of the squish runner. The following diagram illustrates how the individual Squish tools work together.
Squish supports QT based mobile application development. It also has support for native iOS application testing.
Testing iOS native Apps on an actual iPhone or iPad device is less convenient as compared to simulator testing. To do this one must add a Squish-specific wrapper library to Xcode, make a small modification to the application's main function as written below:
Once the main function is setup correctly for Squish, then the developer needs to add a static library “libsquishioswrapper.a” into the Xcode project. This library is shipped with the Squish package and can be found in the package's lib directory.
Frank is an open source UI testing tool for native iOS apps. Frank uses Cucumber and JSON through which once can write structured test/acceptance tests and have them execute against the iOS application. Frank also includes a powerful “app inspector” (called Symbiote) that one can use to get detailed information on the running application.