Tuesday, 2 August, 2016 UTC
Summary
Dear Testdroiders,
Robot Framework is a generic keyword-driven test automation framework for acceptance level testing and acceptance test-driven development (ATDD). Surprisingly, many of you have been using this framework (in the past) for a basic acceptance testing that extend the system level testing capabilities with specifications and test cases associated with the actual app testing. However, if you are not familiar with Robot framework or if you look for a way how to use it in Testdroid Cloud, I’ll walk you through of some basic things in this blog.
What is Robot Framework and How Does It Work
Robot Framework is extremely easy to setup, use and modify to get both Android and iOS apps tested. The test syntax that it uses is based on keywords and these keywords are quick to edit and further configure to make a match with the application under test. Furthermore, testing capabilities provided by Robot Framework can be easily extended with the test libraries which can be implemented using Python, Java and even some other languages are supported.
The nifty thing with Robot Framework is how those mentioned keywords work. Users can quickly create new keywords, either using the existing example ones or by writing everything from the scratch.
If you look for more generic information about Robot Framework, there are plenty of great examples and online documentation in GitHub.
The Basic Setup of Robot Framework
To get started with Robot Framework basically you need Python and pip installed. If you have the environment properly configured you can just execute the installation command lines:
pip install robotframework pip install robotframework-appiumlibrary
After you have installed Robot Framework, you can take a look at the example where we use our Bitbar Sample Application with very basic Robot Framework test examples. For the cloud run you basically need a Python script and shell script that makes sure the test execution on cloud end works fine. For instance, you need to add the pip installation in your shell script to ensure everything gets properly installed on your cloud session as well.
In this example, we’re using Appium as a basis to run the test and make sure everything gets logged for further inspection.
## Cloud setup echo "Starting Appium ..." /opt/appium/appium/bin/appium.js --log-no-colors --log-timestamp --command-timeout 120 >appium.log 2>&1 & # Wait for Appium to fully launch sleep 10 ps -ef|grep appium echo "Extracting tests.zip..." unzip tests.zip echo "Installing pip" curl -O https://bootstrap.pypa.io/get-pip.py python get-pip.py --user echo "Installing requirements" ~/.local/bin/pip install -r ./resources/requirements.txt --target . ## Start test execution echo "Running test" pybot -x TEST-all tests/android_example.robot
This should be all what you need for a shell script. To create a compatible tests you need to add few lines to ensure all paths are recognized and test cases can be found from the right fileset. The file structure for test package can be something as follows:
The following example includes both Android and iOS application in the test package (under resources->app folder). The test definition can be found under ‘tests’ and testing libraries are included under libs->examplelib. In addition, you can easily and quickly tweak and create your own test libraries with the instructions shown on the Github documentation of Robot Framework.
What Data Can Robot Framework Produce
When running a Robot Framework test in Testdroid Cloud users do not need to do any significant tweaks for their test runs (or job configs). The test automation and execution works the same way as with any other framework, and results are quickly available from any of those Android and iOS devices.
If you are starting out test automation with Testdroid Cloud and planning to use Robot Framework, simply create a project (Android or iOS), upload your application (APK or IPA), upload the test package, and select whatever devices you want to use for a test run. Just in few minutes the test run will be done (naturally this depends on the length of your test case) and the following data can be fetched from a test run:
This example includes three basic test cases (we’ll come back to those in the next chapter of this blog). However, after the test run is finalized you have now all the data in one, compact view of Testdroid Cloud. If you click any of those files shown in overview you’ll get full log opened in the view below:
For instance, the shell script file that we introduced in the first chapter of this blog extracted the tests.zip, installed the PIP, setup other tools and finally executed the example tests. The output of the log can be also inspected in this view – and for example it will look something like this:
Imports done ============================================================================== Tests ============================================================================== Tests.Example ============================================================================== Simple Smoke Test - Correct Answer | PASS | ------------------------------------------------------------------------------ Simple Smoke Test - Wrong Answer | PASS | ------------------------------------------------------------------------------ Simple Smoke Test - Wrong Answer 2 | PASS | ------------------------------------------------------------------------------ Tests.Example | PASS | 3 critical tests, 3 passed, 0 failed 3 tests total, 3 passed, 0 failed ============================================================================== Tests | PASS | 3 critical tests, 3 passed, 0 failed 3 tests total, 3 passed, 0 failed ============================================================================== Output: /home/ubuntu/test/output.xml XUnit: /home/ubuntu/test/TEST-all.xml Log: /home/ubuntu/test/log.html Report: /home/ubuntu/test/report.html
In addition, to get comprehensive understanding of how the app (with test) really does on the physical device, you’ll also get performance data about the CPU and memory consumption:
Okay, that’s some fundamentals for creating test runs using Robot Framework and managing the data after tests are done. Let’s take a look of how easy those test cases are to build and what you need to know about Robot Framework to use it efficiently with Testdroid Cloud.
How to Create Robot Framework Compatible Test Cases
First, some generic settings are defined for application and test session.
Resource links to a file where all common procedures are provided in the test package. Test Teardown provides instruction of what system should do when test session is done:*** Settings ***
Resource ...${/}resources${/}common.robot
Test Teardown Close Application
For the variables, creator of a test should define some basic things here. In the Bitbar Sample Application there are name field (editText) and one button (button1). IDs and actions can be defined in this section:
*** Variables ***
${NAME} John Doe
${INPUT_NAME_FIELD} id=com.bitbar.testdroid:id/editText1
${SUBMIT_BUTTON} id=com.bitbar.testdroid:id/button1
The example application looks like this:
Now, let’s look at those test cases (using Android). To create three different test cases for the application, we start with a simple smoke test where correctness of selection is validated. Correct answer in this scenario is when option to use Testdroid Cloud is clicked and name is entered in the text field. Two other test cases indicate the wrong answer with radio buttons (Option to buy 101 devices and ask mom to help you). Test cases are as simple as follows:
*** Test cases ***
Simple Smoke Test - Correct Answer
[Tags] cloud
Set Up And Open Android Application
Input Name ${NAME}
Select Option Use Testdroid Cloud
Submit Selection
Validate Correct Answer
Simple Smoke Test - Wrong Answer
[Tags] 101
Set Up And Open Android Application
Input Name ${NAME}
Select Option Buy 101 devices
Submit Selection
Validate Wrong Answer
Simple Smoke Test - Wrong Answer 2
[Tags] mom
Set Up And Open Android Application
Input Name ${NAME}
Select Option Ask mom for help
Submit Selection
Validate Wrong Answer
When test execution is started, Robot framework first parses the test data. After that it uses keywords provided with the test library/libraries to interact with the system.
In the keyword example, there is an input name, select of options (3 radio buttons) and submit of a selection (=a button press). In addition, keywords section can include validation of correctness. In the example below, selecting the right radio button uses xpath to find a name included in the actual UI element:
*** Keywords ***
Input Name
[Arguments] ${name}
Input Text ${INPUT_NAME_FIELD} ${name}
Hide Keyboard
Select Option
[Arguments] ${option}
Click Element xpath = //*[contains(@text, '${option}')]
Submit Selection
Click Element ${SUBMIT_BUTTON}
Validate Correct Answer
Wait Until Page Contains Congratulations ${NAME} 5s
Validate Wrong Answer
Wait Until Page Contains Wrong Answer 5s
Conclusion and Download Link
Robot Framework is very handy and easy to use. Some of you have been using it in the past so in case you have your tests implemented, and you have your Android / iOS app ready for test, give this framework a try at Testdroid Cloud. You’ll be surprised how smoothly things work and how quickly you can do acceptance testing. Furthermore, here is the full test package available for a download (and tweaks).
Have fun with Robot Framework at Testdroid Cloud!