|How to do host based… on How to do host based testing p…|
|kyranto on Welcome to the MeeGo QA-tools…|
QA-tools development and whatnot
Testplanner is a simple UI tool for creating and maintaining test plans and cases. It is being developed for MeeGo testing but is publicly available for all who wish to use MeeGo QA tools. This article presents the features and usage of the tool that can be considered as the first link in the MeeGo test management toolchain.
Before moving into the tool itself it’s good to understand the underlying concept of MeeGo test plans. Test plans in MeeGo are, in short, XML files that describe the test cases in certain hierarchy and with certain attributes. The basic principle of these XMLs, and the whole MeeGo test tool chain, is that you can use any executable for testing. The verdict of a test case will be determined from the exit code of the executable or, in case of manual testing, from the user input.
A test plan has a hierarchical structure consisting of suite, set, and case. Test cases are naturally the main thing, whereas suite and set are used for grouping the cases to make your life easier. Test cases consist of steps which usually hold commands that are executed on the target device. Test cases also have different attributes, some of which are for informational purposes and some have effect on the test execution. An example of the latter type is the attribute manual that defines the test case as a manual one. You can read more about test plans from this excellent wiki article.
Most of MeeGo QA tools, including Testplanner, are available for Ubuntu variants, Fedora 13, and of course MeeGo from corresponding MeeGo repositories. Follow the instructions to set up repositories and use your package manager to install testplanner.
Test case planning is done on one screen. The screen is divided into two parts: on the left side is a tree view of suite/set/case hierarchy, and on the right side an editor for values. As suites, sets, and cases have partially different properties the editors also are different. Adding and removing test suites, sets, and cases can be done via context menu on the tree view or using the buttons provided in the bottom of the editors.
The highest level of a test plan is test definition (for XML people: I’m talking about the document root here). You can give a description for the whole plan.
To add a test suite, either click the corresponding button or select the item from the tree view context menu.
Select the newly added suite from the tree view and the test suite editor opens up. You can again set a description but also define some attributes. These attributes are inherited by set and case unless not overridden. This is an important feature and you should take this into consideration when deciding how to group the cases! To change a certain attribute for a bunch of test cases from suite level is definitely faster than doing the same for each test case separately.
To add the first test set to your test suite, again click either the button or the corresponding context menu item from the tree view.
The general settings of a test set are similar to those of test suite. If you set some of the general attributes on suite level you will notice that those values are visible on set level as well, and that they can be overridden. In the screen capture below this is the situation with attribute type.
Test set editor however contains two additional tabs that hold test set specific things – Pre and Post Steps and Other. In pre and post steps you can define test steps, usually commands, that are executed before/after test case execution. These are handy for setting up the environment and cleaning it up afterwards.
On the tab Other you can define files to fetch from device after test execution and the environments in which these tests are valid.
Test cases are added to sets the same way sets to suites etc. The test case general editor is again similar to suite and set editors. Test case has one additional tab for defining the test steps.
If the test case’s manual attribute is set to true, the case may contain both manual and automatic steps. Test steps are what testrunner-lite finally executes. For automatic steps these are commands that are executed in the device under test whereas a manual step usually describes in writing what the test engineer is expected to do upon execution.
The expected result of a step is currently valid for automatic steps only. In regular testing this is usually 0 (zero) since binaries return a zero if nothing went wrong. You may also be writing negative tests and want to set this to something else than zero when the step really should fail. So for example if you want to test that command ls works (or that some folder exists or does not exist) you could create a step ls /proc/ with expected result 0, and another step ls /blob/ with expected result 2. Your test case would then pass when /proc/ exists and /blob/ does not.
Undo/Redo functionality is available in Testplanner. It works as you’d expect, that is, either from the Edit menu or using the common keyboard shortcuts.
Drag-and-drop is enabled in the tree view. You can reorder suites, sets, and cases by dragging or move sets under different suites, or cases under different sets, by dragging them.
Once you’ve created your test plan, you will most likely want to save it for further use. Testplanner, when installed from the repository, drags with it the test-definition package that contains a schema for the test XMLs. Upon saving Testplanner will validate the file created against the schema to ensure that it can be finally run with testrunner.
Tests can be executed with testrunner which can be launched directly from Testplanner either by clicking the Play-icon from the toolbar or from the Tools-menu. I will add a separate article on test execution using testrunner, and you can also check the previous post on host based testing.
Notice also that this tool uses the test plan format common for all MeeGo QA Tools, so you can use the same plans with the yet-to-be-released Open Test Service.
One new, awesome feature of Testplanner is the git integration. This means that test designers and test managers can keep their test plans in git version control repository and manage them directly with Testplanner. Git functions can be found from Tools / Git menu.
MeeGo QA team offers a bunch of tests that you can use. These include MeeGo Core Test Suite, Handset-UX tests, and MeeGo Netbook UX Test Suite. If you need to edit the tests (and if you wish to some day merge your changes to the main asset) you need to create a personal clone on gitorious.
After you have cloned the project Testplanner will open up a dialog for you to choose a file to open. When you open a file from a git clone Testplanner will enable also the other Git options: Pull (for updating your clone), Push (for pushing your changes to the server), and Commit (for locally committing your changes) – check the video from Youtube about these features. The editing is otherwise the same as usual.
So there you go – creating and maintaining test plans made easy and available for everyone for no charge. There’s of course still the itsy bitsy thing for you, writing the actual binaries that test the features, but that’s another story.
For further reference check out the demo video on Youtube, and/or install the tool and try it out yourself!