Posts by Leo Arias:

Selenium Grid and the parallel smoke test suite

July 30th, 2010

At the QA team we have been playing for a while with Selenium Grid, a great tool to run Selenium tests in parallel.

The first task was to reduce the execution time of the Smoke Test Suite having more than one browser running the tests. This is a complex suite with a lot of dependencies, so there's a limit on the parallelization that is possible to achieve. Pablo Luján prepared a graph to explain this a little better:



Even though we have reduced the execution time of 114 tests from 160 minutes to 90 minutes. And we are working on some options to reduce this even more.

Now that the integration of our testing code has been thoroughly tested we are ready to promote the changes from the experimental branch to the stable one. But this change will require some adjustments on the jobs and scripts.

We have new components: a Selenium Hub and one or more Grid Remote Controllers. The Remote Controllers register themselves with the Hub and wait for requests. The Hub will send tests to idle RCs or queue them if none is available. And the RCs will execute the actions on his browser.

In order to execute a suite on the Grid, the first step will be to create a Firefox profile for each Remote Controller. Otherwise, all the browsers in a machine will share the Openbravo session, a bad idea because the tests involve starting and closing sessions many times. The following command will create a new profile named rc1:

firefox -no-remote -CreateProfile rc1

Then, the Selenium Hub has to be started:

ant seleniumhub.start < /dev/null &

After that, start the Remote Controllers with an instruction like:

ant -Dport=5555 -DseleniumArgs="-firefoxProfileTemplate /srv/hudson/.mozilla/firefox/g2jc2ulk.rc1"

And finally, run the test suite and enjoy your successful tests:

ant test.integration.smoke -f src-test/com/openbravo/test/integration/erp/testsuites/smoke/build.xml.

More details can be found at the wiki. And we have documented there some options to execute tests sequentially as before.

The second task with the grid will be to use it with independent tests that can be fully parallelized. So if we have 100 tests on a suite and 100 Remote Controllers, the whole suite will be executed... really fast :) More about this tomorrow...

Comments Off

Selenium automation: improvements in test package hirarchies and naming conventions

April 20th, 2010

Hello dear Internet. My name is Leo Arias and this is my first post for the Openbravo Planet.

I work with the QA team helping the rest of developers to build high-quality software.

I spend most of the time maintaining the repository for automated testing. Pablo Lujan, also part of the QA team, recently presented a webinar with more details of the things we have done; you can find the recorded session here.

Recently we have been polishing the automation process in order to make it easier to code new integration tests for the ERP. This post will briefly explain the last changes in the packages hierarchy and the convention we are using to name tests. If you are interested in the automation of your Openbravo installations or the modules you develop this might be useful.

The package hierarchies

We like packages a lot and we are not afraid to use them :)

The main packages used in the automation of an ERP test are GUI, testscripts and testsuites. In order to create a new test case it's highly probable that you will have to touch only classes in this packages, because the rest are maintained by the QA team.

• GUI (com.openbravo.test.integration.erp.gui): this package contains the definitions of ERP GUI elements (textfields, checkboxes, lables, etc.) and functions that perform actions with those elements receiving parameters from test scripts (create a new record, edit a record, process and order, etc.).
  The classes in this package inherit from the Window or PopUp classes, where common functions and attributes were previously defined; so here you add only the things that are specific to the screen you are automating.
  And they are stored following the Openbravo ERP Menu hierarchy, so we have sub-packages for the screens that are accessed through the General Setup menu item, the Master Data Management menu item, and all the rest.
• testscripts (com.openbravo.test.integration.erp.testscripts): this type of classes specify the actions that have to be done on one or more windows in order to complete a test. This includes the steps, the verifications and the assertions that will tell if a test succeeded or failed.
  So basically this classes are a sequence of instantiations of GUI windows and calling of functions of those windows passing variables as parameters.
  And the hierarchy here also follows the Openbravo ERP Menu hierarchy. But, as there may be test scripts that perform actions on more than one menu item, they should be stored under the package named after the menu item where the most important action is executed.
• testsuites (com.openbravo.test.integration.erp.testsuites): On this package we store test suites and the scenarios with actual parameters. Test suites are just classes that group scenarios. And scenarios are classes with the values that will be passed as parameters to testscripts in order to perform an actual test.
  This package follows the hierarchy of testlink because before automating a test it should be documented there.

In addition to that, we have modules package (com.openbravo.test.integration.erp.modules), where tests for Openbravo modules are stored.
So, if you are automating tests for Initial Data Load module for example, you should store them in com.openbravo.test.integration.erp.modules.initialdataload. And that package will have inside the same three packages previously explained, but for the gui elements, scripts and suites that are specific to the module.

Everything related to the packages of the repository is fully explained in the wiki.

Naming convention for test identifiers

We used to have a lot of problems with the identifiers of tests.
We wanted them to be unique and to follow the order of the suite they belong to. Problems came with the frequent task of adding new tests to a suite, because we used to number them with adjacent integers. So, if we had tests 1 and 2, and we required to add a new one between them there wasn't a straightforward way to keep the numbering sequence.

The naming convention we are using now starts by defining a three letter identifier for the suite. For example, we have ADM for Administration master data suite.
Then, the tests that form part of that suite will be identified by the three letters of the suite plus four digits. But the four digits will be a sequence of numbers from ten to ten. Thus, the first test of a suite would be XXX0010 and the second by XXX0020.

This way it would be easy to add a test in the middle of those, identified by XXX0015. And every time that a test suites gets to a stable phase, the tests will be renumbered to start again with a distance of 10.

This is explained with more detail in the wiki.

I'll continue posting things related to QA @ openbravo, tagging them with the Openbravo taxonomy term. Your comments and suggestions are appreciated, as always.

Comments Off