Selenium Remote Control
Selenium RC comes in two parts.
- A server which automatically launches and kills browsers, and acts as a HTTP proxy for web requests from them.
- Client libraries for your favorite computer language.
TestNG is a testing framework inspired from JUnit and NUnit but introducing some new functionality that make it more powerful and easier to use, such as:
- JDK 5 Annotations (JDK 1.4 is also supported with JavaDoc annotations).
- Flexible test configuration.
- Support for data-driven testing (with @DataProvider).
- Support for parameters.
- Allows distribution of tests on slave machines.
- Powerful execution model (no more TestSuite).
- Supported by a variety of tools and plug-ins (Eclipse, IDEA, Maven, etc...).
- Embeds BeanShell for further flexibility.
- Default JDK functions for runtime and logging (no dependencies).
- Dependent methods for application server testing.
TestNG is designed to cover all categories of tests: unit, functional, end-to-end, integration, etc...
Why use TestNG with Selenium RC ?
Selenium IDE by itself is great for creating test cases and executing single tests. TestRunner is ok at running a test suite. But if you want full automation against different server and client platforms, you need a way to invoke the tests from a command line process, flexibility in how you create your test suites, and reports that tell you what happened. TestNG provides these features. You can use Selenium RC with other languages as well, but since Sakai is written in Java it makes sense to stick with the language most of us are familiar with already.
Converting Selenium Tests to the TestNG format
To Convert an html test to your preferred format:
- Add the format to Selenium
- From the IDE menu; Options > Options > Formats > Add
- Type the name of the format (e.g. SakaiTestNG) and then paste the formatting code into the textarea and select OK.
- The format contains certain properties that will need to be set. These include Testname, Description, dependsOn and groups. These properties tell TestNG when to use each test and whether or not the test has any dependencies (i.e. other test that must be run first)
- Close and reopen Selenium IDE
- Selenium IDE will not recognize the properties you've set until it is restarted
- Load your html Selenium Test case
- Export the test into the TestNG format you created in step 1
You can find more information on creating a custom format here: http://wiki.openqa.org/display/SIDE/Adding+Custom+Format
You may want to control the groups or dependencies for tests manually depending on your needs. You can pass this data in from the Selenium IDE, but you may find it easier to manually maintain this in the java file directly. See the annotations in following for an example.
The big issue we ran into was with uploading files. Selenium IDE wants a local file path for the file, this obviously becomes an issue when you try to run the tests from different clients. The Java client API has a method called attachFile() which lets you point to a url for the resource. What happens when the test runs is the file is downloaded to a temp directory on the client and then uploaded to the browser from there. This is great because you don't have to put the resources on all the clients or figure out how to right a single path that will work on windows and linux.
The location of the fileServer is parametrized and gets set at run time, see the sample testng.xml file below.
Tests that execute use cases involving the FCKEditor will need to turn the FCK Editor off. You can do this by installing the Firefox plugin Adblock Plus.
Modify the project.properties file to set the location of your Firefox profile, see http://support.mozilla.com/en-US/kb/Profiles for more info on that.
The default user_extension__file location should work without modification.
Using a custom user-extension.js file
Creating test suites (Using the TestNG xml file)
The great part about TestNG is you can configure what tests you run, in what order, and set the parameters all from a simple xml file. This allows for quite a lot of flexibility.
In the example below, notice the parameters. We are passing in the location of our Sakai instance, where selenium RC is running on what port, what browser we want to invoke, a default timeout, and the location of files we will upload to the Resource tool.
That is followed by a series of tests run in order. We populate the Sakai instance with some data; things like users, sites, and resources. Then we run all tests in a certain package with a group named "Sakai-Reg-Suite". Selenium runs this is any order it chooses, and that is perfectly ok because they are only dependent on the Create User and Create Sites test. Then at the end we tear the whole thing down. That way we can run the test again from a different client (like internet explorer) and we don't have to clean the Sakai instance.
For purposes of reporting, it is essential that all of the tests names <test name="rSmart CLE Regression Test: Delete Users"> be unique. Within a suite you will get a report back with each test highlighted. If you have duplicate names the report will only show the last test with that name that was run.
You can create a series of these tests pointing to different Sakai instances and fire up tests at will. Another handy features I've discovered is the ability to include other testng files. This is helpful because you can maintain different tests that populate, tear down, or do both, and run them separately as needed. Its a great way for developers to seed Sakai data quickly and consistently.
Are We Ready Yet? Checklist
Export ~ Export tests into TestNG format
Addons ~ Install AdBlock Plus and set up a filter for fckeditor.js
Firefox Profile ~ Add the path to your FF Profile to the project.properties file
Create Test Suite ~ Decide what tests are being run
Launching Remote Control
Before you can run a test suite. You need to launch remote control. Basically this process acts as a server proxy which listens for commands, executes the browsers and invokes the commands in the browser. The only requirement is the selenium server jar (attached here).
Once it launches you'll get some output like this:
Running a test suite
OK, now on to the fun part running some tests. Once RC is up and you have your tests and a TestNG xml file, you launch a separate process to run a test suite. TestNG is a java framework, we invoke it from an ant script. Here's a snippet of what we do:
To run this ant task you simply pass in the location of a TestNG xml file like this:
This will fire up the browser, run through the tests, and give you a few nicely formatted HTML reports with the results. The output shows up in the target/reports folder. A full example of what a build.xml might look like can be found here.
Sakai Selenium Tests in Contrib
We are starting to collect Selenium tests in SVN. Anyone can take these and convert them using the process described above into the TestNG format, or any other format.
The TestNG project can be found here. This provides all the test cases currently available and the skeleton to compile and run selenium RC and TestNG. It also has plenty of examples of what TestNG xml files look like, so that can jump start your own project.