Functional testing with Watir

So, we have a public web application accessible 24×7. Even if we do not process financial transactions or control airport traffic, this poses a kind of stress upon us. To relieve it, functional testing is our true companion.

By functional testing I mean simulating an end user’s traversal through the application. This means that, contrary to unit tests, we are testing what the user sees, not what the developer has written. The tests are thus less sensitive to software design changes, internal bug fixes, and even URL refactoring.1

As a tool, we have chosen Watir, an open-source library for controlling a web browser. It is light-weight and it does what it says on the box: “drives the … browser the same way people do. It clicks links, fills in forms, presses buttons… also checks results, such as whether expected text appears on the page.”

Technically, Watir is a library in Ruby2 for controlling Microsoft Internet Explorer on Windows (with support for Linux and Firefox planned in the next major release). The tested web application may be anything (ASP, JSP, PHP, etc.) producing browsable web pages.

Writing tests in Watir is fun in itself. Firstly, you can see the page in the browser and watch the elements being manipulated by an invisible user – buttons, links etc. get highlighted when the test suite accesses them. Secondly, you can even use Ruby console to write the tests interactively so that you command the browser like a starship. 

def fBU1_log_in( username = "Adam", password = "Adam" )
    assert(@ie.pageContainsText("Login"))
    @ie.text_field(:id, "user_username").set(username)
    @ie.text_field(:id, "user_password").set(password)
# Clear the security warnings for HTTPS
    startClearers 
    @ie.button(:value, "Log in").click
# Show English version for test even if the user has 
# another language preference
    if not @ie.pageContainsText("Welcome ") 
        @ie.image(:alt, "English").click 
    end
    assert(@ie.pageContainsText("Welcome "+username))
end

Still, as we soon found out, there needs some home work to be done in order to achieve representative results of the tests:

  1. define test scenarios
  2. prepare test data

Test scenarios are needed to cover the whole functionality landscape, and to agree with the developers, what the application should do. In theory, this should completely be done at the beginning of the application lifecycle, but in an agile approach (as we had it with Recykl.com), you just define the main use cases at the beginning and fine-tune the details based on end user/customer feedback. Anyway, having the scenarios automated in tests is a good validation of the functional design.

Test data for the functional tests are indispensable for the predictability of the test results. In our case, another important requirement was isolation of the data for the individual tests, as we decided to use just one suite of test data for the whole suite of functional tests. For faster test execution the data is loaded once, not at the beginning of each test in the suite (which is a common Ruby test fixtures practice). To achieve isolation, we successively run the tests under different users, so the risk of interfering is reduced to a minimum.

Now, after some time of using Watir for the testing, there are some lessons learned:

  1. the most painful and unpredictable errors result from timing, e.g. in asynchronous AJAX calls – sometimes it’s difficult even for a human to tell if we are still waiting for something to happen or that’s all
  2. use some kind of automated reporting of results (unfortunately Watir currently has a very limited support here)
  3. tests are not carved in stone – you will have to update them as your functional design evolves (and it does in a successful application) – so write them flexibly. For example, do not rely on details like exact wording of a link, but instead use a regular expression matching just the main keyword, e.g. /[Ss]ubmit/
  4. even though it takes long to run the tests, try to run them after each major changeset – you’ll catch bugs and also easier keep up with the intentional changes

Having that in mind, doing functional testing with Watir has significantly increased our confidence in the functioning of our web application and saved us some hot moments in its (sometimes stormy) agile lifecycle.

Note 1: Although this looks like THE testing, other kinds of testing still have their important role: unit testing to make sure the application pieces are runnable after developers’ changes, performance testing to see the behavior of the application under heavier load than just one user, security testing and reviews to identify the possible weak spots exposed to the wild internet.

Note 2: Both Watir and Ruby are open-source products. Ruby is an object-oriented scripting language somewhere between Perl and Smalltalk. It is has garbage collection like C# or Java, but installs from a 5MB Linux RPM or 27MB Windows installer and starts up in an instant. Watir is an 800kB little gem ready to start after download.