Switching from Selenium 1.x to WebDriver/Selenium 2 and HtmlUnit

Recently I became aware of the work that is being done to merge the WebDriver and Selenium codebases. The result is Selenium 2, a project that aims to offer the best bits from both. For more details about the merge, see here.

Having only used Selenium and not WebDriver before, there were a couple of things about WebDriver that piqued my interest. The first is the very clean object-based API that it provides, allowing you to make calls such as:

browser.FindElement(By.Id("q")).SendKeys("Simple Talk");


to type into a text field (in this case the search box on the Google home page). I like the clarity that this syntactical style provides, making it easy to understand existing tests and write new ones.

The second thing was WebDriver’s support for HtmlUnit. HtmlUnit is a Java project that implements a UI-less web browser entirely in memory. This makes the process of browser automation much more efficient, as the overhead of creating a new browser instance is greatly reduced. Unfortunately, since its a Java project, taking advantage of HtmlUnit from .NET hasn’t been a simple prospect up until now (although Steve Sanderson wrote a great blog post about how to do it using IKVM, see here). With the advent of Selenium 2, this process gets a whole lot easier!

The Selenium 2 alpha builds can be found here. I fired up the standalone server in a virtual machine, and create a new WebDriver against that machine as follows:

new RemoteWebDriver(new Uri("http://testVM:4444/wd/hub"), DesiredCapabilities.HtmlUnit());


Note the URL – to create a DefaultSelenium against this instance of the server, the URL would be http://testVM:4444/, whereas to create a WebDriver it is necessary to tack the /wd/hub onto the end.

To compare the performance of using HtmlUnit against Firefox (as I was previously) I copied one of my test fixtures into a new project and migrated the 12 tests over to use a RemoteWebDriver instead of a DefaultSelenium. Since I last wrote about using Selenium Grid, the number of tests has grown from 372 to 2736. To deal with this growth I increased the number of machines in the Grid to 8, the net result of which was that a fixture containing 12 tests running in at most 8 concurrent threads would take about 1 minute 30 seconds to execute. By comparison to this, 12 WebDriver/HtmlUnit-flavour tests running serially (since the Selenium Grid project doesn’t support WebDriver) took only 24 seconds! that’s almost 4x quicker (3.75x, to be exact), despite the lack of parallelism.

This convinced me that it was worth switching all 2736 tests over to use WebDriver, in spite of Selenium Grid’s lack of support for it. So after a couple of hours work migrating my tests and helper methods over to the new API, I was ready to do a full test run. Against the Grid (and its 8 machines), this took almost exactly 3 hours. Using HtmlUnit, the test run completed in 45 minutes for a 4x speedup :)

So even in these early stages, the Selenium 2 project is shaping up to be pretty exciting for .NET web testing. I look forward to getting my hands on an early build of Selenium Grid that supports WebDriver, so I can start taking advantage of HtmlUnit in multiple tests concurrently :D

Regression testing with Selenium GRID

A lot of software teams out there are tasked with supporting and maintaining systems that have grown organically over time, and the web team here at Red Gate is no exception.

We're about to embark on our first significant refactoring endeavour for some time, and as such its clearly paramount that the code be tested thoroughly for regressions. Unfortunately we currently find ourselves with a codebase that isn't very testable - the three layers (database, business logic and UI) are currently tightly coupled. This leaves us with the unfortunate problem that, in order to confidently refactor the code, we need unit tests. But in order to write unit tests, we need to refactor the code :S

To try and ease the initial pain of decoupling these layers, I've been looking into the idea of using UI automation to provide a sort of system-level regression test suite. The idea being that these tests can help us identify regressions whilst we work towards a more testable codebase, at which point the more traditional combination of unit and integration tests can take over. Ending up with a strong battery of UI tests is also a nice bonus :)

Following on from my previous posts (here, here and here) I knew I wanted to use Selenium. I also figured that this would be a good excuse to put my xUnit [Browser] attribute to good use. Pretty quickly, I had a raft of tests that looked like the following (this particular example uses Reflector Pro). In a nut shell the test traverses our shopping cart and, for a particular combination of number of users and months of support, checks that the price calculations all come up with the correct values.

[BrowserTheory]
[Browser(Browsers.Firefox3_6, "http://www.red-gate.com")]
public void Purchase1UserLicenceNoSupport(SeleniumProvider seleniumProvider)
{
    //Arrange
    _browser = seleniumProvider.GetBrowser();
    _browser.Open("http://www.red-gate.com/dynamic/shoppingCart/ProductOption.aspx?Product=ReflectorPro");
            
    //Act
    _browser = ShoppingCartHelpers.TraverseShoppingCart(_browser, 1, 0, ".NET Reflector Pro");

    //Assert
    var priceResult = PriceHelpers.GetNewPurchasePrice(db, "ReflectorPro", 1, 0, Currencies.Euros);
   
    Assert.Equal(priceResult.Price, _browser.GetText("ctl00_content_InvoiceShoppingItemRepeater_ctl01_Price"));
    Assert.Equal(priceResult.Tax, _browser.GetText("ctl00_content_InvoiceShoppingItemRepeater_ctl02_Tax"));
    Assert.Equal(priceResult.Total, _browser.GetText("ctl00_content_InvoiceShoppingItemRepeater_ctl02_Total"));
}


These tests are pretty concise, with much of the common code in the TraverseShoppingCart() and GetNewPurchasePrice() methods.

The (inevitable) problem arose when it came to execute these tests en masse. Selenium is a very slick tool, but it can't mask the fact that UI automation is very slow. To give you an idea, the set of cases that covers all of our products, for all combinations of users and support, came to 372 tests (for now only considering purchases in dollars). In the world of automated integration tests, that's a very manageable number. For unit tests, it's a trifle. However for UI automation, those 372 tests were taking just over two hours to run. Two hours may not sound like a lot, but those cases only cover one of the three currencies we deal with, and only one of the many different ways our systems can be asked to calculate a price.

It was already pretty clear at this point that in order for this approach to be viable, I was going to have to find a way to speed things up. Up to this point I had been using Selenium Remote Control to automate Firefox, as this was the approach I had used previously and it had worked well. Fortunately,  the guys at SeleniumHQ also maintain a tool for executing multiple Selenium RC tests in parallel: Selenium Grid.

Selenium Grid uses a central 'hub' to handle allocation of Selenium tests to individual RCs. The Remote Controls simply register themselves with the hub when they start, and then wait to be assigned work. The (for me) really clever part is that, as far as the client driver library is concerned, the grid hub looks exactly the same as a vanilla remote control. To create a new browser session against Selenium RC, the following C# code suffices:

new DefaultSelenium("localhost", 4444, "*firefox", "http://www.red-gate.com");

This assumes that the RC is running on the local machine, and is listening on port 4444 (the default). Assuming the hub is running on your local machine, then to create a browser session in Selenium Grid, via the hub rather than directly against the control, the code is exactly the same! Behind the scenes, the hub will take this request and hand it off to one of the registered RCs that provides the "*firefox" execution environment. It will then pass all communications back and forth between the test runner and the remote control transparently. This makes running existing RC tests on a Selenium Grid a piece of cake, as the developers intended. For a more detailed description of exactly how Selenium Grid works, see this page.

Once I had a test environment capable of running multiple tests in parallel, I needed a test runner capable of doing the same. Unfortunately, this does not currently exist for xUnit (boo!). MbUnit on the other hand, has the concept of concurrent execution baked right into the framework.

So after swapping out my assembly references, and fixing up the resulting mismatches in assertions, my example test now looks like this:

[Test]
public void Purchase1UserLicenceNoSupport()
{
   //Arrange
   ISelenium browser = BrowserHelpers.GetBrowser();
   var db = DbHelpers.GetWebsiteDBDataContext();
   browser.Start();
   browser.Open("http://www.red-gate.com/dynamic/shoppingCart/ProductOption.aspx?Product=ReflectorPro");
            
   //Act
    browser = ShoppingCartHelpers.TraverseShoppingCart(browser, 1, 0, ".NET Reflector Pro");
   var priceResult = PriceHelpers.GetNewPurchasePrice(db, "ReflectorPro", 1, 0, Currencies.Euros);

   //Assert
    Assert.AreEqual(priceResult.Price, browser.GetText("ctl00_content_InvoiceShoppingItemRepeater_ctl01_Price"));
    Assert.AreEqual(priceResult.Tax, browser.GetText("ctl00_content_InvoiceShoppingItemRepeater_ctl02_Tax"));
    Assert.AreEqual(priceResult.Total, browser.GetText("ctl00_content_InvoiceShoppingItemRepeater_ctl02_Total"));
}


This is pretty much the same as the xUnit version. The exceptions are that the attributes have changed,  the //Arrange phase now has to handle setting up the ISelenium object, as the attribute that previously did this has gone away, and the test now sets up its own database connection. Previously I was using a shared database connection, but this approach becomes more complicated when tests are being executed concurrently. To avoid complexity each test has its own connection, which it is responsible for closing. For the sake of readability, I snipped out the code that closes the browser session and the db connection at the end of the test.

With all that done, there was only one more step required before the tests would execute concurrently. It is necessary to tell the test runner which tests are eligible to run in parallel, via the [Parallelizable] attribute. This can be done at the test, fixture or assembly level. Since I wanted to run all tests concurrently, I marked mine at the assembly level in the AssemblyInfo.cs using the following:

[assembly: DegreeOfParallelism(3)]
[assembly: Parallelizable(TestScope.All)]


The second attribute marks all tests in the assembly as [Parallelizable], whilst the first tells the test runner how many concurrent threads to use when executing the tests. I set mine to three since I was using 3 RCs in separate VMs.

With everything now in place, I fired up the Icarus* test runner that comes with MbUnit. Executing my 372 tests three at a time instead of one at a time reduced the running time from 2 hours 10 minutes, to 55 minutes, that's an improvement of about 58%! I'd like to have seen an improvement of 66%, but I can understand that either inefficiencies in the hub code, my test environment or the test runner code (or some combination of all three most likely) contributes to a slightly diminished improvement. That said, I'd love to hear about any experience you have in upping this efficiency.

Ultimately though, it was a saving that was most definitely worth having. It makes regression testing via UI automation a far more plausible prospect. The other obvious point to make is that this approach scales far better than executing tests serially. So if ever we need to improve performance, we just register additional RC's with the hub, and up the DegreeOfParallelism.

*This was just my personal preference for a GUI runner. The MbUnit/Gallio installer also provides a command line runner, a TestDriven.net runner, and a Resharper 4.5 runner. For now at least, Resharper 5 isn't supported.

[BrowserTheory] - Adding a [URL] attribute to the [Browser] attribute for xUnit.net

EDIT: After a request, the full solution can be obtained from here.

After my last post it was suggested to me that I create a [URL] attribute to deal with the unnecessary repetition of the testStartPageUrl parameter when creating a [Browser] attribute. The example below shows the problem:

[Theory]
[Browser(Browsers.InternetExplorer7, "http://www.google.com")]
[Browser(Browsers.Firefox3_5, "http://www.google.com")]
[Browser(Browsers.GoogleChrome, "http://www.google.com")]
[Browser(Browsers.Opera, "http://www.google.com")]
public void Google_For_SimpleTalk(ISelenium iSelenium)
{
    //Test code
}


In each case it was necessary to specify "http://www.google.com", to be passed through to DefaultSelenium's constructor. However, it would be much cleaner if we were able to write something like the following:

[Theory]
[URL("http://www.google.com")]
[Browser(Browsers.InternetExplorer7)]
[Browser(Browsers.Firefox3_5)]
[Browser(Browsers.GoogleChrome)]
[Browser(Browsers.Opera)]
public void Google_For_SimpleTalk(ISelenium iSelenium)
{
    //Test code
}


Getting this much running was straightforward enough, thanks to a tip I received from Matt Ellis. He suggested that I take advantage of MethodInfo.GetCustomAttributes() from within the [Browser] attribute, to grab the [URL]. So I did just that! I added a private method to the BrowserAttribute responsible for populating its private _url, and then called this in my override to GetData():

private void GetURLFromAttribute(MethodInfo methodInfo)
{
    var urlAttribute = (URLAttribute) methodInfo.GetCustomAttributes(typeof (URLAttribute), false).FirstOrDefault();
    _url = urlAttribute.Url;
}

public override IEnumerable<object[]> GetData(MethodInfo methodUnderTest, Type[] parameterTypes)
{
    GetURLFromAttribute(methodUnderTest);
    yield return new[] { new[] { CreateBrowser() } };
}


You may have noticed that the above code has a problem. Specifically, if I were to omit the [URL] attribute my code would pass null to the DefaultSelenium constructor. This of course causes an exception to be thrown when I attempt to call Start() on the Browser object. I obviously needed a null check, so I inserted one at the beginning of the CreateBrowser() method:

if (_url == null)
throw new NullReferenceException("No [URL] attribute was defined");

At this point I was unfortunate enough to run into a bug in xUnit 1.1 :( Any exception thrown by a derived DataAttribute was not being handled by the test runner, causing it never to proceed beyond that failed attempt to retrieve test data. Fortunately for me, xUnit 1.5 shipped at the weekend, which included the fix for this issue :)

With that problem sorted, however, I immediately encountered another one. Moving from xUnit 1.1 to 1.5 seemed to have re-introduced the "start all the browsers before any test runs are executed" behaviour I eliminated last time out. On looking at the changes made in 1.5, it turns out that the same  change that fixed the exception issue was causing the browser start up problem! Briefly, the code that enumerates test data for DataAttributes went from yield returning new TheoryCommand()'s for each attribute, to adding them to a collection and returning that at the end of the loop.

It was clear that to once again fix this issue, I was going to have to delay calling the browser object's Start() method. Previously, I had been doing this within the BrowserAttribute.CreateBrowser() method, so when the enumeration code called my override to GetData() it was being passed back a started browser object.

To achieve the delayed start, I decided to wrap the browser in object in a new SeleniumProvider object, which I would pass into my tests. This object has one method, GetBrowser(), which calls Start() on it's browser and passes it back:

public class SeleniumProvider
{
    private ISelenium Browser;

    public SeleniumProvider(ISelenium selenium)
    {
        Browser = selenium;
    }

    public ISelenium GetBrowser()
    {
        Browser.Start();
        return Browser;
    }
}


My CreateBrowser() code was then changed thus:

private SeleniumProvider CreateBrowser()
{
    if (_url == null)
        throw new NullReferenceException(
            "No [URL] attribute was defined, either define [URL], or pass the URL of the start page to BrowserAttribute's constructor");

    switch (_browser)
    {
        case "Internet Explorer 6":
            Browser = new DefaultSelenium("testrunner1", 4444, "*iexplore", _url);
            break;
        // <SNIP>
    }
    return new SeleniumProvider(Browser);
}


So I can now happily write tests that look like this:

[Theory]
[URL("http://www.google.com")]
[Browser(Browsers.InternetExplorer7)]
[Browser(Browsers.Firefox3_5)]
[Browser(Browsers.GoogleChrome)]
[Browser(Browsers.Opera)]
public void Google_For_SimpleTalk(ISeleniumProvider seleniumProvider)
{
    Browser = seleniumProvider.GetBrowser();
    //Test code
}

At this point I was challenged by Matt Lee:

"What would be really smart would be if you could define multiple [URL]'s, and have each [Browser] test every [URL]!"

So having come this far, I decided to take on that challenge as well!

After a short period of time thinking, it became clear that I could not achieve this goal within the framework I already had. Despite knowing about all potential [URL]'s from within my BrowserAttribute code, I had no way off passing back multiple browser objects to a test method without re-introducing the kind of TestSetup-style code I was attempting to avoid. Instead, I would have to provide my own implementation of the code responsible for enumerating test data. This code lives in the TheoryAttribute, so to provide my own implementation I created the [BrowserTheory].

The code in question lives in TheoryAttribute.GetData(). This static method takes a MethodInfo, and yield returns an IEnumerable of object arrays containing the test data:

static IEnumerable<object[]> GetData(MethodInfo method)
{
foreach (BrowserAttribute attr in method.GetCustomAttributes(typeof(BrowserAttribute), false))
{
    ParameterInfo[] parameterInfos = method.GetParameters();
    Type[] parameterTypes = new Type[parameterInfos.Length];

    for (int idx = 0; idx < parameterInfos.Length; idx++)
        parameterTypes[idx] = parameterInfos[idx].ParameterType;

    if(attr._url != null)
    {
        IEnumerable<object[]> browserAttrData = attr.GetData(method, parameterTypes);
        if (browserAttrData != null)
            foreach (object[] dataItems in browserAttrData)
                yield return dataItems;
    }

    else foreach (URLAttribute attribute in method.GetCustomAttributes(typeof(URLAttribute), false))
    {
        attr._url = attribute.Url;
        IEnumerable<object[]> browserAttrData = attr.GetData(method, parameterTypes);
        if (browserAttrData != null)
            foreach (object[] dataItems in browserAttrData)
                yield return dataItems;
    }
}
}


Once it has constructed the array of Types required to call GetData() on each BrowserAttribute, the above code is doing one of two things. First it checks to see if the BrowserAttribute already has a URL defined. If it does, then it proceeds using that URL, ignoring any URLAttributes that may have been defined. This is to facilitate the use of the old style [Browser(string browser, string url)].

Otherwise, if there is no URL defined, it goes on to return that BrowserAttribute once for every URLAttribute it discovers attached to the test method.

That was all I needed to do! Now that the [URL]'s are being handled elsewhere, I no longer needed my BrowserAttribute.GetURLFromAttribute() method, so I removed it.

Finally, my example test case (expanded to include all browser versions) now reads like this:

[BrowserTheory]
[URL("http://www.google.co.uk")]
[URL("http://www.google.com")]
[Browser(Browsers.InternetExplorer6)]
[Browser(Browsers.InternetExplorer7, "http://www.google.com")]
[Browser(Browsers.InternetExplorer8)]
[Browser(Browsers.Firefox2)]
[Browser(Browsers.Firefox3)]
[Browser(Browsers.Firefox3_5)]
[Browser(Browsers.GoogleChrome)]
[Browser(Browsers.Opera)]
public void Google_For_SimpleTalk(SeleniumProvider seleniumProvider)
{
    Browser = seleniumProvider.GetBrowser();
    //Test code
}


I've included one example of the old-style [Browser], just to demonstrate how the two co-exist. The above will execute the test case twice for each browser (except Internet Explorer 7), starting from the two Google URLs. One test case execution will occur using IE7, starting from the google.com URL specifically provided to that BrowserAttribute. That makes a grand total of 15 test runs, for one copy of the test code :)

Improving the xUnit.net [Browser] attribute

In yesterday's post I talked about an xUnit [Browser] attribute that wraps Selenium, taking care of setting up the browser objects we pass as parameters into our test methods. Where I left off, my example test case looked as follows:

[Theory]
[Browser("Internet Explorer 7")]
[Browser("Firefox 3.5")]
[Browser("Google Chrome")]
[Browser("Opera")]
public void Google_For_SimpleTalk(ISelenium iSelenium)
{
    Browser = iSelenium;
    Browser.Open("/");
    Browser.Type("q", "Simple Talk");
    Browser.Click("btnG");
    Browser.WaitForPageToLoad("5000");
    Assert.True(Browser.IsTextPresent("www.simple-talk.com"));
}


This worked because the BrowserAttribute class had 'http://www.google.co.uk' hard coded as the URL passed into the DefaultSelenium constructor. Clearly hard-coding the starting point for each test is bad, since you would then need to navigate to the page you are interested in testing at the beginning of every test method.

Instead, it would be far more flexible to be able to pass the start page for any individual test into the DefaultSelenium constructor, via the [Browser] attribute. To do this, we simply change the BrowserAttribute code to the below:

public class BrowserAttribute : DataAttribute
{
private ISelenium Browser { get; set; }

public BrowserAttribute(string browser, string url)
{
switch (browser)
{
    case "Internet Explorer 7":
        Browser = new DefaultSelenium("localhost", 4444, "*iexplore", url);
        Browser.Start();
        break;
    case "Firefox 3.5":
        Browser = new DefaultSelenium("localhost", 4444, "*firefox", url);
        Browser.Start();
        break;
    case "Google Chrome":
        Browser = new DefaultSelenium("localhost", 4444, "*googlechrome", url);
        Browser.Start();
        break;
    case "Opera":
        Browser = new DefaultSelenium("localhost", 4444, "*opera", url);
        Browser.Start();
        break;
}
}

public override IEnumerable<object[]> GetData(MethodInfo methodUnderTest, Type[] parameterTypes)
{
    return new[] { Browser };
}
}


So now the attribute accepts two strings, one each for the browser and start URL. This means our test now becomes:

[Theory]
[Browser("Internet Explorer 7", "http://www.google.co.uk")]
[Browser("Firefox 3.5", "http://www.google.co.uk")]
[Browser("Google Chrome", "http://www.google.co.uk")]
[Browser("Opera", "http://www.google.co.uk")]
public void Google_For_SimpleTalk(ISelenium iSelenium)
{
    Browser = iSelenium;
    Browser.Open("/");
    Browser.Type("q", "Simple Talk");
    Browser.Click("btnG");
    Browser.WaitForPageToLoad("5000");
    Assert.True(Browser.IsTextPresent("www.simple-talk.com"));
}

At first glance this looks like we're unnecessarily duplicating test data. But it feels like a hit worth taking, since we can now decorate our test methods as follows:

[Theory]
[Browser("Google Chrome","http://SiteUnderTest/Account/Login.aspx")]
public void Login_Works_With_Valid_Credentials() {}

[Theory]
[Browser("Firefox 3.5","http://SiteUnderTest/Account/Register.aspx")]
public void Signup_Rejects_Invalid_Email_Addresses(){}

The second issue I highlighted last time was that of browser versions.

One of the nice things that Selenium gives you is the ability to offload the execution of your test cases to another machine or machines. This is done using Selenium RC ("Remote Control").

Briefly, Selenium RC works by running a server process on each machine that you want to execute tests on. This will handle launching browser instances, executing your test cases, and acts as a proxy between the browser and the site being tested, allowing it to intercept and verify the traffic passing between them.

To try this out, I fired up three virtual machines on one of testing servers. Between them, these machines cover:

• Internet Explorer 6
• Internet Explorer 7
• Internet Explorer 8
• Firefox 2
• Firefox 3
• Firefox 3.5
• Opera 9
• Google Chrome 2

This combination covers about ~96% of our total visitors to Simple Talk.

The Selenium RC server is written in Java (for cross-platform purposes), so it was necessary to first ensure that each VM had the Java Runtime installed. I also had to ensure that they were joined to our test network domain, otherwise I would have to resort to referencing them by their (ever-changing) IP address. The final (but not strictly necessary) preparation step was to write a short batch file to start the server process at boot time.

To start using the VMs instead of my local machine, I had to change how I was calling the DefaultSelenium constructor. Where before I was passing "localhost" for the serverUrl parameter, we now specify the machine name of the VM running that particular version of the browser in question. I won't list the entire set of cases in the switch block, as its getting quite long now, but below shows the change for the three versions of Internet Explorer.

public BrowserAttribute(string browser, string url)
        {
            switch (browser)
            {
                case "Internet Explorer 6":
                    Browser = new DefaultSelenium("testrunner1", 4444, "*iexplore", url);
                    Browser.Start();
                    break;
                case "Internet Explorer 7":
                    Browser = new DefaultSelenium("testrunner2", 4444, "*iexplore", url);
                    Browser.Start();
                    break;
                case "Internet Explorer 8":
                    Browser = new DefaultSelenium("testrunner3", 4444, "*iexplore", url);
                    Browser.Start();
                    break;

At this point I also decided to pull the magic browser strings out into a static 'Browsers' class, so that we do not need to know their values, and can instead select browsers via Intellisense:

With those changes made, I looked to address the final issue highlighted last time out - the way the browser objects were being created. By creating the Browser object and calling it's Start() method within the BrowserAttribute's constructor, I was causing xUnit to start up all the requested browser instances before any test run was started. To fix this, I simply need to Start() the browser from within my override of GetData() instead (thanks must go to Mark, for pointing me in this direction). In order to do that, I moved my switch block out of the constructor into a new CreateBrowser() method, which returns the ISelenium object. The constructor now simply accepts and stores the browser and URL strings locally, where they are subsequently picked up by CreateBrowser(). The GetData() method now yield returns a call to CreateBrowser():

public class BrowserAttribute : DataAttribute
{
private ISelenium Browser { get; set; }
private string _browser { get; set; }
private string _url { get; set; }

public BrowserAttribute(string browser, string url)
{
    _browser = browser;
    _url = url;
}

private ISelenium CreateBrowser()
{
switch (_browser)
{
    case "Internet Explorer 6":
        Browser = new DefaultSelenium("testrunner1", 4444, "*iexplore", _url);
        Browser.Start();
        break;
    case "Internet Explorer 7":
        Browser = new DefaultSelenium("testrunner2", 4444, "*iexplore", _url);
        Browser.Start();
        break;
    case "Internet Explorer 8":
        Browser = new DefaultSelenium("testrunner3", 4444, "*iexplore", _url);
        Browser.Start();
        break;
    case "Firefox 2":
        Browser = new DefaultSelenium("testrunner1", 4444, "*firefox", _url);
        Browser.Start();
        break;
    case "Firefox 3":
        Browser = new DefaultSelenium("testrunner2", 4444, "*firefox", _url);
        Browser.Start();
        break;
    case "Firefox 3.5":
        Browser = new DefaultSelenium("testrunner3", 4444, "*firefox", _url);
        Browser.Start();
        break;
    case "Google Chrome":
        Browser = new DefaultSelenium("testrunner1", 4444, "*googlechrome", _url);
        Browser.Start();
        break;
    case "Opera":
        Browser = new DefaultSelenium("testrunner3", 4444, "*opera", _url);
        Browser.Start();
        break;
}
return Browser;
}

public override IEnumerable<object[]> GetData(MethodInfo methodUnderTest, Type[] parameterTypes)
{
    yield return new[] { CreateBrowser() };
}
}


We now have everything we need to write concise integration tests, exercising the vast majority of browsers that our site sees visits from. I should also point out that, whilst the VMs I use are all running XP, Selenium RC works on any platform that can run the Java Runtime Environment. What's more, there is no difference whatsoever from the point of view of your test code. So if you get a lot of traffic from MacOS users browsing in Safari, or Linux-based users with Konquerer, its as simple as starting an instance of the RC server on a MacOS or Linux machine, and using its name when you create your DefaultSelenium object. Selenium will handle the rest!

This is a relatively new pursuit for us, so if anybody has any tips, or can point out potential pit falls, I'm all ears!

Combining xUnit.net and Selenium

There has been a fair amount of discussion about how to improve our test automation here in the Web Team at Red Gate. One of the avenues that we are keen to go down is browser automation.

In the browser automation arena, the two big players for .NET web applications are WATIN (Web Application Testing In .Net) and Selenium. In a recent blog post, Ben Hall describes an xUnit attribute that allows you to run WATIN tests across multiple browsers, simply by providing the desired browser as an argument to your test method.

I took Ben’s code for a test drive, and really liked how concise the resulting tests were. However, there are well-documented advantages to using Selenium as your browser automation framework, so I decided to have a go at re-writing Ben’s [Browser] attribute to target Selenium instead.

Writing your own xUnit attribute is deliberately quite straightforward. Each attribute’s abstract base class is public, allowing you to easily write your own implementation. In this case we’re interested in the DataAttribute class.

The DataAttribute class only contains one abstract method that needs overriding, namely:

public abstract IEnumerable<object[]> GetData(MethodInfo methodUnderTest, Type[] parameterTypes);

So our implementation needs to return an object[] containing the selenium browser object to pass into our test:

public class BrowserAttribute : DataAttribute
{
  private ISelenium Browser { get; set; }
  public BrowserAttribute(string browser)
  {
    string url = "http://www.google.co.uk";
    switch (browser)
    {
      case "Internet Explorer 7":
      Browser = new DefaultSelenium("localhost", 4444, "*iexplore", url);
      Browser.Start();
      break;
      case "Firefox 3.5":
      Browser = new DefaultSelenium("localhost", 4444, "*firefox", url);
      Browser.Start();
      break;
      case "Google Chrome":
      Browser = new DefaultSelenium("localhost", 4444, "*googlechrome", url);
      Browser.Start();
      break;
      case "Opera":
      Browser = new DefaultSelenium("localhost", 4444, "*opera", url);
      Browser.Start();
      break;
    }
  }
  public override IEnumerable<object[]> GetData(MethodInfo methodUnderTest, Type[] parameterTypes)
  {
    return new[] { Browser };
  }
}

The above implementation is enough to allow us to write nice clean tests, like this (note that I pass the iSelenium parameter into the Browser object purely for post test clean up purposes, which I won’t worry about showing here):



[Theory]
[Browser("Internet Explorer 7")]
[Browser("Firefox 3.5")]
[Browser("Google Chrome")]
[Browser("Opera")]
public void Google_For_SimpleTalk(ISelenium iSelenium)
{
    Browser = iSelenium;
    Browser.Open("/");
    Browser.Type("q", "Simple Talk");
    Browser.Click("btnG");
    Browser.WaitForPageToLoad("5000");
    Assert.True(Browser.IsTextPresent("www.simple-talk.com"));
}



This version of the Selenium [Browser] attribute is obviously not very robust, as it has the URL of the Google homepage hard-coded. Moreover, it is currently only able to take advantage of the browser versions currently installed on my local machine. Whilst these versions are the most popular amongst the visitors to our sites, they are by no means the only ones. Another drawback is that, in its current form, the attribute causes all the requested browser instances to be created before any of the executions of a test case occur.

Once I had this basic version up and running, my next task was to extend it to address these issues, so that we can begin to utilise it in our production test environment. In my next post I’ll go through how I did that!