I have been making sure some Selenium UX tests that were originally written against Chrome also work with other browsers. I have had a few problems, the browser under test failing to load or Selenium not being able to find elements.

Turns out the solution is to just use the custom driver start-up options, the default constructors don’t seem to work for browsers other theran Chrome and Firefox.

Hence, I not have helper method that creates a driver from me based the a configuration parameter

 internal static IWebDriver GetWebDriver()
    {
        var driverName = GetWebConfigSetting("webdriver");
        switch (driverName)
        {
            case "Chrome":
                return new ChromeDriver();
            case "Firefox":
                return new FirefoxDriver();
            case "IE":
                InternetExplorerOptions caps = new InternetExplorerOptions();
                caps.IgnoreZoomLevel = true;
                caps.EnableNativeEvents = false;
                caps.IntroduceInstabilityByIgnoringProtectedModeSettings = true;
                caps.EnablePersistentHover = true;
                return new InternetExplorerDriver(caps);
            case "Edge-Chromium":
                var service = EdgeDriverService.CreateDefaultService(Directory.GetCurrentDirectory(), "msedgedriver.exe");
                return new EdgeDriver(service);
            default:
                throw new ConfigurationErrorsException($"{driverName} is not a known Selenium WebDriver");
        }
    }

The Problem

I have a release that installs VSTest and runs some integration tests that target .NET 4.6 x64. All these tests worked fine in Visual Studio. However, I got the following errors for all tests when they were run in a release

2020-04-23T09:30:38.7544708Z vstest.console.exe "C:agent\_workr1aPaymentServicesdroptestartifactsPaymentService.IntegrationTests.dll"

2020-04-23T09:30:38.7545688Z /Settings:"C:agent\_work\_tempuxykzf03ik2.tmp.runsettings"

2020-04-23T09:30:38.7545808Z /Logger:"trx"

2020-04-23T09:30:38.7545937Z /TestAdapterPath:"C:agent\_workr1aPaymentServicesdroptestartifacts"

2020-04-23T09:30:39.2634578Z Starting test execution, please wait...

2020-04-23T09:30:39.4783658Z A total of 1 test files matched the specified pattern.

2020-04-23T09:30:40.8660112Z   X Can\_Get\_MIDs \[521ms\]

2020-04-23T09:30:40.8684249Z   Error Message:

2020-04-23T09:30:40.8684441Z    Test method PaymentServices.IntegrationTests.ControllerMIDTests.Can\_Get\_MIDs threw exception: 

2020-04-23T09:30:40.8684574Z System.BadImageFormatException: Could not load file or assembly 'PaymentServices, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. An attempt was made to load a program with an incorrect format.

2020-04-23T09:30:40.8684766Z   Stack Trace:

2020-04-23T09:30:40.8684881Z       at PaymentServices.IntegrationTests.ControllerMIDTests.Can\_Get\_MIDs()

…

2020-04-23T09:30:40.9038788Z Results File: C:agent\_work\_tempTestResultssvc-devops\_SVRHQAPP027\_2020-04-23\_10\_30\_40.trx

2020-04-23T09:30:40.9080344Z Total tests: 22

2020-04-23T09:30:40.9082348Z      Failed: 22

2020-04-23T09:30:40.9134858Z ##\[error\]Test Run Failed.

Solution

I needed to tell vstest.console.exe to run x64 as opposed to it’s default of x32. This can be done with a command line override –platform:x64

Of late I have certainly seen a bit of a move towards using more functional style tests; still using unit test frameworks, but relying on APIs as access points with real backend systems such as DBs and WebServices being deployed as test environments.

This practice is made far easier than in the past due to cloud services such as Azure and tools to treat creation of complex environments  as code such as Azure Resource Manager and Azure DevTest Labs. Both myself and my colleague RIk Hepworth have posted widely on  the provisioning of such systems.

However, this type of functional testing is still fairly slow, the environments have to be provisioned from scratch, or spun up from saved images, it all takes time. Hence, there is still the space for fast unit tests, and sometimes, usually due to limitations of legacy codebases that were not designed for testing, there is a need to still ‘mock the un-mockable’.

This is where tools like Typemock Isolator and Microsoft Fakes are still needed. 

It has to be said, both are premium products, you need the top Enterprise SKU of Visual Studio to get Fakes or a Typemock Isolator license for Isolator, but when you have a need them their functionality they are the only option. Whether this be to mock out a product like SharePoint for faster development cycles, or to provide a great base to write unit tests on for a legacy code base prior to refactoring.

As I have said before, for me Typemock Isolator easily has the edge over Microsoft Fakes, the syntax is so much easier to use. Hence, it is great to see the Typemock Isolator being have further extended with updated versions for C++ and now Linux.

So in answer to my own question, testing is a layered process. Where you put your investment is going to be down to your systems needs. It is true, I think we are going to all invest a bit more in functional testing on ‘cheap to build and run’ cloud test labs. But you can’t beat the speed of tools like Typemock for those particularly nasty legacy code bases where it is hard to create a copy of the environment in a modern test lab.