Duplicate Test DLLs with vstest.console.exe causes failures

The Problem

I recently did our regular update of our Azure DevOps Private build agents. It is rare we see problems when we do this, but this time one of our very regularly run builds started to fail when running unit tests.

We had not changed the project source code, all the test ran locally in Visual Studio. We had not change the build pipeline YAML

  - task: VSTest@2
        displayName: 'Unit Tests - Services'
        inputs:
          testAssemblyVer2: |
            **\*.unittests.dll
            !**\obj\**            
          searchFolder: '$(System.DefaultWorkingDirectory)/src

And from the pipeline logs, we could see that the CLI command being generated by the VSTest@2 task was also unchanged, finding 39 DLLs that matched the filter.

So the only change was the version of the Visual Studio tools installed on the build agent. Specifically vstest.console.exe had been updated from Version 17.3.2 (x64) to Version 17.4.1 (x64)

Solution

On checking the list of DLLs containing tests to be run we saw that some unit test DLLs were duplicated. They were appearing in their expected folders, but also in other project folders e.g.

vstest.console.exe “E:\Agent_work\3\s\src\BM.Services\BM.Service1.UnitTests\bin\Release\BM.Service1.UnitTests.dll” “E:\Agent_work\3\s\src\BM.Services\BM.Service2.UnitTests\bin\Release\BM.Service2.UnitTests.dll” “E:\Agent_work\3\s\src\BM.Services\BM.Service2.UnitTests\bin\Release\BM.Service1.UnitTests.dll” …

This was due to reference entries in the .csproj files that should not have been there.

So, the fix was to just remove the incorrectly present references to the DLLs in the various .csproj files.

This is one of those problems that makes you ask ‘why has this not been an issue before?’, but at least it is fixed now.