Also see Part 2 on how to address gotcha’s in this process
When using Release Management there is a good chance you will want to run test suites as part of your automated deployment pipeline. If you are using a vNext PowerShell based pipeline you need a way to trigger the tests via PowerShell as there is no out the box agent to do the job.
Step 1 - Install a Test Agent
The first step is to make sure that the Visual Studio Test Agent is installed on the box you wish to run the test on. if you don’t already have a MTM Environment in place with a test agent then this can be done by creating a standard environment in Microsoft Test Manager. Remember you only need this environment to include the VM you want to run the test on, unless you want to also gather logs and events from our machines in the system. The complexity is up to you.
In my case I was using a network isolated environment so all this was already set up.
Step 2 - Setup the Test Suite
Once you have an environment you can setup your test suite and test plan in MTM to include the tests you wish to run. These can be unit test style integration tests or Coded UI it is up to you.
If you have a lot of unit tests to associate for automation remember the TCM.EXE command can make your life a lot easier
This post does not aim to be a tutorial on setting up test plans, have a look at the ALM Rangers guides for more details.
Step 3 - The Release Management environment
This is where it gets a bit confusing, you have already set up a Lab Management environment, but you still need to setup the Release Management vNext environment. As I was using a network isolated Lab management environment this gets even more complex, but RM provides some tools to help
Again this is not a detailed tutorial. The key steps if you are using network isolation are
- Make sure that PowerShell on the VM is setup for remote access by running winrm quickconfig
- In RM create a vNext environment
- Add each a new server, using it’s corporate LAN name from Lab Management with the PowerShell remote access port e.g. VSLM-1002-e7858e28-77cf-4163-b6ba-1df2e91bfcab.lab.blackmarble.co.uk:5985
- Make sure the server is set to use a shared UNC path for deployment.
- Remember you will login to this VM with the credentials for the test domain.
By this point you might be a bit confused as to what you have, well here is a diagram
Step 4 - Wiring the test into the pipeline
The final step is get the release pipeline to trigger the tests. This is done by calling the TCM.EXE command line to instruct the Test Controller trigger the tests. Now the copy of TCM does not have to be in Lab Management environment, but it does need to be on a VM known to RM vNext environment. This will usually mean a VM with Visual Studio Test Manager or Premium (or Enterprise for 2015) installed. In my case this was a dedicated test VM within the environment.
The key to the process is to run a script similar to the one used by the older RM agent based system to trigger the tests. You can extract this PowerShell script from an old release pipeline, but for ease I show my modified version here. The key changes are that I pass in the login credentials required for the call to the TFS server from TCM.EXE to be made from inside the network isolated environment and do a little extra checking of the test results so I can fail the build if the tests fail. These edits might not be required if you trigger TCM from a VM that is in the same domain as your TFS server, or have different success criteria.
param
(
\[string\]$BuildDirectory = $null,
\[string\]$BuildDefinition = $null,
\[string\]$BuildNumber = $null,
\[string\]$TestEnvironment = $null,
\[string\]$LoginCreds = $null,
\[string\]$Collection = $(throw "The collection URL must be provided."),
\[string\]$TeamProject = $(throw "The team project must be provided."),
\[Int\]$PlanId = $(throw "The test plan ID must be provided."),
\[Int\]$SuiteId = $(throw "The test suite ID must be provided."),
\[Int\]$ConfigId = $(throw "The test configuration ID must be provided."),
\[string\]$Title = 'Automated UI Tests',
\[string\]$SettingsName = $null,
\[Switch\]$InconclusiveFailsTests = $false,
\[Switch\]$RemoveIncludeParameter = $false,
\[Int\]$TestRunWaitDelay = 10
)
##################################################################################
\# Output the logo.
write-verbose "Based on the Microsoft Release Management TcmExec PowerShell Script v12.0"
write-verbose "Copyright (c) 2013 Microsoft. All rights reserved.\`n"
##################################################################################
\# Initialize the default script exit code.
$exitCode = 1
##################################################################################
\# Output execution parameters.
write-verbose "Executing with the following parameters:"
write-verbose " Build Directory: $BuildDirectory"
write-verbose " Build Definition: $BuildDefinition"
write-verbose " Build Number: $BuildNumber"
write-verbose " Test Environment: $TestEnvironment"
write-verbose " Collection: $Collection"
write-verbose " Team project: $TeamProject"
write-verbose " Plan ID: $PlanId"
write-verbose " Suite ID: $SuiteId"
write-verbose " Configuration ID: $ConfigId"
write-verbose " Title: $Title"
write-verbose " Settings Name: $SettingsName"
write-verbose " Inconclusive result fails tests: $InconclusiveFailsTests"
write-verbose " Remove /include parameter from /create command: $RemoveIncludeParameter"
write-verbose " Test run wait delay: $TestRunWaitDelay"
##################################################################################
\# Define globally used variables and constants.
\# Visual Studio 2013
$vscommtools = \[System.Environment\]::GetEnvironmentVariable("VS120COMNTOOLS")
if ($vscommtools -eq $null)
{
# Visual Studio 2012
$vscommtools = \[System.Environment\]::GetEnvironmentVariable("VS110COMNTOOLS")
}
if ($vscommtools -eq $null)
{
# Visual Studio 2010
$vscommtools = \[System.Environment\]::GetEnvironmentVariable("VS100COMNTOOLS")
if ($vscommtools -ne $null)
{
if (\[string\]::IsNullOrEmpty($BuildDirectory))
{
$(throw "The build directory must be provided.")
}
if (!\[string\]::IsNullOrEmpty($BuildDefinition) -or !\[string\]::IsNullOrEmpty($BuildNumber))
{
$(throw "The build definition and build number parameters may be used only under Visual Studio 2012/2013.")
}
}
}
else
{
if (\[string\]::IsNullOrEmpty($BuildDefinition) -and \[string\]::IsNullOrEmpty($BuildNumber) -and \[string\]::IsNullOrEmpty($BuildDirectory))
{
$(throw "You must specify the build directory or the build definition and build number.")
}
}
$tcmExe = \[System.IO.Path\]::GetFullPath($vscommtools + "..IDETCM.exe")
##################################################################################
\# Ensure TCM.EXE is available in the assumed path.
if (\[System.IO.File\]::Exists($tcmExe))
{
##################################################################################
# Prepare optional parameters.
$testEnvironmentParameter = "/testenvironment:$TestEnvironment"
if (\[string\]::IsNullOrEmpty($TestEnvironment))
{
$testEnvironmentParameter = \[string\]::Empty
}
if (\[string\]::IsNullOrEmpty($BuildDirectory))
{
$buildDirectoryParameter = \[string\]::Empty
} else
{
# make sure we remove any trailing slashes as the cause permission issues
$BuildDirectory = $BuildDirectory.Trim()
while ($BuildDirectory.EndsWith(""))
{
$BuildDirectory = $BuildDirectory.Substring(0,$BuildDirectory.Length-1)
}
$buildDirectoryParameter = "/builddir:""$BuildDirectory"""
}
$buildDefinitionParameter = "/builddefinition:""$BuildDefinition"""
if (\[string\]::IsNullOrEmpty($BuildDefinition))
{
$buildDefinitionParameter = \[string\]::Empty
}
$buildNumberParameter = "/build:""$BuildNumber"""
if (\[string\]::IsNullOrEmpty($BuildNumber))
{
$buildNumberParameter = \[string\]::Empty
}
$includeParameter = '/include'
if ($RemoveIncludeParameter)
{
$includeParameter = \[string\]::Empty
}
$settingsNameParameter = "/settingsname:""$SettingsName"""
if (\[string\]::IsNullOrEmpty($SettingsName))
{
$settingsNameParameter = \[string\]::Empty
}
##################################################################################
# Create the test run.
write-verbose "\`nCreating test run ..."
$testRunId = & "$tcmExe" run /create /title:"$Title" /login:$LoginCreds /planid:$PlanId /suiteid:$SuiteId /configid:$ConfigId /collection:"$Collection" /teamproject:"$TeamProject" $testEnvironmentParameter $buildDirectoryParameter $buildDefinitionParameter $buildNumberParameter $settingsNameParameter $includeParameter
if ($testRunId -match '.+:s(?<TestRunId>d+).')
{
# The test run ID is identified as a property in the match collection
# so we can access it directly by using the group name from the regular
# expression (i.e. TestRunId).
$testRunId = $matches.TestRunId
write-verbose "Waiting for test run $testRunId to complete ..."
$waitingForTestRunCompletion = $true
while ($waitingForTestRunCompletion)
{
Start-Sleep -s $TestRunWaitDelay
$testRunStatus = & "$tcmExe" run /list /collection:"$collection" /login:$LoginCreds /teamproject:"$TeamProject" /querytext:"SELECT \* FROM TestRun WHERE TestRunId=$testRunId"
if ($testRunStatus.Count -lt 3 -or ($testRunStatus.Count -gt 2 -and $testRunStatus.GetValue(2) -match '.+(?<DateCompleted>d+\[/\]d+\[/\]d+)'))
{
$waitingForTestRunCompletion = $false
}
}
write-verbose "Evaluating test run $testRunId results..."
# We do a small pause since the results might not be published yet.
Start-Sleep -s $TestRunWaitDelay
$testRunResultsTrxFileName = "TestRunResults$testRunId.trx"
& "$tcmExe" run /export /id:$testRunId /collection:"$collection" /login:$LoginCreds /teamproject:"$TeamProject" /resultsfile:"$testRunResultsTrxFileName" | Out-Null
if (Test-path($testRunResultsTrxFileName))
{
# Load the XML document contents.
\[xml\]$testResultsXml = Get-Content "$testRunResultsTrxFileName"
# Extract the results of the test run.
$total = $testResultsXml.TestRun.ResultSummary.Counters.total
$passed = $testResultsXml.TestRun.ResultSummary.Counters.passed
$failed = $testResultsXml.TestRun.ResultSummary.Counters.failed
$inconclusive = $testResultsXml.TestRun.ResultSummary.Counters.inconclusive
# Output the results of the test run.
write-verbose "\`n========== Test: $total tests ran, $passed succeeded, $failed failed, $inconclusive inconclusive =========="
# Determine if there were any failed tests during the test run execution.
if ($failed -eq 0 -and (-not $InconclusiveFailsTests -or $inconclusive -eq 0))
{
# Update this script's exit code.
$exitCode = 0
}
# Remove the test run results file.
remove-item($testRunResultsTrxFileName) | Out-Null
}
else
{
write-error "\`nERROR: Unable to export test run results file for analysis."
}
}
}
else
{
write-error "\`nERROR: Unable to locate $tcmExe"
}
##################################################################################
\# Indicate the resulting exit code to the calling process.
if ($exitCode -gt 0)
{
write-error "\`nERROR: Operation failed with error code $exitCode."
}
write-verbose "\`nDone."
exit $exitCode
Once this script is placed into source control in such a way that it ends up in the drops location for the build you can call it as a standard script item in your pipeline, targeting the VM that has TCM installed. Remember, you get the test environment name and various IDs required from MTM. Check the TCM command line for more details.
However we hit a problem, RM sets PowerShell variable, not the parameters for script . So I find it easiest to use a wrapper script, also stored in source control, that converts the variable to the needed parameters. This also gives the opportunity to use RM set runtime variables and build more complex objects such as the credentials
\# Output execution parameters.
$VerbosePreference ='Continue' # equiv to -verbose
$folder = Split-Path -Parent $MyInvocation.MyCommand.Definition
write-verbose "Running $folderTcmExecWithLogin.ps1"
& "$folderTcmExecWithLogin.ps1" -Collection $Collection -Teamproject $Teamproject -PlanId $PlanId -SuiteId $SuiteId -ConfigId $ConfigId -BuildDirectory $PackageLocation -TestEnvironment $TestEnvironment -LoginCreds "$TestUserUid,$TestUserPwd" -SettingsName $SettingsName
Step 5 – Run it all
If you have everything in place you should now be able to trigger your deployment and have the tests run.
Finishing Up and One final gotcha
I had hoped that my integration test run would be associated with my build. Normally when triggering test via TCM you do this by adding the following parameters to the TCM command line
TCM \[all the other params\] -BuildNumber 'My.Build.CI\_1.7.25.29773' -BuildDefinition 'My.Build.CI'
However this will not work in the scenario above. This is because you can only use these flags to associate with successful builds, at the time TCM is run in the pipeline the build has not finished so it is not marked as successful. This does somewhat limit the end to end reporting. However, I think for now I can accept this limitation as the deployment completing is a suitable marker that the tests were passed.
The only workaround I can think is not to trigger the release directly from the build but to use the TFS events system to allow the build to finish first then trigger the release. You could use my TFS DSL Alert processor for that.