Fixes for issues moving from on-premise Azure DevOps agents to Azure Managed DevOps Pool agents

Background

Our on premise build agents, though using the same image as the Microsoft hosted agents (generated using Packer, as I have previously posted about), have some extra setup done by Lability as they are deployed to a Hyper-V host.

Most of this is specific to the needs of running on Hyper-V e.g. setting up networking, creating a 2nd disk to act as a working store for the build agent, and of course installing the Azure DevOps agent itself.

However, it turns out some of the extra steps it was doing were a barrier to running some of our major software project’s Azure DevOps Pipelines on Microsoft hosted agents.

This was something we had not worried too much about in the past. This was because the performance of the hosted agents was too slow compared to what we can provide on-premise using Hyper-V. However, with the advent of Azure Managed DevOps Pools as discussed in my recent blog post, with their greater options for scaling, this became an issue worth investigating.

So I looked to see how I could get all our pipelines to run on the Microsoft hosted agents and on Managed DevOps Pools.

The Issues

Custom Capabilities

The first issue we hit was that the Microsoft hosted agents and Managed DevOps Pools do not support custom capabilities. With on-premise agents you can add any custom capability you wish, usually used to help route jobs to a suitable agent within an agent pool, but this is not possible with the Microsoft hosted agents or Managed DevOps Pools.

‘Under the hood’ a capability is just an environment variable on the VM that the agent uses to identify the capabilities of the agent. So, though not what they were really designed for, custom capabilities are a useful way to be able to inject an environment variable to all VMs in a pool without having to alter the VM image or add startup scripts.

Historically we have used this technique to add missing environment variables to the agents i.e. to set JAVA and JDK to match the already present JAVA_HOME value. We need to do this as some tools, such as the Azure DevOps XamarinAndroid@1 task, only checks JAVA and JDK, and not JAVA_HOME to find the JDK location.

But, as I mentioned, there is no way to add a custom capability to a Microsoft hosted agent or a Managed DevOps Pool agent at the pool level. The only option is to set it as an environment variable within the VM image. Given we wanted, to minimise maintenance costs, to use the Microsoft hosted build agent, which we have no control over, this technique was not an option.

However, it turns out there is a much better solution in the XamarinAndroid@1 task case, to just pass the JDK version as a parameter to the task. This is a better solution as it is more explicit and does not rely on the agent having the correct environment variables set.

- task: XamarinAndroid@1
displayName: "Build Xamarin.Android Project"
inputs:
    projectFile: src/Droid.csproj
    target: Build
    configuration: "$(BuildConfiguration)"
    clean: true
    msbuildVersionOption: latest
    msbuildArguments: "-m"
    JDKVERSION: 1.11

So your first option should always be to check if you can use a task parameter as opposed to using custom capabilities (environment variables).

If you find that your task does not provide a parameter to set the value you need, then you can still ‘inject’ an environment variable onto an agent by setting an Azure DevOps variable in your pipeline. Remember, this works as Azure DevOps variables at are passed on to agents as environment variables.

To create a variable the following command can be used in a script task.

 echo "##vso[task.setvariable variable=MyEnvVar]MyValue"

Missing folders in PATH

For historic reasons we add the C:\Program Files (x86)\Windows Kits\10\bin folder to the PATH on our on-premise agents. This is because we use the signtool.exe to sign our projects, and this is where it is located by default.

We could of course alter all our pipelines to pass the explicit path to the signtool.exe to the task, but this is a lot of work and would make the pipelines less portable. So we wanted a means to add the path dynamically.

You might think you could use SETX or the PowerShell equivalent to add a folder to the PATH, but this does not work. The path is added to the current CMD/PS session, but not to the next task run by the Azure DevOps agent.

The answer is to use some Azure DevOps magic I was not previously aware of

echo "##vso[task.prependpath]<path to add>"

I used the following PowerShell to check if the required path was present and if not add it. You could of course greatly simplify the script if you only ever run on hosted agents and knew the actual path was always going to be missing, but I might use the script against on-premise agent where the path was already present, hence the guard code

- powershell: |
    # Define the base path where signtool.exe is located
    $basePath = "C:\Program Files (x86)\Windows Kits\10\bin"
    # Filtering via version and architecture, could use just one of the these, depends on needs
    $preferredVersion = "10.0.19041.0\x86"

    $envPath = [Environment]::GetEnvironmentVariable("PATH")
    $targetPath = join-path -path $basepath -childpath $preferredVersion

    # Check to see if the folder is already in the path
    if ($envPath -split ";" -contains $targetPath) {
        Write-Host "The path $targetPath is already present in the PATH environment variable."
    } else {
        Write-Host "The path $targetPath is NOT present in the PATH environment variable."

        # Get the matching signtool path (pick the last in the list if multiple returned)
        $signtoolPath = (Get-ChildItem -Path $basePath -Recurse -Filter "signtool.exe" -File | Where-Object { $_.FullName -like "*\$preferredVersion\*" })[-1] | Select-Object -ExpandProperty FullName

        # Double check the access to the file 
        if (Test-Path -Path $signtoolPath) {
            write-host "SignTool path is $signtoolPath"
            write-host "Updating PATH to include SignTool folder"
            echo "##vso[task.prependpath]$(Split-path -path $signtoolPath)"
        } else {
            write-error"Cannot find SignTool that match the filter $preferredVerison"
        }
    }

Using this technique we can add any path required dynamically at the start of the pipeline, and it will be available to all tasks in the pipeline.

Timezone and Language

The Microsoft hosted agent images are set to the UTC timezone and en-US language. For us this is a problem for execution of some unit tests which have a dependency on GMT Standard Time and en-GB.

Maybe we should alter our tests to make them more robust, but we can address the issue by setting the correct values as part of the pipeline

- powershell: |
    $zone = Get-TimeZone
    if ($zone.Id -eq "GMT Standard Time") {
    write-host "Timezone is correctly set to $($zone.Id)"
    } else {
        write-host "Timezone is incorrectly set to $($zone.Id), setting to GMT Standard Time"
        Set-TimeZone -id "GMT Standard Time"
    }

    $list = Get-WinUserLanguageList
    if ($list[0].LanguageTag -eq "en-GB") {
      write-host "Language is correctly set to $($list[0].LanguageTag)"
    } else {
      write-host "Language is incorrectly set to $($list[0].LanguageTag), setting to en-GB"
      $list[0] = "en-GB"
      Set-WinUserLanguageList -LanguageList $list -Confirm:$false -Force
      Set-ItemProperty -Path "HKCU:\Control Panel\International" -Name sCountry -Value "United kingdom";
      Set-ItemProperty -Path "HKCU:\Control Panel\International" -Name sLongDate -Value "dd MMMM yyyy";
      Set-ItemProperty -Path "HKCU:\Control Panel\International" -Name sShortDate -Value "dd/MM/yyyy";
      Set-ItemProperty -Path "HKCU:\Control Panel\International" -Name sShortTime -Value "HH:mm";
      Set-ItemProperty -Path "HKCU:\Control Panel\International" -Name sTimeFormat -Value "HH:mm:ss";
      Set-ItemProperty -Path "HKCU:\Control Panel\International" -Name sYearMonth -Value "MMMM yyyy";
      Set-ItemProperty -Path "HKCU:\Control Panel\International" -Name iFirstDayOfWeek -Value 0;
    }

This pattern can be used more than once in a single pipeline run to test against multiple timezones and languages, but for us it is just a case of setting the GMT/UK values once.

Summary

It can be seen these fixes are minor and so there should be no technical barrier to us using standard hosted agents, or MDP agents, in the future.

I think that many of these ‘fixes’ will be needed in most of our pipelines. So, it is probably a good idea to have a YAML template that is included at the start of all builds. One that sets the settings above, in effect making hosted agents identical to our on-premise ones.

So moving to hosted agents or MDPs, becomes a question of cost and not technology. One that is a much harder question to answer, balancing capital and servicing costs for on-premise agents against operational costs for the Azure hosted agents.

The reality is everyone’s ‘milage will vary’, but I have a feeling in our case the dynamic scaling of MDP agents, and the management cost reductions this can bring, will make all the difference

Background#

The Issues#

Custom Capabilities#

Missing folders in PATH#

Timezone and Language#

Summary#