When I got home, I thought that this old post & flowchart were worth bringing up to date. So, I also took the chance to make them a little more generic, making them more useful as a guide for a wider range of DevOps toolset migrations, not just Azure DevOps.

The flowchart is still skewed towards Azure DevOps and GitHub Enterprise, as those are the toolset I have most experience with, but the general principle work for any DevOps toolset migration.

Click to download a PDF version of the flowchart  

In the flowchart I mention a few tools, so here are some useful links

• Azure DevOps Server to the Azure DevOps Services migration

  • Azure DevOps Database Migration Tool – the official Microsoft full fidelity TPC migration service

• Source Control migration tools

  • Git TFS – OSS tool to move TFVC into Git
  • Git SVN – OSS tool to move SVN into Git
  • Git VSS – OSS tool to move VSS into Git
  • Azure DevOps Git Import - built-in feature of Azure DevOps to import Git repo into Azure DevOps
  • Azure DevOps TFVC Import - built-in feature of Azure DevOps to import TFVC into Azure DevOps
  • GiHub Git Import - built-in feature of GitHub to import Git repo
  • GitHub Enterprise Import - GitHub CLI tool to import repos from other toolset at scale, and re potentially re-point CI/CD tools for GitHub Enterprise
  • Timely Migrations - Commercial tool to migrate TFVC with history between servers

• Work Tracking tracking migration tools

  • Migration Tools for Azure DevOps –  OSS tool to migrate Azure DevOps Work items between projects/instances
  • Jira to Azure DevOps work item migration tool - OSS tool to migrate Jira tickets to Azure DevOps Work items
  • OpsHub Migration Manager - Commercial migration tool that supports many work tracking toolsets

• CI/CD migration tools

  • GitHub Actions Import - GitHub CLI tool to convert CI/CD definitions from other toolset to another

Hope some of this is useful to anyone considering a migration.

I have been doing the regular maintenance in our Azure DevOps Pipelines of updating the versions of tasks. This usually means you perform one of the following actions

• Just increment the major version number in the YAML e.g task: MyTask@1 to task: MyTask@2 when there is a newer version of a task available.
• If the task is out of support and abandoned, swap to a different task, one that is still being supported, that does the same action.
• Fork the out of date task, and perform the updates to bring it back into support.
• Swap to a PowerShell/Bash script that wrappers a CLI tool to do the same action - increasingly my go to solution.

It is a shame the first option is often not possible, but don’t get me, or other MVPs on the long running subject of abandoned insecure Azure DevOps extensions

The Problem

The problem I hit today was with the Post Build Cleanup extension from Microsoft Premier Services. This had started reporting the warning

##[warning]Task 'Post Build Cleanup' version 4 (PostBuildCleanup@4) is dependent on a Node version (16) that is end-of-life. Contact the extension owner for an updated version of the task. Task maintainers should review Node upgrade guidance: https://aka.ms/node-runner-guidance

On checking the project repo, I saw that version 4 was the latest version of this task. There is an issue #263 that says the extension has no maintainer, and it is recommended you swap to using a PowerShell script. However, I wondered if the extension could be forked, and that is where it got interesting.

The problem is that the Microsoft Premier Services team choose not to open source the code for their extensions, the project repo is only used to track issues and store archived .VSIX packages. However, with Azure DevOps that is not a complete blocker, as you can:

1. Pick the option to get download the extension .VSIX package for Azure DevOps Server use in the Azure DevOps marketplace
2. Rename the .VSIX file to .ZIP
3. Expand the ZIP file into a folder and then you can have a look at the code

As I said, this is where it got interesting. When I looked on the task.json file I could see the task was setup to use any of three Node versions including Node 20

 "execution": {
        "Node10": {
            "target": "enableCleanup.js",
            "argumentFormat": ""
        },
        "Node16": {
            "target": "enableCleanup.js",
            "argumentFormat": ""
        },
        "Node20": {
            "target": "enableCleanup.js",
            "argumentFormat": ""
        }
    },

So why was I getting the warning about Node 16 when it should have been using Node 20?

The Answer

The issue is that the current Azure DevOps agents, and GitHub runners that use a related codebase, no longer ship with Node 20. If you looking a self hosted runners external folder you will see folders for

• node10
• node16
• node20_1
• node24

The task can’t find Node 20, as it is replaced by Node 20.1 with it’s numerous security fixes, so the runner falls back to use Node 16 and hence generates the version warning.

The fix, if you forked the extension, or had your own extension showing the same issues, is to change the execution block in the task.json to target Node 20.1

 "execution": {
        "Node10": {
            "target": "enableCleanup.js",
            "argumentFormat": ""
        },
        "Node16": {
            "target": "enableCleanup.js",
            "argumentFormat": ""
        },
        "Node20_1": {
            "target": "enableCleanup.js",
            "argumentFormat": ""
        }
    },

Once this is done Node 16 is not longer used

Summing Up

In this case swapping to the recommended clean-up template is the obvious answer, but it was an interesting exploration to find the root cause of this confusing warning.

Today, I had a failing integration test in a CI/CD pipeline, but all the tests passed locally. The failing test was looking for a certain number of rows returned from a API with known seed test data. I had revised the seed data and my test, but in my CI/CD pipeline my test was failing as it was still looking for the old number of rows.

Even though I went as far as inspecting the IL code in my test DLL, using dotPeek, to confirm the number of rows expected was the old incorrect value and not the new, it took me far too long to consider that my problem could be a misconfigured cache (again).

My issue was that the folder that the cache was checking was no longer being created due to changes in the pipeline unrelated to my tests. So the cache task could not find it’s target folder, so fell back to caching everything in the working directory. In effect reverting any changes made in the pipeline to the last successful run.

The fix, in this case, was to remove the cache as it was no longer required. However, the real moral of the story is to always consider any caching if your build artifacts are not being updated as you expect.

I recently had an issue trying to add a new Azure DevOps Pipeline Agent Pool to an existing Azure DevOps 2022 Server via the Team Project Collection Settings UI.

When tried to add the agent pool I got the error

Access denied needs Manage permissions to perform this action. For more information, contact the Azure DevOps Server administrator

The problem was that I was the Azure DevOps Server administrator

The Solution

The answer was to create the new Agent Pool at the Team Project level.

The difference when creating the new Agent Pool at the Team Project level is that though you can see the new Agent Pool at the Team Project Collection level, it is only usable in the Team Project it was created in.

For any other Team Projects, you have to add the newly created Agent Pool as an ‘Existing Agent Pool’ via the Team Projects > Settings > Agent Pool menu

So why did this work?

My assumption is that in one or more Team Projects in the Team Project Collection there are some custom permissions that deny access to the Agent Pool settings.

As adding a new Agent Pool at the collection level tries to add it to all existing Team Project, any such deny permission could cause the error I was seeing.

I have been chasing what it turned out to be a non-existent fault when trying to ingest test code coverage data into our SonarQube instance.

I saw my ‘problem’ in a .NET 8.0 solution with XUnit v3 based unit tests, this solution was being built using this Azure DevOps Pipelines YAML

 - task: SonarQubePrepare@7
    inputs:
      SonarQube: "SonarQube"
      scannerMode: "dotnet"
      jdkversion: "JAVA_HOME_17_X64"
      projectKey: "${{ parameters.sonarQubeProjectKey }}"
      projectName: "${{ parameters.sonarQubeProjectName }}"
      projectVersion: "$(GitVersion_Major).$(GitVersion_Minor)"
      extraProperties: |
        # Additional properties that will be passed to the scanner,
        # Put one key=value per line, example:
        sonar.cpd.exclusions=**/AssemblyInfo.cs,**/*.g.cs
        # Ingest the test results and coverage data
        sonar.cs.vscoveragexml.reportsPaths=$(Agent.TempDirectory)/**/*.coveragexml
        sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/**/*.trx

  - task: DotNetCoreCLI@2
    displayName: ".NET Build"
    inputs:
      command: "build"
      arguments: >
        --configuration ${{ parameters.buildConfiguration }}
        --no-restore
      projects: "$(Build.SourcesDirectory)/src/MySolution.sln"

  - task: DotNetCoreCLI@2
    displayName: ".NET Test"
    inputs:
      command: "test"
      projects: "$(Build.SourcesDirectory)/src/MySolution.sln"
      arguments: >
        --configuration ${{ parameters.buildConfiguration }}
        --collect "Code coverage"
        --no-restore
        --no-build

  - task: SonarQubeAnalyze@7
    displayName: 'Complete the SonarQube analysis'
    inputs:
      jdkversion: "JAVA_HOME_17_X64"

  - task: SonarQubePublish@7
    displayName: 'Publish Quality Gate Result'
    inputs:
      pollingTimeoutSec: "300"

At the start of the SonarQubeAnalyze@7 task log I could see that the .coverage file was found and converted into a .coveragexml file. However, there were multiple ‘The device is not ready’ errors when parsing this file later in the process.

Falling back on locating coverage files in the agent temp directory.
Searching for coverage files in E:\Agent\_work\_temp
All matching files: count=2
	E:\Agent\_work\_temp\a3b45684-38dd-4511-a371-1326896d8e57\BMAGENT2025_2026-01-20.16_40_35.coverage
	E:\Agent\_work\_temp\BMAGENT2025_2026-01-20_16_40_32\In\BMAGENT2025\BMAGENT2025_2026-01-20.16_40_35.coverage
Unique coverage files: count=1
	E:\Agent\_work\_temp\a3b45684-38dd-4511-a371-1326896d8e57\BMAGENT2025_2026-01-20.16_40_35.coverage
Converting coverage file 'E:\Agent\_work\_temp\a3b45684-38dd-4511-a371-1326896d8e57\BMAGENT2025_2026-01-20.16_40_35.coverage' to 'E:\Agent\_work\_temp\a3b45684-38dd-4511-a371-1326896d8e57\BMAGENT2025_2026-01-20.16_40_35.coveragexml'.
Coverage report conversion completed successfully.

... other analysis ...

INFO: Sensor C# Tests Coverage Report Import [csharpenterprise]
INFO: Parsing the Visual Studio coverage XML report E:\Agent\_work\_temp\a3b45684-38dd-4511-a371-1326896d8e57\BMAGENT2025_2026-01-20.16_40_35.coveragexml
WARN: Skipping the import of Visual Studio XML code coverage for the invalid file path: D:\a\_work\1\s\src\DotNetWorker.Core\StartupHook.cs at line 3085
java.io.IOException: The device is not ready

Initially my project contained no ‘real’ tests, just some placeholders as I was just getting the CI/CD process in place. These placeholder tests generated no significant test coverage, so I expected the coverage in SonarQube to be near 0%. However, I never bothered to actually check for any coverage value in SonarQube UI, as I assumed these errors in the log meant that coverage data was not being processed at all.

I now know this was not the case, but it took me a while to realise this.

The Exploration

On seeing the errors I started to try to isolate the issue.

• I created a small test project that had a .NET 8.0 class library project with some MSTests. Using the same YAML pipeline settings as above, the test coverage was imported without error.
• I next swapped from MSTest for xUnit, I could still import the coverage without errors

So the issue looked to be due to my codebase, not the basic tools in use.

An Internet search threw up that other people had seen similar issues and an answer was to gather the code coverage data using Coverlet as opposed to the built in dotnet test coverage tooling.

So I did that, and my errors disappeared the test coverage was imported. However, the downside of using Coverlet is that you loose the Coverage tab in the Azure DevOps pipeline summary view. This needs the binary .coverage format file.

As I did not want to loose the built-in coverage tab in the Azure DevOps pipeline summary view, I was back where I started, or so I thought.

It was then I realised that though I was seeing error in the log, I was also seeing a code coverage details in SonarQube. This was easier to seen now as while I had been fiddling tests had got added to the codebase, so I now had significant code coverage.

A Possible Explanation

I am not 100% certain of the problem, there is certainly an issue with SonarQube parsing .coveragexml files, the question is does it matter for my purposes?

I suspect the issue is my .coveragexml file contains coverage data for both my project assemblies and externally referenced ones. I can see this in the Azure DevOps pipeline summary code coverage view.

I think it is external assemblies that are causing the parsing issue, but these errors are not actually important to me as they concern files outside my projects codebase under analysis in SonarQube.

So, I would have saved myself a lot of time if I had noticed the import issues were warnings and not errors, and read the line after the warning block in the SonarQube log that told me what had actually been imported successfully.

WARN: Skipping the import of Visual Studio XML code coverage for the invalid file path: D:\a\_work\1\s\src\DotNetWorker.Core\StartupHook.cs at line 3085
java.io.IOException: The device is not ready

INFO: Coverage Report Statistics: 88 files, 58 main files, 58 main files with coverage, 30 test files, 0 project excluded files, 0 other language files.

The moral of the story is to read the whole of the log before jumping to conclusions, like you were told to do for exam questions at school!

I have some long standing PowerBI reports that I use for summarizing project data. They use a variety of data sources, including Azure hosted SQL instances. I recently moved the Azure hosted SQL databases to a new instance as part of a major tidy up of my Azure resources. This of course caused my reports to break.

I thought swapping the SQL connection details in PowerBI would be easy, and I guess it was, but it took me too long to work out how.

The main issue was that my reports were quick ‘hacked together’ reports, just pushed to my PowerBi personal workspace. I had never considered a formal publication with shared data connectors and gateways. This had come back to bite me.

The Solution

You can’t just change the dataset (now called the semantic view) in the PowerBI web UI, the docs say you should be able to, and I could for a newly created report, but not my very old ones! I guess they must be using some older schema?

After many false starts trying to use the web UI, I ended up using the PowerBi Desktop as follows

1. Open my PowerBI workspace in the browser
2. Download each of my reports as .PBIX
3. Edit the .PBIX in the PowerBI desktop app
4. Go to ‘Transform Data’ (on the top menu bar), a new Window appears
5. Go to ‘Data source settings’ (on the top menu bar), a dialog appears
6. Press the ‘Change Source…’ button, and enter the new connection details
7. When the ‘Change Source…’ dialog is closed you will see a banner message ‘Reauthenticate and reapply’. Press the button to start this process
8. Your local copy of the report should now be using the new data source
9. Save the updated .PBIX, I used a new name but this is not essential
10. Upload the revised .PBIX to your Power BI Workspace.

So I got there in the end. I am sure there are easier ways. I am just not familiar with PowerBI enough to know them (yet)

We have used SonarQube and the OWASP Dependency Checker Plugin for many years to perform analysis and vulnerability checking within our Azure DevOps Pipelines.

Recently, whilst picking up an old project for a new phase of development, I came across a couple of problems due to changes in both tools since the project CI/CD pipelines were last run.

• The OWASP Dependency Checker vulnerabilities were not appearing in SonarQube as issues
• The OWASP Dependency Checker HTML report could not (always) be loaded in SonarQube

The issues were just down to changes in both tools over time. It just goes to show that you can’t just setup a CI/CD system and expect it work forever, changes are always being introduced in cloud based tools.

The Solution

Changes in OWASP Dependency Checker supported output formats

The reason the OWASP Dependency Checker vulnerabilities were not appearing in SonarQube as Issues was because the data file format was incorrect. We were exporting XML, historically correct, when JSON is now required.

The OWASP Dependency Checker Plugin changed the way it handles importing issues in the PR Remove depreacted XML-Report Parser #755. This means the following changes are required

• Get the OWASP Dependency Checker Plugin to produce JSON and HTML reports, as opposed to XML and HTML
• Update the OWASP Dependency Checker Plugin to use the currently supported parameter ie. swap sonar.dependencyCheck.reportPath to sonar.dependencyCheck.jsonReportPath

Viewing HTML Dependency Reports

The issue with viewing the OWASP Dependency Checker HTML report was due to size. If an HTML report that is too big has been uploaded to any branch/PR in a SonarQube project, you get Heap memory issues when you try to access the SonarQube project. This will persist until the branch/PR is removed. This will probably occur due to SonarQube’s housekeeping automation, inactive branches/PRs are removed after 30 days.

You could of course fiddle SonarQube container settings to added memory/performance, but the simple answer is to just not upload the HTML reports to SonarQube.

As long as the JSON report is uploaded all the vulnerabilities are presented within SonarQube system as issues, and as we attach the HTML report as a pipeline artifact it is still available if needed in Azure DevOps.

So the simplest answer is to not define, or comment out, the sonar.dependencyCheck.htmlReportPath SonarQube Dependency Check parameter.

The Final Pipeline

The pipeline I ended up with is as follows

 - task: SonarQubePrepare@7
    inputs:
      SonarQube: "SonarQube Service Connection"
      scannerMode: "dotnet"
      jdkversion: "JAVA_HOME_17_X64"
      projectKey: "MPK"
      projectName: "My Project Name"
      projectVersion: "$(GitVersion_Major).$(GitVersion_Minor)"
      extraProperties: |
        # Additional properties that will be passed to the scanner,
        # Put one key=value per line, example:
        sonar.dependencyCheck.jsonReportPath=$(Build.ArtifactStagingDirectory)/vulnerabilityscan/dependency-check-report.json
        # Comment out the HTML report path to avoid SonarQube heap memory issues
        sonar.dependencyCheck.htmlReportPath=$(Build.ArtifactStagingDirectory)/vulnerabilityscan/dependency-check-report.html
        sonar.cpd.exclusions=**/AssemblyInfo.cs,**/*.g.cs
        sonar.cs.vscoveragexml.reportsPaths=$(Agent.TempDirectory)/**/*.coveragexml
        sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/**/*.trx

  - task: DotNetCoreCLI@2
    displayName: ".NET Build"
    inputs:
      command: "build"
      arguments: >
        --configuration ${{ parameters.buildConfiguration }}
        --no-restore
      projects: "$(Build.SourcesDirectory)/src/MySolution.sln"

  - task: DotNetCoreCLI@2
    displayName: ".NET Test"
    inputs:
      command: "test"
      projects: "$(Build.SourcesDirectory)/src/MySolution.sln"
      arguments: >
        --configuration ${{ parameters.buildConfiguration }}
        --collect "Code coverage"
        --no-restore
        --no-build

  - task: CodeCoverage-Format-Convertor@1
    displayName: "CodeCoverage Format Converter (to allow import to SonarQube)"
    inputs:
      ProjectDirectory: "$(Agent.TempDirectory)"

  # set the version of JAVA required by the dependency-check-build-task
  - task: JavaToolInstaller@0
    inputs:
      versionSpec: '17'
      jdkArchitectureOption: 'x64'
      jdkSourceOption: 'PreInstalled'

  - task: dependency-check-build-task@6
    displayName: "Run OSWAP Vulnerability Scan"
    inputs:
      projectName: "${{ parameters.sonarQubeProjectName }}"
      scanPath: "$(Build.SourcesDirectory)/src/**"
      format: HTML, JSON
      reportsDirectory: "$(Build.ArtifactStagingDirectory)/vulnerabilityscan"
      uploadReports: false # false we publish the reports as an artifact with a name of our choice
      additionalArguments: >
        --nvdApiKey "${{ parameters.nvdApiKey }}"

  # Not needed if uploadReports: true above
  - task: PublishPipelineArtifact@1
    displayName: "Publish Vulnerability Scan Report"
    inputs:
      targetPath: "$(Build.ArtifactStagingDirectory)/vulnerabilityscan"
      publishLocation: "pipeline"
      artifactName: "${{ parameters.artifactName }}VulnerabilityScan"

  - task: SonarQubeAnalyze@7
    displayName: 'Complete the SonarQube analysis'
    inputs:
      jdkversion: "JAVA_HOME_17_X64"

  - task: SonarQubePublish@7
    displayName: 'Publish Quality Gate Result'
    inputs:
      pollingTimeoutSec: "300"

In Summary

It just shows you have to budget some time in keeping you CI/CD automation up to date on any project.

The Problem

I had a working pipeline that as part of its build process ran the OWASP Dependency Checker task. This can be slow to run as it has to download the current vulnerability database. To try to speed my builds I have been using the cache task to cache the current pipeline run’s downloaded vulnerability database, so on the next run the vast majority of the database is already downloaded.

The pipeline YAML is as follows

steps:
- task: PowerShell@2
    displayName: Find the NVD DB path to cache
    inputs:
    targetType: inline
    script: |
        $nvdcachepath = $(get-childitem "$(Agent.WorkFolder)\_tasks\dependency-check-build-task*\*.*.*\dependency-check\data").FullName
        echo "##vso[task.setvariable variable=nvdcachepath;]$nvdcachepath"

- task: Cache@2
    displayName: Cache NVD data
    inputs:
    key: '"NVDCache" | "$(Agent.OS)"'
    path: $(nvdcachepath)
    restoreKeys: |
        NVDCache | "$(Agent.OS)"
        NVDCache

# other build tasks

- task: JavaToolInstaller@0 
    displayName: Install Java needed for the Dependency Check
    inputs:
    versionSpec: '11'
    jdkArchitectureOption: 'x64'
    jdkSourceOption: 'PreInstalled'

- task: dependency-check.dependencycheck.dependency-check-build-task.dependency-check-build-task@6
    displayName: Dependency Check
    inputs:
    projectName: Identity Server
    scanPath: CCC.Web.IdentityAndSSO.IdentityServer
    format: HTML,XML
    additionalArguments: --nvdApiKey $(nvdapikey)

This has all been working well, but the YAML has a potential flaw, can you spot it?

The issue appeared when, due to the server that provides the OWASP vulnerability database being unavailable the dependency check task was temporarily disabled.

This meant that the dependency check task was not downloaded, so the Powershell script to find the vulnerability database folder returned an empty string as it could find no matching folder.

This is important as it is this string that controls what is cached. With no folder passed into the cache task, the whole of the pipeline working folder is cached. In effect the restoring of the cache is checking out the source as it was on the last successful build on the branch. So you are not building the commit you think you are, but an older commit, the source code that was cached.

The Solution

There are a number of options to avoid this problem

1. Don’t use the Cache task
2. Don’t disable any task that produces folders your Cache task is meant caches
3. If you disable a task who’s data you cache also disable the Cache task
4. Put some logic around the cache task e.g

   - task: Cache@2
     condition: and(succeeded(), ne(variables['nvdcachepath'], ''))
   

All are valid for different scenarios, the choice is down to your use case.

The overall point is make sure you are caching what you think you are caching, if not constrained the Cache task will cache everything it can.

Updated 30-Oct-2025 Added more details on Screen Refresh Rate and Teams Room App

The Problem

We have owned a Surface Hub v1 for a number of years, and it has served us well. However, with Microsoft ending support for Windows 10 it was in danger of becoming a large piece of sculpture in the office. This is not just because we did not want to run a Windows 10 device when security patches were not available, but that the embedded version of Teams would not even load.

In theory the answer was easy, flip a switch on the Surface Hub so it uses an external replacement PC as opposed to it’s built in one. However, this process turned out to be harder than we expected as the Surface Hub is very picky over hardware settings.

The Solution

The whole process is very well documented by Ryan Wold, so I won’t repeat the details. However, I will mention the kit and settings we ended up using as we found getting any output on the Surface Hub meant using exactly the correct settings.

We aimed to use spare kit we had around, so our replacement PC was an old Lenovo M910s desktop PC. We picked this particular unit as we had at some point in the past fitted it with an Nvidia Geforce GTX 750Ti as we knew that a dedicated GPU would be needed to drive the 4K screen.

The consistent issue we had trying to get this working was that the PC would boot, we would see the BIOS boot screen, Windows would start to load, but as soon as the loading Windows spinner appeared the Surface Hub screen would go blank. If we had a second display attached, we could see that Windows could not detect the Surface Hub as a display.

The critical item it turned out was the display one, this needed to be an active cable. We ended up with a BENFEI HDMI to DisplayPort Adapter. Once we used this some of our display issues were sorted, Windows could detect the Surface Hub consistently.

The other critical step was that we had to set the resolution to 4K (3840 x 2160) and specifically with the refresh rate set at 30Hz. If we had a more powerful GPU we think we could have run at the default rate of 120Hz, but that was not an option with the kit we had.

30Hz is not the default refresh rate when the Surface Hub is detected as a monitor, so we got the blank screen. To change the refresh rate we had to have a second display attached. This is not a problem you might think, make the edit and remove the second display. However, Windows stores the refresh rates separately for each screen setup e.g. single display, dual display etc. This meant when we disconnected the second display the refresh rate for the Surface Hub returned to its 120Hz default.

So for now we need to keep the second display attached

But not all the setup was so hard, getting the Surface Hub cameras, audio devices, touch screen and pens working was easy. All that needed was a USB cable and the right drivers

Sorting the Refresh Rate

The solution to the refresh rate issue was to use the CRU Display Utility.

With the PC running with with dual monitors:

1. Download, unzip and run the the CRU Utility
2. Selected the Surface Hub (active) display and altered the detailed resolutions order. You need the 30Hz option to be the first entry in the list
3. Repeat Step 2 for any other Surface Hub entries in the list (not sure if this step is essential, but quick to do)
4. Pressed Save and exited CRU
5. This is the important step we initially missed Use the CRU provided restart.exe tool to restart the graphic subsystem and hence read the changed order.
6. I could then disconnect the 2nd monitor, the display a rescan was done, but unlike in the past the working 30Hz refresh rate was picked.

The PC could not be rebooted, with a single monitor, and it still worked.

Trying to Setup the Team Room Client

It is good that we can now use the Surface Hub for Teams calls again, but having to login to start Teams is not as nice as being able to invite the device as location to meetings.

To try to make the device a Teams Room we download the client from the Windows App Store where it is still confusingly call the ‘Skype Team Room’. I guess it must be hard/impossible to change the name of a published package in the store. However, it is the correct one as the release history shows this app being recently updated and has the current Teams branding.

When you install and run this application it initially looks OK, but then fails saying

Microsoft Teams Room device isn’t certified

You must insert the HDMI adaptor that came with this device

You have the option to restart the application, but you get the same result.

The bottom line is that this application seems to be locked down to certified hardware, there is no option for a DIY ‘build your own unit’, or at least not one I have been able to find thus far.

Where we are now

So now we have a working replacement PC connected to Surface Hub. This PC is domain joined, so staff can login for Teams meetings and use all the physical features of the Surface Hub. This is not a perfect replacement, but it at least means the Surface Hub is not just E-Waste.

I will post again, if we have any more successes

I have previously posted about settings up Snipe IT on Azure using a Docker container running in an Azure WebApp with a MySQL DB.

I recently had to revisit my setup as the site was failing to load with a DB connection failure. The issue was that the SSL certificate chain used by Microsoft for their hosted MySQL service had changed and my settings needed to be updated

The Solution

1. In te Azure Portal find your MySQL instance and select the networking blade, click the ‘Download SSL Certificate’ link, you are taken to learn.microsoft.com . Download the DigiCert Global Root G2 certificate and the Microsoft RSA Root Certificate Authority 2017 certificate
2. You need to combine these certificates i.e. put the base64 coded versions of each in a single text file.

-----BEGIN CERTIFICATE-----
(Content from DigiCertGlobalRootCA.crt.pem)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Content from DigiCertGlobalRootG2.crt.pem)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Content from MicrosoftRSARootCertificateAuthority2017.crt.pem)
-----END CERTIFICATE-----

3. SO, I downloaded the DigiCertGlobalRootCA.crt.pem file I was using previously from my Snipe IT’s Azure File Storage and opened it in a text editor (the first block)
4. I then appended the contents of DigiCertGlobalRootG2.crt.pem to this file (the second block)
5. Before I could do the same with the Microsoft RSA Root Certificate Authority 2017.crt file I had to import the .CRT file onto my local Windows machine, then export it is a Base64 encoded file. Once this was done, this too could be add to the DigiCertGlobalRootCA.crt.pem file (the third block)
6. Once all the edits were made I save the edited file and uploaded it into the Azure Storage to overwrite the old version. This avoid the need to edit any other configuration settings that referenced the certificate file name.

Once this was done my Snipe IT instance loaded as expected.

The recent addition to Azure DevOps of the StringList parameter type can be really useful to dynamically create parallel stages or jobs in an Azure DevOps YAML pipeline.

A StringList parameter can be used to present a list of values to the user queuing a pipeline run, thus allowing the selection of one or more values that can be accessed using a YAML expression loop.

This can be combined with YAML templates, where the StringList is passed into a template as an Object. Noting that you can’t use the StringList type for a parameter type in the template definition. However, this is not a problem as there is a dynamic conversion from StringList to Object e.g.

# Azure-Pipeline.yml
parameters:
  - name: environments
    type: stringList
    default: 
      - dev
    values:
      - dev
      - test
      - prod

trigger: none

extends:
  template: template.yml
  parameters:
    environments: ${{parameters.environments }}

# template.yml
parameters:
  - name: environments
    type: object

stages:
  - stage: Deploy
    pool:
      vmImage: ubuntu-latest
    jobs:
      - job: DeployJob
        steps:
          - checkout: none
          - ${{ each value in parameters.environments }}:
            - script: |
                echo "Selected environments: ${{ value }}"

The Problem

Whilst implementing this new parameter type on an existing pipeline we got a very strange set of error messages ‘Unexpected a mapping’, ‘Unexpected at least one-value pair in the mapping’ and ‘Unexpected state while attempting t read the mapping end state’

After much experimentation we found the issue was the indenting of a steps block later in the YAML. When the new loop expression was added only part of the jobs block was correctly indented.

The indenting was

  jobs:
  - ${{ each environment in parameters.Environments }}:
    - job: build_${{ environment }}
    steps:
      ...

When it should have been

  jobs:
  - ${{ each environment in parameters.Environments }}:
    - job: build_${{ environment }}
      steps:
        ...

So as usual when you get an unexpected and unclear error in Azure DevOps YAML, assume the problem is due to whitespace.

A while ago I posted about starting to use Home Assistant. This has been working well for monitoring our SolarEdge PV and Battery system, controlling if we should charge the battery with cheap overnight power if there has not been enough sun to fully charge the battery.

Bitten by the Home Assistant bug, I decided I wanted to add more sensors to the systems, so bought some cheap Zigbee equipment from AliExpress

• SONOFF Zigbee Temperature and Humidity Sensor | SNZB-02P
• SONOFF Zigbee 3.0 USB Dongle Plus | ZBDongle-P

The installation was easy using ZHA, the gateway dongle and sensors were all found. However, the sensor readings I got were strange. On startup a reading was received, this stayed as a flat line for 6 hours then nothing for 6 hours then a new, usually different, value for 6 hours, and on and on.

After looking at forums it seems that I am not alone in having this problem with SONOFF kit, and nothing seemed to fix the problem. The general consensus seemed to be to avoid the brand.

Well, yesterday they fixed themselves, or rather the sensors got a firmware update delivered automatially via ZHA from 0x00002100 to 0x00002200 that seems to have fixed the problem. As the chart at the top of this post shows I now have the type of data I was expecting.

So if you are seeing similar problems with SNZB-02P sensors, look out for a firmware update

Note that this is a repost of my LinkedIn article of the same name

The changing face of DevOps

Over the years the style of DevOps consultancy I have done has changed, along with its name. Changing from the simple ‘source control’, to SDLC, then ALM to now DevOps.

Back in the days of Team Foundation Server (TFS), I used to do a lot of on-premise installs and upgrades. Getting our customers up and running, and helping them improve their adoption of the tools.

With the move to cloud services the profile of this work changed to increasingly helping customers migrate their tooling and processes to the cloud. Specifically doing TFS/Azure DevOps Server to Azure DevOps Services cloud migrations, and then helping them get the best out of this new platform.

Today the profile has changing again. The majority of our clients have already moved to the cloud, so much of my engagements are focused on best practice usage of the tools, and the day to day development lifecycle. As opposed to installation, upgrade and migration.

That does not mean I don’t see installation, upgrade and migration projects in both the Azure DevOps and GitHub space, but certainly not in the volumes I used to.

Who is left on premises and why?

Now we seem to be past the peak of cloud migrations, most clients who want to make the move have moved. Remaining on-premise we now have two broad groups:

• Companies with a governance reason, or maybe just a perceived reason, to stay on-premises. It is worth noting that I do see clients who assume a governance restriction, but don’t check with their regulatory/auditor body for current guidance to see what their options are. They are often surprised that the cloud, when used securely, is a viable option for them.
• Companies who have neglected their on-premise server

It is the latter I want to talk about in this article.

Falling into the gap between IT and Development

I have often seen that it is easy for a TFS/Azure DevOps Server to fall into a gap between the IT and Development teams. Wrongly, the server is not considered a business-critical server to be managed by IT, and not treated with rigor by the Development team as it would be by an IT team.

Because of this, I still see a number of old TFS/Azure DevOps Servers that could well have not been patched since their installation, maybe over a decade ago. The only reason they are being looked at now is the underlying Windows Operating Systems and SQL versions are reaching end of life and company wide IT audits are forcing changes to be made.

I have long said that for the on-premises TFS/Azure DevOps Server you should, on top of the usual OS patching, be looking at patching DevOps at least every 3 to 6 months. This is because this is the rough release cadence that features from the cloud version of Azure DevOps Services get packaged and made available to the on-premise version. These updates include not just new features, but also critical security patches.

It is common that in customers who have not patched their server that they have also not kept their development practices up to date. Often lacking CI/CD processes, which I feel are the core of modern DevOps good practice, the heartbeat of the process, and are commonly still using older source control patterns e.g. still on TFVC source control. This is not in itself wrong, but potentially limiting as modern practices and the skillsets of newer/younger developers are going to be based around the de-facto standard of Git.

But my upgrade is too complex

The issue with upgrades for such old systems is that there is often no direct path to the current version of Azure DevOps Server. Upgrades may require multiple steps, using temporary intermediate environments to address the complexities due to the limitations of supported SQL and OS versions that underpin TFS/Azure DevOps Server.

A common question from clients is ‘how long is this upgrade going to take’? A very difficult one to answer as everyone’s systems are different. You can think of the time required for this upgrade as taking the time for all the smaller updates you should have done over the life of your server and bundling it up into one massive job.

The majority of the time in a TFS upgrade is related to SQL operations, backups, copies, restores and schema/data updates. The sheer volume of data to be moved is usually the limiting factor. Anything involving potentially Terabytes of data takes time.

For major upgrades such as this, I always recommend a dry run to make sure the process is understood, timings known and to give the client’s staff the opportunity for training and adoption of the new tooling available, such as Git and modern CI/CD.

But you can’t get away from the fact that you will probably need an amount of downtime, while your old TFS server is not available, before the new Azure DevOps Server is ready.

But can’t I jump to the cloud now?

Once I start this form of major upgrade engagement it is common for clients to ask “can’t I just bypass all this and go to the cloud?”. Of course the answer is ‘yes, but…’.

If they wish to do a full fidelity migration to Azure DevOps Services then they still have to get to the current version of Azure DevOps Server as the migration start point. So they cannot avoid the on-premise upgrades.

If they don’t need a full fidelity upgrade, or are considering a move to another toolset such as GitHub Enterprise, there are more options that do not require an update to the on-premise server, but these come with constraints and compromises, as you would expect whenever you swap toolsets.

So are you in this position?

The moral of the story is don’t neglect your DevOps tools, a craftsman does not let their tools get rusty and neither should you.

If you are not keeping your DevOps tool chain up to date you are hampering the efficiency and potentially the recruitment/retention of your developers as they look to work on projects using modern tooling.

So, if this sounds like your situation, why not get in touch with me or Black Marble to discuss your DevOps options?

I have recently done a couple of streamable events with our friends at Grey Matter. Both are now available to enjoy on demand:

• Grey Matter Talks Tech podcast ‘Inside GitHub: The Platform Powering Open Sources
• Grey Matter’s on-demand Webinar ‘Inside the Developer’s Toolkit’ with JetBrains and SmartBear

If these sort of subjects are on interest, why not come to an in person event?

I am speaking, as may other great speakers including my colleague Andrew Wilson, at Grey Matter’s Tech Summit in London on the 24th of September, and best news of all it is free.

Hope to see you there

I recently rebuild my trusty Lenovo X1 Carbon laptop. As you usually find, a fresh install of Windows 11 meant a leap in performance.

All was good for a couple of weeks, until I started to get problems. It still was working to supply power and the HDMI connection to my external monitor worked, but the USB devices could not be seen.

I also checked with the Lenovo Dock Manager and it could now no longer see the dock.

The Solution

I am not sure why it happened, but the fix was

1. Open Device Manager: Press the Windows key + X and select “Device Manager”.
2. Show Hidden Devices: Click “View” in the menu bar and select “Show hidden devices”.
3. Locate Thunderbolt Controller: Expand “System Devices” and find the “Thunderbolt Controller” entry.
4. Uninstall: Right-click on “Thunderbolt Controller” and select “Uninstall device”. In the “Uninstall Device” window, check the box that says “Delete the driver software for this device”.

Once this was done I disconnected the dock and reconnected it, and this dialog appeared

I had to approve access to the dock.

My guess is that somehow this access control setting had got set in the wrong state. I have no idea how, as I was never shown this dialog after the Windows reinstall, but at least it is fixed now

The GitHub ‘Azure Pipelines’ App enables some really interesting ‘better together’ scenarios mixing the usage of Azure DevOps Pipelines for CI/CD processes while your source is stored on GitHub. It is particularly useful if an enterprise is migrating towards GitHub Enterprise over a period of time, maybe using GitHub Enterprise Importer (GEI) to do the migration.

I was recently working on such a migration for a client that involved numerous Azure DevOps organisations. In the past someone had setup the GitHub ‘Azure Pipelines’ App on their GitHub organisation and got it working with one of their Azure DevOps organisation.

The question was how do you setup a GitHub App based service connection in their other Azure DevOps organsations, without breaking what was already installed?

Service Connections & GitHub Authentication

The way Azure DevOps Pipelines communicate with GitHub is via a Service Connection.

To create a service connection you goto Project Settings > Pipelines > Service Connections then press the New Service Connection button, picking the service you require.

When you do this and pick the GitHub option you are offered two ways to authenticate

• A PAT - linked to a specific user and will expire after a fixed period
• oAuth - again linked to a specific user, the one creating the service connection, but will not expire, so the preferred solution

But neither of these are authentication via the GitHub App. So the question is how do you create a service connection of that type?

Solution

It turns out adding the ‘right type of service connection’ is easy, but really poorly documented. Every document seems to point to re-installing the Azure Pipelines App in GitHub, this is not required.

The process is as follows, assuming the Azure Pipeline GitHub App is installed and configured in your GitHub organisation.

1. In a Team Project on the Azure DevOps organisation where you need the service connection create a new pipeline
2. Pick GitHub as the source location. You maybe prompted to authenticate, and then will be given a list of repositories to pick from, pick one, the choice of which one is not important.
3. Once this is done you are presented with a list of pipeline templates to use, at this point you can cancel the creation of the pipeline, completing it is not required.
4. If you look in Project Settings > Pipelines > Service Connections you will see a new Git Service Connection has been created with the name of the Azure DevOps Team Project that has the authentication method GitHub App, exactly what we require.

In my scenario I was then able to rewire all the pipelines in my Azure DevOps organisation to point to GitHub using the GEI command gh ado2gh share-service-connection to share the new service connection to other Team Projects, and then gh ado2gh rewire-pipeline to change the source location.

In Azure DevOps Pipelines there is a feature to dynamically configure the build number e.g. at the top of a YAML build add the following to get a version in the form 1.2.25123.1

name: $(major).$(minor).$(year:YY)$(dayofyear).$(rev:r)

• Major & Minor are user created and managed variables
• Year, Dayofyear and Rev are built in Azure DevOps pre-defined variables. The first two are obviously based on the current date, the rev is the number of times a pipeline has been run on the current date.

The problem is that there is no equivalent feature to name in GitHub Actions as build names/numbers are not really a thing in Actions.

However, there are use cases where you want to use some or part of this dynamic build number for other purposes such as version stamping a file.

The Solution

The date related parts of the build number are easy to recreate with PowerShell get-date,

The revision takes a bit more work, but can be done with more PowerShell and the GitHub CLI.

Over the years, I have been involved in numerous migrations of TFS/Azure DevOps Server from on-premises servers to the cloud hosted Azure DevOps Services using the Microsoft Azure DevOps Data Migration Tool. As long as you carefully followed the instructions, the process was relatively straightforward.

I was recently involved in a cloud migration of a set of non-English Azure DevOps Servers i.e. the Azure DevOps Servers had been installed in French not English. Migrating these servers to Azure DevOps Services proved a challenge.

The Problem

The core of the problem with this type of migration is that the target system, Azure DevOps Services, is only available in English.

At this point it is important to explain what is meant by the previous statement. Azure DevOps Services allows you to set a language preference, which you set in your user profile, but the underlying database is always in English. This means that if you have a non-English Azure DevOps Server, where the database is non-English, the migration will have problems.

This problem becomes apparent when you perform the first step of using the Azure DevOps Data Migration Tool, the migrator validate command to validate your TPC (Team Project Collection) prior to migration.

Note Remember you migrate a TPC, not a whole Azure DevOps Server, which could have many TPCs

This command checks the source Azure DevOps Server TPC for issues that will block a migration. In the case of my Non-English server TPC the validation returned many issues.

The first was a warning about the database collation being French_CI_AS, so any migration could result in some characters not being migrated correctly e.g. Non-English characters like ‘é’ might become equivalent to the English ’e’ after migration. However, this was just a warning, and not an error, so can be ignored for now and reviewed as part of the planned dry-run of the migration.

The bigger problem was the many errors in the form

VS403443: In order to migrate successfully, you must rename field 'System.Title' to 'Title'. Given name for 'System.Title' is 'Titre'

When you do these migrations there are often VSxxxxxx or TFxxxxxx errors, especially on TPCs that have been upgraded through various versions of TFS and Azure DevOps over the years. These are usually fixable by editing the process template as discussed in the migration documentation.

However, when I tried to fix these issues I saw two completely separate sets of problems, depending on whether the TPC was using the older XML Process Templates or the newer Inherited Process Templates.

When the TPC is using XML Process Templates

When the migration was for a TPC using the older XML Process templates, I found that once I had fixed the VS403443 errors using commands like

witadmin.exe changefield /collection:http://server:8080/tfs/defaultcollection, /n:System.Title /name:"Title" /noprompt

When I reran the migrator validate command, I got a new set of errors, but they were the same errors as before, just with the field names reversed e.g..

VS403443: In order to migrate successfully, you must rename field 'System.Title' to 'Titre'. Given name for 'System.Title' is 'Title'

I was in an endless loop of errors.

When the TPC is using Inherited Process Templates

When the migration was for a TPC using the newer Inherited Process Templates, the problem was that I could not use the witadmin tool. There is in fact no supported way to update the display name for built-in work item types fields.

This is because the Inherited Process Templates are stored in the database, and not as XML files.

The Solutions

We have to consider the solutions to these problems separately.

XML Process Templates

After much unsuccessful experimentation, I reached out to the community to see if anyone had a solution to the problem and a fellow DevOps MVP, Neno Loje suggested he had had success in the past migrating TPCs originally setup in German, but the process he used came with some constraints.

The process I ended up with, based on his suggestions, was based around migrating the Non-English Team Project Collection (TPC) to an temporary English Azure DevOps Server:

1. Build a new Azure DevOps Server, making sure you pick the default language English
2. Detach the TPC to be migrated on the Non-English Azure DevOps Server and make a SQL Backup of the detached TPC Database on the Non-English Azure DevOps Server

   Note Once the backup is done, you will usually reattach the TPC on the source Non-English server so users can continue to work while you do the dryrun migration

3. Restore the TPC SQL backup on the new English Azure DevOps Server and attach the exported Non-English TPC to the new English Azure DevOps Server
4. On the new English Azure DevOps Server run the migrator validate command to check for errors, and fix them. This should be possible without getting into the error loops

   • Update all WIT fields causing the VS403443 errors, rename display names to the English names as discussed above using the witadmin command)

     Note I wrote this PowerShell script to automate the process of generating the commands to do the renaming of fields in a process template based on the migration error log.

   • Fixing any remaining errors in the migration should be possible with further XML template file edits which are published using the witadmin command as discussed in the migration documentation

     Note Watch out for any for=[group] and not=[group] attributes in the WITD XML files, used to conditionally set field constraints e.g. make a field readonly for all users except those in a specific group. These constraints are not supported in Azure DevOps Services, but are not always picked up by migrator validate. If they are not removed, they will cause the import to fail. An import failure email message will be recieved via email in form

     Step : ProcessValidation - Failure Type - Validation failed : Invalid process template: WorkItem Tracking\TypeDefinitions\Bug.xml:: Object reference not set to an instance of an object.

   • You can also look at using the ConformProject.ps1 script to help with this process. ConformProject will take a defined process template that is on your local machine and apply it, in full, to a specified project. This is a good way to get rid of all the errors, but you will lose all your customisations, so use with care.

5. Once all the errors are fixed, the validation should pass, and hence the rest of the migration process should be possible

However, the fact your migration completed might not be the end of your issues. There is of course the potential issue of the collation of the database, but arguably this is a minor issue. The more important point is something Neno had mentioned. He commented that this process is good for a read-only migration to Azure DevOps Services, but if you wish to continue using the migrated work items more work is required.

The reason for this is not the migration of the process template, it’s the actual work item data. The imported work items are still using strings for state and other fields in the non-English language and need to be “converted” to values that match the constraints of the ‘fixed’ english process template.

To fix this means edits are required for all active work items. This of course could be done one-by-one as Work Items are edited, or via scripts to make the changes at scale, but more the work will be required to make the work item data usable in Azure DevOps Services.

Inherited Process Templates

—— DANGER ZONE ——

The following section describes a process that is not supported by Microsoft and could lead to serious issues with your Azure DevOps Server.

The process should only be used on a copy of a system, not on a live system itself.

I accept no responsibility for any issues that arise from following this process, you follow it at your own risk.

When the migration was for a TPC using Inherited Process Templates, the solution is very different, and can only be described as a massive hack.

There are two reasons for the need for a different process

• With inherited process templates the built-in work item types are by design read only. Your customisations are additions to the built-in work item types. You cannot change the built-in work item types, you can only hide or add new fields to them. There is no equivalent of the witadmin command to change the display names of fields in the Inherited Process Templates, all changes are done via the Azure DevOps UI or REST API.
• The other reason for the different process is that it is not possible to attach a Non-English Inherited Process TPC to an English Azure DevOps Server. Any ‘fixing’ process must be done on a Non-English server.

So, the process I ended up with is as follows:

1. IMPORTANT Create a duplicate of your Non-English Azure DevOps Server, this is the one you will use for the migration process. This step is very important, as you will be making SQL edits to the database which is very unsupported.
2. On the duplicate Non-English Azure DevOps Server load SQL Server Management Studio (SSMS) and connect to the TPC database.
3. For each VS403443 error run the following SQL command to update the display name of the field in the database:

   UPDATE [dbo].[WorkItemTypeField]
   SET [Name] = 'Title',
       [ReportingName] = 'Title'
   WHERE [ReferenceName] = 'System.Title'
   

   Note Again I wrote a PowerShell script to automate the process of generating the SQL commands to do the renaming of fields in a process template based on the migration error log.

4. Rerun the migrator validate command to check for any remaining errors, and fix them.
5. Once all the errors are fixed, the validation should pass, and hence the rest of the migration process should be possible

Again the fact your migration completed is not the end of your issues, in fact the issues are much worse than with the XML Process Templates.

The reason for this is that the Non-English inherited process template tries to match all your work items to the built-in work item types based on the string of the work item type name. So for example, it will try to match the work item type ‘Bogue’ in your French team project to the built-in work item type ‘Bug’, which will of course fail.

This is a serious problem, as it means much of the Azure Boards and Azure Test Plans UI will fail to load any work items, if they even load at all. The work item data is there, you can see it via the REST API e.g. https://dev.azure.com/{organization}/{project}/_apis/wit/workitems/{id}, but the UI will not work as the work item type names are not what the English based Azure DevOps Services instance is expecting.

In theory there must be a way to fix this, but it will require a lot of SQL edits prior to migration to Azure. Far more work than I have the appetite to do.

So in summary, you have to consider this process for migrating Non-English inherited process based TPCs to only be acceptable if your key goal is to migrate your TFVC repositories with no data loss to Azure DevOps Services, and that you are prepared to accept that the work item data will not be usable.

Your only real option if you need the work item data from a Non-English on-premises Inherited process TPC in Azure DevOps Services is to look at using Martin Hinshelwood Azure DevOps Migration Tools to migrate the on-premises work items to a new Azure DevOps Services project on the migrated Azure DevOps Service instance, one that is setup using an English based inherited process template. Given the need to address translation issues during this migration this will also be a not insignificant undertaking.

Conclusion

Migrating a non-English Azure DevOps Server to Azure DevOps Services is possible, but it is not officially supported and requires some extra steps, some of them very ‘hacky’.

If you are considering either of the migration paths in this post, I would recommend in the strongest terms that you perform your migrations on a duplicate of your Azure DevOps Server and not on your live system.

I cannot stress enough, please DO NOT make SQL edits of your live Azure DevOps Server database, as this is not supported and could lead to serious issues with your live system.

I got asked today by a client if there was a way to automate the exporting Azure DevOps Test Plans to Excel files. They knew they could do it manually via the Azure DevOps UI, but had a lot of Test Plans to export and wanted to automate the process.

The Options

I considered a few options:

• TCM CLI - This is a command line tool that can be used to interact with Azure DevOps Test Plans. It can be used to import or clone Test Plans, but not to export them.
• AZ DEVOPS CLI - This is a command line tool that can be used to interact with Azure DevOps. Unfortunately, it does not have any commands to export Test Plans.
• Azure DevOps REST API - This is a powerful API that can be used to interact with Azure DevOps, but the documentation makes no mention of a call to export Test Plans.

But…..

The Solution

I used a process I have used before to write tools for Azure DevOps. I opened the page in the Azure DevOps UI that I wanted to automate, and then used the browser developer tools to inspect the network traffic. I found a call that was made to export the Test Plan to Excel, and then used that as the basis for my tool.

Turns out there is an undocumented API call that can be used to export Test Plans to Excel. A POST call is made to the following URL:

https://dev.azure.com/{organization}/{project}/_apis/testplan/TestCases/TestCaseFile?api-version=7.1-preview.1

with a payload that looks like this:

{
  "testPlanId": 1,
  "testSuiteId": 2,
  "testCaseIds": [3, 4, 5],
  "columnOptions": ["System.Id","System.AssignedTo","System.State"]
}

The Code

So I needed to write a tool get all the test plans and test suites in an Azure DevOps Team project, and then call the API to export them to a set of Excel files.

I usually write this type of tool in PowerShell, but try as I might, even though I appeared to get the expected data back from the API call, I could not write it to disk in such a way Excel could read it. Excel said the file was corrupt.

As I had delivered a webinar today on using GitHub Copilot to write code, I thought I should practice what I preach and use it to fix the tool. As I suspected the issue was not the API call, but the PowerShell encoding of the data, I decided to ask Copilot to convert the code to Python, a language I am not too familiar with.

So with a couple of prompts my simple PowerShell proof of concept was converted to a Python script, and as I had hoped the resultant saved file loaded in Excel without a problem.

It was then a simple matter of adding more code, via Copilot prompts, to loop through all the Test Plans and Test Suites in the Team Project, and then call the API to export them to a set of Excel files.

This is not to say I did not have to make some tweaks to the code, but Copilot got most of it correct, there were a couple of edits I had to make to API URL strings, but Copilot certainly wrote the Python code much faster than I could have done myself.

So certainly a win for GitHub Copilot, and I now have a tool that can be used to export Azure DevOps Test Plans to Excel files at scale.

This can all seem a bit ‘in the future’, but it is a future that is arriving rapidly, and one that will make for a seismic change in what it means to be in a development team.

So to find out what your next steps in this space could be, why not come to the Black Marble ‘GitHub Copilot how to harness the power of AI for Developers’ event I am presenting at next week next week?

I have recently been swapping some Azure DevOps Pipelines to GitHub Actions as part of a large GitHub Enterprise migration. The primary tool I have been using for this is GitHub Copilot in the new Agent Mode

Frankly, Copilot is like magic, it is amazing how close it gets to a valid solution. I say this after suffering years of undelivered marketing promises of no-code “we won’t need developers in the future”.

Thats said, I have yet to find a Copilot translated GitHub Action workflow that works 100% first time. Interestingly they often run without error, but don’t have the desired results of the source pipeline.

So what have I learnt?

Your prompts will not be as ‘definitive’ as you think.

Many steps done using Azure DevOps tasks have no obvious 1-to-1 equivalent in GitHub Actions, so Copilot generates script blocks to perform the same function.

I was moving an Azure DevOps YAML pipeline that was targeting Ubuntu to GitHub Actions, so not unreasonably the generate scripts were in BASH. The problem for me was that I don’t have that much BASH experience, so making my debugging slower.

The solution in my case was as simple as to add an initial prompt stating any generated scripts should be in PowerShell Core, or as I did, use subsequent prompts to alter the generated scripts.

Note If you always want given prompt to be present, such as to favour PowerShell in a given repo, the you could look at adding a .github/copilot-instructions.md file to define prompts that are added to everything.

Silent errors are the root of many issues

Most of the problems I had with migrated pipelines were the ones you experience in any CI/CD solution i.e. you are in the wrong folder, a filter is wrong, the built URI is incorrect etc.

Using Copilot for this migration, these types of errors seemed to be caused by either

• the move from Azure DevOps tasks, that may have hidden behaviors or defaults, to scripts
• or the differences in CLI tools called between the tasks and generated scripts

I good example was the way an archive/ZIP step worked. I ended up creating a ZIP file of static website content with an extra root folder, so all other folders were one level deeper than expected. This deployed without error, but obviously did not work as the old site content had not been overwritten, all the new content was in a sub-folder. A simple problem, but one that took me too lon to realised what the cause was as I got no error messages.

Summary

GitHub Copilot can get you 90% plus of the way to a solution, but as with all coding the ‘devil is in the detail’. That last few percent will take a disproportionate time.

Also there is the danger it generates code you don’t understand, but that is a learning opportunity is it not?

That said, with careful crafting of your prompts you get a better solution faster, as long as you don’t relax the quality checks you should be doing for any solution whether human or AI created.

After changing my configuration of the vNet, I restarted the Logic App to make sure everything was working as expected on startup.

The Logic App appeared to start within a few seconds, the Overview pane in the Azure Portal showing the correct values.

However, if I tried to access any of the workflows in the Logic App, I was presented with a ‘bad gateway’ error. On receiving this error, I assumed that I had made a mistake in the configuration of the vNet and started to investigate.

However, I should not have been so hasty. After waiting a minute or two, the Logic App started to work as expected. It appears that the Logic App Overview pane is updated long before the Logic App is fully operational,so can’t be trusted as a source of truth.

So not a revelatory blog post, but a reminder to myself to be patient when restarting Logic Apps.

I recently was debugging an Execute PowerShell Code Logic App action, so wanted to see it’s output. However, when I reviewed the run history, the output for my code action was empty.

The Cause (and Solution)

The problem turned out to be that the Logic App’s Inbound traffic configuration. You can replicate this issue easily

1. Create a new Logic App Workflow, use any trigger and then add an Execute PowerShell Code action, you don’t need to edit the default code sample.
2. Save the new Action and then run it.
3. Review the run history and you’ll see that the output as expected.
4. Next, change the Logic App’s ‘Inbound traffic configuration > Public Network Access’ to ‘Disabled’ and run the Logic App again, and the output now cannot be seen, as shown above.

You are actually warned about this in the Azure Portal the first time you view a run history the settings that block it

But it is easy to miss, or as I did, just forget about the limitation.

I recently had the need in an Azure Logic App to read a CSV file from an Azure Storage account, parse the file, and then process the data row by row.

Unfortunately, there is no built-in action in Logic Apps to parse CSV files.

So as to avoid having to write an Azure function, or use a number of slow, low level Logic App actions, I decided to use the PowershellCode action to parse the CSV file quickly inline with the rest of the Logic App.

The documented code sample suggests the following should work

# Retrieve outputs from prior steps
$csvdata = Get-ActionOutput -ActionName FileContent

$rows = $csvdata | ConvertFrom-csv

# Push outputs forward to subsequent actions
Push-WorkflowOutput -Output $rows

However this resulted in a null value for $rows.

The Solution

The problem turned out to be that the getFileContentV2 action that I used to load the file from Azure storage, returned the file content as part of an object, not as a simple string. So I had to request the correct property from the object.

# Retrieve outputs from prior steps
$action = Get-ActionOutput -ActionName FileContent
$csvdata = $action.outputs['body'].ToString();

$rows = $csvdata | ConvertFrom-csv

# Push outputs forward to subsequent actions
Push-WorkflowOutput -Output $rows

Once this change was made, the CSV file was parsed correctly and I was able to process the data as required.

In the past, I had avoided IOT and the ‘Intelligent Home’ as I did not want to get to the point where ‘I can’t do the washing up, the Internet is down’. However, I recently got sent an Octopus Mini which meant for the first time I could get real time information from their API, so no longer needing to wait until my Smart Meter got around to transferring data to them, which could be up to 48 hours later.

So I thought it time to look at unifying the management of all these systems, and potentially automating some behaviors such as switching things on or off based the current cost of power and the performance of our PV/Battery system.

So enter Home Assistant, an OSS project I had heard very good things about, and I have to say lived up to the hype.

The installation was very easy, I used an old Intel NUC PC I had lying around.

Once running on my home network, Home Assistant automatically detected far more devices than I had expected. These included an LG TV, AudioCast devices, a old NetGear NAS, and a good few more. Very impressive, using only ‘standard device integrations’ shipped in the basic Home Assistant install, and no additional configuration by me.

Connecting to Octopus Energy and Solaredge took a bit more work as these integration were not shipped as standard.

First, I installed Home Assistant Community Store (HACS). This allows you to easily install any community Home Assistant add-in hosted GitHub. Once this was in place I could install community integration for Octopus Energy and Solaredge Modbus.

Note There is a Solaredge device integration shipped with Home Assistant, but it queries the Solaredge web-hosted API and not the local device, so any data is 15-30 minutes out of date. The community Modbus version queries the Solaredge inverter directly over Ethernet, so provides real time data. Much for useful if planning to do home automation based on current conditions.

Again, this was all very easy, a few clicks and they were installed. The only complexity, where I had to do some reading and learning, was getting the correct data for Home Assistant Energy Dashboard (shown above). This was because the Solaredge Modbus integration provides a lot of low level raw data, but not all the values the Energy Dashboard needs.

However, as I was learning to expect for Home Assistant, the solution was well documented in the Solaredge Modbus integration GitHub Project WIKI. To use Sensor Templates to process the raw data to the forms needed. With hindsight, having to work my way though configuring this was an excellent way to learn Home Assistant better. I find it is always better to learn by trying to fix a problem you are experiencing, rather than just following tutorials

So, my Home Assistant is up and running. It is great to see such a successful and well documented OSS project with a large community of contributors. I am looking forward to seeing what else I can do with Home Assistant, but still fear I am on the slippery slope to being ‘unable to do the washing up due to an Internet outage’.

Edited: 4th Feb 2025 to add detail on Azure DevOps Pipelines Service Connection

Introduction

When automating administration tasks via scripts in Azure DevOps in the past I would have commonly used the Azure DevOps REST API. However, today I tend to favour the Azure DevOps CLI. The reason for this is that the CLI wrappers the REST API in such a way that it is easier to use and more consistent.

One of the most noticeable advantages is in the area of authentication. The CLI supports a number of authentication methods that are not directly available to the REST API.

In this post, I will explore the options available to authenticate the Azure DevOps CLI.

Authentication Options

The Azure CLI has a login command az login, but so does the Azure DevOps extension az devops login. We have the option to use either depending on what we are are trying to do.

The key thing to considers is scope, whether you are trying to autenticate with the whole an Azure subscription or just an Azure DevOps instance.

Azure CLI Personal Access Token (PAT)

PATs have historically been the usual means to authenticate with Azure DevOps . They are still perfectly valid, and the easiest way to authenticate for command line usage. However, it is worth noting that Microsoft are now recommending a move away from PATs.

The basic command to use a PAT is az devops login, when this is run you will be prompted for your PAT.

A script can build on this authentication mechanism and allow the PAT to be passed into the script by echoing it into the az command.

echo  "<my-pat>" | az devops login --organization "https://dev.azure.com/myorg

Interactive Login using Entra ID

An alternative to using a PAT is to login to the Azure CLI using your corporate ID that has access to your Azure Subscription. This can be done outside of any script by using the az login command prior to running the script, or making this command the first one in the script.

When the az login command is run, if it can, it will launch a browser allowing you to authenticate via Entra ID using your corporate ID. Once done, your session will be valid for a period of time, and you can use any az commands.

If a browser is not available, the az login command initiates the device code flow and instructs the user to open a browser page at https://aka.ms/devicelogin on another device. Then, enter the code displayed in the terminal. This flow can be forced by using the command

 az login --use-device-code

The problem both those mechanisms that they are not suitable for a script that needs to run unattended, as they requires a user to be present to authenticate.

Using a Service Principle

A Service Principle is an application registration within Entra ID.

In Azure DevOps, a Service Principle can be granted access to an Azure DevOps Organisation and Projects as you would a user account i.e. the Service Account can be granted a license and given permissions inside a project.

You can create a Service Principle using the command

az ad sp create-for-rbac --name myServicePrincipalName1 --role reader --scopes /subscriptions/00000000-0000-0000-0000-000000000000 

Note: The Service Principle must have at least the role of ‘reader’ in the Azure Subscription associated with the Entra ID

When the above command is run, the result will be block of JSON including an appId, password and tenantID. These values should be stored securely

Once the Service Principle has been created you can grant it permissions to the Azure DevOps Organisation and Projects as needed.

Finally, using the securely stored values, you can now authenticate with the Azure ClI using the form

az login --service-principal -u <appID> -p <password> --tenant <tenantid>

Unlike the interactive login, if you use this mechanism you have the option to pass these values into a script as parameters. So can be used for unattended scripts.

The issue with this mechanism is that the password will expire, and has to be renewed. So there is a maintainance overhead.

Managed Identity

The final option is to use an Azure Managed Identity.

A Managed Identity is created as a resource in Azure. Like a Service Principle, a Managed Identity can be granted a license and permissions in Azure DevOps.

To login with a Managed Identity on an Azure hosted VM you can use the command

az login --identity

This works because the Managed Identity must be associated with the Azure VM you are running the az command on.

The need to be on an Azure resource is the limitation of Managed Identities. So are not appropriate to your needs if you wish to run your script, that uses the Azure CLI, from locations outside Azure.

So what do I use?

My most common use-case is to run a script that uses the AZ CLI from within an Azure DevOps Pipeline. This requirement gives need to two ways of working.

When I need to access an Azure Subscription

If my script needs permissions beyond Azure DevOps, interacting with a range of Azure resource in a subscription. I could create a Service Principle and grant it permissions to the required resources in Azure, and then pass the Service Principle values into the script as parameters from secret Azure DevOps Pipeline variables.

However, that is a lot a work, and there is a better option built into Azure DevOps, Service Connections. When you add an ‘Azure Resource Manager’ service connection you are able to pick from a variety of options such as App Registration (Service Principle), or Managed Identity. Most of the options availble automatically create the required Azure resources required and can support workload identity federation so you don’t have to worry over expiring secrets.

You can then very easily run your scripts on a pre authenticated PowerShell session using the AzurePowerShell task

- task: AzurePowerShell@5
  displayName: Run a Script against Azure Resources
  inputs:
    azureSubscription: 'MyServiceConnection'
    azurePowerShellVersion: 'LatestVersion'
    ScriptType: 'InlineScript'
    Inline: |
      # your az commands, not need to login as it is already done
      az ....
    FailOnStandardError: true
    pwsh: true

When I only need to access Azure DevOps

However, my scripts commonly only need to interact with Azure DevOps, and in this case I just use the Build Agent token as a PAT. This is because it avoids the need to set anything up, you can easily pass the token into the script as an environment variable.

trigger: none

pool:
  vmImage: ubuntu-latest

steps:
- task: PowerShell@2
  displayName: 'Increment PBI count'
  inputs:
    targetType: 'inline'
    pwsh: true
    script: |   
      # I don't need to call az login as it done automaticallly
      az devops ....
  env:
    AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

The Azure DevOps CLI command az boards work-item update can take a list of fields as a set if value pairs e.g.

az boards work-item update  --id 123 --fields Microsoft.VSTS.Scheduling.Effort=10

However, if you try to replace the field name with a variable in PowerShell like this

$fieldname = "Microsoft.VSTS.Scheduling.Effort"
az boards work-item update  --id 123 --fields $fieldname=10

you will get an error like this

The –fields argument should consist of space separated “field=value” pairs.

The Solution

The solution is simple, and the same one required if you wish to pass multiple fields into the command, you need to wrapper the set of key value pairs for the fields in quotes e.g.

$fieldname = "Microsoft.VSTS.Scheduling.Effort"
az boards work-item update  --id 123 --fields "$fieldname=10"

Once this is done the command will work as expected.

I recently wrote about the changes I had had to make to our Azure DevOps pipelines to address the changes required when code signing with a new DigiCert certificate due to new private key storage requirements for Code Signing certificates

Today I had to do the same for a GitHub Actions pipeline. The process is very similar, but there are a few differences in the syntax and the way the secrets are stored.

The Solution

Step 1: Create a Composite Action

I stored theses steps as a Composite Action for easier reuse, but you could put them within a workflow if you prefer.

The composite action installs the DigiCert tools in the first step, and then finds and signs the files in the second.

Yes, I know I could just have a single step, but I wanted to follow the Azure DevOps flow as closely as possible.

name: 'Sign Code with DigiCert'
description: 'Signs the contents of a folder with DigiCert'
inputs:
  digicert-api-key: 
    description: 'The DigiCert API key'
    required: true
  tools-download-url:
    description: 'The URL for the DigiCert Windows MSI'
    required: false
    default: https://one.digicert.com/signingmanager/api-ui/v1/releases/smtools-windows-x64.msi/download
  signer_p12_file: 
    description: 'The DigiCert Signer P12 File as base64 encoded string'
    required: true
  cert_crt_file:
    description: 'The DigiCert Certificate CRT File as base64 encoded string'
    required: true
  digicert_host:
    description: 'The DigiCert Host'
    required: false
    default: https://clientauth.one.digicert.com
  keypair_alias:
    description: 'DigiCert certifiate alias'
    required: true
  password:
    description: 'Digicert password'
    required: true
  file-path:
    description: 'Path to scan for files'
    required: true
runs:
  using: "composite"
  steps:
  - name: Install DigiCert Client Tools
    shell: pwsh
    run: | 
      curl -X GET  ${{ inputs.tools-download-url }} -H "x-api-key:${{ inputs.digicert-api-key }} " -o smtools-windows-x64.msi 
      msiexec /i smtools-windows-x64.msi /quiet /qn 

  - name: Code Sign Files
    shell: pwsh
    run: |
      # Define the base path where signtool.exe is located
      $basePath = "C:\Program Files (x86)\Windows Kits\10\bin"
      # Filtering via version and architecture, could use just one of the these, depends on needs
      $preferredVersion = "x64"

      # Get the matching signtool path (pick the last in the list if multiple returned)
      $signtoolPath = (Get-ChildItem -Path $basePath -Recurse -Filter "signtool.exe" -File | Where-Object { $_.FullName -like "*\$preferredVersion\*" })[-1] | Select-Object -ExpandProperty FullName
      write-host "Found signtool at $signtoolPath"
      
      set-content -Path 'signer.p12.base64' -Value '${{ inputs.signer_p12_file }}' 
      certutil -decode -f 'signer.p12.base64' 'DigiCert Signer Certificate_pkcs12.p12'

      set-content -Path 'cert.crt.base64' -Value '${{ inputs.cert_crt_file }}' 
      certutil -decode -f 'cert.crt.base64' 'cert.crt'

      # Find the files that match 
      write-host "Finding files that match'the path ./src/sample/bin/**/**/*.dll"
      Get-ChildItem -Path "${{ inputs.file-path }}" | ForEach-Object {
        $filePath = $_.FullName
        write-host "Signing file $filePath"
        & $signtoolPath sign /v /tr http://timestamp.digicert.com /td SHA256 /fd SHA256 /csp "DigiCert Signing Manager KSP" /kc "${{ inputs.keypair_alias}}" /f "cert.crt" $filePath
      }   
    env:
      SM_HOST: ${{ inputs.digicert_host }} 
      SM_API_KEY: ${{ inputs.digicert-api-key }} 
      SM_CLIENT_CERT_PASSWORD: ${{ inputs.password }} 
      SM_CLIENT_CERT_FILE: DigiCert Signer Certificate_pkcs12.p12
      SM_TLS_SKIP_VERIFY: true

Step 2: Store the Certificates as Secrets

As with Azure DevOps implementation we now have a pair of files, the .CRT certificate and the .P12 signer’s certificate.

GitHub does not have a feature like Azure DevOps Secure Files. So, we have to store the certificates as secrets.

To store these certificates as GitHub Secrets you need to encode the file content into a base64 string and then add it as a secret in your GitHub repository, or Organisation or Enterprise. Use whichever level works best for you, in my case I chose to store them as Organisation secrets.

Here are the steps:

1. Open Command Prompt or PowerShell.
2. Use the certutil command to encode the file:

   certutil -encode yourfile.crt yourfile.crt.base64
   

3. Open the generated yourfile.crt.base64 file in a text editor (e.g., Notepad) and copy its content.
4. Add the secret to GitHub at the level you chose:
   • Name your secret (e.g. CRT_FILE).
   • Paste the base64-encoded content into the Value field.

Repeat this process for both the .CRT and the .P12 file.

Step 3: Store the other DigiCert settings as secrets and variables

The other DigiCert settings can be stored as a mixture of secrets and variables. Use variables if reading the value in the log is not deemed a security risk.

The secrets:

• DIGICERT_API_KEY
• DIGICERT_SIGNER_P12_FILE as base64 encoded string (see above)
• DIGICERT_CERT_CRT_FILE as base64 encoded string (see above)
• DIGICERT_CLIENT_CERT_PASSWORD

and as a variable:

• DIGICERT_KEYPAIR_ALIAS

Step 4: Using the Composite Action in a Workflow

Finally pull it all together in your workflow

 - name: Sign files with Digicert
   uses: blackmarble/Sign-Code-With-DigiCert@v1
   with:
     digicert-api-key: '${{ secrets.DIGICERT_API_KEY }}'
     signer_p12_file: '${{ secrets.DIGICERT_SIGNER_P12_FILE }}'
     cert_crt_file: '${{ secrets.DIGICERT_CERT_CRT_FILE }}'
     keypair_alias: '${{ vars.DIGICERT_KEYPAIR_ALIAS}}'
     password: '${{ secrets.DIGICERT_CLIENT_CERT_PASSWORD }}'
     file-path: './src/Sample/bin/**/**/*.dll'

This one of those posts that is more a note to self as I keep forgetting how to do this, but I hope it helps others.

Background

I use Microsoft’s Authenticator to provide MFA on a number of accounts. I recently swapped my Android phone and had to, after restoring a backup, re-authenticate some accounts on the new device.

This was a simple process for most accounts, just a case of validating the code generated by the new device, but I had a problem with the entries where my Black Marble Entra ID account was a guest in other company’s Entra ID directories.

I could not remember the process.

The Solution

When the Authenticator app shows the message

Action Required

Scan the QR code provided by your organization to finish recovering your account

and the account is in the form richard_domain.com#EXT#@anothercompany.onmicrosoft.com

The steps are as follows (assuming you still have access to the old MFA device):

1. In a browser open https://mysignins.microsoft.com/security-info
2. Sign in with your own company Entra ID account i.e. in this example richard@domain.com
3. Click the ‘organisation’ button (a icon of a directory tree, in the top right near the help button).
4. A list of organisations you are a member of will appear as a panel on the right of the page
5. Pick the required organisation from the menu on the right, in my richard_domain.com#EXT#@anothercompany.onmicrosoft.com example, this would be an organisation called anothercompany
6. You will need to authenticate with the old MFA device
7. On the refreshed https://mysignins.microsoft.com/security-info page for the selected organisation you can now use the add a new device process to generate the QR code
8. On the new MFA device pick the account to be re-authenticate and scan the QR code. This is all that is required to add the new device as sign-in method (note you do not have to completed the new device wizard in the browser, but it is probably a good idea as a final check.).

If you don’t have access to the old MFA device, you will need to contact the organisation’s IT support to get them to reset the MFA for you.

It is becoming increasingly important to sign files digitally to ensure that they have not been tampered with, to secure the software supply chain. This is something we have done for a good while as a step in our Azure DevOps pipelines. However, recent(ish) changes in the way certificates are issued has meant we have had to revise our approach.

The Problem

We used to use a .PFX file, stored as an Azure DevOps secure file, that contained the public and private keys and was accessed using a password, to sign our files.

However, when we renewed our code signing certificate with DigiCert we found this approach no longer valid due to new private key storage requirements for Code Signing certificates.

The basic issue is that now when we sign a file, the signing tool needs to make a call back to a secure location to validate the certificate. In our case Digicert’s Keylocker service.

The Solution

This change required some changes to our pipelines.

1. Store our .CRT certificate file as a secure file in Azure DevOps
2. Store our .p12 signer’s certificate file as a secure file in Azure DevOps
3. Store our DigiCert account settings as Azure DevOps pipeline variables (a mixture of standard and secret ones)
4. Update our pipeline as follows

- task: SSMClientToolsSetup@1
  displayName: Install DigiCert Client Tools 

- task: DownloadSecureFile@1
  displayName: Download DigiCert Code Signing Certificate File
  inputs:
    secureFile: 'DigicertCodeSigningCert.crt'

- task: DownloadSecureFile@1
  displayName: Download DigiCert Signer Certificate File
  inputs:
    secureFile: 'DigiCertSignerCertificate.p12'

- task: PowerShell@2
  displayName: Code Sign Files
  inputs:
    targetType: 'inline'
    script: |
      # Define the base path where signtool.exe is located
      $basePath = "C:\Program Files (x86)\Windows Kits\10\bin"
      # Filtering via version and architecture, could use just one of the these, depends on needs
      $preferredVersion = "x64"

      # Get the matching signtool path (pick the last in the list if multiple returned)
      $signtoolPath = (Get-ChildItem -Path $basePath -Recurse -Filter "signtool.exe" -File | Where-Object { $_.FullName -like "*\$preferredVersion\*" })[-1] | Select-Object -ExpandProperty FullName

      # Find the files that match filter that need to be signed
      write-host "Finding files to sign"
      Get-ChildItem -Path "$(Build.SourcesDirectory)/myproject/**/*.exe" | ForEach-Object {
         $filePath = $_.FullName
         write-host "Signing file $filePath"
         & $signtoolPath sign /v /tr http://timestamp.digicert.com /td SHA256 /fd SHA256 /csp "DigiCert Signing Manager KSP" /kc "$(SM_KEYPAIR_ALIAS)" /f "$(Agent.TempDirectory)\DigicertCodeSigningCert.crt" $filePath
       }   
   env:
     SM_HOST: $(SM_HOST) 
     SM_API_KEY: $(SM_API_KEY) 
     SM_CLIENT_CERT_PASSWORD: $(SM_CLIENT_CERT_PASSWORD) 
     SM_CLIENT_CERT_FILE: $(Agent.TempDirectory)\DigiCert Signer Certificate_pkcs12.p12
     SM_TLS_SKIP_VERIFY: $(SM_TLS_SKIP_VERIFY)

The gotcha with variables

I have blogged a number of times before about the need to be careful with the syntax for Azure DevOps variables. Guess what, I got caught out by this again!

I had initially used the ${{ variables.XXX }} format for injecting the Azure DevOps variables as PowerShell environment variables e.g.

 SM_HOST: ${{ variables.SM_HOST }}
 SM_API_KEY: ${{ variables.SM_API_KEY }}

This worked fine for the standard variables, but not for the secret ones. The secret ones were not being injected into the script as environment variables so when we tried to sign a file we got the error

SignTool Error: No private key is available.

This was a case of trying to be too clever with the syntax, the correct syntax in this case is to use the standard macro syntax $(XXX) format for all variables, especially the secret ones.

 SM_HOST: $(SM_HOST)
 SM_API_KEY: $(SM_API_KEY)

Once this was done, the signing process worked as expected.

Every project is different, and sometimes you need to be able to inject a step into the Web Deploy package creation process to complete some extra step. This can be done by adding a target to the .csproj project file.

The following example shows how you could sign the assemblies before the Web Deploy package is created.

 <Target Name="SignWebExe" AfterTargets="GenerateMsdeployManifestFiles" BeforeTargets="PackageUsingManifest" Condition="'$(Configuration)' == 'Release'">
    <PropertyGroup>
      <Cmd>signtool.exe sign /debug /f "$(certPath)" /p "$(certPassword)" "$(ProjectDir)obj\$(Configuration)\Package\PackageTmp\bin\MyNamespace*.dll"</Cmd>
    </PropertyGroup>
    <Message Text="Signing web deploy executable with command: $(Cmd)" />
    <Exec Command="$(Cmd)" />
  </Target>

The required parameters for this extra step are then passed in as MSBuild arguments. For example, in an Azure DevOps pipeline, you could use the following task:

- task: VSBuild@1
displayName: 'Build Core Services Solution'
inputs:
    solution: src/MySolution.sln
    msbuildArgs: '/p:DeployOnBuild=true;PublishProfile=Release /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:DeployIisAppPath="__SITENAME__" /p:certPassword="$(SigningPassword)" /p:certPath="$(SigningCertFilePath)"'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    clean: true

We are running our SonarQube instance as an Azure hosted Docker container.

Over the past few weeks we have been seeing intermittent occurrences of the ProjectFileIndexer exception during the SonarQube analysis step in our Azure DevOps pipelines.

##[error]java.lang.IllegalStateException: Unable to load component class org.sonar.scanner.scan.filesystem.ProjectFileIndexer

When I looked closer at the exception stack, I could see at the bottom there was always a timeout error when trying to access the project.protobuf file from SonarQube.

Caused by: java.lang.IllegalStateException: Fail to request url: https://sonarqube.mydomain.co.uk/batch/project.protobuf?key=mykey

If I tried to open the URL in a browser, authenticated with SonarQube, most of the time the expected project.protobuf file was returned, but sometimes it was not.

Also, at random times, the SonarQube UI would take many seconds to refresh when I selected a project.

The Solution

In the past I have usually found any issues with SonarQube are ElasticSearch related, but not so this time.

The issue turned out to be a lack of SQL resources.

Our SonarQube instance was running on

• Azure Web App (Premium V3 P1V3 2vCPU & 8Gb)
• Azure SQL DB (Standard S2 50 DTU)
• Logs/Config/ES indexes etc/ on Azure Standard General Purpose V2) File storage

When I look at the Azure SQL Query Performance Insights, I could see that the SQL DTU resources were maxing out at 100% for long periods of time, this occurred especially when the SonarQube analysis was restarted.

I increased the SQL resource to a Standard S3 100 DTU, the scaling took 15 minutes or so. Once this was done, the issue seemed to be resolved, though it is always hard to be sure with intermittent issues.

My assumption is that there are some SQL queries that are slow to run and are potentially blocking to other queries. By increasing the SQL resources, the these queries could be processed more quickly, and the blocking reduced to a level where it was not an issue.

You can see the impact of the change in the SQL DTU usage graph below. After the increase in SQL resources, we are not maxing out the server DTUs, so the SonarQube analysis can running without issues.

I think the key take away is to make sure you have enough SQL resources, don’t max out your assigned DTUs, as this will cause issues.

As I have blogged previously, we run a SnipeIT instance to manage our IT assets, hosted in Azure using Docker.

This has been working well for us for the past year, but recently we have noticed that the system has become very slow to respond.

Looking on the Azure portal, we can see that around the 15th of October the web app’s response times have gone from milliseconds to 10s of seconds

Investigation

The first question we asked, as you always should, was what had changed?

The answer was nothing obvious.

• We had not changed the Azure Web App or MySQL SKUs
• It is true that we had not updated the SnipeIT version since July, but we were on the current major and minor version and only a few patch versions behind.
• We had not added any significant new assets or users in the last few months.

So it all appeared strange what had changed?

We of course tried restarting the container instance, and cleaning down logfiles. We have seen slow performance in other systems when the logfiles get too large.

We also tried upping the SKU of the Azure Web App, but none of this made a difference.

We then set the SnipeIT log level to debug, this showed the high response times were not due to the web app being slow to respond, but the time taken to get data from the MySQL instance.

Looking again at the metrics in the Azure portal, I noticed that the MySQL instance started showing a higher CPU usage and increased DB connections around the same time as the web app started to slow down.

These new CPU and connection levels did not seem excessive, but they were higher than they had been previously. I then noticed in the Azure Portal a warning message, that we had used up all the MySQL Burstable SKU credits.

Solution

The solution was in fact simple, to move the MySQL instance to a higher SKU, from Standard_B1s to Standard_B2ms, increasing the running cost by only a few pence a month, from £2.17 to £2.25 PCM

This SKU change doubles the resources available to MySQL, but more importantly allows us to build up more burstable CPU credit for the times we need it in a month.

So now I have a working solution, it as all down to the fact that we had been running close to the limit of our MySQL SKU for a while, and it was a matter of time before the issue occurred.

If you are using staged deployment in Azure DevOps, you will probably have multiple Azure Service Connections. So, it makes sense that you might want to use a Service Connection name that is stored in a variable group as a parameter to a templated YAML pipeline.

# the build pipeline
stages:
  - stage: UAT
    jobs:
    - deployment: ARM_Provisioning
      timeoutInMinutes: 0
      environment: 'Staging'
      variables:
      - group: UAT
      pool:
        vmImage: 'windows-latest'
      strategy:
        runOnce:
          deploy:
            steps:
            - template: YAMLTemplates\ProvisionUsingARM.yml
              parameters:
                AzureResourceGroup: $(AzureResourceGroup)
                AzureServiceConnection: $(AzureServiceConnection)
  - stage: PROD
    jobs:
    - deployment: ARM_Provisioning
      timeoutInMinutes: 0
      environment: 'Staging'
      variables:
      - group: PROD
      pool:
        vmImage: 'windows-latest'
      strategy:
        runOnce:
          deploy:
            steps:
            - template: YAMLTemplates\ProvisionUsingARM.yml
              parameters:
                AzureResourceGroup: $(AzureResourceGroup)
                AzureServiceConnection: $(AzureServiceConnection)

With a template YAMLTemplates\ProvisionUsingARM.yml that uses the AzureServiceConnection variable

parameters:
- name: AzureResourceGroup
  type: string 
- name: AzureServiceConnection
  type: string

- task: AzureResourceManagerTemplateDeployment@3
  displayName: 'Azure Deployment: Create Or Update Resource Group'
  inputs:
    deploymentScope: 'Resource Group'
    azureResourceManagerConnection: '${{ parameters.AzureServiceConnection}}'
    action: 'Create Or Update Resource Group'
    resourceGroupName: '${{parameters.azureResourceGroup}}'
    ...

The Issue

If I declared a pair of variables in the UAT and PROD variable groups with the names AzureServiceConnection and AzureResourceGroup I would expect each stage to pick up the appropriate variable group and use the correct service connection.

However, this is not the case. The pipeline will fail with the following error message when you validate or try to queue the pipeline.

There was a resource authorization issue: “The pipeline is not valid. Job ARM_Provisioning: Step input azureResourceManagerConnection references service connection $(AzureServiceConnection) which could not be found. The service connection does not exist, has been disabled or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."

Basically, the $(AzureServiceConnection) variable is not being expanded during template validation.

Analysis

Compile and Runtime variables

I tried all the options for the variable declaration in the YAML

• Standard Macro format $(AzureResourceGroup)
• Expression format $[AzureResourceGroup]
• Runtime expression format ${{ variables.AzureResourceGroup}}

None of these helped.

Hardcoding the connection name

If you hardcode the service connection name in the outer YAML pipeline file, the pipeline will work as expected.

# the build pipeline
  - stage: UAT
    jobs:
    - deployment: ARM_Provisioning
      timeoutInMinutes: 0
      environment: 'Staging'
      variables:
      - group: UAT
      pool:
        vmImage: 'windows-latest'
      strategy:
        runOnce:
          deploy:
            steps:
            - template: YAMLTemplates\ProvisionUsingARM.yml
              parameters:
                AzureResourceGroup: $(AzureResourceGroup)
                AzureServiceConnection: UAT
  - stage: PROD
    jobs:
    - deployment: ARM_Provisioning
      timeoutInMinutes: 0
      environment: 'Staging'
      variables:
      - group: PROD
      pool:
        vmImage: 'windows-latest'
      strategy:
        runOnce:
          deploy:
            steps:
            - template: YAMLTemplates\ProvisionUsingARM.yml
              parameters:
                AzureResourceGroup: $(AzureResourceGroup)
                AzureServiceConnection: PROD

But that is not at all what we are after, but it did at least show that it was possible to pass in the service connection name as a string parameter.

Using a global variable

If I declared a global pipeline variable, at the top of the pipeline file, it could be validated and queued without any issues.

variables:
  AzureServiceConnection: PROD

stages:
   ....

But again this was still not what I was after.

The Solution

In my project I have a number of variable groups, one for each stage of deployment. What I found I needed to do was to declare one of them as a global variable for the whole pipeline. This would allow the pipeline to be validated and queued without any issues, but in at each deployment stage this initial value is overridden by the variable group associated with the stage.

This does rely on the fact that the variable groups for each stage contain the same variable names.

The revised YAML looks as follows

# the build pipeline
variables:
# we have to globally declare this variable group, though it is override at the deployment job level
# if we don't do this the $(AzureServiceConnection) variable used by the Azure Resource deployment fails queue time validation 
- group: UAT

stages:
  - stage: UAT
    jobs:
    - deployment: ARM_Provisioning
      timeoutInMinutes: 0
      environment: 'Staging'
      variables:
      - group: UAT
      pool:
        vmImage: 'windows-latest'
      strategy:
        runOnce:
          deploy:
            steps:
            - template: YAMLTemplates\ProvisionUsingARM.yml
              parameters:
                AzureResourceGroup: $(AzureResourceGroup)
                AzureServiceConnection: $(AzureServiceConnection)
  - stage: PROD
    jobs:
    - deployment: ARM_Provisioning
      timeoutInMinutes: 0
      environment: 'Staging'
      variables:
      - group: PROD
      pool:
        vmImage: 'windows-latest'
      strategy:
        runOnce:
          deploy:
            steps:
            - template: YAMLTemplates\ProvisionUsingARM.yml
              parameters:
                AzureResourceGroup: $(AzureResourceGroup)
                AzureServiceConnection: $(AzureServiceConnection)

So I think a valid workaround for another strange YAML variable expansion issue in Azure DevOps.

The reason I find this so useful is that it allows me to make a series of small related changes to a project without having to clone the repository or using a CodeSpace, very useful when editing the related YAML files of reusable workflows in GitHub Actions.

I was recently working on a templated YAML Azure DevOps Pipeline project and pressed . our of habit and was really pleased to find that this VS Code in the browser feature is also available for Azure DevOps.

So next time you need to make a series of small changes to a project in Azure DevOps why not give it a try.

This is is one of those posts I write so I remember how to do something in the future.

Background

I recently had a need to generate many Visual Studio SQL Database Projects from existing databases. Being a good ’lazy developer’ I wanted to do this from the command line so I could automate the process, but it took me far to long to work out how

The Manual Way

If you only have one database to import you can do this manually by using the Import option in Visual Studio for an individual SQL Database Project.

1. Open Visual Studio
2. Create a new project using the SQL Server -> SQL Server Database Project template
3. Right click on the project in the Solution Explorer and select Import -> Database and follow the wizard.

The Command Line Way

For some reason I really struggled to find the command line syntax to perform the same action, so I am documenting it here for future reference.

The key command is SqlPackage.exe and the syntax needed will be something similar to

sqlpackage /Action:Extract /SourceConnectionString:"Server=tcp:{instance},{port};Initial Catalog={databasename};TrustServerCertificate=True;integrated security=true;" /TargetFile:{projectname} /p:ExtractTarget=SchemaObjectType

This command will generate an database project in a folder of the name specified using /TargetFile, where each SQL object has it’s own file containing the appropriate SQL CREATE script.

The ‘magic’ that took me too long to fine was that the /p:ExtractTarget=SchemaObjectType parameter is required. This instructs SQLpackage to generate a project structure and not extract to a single SQL file or DACPAC.

I am not sure whether my internet search powers were weak, or if this usage is just poorly documented, but I hope this post save future me, and others, some time.

Previously Rik Hepworth has posted on ‘Importing bicep lint output as test results in Azure DevOps pipelines’. In his post he showed how you could move from using the ARM-TTK to validate ARM templates to using the built inBicep Linter.

Rik’s solution involved taking the Bicep Lint output and converting it via the .SARIF format to JUnit so that it could be published to an Azure DevOps pipeline run as a set of test results.

The problem is are linting issues really failed tests?

An Alternative Approach

The Bicep linter, based on its bicepconfig.json defined configuration, will return issues as either an error or a warning when a build is done.

Is it not a better solution to return these as errors and warnings in the pipeline, just as you would for MSbuild?

Returning the linting issues in this manner actually involves less steps than Rik’s solution. Basically we run a Bicep build from within a AzureCLI@2 task and then process the output to return any issues as Azure DevOps errors and warnings. The advantage of using the AzureCLI@2 task is that it is automatically kept up to date to the current Bicep version.

The critical step that is easy to forget is to set the powershellerrorActionPreference: 'continue' so that the script does not fail on the first error

Note: the 2>&1 at the end of the az bicep build command to ensure that the error stream is captured in the $output variable.

steps:
  - task: AzureCLI@2
    displayName: 'Build Bicep files'
    inputs:
      azureSubscription: 'My Azure Subscription'
      scriptType: 'ps'
      scriptLocation: 'inlineScript'
      useGlobalConfig: true
      powershellerrorActionPreference: 'continue' # we handle the errors in the script
      inlineScript: |
        # create folder if it doesn't exist
        if (!(Test-Path -Path $(Build.SourcesDirectory)\Bicep\ARMOutput)) {
          New-Item -ItemType Directory -Path $(Build.SourcesDirectory)\Bicep\ARMOutput
        }
        write-host "Build the Bicep file"
        $output = az bicep build --file $(Build.SourcesDirectory)\Bicep\RootTemplate-Main.bicep --outdir $(Build.SourcesDirectory)\Bicep\ARMOutput 2>&1
        write-host "Process the output"
        $output | foreach-object {
           if ($_ -match 'Error') {
              Write-Host "##vso[task.logissue type=error]$_"
           } 
           if ($_ -match 'Warning') {
               Write-Host "##vso[task.logissue type=warning]$_"
           }
        }

Conclusion

For many projects this solution will give a more actionable result than Rik’s process.

The time that will not be true is if you wish to import the linting issues into a tool such as SonarQube, where using the .SARIF format will be vital.

Last year I posted on how to create your own Azure DevOps maintenance jobs. This solution has been working well for me, until the Azure DevOps Service connection’s Entra ID Service Principle secret expired.

So, I thought it well worth revisiting the creation of this maintenance job but this time using Workload Identity federation to authenticate, and hence never again have to worry about the secret expiring.

Updated Setup Process

Note: This is a modification to the creation of the service connection, but the core the maintenance job setup remains the same as in my original post

1. In the Azure DevOps select the Team Project that the maintenance job will run in, and create a new Service Connection of type Azure Resource Manager and the default type Workload Identity federation (automatic). Pick the Azure Subscription that the Azure DevOps instance is linked to.

2. Once the Service Connection is created, in the Azure Portal check the Entra ID directory for the newest App Registration (Service Principle) that you own. Unfortunately using this setup method, you have no control over the name of this App Registration, but you can probably assume it will be the newest one you own. It will be in the from [AzDo Org]-[Team Project]-[Subscription ID] format e.g. blackmarble-source-bm-3ed896d7-9999-9999-99999999999.

   Important: You could have multiple App Registrations with the same name, so you need to make a note of the Client ID in the Overview blade of the App Registration as you need it later

3. Back at the Azure DevOps organisation level you need to add the newly created Service Principle as a known user. This is where knowing the Client ID is important, as you can use it to identify the correct entry because the Client ID is shown as a subscript on each entry. I granted my new Service Principle Basic level access to be able to query the AZ CLI.

   

   Note: You might get away with Stakeholder access, but I knew my maintenance jobs needed source code access so I had to have Basic level

   Note:

   If you fail to perform this step you see the fairly unhelpful error

   ERROR: TF401444: Please sign-in at least once as {GUID Identifier} in a web browser to enable access to the service.

4. You can now ready to update your scheduled maintenance job trigger YAML to use the new Service Connection, and then attempt it. You will probably get permission errors, these will vary depending on what you are trying to do. In my case I am trying to queue builds so the permissions I need to grant the Service Principle in Azure DevOps are

   • At the Org level reader access to the Agent Pools
   • At the Team Project level reader access
   • At the Team Project level permissions to queue builds and edit build queue configuration

Summary

Moving to Workload Identity federation is a great way to avoid the issue of Service Connection secrets expiring. It is a little more complex to setup, but once done it is a great way to authenticate your maintenance jobs or any other Azure DevOps to Azure interactions.

Whilst refreshing an end-to-end devops demo, one I use for both Azure DevOps and GitHub, I hit a problem. The new Playwright UX Tests, that were replacing old Selenium ones, were failing on the GitHub hosted runner.

The strange thing was the same tests worked perfectly on:

• My local development machine
• The Azure DevOps hosted runner
• And strangest of all, a GitHub self hosted runner

The Solution

Adding some logging to the tests showed the actual issue was that on the GitHub hosted runner the code to count the rows in an HTML table was always returning 0.

I went down a few dead ends, looking at permissions and tooling versions, but the solution was simple. The Playwright tests were running too fast on the GitHub hosted runner.

So, at it’s simplest just adding a wait before my table locator fixed the issue.

await page.waitForTimeout(5000);
int rowCount = await Page.Locator(".dataTable").Locator("tr").CountAsync();

However, spraying waits across the test codebase is not a great solution, but it worked and gave me a starting point to refactor the test to be more robust.

A better solution was to explicitly wait for the table to be visible before counting the rows.

await Expect(Page.Locator(".dataTable")).ToBeVisibleAsync();
int rowCount = await Page.Locator(".dataTable").Locator("tr").CountAsync();

The key takeaway

I think the key here is not the quality of my UX tests, but that this issue only showed up on GitHub hosted agents even though they are built off the same image as the Azure DevOps ones and use effectively the same agent.

The only difference is that Azure DevOps and GitHub hosted runners are provisioned with different virtual hardware specifications

• Azure DevOps: 2 vCPUs, 7 GB RAM
• GitHub: 4 vCPUs, 16 GB RAM

This can be important, as I found, not just for speed of build.

I have been recently refreshing a GitHub end-to-end demo I use for talks and workshops that I had not looked at for a while. It shows how legacy code bases can be deployed with GitHub Actions and Azure App Services. The demo uses MSDeploy to deploy a ASP.NET web application to Azure App Services. The MSDeploy package is created as part of the GitHub Action workflow.

The workflow uses a PowerShell script to do the deployment using the following:

- name: 'Deploy web site with MSDeploy'
  run: |
    $publishProfile = az webapp deployment list-publishing-profiles --resource-group ${{ vars.AzureResourceGroup}} --name ${{ secrets.Sitename }} --query "[?publishMethod=='MSDeploy']" --subscription "${{ secrets.AZURESUBSCRIPTION}}" | convertfrom-json
    $shortPath = resolve-path ./webdeploy
    & "C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -source:package="$shortpath\FabrikamFiber.Web.zip" -setParamFile:"$shortpath\FabrikamFiber.Web.SetParameters.xml" -dest:auto,ComputerName="https://$($publishProfile.msdeploySite).scm.azurewebsites.net/msdeploy.axd?site=$($publishProfile.msdeploySite)",UserName=$($publishProfile.userName),Password=$($publishProfile.userPWD),AuthType='Basic' -verbose -debug -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension
  shell: pwsh

When I last used this demo, over a year ago, it worked fine. However, when I tried to run it again, I found that the MSDeploy was failing with the following error:

Microsoft.Web.Deployment.DeploymentException: Unrecognized argument ‘"-dest:auto,ComputerName=“https://$($publishProfile.msdeploySite).scm.azurewebsites.net/msdeploy.axd?site=$($publishProfile.msdeploySite)",UserName=$($publishProfile.userName),***‘Basic’”’. All arguments must begin with “-”.

The Solution

The difference between when this workflow last ran and now is that the Powershell Core version used in the GitHub Action has been updated from 6 to 7. It appears the token replacement in the -dest parameter is not working as expected in Powershell 7, though it work for Powershell 6.

Once I altered it to the following, putting the whole dest parameter in quotes, it worked:

- name: 'Deploy web site with MSDeploy'
  run: |
    $publishProfile = az webapp deployment list-publishing-profiles --resource-group ${{ vars.AzureResourceGroup}} --name ${{ secrets.Sitename }} --query "[?publishMethod=='MSDeploy']" --subscription "${{ secrets.AZURESUBSCRIPTION}}" | convertfrom-json
    $shortPath = resolve-path ./webdeploy
    & "C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -source:package="$shortpath\FabrikamFiber.Web.zip" -setParamFile:"$shortpath\FabrikamFiber.Web.SetParameters.xml" -dest:"auto,ComputerName=https://$($publishProfile.msdeploySite).scm.azurewebsites.net/msdeploy.axd?site=$($publishProfile.msdeploySite),UserName=$($publishProfile.userName),Password=$($publishProfile.userPWD),AuthType='Basic'" -verbose -debug -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension
  shell: pwsh

Our on premise build agents, though using the same image as the Microsoft hosted agents (generated using Packer, as I have previously posted about), have some extra setup done by Lability as they are deployed to a Hyper-V host.

Most of this is specific to the needs of running on Hyper-V e.g. setting up networking, creating a 2nd disk to act as a working store for the build agent, and of course installing the Azure DevOps agent itself.

However, it turns out some of the extra steps it was doing were a barrier to running some of our major software project’s Azure DevOps Pipelines on Microsoft hosted agents.

This was something we had not worried too much about in the past. This was because the performance of the hosted agents was too slow compared to what we can provide on-premise using Hyper-V. However, with the advent of Azure Managed DevOps Pools as discussed in my recent blog post, with their greater options for scaling, this became an issue worth investigating.

So I looked to see how I could get all our pipelines to run on the Microsoft hosted agents and on Managed DevOps Pools.

The Issues

Custom Capabilities

The first issue we hit was that the Microsoft hosted agents and Managed DevOps Pools do not support custom capabilities. With on-premise agents you can add any custom capability you wish, usually used to help route jobs to a suitable agent within an agent pool, but this is not possible with the Microsoft hosted agents or Managed DevOps Pools.

‘Under the hood’ a capability is just an environment variable on the VM that the agent uses to identify the capabilities of the agent. So, though not what they were really designed for, custom capabilities are a useful way to be able to inject an environment variable to all VMs in a pool without having to alter the VM image or add startup scripts.

Historically we have used this technique to add missing environment variables to the agents i.e. to set JAVA and JDK to match the already present JAVA_HOME value. We need to do this as some tools, such as the Azure DevOps XamarinAndroid@1 task, only checks JAVA and JDK, and not JAVA_HOME to find the JDK location.

But, as I mentioned, there is no way to add a custom capability to a Microsoft hosted agent or a Managed DevOps Pool agent at the pool level. The only option is to set it as an environment variable within the VM image. Given we wanted, to minimise maintenance costs, to use the Microsoft hosted build agent, which we have no control over, this technique was not an option.

However, it turns out there is a much better solution in the XamarinAndroid@1 task case, to just pass the JDK version as a parameter to the task. This is a better solution as it is more explicit and does not rely on the agent having the correct environment variables set.

- task: XamarinAndroid@1
displayName: "Build Xamarin.Android Project"
inputs:
    projectFile: src/Droid.csproj
    target: Build
    configuration: "$(BuildConfiguration)"
    clean: true
    msbuildVersionOption: latest
    msbuildArguments: "-m"
    JDKVERSION: 1.11

So your first option should always be to check if you can use a task parameter as opposed to using custom capabilities (environment variables).

If you find that your task does not provide a parameter to set the value you need, then you can still ‘inject’ an environment variable onto an agent by setting an Azure DevOps variable in your pipeline. Remember, this works as Azure DevOps variables at are passed on to agents as environment variables.

To create a variable the following command can be used in a script task.

 echo "##vso[task.setvariable variable=MyEnvVar]MyValue"

Missing folders in PATH

For historic reasons we add the C:\Program Files (x86)\Windows Kits\10\bin folder to the PATH on our on-premise agents. This is because we use the signtool.exe to sign our projects, and this is where it is located by default.

We could of course alter all our pipelines to pass the explicit path to the signtool.exe to the task, but this is a lot of work and would make the pipelines less portable. So we wanted a means to add the path dynamically.

You might think you could use SETX or the PowerShell equivalent to add a folder to the PATH, but this does not work. The path is added to the current CMD/PS session, but not to the next task run by the Azure DevOps agent.

The answer is to use some Azure DevOps magic I was not previously aware of

echo "##vso[task.prependpath]<path to add>"

I used the following PowerShell to check if the required path was present and if not add it. You could of course greatly simplify the script if you only ever run on hosted agents and knew the actual path was always going to be missing, but I might use the script against on-premise agent where the path was already present, hence the guard code

- powershell: |
    # Define the base path where signtool.exe is located
    $basePath = "C:\Program Files (x86)\Windows Kits\10\bin"
    # Filtering via version and architecture, could use just one of the these, depends on needs
    $preferredVersion = "10.0.19041.0\x86"

    $envPath = [Environment]::GetEnvironmentVariable("PATH")
    $targetPath = join-path -path $basepath -childpath $preferredVersion

    # Check to see if the folder is already in the path
    if ($envPath -split ";" -contains $targetPath) {
        Write-Host "The path $targetPath is already present in the PATH environment variable."
    } else {
        Write-Host "The path $targetPath is NOT present in the PATH environment variable."

        # Get the matching signtool path (pick the last in the list if multiple returned)
        $signtoolPath = (Get-ChildItem -Path $basePath -Recurse -Filter "signtool.exe" -File | Where-Object { $_.FullName -like "*\$preferredVersion\*" })[-1] | Select-Object -ExpandProperty FullName

        # Double check the access to the file 
        if (Test-Path -Path $signtoolPath) {
            write-host "SignTool path is $signtoolPath"
            write-host "Updating PATH to include SignTool folder"
            echo "##vso[task.prependpath]$(Split-path -path $signtoolPath)"
        } else {
            write-error"Cannot find SignTool that match the filter $preferredVerison"
        }
    }

Using this technique we can add any path required dynamically at the start of the pipeline, and it will be available to all tasks in the pipeline.

Timezone and Language

The Microsoft hosted agent images are set to the UTC timezone and en-US language. For us this is a problem for execution of some unit tests which have a dependency on GMT Standard Time and en-GB.

Maybe we should alter our tests to make them more robust, but we can address the issue by setting the correct values as part of the pipeline

- powershell: |
    $zone = Get-TimeZone
    if ($zone.Id -eq "GMT Standard Time") {
    write-host "Timezone is correctly set to $($zone.Id)"
    } else {
        write-host "Timezone is incorrectly set to $($zone.Id), setting to GMT Standard Time"
        Set-TimeZone -id "GMT Standard Time"
    }

    $list = Get-WinUserLanguageList
    if ($list[0].LanguageTag -eq "en-GB") {
      write-host "Language is correctly set to $($list[0].LanguageTag)"
    } else {
      write-host "Language is incorrectly set to $($list[0].LanguageTag), setting to en-GB"
      $list[0] = "en-GB"
      Set-WinUserLanguageList -LanguageList $list -Confirm:$false -Force
      Set-ItemProperty -Path "HKCU:\Control Panel\International" -Name sCountry -Value "United kingdom";
      Set-ItemProperty -Path "HKCU:\Control Panel\International" -Name sLongDate -Value "dd MMMM yyyy";
      Set-ItemProperty -Path "HKCU:\Control Panel\International" -Name sShortDate -Value "dd/MM/yyyy";
      Set-ItemProperty -Path "HKCU:\Control Panel\International" -Name sShortTime -Value "HH:mm";
      Set-ItemProperty -Path "HKCU:\Control Panel\International" -Name sTimeFormat -Value "HH:mm:ss";
      Set-ItemProperty -Path "HKCU:\Control Panel\International" -Name sYearMonth -Value "MMMM yyyy";
      Set-ItemProperty -Path "HKCU:\Control Panel\International" -Name iFirstDayOfWeek -Value 0;
    }

This pattern can be used more than once in a single pipeline run to test against multiple timezones and languages, but for us it is just a case of setting the GMT/UK values once.

Summary

It can be seen these fixes are minor and so there should be no technical barrier to us using standard hosted agents, or MDP agents, in the future.

I think that many of these ‘fixes’ will be needed in most of our pipelines. So, it is probably a good idea to have a YAML template that is included at the start of all builds. One that sets the settings above, in effect making hosted agents identical to our on-premise ones.

So moving to hosted agents or MDPs, becomes a question of cost and not technology. One that is a much harder question to answer, balancing capital and servicing costs for on-premise agents against operational costs for the Azure hosted agents.

The reality is everyone’s ‘milage will vary’, but I have a feeling in our case the dynamic scaling of MDP agents, and the management cost reductions this can bring, will make all the difference

We have for many years based our Azure DevOps build and release pipelines on on-premise agents. This was done to provide a means to scale up each agent using ‘bigger & faster’ local VMs then the Microsoft Hosted agents provided, and to utilise spare Hyper-V hosts we had after moving services such as SharePoint to Office 365.

To provide consistency of build agent experience to our developers, we decided we wished to use the same VM images on our on-premise agents as used by the Microsoft’s hosted agent pools. I have blogged previously on the process of creating these images with Packer and deploying them to Hyper-V with Lability. This is a process we still follow every couple of months to keep the images reasonably up to date.

Enter Azure Managed DevOps Pools

Azure Managed DevOps Pools (MDP) offer an alternative to our on premise agents and have just gone into public preview.

MDP are the logical successor to Azure Virtual Machine Scale Set Agent Pools (VMSS). VMSS are still an option as they are not going away, but MDP are easier to setup and have enhanced features. The key ones being:

• You can pick your Agent Images
  • You can use the same image as the Microsoft Hosted Agent, but with no need to create them yourselves as we do via Packer (a feature not available with VMSS)
  • You can use standard Azure VM images.
  • Or any VM image you create yourself and place in an Azure Image Gallery.
• You can pick the VM size and performance, you are not limited to the fixed performance of the Microsoft hosted pool agents
• You can choose how your pool scales
  • Maximum number of agent VMs
  • Minimum number of agent VMs ready to accept work (can be zero to minimise cost, more VMs are created on-demand with a short start up delay)
• Maybe most important - You can join your MDP to your own Azure VNET, so your VMs can access your private Azure resources (avoiding the need to deploy self managed agents for deployment jobs)
• You only pay for cost of the running agent VMs
• They are licensed using your on-premise agent parallel job licenses

We have been lucky enough to be part of the private preview and have been able to test them out, with great success.

Who are MDPs for?

People who need to connect to private resources on a private VNET

There are many use cases for MDPs, but the one I think we will see most is people who want the ease of hosted agents, but need to connect to private resources on a private VNET to do part of their CI/CD process.

This has been a big win for us, as we, and our clients, have a lot of private resources in secure hybrid Azure architectures.

People who need to scale up their agents

MDPs will also allow us to consider how we will replace our on-premise agents when their host Hyper-V servers reach their end of life.

It will be much easier to manage and scale MDP based agents, as there is no need to rebuild the on premise images every couple of months; each new MDP agent can get an up to date Microsoft hosted agent image upon startup.

For this use case, any decision will be based on the cost of the MDP VMs vs the cost of the time to maintain the images and run them on-premises. From initial estimates we think it will be a close call, we need to do some more detailed analysis as to the actual utilisation of the agents, how many are actually active at any given time, and how long they are active for. However, a key factor will be that MDPs mean we can scale down the number of VMs to zero, so we only pay for what we use.

But we have to remember everyone’s milage will vary as to whether MDPs are cheaper than running on-premise agents over a multi-year purchase cycle, but my suspicion is that for many the MDPs will be the cheaper option.

Summing Up

So I urge you irrespective of how you use Azure DevOps build agents to take a look at the Managed DevOps Pools. They could be a game changer for you in both functionality and cost management.

“TF255146: The SQL Server instance you specified (nnnnn) is version ‘SQL Server 2017 RTM’, which is not supported by this version of Azure DevOps Server. For more information about supported versions of SQL Server, visit https://www.visualstudio.com/docs/setup-admin/requirements "

This issue has been detailed on the Visual Studio Developer Community site, and Microsoft have retrospectively updated the product documentation.

However, it is fair to say Microsoft have not done a great job with the messaging here

• Unusually, there was no prior announcement that the system requirements were changing with a minor .X update. In the past supported version changes have been made only with a major update.
• There was also nothing in the 2022.2 official release notes about the SQL change. Only, belatedly, in the primary product documentation, a place easily missed when doing an upgrade.
• The information in the primary documentation is only partially correct. It states SQL 2017 does not work with 2022.X, but it does for 2022 RTM and 2022 Update 1. This is confusing.

So the key point to take away is that any future updates from Azure DevOps 2022.1 will require a migration to a newer SQL version than 2017.

You will want to consider this SQL change if you are planning to migrate to Azure DevOps Services as only the current and current -1 version of Azure DevOps Server are supported, hence this SQL version limitation will be important.

The key point is that the clock is ticking before the migration tools from 2022.1 are retired and you are forced to move to 2022.2 or later prior to yur migration. On past update history my guess is you have around 6 months before this occurs.

So if a migration is on your horizon, and you are on Azure DevOps Server 2022.1 you might want to start planning it now.

Yesterday I posted about converting ARM output variables to Azure DevOps pipeline variables

Whilst using the pattern I discussed we hit an interesting problem. On my test pipeline I had the following YAML and it was working as expected.

- task: PowerShell@2
  displayName: Obtain Azure Deployment outputs
  inputs:
    targetType: 'inline'
    script: |
      if (![string]::IsNullOrEmpty( $env:deploymentOutputs )) {
        $DeploymentOutputs = convertfrom-json $env:deploymentOutputs

        $DeploymentOutputs.PSObject.Properties | ForEach-Object {
            $keyname = $_.Name
            $value = $_.Value.value
            Write-Host "The value of [$keyName] is [$value]"
            Write-Host "##vso[task.setvariable variable=$keyname]$value"
        }
      }      

However, on the first production project I tried this on, the script ran but did not create the expected variables. The issue was that the variable $env:deploymentOutputs was empty, even though the ARM deployment had completed successfully and the outputs were available in the pipeline debug logs.

The Solution

Turns out the problem was the type of pipeline agent being used. The test pipeline was using a Windows agent, the production pipeline an Ubuntu agent.

The fix was to swap to use the $(deploymentOutputs) syntax, as opposed to the $env:deploymentOutputs syntax, as shown below

- task: PowerShell@2
  displayName: Obtain Azure Deployment outputs
  inputs:
    targetType: 'inline'
    script: |
      if (![string]::IsNullOrEmpty( '$(deploymentOutputs)' )) {
        $DeploymentOutputs = convertfrom-json '$(deploymentOutputs)'

        $DeploymentOutputs.PSObject.Properties | ForEach-Object {
            $keyname = $_.Name
            $value = $_.Value.value
            Write-Host "The value of [$keyName] is [$value]"
            Write-Host "##vso[task.setvariable variable=$keyname]$value"
        }
      }

Interestingly, I had assumed the issue was actually whether PowerShell or PowerShell Core was being used, but this was not the case. Both versions of PowerShell correctly resolved the $env:deploymentOutputs syntax on a Windows agent.

So it was the agent OS type that was the issue. Yet another strange quirk of Azure DevOps pipeline variable expansion.

Historically, we have used Kees Schollaart’s ARM Outputs Azure DevOps task to convert the output from an ARM template deployment into a variable that can be used in a subsequent Azure DevOps pipeline task, using the general form

- task: AzureResourceManagerTemplateDeployment@3
  displayName: Deploy the main template
  inputs:
    deploymentScope: 'Resource Group'
    azureResourceManagerConnection: 'ARMConnEndpoint'
    subscriptionId: '$(SubscriptionId)'
    action: 'Create Or Update Resource Group'
    resourceGroupName: '$(ResourceGroup)'
    location: '$(AzureRegion)'
    templateLocation: 'Linked artifact'
    csmFile: '$(Pipeline.Workspace)/ARMtemplates/azuredeploy.json'
    overrideParameters: >-
      -staticSitelocation "westeurope"
      -projectName "$(projectName)"
      -env "$(environment)"
    deploymentMode: 'Incremental'

- task: ARM Outputs@6
  displayName: Obtain outputs from the deployment of the Main Deploy
  name: 'MainDeployOutput'
  inputs:
    ConnectedServiceNameSelector: 'ConnectedServiceNameARM'
    ConnectedServiceNameARM: 'ARMConnEndpoint'
    resourceGroupName: '$(ResourceGroup)'
    whenLastDeploymentIsFailed: 'fail'

This process has been working well until we upgraded our service connections to workload identity federation. As soon as we did this the ARM Outputs@6 task started failing with the error message

Logging in using ApplicationTokenCredentials, authScheme is ‘WorkloadIdentityFederation’
Unhandled exception during ARM Outputs Task Error: secret must be a non empty string.

A quick look on the task’s GitHub repository showed we were not alone in seeing this error, but also that the task had not been updated in a good while.

To Fork or not to Fork

This task, like so many others in the Azure DevOps marketplace, has ceased to be under active support.

Our first thought was to fork the repository and fix the issue ourselves. However, after a bit of research we found the fix was non-trivial and would probably need a complete swap out of the authentication libraries to update to ones that supported WorkloadIdentityFederation as the existing libraries were deprecated.

However, more than the work involved, there was also the issue of what to do with any fix. We could see that, as the task was no longer under support, no PRs had been merged into the original repo for a number of years. So we had no mechanism to get our fix out to the community via the original GitHub repo and the Azure DevOps Marketplace listing. Our only option was republishing the task under a new ID. Not an easily discoverable solution or desirable as this confuses the marketplace and makes it harder for users to find the right task.

Also we did not really have the appetite to maintain the fork of the task, and keep it up to date with the latest changes in the Azure DevOps API. This lack of support commitment is just the problem that got us to the current position.

The Solution

The solution was a pattern we find ourselves using more and more i.e. Don’t create an Azure DevOps task and publish it via the Azure DevOps Marketplace, but rather use a generic script task and some BASH/PowerShell to get the job done within the code the team has direct access too in their repos. Whether it be a project repo or some shared internal repo that is used by many projects.

So the above pipeline was replaced with the following

- task: AzureResourceManagerTemplateDeployment@3
  displayName: Deploy the main template
  inputs:
    deploymentScope: 'Resource Group'
    azureResourceManagerConnection: 'ARMConnEndpoint'
    subscriptionId: '$(SubscriptionId)'
    action: 'Create Or Update Resource Group'
    resourceGroupName: '$(ResourceGroup)'
    location: '$(AzureRegion)'
    templateLocation: 'Linked artifact'
    csmFile: '$(Pipeline.Workspace)/ARMtemplates/azuredeploy.json'
    overrideParameters: >-
      -staticSitelocation "westeurope"
      -projectName "$(projectName)"
      -env "$(environment)"
    deploymentMode: 'Incremental'
    deploymentOutputs: deploymentOutputs # need this parameter adding to get the outputs as a JSON string in an environment variable

# Then process the outputs from the ARM deployment and promote them to pipeline variables
- task: PowerShell@2
  displayName: Obtain Azure Deployment outputs
  inputs:
    targetType: 'inline'
    script: |
      if (![string]::IsNullOrEmpty( '$(deploymentOutputs)' )) {
        $DeploymentOutputs = convertfrom-json '$(deploymentOutputs)'

        $DeploymentOutputs.PSObject.Properties | ForEach-Object {
            $keyname = $_.Name
            $value = $_.Value.value
            Write-Host "The value of [$keyName] is [$value]"
            Write-Host "##vso[task.setvariable variable=$keyname]$value"
        }
      }

Updated 23rd July In the inline PowerShell the $(deploymentOutputs) variable syntax is required, as opposed to the $env:deploymentOutputs syntax, if you are using an Ubuntu agent. So it is probably best to always use the $(deploymentOutputs) syntax to ensure cross platform compatibility.

This solution works, is maintainable and is under our control. Hope the same YAML unblocks some other teams out there.

I have previously blogged a good deal on Azure DevOps variable evaluation, see here and here, but the saga continues…

When variables exist

Today I realised, something I guess should have been obvious, that when you manually queue a run, pre-defined variables such as $(Build.SourceBranchName) are not available until the pipeline is compiled and starts running. This is because though there is a value in UI branch combo but this value is not in $(Build.SourceBranchName) until the pipeline starts running.

This would seem unimportant, and it is, except in the case when you are manually picking which stages to run via the run pipeline dialog. Only stages that have no conditional expressions, or ones where the $(Build.SourceBranchName) name is empty are shown in the list of stages to run.

Updated 22 July 2024 - Rik Hepworth pointed out to me that a good use of the Stage Picker UI is to do a ‘richer’ validation of the YAML pipeline.

This is useful as though the Azure DevOps UI does a validation of any YAML edits, it only shows compile time issues, mostly typos and indenting issues, it does not validate any runtime values.

Rik’s suggested that he had found loading the Stage Picker UI is a good way to do further validation of “some of the runtime values”. This is a good suggestion, it is not perfect validation, but allows you to do a bit more checking before queuing a run.

trigger: none

pool:
  vmImage: ubuntu-latest

stages:
  - stage: No_Conditions
    jobs:
      - job: Job1
        steps:
        - script: echo 'Running on $(Build.SourceBranchName)'

  - ${{ if eq(variables['Build.SourceBranchName'], 'main') }}:
    - stage: BranchName_is_main
      jobs:
        - job: Job2
          steps:
          - script: echo 'Running on $(Build.SourceBranchName)'

  - ${{ if or(eq(variables['Build.SourceBranchName'], 'main'), eq(variables['Build.SourceBranchName'], '')) }}:
    - stage: BanchName_is_main_or_empty
      jobs:
        - job: Job3
          steps:
          - script: echo 'Running on $(Build.SourceBranchName)'

  - stage: Using_Conditions
    condition: eq(variables['Build.SourceBranchName'], 'main')
    jobs:
      - job: Job4
        steps:
        - script: echo 'Running on $(Build.SourceBranchName)'

Remember if you wish to if a {{if }} expression to actually filter on a branch name you need to use an Or to handle both the UI and runtime conditions

Basically, you cannot rely on the value of Build.SourceBranchName and -{{ if }} expressions to trim the list of available stages in the run build UI. In my opinion, best to avoid even trying to do this.

What can I do with conditions?

If you want to be able to pick the stages to run in the UI, and need checks against items like branch names, the simple answer is to use conditions on the stages. These are only evaluated at the moment the stage is started, unlike expressions which are evaluated when the run is queued.

But how can we give users an indication of what will be run, or not, as soon as the run is started as opposed to only finding out as it completes?

Variables and expressions are the answer here. Using expressions we can set a variable based on the branch name at queue time. This can be used in a displayname for the stage so in the UI it shows if a stage will be run or not

variables:
  - ${{ if eq(variables['Build.SourceBranchName'], 'main') }}:
    - name: willdeployToProdString
      value: Yes
  - ${{ if ne(variables['Build.SourceBranchName'], 'main') }}:
    - name: willdeployToProdString
      value: No
stages:
   # other stages shown above left of for clarity of the example

  - stage: Using_Conditions
    displayName: Will deploy ${{variables.willdeployToProdString}}
    condition: eq(variables['Build.SourceBranchName'], 'main')
    jobs:
      - job: Job4
        steps:
        - script: echo 'Running on $(Build.SourceBranchName)'

Remember if you are skipping steps with conditions, you may need to set the condition on subsequent stages to run. The default (if no condition is set) is to only run on success of the previous stage. See the official docs for more details on your options

Conclusion

As I have written about before, use compile time expressions in Azure DevOps Pipelines with care, their evaluation can get complex.

So a bit of a niche blog post, hope it is of use to someone

Whilst working on an Azure DevOps YAML pipeline for a solution that used Git Submodules we hit a problem with the checkout of the repo and submodule using the YAML

jobs:
  - job: 'Build'
    steps:
    - checkout: self
        submodules: true

It got the main Git repo, but failed with the following error

git submodule sync
git --config-env=http.https://myorg@dev.azure.com.extraheader=env_var_http.https://myorg@dev.azure.com.extraheader submodule update --init --force
Submodule 'Library' (https://myorg@dev.azure.com/myorg/myproject/_git/Library) registered for path 'Library'
Cloning into 'D:/a/1/s/Library'...
remote: TF401019: The Git repository with name or identifier Library does not exist or you do not have permissions for the operation you are attempting.

The Analysis

The issue is that the build agent access token was scoped to only the repo containing the YAML pipeline and not the submodule repo, even though they are in the same Azure DevOps Team Project.

This was not because of a lack of permissions for the build service account, but that the Azure DevOps Team Project pipelines setting ‘Protect access to repositories in YAML pipelines’ was enabled. This setting restricts the access of the build agent to only the repo containing the YAML pipeline, or ones referenced explicitly in the pipeline.

The Solution

The obvious solution is to disable the ‘Protect access to repositories in YAML pipelines’ setting for the Team Project. However, this may not be possible due to security requirements. In our case this setting was being enforced at the Team Project Collection level and we could not alter it.

But luckily there is a workaround.

1. In the YAML reference the Git Submodule and use the checkout task to clone into some folder (the folder is unimportant as we will never actually use it).

2. You can then call the main checkout with the submodules parameter set to true. This will now work because the agent access token is now scoped to both the repo the YAML is in and the submodule you manually referenced

name: $(Build.DefinitionName)
 
trigger:
  branches:
    include: [ main ] 
 
resources:
  repositories:
    - repository: Library # the submodule
      type: git
      name: Library
 
pool:
  vmImage: windows-latest
 
stages:
  - stage: 'Build_Packages'
    jobs:
      - job: 'Build'
        steps:
        - checkout: Library # the throwaway checkout
          path: './s/SomeTmpPath/Library'
 
        - checkout: self # the main checkout
          submodules: true

Note: You can’t seem to get away with just adding the reference, the explicit checkout of the referenced submodule repo is required, else you still get the permissions error.

So, a hacky solution, but it works!

I recently upgraded a clients Azure DevOps server from 2019 to 2022. This required a new application tier VM due to the change in supported versions of Windows Server between the two versions.

Unfortunately, the client’s developers had always accessed the old Azure DevOps Server using the VMs FQDN, as opposed to a DNS managed alias. Hence, given they wanted to minimise change, the plan was to create a DNS Alias so they could continue to use the same URLs and TFVC workspaces mappings.

The upgrade went well, and once complete I could access the upgraded server using the URLs http://localhost:8080/tfs or http://newserver:8080/tfs.

However, once the DNS Alias (an A record) was added and we tried to access http://oldserver:8080/tfs the developers were being shown a login dialog and when they entered their valid credentials were getting a TF30063 error.

The Solution (well sort of)

I went down a rabbit hole of loopback check settings, but to no avail.

I then tried adding an entry in the local host file for my new server VM on my test client. Strangely this worked.

So my working assumption was either

1. The DNS Alias was not working correctly, maybe a CNAME record is required as opposed to an A record?
2. As the DNS Alias is replacing what was a ‘real’ server name, had we missed some special settings?

To work through these ideas, I got a new CNAME added to the DNS ‘devops’ that pointed to the new server (arguably a CNAME that should have been in use all along). For some reason this DNS change took much longer to propagate than the A record, but once it did the TF30063 error had completely gone. I could now access the new server using the URLs

• http://localhost:8080/tfs
• http://newserver:8080/tfs
• http://oldserver:8080/tfs (the DNS A record)
• and http://devops:8080/tfs (the DNS CNAME)

So as usual the problem was DNS, I assume due to DNS propagation delays and cache timeouts.

Why is it always DNS?

Last week I noticed that the staging URL that is normally output as a comment was missing from new GitHub PRs. Previously, this URL was added automatically by the Azure/static-web-apps-deploy GitHub Action for PRs in our Hugo based websites.

After a bit of digging, I noticed a warning message in the logs of the Action that said:

…
Done Zipping App Artifacts
Uploading build artifacts.
Finished Upload. Polling on deployment.
Status: InProgress. Time: 0.178533(s)
Status: Succeeded. Time: 15.3731517(s)
Deployment Complete :)
Visit your site at: https://white-glacier-0d2380f03-300.westeurope.2.azurestaticapps.net Unexectedly failed to add GitHub comment.
Thanks for using Azure Static Web Apps!
Exiting

The Solution

Initially I thought the problem might be a change in functionality of the Azure/static-web-apps-deploy action. However, it turns out it has not altered since May 2021.

So next I tried to add my own PR comment using the actions/github-script action

- uses: actions/github-script@v6
  if: github.event_name == 'pull_request'
  with:
    script: |
      github.rest.issues.createComment({
        issue_number: context.issue.number,
        owner: context.repo.owner,
        repo: context.repo.repo,
        body: 'Azure Static Web Apps: Your staging site is ready at: ${{ steps.builddeploy.outputs.static_web_app_url }}'
      })

This failed with a 403 error, so I realised my problem was missing permissions. So added a permissions block to the job

jobs:
  build_and_deploy_job:
    if: github.event_name == 'schedule' || github.event_name == 'push' || (github.event_name == 'pull_request' && github.event.action != 'closed')
    runs-on: ubuntu-latest
    permissions:
      contents: read   # This is required read the repo
      pull-requests: write  # This is required to comment on the PR
    ...

Note: As soon as you set any permissions you have to set all the ones you need, as setting a permission removes the defaults. So in this case, if you just set the pull-requests: write permission but not the contents: read permission, the workflow would not be able to clone the repo

This worked, but then it occured to me, was the original error just permissions related?

So I removed the actions/github-script action but left the permissions block and as I hoped the staging URL appeared in the PR comment.

So my assumption is that default permissions have recently changed. It just shows it is always a good idea to be explicit with permissions in your GitHub Actions workflows.

If you forget you can expect to waste plenty of time, like we did last week, with environment agents not picking up queued jobs with no diagnostic logging messages to give you a clue as to what is going on.

This issue has been reported on StackOverflow where it was pointed out that the official documentation used ‘deployment’ as the stage name. The good news is that at least the documentation is now fixed, but there is still the chance you can make this naming mistake all on your own.

So, future me, I hope this post save you some time.

I just upgraded our Azure/Docker container hosted SonarQube instance from 10.5.1 to 10.6.0. This was partly due to our usual upgrade process, we try to upgrade within a week or so of a new release, but also to address a specific Java issue we started to see when using the SonarQube@6 Azure DevOps tasks.

The error was the SonarQube analysis completed successfully, but the clean up process failed with this JVM error.

INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 4:20.150s
INFO: Final Memory: 55M/194M
INFO: ------------------------------------------------------------------------
##[error]ERROR: Cleanup during JVM shutdown failed
ERROR: Cleanup during JVM shutdown failed
##[error]java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError

The Issue

The upgraded appeared to go well

• The container restarted
• The DB schema was updated
• The SonarQube instance started and became available.
• As expected, a background re-index started on our 89 projects

88 were re-indexed without issue, but one failed with the following error.

Error Details: A Customer Function [Project Data Reload]
Error Details
java.lang.IllegalStateException: Unrecoverable indexing failures: 7 errors among 7 requests. Check Elasticsearch logs for further details.

The Elasticsearch logs showed nothing more.

Solution

A search of the ever useful SonarSource Community forums suggested this error can be seen when there is a lack of disk space. However given we are running on Azure Storage and are using a tiny fraction of our available 100Tb of disk space, that did not seem to be the likely issue.

So I adopted my usual approach for SonarQube issues involving Elasticsearch. I stopped the container, deleted the index sonarqube-data/es6 folder, restarted the container and let SonarQube rebuild the index to see of this fixed the problem.

An as is so often the case, this fixed the issue, all the projects were re-indexed successfully.

I have an Azure DevOps YAML template that does some deployment actions using PowerShell. This template is used multiple locations. The problem is that the PowerShell step within the template needs a variable number of environment variables, set from values stored in an Azure DevOps variable Group. The number of variables is not fixed because they depend on the underlying commands the PowerShell script is triggering to do the deployment.

Hence, the problem was how to pass the values of these variables to the template in a way that was easy to maintain and understand, as this is not something that is well documented.

The Solution

The solution, as you might expect if you have used YAML templates, is to use the object type for the parameter in the template. This allows you to pass a single object to the template that contains all the values you need.

This can be seen in this simple example

parameters:
  - name: envparams
    type: object
    default:

steps:
- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: |
      Get-ChildItem -Path Env:
  env:  ${{ parameters.envparams }}

The question then becomes, how to you wire this up in the calling pipeline?

The answer is you pass a JSON block to the template. Just make sure you get the escaping correct, I have found this is the most common reason for a pipeline to fail to run.

trigger:
- main

pool:
  vmImage: ubuntu-latest

variables:
  # Link to the variable group
  - group: ExpandTest
steps:
- template: base.yml
  parameters:
    # match the variable group values with name of the environment variables you wish to create
    envparams: {
      'MY-PARAM1': '$(MY-PARAM1)',
      'MY-PARAM2': '$(MY-PARAM2)'
    }

However, today I was surprised to find I was getting cache misses on my pipeline, even though I was sure the cache should have been hit.

There are rules over how the cache is used:

• The cache is specific to a pipeline definition, so there is no sharing of the cache between pipeline definitions
• The cache is only created if the pipeline is successful (running the post run tasks)
• The cache only lasts 7 days
• but what I had not realised was the cache is also specific to the branch in a not so obvious way.

My pipeline was triggered off a PR, so the cache was being created on the ‘branch’ PR #123. This was working as expected, all runs of the PR triggered build used the cache after the initial run. However, if I manually triggered pipeline run of the same branch as the PR was using, there was a cache miss.

As far as the Azure DevOps cache task is concerned, the branch used for PR #123 and a manual run off the underlying branch used in the PR are different things, so the cache created by the PR was not being used by the manual branch build.

That explains a few slow builds I have had.

The Global DevOps Experience is a day full of learning and fun, as we immerse you in the world of DevOps and AI. You will learn about the latest trends and technologies, and work in a team to solve challenging exercises based around a realistic business scenario.

You can register for free on the GDEX site, hope to see you on the 15th

Whilst migrating some Azure DevOps classic pipelines to templated multi-stage YAML, I hit a problem that a job running on a self-hosted agent would not start.

The YAML stage, which required approval, would be queued and approved, but the agent would just sit there with the message Starting job but would never start. The strange thing was even though the job was not started, the pipeline instantly showed as failed with no error message.

The Solution

After much fiddling, as there was no diagnostic information in the logs, I found the issue was a YAML syntax error, I had made an error that not want picked up at queue time or run time.

I had written in a template file

steps:
- download: self

When it should have been

steps:
- download: current

Once I fixed this the jobs started as expected. I am not sure why this issue did not give a clearer error message, but hope this post save someone else a bit of time.

Whilst updating a client’s Duende Identity Server from versions 6 to 7, we experienced a problem. We followed the upgrade steps and all was working fine against our development instance i.e. the Identity Server Db was upgraded to the current schema and we could login from our test MVC web client without issues.

The problem occured when we started to test using our UAT (a production replica) DB. On loading the Identity Server, it did the expected EF migrations and appeared to start, but when a client tried to connect we got an exception in the Identity Server logs in the form

OPENJSON - Incorrect syntax near the keyword '$'

The Solution

The issue it turns out was the SQL DB Compatibility Level as OPENJSON requires Compatibility Level 130. The UAT (and production) DB’s compatibility level had not been updated when the SQL server instance was upgraded, but it had not been an issue until now.

Once the DB compatibility level was updated, the Identity Server started working as expected. We did this via SQL Management Studio, right-clicking on the DB, selecting properties, and then setting the compatibility level to 150 (SQL 2019), but you could use the command

ALTER DATABASE IdentityServer SET COMPATIBILITY_LEVEL = 150

Within one of our Azure DevOps builds we today started to see the following error when running the SonarQube analysis step:

##[error][SQ] Task failed with status FAILED, Error message: Unrecoverable indexing failures: 1 errors among 1 requests. Check Elasticsearch logs for further details.

Nothing obvious pointed to why this should have started to occur, the SonarQube logs showed nothing more than a longer version of the same message.

The Solution

The solution, as it has been previously, was to clear the SonarQube index.

This can be done by deleting the ES8 folder from the data folder of the SonarQube installation. In my case this was withi an Azure storage account.

In past years, pre Covid, Black Marble hosted a venue for the Global DevOps Bootcamp, a community run in person hackathon event.

After a short hiatus, I am pleased to be able to say this event has a successor, the Global DevOps Experience which is being run on the 15th of June at many venues around the world, including Black Marble’s offices.

What will be involved?

The Global DevOps Experience is a day full of learning and fun, as we immerse you in the world of DevOps and AI. You will learn about the latest trends and technologies, and work in a team to solve challenging exercises based around a realistic business scenario.

• Learn about the latest trends. We will offer you an environment to learn about the latest technologies and trends.
• Get help. We make sure you don’t get stuck. It is all about learning and fun, so proctors will be around and the system will guide you.
• Work in a team. We will make sure you are able to work together in a team of peers.
• Learn about GitHub. Experience a fully provisioned GitHub environment and see how you can benefit from all its features.
• Explore the use of AI. Both within the develop tools, and from a developer point of view.
• Have fun. Most importantly, have fun by working together on challenging exercises.

How do you get involved?

As this is a free event, all you have do do is register for the Black Marble venue and turn up on the day.

Or, of course, you can register for any other venue around the world

Hope to see you on the 15th

We have a number of projects that use the old style SQL Server Reporting Services (SSRS) .RPTProj project format. These projects are not supported in Visual Studio 2022 out of the box, but there is an extension in the Marketplace that adds the functionality back so you can build them in the IDE.

However, we want to build our RDL files as part of our CI process, and this is where we hit a problem. When we attempt a build with MSBuild it fails with an error about missing .NET 4.0 SDK/Targeting Pack.

"C:\projects\src\Reports.sln" (default target) (1) ->
"C:\projects\src\Reporting\Reporting.rptproj" (default target) (2) ->
(GetReferenceAssemblyPaths target) ->
  C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(12
29,5): error MSB3644: The reference assemblies for .NETFramework,Version=v4.0 were not found. To resolve this, install th
e Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Fram
ework Developer Packs at https://aka.ms/msbuild/developerpacks [C:\projects\src\Reporting\Reporting.rptproj]

    0 Warning(s)
    1 Error(s)

Interestingly, if we try to build from the command line on a developer PC we get the same error, so it is not that the reporting service VS extension is not installed on the build agent, it is that the required MSBuild targets are not installed.

The Solution

You would assume the solution would be to install the missing SDK/Targeting Pack on the agent. However, there is no C# compilation in our .RPTProj files, so why do we even need the SDK/Targeting Pack?

The fix was to just update the .RPTProj file with a TargetFrameworkVersion>v4.8</TargetFrameworkVersion> block for each <PropertyGroup Condition= .... i.e. to tell MSBuild to use the .NET version installed.

  <PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
    <FullPath>Release</FullPath>
    <OutputPath>bin\Release</OutputPath>
    <ErrorLevel>2</ErrorLevel>
    <OverwriteDatasets>False</OverwriteDatasets>
    <OverwriteDataSources>False</OverwriteDataSources>
    <TargetServerVersion>SSRS2008R2</TargetServerVersion>
    <Platform>Win32</Platform>
    <TargetReportFolder>Reporting</TargetReportFolder>
    <TargetDatasetFolder>Datasets</TargetDatasetFolder>
    <TargetDatasourceFolder>Data Sources</TargetDatasourceFolder>
    <TargetReportPartFolder>Report Parts</TargetReportPartFolder>
    <TargetFrameworkVersion>v4.8</TargetFrameworkVersion>
  </PropertyGroup>

Once this was done the command line build worked fine on both a developers PC and Microsoft hosted build agent. So, avoiding the need to install an out of support version of .NET.

One of my client’s recently had a problem that a large number of Azure DevOps work items had been updated via Excel in error.

They asked if there was a means to undo these edits. Unfortunately, a feature Azure DevOps does not provide.

Solution

So, I wrote a PowerShell script to do it. The script…

1. Gets a list of work items edited today by a specific user, the one who did the problematic bulk update, using a Work Item Query
2. Get the last update of each work item and check it was made by the user who did the bulk edit, incase someone manually fixed the problematic update already.
3. For a limited list of fields, revert the change to the value prior to the last update
4. Save the updated work item, or if the -whatif flag is set just validate the update against the Azure DevOps instance

For my client, the script worked well enough, reverting over 1000 work items in about 5 minutes. The few work items it could not revert were fixed manually.

The common factor in the work items that it could not revert was that they all had rich text/HTML based descriptions, though so did many that successfully reverted.

I suspect there is an edge case related to the encoding of some character(s) content. However, I have not been able to reproduce the problem as yet on my test rig.

The good news is that if the revert of a work item does fail, the target work item is left unchanged. So allowing either repeated revert attempts with an updated version of the script, or for a manual fix can be done.

You can find the PowerShell script here as a Gist

Programmatic connection to Azure DevOps cannot be done with your Active Directory credentials. This is because this involves a dialog being shown, and these days usually an MFA check too.

Historically, the solution to this problem was to enable Alternate Credentials, which could be passed as username and password, without the dialog being shown. However, the use of these has been deprecated since 2020, and they have been completely removed since Jan 2024.

So for a number of years the recommended way to programmatically authenticate with Azure DevOps has been to use a Personal Access Token (PAT). However, PATs are not without their own issues.

The problem with PATs

The most critical problem is that PATs expire. By default they expire after 30 days, and the longest you can set them to expire after is a maximum of 1 year. This means that you have to remember to renew them, and if you forget then your automation that rely on them will stop working.

Obviously it is very tempting to just set the expiry to 1 year, but this is not the best idea. The longer the expiry the more time there is for the PAT to be compromised. Just like SSL certificates, or any access token, the shorter period before the expiry the better.

Of course like any authentication and permission model you can grant too many permissions to a PAT, and then if it is compromised you have a problem, but that is on you not following the principle of least privilege, not a fundamental issue of PATs.

But what is the alternative?

Azure DevOps now offers support for Managed Identity & Service Principals. These are a much better way to provide a means to programmatically authenticate. Critically, they do not expire, and the tokens they generate for the actual connections are short lived, and you can of course grant them privileges in line with the principle of least privilege.

When used to access Azure DevOps resource, the only downside is they consume a basic user license.

This form of authentication is perfect for use in automation from locations like Azure Functions, or from the Az CLI.

What about GitHub?

The issue of PATs is not just one for Azure DevOps, the same problems are true for GitHub. On GitHub the solution is to use a GitHub App to authenticate with the GitHub API, and then use the short term token it can generate to authenticate with the GitHub API.

What about other outgoing services from Azure DevOps?

Another place you will see a move away from expiring secrets in Azure DevOps are Service Connections to services to such as Azure. You can now use Workload Identity Federation to authenticate with Azure.

Converting to use Workload Identity Federation is simple, you just press a button in the Azure DevOps UI, and it will create a Service Principal in Azure AD, and then update the Service Connection to use this to authenticate with Azure.

Once this is done there will be no need to remember to renew the Service Connection, as it will use the Service Principal to authenticate, and this does not expire.

Summary

So, when using Azure DevOps in many cases you can now avoid the use of PATs, and hence the need to remember to renew them. Giving you one less thing to remember to do, and one less thing to go wrong.

Team Foundation Version Control (TFVC) has been around since 2005, since the first release of Team Foundation Server (TFS) in 2005. In 2013, over 10 years ago, Microsoft added Git support to TFS (later renamed as Azure DevOps), Git had already been around for 8 years at that point.

10 years is a long time in the software industry, I always think of ‘IT years’ like ‘dog years’ i.e. 7 to 1, so over 70 years has passed, a lifetime. Over this period Git has become the de facto standard for source control. So if you are still using TFVC as your source control system you really need to ask yourself why?

The advantages of Git over TFVC

The advantages of Git over TFVC are numerous, but here are a few:

• Git is distributed, meaning you can work offline and commit changes locally
• Git has better branching and merging (pull request) support
• Many newer features of Azure DevOps are Git only i.e YAML based pipelines
• Cloud services like Azure Functions, Azure App Service, Azure Logic Apps, Azure API Management etc. all have built-in support for Git
• Git repositories are portable, they are easy to migrate to new systems e.g GitHub, GitLab, Bitbucket etc.
• Any new hires/students will probably have Git experience

From talking to clients who are still on TFVC, the main reason they are still using TFVC is just because they have always used it. There is a fear a change, something you might not expect to see in the technology sector, but all too common.

This fear might be wrappered in the perceived complexity of migrating from TFVC to Git, or that an audit or regulatory requirement means they can’t change processes. But these are just excuses. Updates to process and tooling cannot be ignored, you don’t have to be the first to move, but the longer you leave it the harder it will be to change.

As an example, one of the most obvious limitations for me of TFVC is how it works with Azure DevOps Pipelines. It is true you have to use the ‘classic’ pipelines, which are not as flexible as the YAML based pipelines, and have reached a ‘done’ state and so will get no further development. However, more me the big limitations are not the editor experience but that:

• For each TFVC branch you have to create/clone a new pipeline
• You cannot make CI/CD part of your branch protection
• You cannot source control your pipeline definitions and review them via a PR

These reasons alone should be enough to make you want to move to Git from TFVC.

So how can I migrate?

You have three basic options to migrate from TFVC to Git:

• ‘Tip’ migration: This is the simplest, you just migrate the latest version of your code to a new Git repository. This is the quickest and easiest, but you lose all the history of your code.
• Azure DevOps Importer: This is a tool that can be used to import a TFVC repository into a new Git repository, with the last 180 days of history. Why you would want neither none of the history or all of the history has always been beyond me!
• Git TFS: A command line extension for Git that can be used to migrate a TFVC repository to a new Git repository, with all the history. A powerful and flexible tool, but it can be complex to use, and is ‘brittle’ in that it can fail if the TFVC repository has a complex history of branching and renames.

I would always favour the first option. It is the simplest and quickest, and you can always keep the old TFVC repository around for historical purposes. Also remember Git is not TFVC, so what is a good structure in TFVC might not be the best in Git, so you might want to take the opportunity to restructure your codebase.

The other important factor to consider in any migration is that of training. You need to make sure your team is comfortable with Git before you start the migration. This is not just about the commands, but also the concepts of distributed source control, and the different branching and merging strategies that Git enables.

A TFVC to Git migration will be a major change and should be considered as a project in its own right, with planning, training, pilot studies as well as the main migration itself. But remember you don’t have to move all your TFVC source control to Git at once, you can do it incrementally, maybe as you start a new version of a product.

Summary

So if you are still using TFVC I urge you to consider migrating to Git. It is not as hard as you might think, and the benefits are numerous.

Happy to discuss this further, please reach out to me via the usual channels

Since reading this I have made much more of an effort to keep my Azure DevOps Extensions up to date. Dependabot generated PRs have been a great help in this regard. Creating PRs for vulnerabilities and out of date dependencies.

However, no level of AI can protect you from stupidity. Recently, as part of addressing a vulnerability via a refactoring to change testing framework, I wasted far too long trying to work out why all my test passed in my Codespace, but failed in the CI build. I had forgotten to commit the jest.config.js file as my .gitignorewas set to ignore all .JS files as I was working in TypeScript.

Very frustration, but a good reminder to always check the files you are committing are the ones you expect.

We saw an issue with our SonarQube 10.3 Developer Edition (that is running as a Docker image hosted in Azure) when it was doing the analysis of a project that included Azure Bicep files.

The Azure DevOps pipeline that triggered the SonarQube analysis was not failing, but within the SonarQube analysis step an error was reported in the task log

INFO: Sensor IaC AzureResourceManager Sensor is restricted to changed files only
INFO: 1 source file to be analyzed
##[error]ERROR: Cannot parse 'AzureServices/QueryPack.bicep:89:1'

The Solution

Turns out the problem was related to parsing Bicep files for App Insights Query packs.

If the Bicep resource for the query contains a body property that starts with a comment e.g.

resource querypacks_DefaultQueryPack 'microsoft.operationalInsights/querypacks/queries@2019-09-01-preview' = {
  parent: QueryPack
  name: ...
  properties: {
    displayName: ...
    description: ..
    body: '// 35 is ABC\r\n// 40 is XYZ \r\nrequests\r\n| where name has "myfacade.svc"\r\n| order by timestamp desc\r\n| where name !has "GET"\r\n| summarize count() by name, resultCode\r\n| render columnchart'

We get the error ##[error]ERROR: Cannot parse 'AzureServices/QueryPack.bicep:89:1

We can fix this by not starting the body with a comment, just moving the comment to the end of the body i.e.

resource querypacks_DefaultQueryPack 'microsoft.operationalInsights/querypacks/queries@2019-09-01-preview' = {
  parent: QueryPack
  name: ...
  properties: {
    displayName: ...
    description: ..
    body: 'requests\r\n| where name has "myfacade.svc"\r\n| order by timestamp desc\r\n| where name !has "GET"\r\n| summarize count() by name, resultCode\r\n| render columnchart\r\n// 35 is ABC\r\n// 40 is XYZ'