Updated: 19 Jul 2023 - Revised the post to use Az CLI Task as opposed to a PowerShell Task

Updated: 29 Aug 2024 - Revised the PowerShell as original version was not working. Also see follow up post on using Workload Identity federation

Background

Azure DevOps Pipelines have a built in mechanism to run maintenance jobs on a schedule. This is great for cleaning out old temporary data, but what if you want to run your own maintenance job?

Writing a pipeline to run on a schedule is not in itself difficult. The problem is how to schedule so it runs on all available agents in all available agent pools.

Implementation

There are a number of ways to tackle this, but the way I chose was to use two YAML pipelines and the Az DevOps CLI

The Maintenance Job Pipeline

The contents of the pipeline to run on each agent is down to your requirements. In my case I wanted to run a PowerShell script to update the cached vulnerability databases for the OWSAP Dependency checker.

The key point to note is that I expose a pair of parameters , aliased into variables, to target the pool and agent. This is so I can pass them in from the calling pipeline.

# This is the pipeline that runs any scheduled maintenance jobs
# we wish to run in addition to the built in Azure DevOps Maintenance jobs

# The parameters to target each pool and agent
parameters:
  - name: pool
  - name: agent

# We cannot use to the parameters directly else we get a 'A template expression is not allowed in this context'
# However, if we alias them with a variable they work
variables:
  - name: pool
    value: ${{parameters.pool}}
  - name: agent
    value: ${{parameters.agent}}
 
trigger: none

pool:
  name: "$(pool)"
  demands: Agent.Name -equals $(agent)

steps:
- task: dependency-check-build-task@6
  displayName: "Vunerability Scan Exploited Vulnerabilities update check"
  inputs:
    projectName: 'Maintainance'
    scanPath: '.'
    format: 'HTML'

The Scheduler Job Pipeline

Note: Previously I had run the script using an Azure DevOps PowerShell task, but this had the limitation I had to pass a personal PAT of a user with enough permissions to access the organisation level agent pools using the AZ CLI. This was done using an environment variable injected into the PowerShell.

- task: PowerShell@2
 inputs:
   targetType: 'inline'
   script: |
     # all the script lines     
  displayName: 'Trigger maintainance builds on all active agents'
 env:
   AZURE_DEVOPS_EXT_PAT: $(PAT)

I had hoped to use the $(System.AccessToken), but though this works for some az pipeline commands it returns an empty set when querying the agent pools. I had assumed it was a permissions issue, but couldn’t find the solution. Hence the use of a PAT, until I swapped to this solution using the Az CLI task.

The maintenance pipeline, shown above, is called from a scheduled pipeline, shown below, that runs a script that calls the AZ CLI to find all the agents to target using the Azure CLI Task

# Scheduler Pipeline
variables:
  # filter to limit the scope of agent pools to consider
  - name: PoolNamePrefix
    value: "BM-"
  # the pipeline to run on each agent
  - name: BuildDefintion 
    value: "ScheduledMaintenanceBuild"

trigger: none

schedules:
- cron: '0 0 * * *'
  displayName: Daily midnight build
  branches:
    include:
    - main

pool:
  name: MyAgentPool

steps:
- task: AzureCLI@2
  inputs:
    # The Azure Subscription is the link to a Service Principle
    # with permission to access the agent pools and queue builds
    azureSubscription: 'AzureDevOpsAgentAutomation'
    scriptType: 'pscore'
    scriptLocation: 'inlineScript'
    inlineScript: |
      write-host "Find the agent pools with the prefix '$(PoolNamePrefix)'"
      $Pools = $(az pipelines pool list --organization $(System.TeamFoundationCollectionUri) --query "[? (starts_with(name,'$(PoolNamePrefix)'))].[id,name]" --output tsv)

      write-host "$($Pools.count) found"
      foreach ($pool in $pools) {
          # ugly but works with tsv format data
          $poolSplit = $pool.Split("`t")
          $poolID = $poolSplit[0]
          $poolName = $poolSplit[1]
          
          write-host "Find the agents the pool '$poolName'"

          $Agents = az pipelines agent list --organization $(System.TeamFoundationCollectionUri) --pool-id $PoolID --query "[?enabled].name"  --output tsv 

          foreach ($Agent in $Agents) {
              $buildNameid = $(az pipelines run --organization $(System.TeamFoundationCollectionUri) --name $(BuildDefintion) --project $(System.TeamProject) --parameters "pool=$PoolName" "agent=$Agent" "nvdapikey=$(nvdapikey)" --query "name"  --output tsv )
              Write-host "Queued build $buildNameid on $agent"
          }
      }      
  displayName: 'Trigger maintainance builds on all active agents'

Note: A change required if running this script in the AZ CLI task, as opposed to the PowerShell task, is that you need to specify the --organization parameter. It is not picked up automatically. If this is not done you get the somewhat confusing error

ERROR: TF401019: The Git repository with name or identifier <name of the Git repo> does not exist or you do not have permissions for the operation you are attempting. Operation returned a 404 status code.

For this pipeline to run you need to setup a Service Principle and Service Connection for the Az CLI task to use. This is done as follows

  1. Create a Service Principle in Azure AD and add it to the Azure DevOps organisation as an Azure DevOps Service connection

Note: I used the az ad sp create-for-rbac command to create the service principle, but when I tried to add it as a service connection I got a 404 error about lack of permissions in the subscription. It seems the issue was the new service principle must be granted at least read permissions in the subscription being used to register it in Azure DevOps. This is not done by default

  1. Add the new service principle as a user to the Azure DevOps organisation
  2. Grant the new service principle reader permissions at the organisation Agent Pool level
  3. In the Team Project that contains your maintenance pipelines, in the Pipelines > manage permissions, grant the new service principle the queue builds permissions

Summary

This is a simple way to run your own maintenance jobs on all agents in all pools. It is not the only way, but it is one that works for my current needs.