Background

If you are using staged deployment in Azure DevOps, you will probably have multiple Azure Service Connections. So, it makes sense that you might want to use a Service Connection name that is stored in a variable group as a parameter to a templated YAML pipeline.

# the build pipeline
stages:
  - stage: UAT
    jobs:
    - deployment: ARM_Provisioning
      timeoutInMinutes: 0
      environment: 'Staging'
      variables:
      - group: UAT
      pool:
        vmImage: 'windows-latest'
      strategy:
        runOnce:
          deploy:
            steps:
            - template: YAMLTemplates\ProvisionUsingARM.yml
              parameters:
                AzureResourceGroup: $(AzureResourceGroup)
                AzureServiceConnection: $(AzureServiceConnection)
  - stage: PROD
    jobs:
    - deployment: ARM_Provisioning
      timeoutInMinutes: 0
      environment: 'Staging'
      variables:
      - group: PROD
      pool:
        vmImage: 'windows-latest'
      strategy:
        runOnce:
          deploy:
            steps:
            - template: YAMLTemplates\ProvisionUsingARM.yml
              parameters:
                AzureResourceGroup: $(AzureResourceGroup)
                AzureServiceConnection: $(AzureServiceConnection)

With a template YAMLTemplates\ProvisionUsingARM.yml that uses the AzureServiceConnection variable

parameters:
- name: AzureResourceGroup
  type: string 
- name: AzureServiceConnection
  type: string

- task: AzureResourceManagerTemplateDeployment@3
  displayName: 'Azure Deployment: Create Or Update Resource Group'
  inputs:
    deploymentScope: 'Resource Group'
    azureResourceManagerConnection: '${{ parameters.AzureServiceConnection}}'
    action: 'Create Or Update Resource Group'
    resourceGroupName: '${{parameters.azureResourceGroup}}'
    ...

The Issue

If I declared a pair of variables in the UAT and PROD variable groups with the names AzureServiceConnection and AzureResourceGroup I would expect each stage to pick up the appropriate variable group and use the correct service connection.

However, this is not the case. The pipeline will fail with the following error message when you validate or try to queue the pipeline.

There was a resource authorization issue: “The pipeline is not valid. Job ARM_Provisioning: Step input azureResourceManagerConnection references service connection $(AzureServiceConnection) which could not be found. The service connection does not exist, has been disabled or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."

Basically, the $(AzureServiceConnection) variable is not being expanded during template validation.

Analysis

Compile and Runtime variables

I tried all the options for the variable declaration in the YAML

  • Standard Macro format $(AzureResourceGroup)
  • Expression format $[AzureResourceGroup]
  • Runtime expression format ${{ variables.AzureResourceGroup}}

None of these helped.

Hardcoding the connection name

If you hardcode the service connection name in the outer YAML pipeline file, the pipeline will work as expected.

# the build pipeline
  - stage: UAT
    jobs:
    - deployment: ARM_Provisioning
      timeoutInMinutes: 0
      environment: 'Staging'
      variables:
      - group: UAT
      pool:
        vmImage: 'windows-latest'
      strategy:
        runOnce:
          deploy:
            steps:
            - template: YAMLTemplates\ProvisionUsingARM.yml
              parameters:
                AzureResourceGroup: $(AzureResourceGroup)
                AzureServiceConnection: UAT
  - stage: PROD
    jobs:
    - deployment: ARM_Provisioning
      timeoutInMinutes: 0
      environment: 'Staging'
      variables:
      - group: PROD
      pool:
        vmImage: 'windows-latest'
      strategy:
        runOnce:
          deploy:
            steps:
            - template: YAMLTemplates\ProvisionUsingARM.yml
              parameters:
                AzureResourceGroup: $(AzureResourceGroup)
                AzureServiceConnection: PROD

But that is not at all what we are after, but it did at least show that it was possible to pass in the service connection name as a string parameter.

Using a global variable

If I declared a global pipeline variable, at the top of the pipeline file, it could be validated and queued without any issues.

variables:
  AzureServiceConnection: PROD

stages:
   ....

But again this was still not what I was after.

The Solution

In my project I have a number of variable groups, one for each stage of deployment. What I found I needed to do was to declare one of them as a global variable for the whole pipeline. This would allow the pipeline to be validated and queued without any issues, but in at each deployment stage this initial value is overridden by the variable group associated with the stage.

This does rely on the fact that the variable groups for each stage contain the same variable names.

The revised YAML looks as follows

# the build pipeline
variables:
# we have to globally declare this variable group, though it is override at the deployment job level
# if we don't do this the $(AzureServiceConnection) variable used by the Azure Resource deployment fails queue time validation 
- group: UAT

stages:
  - stage: UAT
    jobs:
    - deployment: ARM_Provisioning
      timeoutInMinutes: 0
      environment: 'Staging'
      variables:
      - group: UAT
      pool:
        vmImage: 'windows-latest'
      strategy:
        runOnce:
          deploy:
            steps:
            - template: YAMLTemplates\ProvisionUsingARM.yml
              parameters:
                AzureResourceGroup: $(AzureResourceGroup)
                AzureServiceConnection: $(AzureServiceConnection)
  - stage: PROD
    jobs:
    - deployment: ARM_Provisioning
      timeoutInMinutes: 0
      environment: 'Staging'
      variables:
      - group: PROD
      pool:
        vmImage: 'windows-latest'
      strategy:
        runOnce:
          deploy:
            steps:
            - template: YAMLTemplates\ProvisionUsingARM.yml
              parameters:
                AzureResourceGroup: $(AzureResourceGroup)
                AzureServiceConnection: $(AzureServiceConnection)

So I think a valid workaround for another strange YAML variable expansion issue in Azure DevOps.