Introduction

I have recently been swapping some Azure DevOps Pipelines to GitHub Actions as part of a large GitHub Enterprise migration. The primary tool I have been using for this is GitHub Copilot in the new Agent Mode

Frankly, Copilot is like magic, it is amazing how close it gets to a valid solution. I say this after suffering years of undelivered marketing promises of no-code “we won’t need developers in the future”.

Thats said, I have yet to find a Copilot translated GitHub Action workflow that works 100% first time. Interestingly they often run without error, but don’t have the desired results of the source pipeline.

So what have I learnt?

Your prompts will not be as ‘definitive’ as you think.

Many steps done using Azure DevOps tasks have no obvious 1-to-1 equivalent in GitHub Actions, so Copilot generates script blocks to perform the same function.

I was moving an Azure DevOps YAML pipeline that was targeting Ubuntu to GitHub Actions, so not unreasonably the generate scripts were in BASH. The problem for me was that I don’t have that much BASH experience, so making my debugging slower.

The solution in my case was as simple as to add an initial prompt stating any generated scripts should be in PowerShell Core, or as I did, use subsequent prompts to alter the generated scripts.

Note If you always want given prompt to be present, such as to favour PowerShell in a given repo, the you could look at adding a .github/copilot-instructions.md file to define prompts that are added to everything.

Silent errors are the root of many issues

Most of the problems I had with migrated pipelines were the ones you experience in any CI/CD solution i.e. you are in the wrong folder, a filter is wrong, the built URI is incorrect etc.

Using Copilot for this migration, these types of errors seemed to be caused by either

  • the move from Azure DevOps tasks, that may have hidden behaviors or defaults, to scripts
  • or the differences in CLI tools called between the tasks and generated scripts

I good example was the way an archive/ZIP step worked. I ended up creating a ZIP file of static website content with an extra root folder, so all other folders were one level deeper than expected. This deployed without error, but obviously did not work as the old site content had not been overwritten, all the new content was in a sub-folder. A simple problem, but one that took me too lon to realised what the cause was as I got no error messages.

Summary

GitHub Copilot can get you 90% plus of the way to a solution, but as with all coding the ‘devil is in the detail’. That last few percent will take a disproportionate time.

Also there is the danger it generates code you don’t understand, but that is a learning opportunity is it not?

That said, with careful crafting of your prompts you get a better solution faster, as long as you don’t relax the quality checks you should be doing for any solution whether human or AI created.