Note that this is a repost of my LinkedIn article of the same name

The changing face of DevOps

Over the years the style of DevOps consultancy I have done has changed, along with its name. Changing from the simple ‘source control’, to SDLC, then ALM to now DevOps.

Back in the days of Team Foundation Server (TFS), I used to do a lot of on-premise installs and upgrades. Getting our customers up and running, and helping them improve their adoption of the tools.

With the move to cloud services the profile of this work changed to increasingly helping customers migrate their tooling and processes to the cloud. Specifically doing TFS/Azure DevOps Server to Azure DevOps Services cloud migrations, and then helping them get the best out of this new platform.

Today the profile has changing again. The majority of our clients have already moved to the cloud, so much of my engagements are focused on best practice usage of the tools, and the day to day development lifecycle. As opposed to installation, upgrade and migration.

That does not mean I don’t see installation, upgrade and migration projects in both the Azure DevOps and GitHub space, but certainly not in the volumes I used to.

Who is left on premises and why?

Now we seem to be past the peak of cloud migrations, most clients who want to make the move have moved. Remaining on-premise we now have two broad groups:

• Companies with a governance reason, or maybe just a perceived reason, to stay on-premises. It is worth noting that I do see clients who assume a governance restriction, but don’t check with their regulatory/auditor body for current guidance to see what their options are. They are often surprised that the cloud, when used securely, is a viable option for them.
• Companies who have neglected their on-premise server

It is the latter I want to talk about in this article.

Falling into the gap between IT and Development

I have often seen that it is easy for a TFS/Azure DevOps Server to fall into a gap between the IT and Development teams. Wrongly, the server is not considered a business-critical server to be managed by IT, and not treated with rigor by the Development team as it would be by an IT team.

Because of this, I still see a number of old TFS/Azure DevOps Servers that could well have not been patched since their installation, maybe over a decade ago. The only reason they are being looked at now is the underlying Windows Operating Systems and SQL versions are reaching end of life and company wide IT audits are forcing changes to be made.

I have long said that for the on-premises TFS/Azure DevOps Server you should, on top of the usual OS patching, be looking at patching DevOps at least every 3 to 6 months. This is because this is the rough release cadence that features from the cloud version of Azure DevOps Services get packaged and made available to the on-premise version. These updates include not just new features, but also critical security patches.

It is common that in customers who have not patched their server that they have also not kept their development practices up to date. Often lacking CI/CD processes, which I feel are the core of modern DevOps good practice, the heartbeat of the process, and are commonly still using older source control patterns e.g. still on TFVC source control. This is not in itself wrong, but potentially limiting as modern practices and the skillsets of newer/younger developers are going to be based around the de-facto standard of Git.

But my upgrade is too complex

The issue with upgrades for such old systems is that there is often no direct path to the current version of Azure DevOps Server. Upgrades may require multiple steps, using temporary intermediate environments to address the complexities due to the limitations of supported SQL and OS versions that underpin TFS/Azure DevOps Server.

A common question from clients is ‘how long is this upgrade going to take’? A very difficult one to answer as everyone’s systems are different. You can think of the time required for this upgrade as taking the time for all the smaller updates you should have done over the life of your server and bundling it up into one massive job.

The majority of the time in a TFS upgrade is related to SQL operations, backups, copies, restores and schema/data updates. The sheer volume of data to be moved is usually the limiting factor. Anything involving potentially Terabytes of data takes time.

For major upgrades such as this, I always recommend a dry run to make sure the process is understood, timings known and to give the client’s staff the opportunity for training and adoption of the new tooling available, such as Git and modern CI/CD.

But you can’t get away from the fact that you will probably need an amount of downtime, while your old TFS server is not available, before the new Azure DevOps Server is ready.

But can’t I jump to the cloud now?

Once I start this form of major upgrade engagement it is common for clients to ask “can’t I just bypass all this and go to the cloud?”. Of course the answer is ‘yes, but…’.

If they wish to do a full fidelity migration to Azure DevOps Services then they still have to get to the current version of Azure DevOps Server as the migration start point. So they cannot avoid the on-premise upgrades.

If they don’t need a full fidelity upgrade, or are considering a move to another toolset such as GitHub Enterprise, there are more options that do not require an update to the on-premise server, but these come with constraints and compromises, as you would expect whenever you swap toolsets.

So are you in this position?

The moral of the story is don’t neglect your DevOps tools, a craftsman does not let their tools get rusty and neither should you.

If you are not keeping your DevOps tool chain up to date you are hampering the efficiency and potentially the recruitment/retention of your developers as they look to work on projects using modern tooling.

So, if this sounds like your situation, why not get in touch with me or Black Marble to discuss your DevOps options?

Programmatic connection to Azure DevOps cannot be done with your Active Directory credentials. This is because this involves a dialog being shown, and these days usually an MFA check too.

Historically, the solution to this problem was to enable Alternate Credentials, which could be passed as username and password, without the dialog being shown. However, the use of these has been deprecated since 2020, and they have been completely removed since Jan 2024.

So for a number of years the recommended way to programmatically authenticate with Azure DevOps has been to use a Personal Access Token (PAT). However, PATs are not without their own issues.

The problem with PATs

The most critical problem is that PATs expire. By default they expire after 30 days, and the longest you can set them to expire after is a maximum of 1 year. This means that you have to remember to renew them, and if you forget then your automation that rely on them will stop working.

Obviously it is very tempting to just set the expiry to 1 year, but this is not the best idea. The longer the expiry the more time there is for the PAT to be compromised. Just like SSL certificates, or any access token, the shorter period before the expiry the better.

Of course like any authentication and permission model you can grant too many permissions to a PAT, and then if it is compromised you have a problem, but that is on you not following the principle of least privilege, not a fundamental issue of PATs.

But what is the alternative?

Azure DevOps now offers support for Managed Identity & Service Principals. These are a much better way to provide a means to programmatically authenticate. Critically, they do not expire, and the tokens they generate for the actual connections are short lived, and you can of course grant them privileges in line with the principle of least privilege.

When used to access Azure DevOps resource, the only downside is they consume a basic user license.

This form of authentication is perfect for use in automation from locations like Azure Functions, or from the Az CLI.

What about GitHub?

The issue of PATs is not just one for Azure DevOps, the same problems are true for GitHub. On GitHub the solution is to use a GitHub App to authenticate with the GitHub API, and then use the short term token it can generate to authenticate with the GitHub API.

What about other outgoing services from Azure DevOps?

Another place you will see a move away from expiring secrets in Azure DevOps are Service Connections to services to such as Azure. You can now use Workload Identity Federation to authenticate with Azure.

Converting to use Workload Identity Federation is simple, you just press a button in the Azure DevOps UI, and it will create a Service Principal in Azure AD, and then update the Service Connection to use this to authenticate with Azure.

Once this is done there will be no need to remember to renew the Service Connection, as it will use the Service Principal to authenticate, and this does not expire.

Summary

So, when using Azure DevOps in many cases you can now avoid the use of PATs, and hence the need to remember to renew them. Giving you one less thing to remember to do, and one less thing to go wrong.

Team Foundation Version Control (TFVC) has been around since 2005, since the first release of Team Foundation Server (TFS) in 2005. In 2013, over 10 years ago, Microsoft added Git support to TFS (later renamed as Azure DevOps), Git had already been around for 8 years at that point.

10 years is a long time in the software industry, I always think of ‘IT years’ like ‘dog years’ i.e. 7 to 1, so over 70 years has passed, a lifetime. Over this period Git has become the de facto standard for source control. So if you are still using TFVC as your source control system you really need to ask yourself why?

The advantages of Git over TFVC

The advantages of Git over TFVC are numerous, but here are a few:

• Git is distributed, meaning you can work offline and commit changes locally
• Git has better branching and merging (pull request) support
• Many newer features of Azure DevOps are Git only i.e YAML based pipelines
• Cloud services like Azure Functions, Azure App Service, Azure Logic Apps, Azure API Management etc. all have built-in support for Git
• Git repositories are portable, they are easy to migrate to new systems e.g GitHub, GitLab, Bitbucket etc.
• Any new hires/students will probably have Git experience

From talking to clients who are still on TFVC, the main reason they are still using TFVC is just because they have always used it. There is a fear a change, something you might not expect to see in the technology sector, but all too common.

This fear might be wrappered in the perceived complexity of migrating from TFVC to Git, or that an audit or regulatory requirement means they can’t change processes. But these are just excuses. Updates to process and tooling cannot be ignored, you don’t have to be the first to move, but the longer you leave it the harder it will be to change.

As an example, one of the most obvious limitations for me of TFVC is how it works with Azure DevOps Pipelines. It is true you have to use the ‘classic’ pipelines, which are not as flexible as the YAML based pipelines, and have reached a ‘done’ state and so will get no further development. However, more me the big limitations are not the editor experience but that:

• For each TFVC branch you have to create/clone a new pipeline
• You cannot make CI/CD part of your branch protection
• You cannot source control your pipeline definitions and review them via a PR

These reasons alone should be enough to make you want to move to Git from TFVC.

So how can I migrate?

You have three basic options to migrate from TFVC to Git:

• ‘Tip’ migration: This is the simplest, you just migrate the latest version of your code to a new Git repository. This is the quickest and easiest, but you lose all the history of your code.
• Azure DevOps Importer: This is a tool that can be used to import a TFVC repository into a new Git repository, with the last 180 days of history. Why you would want neither none of the history or all of the history has always been beyond me!
• Git TFS: A command line extension for Git that can be used to migrate a TFVC repository to a new Git repository, with all the history. A powerful and flexible tool, but it can be complex to use, and is ‘brittle’ in that it can fail if the TFVC repository has a complex history of branching and renames.

I would always favour the first option. It is the simplest and quickest, and you can always keep the old TFVC repository around for historical purposes. Also remember Git is not TFVC, so what is a good structure in TFVC might not be the best in Git, so you might want to take the opportunity to restructure your codebase.

The other important factor to consider in any migration is that of training. You need to make sure your team is comfortable with Git before you start the migration. This is not just about the commands, but also the concepts of distributed source control, and the different branching and merging strategies that Git enables.

A TFVC to Git migration will be a major change and should be considered as a project in its own right, with planning, training, pilot studies as well as the main migration itself. But remember you don’t have to move all your TFVC source control to Git at once, you can do it incrementally, maybe as you start a new version of a product.

Summary

So if you are still using TFVC I urge you to consider migrating to Git. It is not as hard as you might think, and the benefits are numerous.

Happy to discuss this further, please reach out to me via the usual channels

So I am expecting a degree of maintenance work on my Azure DevOps pipelines. However, this is becoming more awkward as many of the tasks used by Azure Pipelines, even ones from Microsoft, are themselves not being maintained.

As well as tasks that are running out of date versions of underlying tools, there also seems to be no end of them that when run report the following warning. A classic sign of a lack of maintenance as this guidance is a few years old:

##[warning]Task 'Some task' (1.2.3) is using deprecated task execution handler. The task should use the supported task-lib: https://aka.ms/tasklib

Because of this lack of maintenance, I am becoming more reticent to use tasks from the Azure DevOps Marketplace, especially if they are not from maintainers I trust. Rather I am starting to favour using simple PowerShell or Bash scripts to do the work. So I can control the versioning of the underlying tools.

In the past I would have written my own custom tasks to provide these wrapper functions, but of late it does not seem worth the effort.

So of late, it seems an approach of writing scripts and including them, with the tools they run, via a YAML Pipeline Repository Resource is just easier.

Am I alone in this change of approach?

So, I was really please to discover the 2nd Edition of Charles Petzold’s book Code: The Hidden Language of Computer Hardware and Software.

This book covers exactly the journey I discussed. Starting with simple codes and electronics, building up to how a computer actually functions.

Highly recommended for anyone who wants to understand how a computer works, so to better diagnose problems.

Background

Back in 2004 I completed my BSc Degree in Computer Science at Bradford University; only 20 years after I started. My year in industry got out of hand!

One module I did involved me writing a paper on a legal area concerning computing. I chose to discuss licensing of engineers.

Whilst at the DDDNorth community conference last weekend (Dec 2022) I got chatting on aspects of staff development, problem solving and training with a fellow MVP Matteo Emili and I mentioned this paper I had written. We both thought it might be interesting to revisit it and see if it still held true, best part of 20 years on.

So enjoy, remembering this was originally written 18 years ago…

Introduction

Throughout the history of science and technology there has been a trend in the development spectrum away from the gentleman amateur and towards the professional engineer. This report explores at which point on this spectrum Software Engineering currently lies and, if it is found to lie at the fully professional end, whether or not there is a need for licenses to practice.

Software Engineering Professionalism

Computing in general, and software engineering in particular, is a young discipline. Its professional societies are only around 50 years old, the BCS was formed in 1957; it is now only reaching a point where it is not uncommon that when retiring, practitioners have had their whole career in the field.

Due to this relative youth the formal call for professionalisation is only recently being brought to the fore. This is not to say that people working in computing have not had the need to be professional - this has always been important - but since the mid 1990s, there has been serious talk of some formality, maybe even of a license to practice, as in other professions. This debate can be seen at least as a sign of puberty, if not one of maturity, for the industry.

What are the Real Benefits of Licensing?

The key factor behind any call to licensing is one of public confidence. The license to practice provides a standard by which the purchaser of any service can see that a practitioner has met a minimum level of competence. This does not mean that they are a good Doctor or Accountant, bad practitioners still exist, but, if they do not maintain this standard, there are processes through which their license may be revoked.

Perception of the Computing Industry

Currently such a position of public confidence is not enjoyed by the computing industry. The media coverage is usually concentrated on perceived-failing systems, such as the Swanwick Air Traffic Control Centre and the new NPfIT NHS records project (even before the project has really started). The media does not talk of computing success, it is not newsworthy.

This media slant is not unsurprising, given the project success rates shown in Figure 1. If this were figures in the Civil Engineering industry, would anyone be crossing bridges?

In fact, the route to licensing of Civil Engineering came from failing constructions such as bridges. Canadian engineering folk law [SM99a] has it that the iron used to make the rings given to graduating engineers comes from the collapsed Quebec City Bridge of 1907, as a reminder of their public duty. Licensing of engineers in Texas came about after a boiler explosion that killed 300 school children [SM99a].

Another factor in the perception of computing in many people’s eyes is that they say ‘I have a computer and use it OK, so how hard can this software thing be?’

This viewpoint is not limited to the ‘man in the street’, but also to many corporate managers.

A figure regularly reported at Microsoft conferences is that 70% of corporate development is done using the Visual Basic family of languages, commonly not by specialist programmers but by end users. These users have little knowledge of software engineering principles, and hence often produce initially functional but poorly maintainable or un-scalable systems. It is often the reliability of these systems that shape public perception of computing.

Is there need for universal licensing?

Given these facts, would requiring all IT professionals to have a license be the right move? The answer has to be no.

Table 1: Percentage of licensed Engineering graduates in the US as of 1996 [SM99a]

In other engineering professions only a small percentage of people are licensed [Table 1]. The remainder of the workforce are working towards licensing, or are responsible to a licensed engineer.

Given this data Steve McConnell suggest only 5-10% of software engineers need to be licensed, specifically those working on safety-critical systems [SM99b].

A Move to Licensing

In 1999 the Texas Society for Professional Engineers passed a resolution to license Software Engineers [LHW99]. The scheme was based on applicants providing nine references and meeting one of the following criteria:

• 16 years’ engineering experience
• 12 years’ engineering experience, plus a bachelor’s degree from an accredited program.
• 6 years’ engineering experience, plus a PhD in engineering or related subject from a university with an accredited undergraduate program.

In June 2001 the Canadian Engineering Accreditation Board (CEAB) accredited three degree programs [DLP02a] and instigated a licensing program with similar requirements to that in Texas.

The Current BCS Position

Currently there is no plan to license Software Engineers in the UK or the EU, only voluntary professional schemes are in place.

Historically the BCS had a membership system which provided a means to get an Engineering Council Charter Engineer (CEng) qualification. In 2002 the BCS underwent a radical restructure to provide a more outward-looking system in an aim to promote professionalism in the industry. The major change in the membership system was a new grade of Chartered Information Technology Professional (MBCS CITP) that sits along side the CEng qualification.

Initial results seem to show that the changes have been approved by the industry as ‘The British Computer Society has reported a 200% leap in membership applications since the launch of its new grading structure, - reports chief executive David Clarke’ [BCS04a]

There has also been a change in the culture at the BCS. At the West Yorkshire BCS branch AGM (25th May 2004) David Clarke the Chief Executive of the BCS gave an example of how, when he joined the BCS, there was no standard way for the public to ask it questions; they had been asked by a national newspaper for comments on the Home Secretary’s proposed ID card scheme. It had taken three weeks for his team to find a person willing to speak on the subject. Changes to the BCS now see it taking a more active role in advising bodies such as government committees on IT related issues [BCS04b]. This is a key area that a professional organisation must address in the current media-centric world.

For the foreseeable future the BCS is not making moves toward licensing, preferring certification based on internationally recognised products that it provides such as European Computer Driving License (ECDL) and the European Certification for IT Professionals (EUCIP).

Certification as an alternative

Usually certification is not compulsory, unlike licensing, and is often vendor-led. Other than the BCS, the major certification schemes are from vendors such as Microsoft, Cisco and Novell. Certification is well established, and aims to give all parties benefits; the customer knows the engineer has been trained, the vendor reduces their support calls due to engineer training, and the engineer gets a way to differentiate themselves from their competitors.

The problem is that such qualifications age rapidly - usually re-certification is required every 3 to 5 years as vendor products are dropped and new ones replace them. Also, some of the qualifications have become devalued due to ‘boot camps’ and ‘brain-dumps’ , passing the certification exams is often dependant on the students budget and ability to cram, not their fundamental knowledge of the subject.

Certification is a valuable tool to the industry, providing good specific product training, but not a replacement for good basic grounding provided by a degree or similar.

Has licensing worked thus far?

A view on the success or failure of licensing can be found in the ‘Communications of ACM’ [KK02]

“…after four years, what has been the overall impact of software engineering licensing in Texas? In our view, the news is good: licensing had practically no effect…”

The key factor cited in reviews of the state of licensing is the lack of a stable core of knowledge that a practitioner is required to study [KK02] [DLP02b]. In other engineering disciplines there is a well defined boundary between research science and engineering. For example if you wished to make TVs for the consumer market you could hire a physicist; they could make you a TV, but it would no doubt be expensive, unreliable but revolutionary. The normal practice would be to hire experienced electronics and production engineers, who would draw on the core knowledge of their fields, originated by research scientists, to make a suitable specified product on time and on budget.

Are we tackling the right problem?

An interesting point of view is put forward by Pete McBreen in ‘Software Craftsmanship’, this again states the problems with lack of a stable core of knowledge, but highlights that ‘The concept of a single, responsible engineer signing off a complete work is not feasible for software’. [PM02a]. Software projects are just too large and complex and, unlike Civil Engineering building codes, there is no agreed legal standard to sign off against anyway.

McBreen goes on to propose that Software Engineering, not just licensing, fails not because of the lack of experience or skill, but due to the lack of personal responsibility. He argues that it is better to have a master-apprentice approach to software development, with a senior programmer taking juniors under their wing for a number of years, with their systems being developed and handed on like a prize heirloom - the EMACS product is held up as an example [PM02b].

This structure has its appeal. It seems to fit well with other professions, and while they may not be called master and apprentice similar structures occur in the legal, medical and accountancy professions. Though, interestingly, they appear less frequently in traditional engineering, where in many ways the personal craftsman’s responsibility has been replaced by a corporate regulation framework.

Is Licensing Round the Corner?

Software engineering is not ready for universal licensing, the core body of knowledge is still growing too fast to allow any true foundation engineering degrees to be stabilised. It could reasonably be argued that this stabilisation may never occur, the rate of technology change and tre ihe huge variety of projects for which software is developed may defeat all efforts for universal standards for licensing. However, this does not mean that licensing may not appear in specialised areas, if they can be formalised to a sufficient degree. The most likely area is safety-critical systems.

In general, for the foreseeable future the industry will have to rely on voluntary membership of organisations such as the BCS and vendor certifications. This in effect means that the onus of good practice and professionalism falls, as always, on the shoulders of the individual. Maybe what is required is the personal responsibility of the gentleman amateur and craftsman to build the confidence of the general public rightly expects.

.. And 18 years on

So what has changed 18 years on, well not a lot that I can see. There is no clamour for licensing, our media only cover stories of failure, and projects of all sizes are still failing.

There is still no agreement on best practice, at least to the level required for licensing. IT is still just too fast moving, another week another framework, language or pattern.

We as individuals and teams are still, and always will be, responsible for our own actions, and we still have to strive to deliver high quality solutions that meet our client business needs.

What are the chances of a change in the next 20 years? Or will it become an irrelevance as we move to the use of more AI tools?

Maybe I will come back to this post around 2043 and see what has changed.

References

In the recent past, I have seen confusion from our clients as to what is Microsoft’s recommended DevOps solution, given they have both Azure DevOps and GitHub.

It is true that Microsoft have said, and continue to say, that GitHub is the ’north star’ the long term destination for all users. However, that does not help clients today. Many of mine ask ‘but I am using Azure DevOps, but all Microsoft seem to talk about is GitHub, is Azure DevOps dead?’.

Well today’s announcements in the conference session ‘Accelerate innovation with the world’s most complete cloud developer platform’ and the blog post ‘Integrate security into your developer workflow with GitHub Advanced Security for Azure DevOps’ goes a long way to answering that question with a positive answer.

‘Cool features’ from GitHub Enterprise, specifically Code Advanced Security, are coming to Azure DevOps, along with other Azure DevOps specific investments. Thus greatly re-enforcing the ‘better together’ story for Azure DevOps and GitHub, and that Azure DevOps has a future.

These changes could be taken two ways, depending on your position

• You could argue it makes it much clearer for both existing Azure DevOps and GitHub users to see their road-map, with both products sharing a common set of share services
• Conversely you could argue it muddles the waters and slows the inevitable move to GitHub

I suspect which position you take is dependent as to whether you are an Azure DevOps or GitHub Enterprise user.

If this is an area of interest, I will be talking more about these changes at Black Marble’s next free webinar titled ‘Which is best for me… GitHub, Azure DevOps, or Better Together?’ on 26th October 2022.

Each section is delivered in broadly three parts, which I found really effective

• the history as to why we got to where we are
• theory as to how we should use a process/technology
• and what practical we can do to best adopt this practice with GitHub within our organisation.

Taken all together it gives coherent examples of what does, and does not work, based on different teams sizes and common external/company constraints.

As with all books based on a technology stack, in this case GitHub, it is going to age quickly. Cloud based tools move on, walkthroughs become outdated. However, because of the emphasis this book places on background history and core theories as to why we use a technology, it will stand the test of time better than most.

So I think it is well worth a read, irrespective of your experience with DevOps and GitHub. There is something in it for any individual or company no matter where you are on your DevOps journey

I had an error when trying to digitally sign a package. On searching on the error code I came across my own blog post with the solution. This was, as usual, one I had no recollection of writing.

I find this happens all the time. It is a little disturbing when you search for an issue and the only reference is to a post you made and have forgotten, so you are the defacto expert, nobody knows anymore on the subject, but better than having no solution.

Too often I ask people if they have documented the hints, tips and solutions they find and the response I get is ‘I will remember’. Trust me you won’t. Write something down where it is discoverable for your team and your future self. This can be any format that works for you: an Email, OneNote, a Wiki or the one I find most useful a blog. Just make sure it is easily searchable.

Your future self will thank you.

In December 2017 I had major surgery. This was to remove an adrenal cortical carcinoma (ACC) that had grown on one of my adrenal glands and then up my inferior vena cava (IVC) into my heart.

Early on I decided, though not hiding the fact I was ill, to not live every detail on social media. So, it is only now that I am back to a reasonable level of health and with some distance that I feel I can write about my experiences. I hope they might give people some hope that there can be a good outcome when there is a cancer diagnosis.

I had known I was ill for a good while before I was diagnosed in May 2017. I had seen my Parkrun times slowing week on week to the point where I could not run at all, and I had also had a couple of failed blood donations due to low haemoglobin levels.

It was clear I was unwell, and getting worse, but there was no obvious root cause. All sorts of things had been considered from heart to thyroid. Cancer was suspected, but a tumour could not be found. Try as they might, my GP had failed to find a test that showed anything other than my blood numbers were not right. I was just continuing to get weaker, by that spring I was unable to walk more than a few hundred meters without getting out of breath with my heart beating at well over 170 BPM.

The problem was that ACC is a rare form of cancer and mine had presented in a hard to find way. There are two basic forms of ACC. One shuts down your adrenal system, and you notice this very quickly. The other form shows no symptoms until the tumour starts to physically impact something. This was the form I had. In my case, the tumour was increasingly blocking blood flow in my IVC and heart.

In the end, the tumour was found because of a lower abdominal ultrasound. By the time I had the ultrasound scan it was about the only diagnostic that had not been tried. It was a strange mixture of shock and relief to be immediately told after the scan by the sonographer that ‘the doctor would like a word before you go home’. So, at least I knew the cause of why I felt so ill. I left the hospital that day with a diagnosis of an adrenal tumour that was most likely benign but may be malignant, on blood thinning injections and with a whole set of appointments to find out just how bad it was.

At this point the NHS did what it does best, react to a crisis. Over the next couple of weeks, I seemed to live at the regional cancer centre at St James Hospital in Leeds having all sorts of tests.

My health, and the time I was spending at the hospital, meant there was no way I could continue to work. I was lucky I was able to transition quickly onto long term sick in such a way that meant I did not have the financial worries many cancer patients have to contend with on top of their illness. I would not be seeing work again for over 9 months.

The next phase of diagnostic tests were wide ranging. Plenty of blood was taken, I had to collect my urine for 48 hours, there were CT scans and PET Scans, all to get a clearer idea of how bad it was. The real clincher test as to whether the tumour was benign or malignant was a biopsy. One of those strangely pain free tests, due to the local anaesthetics, but accompanied by much poking, pushing and strange crunching noises. Then a 6 hour wait flat on my back on a recovery ward before I could sit up, let alone go home.

It was whilst laying down post-test I had probably my best meal on the NHS. Having just missed the lunch service on the recovery ward, a good move from past experience, a nurse produced a huge pile of toast and jam. A perfect meal for the reclined patient.

It was also during this post test recovery time that I first met other cancer patients and had a chance to have a proper chat with them. No matter how bad your case seems to be you always seem to be meeting people with a worse prognosis. Whilst on the biopsy recovery ward I met a man who told me his story. A check-up because he did not feel well led to the discovery of a large brain tumour which then spread throughout his body. He knew he only had a short time left. The conversation opened my eyes to the reality of my and other patients’ situations.

A couple of weeks later we got the bad news that the cancer was malignant and very advanced. We had clung onto the hope it was benign. The news was delivered in a very matter of fact way, that I probably would not see Christmas unless a treatment plan could be found, and the options were not good. There were tears.

However, there was at least some good news, the tumour was a single mass, it had not spread around my body. The problem was that there was no obvious surgical option due to its size and position. All that could be done was to start chemotherapy to see if the tumour could be shrunk. So, a very ‘old school’, and hence harsh, three cycle course of chemotherapy was started in July 2017.

I dealt with all of this in a very step by step way. People seemed surprised by this, that I was not more emotionally in pieces. I assume that is just my nature. I think this whole phase of my illness was much harder on my partner and family. They had to watch me getting more ill with no obvious route to recovery. For me it was just a case of get up and doing whatever the tasks were for the day. Whether they be tests, treatments or putting things in place like a Lasting Power of Attorney.

Life became a cycle of three-day eight-hour blocks of chemotherapy, then a month to try to recover. On each cycle I recovered less than the previous one.

The chemotherapy ward is strangely like flying business class. The seats look comfortable, but after eight hours they are not. You can’t go to the toilet without issues, on an airplane it is getting out of the row, on the chemotherapy ward it is taking the drip with you. In both cases, the toilet is too small. You feel tired all the time, just like jet lag, and of course, the food is questionable at best.

As I had seen on other wards, there was a strong camaraderie on the chemotherapy ward. Everyone is going through life changing treatment. Some people looked very ill, others as if there is nothing obviously wrong with them, but irrespective of their condition I found the patients, as well as the staff, very supportive. It was far from an unhappy place. Not something I had expected.

In many ways the worst side effect of chemotherapy, beyond the expected weight loss, hair loss, nausea and lack of energy was that my attention span disappeared. For the first time in my adult life I stopped reading. I struggled to make it through a single paragraph without forgetting where I was. I remember one afternoon in a hospital waiting room, whilst waiting for yet more test results, trying to read a page in a novel. I never got to the end of the page, just starting it over and over. It was also at this time I realised I had to stop driving, I felt my attention was too poor and my reactions too slow.

As I said, by this point I was very weak. This made most day-to-day activities very hard, but the strange thing was I found I could still swim. I had had the theory that though my IVC was blocked, hence not bringing blood from the lower half of my body, if I swam with a pull-buoy just using my arms, I would be OK. This turned out to be correct, much to the surprise of the medical professionals. So, I started to do some easy swimming in the recovery phases between chemotherapy cycles when I was able. It turned out the biggest issue was I got cold quickly due to my weight loss. So, swim sessions were limited to 15 to 20 minutes and just a few hundred metres.

After the planned three chemotherapy cycles all the tests were rerun and it was found that the tumour seemed unaffected. It was always a very low chance of success. I had already decided I was unlikely to start a 4th cycle as I felt so ill, it was just no life. I did not want any more chemotherapy when the chance of success was so low. Better to have some quality of life before the end.

This is where I got lucky because I was being treated at a major cancer research centre. I had been told there was no adrenal cancer surgical option for the way my ACC had presented. However, the hospital’s renal cancer surgical team had seen something similar and were willing to operate with the support of the cardiac and vascular teams. A veritable who’s who of senior surgeons at St James as I was informed by the nurse when I was being admitted for the operation in December 2017.

My operation meant stopping the heart, removing the tumour along with an adrenal gland, and a kidney (collateral damage as there was nothing wrong with it other than its proximity to the tumour) and then patching me all back together. Over 10 hours on the operating table and a transfusion of a couple of pints of blood.

When you see a very similar version of your operation on the BBC series on cutting edge surgery ‘Edge of Life’ you realise how lucky you are. Just a few years ago or living in another city and the operation would not have been possible.

Given my heart had to be stopped, I was treated as a cardiac patient, and the cardiac department moves you through recovery fast. Most of the people on the ward were having heart bypasses, so I was ’the interestingly different’ case to many of the staff. I did take longer than the usual 5 days on the ward taken by bypass patients, but I still managed to get out of hospital in 10 days, in time for Christmas. It is surprising how fast you can get over being opened up from the top of your chest to your groin, and how little pain there was.

At this point I was in theory cured, the tumour was removed, blood was flowing again but I was very weak and recovery was going to be a long road. I started with walks of only a few minutes and then the rest of the day resting. The great news was that I could walk again without getting out of breath and my heart rate going through the roof.

So, over the next few months, I gradually regained my health, some weight, some hair and my attention span. I was able to ease back into work part time in the early summer of 2018.
However, the surgery was not the end of my treatment. The surgeons were confident they had got all the tumour they could see. They said it was well defined, so cancerous and normal tissue could be differentiated, but there was always the chance of microscopic cancerous cells remaining. So, I was put on two years of Mitotane tablet-based chemotherapy. This was the treatment with the best evidence base, but that is not saying much. There are not that many research studies into ACC treatment options as it is so rare. My treatment plan was based on a small Italian and German study of 177 people, most of which did not complete the plan, but it did show a statistically significant reduction in the chance of remission after 5 years.

Mitotane stops cell division and I had not realised how hard this would make my recovery and specifically regaining some fitness. I was OK for day to day living, but an activity like running was not possible. I twice started Couch to 5K but had to give up as I could not progress beyond the walking stages.

The mental weight of everything did not catch up with me until a good year or so after surgery, by which time I was back at work and living a ‘normal’ life. Previously people had kept asking ‘how are you doing?’. As I said, I felt they expected me to be in pieces, and I was just going step by step. It is only when the main treatment stopped and life returned to normal that everything that had occurred hit me. A seemingly unrelated fairly small in the scheme of things family incident caused it all to come flooding back and completely stopped me in my tracks.

It was that this time I reached out to the support services of the Macmillan charity and specifically the Robert Ogden Centre at St James for help. This was something I had not done prior to this time, though my partner had used their family support services earlier in my treatment. With their counselling help, I worked my way through my problems and got back to some form of normal.

In the autumn of 2019 I came off Mitotane and once it was out of my system I could at last try to get fit again. So, it was back to Couch to 5K and with a few repeated weeks I was able to run 5K again. I was back running Parkrun in November 2019. It was great to get back to my local Roundhay Parkrun community, though I had been volunteering whenever my health allowed throughout my illness. I was running much slower than before I was ill, but running.

Since then, I have to say Covid lockdown has helped me, giving me a structure to my training. I have certainly got a reasonable level of endurance back, but any speed seems to elude me.

I have always had a fairly high maximum heart rate, over 200 well into my 40s, and before getting cancer it was still in the 190s. Now, post illness, I struggle to reach 160 and my bike and run maximum heart rates are very similar. I have tried to do a maximum heart rate test, it is as if I get to a heart rate around 150-160 for a tempo run, but it barely goes any higher when I sprint. So, I have a question for anyone with experience of training after cancer and heart surgery. Is it expected after stopping the heart that my maximum heart rate should be way lower? Or is the problem my hormone levels are different due to the lack of one of my adrenal glands? Or is it just I am getting older and have just lost muscle mass? I am not sure I will ever know the answer to that one, it is not exactly a question the NHS is set up to answer. All their post-operative guidance is aimed at day-to-day levels of exertion not the elevated levels caused by sports.

But that is a minor gripe, I am reasonably fit again. I have recently completed my first triathlon in 5 years and between lockdowns walked the 268 miles of the Pennine Way with my partner. I am not as fast as I was, but I am 5 years older and have had major heart surgery. Hell, I am alive.

Like all cancer patients, this is not the end of the road for my treatment. I am still on steroids and have annual CT scans, but all the signs seem good that the surgery got the tumour and there is no reason I should not live to a ripe old age.

I would not have got here without the support of my partner and family, and the unbelievable work of the NHS and the support services I have used. I can’t thank you all enough.

Leeds Hospital Charity - the charity of Leeds Teaching Hospitals
Macmillan Cancer Support - support or cancer patients and their families
NHS Blood Transfusion Service – please consider giving blood, without regular donations surgery like mine is not possible.

Update 18 Aug 2022

I have just had my five year checkup at the Leeds Cancer Centre and I am pleased to say all is good, there are no signs of any reoccurrence of my cancer.

For my type of adrenal cancer, the statistics say if you survive to five years after treatment without problems, your reoccurrence risk levels fall back to the baseline level for the general population.

So, I am currently cancer free and my chances of a reoccurrence are no higher than that for the general population. This is a real success story, and I hope gives hope to others starting their cancer journey

Of late I have certainly seen a bit of a move towards using more functional style tests; still using unit test frameworks, but relying on APIs as access points with real backend systems such as DBs and WebServices being deployed as test environments.

This practice is made far easier than in the past due to cloud services such as Azure and tools to treat creation of complex environments  as code such as Azure Resource Manager and Azure DevTest Labs. Both myself and my colleague RIk Hepworth have posted widely on  the provisioning of such systems.

However, this type of functional testing is still fairly slow, the environments have to be provisioned from scratch, or spun up from saved images, it all takes time. Hence, there is still the space for fast unit tests, and sometimes, usually due to limitations of legacy codebases that were not designed for testing, there is a need to still ‘mock the un-mockable’.

This is where tools like Typemock Isolator and Microsoft Fakes are still needed. 

It has to be said, both are premium products, you need the top Enterprise SKU of Visual Studio to get Fakes or a Typemock Isolator license for Isolator, but when you have a need them their functionality they are the only option. Whether this be to mock out a product like SharePoint for faster development cycles, or to provide a great base to write unit tests on for a legacy code base prior to refactoring.

As I have said before, for me Typemock Isolator easily has the edge over Microsoft Fakes, the syntax is so much easier to use. Hence, it is great to see the Typemock Isolator being have further extended with updated versions for C++ and now Linux.

So in answer to my own question, testing is a layered process. Where you put your investment is going to be down to your systems needs. It is true, I think we are going to all invest a bit more in functional testing on ‘cheap to build and run’ cloud test labs. But you can’t beat the speed of tools like Typemock for those particularly nasty legacy code bases where it is hard to create a copy of the environment in a modern test lab.