I have posted on previously how we used TFS Lab Management to provision our test and development environments. With our move to VSTS, where Lab Management does not exist, we needed to look again at how to provision these labs. There are a few options…
Move to the Cloud – aka stop using Lab Management
Arguably the best option is to move all your lab VMs up to the cloud. Microsoft even has the specific service to help with this Azure DevTest Labs. This service allows you to create single VMs or sets of VMs for more complex scenarios using of ARM templates.
All good it seems, but the issue is that adoption of a cloud solution moves the cost of running the lab from a capital expenditure (buying the VM host server) to an operational cost (monthly cloud usage bill). This can potentially be a not insignificant sum; in our case we have up to 100 test VMs of various types running at any given time. A sizeable bill.
Also we need to consider that this is a different technology to Lab management, so we would need to invest time to rebuild our test environments using newer technologies such as ARM, DSC etc. A thing we should be doing, but I would to avoid doing it for all our projects today.
Now it is fair to say that we might not need all the VMs keep running all the time, better VM management could help alleviate the costs, and DevTest Labs has tools to help here, but it won’t remove all the costs.
So is there a non-cloud way?
Move to Systems Center
Microsoft’s current on premises recommended solution is to use System Center, using tasks within your build and release pipeline to trigger events via SC-VMM.
Now as Lab Management also makes use of System Center SC-VMM this might initially sound a reasonable step. Problem is that the way Lab Management uses System Center is ‘special’. It does not leverage any of the standard System Center tools really. Chances are anyone who investing time in using Lab Management makes little or no use of System Center own tools directly.
So if you want to use System Center without Lab Management you need to work in a very different way. You are into the land of System Center orchestrations etc.
So again you are looking at a new technology, this might be appealing for you, especially if you are using System Center to manage your on premised IT estate, but it was not a route I wanted to take.
Keeping Lab Management running
So the short term answer for us was to keep our Lab Management system running, it does what we need (network isolation the key factor for us), we have a library of ‘standard VMs’ built and we have already paid for the Hyper-V hosts. So the question became how to bridge the gap to VSTS?
Step 1 – Leave Lab Management Running
When we moved to VSTS we made the conscious choice to leave our old TFS 2015.3 server running. We removed access for most users, only leaving access for those who needed to manage Lab Management. This provided us with a means to start, stop, deploy network isolated Lab Environments.
KEY POINT HERE – The only reason our on-premised TFS server is running is to allow a SC-VMM server and a Test Controller to connect to it to allow Lab Management operations.
Another important fact to remember is that network isolation in each labs is enabled by the Lab Test Agents running on the VMs in the Lab; so as well as communicating with the Test Controller the agents in the environments also manage the reconfiguration of the VMs network adapters to provide the isolation. Anything we do at this point has to be careful not to ‘mess up’ this network configuration.
Problem is you also use this Test Agent to run your tests, how do you make sure the Test Agent runs the right tests and send the results to the right place?
We had already had to build some custom scripts to get these agents to work the TFS vNext build against the on-prem TFS server. We were going to need something similar this time too. The key was we needed to be able to trigger tests in the isolated environment and get the results back out and up to VSTS all controlled within a build and release pipeline.
We came up with two options.
Option 1 Scripts
First option is to do everything with PowerShell Scripts Tasks within the release process.
- Copy the needed files onto the VM using the built in tasks
- Use PowerShell remoting to run MSTest (previously installed on the target VM) – remember you have to delete any existing .TRX result file by hand, it won’t overwrite.
- Copy the test results back from the VM (RoboCopy again)
- Publish the test results TRX file using the standard VSTS build task for that job.
There is nothing too complex, just a couple of PowerShell scripts, and it certainly does not effect the network isolation.
However, there is a major issue if you want to run UX tests. MSTest is running on a background thread, so your test will fail it cannot access the UI thread.
That said, this is a valid technique as long as either
- Your tests are not UX based e.g. integration tests that hit an API
- You can write your UX test to use Selenium PhantomJS
Option 2 do it the ‘proper’ VSTS way
VSTS has tasks built in to deploy a Test Agent to a machine and run tests remotely, including UX tests. The problem was I had assumed these tasks could not be used as they would break the network isolation, but I thought I would give it try anyway. That is what test labs are for!
Inside my release pipeline I added
- Copy the needed files onto the VM using the built in tasks, as before
- A deploy Test Agent Task
- Run functional tests Task, which also handles the publish
When this was run the deploy Test Agent task de-configures (and removes) the old TFS 2015 Test Agent put on by Lab Management and installs the current version. However, and this is important, it does not break the network isolation as this is all setup during VM boot and/or repair. The Lab will report itself a broken in the Lab Management UI as the Test Agent will not be reporting to the Test Controller, but it is still working
Once the new agent is deployed, it can be used to run the test and the results get published back to VSTS, whether they be UX tests or not.
If you restart, redeploy, or repair the Network Isolated environment the 2015 Test Agent gets put back in place, so making sure the network isolation is fixed.
Conclusion
So Option 2 seems to deliver what I needed for now
- I can use the old tech to manage the deployment of the VMs
- and use the new tech to run my tests and get the results published to the right place.
Now this does not means I should not be looking at DevTest Labs to replace some of my test environments, also Azure Stack might provide an answer in the future.
But for now I have a workable solution that protects my past investments while I move to a longer term future plan.