Managing Variables in Microsoft DevOps

Automating tasks in Microsoft DevOps using Python

DevOps pipelines

I recently spent time trying to piece together a decent way of managing variables in Microsoft DevOps for the purpose of maintaining standard software versions.

In general, software versions use the following naming convention: MajorVersion.MinorVersion.PatchNumber

Which would translate to 10.2.4
In my case, I also append the git commit to the end of the build (i.e. 10.2.4–6rfds8) so there is traceability between the code running in a given environment and the source code.

The problem

In my DevOps environment, I maintain pipelines and git branches for Dev, Test and Staging / Prod. I also use the variables library to store the version info for the different environments, it’s a nice way to keep everything organised.

When I build my Dev branch, I wanted my patch variables to automatically increment when the dev build pipeline runs. Then, when code is merged into the upper branches (Test and Staging) the version variables for those environments should update based on the current Dev variables.

This means that for a given drop, all environments should be in sync, showing a matching version number and git commit id.

After scratching around for a while there were a few different implementations with PowerShell that came up. But, there were 2 problems:

1. Copying code from the internet without understanding exactly what it does is always a bad idea (sometimes even if you do understand, it’s still a bad idea… oh the irony of what follows…).
2. I don’t like PowerShell. Sorry PowerShell fans, I just find it fiddly and unintuitive, particularly when manipulating values.

The Solution

I decided to translate the various PowerShell implementations I found into Python (which for me is much easier to debug and test) and make the alterations necessary for my particular use case.

Before we get to the code, you will need the following:

• Your Organisation name — you can find this in the URL when you log into DevOps (e.g. https://dev.azure.com/edlittle)
• Your Project name —for the pipeline you’re working in. Again, you can find this in the URL in DevOps.
• The ID of the Variable Group in the Library — this too, you can get from the URL. If you don’t have any setup yet, navigate to Pipelines in the sidebar, and select Library in the list and create a new Variable Group. View your Variable Group and in the URL you should find a value for variableGroupId= and then some number, that number is what you need.
• A Personal Access Token (PAT)— you’ll need this to interact with the DevOps API and save the new variable values. When you create your PAT, make sure to only allow access to the scope for updating Variables.
• A Variable name — well, specifically the name of the variable you want to increment, in my case, it was SystemDevPatchVersion.

The Code

I’ll walk through the script in sections, and then at the end, I’ll leave you with the full script to see it all in context.

Setting up:
First up, we need to import the relevant libraries and set up some variables that we’ll use in our code. This is where most of the items we collected a moment ago get subbed into the code. The more sensitive stuff like your PAT will be handled through an argument rather than saving it into the source code.

import requests
import base64
import argparse
URI_BASE = "https://dev.azure.com/{orgname}"
PROJECT = "{projectname}"
VARIABLE_GROUP_ID = {variablegroupid}

Arguments / Parameters:
Next up, we need to be able to pass in some arguments to the script. There are several ways of doing this in Python, but my preference is using the argparse library. It’s a really simple, clear and versatile way of handling parameters in a script.

parser = argparse.ArgumentParser(description='Update the environment file for testing.')
parser.add_argument("--variable", type=str, dest="variable", help="Variable to update", required=True)
parser.add_argument("--PAT", type=str, dest="pat", help="Personal Access Token (PAT) with permissions to update the Variable library", required=True)
args = parser.parse_args()

Base64 encoding:
To access the API, we’re going to need to encode the authorisation data in base64. To make this simple and easier to test, I wrote a simple function.
There is most certainly a more succinct way of doing this, but for the sake of simplicity and clarity, this is fine.

def getB64String(string: str):
    sbytes = string.encode("ascii")
    b64bytes = base64.b64encode(sbytes)
    return b64bytes.decode("ascii")

Prepping the request:
In my experience, it’s much easier to work out where something is going wrong by splitting things up a little, so to prepare for the API request, I’m gluing all the pieces together first before attempting the request.
In this example, I’m using “f-strings” which allow me to concatenate variables easily in my string. For those interested, f-strings were introduced with Python 3.6. For context, at the time of writing, we’re up to Python 3.9.

encoded_authorisation = getB64String(f":{args.pat}")
base_uri = f"{URI_BASE}/{PROJECT}/_apis/distributedtask/variablegroups/{VARIABLE_GROUP_ID}?api-version=5.0-preview.1"
headers = {
    "Authorization": f"Basic {encoded_authorisation}"
}

Making the first request:
The requests library makes executing HTTP requests a breeze, if you’re not familiar with it already, it’s well worth a look.
Here, we make a GET request to the API using the URI we defined above, and the authentication headers dictionary we setup. When the response is returned, we should be able to parse the data as JSON.

response = requests.get(base_uri, headers=headers)
response_json = response.json()

A quick note here: If you get an error at this point, it’s most likely because the PAT is either incorrect or does not have the correct permissions to read the Variables data. You can add a line after the GET request print(response.status_code, response.text) that will display the response from the API. If it looks like HTML, it’s likely an issue with the token that was provided, make sure you’re not missing any characters.

If you’re interested in the full JSON structure of the response, you can also print the response_json object, or write it out to a file to look at.

Updating the values:
The only complicated part here is making sure the hierarchy within the JSON object is understood. To make things easier, we just use the variable parameter from earlier. This means we can apply the same script to any variable.

response_json["variables"][f"{args.variable}"]["value"] = int(response_json["variables"][f"{args.variable}"]["value"]) + 1

Storing the updated values:
Lastly, because the variables are loaded from the library when the pipeline starts up, we need to update both the library and the pipeline value so that any downstream scripts or steps within the pipeline are using the latest data.

The first line updates the API using a PUT request and the second line prints some text that contains a command that can be interpreted by the pipeline agent to update the local pipeline variable.

update_response = requests.put(base_uri, json=response_json, headers=headers)
print(f"##vso[task.setvariable variable={args.variable}]{new_value}")

That’s it!

All Together

Let’s look at the full script with all the parts together in context.

import requests
import base64
import argparse
URI_BASE = "https://dev.azure.com/{orgname}"
PROJECT = "{projectname}"
VARIABLE_GROUP_ID = {variablegroupid}

parser = argparse.ArgumentParser(description='Update the environment file for testing.')
parser.add_argument("--variable", type=str, dest="variable", help="Variable to update", required=True)
parser.add_argument("--PAT", type=str, dest="pat", help="Personal Access Token (PAT) with permissions to update the Variable library", required=True)
args = parser.parse_args()
def getB64String(string: str):
    sbytes = string.encode("ascii")
    b64bytes = base64.b64encode(sbytes)
    return b64bytes.decode("ascii")
encoded_authorisation = getB64String(f":{args.pat}")
base_uri = f"{URI_BASE}/{PROJECT}/_apis/distributedtask/variablegroups/{VARIABLE_GROUP_ID}?api-version=5.0-preview.1"
headers = {
    "Authorization": f"Basic {encoded_authorisation}"
}

print("URI", base_uri)
response = requests.get(base_uri, headers=headers)
response_json = response.json()
old_value = response_json["variables"][f"{args.variable}"]["value"]
response_json["variables"][f"{args.variable}"]["value"] = int(response_json["variables"][f"{args.variable}"]["value"]) + 1
new_value = response_json["variables"][f"{args.variable}"]["value"]
print(f"Updating '{args.variable}' from '{old_value}' to '{new_value}'")
update_response = requests.put(base_uri, json=response_json, headers=headers)
print("Update Response:", update_response.status_code)
print(f"##vso[task.setvariable variable={args.variable}]{new_value}")

Thanks for making it this far. I hope you found this useful.