# Kubernetes Canary Deployments with GitHub Actions

- URL: https://baeke.info/2021/01/11/kubernetes-canary-deployments-with-github-actions/
- Author: Geert Baeke
- Published: January 11, 2021
- Topics: cicd, github, kubernetes

In the [previous post](https://baeke.info/2021/01/06/a-look-at-github-actions-for-azure-and-aks-deployments/), we looked at some of the GitHub Actions you can use with Microsoft Azure. One of those actions is the [azure/k8s-deploy](https://github.com/Azure/k8s-deploy) action which is currently at v1.4 (January 2021). To use that action, include the following snippet in your workflow:

```text
- uses: azure/k8s-deploy@v1.4
  with:
    namespace: go-template
    manifests: ${{ steps.bake.outputs.manifestsBundle }}
    images: |
      ghcr.io/gbaeke/go-template:${{ env.IMAGE_TAG }}
```

The above snippet uses baked manifests from an earlier azure/k8s-bake action that uses kustomize as the render engine. This is optional and you can use individual manifests without kustomize. It also replaces the image it finds in the baked manifest with an image that includes a specific tag that is set as a variable at the top of the workflow. Multiple images can be replaced if necessary.

The azure/k8s-deploy action supports different styles of deployments, defined by the **strategy** action input:

-   none: if you do not specify a strategy, a standard Kubernetes [rolling update](https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/#:~:text=In%20Kubernetes%20this%20is%20done,on%20Nodes%20with%20available%20resources.) is performed
-   [canary](https://martinfowler.com/bliki/CanaryRelease.html): deploy a new version and direct a part of the traffic to the new version; you need to set a **percentage** action input to control the traffic split; in essence, a percentage of your users will use the new version
-   [blue-green](https://martinfowler.com/bliki/BlueGreenDeployment.html): deploy a new version next to the old version; after testing the new version, you can switch traffic to the new version

In this post, we will only look at the canary deployment. If you read the description above, it should be clear that we need a way to split the traffic. There are several ways to do this, via the **traffic-split-method** action input:

-   pod: the default value; by tweaking the amount of “old version” and “new version” pods, the standard load balancing of a Kubernetes service will approximate the percentage you set; pod uses standard Kubernetes features so no additional software is needed
-   [smi](https://smi-spec.io/): you will need to implement a service mesh that supports [TrafficSplit](https://github.com/servicemeshinterface/smi-spec/blob/master/apis/traffic-split/v1alpha3/traffic-split.md); the traffic split is controlled at the request level by the service mesh and will be precise

Although pod traffic split is the easiest to use and does not require additional software, it is not very precise. In general, I recommend using [TrafficSplit](https://github.com/servicemeshinterface/smi-spec/blob/master/apis/traffic-split/v1alpha3/traffic-split.md) in combination with a service mesh like [linkerd](https://linkerd.io/), which is very easy to implement. Other options are [Open Service Mesh](https://openservicemesh.io/) and of course, [Istio](https://istio.io/).

With this out of the way, let’s see how we can implement it on a standard Azure Kubernetes Service (AKS) cluster.

## Installing linkerd

Installing linkerd is easy. First install the cli on your system:

```bash
curl -sL https://run.linkerd.io/install | sh
```

Alternatively, use brew to install it:

```bash
brew install linkerd
```

Next, with kubectl configured to connect to your AKS cluster, run the following commands:

```bash
linkerd check --pre
linkerd install | kubectl apply -f -
linkerd check
```

## Preparing the manifests

We will use three manifests, in combination with kustomize. You can find them on [GitHub](https://github.com/gbaeke/azure-deploy/tree/main/deploy). In [namespace.yaml](https://github.com/gbaeke/azure-deploy/blob/main/deploy/namespace.yaml), the **linkerd.io/inject** [annotation](https://linkerd.io/2/features/proxy-injection/) ensures that the entire namespace is meshed. Every pod you create will get the linkerd [sidecar](https://linkerd.io/2/reference/architecture/) injected, which is required for traffic splitting.

In the GitHub workflow, the manifests will be “baked” with kustomize. The result will be one manifest file:

```text
- uses: azure/k8s-bake@v1
  with:
    renderEngine: kustomize
    kustomizationPath: ./deploy/
  id: bake
```

The action above requires an id. We will use that id to refer to the resulting manifest later with:

```
${{ steps.bake.outputs.manifestsBundle }}
```

**Important note**: I had some trouble using the baked manifest and later switched to using the individual manifests; I also deployed namespace.yaml in one action and then deployed service.yaml and deployment.yaml is a separate action; to deploy multiple manifests, use the following syntax:

```text
- uses: azure/k8s-deploy@v1.4
  with:
    namespace: go-template
    manifests: |
      ./deploy/service.yaml
      ./deploy/deployment.yaml
```

## First run

We got to start somewhere so we will deploy version 0.0.1 of the [ghcr.io/gbaeke/go-template](https://github.com/users/gbaeke/packages/container/package/go-template) image. In the [deployment workflow](https://github.com/gbaeke/azure-deploy/blob/main/.github/workflows/app.yml), we set the IMAGE\_TAG variable to 0.0.1 and have the following action:

```text
- uses: azure/k8s-deploy@v1.4
  with:
    namespace: go-template
    manifests: ${{ steps.bake.outputs.manifestsBundle }}
    # or use individual manifests in case of issues 🙂
    images: |
      ghcr.io/gbaeke/go-template:${{ env.IMAGE_TAG }}
    strategy: canary
    traffic-split-method: smi
    action: deploy #deploy is the default; we will later use this to promote/reject
    percentage: 20
    baseline-and-canary-replicas: 2
```

Above, the action inputs set the **canary** strategy, using the **smi** method with 20% of traffic to the new version. The **deploy** action is used which results in “canary” pods of version 0.0.1. It’s not actually a canary because there is no stable deployment yet and all traffic goes to the “canary”.

This is what gets deployed:

![](https://baeke.info/_astro/2021-01-image-6.Bz1fn8oE_2nSLJD.webp)

*Initial, canary-only release*

There only is a canary deployment with 2 canary pods in the deployment (we asked for 2 explicitly in the action). There are four services: the main go-template-service and then a service for baseline, canary and stable. Instead of deploy, you can use promote directly (action: promote) to deploy a stable version right away.

If we run **linkerd dashboard** we can check the namespace and the Traffic Split:

![](https://baeke.info/_astro/2021-01-image-7.BkoMvLmv_2371pk.webp)

*TrafficSplit in linkerd; all traffic to canary*

Looked at in another way:

![](https://baeke.info/_astro/2021-01-image-8.DpeUUGP1_11hCT0.webp)

*TrafficSplit*

All traffic goes to the canary. In the Kubernetes TrafficSplit object, the weight is actually set to 1000m which is shown as 1 above.

## Promotion

We can now modify the pipeline, change the action input action of azure/k8s-deploy to **promote** and trigger the workflow to run. This is what the action should look like:

```text
- uses: azure/k8s-deploy@v1.4
  with:
    namespace: go-template
    manifests: ${{ steps.bake.outputs.manifestsBundle }}
    images: |
      ghcr.io/gbaeke/go-template:${{ env.IMAGE_TAG }}
    strategy: canary
    traffic-split-method: smi
    action: promote  #deploy is the default; we will later use this to promote/reject
    percentage: 20
    baseline-and-canary-replicas: 2
```

This is the result of the promotion:

![](https://baeke.info/_astro/2021-01-image-10.CmUp4NnG_2wObdw.webp)

*After promotion*

As expected, we now have 5 pods of the 0.0.1 deployment. This is the stable deployment. The canary pods have been removed. We get five pods because that is the number of replicas in deployment.yaml. The baseline-and-canary-replicas action input is not relevant now as there are no canary and baseline deployments.

The TrafficSplit now directs 100% of traffic to the stable service:

![](https://baeke.info/_astro/2021-01-image-11.B04jHoKc_uhaHm.webp)

*All traffic to “promoted” stable service*

## Deploying v0.0.2 with 20% split

Now we can deploy a new version of our app, version 0.0.2. The action is the same as the initial deploy but IMAGE\_TAG is set to 0.0.2:

```text
- uses: azure/k8s-deploy@v1.4
  with:
    namespace: go-template
    manifests: ${{ steps.bake.outputs.manifestsBundle }}
    images: |
      ghcr.io/gbaeke/go-template:${{ env.IMAGE_TAG }}
    strategy: canary
    traffic-split-method: smi
    action: deploy #deploy is the default; we will later use this to promote/reject
    percentage: 20
    baseline-and-canary-replicas: 2
```

Running this action results in:

![](https://baeke.info/_astro/2021-01-image-12.NDV9QwM0_ZVsMVc.webp)

*Canary deployment of 0.0.2*

The stable version still has 5 pods but canary and baseline pods have been added. More info about baseline below.

TrafficSplit is now:

![](https://baeke.info/_astro/2021-01-image-13.BAZKs_uQ_N9JAs.webp)

*TrafficSplit: 80% to stable and 20% to baseline & canary*

Note that the baseline pods uses the same version as the stable pods (here 0.0.1). The baseline should be used to compare metrics with the canary version. You should not compare the canary to stable because factors such as caching might influence the comparison. This also means that, instead of 20%, only 10% of traffic goes to the new version.

## Wait… I have to change the pipeline to promote/reject?

Above, we manually changed the pipeline and ran it manually from VS Code or the GitHub website. This is possible with triggers such as **repository\_dispatch** and **workflow\_dispatch**. There are (or should be) some ways to automate this better:

-   GitHub environments: with Azure DevOps, it is [possible](https://docs.microsoft.com/en-us/azure/devops/pipelines/ecosystems/kubernetes/canary-demo?view=azure-devops&tabs=yaml#initiate-canary-workflow) to run jobs based on environments and the approve/reject status; I am still trying to figure out if this is possible with GitHub Actions but it does not look like it (yet); if you know, drop it in the comments; I will update this post if there is a good solution
-   workflow\_dispatch inputs: if you do want to run the workflow manually, you can use workflow\_dispatch inputs to approve/reject or do nothing

## Should you use this?

While I think the GitHub Action works well, I am not in favor of driving all this from GitHub, Azure DevOps and similar solutions. There’s just not enough control imho.

Solutions such as [flagger](https://flagger.app/) or [Argo Rollouts](https://argoproj.github.io/argo-rollouts/) are progressive delivery operators that run inside the Kubernetes cluster. They provide more operational control, are fully automated and can be integrated with Prometheus and/or service mesh metrics. For an example, check out [one of my videos](https://www.youtube.com/watch?v=y2dYlNnZrtM&t=532s). When you need canary and/or blue-green releases and you are looking to integrate the progression of your release based on metrics, surely check them out. They also work well for manual promotion via a CLI or UI if you do not need metrics-based promotion.

## Conclusion

In this post we looked at the “mechanics” of canary deployments with GitHub Actions. An end-to-end solution, fully automated and based on metrics, in a more complex production application is quite challenging. If your particular application can use simpler deployment methods such as standard Kubernetes deployments or even blue-green, then use those!
