# First steps with Crossplane

- URL: https://baeke.info/2022/07/23/first-steps-with-crossplane/
- Author: Geert Baeke
- Published: July 23, 2022 (updated July 24, 2022)
- Topics: Automation, Azure, Cloud, Containers, GitHub, Kubernetes, azure, crossplane, gitops, kubernetes
- Outdated: provider-jet-azure was replaced by the Upbound Azure providers.

Image Source: crossplane.io

Although [Crossplane](https://crossplane.io/) has been around for a while, I never got around to trying it. Crossplane has many capabilities. However, in this post, I will focus on the following aspects:

-   Installing Crossplane on a Kubernetes cluster (AKS); you can install on a local cluster as well (e.g., k3s, kind, minikube, …) but then you would need Azure Arc for Kubernetes to install the microsoft.flux extension (I will be using GitOps with Flux via that extension)
-   Adding and configuring providers for Azure and Kubernetes: providers allow you to deploy to Azure and Kubernetes (and much more) from Crossplane
-   Deploying Azure infrastructure with Crossplane using a fully declarative GitOps approach

## Introduction

Crossplane basically allows you to build a control plane that you or your teams can use to deploy infrastructure and applications. This control plane is built on Kubernetes. In short, suppose I want to deploy an Azure resource group with Crossplane, I would create the below YAML file and apply it with `kubectl apply -f filename.yaml`.

This is, in essence, a fully declarative approach to deploying Azure infrastructure using Kubernetes. There are other projects, such as the [Azure Servic](https://github.com/Azure/azure-service-operator)[e](https://github.com/Azure/azure-service-operator) [Operator v2](https://github.com/Azure/azure-service-operator), that do something similar.

```yaml
apiVersion: azure.jet.crossplane.io/v1alpha2
kind: ResourceGroup
metadata:
  name: rg-crossplane
spec:
  forProvider:
    location: "westeurope"
    tags:
      provisioner: crossplane
  providerConfigRef:
    name: default
```

In order to enable this functionality, you need the following:

-   Install Crossplane on your Kubernetes cluster
-   Add a provider that can create Azure resources; above the [jet provider for Azure](https://github.com/crossplane-contrib/provider-jet-azure) is used; more about providers later
-   Configure the provider with credentials; in this case [Azure credentials](https://crossplane.io/docs/v1.9/cloud-providers/azure/azure-provider.html)

In a diagram:

![](https://baeke.info/_astro/2022-07-image-10.4mcq8Toe_1tq8BX.webp)

*Install Crossplane from git with Flux on AKS; deploy an Azure resource group and another AKS cluster from Crossplane; create a namespace on that new cluster*

## Combination with GitOps

Although you can install and configure Crossplane manually and just use kubectl to add custom resources, I wanted to add Crossplane and custom resources using GitOps. To that end, I am using Azure Kubernetes Service (AKS) with the `microsoft.flux` extension. For more information to enable and install the extension, see my Flux v2 [quick guide](https://github.com/gbaeke/quick-guides/blob/main/fluxv2/README.md).

⚠️ The git repository I am using with Flux v2 and Crossplane is here: [https://github.com/gbaeke/crossplane/tree/blogpost](https://github.com/gbaeke/crossplane/tree/blogpost). This refers to the **blogpost** branch, which should match the content of this post. Tbe **main** branch might be different.

The repo contains several folders that match Flux kustomizations:

-   **infra folder**: installs Crossplane and Azure Key Vault to Kubernetes; an infra kustomization will point to this folder
-   **secrets folder**: creates a secret with Azure Key Vault to Kubernetes from Azure Key Vault; the secrets kustomization will point to this folder
-   **crossplane-apps folder**: installs Azure resources and Kubernetes resources with the respective Crossplane providers; the apps kustomization will point to this folder

**Note**: if you do not know what Flux kustomizations are and how Flux works, do check my Flux playlist: [https://www.youtube.com/playlist?list=PLG9qZAczREKmCq6on\_LG8D0uiHMx1h3yn](https://www.youtube.com/playlist?list=PLG9qZAczREKmCq6on_LG8D0uiHMx1h3yn). The videos look at the open source version of Flux and not the microsoft.flux extension. To learn more about that extension, see [https://www.youtube.com/watch?v=w\_eoJbgDs3g](https://www.youtube.com/watch?v=w_eoJbgDs3g).

## Installing Crossplane

The **infra** customization installs Crossplane and [Azure Key Vault to Kubernetes](https://github.com/SparebankenVest/azure-key-vault-to-kubernetes). The latter is used to sync a secret from Key Vault that contains credentials for the Crossplane Azure provider. More details are in the diagram below:

![](https://baeke.info/_astro/2022-07-image.DhdsQ0WU_2jA6vX.webp)

As noted above, the installation of Crossplane is done with Flux. First, there is the HelmRepository resource that adds the Crossplane Helm repository to Flux.

```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  namespace: config-infra
  name: crossplane
spec:
  interval: 1m0s
  url: https://charts.crossplane.io/stable
```

Next, there is the HelmRelease that installs Crossplane. Important: target namespace is **crossplane-system** (bottom line):

```yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: crossplane
  namespace: config-infra
spec:
  chart:
    spec:
      chart: crossplane
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: crossplane
        namespace: config-infra
  install:
    createNamespace: true
  interval: 1m0s
  targetNamespace: crossplane-system
```

For best results, in the YAML above, set the namespace of the resource to the namespace you use with the AKS k8s-configuration. The resources to install Azure Key Vault to Kubernetes are similar.

To install the Crossplane jet provider for Azure:

```yaml
---
apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: jet-azure-config
  labels:
    app: crossplane-provider-jet-azure
spec:
  image: crossplane/provider-jet-azure-controller:v0.9.0
  args: ["-d"]
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: crossplane-provider-jet-azure
spec:
  package: crossplane/provider-jet-azure:v0.9.0
  controllerConfigRef:
    name: jet-azure-config
```

Above, debugging is turned on for the provider. This is optional. The provider actually runs in the **crossplane-system** namespace:

![](https://baeke.info/_astro/2022-07-image-1.BPSPYG3e_Z1CyXqk.webp)

*jet provider*

The provider is added via the **Provider** resource (second resource in the YAML manifest).

We can now create the AKS **k8s-configuration**, which creates a Flux source and a kustomization:

```bash
RG=your AKS resource group
CLUSTER=your AKS cluster name (to install Crossplane to)

az k8s-configuration flux create -g $RG -c $CLUSTER \
  -n cluster-config --namespace config-infra -t managedClusters \
  --scope cluster -u https://github.com/gbaeke/crossplane \
  --branch main  \
  --kustomization name=infra path=./infra prune=true
```

The Flux source will be the repo specified with -u. There is one kustomization: infra. Pruning is turned on. With pruning, removing manifests from the repo results is removing them from Kubernetes.

The k8s-configuration should result in:

![](https://baeke.info/_astro/2022-07-image-2.DUJyh7_y_15IQyF.webp)

*Don’t mind the other Kustomizations; will be added later; this is the GitOps view in the properties of the cluster in the Azure Portal*

Crossplane is now installed with two providers. We can now configure the Azure provider with credentials.

## Configuring Azure Credentials

You need to create a service principal by following the steps in [https://crossplane.io/docs/v1.9/cloud-providers/azure/azure-provider.html](https://crossplane.io/docs/v1.9/cloud-providers/azure/azure-provider.html). I compacted the resulting JSON with:

```bash
cat <path-to-JSON> | jq -c
```

The output of the above command was added to Key Vault:

![](https://baeke.info/_astro/2022-07-image-3.B3fJGR2v_Z2olcDD.webp)

*Azure creds in Key Vault*

The Key Vault I am using uses the **Azure RBAC** permission model. Ensure that the AKS cluster’s kubelet identity has at least the **Key Vault Secrets User** role. It is a user-assigned managed identity with a name like **clustername-agentpool**.

To actually create a Kubernetes secret from this Key Vault secret, the **secrets** folder in the git repo contains the manifest below:

```yaml
apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: azure-creds 
  namespace: crossplane-system
spec:
  vault:
    name: kvgebadefault # name of key vault
    object:
      name: azure-creds # name of the akv object
      type: secret # akv object type
  output: 
    secret: 
      name: azure-creds # kubernetes secret name
      dataKey: creds # key to store object value in kubernetes secret
```

This creates a Kubernetes secret in the **crossplane-system** namespace with name **azure-creds** and a key **creds** that holds the credentials JSON.

![](https://baeke.info/_astro/2022-07-image-4.DonUpP05_Z11cqha.webp)

*Secret as seen in k9s*

![](https://baeke.info/_astro/2022-07-image-5.78CIRzBI_Z1zl0K2.webp)

*the decoded secret as shown in k9s*

To add the secret(s) as an extra kustomization, run:

```bash
RG=your AKS resource group
CLUSTER=your AKS cluster name

az k8s-configuration flux create -g $RG -c $CLUSTER \
  -n cluster-config --namespace config-infra -t managedClusters \
  --scope cluster -u https://github.com/gbaeke/crossplane \
  --branch main  \
  --kustomization name=infra path=./infra prune=true \
  --kustomization name=secrets path=./secrets prune=true dependsOn=["infra"]
```

Note that the secrets kustomization is **dependent on** the infra kustomization. After running this command, ensure the secret is in the **crossplane-system** namespace. The k8s-configuration uses the same source but now has two kustomizations.

## Deploying resources with the Jet provider for Azure

Before explaining how to create Azure resources, a note on providers. As a novice Crossplane user, I started with the following Azure provider: [https://github.com/crossplane-contrib/provider-azure](https://github.com/crossplane-contrib/provider-azure). This works well but it is not so simple for contributors to ensure the provider is up-to-date with the latest and greatest Azure features. For example, if you deploy AKS, you cannot use managed identity, the cluster uses availability sets etc…

To improve this, [Terrajet](https://blog.crossplane.io/announcing-terrajet/) was created. It is a code generation framework that can generate Crossplane CRDs (custom resource definitions) and sets up the provider to use Terraform. Building on top of Terraform is an advantage because it is more up-to-date with new cloud features. That is the reason why this post uses the jet provider. When we later create an AKS cluster, it will take advantage of managed identity and other newer features.

**Note:** there is also a Terraform provider that can take Terraform HCL to do anything you want; we are not using that in this post

Ok, let’s create a resource group and deploy AKS. First, we have to configure the provider with Azure credentials. The **crossplane-apps** folder contains a file called **jet-provider-config.yaml**:

```yaml
apiVersion: azure.jet.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: azure-creds
      key: creds
```

The above ProviderConfig tells the provider to use the credentials in the Kubernetes secret we created earlier. We know we are configuring the jet provider from the apiVersion: **azure.jet.crossplane.io/v1alpha1**.

With that out of the way, we can create the resource group and AKS cluster. Earlier in this post, the YAML to create the resource group was already shown. To create a basic AKS cluster called **clu-cp** in this group, **aks.yaml** is used:

```yaml
apiVersion: containerservice.azure.jet.crossplane.io/v1alpha2
kind: KubernetesCluster
metadata:
  name: clu-cp
spec:
  writeConnectionSecretToRef:
    name: example-kubeconfig
    namespace: crossplane-system
  forProvider:
    location: "westeurope"
    resourceGroupNameRef:
      name: rg-crossplane
    dnsPrefix: "clu-cp"
    defaultNodePool:
      - name: default
        nodeCount: 1
        vmSize: "Standard_D2_v2"
    identity:
      - type: "SystemAssigned"
    tags:
      environment: dev
  providerConfigRef:
    name: default
```

Above, we refer to our resource group by name (resourceGroupNameRef) and we write the credentials to our cluster to a secret (writeConnectionSecretToRef). That secret will contain keys with the certificate and private key, but also a **kubeconfig** key with a valid kubeconfig file. We can use that later to connect and deploy to the cluster.

To see an example of connecting to the deployed cluster and creating a namespace, see **k8s-provider-config.yaml** and **k8s-namespace.yaml** in the repo. The resource **k8s-provider-config.yaml** will use the **example-kubeconfig** secret created above to connect to the AKS cluster that we created in the previous steps.

To create a kustomization for the **crossplane-apps** folder, run the following command:

```bash
RG=your AKS resource group
CLUSTER=your AKS cluster name

az k8s-configuration flux create -g $RG -c $CLUSTER \
  -n cluster-config --namespace config-infra -t managedClusters \
  --scope cluster -u https://github.com/gbaeke/crossplane \
  --branch main  \
  --kustomization name=infra path=./infra prune=true \
  --kustomization name=secrets path=./secrets prune=true dependsOn=["infra"] \
  --kustomization name=apps path=./crossplane-apps prune=true dependsOn=["secrets"]
```

This folder does not contain a kustomization.yaml file. Any manifest you drop in it will be applied to the cluster! The k8s-kustomization now has the same source but three kustomizations:

![](https://baeke.info/_astro/2022-07-image-6.bZuOW8LG_E109w.webp)

*infra, secrets and apps kustomizations*

After a while, an AKS cluster **clu-cp** should be deployed to resource group **rg-crossplane**:

![](https://baeke.info/_astro/2022-07-image-7.Cvd6xbCh_Z9OFHg.webp)

*AKS deployed by Crossplane running on another AKS cluster*

To play around with this, I recommend using Visual Studio Code and the GitOps extension. When you make a change locally and push to main, to speed things up, you can reconcile the git repository and the apps kustomization manually:

![](https://baeke.info/_astro/2022-07-image-8.dgInRVFT_Z16Asn4.webp)

*Reconcile the GitRepository source and kustomization from the GitOps extension for Visual Studio Code*

## Conclusion

In this post, we looked at installing and configuring Crossplane on AKS via GitOps and the microsoft.flux extension. In addition, we deployed a few Azure resources with Crossplane and its jet provider for Azure. We only scratched the surface here but I hope this gets you started quickly when evaluating Crossplane for yourself.
