Continuous integration and continuous delivery (CI/CD) is a very important aspects of DevOps practice. When it comes to the Kubernetes environment, a trend that applies to the practice of CI/CD is called GitOps. GitOps means that Git is the source of truth for the application deployment. In nutshell, GitOPs steps are as follows:
GitOps is summarized in the GitOps principles listed below, by the OpenGitOps working group. Here are the 4 GitOps principles.
There are 2 major tools for doing GitOps on a Kubernetes platform-- Flux and ArgoCD. Here we will mainly discuss Flux. According to the creators of Flux:
Flux is a set of continuous and progressive delivery solutions for Kubernetes that are open and extensible.
Once we understood the concept of GitOps, the next thing is to understand the architecture of Flux. At the moment, Flux has both version 1 and version 2 but version 1 is legacy. In these mini-workshops, we will be using Flux version 2. In version 2, Flux is developed using the GitOPs toolkit. GitOps toolkit is a set of controllers that can be independently used for doing different things around GitOPs operations. Here is the architecture of Flux, showing the controllers. At the moment, there are 5 controllers included in the GitOps toolkit:
The image-reflector-controller and image-automation-controller work together to update a Git repository when new container images are available.
Prerequisite:
Here is the step-by-step Plan:
Day 1: Set up your environment
Day 2: Understand Kustomize
Day 3: Understand Helm Charts
Day 4: Continuous delivery with Flux
Step 1: Flux Bootstrap
Step 2: Create Sources
Step 3: Deploy your application
Step 4: Continuous Delivery in action
Day 5: Conclusion
Day 1
Today we will handle the setup of the environment. We will install Kubernetes Cluster, install Flux, and Install Kustomize. Before we dedicate much time to these tools, let's understand what they do.
kustomize : as the name indicates, this is to customize k8s configuration files from the command line. It can be used as a standalone or used with kubectl -k. This is an overlay, meaning that kustomize can help replace some data/tests in your yaml files. If you have the same set of YAML files that you would like to use in different environments like dev, staging and production, kustomize can help you make that happen with minimal effort.
Helm: Helm is like apt or yum for Linux distributions. When it comes to deploying Kubernetes applications with ease, helm is the best option. You must create charts for your applications before you can install them with helm just as you need to create a .rpm pr .deb files before you can use yum and apt respectively. Helm is a templating system as comared to kustomize that is an overlay system.
Kind: For installing the Kuberntes we will use Kind (Kubernetes in Docker). Kubernetes has a way to quickly get up and running by deploying a cluster. Minikube, Docker for Desktop, and Kind are very popular. These days, I like using KIND for quick demos.
Here is the video on how to set up your environment on Mac.
Day 2
Kustomize can help you with customizing your Kubernetes manifest files for different scenarios/environments. For example, you can have a set of base YAML files and then modify them for use in Preprod, Staging, and Production environments.
Here is the directory structure we are going to build, step by step:
Have a base directory with the configuration files you would like to customize. In this directory, you include a file called kustomization.yaml. You can generate custom files here, but if you needed to create overlay folders, you can do the following still.
You can create 'Overlay directories' (folders for each of the environments you want to customize, for example, preprod, staging, and prod).
In each overlay folder, you would add another kustomization.yaml file, referencing the base kustomization file.
To generate the files, you would run any of the following commands. The directory you run the commands matters, or you can specify the location of kustomization.yaml file.
kustomize build > new-config-1.yaml or
kustomize build . > new-config-2.yaml
kubectl kustomize > new-config-3.yaml
kubectl kustomize /home/igbedo/lms-kustomize/ >new-config-4.yaml
When you run the command without redirecting to a file, the output will be on the screen.
For example:
kustomize build
After you generate the files, you can now use kubectl to create your files as usual. For example,
kubectl create -f new-config-4.yaml
1. Automatically generate kustomization files. You can do that with:
kustomize create --resources ../base
kustomize create --autodetect
Hint: type kustomize create <enter> and you'll see examples
2. kustomize edit set (remember to run kustomize build after edit set)
kustomize edit set nameprefix <prefix-value>
kustomize edit set image monopole/hello:1=monopole/hello:latest
Hint: type kustomize edit set <enter> and you'll see examples
3. kustomize edit add (Remember to run kustomize build after edit add)
kustomize edit add patch --kind Deployment --path patch.yaml
Hint: type kustomize edit add <enter>
With the above notes, you can get started working with kustomize.
References:
Day 3
There are a couple of ways you can manage your workload in a Kubernetes environment.
You can sometimes use a combination of the above methods in managing your application. Here we will package our application into a helm chart and then use helm to install it. This will set us up to understand how to use Flux in the next series of steps.
This tutorial is divided into 2 parts. In Part 1, we will learn how to use helm to install applications and in part 2 we will create a helm chart and use it.
To use helm to install applications, you need to find and add the helm repository where the application is located. The same application can be found in several repositories, depending on who created the package. Here we will install Nginx from different helm repos.
Here are the step-by-step instructions
helm repo add nginx-stable https://helm.nginx.com/stable
helm repo list
helm repo update
helm install my-release nginx-stable/nginx
helm list
helm uninstall my-release
to search a repo
helm search repo nginx
To Package our application into a helm chart, let's follow the steps below. Of course, make sure that helm is installed.
helm create secure-website
This will create a hierarchy of files and folders as seen below. You can customize any of the files to suite your purpose. In our own case, we need to customize the deployment and the service files. However, doing that requires an understanding of the way Helm creates YAML files for Kubernetes. Alternatively, the easier way is to use a utility to convert our YAML files into Helm format. Let's do the latter and make our life easy.
If you are on a mac laptop, you can use a tool called hemify to convert your YAML files into Helm format. Let's first install the hemify utility using brew thus:
brew install arttor/tap/helmify
awk 'FNR==1 && NR!=1 {print "---"}{print}' secure-website/*.yaml | helmify secure-website
We passed the directory for our YAML files (secure-website-yaml/*.yaml) to hemify and ask it to create a helm chart in secure-website directory.
helm install secure-website --generate-name
kubectl get pods
kubectl get svc
Now we have been able to convert our YAML files into a helm chart. If we want, we can share this chart with people all over the world, but that is not the aim of this lab.
At this point, we know what Helm does. It is an alternative to managing Kubernetes objects. Here we packaged our secure website into a Helm chart, and we installed it locally with helm install. We can uninstall with helm uninstall etc
Day 4:
With Flux, you can:
We will cover both steps in this blog.
For the setup, we need to have 2 Git repositories and of course, you need to have an account in GitHub or GitLab, or BitBucket. I am using GitHub here.
2. Github Repo 2: Application repo called edukate-gitops -- This contains the application that you are deploying with Flux. This repo will typically consist of :
The steps are:
Step 1: Flux Bootstrap (Done once though idempotent so can be run several times with no issues)
Step 2: Clone the Repo in step 1 above
Step 3: Deploy your application
Step 3.1: Create Sources (For every application you want to deploy)
Step 3.2: Deploy your application (For every application you want to deploy)
Step 4: Watch Flux sync the application
Before we start using flux we have to bootstrap Flux. This means that we have to get flux installed into our Kubernetes cluster. Remember that we installed Flux CLI earlier but this time we are installing Flux into the Kubernetes cluster. Flux will be installed into the Kubernetes cluster in the flux-system namespace. Flux bootstrap will, first of all, create a Git repository and deploy some components to it. To ease connecting to the cluster, we will set the credentials for our Git login by exporting the $USERNAME and $TOKEN for our Git account.
You can fill in the details thus:
export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>
We also need to create the ssh keys to be used for connecting to the repo. Follow the steps to create the keys:
The bootstrap step is required to be done only once in a cluster. However the command is idempotent so even if you run it several times, it won't hurt. I have discovered that if your cluster doesn't seem to be behaving as expected, running it helps to see what is going on and to help troubleshoot the problem.
Here is the command to bootstrap the cluster:
flux bootstrap
flux bootstrap github \
--components-extra=image-reflector-controller,image-automation-controller \--owner=$GITHUB_USER \ --repository=flux-edukate \ --branch=main \ --path=./clusters/k8scluster \ --personal
This command does the following:
flux-edukate
on your GitHub account/clusters/k8scluster/
in the repository. Note that you can specify any path you want.
Now that we have a repo created on our GitHub account, we need to clone it to our laptop or work environment. By cloning it, whenever we add our configuration files to the local repo and push the changes to the repository, Flux will ensure that our Kubernetes application is synced to reflect the state of the GitHub repo. Remember that our GitHub repo is the source of truth.
git clone ssh://git@github.com/$GITHUB_USER/flux-edukate
cd flux-edukate
At this point, we are done with installing Flux and cloning the flux bootstrap repo. We can now go ahead and deploy our applications to our cluster.
We can deploy as many applications as we want. For each application that we would like to deploy, we need to do 2 things:
1. Add the source for the application. If our application is stored in a Git repo, we can indicate that as the source. This application should normally contain a kustomize directory where the Kubernetes deployment objects (like deployment and service objects) are stored. The kustomize folder should also contain the kustomization.yaml file.
2. Create a Kustomization for the application. This kustomization file will be used to deploy the application in step 1.
We must do the above 2 steps from the directory that we cloned earlier. You can create the object in 3 different ways as shown in 1a, 1b and 1c below.
Take note of this ssh URL from edukate-gitops repository ssh://git@github.com/igbedo/edukate-gitops
First, I'll set up a GitRepository
giving access to the git repo. For read/write access, I need a deploy key (or some other means of authenticating, but a deploy key will be easiest). To make a key (give an empty passphrase):
ssh-keygen -f identity
You also need the host keys from github. To get the host keys and verify them:
ssh-keyscan github.com > known_hosts
ssh-keygen -l -f known_hosts
Now you can make a secret with the deploy key and known_hosts file:
kubectl create secret generic edukate-gitops --from-file=identity --from-file=known_hosts -n flux-system
Those two filenames -- identity
and known_hosts
-- are what the source controller library code expects, which makes it easier for the automation controller to use the GitRepository
type.
You also need to install the deploy key in GitHub. Copy it from identity.pub
(that's the public part of the key):
$ cat identity.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKM2wTSz5VyL2UCLh3ke9XUO1WUmAf
[...]w2FFnV24AGhWdP5lPOS/Jv64+OfMSF5E/e4dwVs= damian@macbook pro
and add under Settings / Deploy keys
for your fork on GitHub, giving it write access.
Now you can create a GitRepository
which will provide access to the git repository within the cluster. We can do this in 3 ways. All are valid so use whichever you prerfer.
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
name: edukate-gitops
namespace: flux-system
spec:
interval: 30s
ref:
branch: main
url: ssh://git@github.com/igbedo/edukate-gitops
secretRef:
name: edukate-gitops
$ kubectl apply -f gitrepo.yaml
gitrepository.source.toolkit.fluxcd.io/cuttlefacts-repo created
$ kubectl get gitrepository
NAME URL READY STATUS AGE
cuttlefacts-repo ssh://git@github.com/squaremo/cuttlefacts-app 9s
flux create source git edkukate-gitops \
--url=https://github.com/igbedo/edukate-gitops \
--branch=main \
--interval=20s \
--secret-ref edukate-gitops \
--export > ./clusters/k8scluster/edukate-gitops-source.yaml
The above command will generate the file below but if you skip step 1b, you can just create this file (like any other Kubernetes manifest file)
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
name: edukate-gitops
namespace: flux-system
spec:
interval: 30s
ref:
branch: main
url: ssh://git@github.com/igbedo/edukate-gitops
secretRef:
name: edukate-gitops
This file will be located in flux-edukate/clusters/k8scluster/
as indicated in step 1b but if you create the file manually, ensure to place it in the same location.
Commit and push the
edukate-gitops-source.yaml
file to the flux-edukate
repository:
git add -A && git commit -m "Add the source file to GitRepository"
git push
Configure Flux to build and apply the kustomize directory located in the edukate repository.
Use the flux create
command to create a kustomization that applies the edukate deployment.
flux create kustomization edukate-gitops \
--target-namespace=default \
--source=edukate-gitops \
--path="./kustomize" \
--prune=true \
--interval=5m \
--export > ./clusters/k8scluster/edukate-gitops-kustomization.yaml
The output is similar to the file below but you can also create this file manually like any other Kubernetes manifest file:
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: edukate-gitops
namespace: flux-system
spec:
interval: 4m0s
path: ./kustomize
prune: true
sourceRef:
kind: GitRepository
name: edukate-gitops
targetNamespace: default
Commit and push the Kustomization
manifest to the repository:
git add -A && git commit -m "Add edukate gitops Kustomization"
git push
The structure of the flux-edukate
repo should be similar to:
flux-edukate
└── clusters/
└── k8cluster/
├── flux-system/
│ ├── gotk-components.yaml
│ ├── gotk-sync.yaml
│ └── kustomization.yaml
├── edukate-gitops-kustomization.yaml
└── edukate-gitops-source.yaml
Use the flux get
command to watch the edukate app
flux get kustomizations --watch
The output is similar to:
NAME REVISION SUSPENDED READY MESSAGE
flux-system main/680d4f8 False True Applied revision: main/680d4f8
flux-edukate master/49f414f False True Applied revision: master/c9f414f
Check edukate has been deployed on your cluster:
kubectl get deployments,services
The output is similar to:
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/edukate 2/2 2 2 50s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/edukate-service NodePort 10.111.214.212 <none> 80:30004/TCP 50s
Changes made to the edukate Kubernetes manifests in the main branch are reflected in your cluster. Take note of the following points:
Kustomization
from the flux-edukate repository, Flux removes all Kubernetes objects previously applied from that Kustomization
.kubectl edit
, the changes are reverted to match the state described in Git.
This is the workflow
The major steps are:
To create these files, you have 2 main methods:
Method 1. Use kubectl create -f to create them from Kubernetes manifests files
Method 2. Create the files and upload them to the flux-edukate repo. Once pushed, the objects are created on the Kubernetes cluster.
Let's go through the step-by-step procedure for this configuration.
At some stage, Github Actions would need to push images to DockerHub. To make this possible, we need to create secrets for GitHub Action to use and insert those in the configuration file for GitHub Actions. This is the section of the file where the secrets are configured.
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
This process will be done both in DockerHub and in GitHub.
For this process, go to Dockerhub and create Access token as follows:
Here are the steps:
Repeat the above steps to create DOCKERHUB_USERNAME and paste your DockerHub username.
Here is the file that we will use for GitHub Action. You can find a sample in Flux documentation on this Flux GitHub Action Example and modify it but even the default will work for what I want to do here. This file must be placed under .github/workflows/edukate-ci.yaml
name: CI
on:
push:
branches: [main]
jobs:
build-push:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Generate build ID
id: prep
run: |
branch=${GITHUB_REF##*/}
sha=${GITHUB_SHA::8}
ts=$(date +%s)
echo "::set-output name=BUILD_ID::${branch}-${sha}-${ts}"
- name: Set up QEMU
uses: docker/setup-qemu-action@v1
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build and publish container image
uses: docker/build-push-action@v2
with:
push: true
context: .
file: ./Dockerfile
tags: |
igbedo/edukate:${{ steps.prep.outputs.BUILD_ID }}
The deployment in edukate-gitops uses the image igbedo/edukate
. We'll automate that so it gets updated when there's a new image tag available, e.g., igbedo/edukate-main-d5a8892b-1668543264
Keeping track of the most recent image takes two resources: an ImageRepository
, to scan DockerHub for the image's tags, and an ImagePolicy
, to give the particular policy for selecting an image (here, a semver range).
The ImageRepository
:
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
name: edukate-gitops
namespace: flux-system
spec:
image: igbedo/edukate
interval: 1m0s
... and the policy:
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
name: edukate-gitops
namespace: flux-system
spec:
imageRepositoryRef:
name: edukate-gitops
filterTags:
pattern: "^main-[a-f0-9]+-(?P<ts>[0-9]+)"
extract: "$ts"
policy:
numerical:
order: asc
Apply these into the cluster, and the image reflector controller (installed as a prerequisite, above) will scan for the tags of the image and figure out which one to use. You can see this by asking for the status of the image policy:
$ kubectl get imagepolicy app-policy
NAME LATESTIMAGE
app-policy cuttlefacts/cuttlefacts-app:1.0.0
Now we have an image policy, which calculates the most recent image, and a git repository to update, and we've marked the field to update, in a file. The last ingredient is to tie these together with an ImageUpdateAutomation
resource:
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
name: edukate-gitops
namespace: flux-system
spec:
interval: 1m
sourceRef:
kind: GitRepository
name: edukate-gitops
git:
checkout:
ref:
branch: main
commit:
author:
email: fluxcdbot@users.noreply.github.com
name: fluxcdbot
messageTemplate: |
Automated image updated by Flux
[ci skip]
push:
branch: main
update:
path: ./kustomize
strategy: Setters
The git repository object is mentioned, and the setters
value gives the paths to apply updates under.
Apply the file to create the automation object:
kubectl apply -f update.yaml
Once that's created, it should quickly commit a change to the git repository, to make the image in the deployment match the most recent given by the image policy.
Here is the file that will create all the required policies. This file must be pushed to the flux-edukate repo.
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
name: edukate-gitops
namespace: flux-system
spec:
imageRepositoryRef:
name: edukate-gitops
filterTags:
pattern: "^main-[a-f0-9]+-(?P<ts>[0-9]+)"
extract: "$ts"
policy:
numerical:
order: asc
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
name: edukate-gitops
namespace: flux-system
spec:
image: igbedo/edukate
interval: 1m0s
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
name: edukate-gitops
namespace: flux-system
spec:
interval: 1m
sourceRef:
kind: GitRepository
name: edukate-gitops
git:
checkout:
ref:
branch: main
commit:
author:
email: fluxcdbot@users.noreply.github.com
name: fluxcdbot
messageTemplate: |
Automated image updated by Flux
[ci skip]
push:
branch: main
update:
path: ./kustomize
strategy: Setters
To tell the controller what to update, you add some markers to the files to be updated. Each marker says which field to update, and which image policy to use for the new value.
In this case, it's the image in the deployment that needs to be updated, with the latest image from the image policy made earlier. Edit the file either locally or through GitHub, and add a marker to the file kustomize/deployment.yaml
at the line with the image field, image: igbedo/edukate
. The surrounding lines look like this:
containers:
- name: edukate
image: igbedo/edukate
imagePullPolicy: IfNotPresent
With the marker, they look like this:
containers:
- name: edukate
image: igbedo/edukate:main-447331f1-1668215829 # {"$imagepolicy": "flux-system:edukate-gitops"}
imagePullPolicy: IfNotPresent
The marker is a comment at the end of the image:
line, with a JSON value (so remember the double quotes), naming the image policy object to use for the value. A :
character separates the namespace from the name of the ImagePolicy
object. (The namespace is flux-system as specified in the manifest.
Commit that change, and push it if you made the commit locally.
apiVersion: apps/v1
kind: Deployment
metadata:
name: edukate
namespace: default
labels:
app: edukate
spec:
replicas: 2
selector:
matchLabels:
app: edukate
template:
metadata:
labels:
app: edukate
spec:
containers:
- name: edukate
image: igbedo/edukate:main-447331f1-1668215829 # {"$imagepolicy": "flux-system:edukate-gitops"}
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
images:
- name: igbedo/edukate
newName: igbedo/edukate # {"$imagepolicy": "flux-system:edukate-gitops:name"}
newTag: main-447331f1-1668215829 # {"$imagepolicy": "flux-system:edukate-gitops:tag"}
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: edukate-gitops
namespace: flux-system
spec:
interval: 5m0s
path: ./kustomize
prune: true
sourceRef:
kind: GitRepository
name: edukate-gitops
targetNamespace: default
After the above steps, we have a fully functional CI/CD using Flux. We can test by making changes to our application and pushing it to a repo. After that, the steps enumerated earlier will follow. Below is the workflow:
I have shown you how to use FluxCD for managing your applications and performing CI/CD . All the steps are documented to make this a smooth ride for you. I have come to Love Flux, I think it's a valuable tool for Kubernetes and once you start playing with it, you may never go back to how you were deploying your applications anymore. I hope you found this useful.