Testing the Operator SDK and making a prefetch mechanism for Kubernetes


In this article we will explore how to create an operator that can prefetch our images (from our deployments to all nodes) using the Operator SDK, you might be wondering why would you want to do this? the main idea is to get the images in advance so you don’t have to pull them when the pod actually needs to start running in a given node, this can speed up things a bit and it’s also an interesting exercise.

If you have read the article Cloud native applications with kubebuilder and kind aka kubernetes operators you will note that the commands are really similar between each other, since now the operator-sdk uses kubebuilder, you can read more here.

The source for this article is here


Creating our local cluster

Kind config for multi-cluster

This is the kind config necessary to have a multi-node setup locally: kind create cluster --config kind.yaml

Creating the cluster

We will need a cluster to run and test our operator, so kind is pretty straight forward and lightweight enough to run anywhere.

Creating our operator

Here we bootstrap our go project aka as kubernetes operator

Creating our API

This will be the object that it will hold all the important information for a given image, the files that we need to modify at first hand are in: controllers/*_controller.go and api/v1/*_types.go

Building and pushing (docker image)

Basic build and push of the operator image with the projects helper


Now that we have the project built into a docker image and stored in dockerhub then we can install our CRD and then deploy the operator

Deploy the operator

Then we can deploy our operator

Validate that our operator was deployed

Check that our pods are running

So far everything is peachy but our operator is kind of useless at the moment, so let’s drop some code to make it do what we want…

Our code

A lot of what we use is generated however we need to give it some specific permissions and behaviour to our operator so it does what we want when we create an object in kubernetes

Our manifest

This will be the manifest that we will be using to tell our operator which deployments we want to prefetch images for

Sample nginx deployment

This nginx deployment will be used to validate that the images are fetched in all nodes We don’t actually need to do this, but this way it’s easy to make sure that a pod won’t be scheduled if the label is not present: kubectl label nodes kind-worker3 nginx-schedulable="true"

Our actual logic (this made me chuckle so much bootstrap just to get here, but imagine having to do all that by yourself)

This is where things actually happen, first we get our Spec updated: You can find this file here

Then we can put some code, I will add more comments later in the code to explain what everything does: Basically what we do is set a timer to create a pod in each node to force it fetch the image that the deployments (that we filter by labels) needs or is going to use, by doing this if the node already has the image nothing happens and it will be removed in the next run, however if the image is not there it will be fetched so if anything happens and a pod needs to be actually scheduled there it won’t need to download everything so it should be relatively faster. You can find this file here

What we should be seeing in our cluster

Cleaning up

To clean up the operator from the cluster you can do, and also remember to clean up your clusters or whatever you are using if it’s in the cloud to avoid unexpected bills

Closing notes

Be sure to check the links if you want to learn more about the project and I hope you enjoyed it, see you on twitter or github!

The source for this article is here


If you spot any error or have any suggestion, please send me a message so it gets fixed.

Also, you can check the source code and changes in the generated code and the sources here