chrisallmark.dev | Adding a private Docker registry to your RPI Kubernetes cluster

Adding a private Docker registry to your RPI Kubernetes cluster

docker

In my previous post, I described how you can build your own Raspberry Pi Kubernetes Cluster using Rancher’s K3s for local development. By default, the cluster pulls public images from Docker Hub which, for personal projects, may be acceptable but if you want to work on anything more commercially sensitive then you’ll want to use a private docker registry to host your images.

If you recall in my last post, I mentioned that I purchased a larger (256 GB) SD card for the master node as I intended it to also house a private docker registry. In this post, I’ll describe the steps you need to take to set that up.

Create the Registry

To create a registry, we need a ReplicationController that mounts a registry container on the master node.

We also need to make sure that the registry is only ever created on the master node and to do that we use a nodeSelector that targets the preferred node with a given label; fortunately, K3s already labels the master node with node-role.kubernetes.io/master=true so targeting it becomes a trivial task.

Images will be stored in a volume at /var/lib/registry which will persist the registry state after a system restart.

We’ll also create a Service to expose the registry on port 5000.

Save the following script as kube-registry.yaml:

	apiVersion: v1
	kind: ReplicationController
	metadata:
	name: kube-registry
	namespace: kube-system
	spec:
	replicas: 1
	selector:
	app: kube-registry
	template:
	metadata:
	name: kube-registry
	labels:
	app: kube-registry
	spec:
	containers:
	- name: registry
	image: registry:2
	resources:
	limits:
	cpu: 100m
	memory: 200Mi
	env:
	- name: REGISTRY_HTTP_ADDR
	value: :5000
	- name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
	value: /var/lib/registry
	- name: REGISTRY_STORAGE_DELETE_ENABLED
	value: "true"
	volumeMounts:
	- name: registry
	mountPath: /var/lib/registry
	ports:
	- containerPort: 5000
	nodeSelector:
	node-role.kubernetes.io/master: "true"
	volumes:
	- name: registry
	hostPath:
	path: /var/lib/registry
	---
	apiVersion: v1
	kind: Service
	metadata:
	name: kube-registry
	namespace: kube-system
	spec:
	selector:
	app: kube-registry
	ports:
	- port: 5000
	targetPort: 5000
	type: LoadBalancer

view raw kube-registry.yaml hosted with ❤ by GitHub

Now apply this to your cluster with:

kubectl apply -f kube-registry.yaml

Configure K3s

As you access the cluster using a hostname from your local machine you’ll need to configure K3s so that it can resolve the repository to an IP address:

Do this by adding the following /etc/rancher/k3s/registries.yaml file on each node of your cluster:

mirrors:
  "<hostname>:5000":
    endpoint:
      - "http://<ip-address>:5000"

I used k3sas an alternative hostname for my master node in /etc/hosts:

192.168.0.100 k3s-0 k3s
192.168.0.101 k3s-1
192.168.0.102 k3s-2
192.168.0.103 k3s-3

So my registries.yaml looks like this:

mirrors:
  "k3s:5000":
    endpoint:
      - "http://192.168.0.100:5000"

Restart your cluster with sudo systemctl restart ks3 on the master node.

Configure Docker

Finally, configure your local Docker client by adding:

"insecure-registries": ["<hostname>:5000"]

...to your Docker Engine preferences.

Test It Out

We’ll reuse the demo application that you can download from https://github.com/chrisallmark/k3s-cluster-demo from my previous post. It’s a simple React client with a server API that just returns the server’s hostname for display.

Now that we’ve got a private registry we’ll need to configure the infra scripts to point to it with:

./configure.sh <hostname>:5000

Now you should now be able to push images into your private registry and deploy them to your cluster private with:

skaffold dev

And that's it — now you can list the images in your private registry with:

curl --request GET --url http://<hostname>:5000/v2/_catalog

...and you should get the following response:

{
  "repositories": [
    "k3s-cluster-demo_client",
    "k3s-cluster-demo_server"
  ]
}