Using Minikube On Fedora With The Registry Addon

Posted on wo 05 februari 2020 in dev

Table of Contents

  1. Setting up minikube on Fedora 31
    1. Setting up a minikube VM
      1. Downloading the required binaries
      2. Configuring minikube
    2. Starting the minikube VM
      1. Enabling the registry addon
  2. Pushing images into the registry and running them!
    1. Starting the container

Update (March 6, 2020)

I have updated the below description for minikube 1.7.3, which is a little different than the 1.6.2 version that I originally wrote this tutorial for.

The main difference is in what minikube config is done (I'm adding a vm-driver config option, and an insecure-registry option), though I'm not 100% if this is working, because I'm also still explicitly using the --insecure-registry flag and enabling the registry addon. Without these, pushing an image into the registry (or pulling from it) for a deployment, does not work.

Enabling the registry addon needs to happen after starting the minikube VM now though, and not before.

Setting up minikube on Fedora 31

The article on setting up OpenShift 4.3 on libvirt VMs is not quite done yet. In the meantime though, I've been playing around with minikube as well. Just to get a feel of plain Kubernetes without the productive / useful abstractions OpenShift uses (OpenShift being certified Kubernetes, btw).

Setting up a minikube VM

Downloading the required binaries

Setting up minikube on Fedora is really, really easy. It's not packaged as an RPM, but you only have to go here and download the latest release. I prefer to stick with stable, which is v1.7.3 at the moment of writing this.

Together with the main minikube binary, you need to download the docker-machine-driver-kvm2 binary. Drop both that file and the minikube binary itself into your PATH and you are ready to go.

(There is an RPM available as well, but I use the binary downloads.)

Configuring minikube

Minikube should work fine without any additional configuration at this point, but I usually make some changes.

The first thing I like to do, is increase the default memory. According to the doc, by default, minikube only allocates 2GiB of RAM, which is next to nothing. I increase it to at least 10GiB instead.

I also like to switch the amount of vCPUs to at least 4, but sometimes more, depending on what I'm playing around with (I have a ton of CPUs in my machine anyway). The default is 2 vCPUs; if that's enough for you, leave it as is.

Let's also increase the size of the disk to 30GiB so we can store some images. 20GiB is the default here, and it will be created as a thin provisioned, raw disk image in ~/.minikube/machines by default.

Finally, in order to make sure we use the container runtime that aligns best to Kubernetes itself, we set the container-runtime configuration property to crio.

$ minikube config set memory 10240
$ minikube config set cpus 4
$ minikube config set disk-size 30000
$ minikube config set container-runtime crio
$ minikube config set vm-driver kvm2
$ minikube config set insecure-registry 192.168.39.0/24

Starting the minikube VM

This should be a fairly short paragraph, but there's one thing to talk about here. Minikube allows you to run different instances of minikube in different profiles. Profiles are basically named instances of minikube.

For example, you can have a minikube instance to play around with to learn Kubernetes itself, and you can have one to - I don't know - test your app in. The first could be called k8s-test, the second one myapp.

In order to start a named minikube instance, we run it with the -p option:

$ minikube start -p mytest

Start a minikube instance for real, and let's push some images into the registry living on it! Make sure to use the --insecure-registry option like below. You'll need that to have minikube pull from it!

$ minikube start -p mytest --insecure-registry "192.168.39.0/24"

If you minikube stop -p mytest, in order to start your minikube VM again, you will need to use the --insecure-registry flag again!

Enabling the registry addon

I want to be able to push images into minikube, so I need to run a registry in it. We can get a list of available addons to minikube by running:

$ minikube addons list

This will show you all the available options. To enable the registry addon, we only need to run while our mytest minikube VM is running (which is different from previous versions of minikube!):

$ minikube addons enable registry -p mytest

That's all, we good to go!

Pushing images into the registry and running them!

I want to be able to build images of my app locally, then push them into the minikube registry and run them from there. The registry on minikube does not have TLS enabled. To get images in there, for me the easiest way to do that was to wield skopeo.

First, let's see what images we have, locally:

$ podman images
REPOSITORY                                                         TAG      IMAGE ID       CREATED        SIZE
registry.redhat.io/rhel8/nodejs-10                                 latest   72b60f3a377b   6 weeks ago    788 MB
registry.redhat.io/openjdk/openjdk-11-rhel7                        latest   9a6cd285b6c8   6 weeks ago    510 MB
registry.access.redhat.com/redhat-openjdk-18/openjdk18-openshift   latest   963af2cdd61f   6 weeks ago    491 MB
registry.redhat.io/rhel8/postgresql-10                             latest   58950bf238bf   8 weeks ago    407 MB
registry.access.redhat.com/ubi8/ubi                                latest   096cae65a207   8 weeks ago    239 MB
registry.redhat.io/ubi8/ubi                                        latest   096cae65a207   8 weeks ago    239 MB
registry.access.redhat.com/ubi8/ubi                                8.1      096cae65a207   8 weeks ago    239 MB
registry.access.redhat.com/ubi8/ubi-minimal                        latest   469119976c56   3 months ago   107 MB
registry.fedoraproject.org/fedora-minimal                          latest   7bb5a60e8a78   3 months ago   167 MB
registry.fedoraproject.org/f29/httpd                               latest   25c76f9dcdb5   9 months ago   482 MB

As an example, let's push the fedora-minimal image into our registry:

$ skopeo copy --dest-tls-verify=false docker://registry.fedoraproject.org/f29/httpd docker://$(minikube ip -p mytest):5000/f29/httpd
Getting image source signatures
Copying blob 7692efc5f81c skipped: already exists
Copying blob d77ff9f653ce skipped: already exists
Copying blob aaf5ad2e1aa3 skipped: already exists
Copying config 25c76f9dcd done
Writing manifest to image destination
Storing signatures

And that's it. We can now start that container.

Starting the container

We point the deployment to the local registry and everything from that point on is automatic:

$ kubectl run apache --image=$(minikube -p mytest ip):5000/f29/httpd

Let's check what's going on:

$ kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
apache-f698479d9-ltqvf     1/1     Running   0          11m
$ kubectl logs apache-f698479d9-ltqv
=> sourcing 10-set-mpm.sh ...
=> sourcing 20-copy-config.sh ...
=> sourcing 40-ssl-certs.sh ...
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.1.0.18. Set the 'ServerName' directive globally to suppress this message
[Wed Feb 05 09:30:46.294280 2020] [ssl:warn] [pid 1:tid 140049696796032] AH01882: Init: this version of mod_ssl was compiled against a newer library (OpenSSL 1.1.1b FIPS  26 Feb 2019, version currently loaded is OpenSSL 1.1.1 FIPS  11 Sep 2018) - may result in undefined or erroneous behavior
[Wed Feb 05 09:30:46.294874 2020] [ssl:warn] [pid 1:tid 140049696796032] AH01909: 10.1.0.18:8443:0 server certificate does NOT include an ID which matches the server name
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.1.0.18. Set the 'ServerName' directive globally to suppress this message
[Wed Feb 05 09:30:46.352319 2020] [ssl:warn] [pid 1:tid 140049696796032] AH01882: Init: this version of mod_ssl was compiled against a newer library (OpenSSL 1.1.1b FIPS  26 Feb 2019, version currently loaded is OpenSSL 1.1.1 FIPS  11 Sep 2018) - may result in undefined or erroneous behavior
[Wed Feb 05 09:30:46.352919 2020] [ssl:warn] [pid 1:tid 140049696796032] AH01909: 10.1.0.18:8443:0 server certificate does NOT include an ID which matches the server name
[Wed Feb 05 09:30:46.353031 2020] [lbmethod_heartbeat:notice] [pid 1:tid 140049696796032] AH02282: No slotmem from mod_heartmonitor
[Wed Feb 05 09:30:46.354288 2020] [mpm_event:notice] [pid 1:tid 140049696796032] AH00489: Apache/2.4.39 (Fedora) OpenSSL/1.1.1 configured -- resuming normal operations
[Wed Feb 05 09:30:46.354302 2020] [core:notice] [pid 1:tid 140049696796032] AH00094: Command line: 'httpd -D FOREGROUND'

Excellent! Happy containering from this point on!