Setting up a Fedora CoreOS Kubernetes Cluster

How to setup a container-optimized OS Cluster to use it with Kubernetes.

Host: macOS 12
Virtualization system: VirtualBox
Specs: 64 GB of RAM memory, around 500 GiB of storage, 12 cores.

Get Fedora CoreOS

Get the image file here. I use the OVA.

Create a host-only network

On VirtualBox we should create vboxnet0 with range 192.168.99.0/24.

The gateway (.1) should be our host, so we'll create a mini HTTP server with Python to serve a file.

Butane file

As the file would be bigger than VBox maximum, we ought to use a remote file. This is the Butane file used on the VirtualBox VM:

variant: fcos
version: 1.4.0
ignition:
  config:
    replace:
      source: http://192.168.99.1:8080/fcos.ign

And the remote one:

variant: fcos
version: 1.4.0
storage:
  files:
    # CRI-O DNF module
    - path: /etc/dnf/modules.d/cri-o.module
      mode: 0644
      overwrite: true
      contents:
	inline: |
	  [cri-o]
	  name=cri-o
	  stream=1.17
	  profiles=
	  state=enabled
    # YUM repository for kubeadm, kubelet and kubectl
    - path: /etc/yum.repos.d/kubernetes.repo
      mode: 0644
      overwrite: true
      contents:
	inline: |
	  [kubernetes]
	  name=Kubernetes
	  baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
	  enabled=1
	  gpgcheck=0
	  repo_gpgcheck=0
	  gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
	    https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
    # configuring automatic loading of br_netfilter on startup
    - path: /etc/modules-load.d/br_netfilter.conf
      mode: 0644
      overwrite: true
      contents:
	inline: br_netfilter
    # setting kernel parameters required by kubelet
    - path: /etc/sysctl.d/kubernetes.conf
      mode: 0644
      overwrite: true
      contents:
	inline: |
	  net.bridge.bridge-nf-call-iptables=1
	  net.ipv4.ip_forward=1
passwd: # setting login credentials
  users:
    - name: core
      ssh_authorized_keys:
	- {YOUR_SSH_KEY}

Ignition

We create the two files: the one injected in VirtualBox and the "remote" one served on the host:

docker run --interactive --rm \
quay.io/coreos/butane:release \
--pretty --strict < fcos.bu > fcos.ign

docker run --interactive --rm \
quay.io/coreos/butane:release \
--pretty --strict < fcos_vbox.bu > fcos_vbox.ign

You could use podman instead of docker.

Server

On the folder you store the files, execute this:

python3 -m http.server 8080

Launch the machines

Here's a script I use to create the VMs:

#!/usr/bin/env sh

DIR="$HOME/Kubernetes"
IGN_PATH="$DIR/fcos_vbox.ign"
VM_NAME=minion$1
VBoxManage import --vsys 0 --vmname "$VM_NAME" fedora-coreos-36.20220918.3.0-virtualbox.x86_64.ova
VBoxManage guestproperty set "$VM_NAME" /Ignition/Config "$(cat $IGN_PATH)"
VBoxManage modifyvm "$VM_NAME" --natpf1 "guestssh,tcp,,222$1,,22"
VBoxManage modifyvm "$VM_NAME" --nic2 hostonlynet --host-only-net2 vboxnet0
VBoxManage startvm "$VM_NAME" --type headless

Launch it with ./script.sh N, with N being a number between 0 and 9.

Configure the machines

Change their names:

sudo -i
echo "minionN" > /etc/hostname

Then on the main node, install the tools:

sudo rpm-ostree install kubelet kubeadm kubectl cri-o

We don't need kubectl on the other nodes:

sudo rpm-ostree install kubelet kubeadm kubectl cri-o

Reboot the nodes. Then enable and start the crio and kubelet systemd's services on each node:

sudo systemctl enable --now crio kubelet

Edit Hosts file

Add the IPs and its hostnames. Trivial thing to do.

Init the cluster

On the main node (Control Plane) create this file and save it as clusterconfig.yaml:

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.25.0
controllerManager:
  extraArgs: # specify a R/W directory for FlexVolumes (cluster won't work without this even though we use PVs)
    flex-volume-plugin-dir: "/etc/kubernetes/kubelet-plugins/volume/exec"
networking: # pod subnet definition
  podSubnet: 10.244.0.0/16
apiServer:
  certSANs:
  - "10.0.2.15"
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "192.168.99.8"
  bindPort: 6443

Notice you should change the advertiseAddress to yours.

Launch the kubeadm:

kubeadm init --config clusterconfig.yaml

Annotate the join command, the output should be similar to this:

kubeadm join 192.168.99.8:6443 --token wmjfhl.mougk1kpm7h0bx2c \
        --discovery-token-ca-cert-hash sha256:1bdd4762665258142cc2c06f1084a6d81cf02199bc5f498bead13ed48cc07647

Change the IP to the 192.168.99.0/24 scoped one.

Network initialization

kubectl apply -f \
https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml

Deploy something

Now it's up to you! Don't forget to get the kubeconfig file.