Setup a cluster of Fedora CoreOS with Kubespray and Terraform

In this article we will create a cluster of a Container Linux distro, like Fedora CoreOS, with Terraform (using libvirt for testing purposes), then create the k8s cluster in it using Kubespray and Ansible.

Install the necessary toolset

First install Terraform using their guide. I use Fedora 37 so I had to tweak a little the repo file.

For the Ignition file, we'll use the Podman command, and it's aliased (add it into your .bashrc, .zshrc, whatever fish has…):

alias butane='podman run --rm --interactive         \
              --security-opt label=disable          \
              --volume "${PWD}":/pwd --workdir /pwd \
              quay.io/coreos/butane:release'

Needed files

Here we'll get the needed files:

Butane file

This is the Butane YAML file to generate the Ignition file:

variant: fcos
version: 1.4.0
storage:
  files:
    # CRI-O DNF module
    - path: /etc/dnf/modules.d/cri-o.module
      mode: 0644
      overwrite: true
      contents:
	inline: |
	  [cri-o]
	  name=cri-o
	  stream=1.17
	  profiles=
	  state=enabled
      # configuring automatic loading of br_netfilter on startup
    - path: /etc/modules-load.d/br_netfilter.conf
      mode: 0644
      overwrite: true
      contents:
	inline: br_netfilter
    # setting kernel parameters required by kubelet
    - path: /etc/sysctl.d/kubernetes.conf
      mode: 0644
      overwrite: true
      contents:
	inline: |
	  net.bridge.bridge-nf-call-iptables=1
	  net.ipv4.ip_forward=1
    # GOOGLE CLOUDFLARE DNS FFS
    - path: /etc/systemd/resolved.conf.d/dns_servers.conf
      mode: 0644
      overwrite: true
      contents:
	inline: |
	  [Resolve]
	  DNS=8.8.8.8 1.1.1.1
passwd: # setting login credentials
  users:
    - name: core
      ssh_authorized_keys:
	- THIS_IS_REDACTED

Terraform file

Let's create the terraform file! This file will create N nodes with C cores and M memory, that would be prompted while we use the terraform apply command.

terraform {
  # Import required providers, we're using libvirt on local, this lib manages Ignition files
  # The .ign will be created before the Terraform launch.
  required_providers {
    libvirt = {
      source  = "dmacvicar/libvirt"
      version = "0.7.0"
    }
  }
}

# Set Provider
provider "libvirt" {
  uri = "qemu:///system"
}

# Variables to set
variable "vms_amount" {
  type        = number
  description = "Number of nodes to create"
}

variable "ram" {
  type        = number
  description = "Amount of memory for each node"
}

variable "vcpu" {
  type        = number
  description = "How many cores would the machines use"
}

# Libvirt resources
# Ignition
resource "libvirt_ignition" "ignition" {
  name    = "coreos.ign"
  content = "coreos.ign"
}

# Master CoreOS Volume
resource "libvirt_volume" "fcos" {
  name   = "fcos"
  source = "./fedora-coreos-36.20221014.3.1-qemu.x86_64.qcow2"
}

# Minion volumes
resource "libvirt_volume" "minionvols" {
  name           = format("minion-%02d.qcow2", count.index)
  count          = var.vms_amount
  base_volume_id = libvirt_volume.fcos.id
}

# Networking
resource "libvirt_network" "minionet" {
  name      = "minionet"
  mode      = "nat"
  addresses = ["172.25.52.0/24"]

}

# Domains
resource "libvirt_domain" "minions" {
  name            = format("minion-%02d", count.index)
  memory          = var.ram
  vcpu            = var.vcpu
  count           = var.vms_amount
  coreos_ignition = libvirt_ignition.ignition.id

  disk {
    volume_id = libvirt_volume.minionvols[count.index].id
  }

  network_interface {
    network_id     = libvirt_network.minionet.id
    hostname       = format("minion-%02d", count.index)
    addresses      = [format("172.25.52.%d", count.index + 10)]
    wait_for_lease = true
  }
}

output "nodes" {
  value = zipmap(libvirt_domain.minions.*.network_interface.0.hostname,
  flatten(libvirt_domain.minions.*.network_interface.0.addresses))
  depends_on = [libvirt_domain.minions]
}

Create the VMs with Terraform

We execute this command:

butane < coreos.yaml > coreos.ign
TF_VAR_ram=4096 TF_VAR_vcpu=2 TF_VAR_vms_amount=6 terraform apply

This command will create six nodes with two vCores and 4 GiB of memory for each node.

We'll get the node's IPs too. This will be useful next.

NOTE: If yopu want to destroy the cluster, replace apply with destroy in the command above.

Setting up Kubespray

First of all, install Python3 developer libraries, on Fedora this could be achieved installing the python-devel package. Also, make sure you DON'T HAVE INSTALLED ANSIBLE ON YOUR MACHINE, as it will conflict with the virtual environment.

Then, we'll follow this guide.

Clone and initialize Kubespray

First, we clone the repo:

git clone https://github.com/kubernetes-sigs/kubespray

Then we set the environment variables:

VENVDIR=kubespray-venv
KUBESPRAYDIR=kubespray
ANSIBLE_VERSION=2.12

After that, we create a venv and we'll use it

virtualenv  --python=$(which python3) $VENVDIR
source $VENVDIR/bin/activate
cd $KUBESPRAYDIR

Finally we install Ansible on this venv

pip install -U -r requirements-$ANSIBLE_VERSION.txt
test -f requirements-$ANSIBLE_VERSION.yml && \
  ansible-galaxy role install -r requirements-$ANSIBLE_VERSION.yml && \
  ansible-galaxy collection -r requirements-$ANSIBLE_VERSION.yml

Initialize inventory

We need the IPs of the nodes we created with Terraform. If we set 6 nodes, we start from 10 to 15.

cp -rfp inventory/sample inventory/coreos-cluster
IPS=($(for i in $(seq 10 15); do echo -n "172.25.52.$i ";done))
declare -a IPS=($IPS)
CONFIG_FILE=inventory/coreos-cluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

Also, we must change some properties prior to the execution of this Ansible playbooks.

We need to use a RW directory like /opt:

sed -i 's/usr\/local/opt/' inventory/coreos-cluster/group_vars/all/all.yml

Also, the DNS must be changed, othwerwise we won't be able to get anything downloaded on the cluster. The file inventory/coreos-cluster/group_vars/k8s_cluster.yaml needs to get appended this after resolvconf_mode: host_resolvconf:

nameservers:
- 8.8.8.8

Or the nameserver's IP you like, but we must specify it/them.

Kubernetes installation with Kubespray

Just this command.

ansible-playbook -i inventory/coreos-cluster/hosts.yaml -u core --become --become-user=root cluster.yml

Then wait. Et voilà.

Getting the `kubeconfig`

It's on the master node.

sudo -i
cat /etc/kubernetes/admin.conf

Or whatever you use to get it like SCP, it's up to you.