Setup a cluster of Fedora CoreOS with Kubespray and Terraform

In this article we will create a cluster of a Container Linux distro, like Fedora CoreOS, with Terraform (using libvirt for testing purposes), then create the k8s cluster in it using Kubespray and Ansible.
Install the necessary toolset
First install Terraform using their guide. I use Fedora 37 so I had to tweak a little the repo file.
For the Ignition file, we'll use the Podman command, and it's aliased (add it into your .bashrc
, .zshrc
, whatever fish
has…):
alias butane='podman run --rm --interactive \ --security-opt label=disable \ --volume "${PWD}":/pwd --workdir /pwd \ quay.io/coreos/butane:release'
Needed files
Here we'll get the needed files:
Butane file
This is the Butane YAML file to generate the Ignition file:
variant: fcos version: 1.4.0 storage: files: # CRI-O DNF module - path: /etc/dnf/modules.d/cri-o.module mode: 0644 overwrite: true contents: inline: | [cri-o] name=cri-o stream=1.17 profiles= state=enabled # configuring automatic loading of br_netfilter on startup - path: /etc/modules-load.d/br_netfilter.conf mode: 0644 overwrite: true contents: inline: br_netfilter # setting kernel parameters required by kubelet - path: /etc/sysctl.d/kubernetes.conf mode: 0644 overwrite: true contents: inline: | net.bridge.bridge-nf-call-iptables=1 net.ipv4.ip_forward=1 # GOOGLE CLOUDFLARE DNS FFS - path: /etc/systemd/resolved.conf.d/dns_servers.conf mode: 0644 overwrite: true contents: inline: | [Resolve] DNS=8.8.8.8 1.1.1.1 passwd: # setting login credentials users: - name: core ssh_authorized_keys: - THIS_IS_REDACTED
Terraform file
Let's create the terraform file! This file will create N nodes with C cores and M memory, that would be prompted while we use the terraform apply
command.
terraform { # Import required providers, we're using libvirt on local, this lib manages Ignition files # The .ign will be created before the Terraform launch. required_providers { libvirt = { source = "dmacvicar/libvirt" version = "0.7.0" } } } # Set Provider provider "libvirt" { uri = "qemu:///system" } # Variables to set variable "vms_amount" { type = number description = "Number of nodes to create" } variable "ram" { type = number description = "Amount of memory for each node" } variable "vcpu" { type = number description = "How many cores would the machines use" } # Libvirt resources # Ignition resource "libvirt_ignition" "ignition" { name = "coreos.ign" content = "coreos.ign" } # Master CoreOS Volume resource "libvirt_volume" "fcos" { name = "fcos" source = "./fedora-coreos-36.20221014.3.1-qemu.x86_64.qcow2" } # Minion volumes resource "libvirt_volume" "minionvols" { name = format("minion-%02d.qcow2", count.index) count = var.vms_amount base_volume_id = libvirt_volume.fcos.id } # Networking resource "libvirt_network" "minionet" { name = "minionet" mode = "nat" addresses = ["172.25.52.0/24"] } # Domains resource "libvirt_domain" "minions" { name = format("minion-%02d", count.index) memory = var.ram vcpu = var.vcpu count = var.vms_amount coreos_ignition = libvirt_ignition.ignition.id disk { volume_id = libvirt_volume.minionvols[count.index].id } network_interface { network_id = libvirt_network.minionet.id hostname = format("minion-%02d", count.index) addresses = [format("172.25.52.%d", count.index + 10)] wait_for_lease = true } } output "nodes" { value = zipmap(libvirt_domain.minions.*.network_interface.0.hostname, flatten(libvirt_domain.minions.*.network_interface.0.addresses)) depends_on = [libvirt_domain.minions] }
Create the VMs with Terraform
We execute this command:
butane < coreos.yaml > coreos.ign TF_VAR_ram=4096 TF_VAR_vcpu=2 TF_VAR_vms_amount=6 terraform apply
This command will create six nodes with two vCores and 4 GiB of memory for each node.
We'll get the node's IPs too. This will be useful next.
NOTE: If yopu want to destroy the cluster, replace apply
with destroy
in the command above.
Setting up Kubespray
First of all, install Python3 developer libraries, on Fedora this could be achieved installing the python-devel
package. Also, make sure you DON'T HAVE INSTALLED ANSIBLE ON YOUR MACHINE, as it will conflict with the virtual environment.
Then, we'll follow this guide.
Clone and initialize Kubespray
First, we clone the repo:
git clone https://github.com/kubernetes-sigs/kubespray
Then we set the environment variables:
VENVDIR=kubespray-venv KUBESPRAYDIR=kubespray ANSIBLE_VERSION=2.12
After that, we create a venv and we'll use it
virtualenv --python=$(which python3) $VENVDIR source $VENVDIR/bin/activate cd $KUBESPRAYDIR
Finally we install Ansible on this venv
pip install -U -r requirements-$ANSIBLE_VERSION.txt test -f requirements-$ANSIBLE_VERSION.yml && \ ansible-galaxy role install -r requirements-$ANSIBLE_VERSION.yml && \ ansible-galaxy collection -r requirements-$ANSIBLE_VERSION.yml
Initialize inventory
We need the IPs of the nodes we created with Terraform. If we set 6 nodes, we start from 10 to 15.
cp -rfp inventory/sample inventory/coreos-cluster IPS=($(for i in $(seq 10 15); do echo -n "172.25.52.$i ";done)) declare -a IPS=($IPS) CONFIG_FILE=inventory/coreos-cluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
Also, we must change some properties prior to the execution of this Ansible playbooks.
We need to use a RW directory like /opt
:
sed -i 's/usr\/local/opt/' inventory/coreos-cluster/group_vars/all/all.yml
Also, the DNS must be changed, othwerwise we won't be able to get anything downloaded on the cluster. The file inventory/coreos-cluster/group_vars/k8s_cluster.yaml
needs to get appended this after resolvconf_mode: host_resolvconf
:
nameservers: - 8.8.8.8
Or the nameserver's IP you like, but we must specify it/them.
Kubernetes installation with Kubespray
Just this command.
ansible-playbook -i inventory/coreos-cluster/hosts.yaml -u core --become --become-user=root cluster.yml
Then wait. Et voilĂ .
Getting the kubeconfig
It's on the master node.
sudo -i cat /etc/kubernetes/admin.conf
Or whatever you use to get it like SCP, it's up to you.