This how-to is based on the work of Techno Tim: https://github.com/techno-tim/k3s-ansible combined with multiple other online sources, after watching Hardware Haven's video about it: https://youtu.be/S_pp_nc5QuI.
For more information on why and how I came to this particular setup, read the article on my blog: https://blog.joeplaa.com/highly-available-kubernetes-cluster-on-proxmox.
Download a Debian iso and upload to your Proxmox storage: https://www.debian.org/distrib/.
In Proxmox create a VM and give it a name: jpl-k3s-hanode1. Add the Debian ISO as CD drive, assign at least 4 cores, CPU type x86-64-v2-AES or host (or whatever best fits your servers), at least 4 GB RAM (I increased this to 6 GB, 8 would be even better), at least 20 GB (I ended up increasing this to 32 GB) VirtIO disk and obviously a network adapter.
agent: 1
balloon: 2048
boot: order=ide2;scsi0;net0
cores: 4
cpu: x86-64-v2-AES
ide2: none,media=cdrom
machine: q35
memory: 4096
meta: creation-qemu=8.0.2,ctime=1700047450
name: jpl-k3s-hanode1
net0: virtio=da:cb:9e:5f:58:78,bridge=vmbr0,firewall=1,tag=50
numa: 0
onboot: 1
ostype: l26
scsi0: proxmox2-data-lvm:vm-111-disk-0,discard=on,iothread=1,size=20G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=601349d9-7946-4cff-9ef7-24faef61e094
sockets: 1
startup: order=10
vmgenid: 746fa4ba-b164-449a-ae64-935ba2f7fa31
Boot the VM and install Debian. When asked, enter the VM name as hostname during installation. Also add a root password and create a user ansible or k3s or ....
After installation, remove the ISO from the VM config and reboot the VM. (But first go into pfSense (or your router/DHCP server) and assign a DHCP Static Mapping. Alternatively assign a static IP address inside the VM.)
On your local machine create a key-pair jpl-k3s, and copy the key to the VM.
Add the machine to your ssh config file.
Ssh into the machine using the root user:
ssh root@jpl-k3s-hanode1
Add the created user to the sudo'ers group:
usermod -aG sudo ansible
Logout
Ssh into the machine using the created user:
ssh jpl-k3s-hanode1
Optional: Install dependency ca-certificates (if you need https access to your apt mirror):
sudo apt update && sudo apt install ca-certificates
Optional: Update your apt mirror if you want a faster one or use a local mirror or proxy, for example Nexus.
sudo nano /etc/apt/sources.list
#deb cdrom:[Debian GNU/Linux 12.2.0 _Bookworm_ - Official amd64 NETINST with firmware 20231007-10:28]/ bookworm main non-free-firmware
#deb http://ftp.nl.debian.org/debian/ bookworm main non-free-firmware
#deb-src http://ftp.nl.debian.org/debian/ bookworm main non-free-firmware
deb https://sonatype.jodibooks.com/repository/debian-bookworm/ bookworm main non-free-firmware
#deb http://security.debian.org/debian-security bookworm-security main non-free-firmware
#deb-src http://security.debian.org/debian-security bookworm-security main non-free-firmware
deb https://sonatype.jodibooks.com/repository/debian-bookworm-security/ bookworm-security main non-free-firmware
# bookworm-updates, to get updates before a point release is made;
# see https://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_updates_and_backports
#deb http://ftp.nl.debian.org/debian/ bookworm-updates main non-free-firmware
#deb-src http://ftp.nl.debian.org/debian/ bookworm-updates main non-free-firmware
deb https://sonatype.jodibooks.com/repository/debian-bookworm/ bookworm-updates main non-free-firmware
# This system was installed using small removable media
# (e.g. netinst, live or single CD). The matching "deb cdrom"
# entries were disabled at the end of the installation process.
# For information about how to configure apt package sources,
# see the sources.list(5) manual.
Install Ansible dependencies:
sudo apt update
sudo apt install python3
Install Longhorn dependencies:
sudo apt install nfs-common open-iscsi
Optional: sync clock with central server on pfSense:
Install chrony:
sudo apt install chrony
Edit Chrony config:
sudo nano /etc/chrony/chrony.conf
Comment out debian pools and add local server:
# Use local time server
server 10.33.50.1 iburst
Shutdown VM:
sudo shutdown
Now either convert this VM into a template or clone directly. As I'm experimenting I skipped the template step.
Clone the VM to a new one and name that jpl-k3s-hanode2
Because we cloned it, the hostname of this machine is still jpl-k3s-hanode1, we don't want that. Start the VM, ssh into it and rename it in two files:
sudo nano /etc/hostname
sudo nano /etc/hosts
Save the changes and shutdown the machine.
Repeat this for jpl-k3s-hanode3, jpl-k3s-node1, jpl-k3s-node2, etc..
Now that we have booted all the clones, they have gotten a unique mac-address and we can also give them a DHCP Static Mapping.
This is a good moment to create a snapshot, named base_install or pre-k3s, of all the created VM's.
You can create another (debian) VM for this, but I chose to install all of this on my local workstation/dev machine.
Make sure git is installed:
sudo apt install git
Clone git repo joeplaa/joeplaa-k3s-ansible or the source I used: techno-tim/k3s-ansible and browse to that folder:
git clone git@github.com:joeplaa/joeplaa-k3s-ansible.git
cd joeplaa-k3s-ansible
Create your inventory directory:
cp -R inventory/example inventory/jpl-cluster
Edit inventory/jpl-cluster/hosts.ini (the ansible_ssh_private_key_file is the private key jpl-k3s created earlier):
[master]
jpl-k3s-hanode1
jpl-k3s-hanode2
jpl-k3s-hanode3
[master:vars]
ansible_user=ansible
ansible_ssh_private_key_file=~/.ssh/jpl-k3s
[node]
jpl-k3s-node1
jpl-k3s-node2
[node:vars]
ansible_user=ansible
ansible_ssh_private_key_file=~/.ssh/jpl-k3s
[k3s_cluster:children]
master
node
Create a file secrets.txt with the password to become sudo on the nodes. Add this file to .gitignore, so it isn't committed to git, should you want to clone and commit.
Edit inventory/jpl-cluster/group_vars/all.yml:
k3s_version (check for rancher compatibility)ansible_user: ansiblesystem_timezone: Europe/Amsterdam or the one you selected during Debian installation.flannel_iface: enp6s18 (run ip a on the node to get this value)apiserver_endpoint: 10.33.50.50 (a free IP address in the vlan/subnet range of your network)k3s_token: some-random-string-with-only-letters-and-numbers, see https://docs.k3s.io/cli/token?_highlight=k3s_tokenmetal_lb_ip_range: 10.33.50.60-10.33.50.69 (IP range in the vlan/subnet range of your network for exposed services)--skip-tags flag later in step 3.---
# if you want to install Rancher check which is the latest k3s version that is supported
k3s_version: v1.28.3+k3s2
# this is the user that has ssh access to these machines
ansible_user: ansible
systemd_dir: /etc/systemd/system
# Set your timezone
system_timezone: "Europe/Amsterdam"
# interface which will be used for flannel
flannel_iface: "enp6s18"
# apiserver_endpoint is virtual ip-address which will be configured on each master
apiserver_endpoint: "10.33.50.50"
# k3s_token is required masters can talk together securely
# this token should be alpha numeric only
k3s_token: "some-random-string-with-only-letters-and-numbers"
# The IP on which the node is reachable in the cluster.
# Here, a sensible default is provided, you can still override
# it for each of your hosts, though.
k3s_node_ip: '{{ ansible_facts[flannel_iface]["ipv4"]["address"] }}'
# Disable the taint manually by setting: k3s_master_taint = false
k3s_master_taint: "{{ true if groups['node'] | default([]) | length >= 1 else false }}"
# these arguments are recommended for servers as well as agents:
extra_args: >-
--flannel-iface={{ flannel_iface }}
--node-ip={{ k3s_node_ip }}
# change these to your liking, the only required are: --disable servicelb, --tls-san {{ apiserver_endpoint }}
extra_server_args: >-
{{ extra_args }}
{{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }}
--tls-san {{ apiserver_endpoint }}
--disable servicelb
--disable traefik
extra_agent_args: >-
{{ extra_args }}
# image tag for kube-vip
kube_vip_tag_version: "v0.5.12"
# metallb type frr or native
metal_lb_type: "native"
# metallb mode layer2 or bgp
metal_lb_mode: "layer2"
# bgp options
# metal_lb_bgp_my_asn: "64513"
# metal_lb_bgp_peer_asn: "64512"
# metal_lb_bgp_peer_address: "192.168.30.1"
# image tag for metal lb
metal_lb_speaker_tag_version: "v0.13.9"
metal_lb_controller_tag_version: "v0.13.9"
# metallb ip range for load balancer
metal_lb_ip_range: "10.33.50.60-10.33.50.69"
# Only enable if your nodes are proxmox LXC nodes, make sure to configure your proxmox nodes
# in your hosts.ini file.
# Please read https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185 before using this.
# Most notably, your containers must be privileged, and must not have nesting set to true.
# Please note this script disables most of the security of lxc containers, with the trade off being that lxc
# containers are significantly more resource efficent compared to full VMs.
# Mixing and matching VMs and lxc containers is not supported, ymmv if you want to do this.
# I would only really recommend using this if you have partiularly low powered proxmox nodes where the overhead of
# VMs would use a significant portion of your available resources.
proxmox_lxc_configure: false
# the user that you would use to ssh into the host, for example if you run ssh some-user@my-proxmox-host,
# set this value to some-user
proxmox_lxc_ssh_user: root
# the unique proxmox ids for all of the containers in the cluster, both worker and master nodes
proxmox_lxc_ct_ids:
- 200
- 201
- 202
- 203
- 204
# Only enable this if you have set up your own container registry to act as a mirror / pull-through cache
# (harbor / nexus / docker's official registry / etc).
# Can be beneficial for larger dev/test environments (for example if you're getting rate limited by docker hub),
# or air-gapped environments where your nodes don't have internet access after the initial setup
# (which is still needed for downloading the k3s binary and such).
# k3s's documentation about private registries here: https://docs.k3s.io/installation/private-registry
custom_registries: false
# The registries can be authenticated or anonymous, depending on your registry server configuration.
# If they allow anonymous access, simply remove the following bit from custom_registries_yaml
# configs:
# "registry.domain.com":
# auth:
# username: yourusername
# password: yourpassword
# The following is an example that pulls all images used in this playbook through your private registries.
# It also allows you to pull your own images from your private registry, without having to use imagePullSecrets
# in your deployments.
# If all you need is your own images and you don't care about caching the docker/quay/ghcr.io images,
# you can just remove those from the mirrors: section.
custom_registries_yaml: |
mirrors:
docker.io:
endpoint:
- "https://registry.domain.com/v2/dockerhub"
quay.io:
endpoint:
- "https://registry.domain.com/v2/quayio"
ghcr.io:
endpoint:
- "https://registry.domain.com/v2/ghcrio"
registry.domain.com:
endpoint:
- "https://registry.domain.com"
configs:
"registry.domain.com":
auth:
username: yourusername
password: yourpassword
# Only enable and configure these if you access the internet through a proxy
# proxy_env:
# HTTP_PROXY: "http://proxy.domain.local:3128"
# HTTPS_PROXY: "http://proxy.domain.local:3128"
# NO_PROXY: "*.domain.local,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
Update ansible.cfg:
[defaults]
inventory = inventory/jpl-cluster/hosts.ini ; Adapt this to the path to your inventory file
Install ansible:
sudo apt-add-repository ppa:ansible/ansible
sudo apt install ansible
ansible-galaxy collection install -r collections/requirements.yml
Test if ansible works:
ansible all -m ping -v
should return:
jpl-k3s-hanode1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
jpl-k3s-hanode2 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
jpl-k3s-hanode3 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
jpl-k3s-node1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
jpl-k3s-node2 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
Run cluster setup:
--become-password-file=secrets.txt points to the file with the sudo password needed to install stuff on the nodes.--skip-tags metallb skips installing metallb. If you only want to use a single IP range, don't use this flag.ansible-playbook site.yml --become-password-file=secrets.txt --skip-tags metallb
Reboot VM's:
ansible-playbook reboot.yml
Copy kube config to your local machine:
mkdir ~/.kube
scp jpl-k3s-hanode1:~/.kube/config ~/.kube/config
Test if nodes are up:
kubectl get nodes
I want to use multiple VLAN's for my services. I don't know how yet, by at least I figured out how to supply MetalLB with multiple ranges.
Create config file metallb/resources.yaml:
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
creationTimestamp: null
name: default
namespace: metallb-system
spec:
addresses:
- 10.33.50.60-10.33.50.69
autoAssign: true
avoidBuggyIPs: false
serviceAllocation:
priority: 50
status: {}
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
creationTimestamp: null
name: iot
namespace: metallb-system
spec:
addresses:
- 10.33.40.60-10.33.40.69
autoAssign: true
avoidBuggyIPs: false
serviceAllocation:
priority: 50
namespaceSelectors:
- matchLabels:
group: iot
status: {}
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
creationTimestamp: null
name: l2advertisement1
namespace: metallb-system
spec:
ipAddressPools:
- default
- iot
status: {}
---
Install:
helm repo add metallb https://metallb.github.io/metallb
helm repo update
helm install metallb metallb/metallb -n metallb-system --create-namespace
kubectl apply -f metallb/resources.yaml
Because I use an external reverse proxy, I want the Longhorn webUI to have a fixed IP address. This is done by passing the --set service.ui.loadBalancerIP="10.33.50.60" option.
Install (if you don't need a fixed IP for the WebUI, don't use --set service.ui.loadBalancerIP="10.33.50.60"):
kubectl create namespace longhorn-system
helm repo add longhorn https://charts.longhorn.io
helm repo update
helm install longhorn longhorn/longhorn -n longhorn-system --set service.ui.type="LoadBalancer" --set service.ui.loadBalancerIP="10.33.50.60"
Check/find IP address for longhorn-frontend:
kubectl get service longhorn-frontend -n longhorn-system
should return something like:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
longhorn-frontend LoadBalancer 10.43.254.112 10.33.50.60 80:32651/TCP 25h
Because I use an external reverse proxy, I want the Grafana webUI to have a fixed IP address. I couldn't figure out how to to this by passing the --set service....loadBalancerIP="10.33.50.61" option. So we do it manually in step 2 below.
Install:
kubectl create namespace monitoring
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring
kubectl edit service prometheus-grafana -n monitoring
Edit prometheus-grafana service:
kubectl edit service prometheus-grafana -n monitoring
Change type to LoadBalancer and optionally add loadBalancerIP: 10.33.50.61 if you need a static IP:
...
spec:
ports:
- name: http-web
protocol: TCP
port: 80
targetPort: 3000
nodePort: 31167
selector:
app.kubernetes.io/instance: prometheus
app.kubernetes.io/name: grafana
clusterIP: 10.43.11.220
clusterIPs:
- 10.43.11.220
type: LoadBalancer
sessionAffinity: None
loadBalancerIP: 10.33.50.61
externalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
allocateLoadBalancerNodePorts: true
internalTrafficPolicy: Cluster
Check/find IP address for prometheus-grafana:
kubectl get service prometheus-grafana -n monitoring
should return something like:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
prometheus-grafana LoadBalancer 10.43.11.220 10.33.50.61 80:30796/TCP 23h
Because I use an external reverse proxy, I want the Netdata dashboard to have a fixed IP address. I couldn't figure out how to to this by passing the --set service....loadBalancerIP="10.33.50.62" option. So we do it manually in step 2 below.
Install:
kubectl create namespace netdata
helm repo add netdata https://netdata.github.io/helmchart/
helm repo update
helm install netdata netdata/netdata -n netdata \
--set image.tag=stable \
--set parent.claiming.enabled="true" \
--set parent.claiming.token=<claim token> \
--set parent.claiming.rooms=<room id> \
--set child.claiming.enabled="true" \
--set child.claiming.token=<claim token> \
--set child.claiming.rooms=<room id>
Edit netdata service:
kubectl edit service netdata -n netdata
Change type to LoadBalancer and optionally add loadBalancerIP: 10.33.50.62 if you need a static IP:
...
spec:
ports:
- name: http
protocol: TCP
port: 19999
targetPort: http
nodePort: 30656
selector:
app: netdata
release: netdata
role: parent
clusterIP: 10.43.163.10
clusterIPs:
- 10.43.163.10
type: LoadBalancer
sessionAffinity: None
loadBalancerIP: 10.33.50.62
externalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
allocateLoadBalancerNodePorts: true
internalTrafficPolicy: Cluster
Check/find IP address for netdata:
kubectl get service netdata -n netdata
should return something like:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
netdata LoadBalancer 10.43.163.10 10.33.50.62 19999:30656/TCP 23h