I have three small proxmox nodes (A) that I want to use to provide highly-available storage. I chose to use Linstor/DRBD in favor of CEPH. My cluster will be small and thus manageble and Linstor promises to be faster than CEPH.
The fourth node (B) will be "diskless" as in not participating in the ha-storage part. It will run the controller and will use the storage for (some of) its virtual machines and containers.
Follow this tutorial to get a drbd-storage pool in Proxmox and access it from all nodes in the Proxmox cluster, also "diskless" nodes.
I added A or B in front of each step if they only need to be performed on either the storage nodes (A) or the diskless controller node (B). If no letter is shown, the step is to be performed on both.
A: jpl-proxmox7, jpl-proxmox8, jpl-proxmox9
B: jpl-proxmox6
Sources
- Blogpost (8-8-2023): How to Setup LINSTOR on Proxmox VE
- Official docs: 8. LINSTOR Volumes in Proxmox VE
- Official docs: 2.5. Initializing Your Cluster
- Blogpost (16-10-2022): Creating a Self-healing Cluster Using LINSTOR’s Auto-evict Feature
Add repositories:
wget -O /tmp/package-signing-pubkey.asc https://packages.linbit.com/package-signing-pubkey.asc
gpg --yes -o /etc/apt/trusted.gpg.d/linbit-keyring.gpg --dearmor /tmp/package-signing-pubkey.asc
PVERS=8 && echo "deb [signed-by=/etc/apt/trusted.gpg.d/linbit-keyring.gpg] http://packages.linbit.com/public/ proxmox-$PVERS drbd-9" > /etc/apt/sources.list.d/linbit.list
apt update
Install dependencies on storage nodes:
apt install proxmox-default-headers drbd-dkms drbd-utils linstor-proxmox
Install Linstor services (best to install the controller on all nodes, even if it will only run on one):
apt install linstor-controller linstor-satellite linstor-client
systemctl enable linstor-satellite --now
B: Enable controller on a single node
systemctl enable linstor-controller --now
All
linstorcommands need to be entered on the controller node.
Most
linstorcommands can be shortened. E.g.node create-->n corstorage-pool list-->sp l
Add nodes, it's important to use the hostname as stored in /etc/hostname. Enter the IP address from the subnet on which you want communication between controller and satellite, i.e. also on which the diskless node reads and writes data. I will use a dedicated subnet for the storage nodes to sync the storage.
linstor node create jpl-proxmox6 10.33.80.116
linstor node create jpl-proxmox7 10.33.80.117
linstor node create jpl-proxmox8 10.33.80.118
linstor node create jpl-proxmox9 10.33.80.119
A: Either create a LVM volume group or ZFS pool on the storage devices reserved for drbd. In this example a thinpool on top of a single SSD per node. So on each node first find the disk:
lsblk
A: Then create the pool:
vgcreate linstor_vg /dev/sda
lvcreate -l 80%FREE -T linstor_vg/thinpool
A: And create a storage pool in Linstor:
linstor storage-pool create lvmthin jpl-proxmox7 pve-storage linstor_vg/thinpool
linstor storage-pool create lvmthin jpl-proxmox8 pve-storage linstor_vg/thinpool
linstor storage-pool create lvmthin jpl-proxmox9 pve-storage linstor_vg/thinpool
Create a resource group and volume group:
linstor resource-group create pve-rg --storage-pool=pve-storage --place-count=3
linstor volume-group create pve-rg
Open /etc/pve/storage.cfg and add:
drbd: linstor_storage
content images, rootdir
controller 10.33.80.116
resourcegroup pve-rg
Restart Proxmox services:
systemctl restart pve-cluster pvedaemon pvestatd pveproxy pve-ha-lrm
Edit satellite service: systemctl edit linstor-satellite.service and add:
[Service]
Type=notify
TimeoutStartSec=infinity
Add repository
sudo apt-get install python3-launchpadlib software-properties-common
sudo add-apt-repository ppa:linbit/linbit-drbd9-stack
Change repository => change "bookworm" to "jammy"
sudo nano /etc/apt/sources.d/linbit-ubuntu-linbit-drbd9-stack-bookworm.list
Install GUI
sudo apt update
sudo apt install linstor-gui
Sources
In theory you can now create a VM. In my short experience this (only) works when creating the first VM or each VM on one of the storage nodes and then migrating it to the diskless node.
linstor node restore <node>
linstor n r <node>