I have three small proxmox nodes (A) that I want to use to provide highly-available storage.
The fourth node (B) will be "diskless" as in not participating in the ha-storage part. It will use the storage for (some of) its virtual machines and containers.
Follow this tutorial to get a vitastor storage pool in Proxmox and access it from all nodes in the Proxmox cluster, also "diskless" nodes.
I added A or B in front of each step if they only need to be performed on either the storage nodes (A) or the diskless node (B). If no letter is shown, the step is to be performed on both.
A: jpl-proxmox7, jpl-proxmox8, jpl-proxmox9
B: jpl-proxmox6
Sources
- Official docs: Quick Start
- Official docs: Proxmox VE
Add repositories:
wget https://vitastor.io/debian/pubkey.gpg -O /etc/apt/trusted.gpg.d/vitastor.gpg
nano /etc/apt/sources.list.d/vitastor.list
A: Install dependencies on storage nodes:
apt update
apt install vitastor lp-solve etcd vitastor-client pve-storage-vitastor
B: Install dependencies on non-storage (diskless) node:
apt update
apt install vitastor lp-solve vitastor-client pve-storage-vitastor
etcd / monitor nodesAdd etcd addresses and osd-network on all nodes:
nano /etc/vitastor/vitastor.conf
{
"etcd_address": ["192.168.100.117:2379","192.168.100.118:2379","192.168.100.119:2379"],
"osd_network": "192.168.100.0/24"
}
A: Initialize etcd on each monitor:
/usr/lib/vitastor/mon/make-etcd
Enable each monitor:
systemctl enable vitastor-mon.service
A: On each storage node prepare disks:
wipefs -a /dev/sda
dd if=/dev/zero of=/dev/sda bs=1M oflag=direct count=1000
vitastor-disk prepare /dev/sda --disable_data_fsync off
Info: the
vitastor-disk preparecommand currently (v1.9.3) has a bug: https://github.com/vitalif/vitastor/issues/73, which will be fixed in an upcoming release. Running theddcommand above is a workaround for this issue, but shouldn't be needed in a future version.
Check is OSD's are enabled:
vitastor-cli status
A: On each storage node enable OSD service:
systemctl enable vitastor-osd@<x>.service
Add a pool:
vitastor-cli create-pool testpool --pg_size 3 --pg_count 32
Check if pool is active:
vitastor-cli status
vitastor-cli ls-pools
vitastor-cli df
vitastor-cli osd-tree
Open /etc/pve/storage.cfg and add:
vitastor: vitastor_test
vitastor_pool testpool
content images,rootdir
Restart Proxmox services:
systemctl restart pvedaemon pveproxy