How to Sync Proxmox VE Cloud-Init Snippets Between Nodes in a Cluster

In this article, I am going to show you how to sync Proxmox VE cloud-init snippets stored in local storage between Proxmox VE nodes in a Proxmox VE cluster using a cron job. This is helpful when you want to automate tasks, do VM migrations between nodes in a Proxmox cluster, and so on.

In Proxmox VE, the cloud-init snippets are stored in the path /var/lib/vz/snippets. To sync the snippets, we need to sync this directory between all the Proxmox VE nodes in the cluster. We can use the rsync program and some shell scripts for that.

root@pve-01:~# ls -lh /var/lib/vz/snippets/
total 12K
-rw-r--r-- 1 root root 264 Jan 14 13:21 init-debian-vps.yaml
-rw-r--r-- 1 root root 265 Jan 14 13:20 init-ubuntu-vps.yaml
-rw-r--r-- 1 root root 488 Jan 14 14:05 userconfig-80.yaml
root@pve-01:~# 

Adding Proxmox VE Nodes to the /etc/hosts File

To make referencing Proxmox VE nodes easier, we can use the /etc/hosts file. This way we won’t have to type in the IP addresses of the Proxmox VE nodes in our shell script.

Open the /etc/hosts file for editing with the nano text editor:

root@pve-01:~# nano /etc/hosts

Add the following lines to the /etc/hosts file and make necessary adjustments (i.e. change the IP addresses to your Proxmox VE node IP addresses).

10.10.10.11 pve-01
10.10.10.12 pve-02
10.10.10.13 pve-03 

Shell Script to Sync Proxmox VE Snippets

Create a shell script sync-snippets.sh in the /usr/local/bin directory with the following command:

root@pve-01:~# nano /usr/local/bin/sync-snippets.sh

Type in the following lines in the sync-snippets.sh file:

#!/bin/bash

# Configuration
SNIPPETS_DIR="/var/lib/vz/snippets/"
# List your other node IPs or hostnames here
NODES=("pve-02" "pve-03") 
LOCK_FILE="/tmp/snippet_sync.lock"
LOG_FILE="/var/log/sync-snippets.log"

# Ensure the script doesn't run multiple times simultaneously
if [ -e ${LOCK_FILE} ]; then
    echo "[$(hostname)] syncing snippets..." >> $LOG_FILE
    exit 1
fi

touch ${LOCK_FILE}

# Perform sync to each target node
for NODE in "${NODES[@]}"; do
    rsync -auqz --delete "$SNIPPETS_DIR" "root@${NODE}:${SNIPPETS_DIR}"
    echo "[${NODE}] synced snippets." >> $LOG_FILE
done

rm -f ${LOCK_FILE}

Make the script executable with the following command:

root@pve-01:~# chmod +x /usr/local/bin/sync-snippets.sh

Make sure to test the script throughly before adding it to crontab.

Adding the Snippet Syncing Script to Crontab

Open crontab with the following command:

root@pve-01:~# crontab -e

Since snippet files are small, I want to run the sync script sync-snippets.sh every 5 seconds to make sure the snippets are synced between the Proxmox nodes as soon as possible. So, I added the following line to crontab file.

* * * * * for i in {1..12}; do /usr/local/bin/sync-snippets.sh; sleep 5; done

If you want to run every 10 seconds (let’s say), add the following line instead:

* * * * * for i in {1..6}; do /usr/local/bin/sync-snippets.sh; sleep 10; done

The logic here is that we can’t run a script every 5 or 10 seconds using cron job, we can only run a script every minute. So, in a minute we have 60 seconds, and if we want to run the script every 5 seconds, we need to run the script 12 times every minute. 5s * 12 = 60s = 1 minute.

In the same way, if we want to run a script every 10 seconds, we need to run the script 6 times every minute. 10s * 6 = 60s = 1 minute.

Now, you should be able to adjust the sync interval as required.

Checking Snippet Sync Logs

The snippets sync script stores sync logs in the /var/log/sync-snippets.log file.

To check if the snippets are being synced correctly, you can check the snippet sync logs with the following command:

root@pve-01:~# tail -f /var/log/sync-snippets.log 
[pve-02] synced snippets.
[pve-03] synced snippets.
[pve-02] synced snippets.
[pve-03] synced snippets.
[pve-02] synced snippets.
[pve-03] synced snippets.