dotlinux blog

How to Set Up Clustering and High Availability in Proxmox

Proxmox Virtual Environment (Proxmox VE) is an open-source server virtualization platform that combines KVM (virtual machines) and LXC (containers) for efficient resource utilization. Clustering in Proxmox pools multiple physical servers (nodes) into a single logical entity, enabling resource sharing, load balancing, and High Availability (HA). HA ensures services (VMs/containers) remain accessible during node failures by automatically migrating workloads to healthy nodes using watchdog-based fencing. This guide walks you through setting up a Proxmox cluster and configuring HA, with practical steps and best practices.

Last Updated: 2026-05

Table of Contents#

Prerequisites#

Before setting up a Proxmox cluster, ensure:

Hardware#

  • At least 3 physical servers (nodes) for reliable quorum (a 2-node cluster requires an additional QDevice for the third vote; see the Proxmox Administration Guide). Nodes should have compatible CPUs (for live migration) and sufficient CPU/RAM/storage.
  • A dedicated network for cluster communication (recommended) and VM traffic (if needed).
  • Shared storage (NFS, Ceph, iSCSI) accessible by all nodes (required for HA).
  • A hardware watchdog timer (optional but recommended; the Linux kernel software watchdog softdog is used as fallback).

Software#

  • Proxmox VE 8.x or later installed on all nodes (download). As of 2025, Proxmox VE 9.0 (based on Debian 13 "Trixie") is the latest major release.
  • Static IPs (or reserved DHCP) and network connectivity between nodes (ping/SSH working).
  • Time synchronization (NTP/Chrony) to avoid clock drift.

Setting Up a Proxmox Cluster#

Preparing Nodes for Clustering#

1. Hostname and IP Configuration#

  • Set unique hostnames (e.g., pve1, pve2) in /etc/hostname and /etc/hosts: 
    # On pve1:
echo "pve1" > /etc/hostname
echo "192.168.1.101 pve1" >> /etc/hosts
echo "192.168.1.102 pve2" >> /etc/hosts
# Repeat for pve2, updating IP/hostname.
  • Restart networking: systemctl restart networking.

2. Time Synchronization#

  • Install NTP (Debian-based Proxmox): 
    apt update && apt install ntp
systemctl enable --now ntp
  • Verify with ntpq -p or chronyc sources.

3. Network Connectivity#

  • Ensure nodes can ping each other (e.g., ping pve2 from pve1).
  • Check firewall rules (Corosync v3+ uses UDP ports 5405–5412; open by default in Proxmox).

Creating the Cluster#

1. Initialize the First Node#

On the first node (e.g., pve1), create the cluster:

pvecm create my-proxmox-cluster  # Replace with your cluster name

This generates a Corosync configuration. Save the join command/password for other nodes.

Joining Additional Nodes#

On the second node (e.g., pve2), join the cluster:

pvecm join 192.168.1.101  # IP of the first node

Enter the root password of the first node. Repeat for more nodes.

Verifying Cluster Health#

Check cluster status with:

pvecm status

Output should show:

  • Quorum — indicates whether the cluster has a majority of nodes online (e.g., 2 of 3 nodes required for quorum in a 3-node cluster).
  • Nodes in Online state.

Configuring High Availability (HA)#

Setting Up HA Resources#

1. Verify HA Services Are Running#

The HA daemons (pve-ha-lrm and pve-ha-crm) start automatically on boot. Verify they are active:

systemctl status pve-ha-lrm
systemctl status pve-ha-crm

2. Add VMs/Containers to HA#

  • Web GUI: Go to Datacenter → HA → Add, select a VM/CT, and set the desired State (e.g., started).
  • CLI: 
    # For VMs (VMID):
ha-manager add vm:VMID --state started
# For LXC containers (CTID):
ha-manager add ct:CTID --state started

HA Groups and Restrictions#

HA groups restrict VMs to specific nodes (affinity/anti-affinity):

Note: In Proxmox VE 9, HA groups are deprecated in favor of HA rules (ha-manager rules), which support both node-affinity and resource-affinity rules. The ha-manager groupadd command still works but is considered legacy.

1. Create an HA Group#

ha-manager groupadd group1 --nodes pve1,pve2  # Nodes allowed to run VMs in this group

2. Assign VMs to the Group#

ha-manager set vm:VMID --group group1

3. Using HA Rules (PVE 9+)#

For newer deployments, use HA rules for more flexible control:

# Node affinity rule (restrict a VM to specific nodes)
ha-manager rules add node-affinity my-rule --nodes pve1:2,pve2:1 --resources vm:100
# Resource affinity rule (keep VMs together or apart)
ha-manager rules add resource-affinity keep-together --affinity positive --resources vm:100,vm:101

Fencing (Watchdog-Based)#

Fencing ensures a failed node is guaranteed to be offline before its services are recovered elsewhere, preventing split-brain scenarios where the same VM could run on two nodes simultaneously.

Modern Proxmox VE uses software watchdog-based fencing out of the box — no external fencing device configuration is required. Here is how it works:

  1. The HA stack (pve-ha-lrm) regularly resets a watchdog timer on each node.
  2. If a node loses quorum or the HA daemon stops responding, the watchdog elapses and triggers an automatic reboot of the node.
  3. Once fenced, the CRM recovers the failed node's services on remaining healthy nodes.

Configuring a Hardware Watchdog (Optional)#

By default, Proxmox uses the Linux kernel software watchdog (softdog). For higher reliability, you can configure a hardware watchdog module. Edit /etc/default/pve-ha-manager on each node:

# Select watchdog module (default is softdog)
WATCHDOG_MODULE=iTCO_wdt

Common hardware watchdog modules include iTCO_wdt (Intel), hpwdt (HP), and ipmi_watchdog (IPMI). The watchdog-mux service loads the specified module at startup.

Shared Storage Setup#

HA requires shared storage (VMs/CTs must be on storage accessible by all nodes).

1. Example: NFS#

  • NFS Server: Export a share (e.g., /nfs-share in /etc/exports).
  • Proxmox Nodes: Mount and add as storage: 
    mount -t nfs 192.168.1.50:/nfs-share /mnt/pve/nfs-share  # Replace with NFS server IP
    In the web GUI: Datacenter → Storage → Add → NFS to configure.

Testing High Availability#

Simulating Node Failure#

To test HA:

1. Graceful Shutdown#

shutdown -h now  # On a node (e.g., pve1)

2. Hard Reset#

Power off the node (simulate a crash).

Verifying VM Migration#

After node failure, check VM status:

ha-manager status  # HA resource status
qm list  # VM status on nodes

VMs should migrate to a healthy node.

HA Status Checks#

Check HA manager status:

ha-manager status

Resources should show active on a healthy node.

Best Practices for Clustering and HA#

Network Redundancy#

  • Bonded Interfaces: Use LACP bonding for cluster networks (edit /etc/network/interfaces).
  • Isolate Cluster Traffic: Use a dedicated VLAN/network for Corosync.

Quorum Management#

  • Odd Number of Nodes: Use 3/5 nodes to avoid split-brain.
  • QDevice: For 2-node clusters, add a quorum device (e.g., a Raspberry Pi) with pvecm qdevice setup <QDEVICE-IP>. See the Proxmox Administration Guide for QDevice setup details.

Resource Allocation#

  • Limit Resources: Set CPU/RAM limits for VMs to prevent overcommitment.
  • Prioritize Critical VMs: Assign higher priority to critical workloads.

Shutdown Policy#

Configure the HA shutdown policy under Datacenter → Options → HA Settings:

  • Migrate: Live-migrates HA services to other nodes before shutdown (recommended for planned maintenance).
  • Failover: Stops services but ensures they are recovered on other nodes.
  • Freeze: Stops services and freezes them until the node comes back online.
  • Conditional (default): Automatically detects shutdown vs. reboot and acts accordingly.

Node Maintenance Mode#

Before performing hardware maintenance, enable maintenance mode to gracefully migrate all HA services off a node:

ha-manager crm-command node-maintenance enable NODENAME

Disable when maintenance is complete:

ha-manager crm-command node-maintenance disable NODENAME

Regular Backups and Monitoring#

  • Backups: Use vzdump to back up VMs to external storage. Proxmox Backup Server (PBS) provides a dedicated, enterprise-grade backup solution with deduplication and incremental backups.
  • Monitoring: Use the built-in Proxmox VE web dashboard, pveperf (storage benchmarking), or integrate with Prometheus/Grafana for cluster-wide monitoring.
  • HA Simulator: Test HA behavior without a real cluster by installing pve-ha-simulator (apt install pve-ha-simulator). It simulates a 3-node cluster with sample VMs for learning and testing.

Troubleshooting Common Issues#

Network Connectivity Problems#

  • Check ping/SSH between nodes.
  • Verify firewall rules (open UDP ports 5405–5412 for Corosync).

Quorum Loss#

  • Symptom: No quorum in pvecm status.
  • Fix:
    • Check network connectivity.
    • Restart Corosync: systemctl restart corosync on all nodes.
    • Add a QDevice (for even nodes).

HA Resource Failures#

  • Check HA logs: journalctl -u pve-ha-lrm (on the node where the service runs) and journalctl -u pve-ha-crm (on the CRM master node).
  • Review /var/log/syslog for additional context.
  • Ensure target nodes have enough resources (CPU/RAM/storage).
  • If a service enters the error state, disable it first (ha-manager set vm:VMID --state disabled), fix the underlying issue, then re-enable it.

Conclusion#

Proxmox clustering and HA provide a robust virtualization environment with redundancy and failover capabilities. By following this guide—from cluster setup to HA configuration and testing—you can build a reliable infrastructure. Remember to follow best practices (network redundancy, quorum management, backups) to ensure long-term stability.

References#