A Guide to High Availability on Linux: Failover Strategies and Tools

In today’s digital landscape, downtime is costly. Whether it’s an e-commerce platform losing sales during peak hours or a critical database failing to serve customer requests, unplanned outages erode trust, revenue, and productivity. High Availability (HA) is the practice of designing systems to minimize downtime by ensuring continuous operation even when hardware, software, or network components fail. For Linux systems, HA is achieved through a combination of redundancy (duplicating critical components) and failover (automatically shifting workloads from a failed component to a healthy one). This guide demystifies HA on Linux, exploring core concepts, failover strategies, essential tools, implementation steps, and best practices to help you build resilient systems.

Table of Contents

1. Understanding High Availability and Failover
   • Key HA Metrics
   • What is Failover?
   • Failover Detection Mechanisms
2. Key Failover Strategies
   • Active-Passive (Standby)
   • Active-Active (Load-Sharing)
   • N+1 Redundancy
   • Comparing Strategies
3. Essential Linux HA Tools
   • Corosync + Pacemaker: The Industry Standard
   • Keepalived: Lightweight IP Failover
   • DRBD: Block-Level Storage Replication
   • HAProxy: Load Balancing with Health Checks
4. Implementation: Step-by-Step Examples
   • Example 1: Active-Passive Web Server with Keepalived
   • Example 2: Database Cluster with Pacemaker/Corosync
5. Common Practices and Pitfalls
   • Testing Failover Regularly
   • Monitoring Cluster Health
   • Fencing (STONITH): Preventing Split-Brain
   • Pitfalls to Avoid
6. Best Practices for HA on Linux
7. Conclusion
8. References

1. Understanding High Availability and Failover

Key HA Metrics

HA systems are measured by their ability to maintain uptime. Common metrics include:

• Uptime: Percentage of time the system is operational (e.g., 99.9% uptime allows ~8.76 hours of downtime/year; 99.999% allows ~5.25 minutes/year).
• MTBF (Mean Time Between Failures): Average time between system failures (higher = more reliable).
• MTTR (Mean Time to Recovery): Average time to restore service after a failure (lower = better).

What is Failover?

Failover is the process of automatically or manually transferring workloads from a failed “primary” component to a healthy “secondary” component. It ensures uninterrupted service by masking failures.

Failover Detection Mechanisms

• Heartbeats: Periodic network messages between nodes (e.g., Corosync uses UDP/TCP heartbeats). If a node stops sending heartbeats, it’s marked as failed.
• Health Checks: Active monitoring of resources (e.g., “Is the web server process running?” or “Is the database responding to queries?”). Tools like Pacemaker or Keepalived use scripts for health checks.
• Quorum: In clusters with ≥3 nodes, a majority vote determines which nodes are active (prevents “split-brain,” where nodes can’t communicate and both claim to be primary).

2. Key Failover Strategies

Active-Passive (Standby)

• How it works: One “active” node handles all traffic; a “passive” (standby) node remains idle until the active node fails.
• Use Case: Critical but low-to-moderate traffic services (e.g., small databases, internal APIs).
• Pros: Simple to implement; minimal resource overhead on the passive node.
• Cons: Wasted capacity (passive node is idle); potential latency during failover.

Active-Active (Load-Sharing)

• How it works: Multiple nodes actively handle traffic (e.g., via load balancing). If one node fails, others absorb its workload.
• Use Case: High-traffic services (e.g., web servers, caching layers like Redis).
• Pros: Maximizes resource utilization; no idle nodes; better scalability.
• Cons: Complex coordination (e.g., session persistence, data consistency); higher resource requirements.

N+1 Redundancy

• How it works: “N” active nodes are paired with “1” standby node. If any active node fails, the standby takes over.
• Use Case: Scalable systems with multiple identical components (e.g., application servers in a microservices architecture).
• Pros: Cost-effective balance of redundancy and capacity.
• Cons: Single point of failure if the standby node itself fails.

Comparison Table

3. Essential Linux HA Tools

Corosync + Pacemaker: The HA Cluster Stack

• Corosync: A messaging layer that provides cluster membership, quorum, and reliable communication between nodes (replaces older tools like Heartbeat).
• Pacemaker: A resource manager that orchestrates failover. It monitors resources (e.g., IP addresses, databases) and moves them to healthy nodes.

Use Case: Enterprise-grade clusters (e.g., databases, virtualization hosts).
Example Workflow: Corosync detects a failed node → Pacemaker initiates failover → Resources (IP, database) are moved to the standby node.

Keepalived

• What it does: Lightweight tool for IP failover using VRRP (Virtual Router Redundancy Protocol). VRRP assigns a “virtual IP” (VIP) to the active node; if it fails, the VIP moves to the standby node.
• Use Case: Simple web servers, load balancers, or edge routers.
• Pros: Minimal overhead; easy to configure; no dependency on complex cluster stacks.

DRBD (Distributed Replicated Block Device)

• What it does: Synchronously or asynchronously replicates block storage (e.g., /dev/sda) between nodes. If the primary node’s disk fails, the secondary node’s replicated disk takes over.
• Use Case: Storage HA (e.g., databases, file servers).
• Pros: Block-level replication (works with any filesystem); integrates with Pacemaker for automated failover.

HAProxy

• What it does: A load balancer with built-in health checks. It routes traffic to active backend servers and removes failed servers from the pool.
• Use Case: Active-active web clusters; API gateways.
• Pros: Layer 4/7 load balancing; supports TCP (databases) and HTTP/HTTPS (web).

4. Implementation: Step-by-Step Examples

Example 1: Active-Passive Web Server with Keepalived

Goal: Use Keepalived to fail over a VIP between two Nginx web servers (Node A = primary, Node B = standby).

Prerequisites

• Two Linux nodes (e.g., Ubuntu 22.04):
  • Node A: 192.168.1.10 (primary)
  • Node B: 192.168.1.11 (standby)
• VIP: 192.168.1.100 (shared IP for clients).
• Nginx installed on both nodes (sudo apt install nginx).

Step 1: Install Keepalived

# On both nodes
sudo apt update && sudo apt install -y keepalived

Step 2: Configure Keepalived on Node A (Primary)

Create /etc/keepalived/keepalived.conf:

vrrp_instance VI_1 {
  state MASTER           # This node is primary
  interface eth0         # Network interface to use (check with `ip link`)
  virtual_router_id 51   # Must be the same on all nodes in the VRRP group
  priority 100           # Higher priority = more likely to be master (100 > 90)
  advert_int 1           # Send VRRP heartbeat every 1 second

  # Authentication (optional but recommended)
  authentication {
    auth_type PASS
    auth_pass mysecretkey  # Must match on all nodes
  }

  # Virtual IP (VIP) to assign to the active node
  virtual_ipaddress {
    192.168.1.100/24 dev eth0  # VIP/CIDR and interface
  }

  # Health check: Restart Nginx if it fails; if it can't restart, lower priority
  track_script {
    chk_nginx
  }
}

# Nginx health check script
vrrp_script chk_nginx {
  script "/usr/bin/pgrep nginx"  # Check if nginx process is running
  interval 2                     # Run check every 2 seconds
  weight -20                     # If check fails, lower priority by 20 (100-20=80 < 90)
}

Step 3: Configure Keepalived on Node B (Standby)

Create /etc/keepalived/keepalived.conf (only differences from Node A are state and priority):

vrrp_instance VI_1 {
  state BACKUP          # This node is standby
  interface eth0
  virtual_router_id 51
  priority 90           # Lower than master (90 < 100)
  advert_int 1

  authentication {
    auth_type PASS
    auth_pass mysecretkey
  }

  virtual_ipaddress {
    192.168.1.100/24 dev eth0
  }

  track_script {
    chk_nginx
  }
}

vrrp_script chk_nginx {
  script "/usr/bin/pgrep nginx"
  interval 2
  weight -20
}

Step 4: Start Keepalived and Test Failover

# On both nodes
sudo systemctl enable --now keepalived

# Verify VIP on Node A (primary)
ip addr show eth0 | grep 192.168.1.100  # Should show the VIP

# Test failover: Stop Nginx on Node A
sudo systemctl stop nginx

# Check VIP on Node B after ~5 seconds (VRRP advert_int + health check interval)
ip addr show eth0 | grep 192.168.1.100  # VIP should now be on Node B

Example 2: Database Cluster with Pacemaker/Corosync

Goal: Create a 2-node active-passive cluster for PostgreSQL using Pacemaker (resource manager) and Corosync (messaging layer).

Prerequisites

• Two nodes: db1 (192.168.1.20) and db2 (192.168.1.21).
• Passwordless SSH between nodes (for cluster setup).
• PostgreSQL installed on both nodes (data will be replicated with DRBD, not covered here for brevity).

Step 1: Install Corosync and Pacemaker

# On both nodes
sudo apt install -y corosync pacemaker pcs fence-agents  # pcs = Pacemaker CLI
sudo systemctl enable --now pcsd  # PCS daemon for cluster management
sudo passwd hacluster  # Set password for the 'hacluster' admin user (same on both nodes)

Step 2: Authenticate Nodes and Create Cluster

# On db1
sudo pcs cluster auth db1 db2 -u hacluster -p <hacluster-password>
sudo pcs cluster setup --name postgres-cluster db1 db2
sudo pcs cluster start --all
sudo pcs cluster enable --all  # Start on boot

Step 3: Configure Fencing (STONITH)

Fencing (Shoot The Other Node In The Head) prevents split-brain by powering off unresponsive nodes. Use a fence agent (e.g., fence_ipmilan for IPMI, or fence_virsh for VMs).

# Example: Configure fence_virsh for KVM VMs (adjust for your environment)
sudo pcs stonith create fence-db1 fence_virsh ipaddr=hypervisor-ip login=root vm_name=db1
sudo pcs stonith create fence-db2 fence_virsh ipaddr=hypervisor-ip login=root vm_name=db2

# Enable fencing (critical for cluster safety)
sudo pcs property set stonith-enabled=true

Step 4: Add Cluster Resources

Add a VIP and PostgreSQL as managed resources:

# Add virtual IP (VIP)
sudo pcs resource create VirtualIP IPaddr2 ip=192.168.1.30 cidr_netmask=24

# Add PostgreSQL service (systemd-based)
sudo pcs resource create PostgreSQL systemd:postgresql

# Configure dependencies: PostgreSQL depends on VIP (start VIP first, then PostgreSQL)
sudo pcs constraint colocation add PostgreSQL with VirtualIP INFINITY
sudo pcs constraint order VirtualIP then PostgreSQL

# Verify resources
sudo pcs status

Step 5: Test Failover

# Simulate failure on db1 (primary node)
sudo pcs node standby db1  # Mark db1 as standby (triggers failover)

# Check status: VIP and PostgreSQL should move to db2
sudo pcs status

# Bring db1 back online
sudo pcs node unstandby db1

5. Common Practices and Pitfalls

Common Practices

• Test Failover Regularly: Use tools like pcs node standby (Pacemaker) or systemctl stop keepalived to simulate failures. Automate tests with cron jobs or chaos engineering tools (e.g., Chaos Monkey for Linux).
• Monitor Cluster Health: Use Prometheus + Grafana (with prometheus-pacemaker-exporter) or Nagios to track node status, resource usage, and failover events.
• Document Everything: Cluster topology, resource dependencies, fencing configuration, and failover runbooks.
• Use Quorum for ≥3 Nodes: In clusters with 3+ nodes, enable quorum (default in Corosync) to prevent split-brain.

Pitfalls to Avoid

• No Fencing: Without STONITH, split-brain can cause data corruption (e.g., two nodes writing to the same disk).
• Poor Network Latency: Corosync/heartbeats require low-latency, high-bandwidth networks (use dedicated cluster networks if possible).
• Unreliable Health Checks: Flaky scripts (e.g., “ping” instead of application-level checks) cause false failovers.
• Overcomplicating Resources: Avoid adding too many resources (e.g., 10+ services) to a single cluster; split into smaller clusters instead.

6. Best Practices for HA on Linux

1. Prioritize Fencing: Always enable STONITH. Use hardware-based fencing (e.g., IPMI) over software-based (e.g., fence_virsh) for reliability.
2. Keep Clusters Small: 2-3 nodes are easier to manage than large clusters. Use active-active for scaling instead of adding more nodes.
3. Secure the Cluster: Encrypt Corosync traffic with TLS (use corosync-cfgtool to configure), restrict VIP access with firewalls (e.g., ufw allow from 192.168.1.0/24 to 192.168.1.30), and audit resource agents.
4. Update Regularly: Patch Corosync, Pacemaker, and OS packages – but test updates in a staging cluster first.
5. Avoid Single Points of Failure (SPOFs): Redundant power, network switches, and storage (use DRBD or shared storage like Ceph).

7. Conclusion

High availability on Linux is achievable with the right strategies and tools. By combining redundancy (active-passive/active-active) with failover mechanisms (Corosync, Keepalived) and tools like Pacemaker for orchestration, you can minimize downtime and ensure service reliability.

Start small: Deploy a simple Keepalived setup for web servers, then scale to Pacemaker/DRBD for databases. Always test failover, monitor aggressively, and prioritize fencing to avoid split-brain. With these practices, you’ll build systems that keep serving users – even when components fail.

8. References

• Pacemaker Documentation
• Corosync Documentation
• Keepalived Man Page
• DRBD User’s Guide
• Linux High Availability Administration (Book by SUSE)
• VRRP Protocol (RFC 5798)