dotlinux blog

20 Useful Commands of ‘Sysstat’ Utilities (mpstat, pidstat, iostat and sar) for Linux Performance Monitoring

Sysstat is a powerful suite of Linux utilities for system performance monitoring, troubleshooting, and capacity planning. It includes four core tools:

  • mpstat: Analyzes CPU (and core) utilization.
  • pidstat: Tracks process-level metrics (CPU, memory, I/O, context switches).
  • iostat: Monitors disk/partition I/O performance.
  • sar (System Activity Reporter): Collects historical and real-time stats (CPU, memory, network, load, etc.).

These tools help identify bottlenecks (e.g., CPU saturation, disk I/O latency, memory pressure) and optimize system resources. Below, we explore 20 essential commands from the Sysstat suite.

2026-05

Table of Contents#

1. mpstat Commands (Multiprocessor Statistics)#

mpstat focuses on CPU (and core) utilization, including user/system time, interrupts, and idle time.

1.1 mpstat – Real-Time CPU Usage (All Cores)#

  • Purpose: Monitor average CPU usage across all cores (real-time or historical).
  • Syntax: mpstat [interval] [count]
    • interval: Time (seconds) between updates.
    • count: Number of updates to display.
  • Example: mpstat 2 5
    • Prints CPU stats every 2 seconds, 5 times.
  • Output Interpretation:
    • Columns: %user (user-space CPU), %system (kernel-space), %iowait (CPU waiting for disk I/O), %idle (idle CPU).
    • Use: Quick check of overall CPU load.

1.2 mpstat -P ALL – Per-Core CPU Monitoring#

  • Purpose: Analyze individual CPU cores (e.g., identify a single core bottleneck).
  • Syntax: mpstat -P ALL [interval] [count]
  • Example: mpstat -P ALL 1 3
    • Displays stats for each core (e.g., CPU 0, CPU 1, ...) every 1 second, 3 times.
  • Output Interpretation:
    • Spot cores with high %user/%system (e.g., a single core at 100% while others are idle).

1.3 mpstat -u – Historical CPU Utilization (with sar)#

  • Purpose: View historical CPU stats (requires sar to collect data).
  • Syntax: mpstat -u -f /var/log/sa/saXX
    • /var/log/sa/saXX: sar’s log file (XX = day of month).
  • Example: mpstat -u -f /var/log/sa/sa15
    • Reads CPU stats from the 15th day.
  • Note: Enable sar’s daily logging (default in most distros) to use this.

1.4 mpstat -I SUM – Interrupt Statistics#

  • Purpose: Monitor CPU interrupts (hardware/software).
  • Syntax: mpstat -I SUM [interval] [count]
  • Example: mpstat -I SUM 1 5
    • Displays interrupt counts (per second) for 5 iterations.
  • Output Interpretation:
    • %irq (hardware interrupts), %soft (software interrupts).
    • Use: Diagnose “interrupt storms” (e.g., high %soft from network drivers).

1.5 mpstat -A – All CPU/Interrupt Statistics#

  • Purpose: Display all available mpstat metrics (CPU, interrupts, softirqs).
  • Syntax: mpstat -A [interval] [count]
  • Example: mpstat -A 2 1
    • Prints full stats (once, 2-second interval).
  • Use: Comprehensive CPU/interrupt analysis (e.g., debug kernel-level issues).

2. pidstat Commands (Process Statistics)#

pidstat tracks process-level metrics (CPU, memory, I/O, context switches) to identify resource-hungry processes.

2.1 pidstat – Real-Time Process CPU Usage#

  • Purpose: Monitor per-process CPU usage (real-time).
  • Syntax: pidstat [interval] [count]
  • Example: pidstat 1 5
    • Displays CPU usage for active processes every 1 second, 5 times.
  • Output Interpretation:
    • Columns: %usr (user-space CPU for process), %system (kernel-space), PID (process ID).
    • Use: Identify CPU-intensive processes (e.g., a database or web server).

2.2 pidstat -r – Process Memory Usage#

  • Purpose: Analyze process memory (resident set size, page faults).
  • Syntax: pidstat -r [interval] [count]
  • Example: pidstat -r 1 3
    • Displays memory stats (e.g., RSS (resident memory), %MEM (percent of RAM)) every 1 second, 3 times.
  • Output Interpretation:
    • High RSS or page faults → memory-heavy or inefficiently coded process.

2.3 pidstat -d – Process-Level Disk I/O#

  • Purpose: Track process-specific disk I/O (reads, writes, throughput).
  • Syntax: pidstat -d [interval] [count]
  • Example: pidstat -d 2 5
    • Displays I/O stats (e.g., KB_rd/s (read rate), KB_wr/s (write rate)) every 2 seconds, 5 times.
  • Output Interpretation:
    • High KB_rd/s/KB_wr/s → process is I/O-bound (e.g., a database backup).

2.4 pidstat -w – Process Context Switches#

  • Purpose: Monitor context switches (voluntary/involuntary) for processes.
  • Syntax: pidstat -w [interval] [count]
  • Example: pidstat -w 1 5
    • Displays voluntary_ctxt_switches (process yields CPU) and nonvoluntary_ctxt_switches (OS preempts process) every 1 second, 5 times.
  • Output Interpretation:
    • High nonvoluntary switches → process is “fighting” for CPU time (e.g., a misconfigured app).

2.5 pidstat -h – Historical Process Stats (with sar)#

  • Purpose: View historical process metrics (requires sar to log process data).
  • Syntax: pidstat -h -f /var/log/sa/saXX
  • Example: pidstat -h -f /var/log/sa/sa15
    • Reads historical process stats from the 15th day.
  • Note: Enable sar’s process accounting (e.g., sar -P ALL ...) to collect data.

3. iostat Commands (Input/Output Statistics)#

iostat focuses on disk/partition I/O performance (throughput, latency, utilization).

3.1 iostat – Real-Time Disk I/O#

  • Purpose: Monitor disk I/O rates (reads, writes, transactions per second).
  • Syntax: iostat [interval] [count]
  • Example: iostat 1 10
    • Displays disk stats (e.g., tps (transactions per second), KB_read/s, KB_wrtn/s) every 1 second, 10 times.
  • Output Interpretation:
    • High tps or KB_read/s → disk is busy (e.g., during a large file transfer).

3.2 iostat -x – Extended Disk I/O Details#

  • Purpose: Get advanced disk stats (latency, service time, utilization).
  • Syntax: iostat -x [interval] [count]
  • Example: iostat -x 2 5
    • Displays extended stats (e.g., await (avg wait time), svctm (service time), %util (disk busy)) every 2 seconds, 5 times.
  • Output Interpretation:
    • High await/%util → disk is slow or saturated (e.g., a failing HDD).

3.3 iostat -p ALL – Per-Partition I/O#

  • Purpose: Analyze individual partitions (e.g., /dev/sda1, /dev/sdb2).
  • Syntax: iostat -p ALL [interval] [count]
  • Example: iostat -p ALL 1 3
    • Displays I/O stats for all partitions (not just disks) every 1 second, 3 times.
  • Use: Troubleshoot partition-specific issues (e.g., a full partition causing I/O bottlenecks).

3.4 iostat -z – Exclude Idle Devices#

  • Purpose: Omit idle disks/partitions (reduces clutter in output).
  • Syntax: iostat -z [interval] [count]
  • Example: iostat -z 1 5
    • Displays stats only for devices with active I/O.
  • Use: Focus on busy disks (e.g., in a system with many unused disks).

3.5 iostat -t – Include Timestamp#

  • Purpose: Add a timestamp to each output line (e.g., correlate I/O with time).
  • Syntax: iostat -t [interval] [count]
  • Example: iostat -t 2 5
    • Displays disk stats with timestamps (e.g., 21:45:01).
  • Use: Analyze I/O during peak hours (e.g., database backups at 2 AM).

4. sar Commands (System Activity Reporter)#

sar collects historical and real-time stats for CPU, memory, network, load, and more.

4.1 sar -u – Historical/Real-Time CPU Utilization#

  • Purpose: Track CPU usage (real-time or historical).
  • Syntax:
    • Real-time: sar -u [interval] [count]
    • Historical: sar -u -f /var/log/sa/saXX
  • Example:
    • Real-time: sar -u 1 5 (CPU stats every 1 second, 5 times).
      • Historical: sar -u -f /var/log/sa/sa15 (CPU stats from the 15th day).
  • Output Interpretation:
    • Columns: %user, %system, %iowait, %idle (similar to mpstat).

4.2 sar -r – Memory Utilization#

  • Purpose: Analyze RAM usage (free, used, buffers, cache).
  • Syntax:
    • Real-time: sar -r [interval] [count]
    • Historical: sar -r -f /var/log/sa/saXX
  • Example: sar -r 2 5
    • Displays memory stats (e.g., kbmemfree (free RAM), %memused (percent used)) every 2 seconds, 5 times.
  • Output Interpretation:
    • Low kbmemfree + high %memused → memory pressure (e.g., swap usage may follow).

4.3 sar -b – I/O and Transfer Rates#

  • Purpose: Monitor disk I/O operations (reads, writes) and transfer rates.
  • Syntax:
    • Real-time: sar -b [interval] [count]
    • Historical: sar -b -f /var/log/sa/saXX
  • Example: sar -b 1 10
    • Displays rtps (read operations per second), wtps (write ops), bread/s (read throughput), bwrtn/s (write throughput) every 1 second, 10 times.
  • Use: Track disk I/O throughput (e.g., during a large file copy).

4.4 sar -n DEV – Network Interface Stats#

  • Purpose: Analyze network interface traffic (bytes, packets).
  • Syntax:
    • Real-time: sar -n DEV [interval] [count]
    • Historical: sar -n DEV -f /var/log/sa/saXX
  • Example: sar -n DEV 1 5
    • Displays stats for each interface (e.g., eth0, lo) every 1 second, 5 times.
  • Output Interpretation:
    • Columns: rxkB/s (received bytes/s), txkB/s (transmitted bytes/s), rxpck/s (received packets/s).
    • Use: Identify network bottlenecks (e.g., a saturated eth0).

4.5 sar -q – System Load Averages#

  • Purpose: Monitor system load averages (1, 5, 15 minutes) and runnable processes.
  • Syntax:
    • Real-time: sar -q [interval] [count]
    • Historical: sar -q -f /var/log/sa/saXX
  • Example: sar -q 1 5
    • Displays runq-sz (runnable processes), plist-sz (process list size), and load averages every 1 second, 5 times.
  • Output Interpretation:
    • High runq-sz + load averages → system is overloaded (e.g., too many concurrent tasks).

Conclusion#

Sysstat’s utilities (mpstat, pidstat, iostat, sar) provide granular, actionable insights into Linux system performance. The 20 commands above cover:

  • CPU/core utilization (e.g., mpstat -P ALL).
  • Process-level metrics (e.g., pidstat -d for I/O-bound processes).
  • Disk/partition I/O (e.g., iostat -x for latency).
  • System-wide trends (e.g., sar -r for memory, sar -n DEV for network).

By mastering these commands, you can proactively identify bottlenecks, optimize resources, and ensure system stability. Experiment with them to tailor monitoring to your environment!

References#