dotlinux blog

20 Useful Commands of ‘Sysstat’ Utilities (mpstat, pidstat, iostat and sar) for Linux Performance Monitoring

Sysstat is a powerful suite of Linux utilities for system performance monitoring, troubleshooting, and capacity planning. It includes four core tools:

  • mpstat: Analyzes CPU (and core) utilization.
  • pidstat: Tracks process-level metrics (CPU, memory, I/O, context switches).
  • iostat: Monitors disk/partition I/O performance.
  • sar (System Activity Reporter): Collects historical and real-time stats (CPU, memory, network, load, etc.).

These tools help identify bottlenecks (e.g., CPU saturation, disk I/O latency, memory pressure) and optimize system resources. As of 2026, the latest stable release is sysstat 12.7.9, which includes JSON output support for all commands and improved kernel compatibility. Below, we explore 20 essential commands from the Sysstat suite.

Last Updated: 2026-05

Installing Sysstat#

Before running these commands, ensure sysstat is installed on your system:

# Debian/Ubuntu
sudo apt install sysstat
sudo dpkg-reconfigure sysstat  # Enable data collection
 
# RHEL/CentOS/Fedora
sudo dnf install sysstat
sudo systemctl enable --now sysstat
 
# Arch Linux
sudo pacman -S sysstat

Verify the installation with mpstat -V.

Table of Contents#

1. mpstat Commands (Multiprocessor Statistics)#

mpstat focuses on CPU (and core) utilization, including user/system time, interrupts, and idle time.

1.1 mpstat – Real-Time CPU Usage (All Cores)#

  • Purpose: Monitor average CPU usage across all cores (real-time or historical).
  • Syntax: mpstat [interval] [count]
    • interval: Time (seconds) between updates.
    • count: Number of updates to display.
  • Example: mpstat 2 5
    • Prints CPU stats every 2 seconds, 5 times.
  • Output Interpretation:
    • Columns: %user (user-space CPU), %system (kernel-space), %iowait (CPU waiting for disk I/O), %idle (idle CPU).
    • Use: Quick check of overall CPU load.

1.2 mpstat -P ALL – Per-Core CPU Monitoring#

  • Purpose: Analyze individual CPU cores (e.g., identify a single core bottleneck).
  • Syntax: mpstat -P ALL [interval] [count]
  • Example: mpstat -P ALL 1 3
    • Displays stats for each core (e.g., CPU 0, CPU 1, ...) every 1 second, 3 times.
  • Output Interpretation:
    • Spot cores with high %user/%system (e.g., a single core at 100% while others are idle).

1.3 mpstat -u – CPU Utilization Statistics#

  • Purpose: Display CPU utilization statistics (this is the default display mode).
  • Syntax: mpstat -u [interval] [count]
  • Example: mpstat -u 2 5
    • Displays CPU utilization every 2 seconds, 5 times.
  • Output Interpretation:
    • Columns: %usr, %nice, %sys, %iowait, %irq, %soft, %steal, %guest, %gnice, %idle.
  • Note: To view historical stats from sar data files, use mpstat -f /var/log/sa/saXX (where XX is the day of month).

1.4 mpstat -I SUM – Interrupt Statistics#

  • Purpose: Monitor CPU interrupts (hardware/software).
  • Syntax: mpstat -I SUM [interval] [count]
  • Example: mpstat -I SUM 1 5
    • Displays interrupt counts (per second) for 5 iterations.
  • Output Interpretation:
    • %irq (hardware interrupts), %soft (software interrupts).
    • Use: Diagnose “interrupt storms” (e.g., high %soft from network drivers).

1.5 mpstat -A – All CPU/Interrupt Statistics#

  • Purpose: Display all available mpstat metrics (CPU, interrupts, softirqs).
  • Syntax: mpstat -A [interval] [count]
  • Example: mpstat -A 2 1
    • Prints full stats (once, 2-second interval).
  • Use: Comprehensive CPU/interrupt analysis (e.g., debug kernel-level issues).

2. pidstat Commands (Process Statistics)#

pidstat tracks process-level metrics (CPU, memory, I/O, context switches) to identify resource-hungry processes.

2.1 pidstat – Real-Time Process CPU Usage#

  • Purpose: Monitor per-process CPU usage (real-time).
  • Syntax: pidstat [interval] [count]
  • Example: pidstat 1 5
    • Displays CPU usage for active processes every 1 second, 5 times.
  • Output Interpretation:
    • Columns: %usr (user-space CPU for process), %system (kernel-space), PID (process ID).
    • Use: Identify CPU-intensive processes (e.g., a database or web server).

2.2 pidstat -r – Process Memory Usage#

  • Purpose: Analyze process memory (resident set size, page faults).
  • Syntax: pidstat -r [interval] [count]
  • Example: pidstat -r 1 3
    • Displays memory stats (e.g., RSS (resident memory), %MEM (percent of RAM)) every 1 second, 3 times.
  • Output Interpretation:
    • High RSS or page faults → memory-heavy or inefficiently coded process.

2.3 pidstat -d – Process-Level Disk I/O#

  • Purpose: Track process-specific disk I/O (reads, writes, throughput).
  • Syntax: pidstat -d [interval] [count]
  • Example: pidstat -d 2 5
    • Displays I/O stats (e.g., KB_rd/s (read rate), KB_wr/s (write rate)) every 2 seconds, 5 times.
  • Output Interpretation:
    • High KB_rd/s/KB_wr/s → process is I/O-bound (e.g., a database backup).

2.4 pidstat -w – Process Context Switches#

  • Purpose: Monitor context switches (voluntary/involuntary) for processes.
  • Syntax: pidstat -w [interval] [count]
  • Example: pidstat -w 1 5
    • Displays voluntary_ctxt_switches (process yields CPU) and nonvoluntary_ctxt_switches (OS preempts process) every 1 second, 5 times.
  • Output Interpretation:
    • High nonvoluntary switches → process is “fighting” for CPU time (e.g., a misconfigured app).

2.5 pidstat -h – Horizontal Output Format#

  • Purpose: Display all activities horizontally on a single line, making it easier to parse with other programs. No average statistics are shown at the end of the report.
  • Syntax: pidstat -h [interval] [count]
  • Example: pidstat -rh 2 3
    • Displays memory stats in compact horizontal format every 2 seconds, 3 times.
  • Output Interpretation:
    • All metrics appear on one line per sample, ideal for scripting and log parsing.
  • Note: To view historical process stats from sar data files, use pidstat -f /var/log/sa/saXX (where XX is the day of month).

3. iostat Commands (Input/Output Statistics)#

iostat focuses on disk/partition I/O performance (throughput, latency, utilization).

3.1 iostat – Real-Time Disk I/O#

  • Purpose: Monitor disk I/O rates (reads, writes, transactions per second).
  • Syntax: iostat [interval] [count]
  • Example: iostat 1 10
    • Displays disk stats (e.g., tps (transactions per second), KB_read/s, KB_wrtn/s) every 1 second, 10 times.
  • Output Interpretation:
    • High tps or KB_read/s → disk is busy (e.g., during a large file transfer).

3.2 iostat -x – Extended Disk I/O Details#

  • Purpose: Get advanced disk stats (latency, queue depth, utilization).
  • Syntax: iostat -x [interval] [count]
  • Example: iostat -x 2 5
    • Displays extended stats (e.g., await (avg wait time), aqu-sz (average queue size), %util (disk busy)) every 2 seconds, 5 times.
  • Output Interpretation:
    • High await/%util → disk is slow or saturated (e.g., a failing HDD).
    • Note: The svctm (service time) column has been deprecated and removed in sysstat 12.5.5+ and RHEL 9+. Use await and %util instead for latency analysis.

3.3 iostat -p ALL – Per-Partition I/O#

  • Purpose: Analyze individual partitions (e.g., /dev/sda1, /dev/sdb2).
  • Syntax: iostat -p ALL [interval] [count]
  • Example: iostat -p ALL 1 3
    • Displays I/O stats for all partitions (not just disks) every 1 second, 3 times.
  • Use: Troubleshoot partition-specific issues (e.g., a full partition causing I/O bottlenecks).

3.4 iostat -z – Exclude Idle Devices#

  • Purpose: Omit idle disks/partitions (reduces clutter in output).
  • Syntax: iostat -z [interval] [count]
  • Example: iostat -z 1 5
    • Displays stats only for devices with active I/O.
  • Use: Focus on busy disks (e.g., in a system with many unused disks).

3.5 iostat -t – Include Timestamp#

  • Purpose: Add a timestamp to each output line (e.g., correlate I/O with time).
  • Syntax: iostat -t [interval] [count]
  • Example: iostat -t 2 5
    • Displays disk stats with timestamps (e.g., 21:45:01).
  • Use: Analyze I/O during peak hours (e.g., database backups at 2 AM).

4. sar Commands (System Activity Reporter)#

sar collects historical and real-time stats for CPU, memory, network, load, and more.

4.1 sar -u – Historical/Real-Time CPU Utilization#

  • Purpose: Track CPU usage (real-time or historical).
  • Syntax:
    • Real-time: sar -u [interval] [count]
    • Historical: sar -u -f /var/log/sa/saXX
  • Example:
    • Real-time: sar -u 1 5 (CPU stats every 1 second, 5 times).
      • Historical: sar -u -f /var/log/sa/sa15 (CPU stats from the 15th day).
  • Output Interpretation:
    • Columns: %user, %system, %iowait, %idle (similar to mpstat).

4.2 sar -r – Memory Utilization#

  • Purpose: Analyze RAM usage (free, used, buffers, cache).
  • Syntax:
    • Real-time: sar -r [interval] [count]
    • Historical: sar -r -f /var/log/sa/saXX
  • Example: sar -r 2 5
    • Displays memory stats (e.g., kbmemfree (free RAM), %memused (percent used)) every 2 seconds, 5 times.
  • Output Interpretation:
    • Low kbmemfree + high %memused → memory pressure (e.g., swap usage may follow).

4.3 sar -b – I/O and Transfer Rates#

  • Purpose: Monitor disk I/O operations (reads, writes) and transfer rates.
  • Syntax:
    • Real-time: sar -b [interval] [count]
    • Historical: sar -b -f /var/log/sa/saXX
  • Example: sar -b 1 10
    • Displays rtps (read operations per second), wtps (write ops), bread/s (read throughput), bwrtn/s (write throughput) every 1 second, 10 times.
  • Use: Track disk I/O throughput (e.g., during a large file copy).

4.4 sar -n DEV – Network Interface Stats#

  • Purpose: Analyze network interface traffic (bytes, packets).
  • Syntax:
    • Real-time: sar -n DEV [interval] [count]
    • Historical: sar -n DEV -f /var/log/sa/saXX
  • Example: sar -n DEV 1 5
    • Displays stats for each interface (e.g., eth0, lo) every 1 second, 5 times.
  • Output Interpretation:
    • Columns: rxkB/s (received bytes/s), txkB/s (transmitted bytes/s), rxpck/s (received packets/s).
    • Use: Identify network bottlenecks (e.g., a saturated eth0).

4.5 sar -q – System Load Averages#

  • Purpose: Monitor system load averages (1, 5, 15 minutes) and runnable processes.
  • Syntax:
    • Real-time: sar -q [interval] [count]
    • Historical: sar -q -f /var/log/sa/saXX
  • Example: sar -q 1 5
    • Displays runq-sz (runnable processes), plist-sz (process list size), and load averages every 1 second, 5 times.
  • Output Interpretation:
    • High runq-sz + load averages → system is overloaded (e.g., too many concurrent tasks).

Conclusion#

Sysstat’s utilities (mpstat, pidstat, iostat, sar) provide granular, actionable insights into Linux system performance. The 20 commands above cover:

  • CPU/core utilization (e.g., mpstat -P ALL).
  • Process-level metrics (e.g., pidstat -d for I/O-bound processes).
  • Disk/partition I/O (e.g., iostat -x for latency).
  • System-wide trends (e.g., sar -r for memory, sar -n DEV for network).

Modern Sysstat Features (v12.7.7+)#

Recent sysstat versions include useful enhancements:

  • JSON output: All commands support -o JSON for machine-readable output, useful for integration with monitoring systems like Prometheus or Grafana.
  • Human-readable format: Use --human to display sizes in KB, MB, GB instead of raw bytes.
  • Color output: Statistics are displayed in color by default when connected to a terminal (configurable via S_COLORS environment variable).

By mastering these commands, you can proactively identify bottlenecks, optimize resources, and ensure system stability. Experiment with them to tailor monitoring to your environment!

References#