dotlinux blog

Sysstat – All-in-One System Performance and Usage Activity Monitoring Tool For Linux

Sysstat is a comprehensive, open-source toolkit for monitoring Linux system performance, resource utilization, and activity over time. It empowers system administrators, DevOps engineers, and developers to:

  • Troubleshoot performance bottlenecks (e.g., CPU, disk, memory, or network issues).
  • Analyze historical system behavior for capacity planning.
  • Generate real-time or historical reports on system activity.

At its core, Sysstat includes utilities like sar (System Activity Reporter), iostat (I/O Statistics), mpstat (Multi-Processor Statistics), pidstat (Process-Level Statistics), and sadf (System Activity Data Formatter). These tools work together to provide granular insights into system health and resource usage.

2026-04

Table of Contents#

  1. Introduction to Sysstat
  2. Installation on Major Linux Distributions
  3. Key Components of Sysstat
  4. Using sar for Comprehensive Monitoring
  5. Analyzing Storage with iostat
  6. Multi-Core Insights with mpstat
  7. Process-Level Troubleshooting with pidstat
  8. Exporting Data with sadf
  9. Sysstat Configuration and Data Management
  10. Real-World Use Cases
  11. Advanced Tips and Tricks
  12. Conclusion
  13. References

Installation on Major Linux Distributions#

Sysstat is available in the official repositories of most Linux distributions. Install it using the package manager for your system:

Debian/Ubuntu:#

sudo apt update && sudo apt install sysstat

RHEL/CentOS (7+/8+):#

sudo yum install sysstat  # For RHEL/CentOS 7
sudo dnf install sysstat  # For RHEL/CentOS 8+

Fedora:#

sudo dnf install sysstat

Arch Linux:#

sudo pacman -S sysstat

After installation, enable the sysstat service to start collecting system activity data:

sudo systemctl enable --now sysstat

Key Components of Sysstat#

Sysstat bundles five core utilities, each serving a specific monitoring purpose:

sar: System Activity Reporter#

sar (System Activity Reporter) is the backbone of Sysstat. It collects and reports on:

  • CPU, memory, disk, and network usage.
  • System load, swap space, and I/O statistics.
  • Historical data (stored in /var/log/sa/ by default).

Example: Real-time CPU monitoring#

To view CPU usage every 1 second for 5 intervals:

sar -u 1 5

iostat: Input/Output Statistics#

iostat focuses on disk I/O performance and CPU utilization related to I/O operations. It helps identify slow disks, I/O bottlenecks, or misconfigured storage.

Example: Extended disk I/O stats (every 1 second, 5 times)#

iostat -x 1 5

mpstat: Multi-Processor Statistics#

mpstat reports per-CPU or per-core statistics, ideal for multi-CPU/multi-core systems. It reveals imbalanced load across processors.

Example: Monitor all CPU cores (every 1 second, 5 times)#

mpstat -P ALL 1 5

pidstat: Process-Level Statistics#

pidstat monitors individual processes, showing CPU, memory, I/O, and other metrics per process. Use it to identify resource-hungry applications.

Example: Monitor all processes (every 1 second, 5 times)#

pidstat 1 5

sadf: System Activity Data Formatter#

sadf formats and exports sar data into various formats (CSV, JSON, XML) for analysis in external tools (e.g., Excel, Grafana, Python scripts).

Example: Export CPU data to CSV#

sadf -o csv /var/log/sa/sa$(date +%d) -- -u

Using sar for Comprehensive Monitoring#

sar is the most versatile Sysstat tool. Let’s explore its key use cases:

CPU Utilization#

To analyze CPU usage (user, system, idle, and wait times):

sar -u 2 10  # 2-second intervals, 10 times

For per-CPU core stats (useful in multi-core systems):

sar -P ALL 2 10

Memory Usage#

Monitor memory (RAM) and swap space:

sar -r 2 5  # 2-second intervals, 5 times

This reports:

  • kbmemfree: Free memory.
  • kbmemused: Used memory.
  • %memused: Memory utilization percentage.
  • kbswpfree/kbswpused: Swap space stats.

Disk I/O#

Track disk read/write rates, I/O operations, and transfer times:

sar -b 1 5  # 1-second intervals, 5 times

For per-disk I/O stats:

sar -d 1 5

Network Statistics#

Monitor network interface activity (bytes sent/received, errors, collisions):

sar -n DEV 1 5  # "DEV" = network devices

Analyzing Storage with iostat#

iostat is critical for diagnosing storage bottlenecks. Key metrics:

  • %util: Percentage of time the disk was busy (high values = bottleneck).
  • await: Average time (ms) for I/O requests to complete (high = slow disk).
  • r/s/w/s: Read/write requests per second.

Example: Identify slow disks#

Run:

iostat -x 1 10

Look for disks with:

  • High %util (e.g., > 80%).
  • High await (e.g., > 100 ms).

Multi-Core Insights with mpstat#

In multi-CPU systems, mpstat reveals load imbalances. For example:

  • If one core is at 100% usage while others are idle, your application may not be optimized for multi-threading.

Example: Check core-specific load#

mpstat -P ALL 1 5

Look for discrepancies in %usr (user CPU), %sys (system CPU), or %idle (idle CPU) across cores.

Process-Level Troubleshooting with pidstat#

pidstat helps isolate processes consuming excessive resources.

Example: Monitor a specific process (e.g., PID 1234)#

pidstat -p 1234 1 5

Example: Find top CPU-consuming processes#

Combine pidstat with sort and head:

pidstat 1 5 | sort -k 10 -r | head -5  # Sort by CPU% (column 10)

Exporting Data with sadf#

sadf converts sar’s binary logs into human-readable or machine-friendly formats (e.g., CSV, JSON).

Example: Export 7-day historical CPU data to CSV#

sadf -o csv -- -u -s 00:00:00 -e 23:59:59 /var/log/sa/sa??

Example: Export to JSON for Grafana/Prometheus#

sadf -o json /var/log/sa/sa$(date +%d) -- -u

Sysstat Configuration and Data Management#

Customize Sysstat to fit your monitoring needs:

Configuring Data Collection#

Edit /etc/sysstat/sysstat (or /etc/default/sysstat on some systems) to:

  • Set data collection interval (INTERVAL).
  • Adjust data retention (HISTORY = number of days to keep logs).

Example: Collect data every 10 minutes#

In /etc/sysstat/sysstat:

INTERVAL=600  # 10 minutes (in seconds)
HISTORY=28    # Keep data for 28 days

Managing Data Retention#

Sysstat stores daily reports in /var/log/sa/ (e.g., sa01, sa02, ..., sa31). To clear old data:

sudo find /var/log/sa/ -mtime +30 -delete  # Delete files older than 30 days

Service Management#

Control the sysstat service:

  • Start: sudo systemctl start sysstat
  • Stop: sudo systemctl stop sysstat
  • Status: sudo systemctl status sysstat

Real-World Use Cases#

Sysstat solves common system administration challenges:

Performance Troubleshooting#

  • High CPU: Use sar -u and pidstat to identify processes.
  • Slow Disk: Use iostat -x to check %util and await.
  • Memory Leak: Use sar -r to track memory usage over time.

Capacity Planning#

Analyze historical sar data to:

  • Predict when memory/CPU will hit capacity.
  • Justify hardware upgrades (e.g., “CPU usage averages 90% during peak hours”).

Long-Term Monitoring#

Review daily/weekly sar logs (in /var/log/sa/) to:

  • Identify seasonal trends (e.g., higher load during business hours).
  • Audit system health (e.g., “Disk I/O increased after new application deployment”).

Advanced Tips and Tricks#

Automating Reports with Cron#

Schedule sar reports (e.g., daily CPU summary) via cron:

  1. Edit cron (as root):

    sudo crontab -e

  2. Add a job to generate a daily CPU report at 23:59:

    59 23 * * * sar -u -f /var/log/sa/sa$(date +\%d -d "yesterday") > /var/log/sar_daily_cpu_$(date +\%Y\%m\%d -d "yesterday").txt

Integrating with Scripts/Monitoring Tools#

Use sadf to export data for visualization:

  • Grafana: Export sar data to InfluxDB (via a script) and build dashboards.
  • Python: Parse sadf’s CSV output to generate custom reports.

Example: Python script to plot CPU usage (using matplotlib)#

import pandas as pd
import matplotlib.pyplot as plt
 
# Read sadf CSV output
df = pd.read_csv('sar_cpu.csv', skiprows=2)  # Adjust skiprows
 
# Plot %idle over time
df.plot(x='timestamp', y='%idle', title='CPU Idle Time')
plt.show()

Conclusion#

Sysstat is an essential toolkit for Linux system monitoring. Its utilities (sar, iostat, mpstat, pidstat, sadf) provide deep insights into system performance, enabling:

  • Proactive troubleshooting.
  • Data-driven capacity planning.
  • Long-term system health tracking.

By mastering Sysstat, you gain the ability to optimize resources, resolve bottlenecks, and ensure your Linux systems run efficiently.

References#