In the world of Linux systems—whether you’re managing servers, debugging application bottlenecks, or optimizing resource usage—understanding system performance is critical. While graphical tools exist, the Linux command line (CLI) offers lightweight, scriptable, and universally available utilities that provide deep insights into system behavior. This blog will guide you through the fundamentals of Linux performance analysis, essential CLI tools, common workflows, and best practices to diagnose and resolve performance issues efficiently.
Table of Contents
- Key Performance Metrics
- Essential CLI Tools for Performance Analysis
- Common Analysis Workflows
- Best Practices
- Conclusion
- References
Key Performance Metrics
Before diving into tools, it’s critical to understand the core metrics that define system performance. These metrics help you identify bottlenecks (CPU, memory, disk, or network) and prioritize troubleshooting:
| Metric Category | Key Metrics | What They Indicate |
|---|---|---|
| CPU | Utilization (user/system/idle), load average, context switches, per-CPU usage | How busy the CPU is, waiting processes, and thread/process scheduling efficiency. |
| Memory | Used/free/available memory, swap usage, page faults (major/minor), slab usage | Physical memory exhaustion, swap thrashing, or inefficient memory allocation. |
| Disk I/O | Throughput (read/write), latency (await), queue length (aqu-sz), I/O per process | Slow disk response, I/O contention, or misconfigured storage (e.g., spinning vs. SSD). |
| Network | Bandwidth (RX/TX), packet loss, TCP connections, socket states | Network saturation, misconfigured firewalls, or application-level network inefficiencies. |
Essential CLI Tools for Performance Analysis
Linux offers a rich ecosystem of CLI tools to monitor these metrics. Below are the most critical utilities, with usage examples and explanations.
CPU Monitoring
1. uptime – Quick System Health Check
uptime provides a high-level snapshot of system load and uptime, making it the first tool to run when diagnosing issues.
Usage:
uptime Sample Output:
14:32:45 up 10 days, 2:15, 1 user, load average: 0.85, 0.92, 0.78 Explanation:
load average: 0.85, 0.92, 0.78: Average number of processes in the runqueue (waiting for CPU) or uninterruptible sleep (waiting for I/O) over 1, 5, and 15 minutes. A load average > number of CPU cores indicates potential saturation.
2. top – Real-Time Process Monitor
top displays real-time CPU, memory, and process data, sorted by resource usage. It’s ideal for identifying resource-hungry processes.
Basic Usage:
top Key Options:
- Press
Pto sort processes by CPU usage. - Press
Mto sort by memory usage. top -b -n 1: Batch mode (non-interactive, useful for scripting).
Sample Output Excerpt:
top - 14:35:22 up 10 days, 2:18, 1 user, load average: 0.89, 0.93, 0.79
Tasks: 187 total, 1 running, 186 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.2 us, 2.1 sy, 0.0 ni, 92.5 id, 0.0 wa, 0.0 hi, 0.2 si, 0.0 st
MiB Mem : 31997.7 total, 18234.3 free, 5123.4 used, 8640.0 buff/cache
MiB Swap: 2048.0 total, 2048.0 free, 0.0 used. 25967.3 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12345 appuser 20 0 2250240 456780 89232 R 35.2 1.4 5:23.10 java Key Fields:
%CPU: CPU usage by the process (e.g.,35.2= 35.2% of a core).%MEM: Memory usage (resident set size,RES).TIME+: Total CPU time consumed by the process.
3. htop – Enhanced top Alternative
htop improves on top with a color-coded interface, mouse support, and easier process management (e.g., killing processes with F9).
Installation (if missing):
sudo apt install htop # Debian/Ubuntu
sudo dnf install htop # RHEL/CentOS Usage:
htop Key Features:
- Per-CPU utilization graph (top of the screen).
- Sort processes by CPU/memory with
F6.
4. mpstat – Per-CPU and Overall CPU Statistics
mpstat (part of the sysstat package) provides detailed CPU usage breakdowns, including per-core metrics—critical for identifying uneven CPU utilization.
Installation:
sudo apt install sysstat # Debian/Ubuntu
sudo dnf install sysstat # RHEL/CentOS Usage:
mpstat 1 5 # Report every 1 second, 5 times total Sample Output:
Linux 5.15.0-78-generic (server-01) 09/20/2023 _x86_64_ (8 CPU)
14:40:01 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
14:40:02 all 4.88 0.00 1.88 0.00 0.00 0.12 0.00 0.00 0.00 93.12
14:40:02 0 6.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 92.00
14:40:02 1 3.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 96.00
... Explanation:
%usr: User-space CPU usage (e.g., applications).%sys: Kernel-space CPU usage (e.g., system calls).%iowait: Time CPU spends waiting for I/O (high values indicate disk/network bottlenecks).
5. vmstat – Virtual Memory and System Activity
vmstat reports on processes, memory, paging, block I/O, traps, and CPU activity. It’s useful for spotting trends like excessive context switches or page faults.
Usage:
vmstat 2 5 # Report every 2 seconds, 5 times Sample Output:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 18234360 123456 8923456 0 0 5 23 123 456 5 2 92 0 0
0 0 0 18234210 123456 8923456 0 0 0 18 118 432 4 1 95 0 0 Key Fields:
r: Number of runnable processes (high values = CPU saturation).cs: Context switches per second (high values may indicate excessive multitasking).us/sy: User/kernel CPU time (same asmpstat).
Memory Usage
1. free – Memory Overview
free displays total, used, free, and available memory (including swap).
Usage:
free -h # Human-readable units (GB, MB) Sample Output:
total used free shared buff/cache available
Mem: 31Gi 5.0Gi 17Gi 234Mi 8.4Gi 25Gi
Swap: 2.0Gi 0B 2.0Gi Explanation:
available: Estimated memory available for new processes (accounts for buffers/cache, which can be freed).buff/cache: Memory used for disk buffers and page cache (improves I/O performance).
2. slabtop – Kernel Slab Allocation
Linux kernels use “slabs” to manage memory for internal objects (e.g., inodes, file descriptors). slabtop helps diagnose kernel memory leaks.
Usage:
slabtop -o # One-shot mode (non-interactive) Sample Output Excerpt:
Active / Total Objects (% used) : 123456 / 156789 (78.7%)
Active / Total Slabs (% used) : 5678 / 6789 (83.6%)
Active / Total Caches (% used) : 123 / 156 (78.9%)
Active / Total Size (% used) : 456.7MiB / 567.8MiB (79.9%)
Minimum / Average / Maximum Object : 0.01KiB / 3.45KiB / 128.00KiB
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
89234 78210 87% 1.00KiB 2789 32 89.25MiB ext4_inode_cache
56789 50123 88% 0.50KiB 1775 32 28.40MiB dentry_cache Disk I/O
1. iostat – Disk Input/Output Statistics
iostat (part of sysstat) reports on CPU and disk I/O utilization, including throughput, latency, and queue length.
Usage:
iostat -x 2 5 # Extended stats, every 2s, 5 times Sample Output:
avg-cpu: %user %nice %system %iowait %steal %idle
4.88 0.00 1.88 0.00 0.00 93.24
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
sda 0.50 2.30 20.00 115.00 0.00 0.10 0.00 4.17 0.80 2.10 0.01 40.00 50.00 0.50 0.14 Key Fields:
%util: Percentage of time the disk is busy (values >70% indicate saturation).await: Average time (ms) for I/O requests to complete (includes queueing + service time).aqu-sz: Average number of pending I/O requests (high values = I/O contention).
2. iotop – Per-Process Disk I/O
iotop identifies processes causing high disk I/O, similar to how top identifies CPU hogs.
Usage:
sudo iotop # Requires root Sample Output Excerpt:
Total DISK READ: 0.00 B/s | Total DISK WRITE: 123.45 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
1234 be/4 appuser 0.00 B/s 98.76 K/s 0.00 % 5.43 % java -jar app.jar Network Activity
1. iftop – Bandwidth Usage by Interface
iftop monitors real-time network bandwidth usage per interface, showing traffic by source/destination IP.
Installation:
sudo apt install iftop # Debian/Ubuntu Usage:
sudo iftop -i eth0 # Monitor interface eth0 Sample Output Excerpt:
192.168.1.100 => 192.168.1.200 123Kb 145Kb 112Kb
<= 45Kb 32Kb 28Kb
192.168.1.100 => 203.0.113.5 89Kb 76Kb 65Kb
<= 12Kb 10Kb 9Kb
-------------------------------------------------------------------------------------------------
TX: cum: 1.23GB peak: 234Kb/s rates: 212Kb 221Kb 177Kb
RX: 456MB 89Kb/s 57Kb 42Kb 37Kb
TOTAL: 1.68GB 323Kb/s 269Kb 263Kb 214Kb 2. ss – Socket Statistics
ss replaces netstat (deprecated) and shows active sockets, listening ports, and process IDs (PIDs) using them.
Usage:
ss -tulpn # Show TCP/UDP listening ports with PIDs Sample Output:
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=5678,fd=3))
LISTEN 0 128 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=5678,fd=4)) Common Analysis Workflows
Performance analysis is rarely a single-tool task. Below is a step-by-step workflow to diagnose bottlenecks:
Step 1: Assess Overall Health
Start with high-level tools to identify anomalies:
uptime # Check load average
top # Identify top CPU/memory consumers
free -h # Check memory availability
iostat -x 1 # Check disk I/O Step 2: Drill Into CPU Issues
If load average is high or top shows CPU saturation:
- Use
mpstat -P ALL 1to check for per-CPU imbalance (e.g., one core at 100% while others are idle). - Use
vmstat 1to checkcs(context switches) andus/sy(user vs. kernel CPU). Highsymay indicate excessive system calls. - Use
pidstat -u 1(part ofsysstat) to get per-process CPU usage over time