dotlinux guide

Debugging Linux Systems with Command Line Utilities

Linux systems power everything from embedded devices to enterprise servers, but even the most robust systems encounter issues—crashes, slowdowns, network failures, or misbehaving applications. Debugging these issues requires a systematic approach, and command-line utilities are the cornerstone of this process. Unlike graphical tools, command-line utilities are lightweight, scriptable, and available on even the most minimal Linux installations, making them indispensable for system administrators, developers, and DevOps engineers. This blog explores the fundamental concepts, essential tools, common workflows, and best practices for debugging Linux systems using command-line utilities. By the end, you’ll have a structured toolkit to diagnose and resolve issues efficiently.

Table of Contents

  1. Fundamental Concepts of Linux System Debugging
  2. Essential Command-Line Debugging Utilities
  3. Common Debugging Workflows
  4. Best Practices for Effective Debugging
  5. Conclusion
  6. References

Fundamental Concepts of Linux System Debugging

What is System Debugging?

System debugging is the process of identifying, isolating, and resolving issues in a Linux system. Issues may include:

  • Application crashes or hangs
  • High CPU/memory/disk usage
  • Network connectivity problems
  • File system corruption
  • Service failures (e.g., nginx, ssh)

Debugging involves gathering data (logs, metrics, traces), analyzing patterns, and testing hypotheses to pinpoint the root cause.

Why Command-Line Utilities?

Command-line tools offer unique advantages for Linux debugging:

  • Ubiquity: Preinstalled on nearly all Linux distributions (no extra dependencies).
  • Efficiency: Lightweight and fast, even on resource-constrained systems (e.g., embedded devices).
  • Scriptability: Automate repetitive tasks with bash/Python scripts.
  • Depth: Access low-level system details (e.g., kernel logs, syscalls) unavailable to graphical tools.

Essential Command-Line Debugging Utilities

Process Management & Tracing

Processes are the building blocks of Linux systems. Tools in this category help monitor, analyze, and trace misbehaving processes.

ps – List Running Processes

ps (process status) displays active processes. Use it to identify PID (Process ID), CPU/memory usage, and parent-child relationships.

Examples:

  • List all processes (BSD style): 
    ps aux  # a: all users, u: user details, x: include daemons
  • Filter by process name (e.g., nginx): 
    ps aux | grep nginx

top/htop – Real-Time Process Monitoring

top provides a dynamic view of system resource usage (CPU, memory, load). htop (an enhanced version) adds color, mouse support, and easier navigation.

Examples:

  • Launch top and sort by CPU usage: 
    top -o %CPU  # Press 'q' to exit
  • Launch htop (install with sudo apt install htop on Debian/Ubuntu): 
    htop

strace – Trace System Calls

strace intercepts and logs syscalls (e.g., open(), read(), connect()) made by a process. Use it to diagnose issues like missing files, permission errors, or network timeouts.

Examples:

  • Trace syscalls of a failing application: 
    strace ./myapp 2>&1 | grep -i "error\|fail"  # 2>&1 redirects stderr to stdout
  • Count syscall frequency (to identify bottlenecks): 
    strace -c ./myapp  # -c: summary of syscall counts and time

gdb – GNU Debugger

gdb debugs binary executables (C/C++/Rust) by setting breakpoints, inspecting memory, and analyzing crashes (via core dumps).

Example: Debug a crashed program with a core dump:

# Enable core dumps (temporarily)
ulimit -c unlimited  

# Run the program until it crashes (generates core.<PID>)
./myapp  

# Debug the core dump
gdb ./myapp core.12345  
(gdb) bt  # Print backtrace to identify crash location

System Log Analysis

Logs are critical for debugging. Linux systems store logs in /var/log/ (traditional) or via systemd-journald (modern).

journalctl – Query systemd Logs

journalctl accesses logs from systemd-journald, which aggregates logs from services, the kernel, and applications.

Examples:

  • Show logs for the nginx service (last hour, errors only): 
    journalctl -u nginx.service --since "1 hour ago" --priority=err
  • Follow live logs (like tail -f): 
    journalctl -f  # Press Ctrl+C to exit

dmesg – Kernel Ring Buffer

dmesg prints the kernel’s ring buffer, which contains boot messages, hardware errors (e.g., disk failures), and driver issues.

Example: Check for disk I/O errors:

dmesg | grep -i "io error\|sda"  # sda: first SATA disk

Network Debugging

Network issues (e.g., timeouts, packet loss) require tools to inspect traffic, resolve DNS, and test connectivity.

ping/traceroute – Test Connectivity

  • ping: Checks if a host is reachable (ICMP echo requests). 
    ping -c 4 google.com  # -c: number of packets
  • traceroute: Maps the path packets take to a host (identifies network hops with delays). 
    traceroute google.com

ss – Socket Statistics

ss replaces netstat (deprecated) to display active network connections, ports, and protocols (TCP/UDP).

Examples:

  • List all listening TCP ports: 
    ss -ltn  # l: listening, t: TCP, n: numeric (no DNS lookup)
  • Show UDP connections for a process (e.g., PID 5678): 
    ss -uap | grep 5678  # u: UDP, a: all, p: process info

tcpdump – Capture Network Traffic

tcpdump captures raw network packets for deep analysis (e.g., malformed HTTP requests, DNS hijacking).

Example: Capture HTTP traffic on port 80:

sudo tcpdump -i eth0 port 80 -A  # -i: interface, -A: ASCII output

File System & Storage Issues

Tools to diagnose disk fullness, file locks, and corruption.

df/du – Disk Usage

  • df: Check free disk space on mounted filesystems. 
    df -h  # -h: human-readable (GB/MB)
  • du: Find large files/directories (e.g., to identify what’s filling the disk). 
    du -sh /var/log/*  # -s: summary, -h: human-readable

lsof – List Open Files

lsof identifies which processes are using a file (useful for resolving “file in use” errors when unmounting drives).

Example: Find which process is using /var/log/syslog:

lsof /var/log/syslog

fsck – File System Check

fsck (file system check) repairs corrupted filesystems (run unmounted or in single-user mode).

Example: Check and repair /dev/sda1 (ext4 filesystem):

sudo fsck -y /dev/sda1  # -y: auto-answer "yes" to repairs

Memory & Resource Usage

Tools to diagnose memory leaks, swap usage, and OOM (Out-of-Memory) errors.

free – Memory Statistics

free shows total/used/free RAM and swap space.

Example:

free -h  # -h: human-readable

vmstat – Virtual Memory Statistics

vmstat reports on memory, processes, and I/O (useful for identifying swap thrashing).

Example: Monitor memory every 2 seconds:

vmstat 2  # Refresh every 2 seconds (Ctrl+C to stop)

Performance Monitoring

Tools to analyze long-term system performance trends.

iostat – Disk I/O Statistics

iostat monitors disk read/write rates and latency (install via sysstat package).

Example: Check disk I/O for sda:

iostat -x sda 5  # -x: extended stats, 5: refresh every 5s

sar – System Activity Reporter

sar collects and displays historical performance data (CPU, memory, network) stored in /var/log/sysstat/.

Example: Show CPU usage from yesterday:

sar -u -f /var/log/sysstat/sa23  # -u: CPU, -f: file (sa23 = 23rd of month)

Advanced Debugging Tools

For complex issues (e.g., kernel-level bugs, performance bottlenecks).

perf – Performance Profiler

perf analyzes CPU usage, cache misses, and kernel events (e.g., to find why an app is slow).

Example: Profile CPU usage of myapp:

sudo perf record -g ./myapp  # -g: capture call graphs
sudo perf report  # Analyze the recorded data

bpftrace – Dynamic Tracing

bpftrace (based on eBPF) enables low-overhead tracing of kernel/user-space events (e.g., track file opens system-wide).

Example: Count file opens by process:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_open { @[comm] = count(); }'

Common Debugging Workflows

Debugging is iterative. Here’s a typical workflow for a “slow server” issue:

  1. Reproduce the Issue: Confirm the problem is consistent (e.g., “server slows down at 3 PM daily”).
  2. Check Resource Usage: Use top/htop to identify CPU/memory hogs (e.g., a misbehaving nodejs process).
  3. Inspect Logs: Run journalctl -u nodejs.service --since "1 hour ago" to look for errors.
  4. Trace the Process: Use strace -p <PID> to see if the process is stuck on a syscall (e.g., connect() to a down database).
  5. Verify Dependencies: Check if the database is reachable with ping/telnet <db-ip> 5432.
  6. Fix and Validate: Restart the database, then confirm the server speed improves with iostat/vmstat.

Best Practices for Effective Debugging

  1. Document Everything: Log steps, commands, and outputs (e.g., in a text file or ticketing system).
  2. Start with Logs: Always check journalctl, dmesg, and application logs first—they often contain direct error messages.
  3. Isolate the Problem: Use tools like systemctl isolate multi-user.target (single-user mode) to rule out conflicting services.
  4. Avoid Production Disruption: Test fixes in staging first. Use strace -p <PID> cautiously (it adds overhead).
  5. Automate Repetitive Tasks: Write scripts to parse logs or monitor resources (e.g., a bash script to alert on high disk usage).

Conclusion

Debugging Linux systems with command-line utilities is a skill that combines technical knowledge with systematic problem-solving. Tools like strace, journalctl, and tcpdump provide unparalleled visibility into system behavior, while workflows and best practices ensure efficiency.

By mastering these utilities, you’ll be able to diagnose everything from simple service failures to complex kernel-level issues. Remember: debugging is iterative—start small, validate hypotheses, and leverage the Linux ecosystem’s rich tooling.

References