Practical Linux Command Line Troubleshooting for System Administrators

For system administrators, the Linux command line is both a Swiss Army knife and a diagnostic toolkit. While graphical tools have their place, the command line offers unparalleled speed, precision, and depth when troubleshooting system issues—from minor performance glitches to critical service outages. This blog aims to demystify practical Linux CLI troubleshooting, equipping you with a systematic approach, essential tools, and best practices to resolve problems efficiently. Whether you’re debugging a misbehaving service, diagnosing resource bottlenecks, or untangling network issues, mastering these skills will transform you into a more confident and effective administrator.

Table of Contents

1. Fundamentals of Linux CLI Troubleshooting
   • 1.1 The Troubleshooting Mindset
   • 1.2 Key Tools for Initial System Assessment
   • 1.3 Log Analysis: The First Step to Diagnosing Issues
2. Troubleshooting Common System Issues
   • 2.1 High CPU Usage
   • 2.2 Memory (RAM) Shortages
   • 2.3 Disk Space and I/O Bottlenecks
   • 2.4 Network Connectivity and Performance
   • 2.5 Service and Process Failures
3. Advanced Troubleshooting Techniques
   • 3.1 Debugging with strace and ltrace
   • 3.2 Analyzing System Calls with perf
   • 3.3 Kernel Logs and dmesg
4. Best Practices for Efficient Troubleshooting
   • 4.1 Documentation and Note-Taking
   • 4.2 Scripting for Repetitive Tasks
   • 4.3 Safety First: Avoiding Destructive Commands
   • 4.4 Version Control for Configuration Files
5. Conclusion
6. References

1. Fundamentals of Linux CLI Troubleshooting

1.1 The Troubleshooting Mindset

Effective troubleshooting begins with a structured approach:

1. Identify Symptoms: Gather user reports or system alerts (e.g., “server is slow” or “API requests timing out”).
2. Gather Data: Use CLI tools to collect metrics, logs, and process information.
3. Hypothesize Causes: Narrow down potential issues (e.g., “high CPU usage” or “disk I/O saturation”).
4. Test Hypotheses: Validate theories with targeted commands (e.g., check top for CPU hogs).
5. Resolve and Verify: Apply fixes and confirm resolution (e.g., restart a service and monitor metrics).

Avoid “shotgun debugging” (randomly changing settings). Instead, iterate methodically to minimize downtime.

1.2 Key Tools for Initial System Assessment

Before diving into specific issues, start with these commands to understand the system’s state:

Example Workflow:
Run uptime to check load averages. If high (> number of CPU cores), use htop to identify resource-heavy processes.

1.3 Log Analysis: The First Step to Diagnosing Issues

Logs are critical for understanding past events. Key log directories and tools:

• /var/log/: Central log directory (e.g., syslog, auth.log (Debian) or secure (RHEL), nginx/access.log).
• journalctl: Query systemd logs (replaces traditional syslog on systemd-based distros like Ubuntu 16.04+ and RHEL 7+).

Essential Log Commands:

# Tail a log in real-time (e.g., Apache errors)
tail -f /var/log/apache2/error.log

# Search for "error" in authentication logs (Debian/Ubuntu)
grep -i "error" /var/log/auth.log

# View logs for the nginx service (systemd)
journalctl -u nginx.service

# Filter logs from the last hour
journalctl --since "1 hour ago" --until "now"

Tip: Use journalctl -p err to show only error-level logs for quick issue spotting.

2. Troubleshooting Common System Issues

2.1 High CPU Usage

Symptoms: Slow response, high load averages, processes stuck in “R” (running) state.

Diagnosis Steps:

1. Identify top CPU-consuming processes:

   # List top 5 CPU-heavy processes (descending order)
   ps aux --sort=-%cpu | head -6  # Skip header with head -6

   Output might show a misbehaving python3 script using 90% CPU.

2. Drill into per-thread CPU usage with htop:
   Press H in htop to toggle threads. Look for threads within a process consuming excessive CPU.

3. Check for CPU core saturation with mpstat (part of sysstat package):

   mpstat -P ALL 2  # Monitor all cores every 2 seconds

   A core with 100% %usr or %sys indicates saturation.

Resolution:

• If the process is non-essential, kill it with kill -9 <PID>.
• If essential, optimize the process (e.g., add caching to a script) or scale resources (e.g., upgrade CPU).

2.2 Memory (RAM) Shortages

Symptoms: OOM (Out-of-Memory) killer terminating processes, free -h showing low available RAM, swap thrashing.

Diagnosis Steps:

1. Check memory usage with free -h:

   free -h

   Output:

                 total        used        free      shared  buff/cache   available
   Mem:           15Gi       12Gi       512Mi       256Mi       2.5Gi       2.1Gi
   Swap:          4.0Gi       3.8Gi       200Mi

   Low available RAM (<10% of total) and high swap usage indicate memory pressure.

2. Identify memory-hungry processes:

   ps aux --sort=-%mem | head -6  # Sort by memory usage

3. Check for OOM kills in logs:

   dmesg | grep -i "out of memory"  # Kernel logs OOM events

Resolution:

• Terminate non-critical processes.
• Add swap space temporarily (e.g., sudo fallocate -l 2G /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile).
• Upgrade RAM or optimize memory leaks in applications.

2.3 Disk Space and I/O Bottlenecks

Symptoms: “No space left on device” errors, slow file operations, high iowait in top.

Diagnosis Steps:

1. Check disk usage with df -h:

   df -h  # Show mounted filesystems and usage

   A filesystem at 100% usage needs immediate attention.

2. Find large files/directories with du:

   # List sizes of directories in /var (human-readable, summarize)
   du -sh /var/* | sort -rh | head -10

   Often, /var/log/ or /var/lib/docker/ grows unchecked.

3. Check for deleted but open files (still consuming space):

   lsof | grep deleted  # Files marked as deleted but held open by processes

   Restart the process to free space (e.g., sudo systemctl restart nginx).

4. Monitor disk I/O with iostat (sysstat package):

   iostat -x 2  # Check I/O stats every 2 seconds

   High %iowait (>20%) indicates slow storage (e.g., a failing HDD).

Resolution:

• Delete large, unnecessary files (e.g., old logs with truncate -s 0 /var/log/syslog).
• Move data to a larger disk or enable log rotation (e.g., logrotate).

2.4 Network Connectivity and Performance

Symptoms: Timeouts, slow transfers, “connection refused” errors.

Diagnosis Steps:

1. Test basic connectivity with ping and traceroute:

   ping google.com  # Check DNS and ICMP reachability
   traceroute google.com  # Identify hop-by-hop latency issues

2. Check open ports and listening services:

   # List all TCP/UDP ports with processes (requires root)
   ss -tulpn  # t: TCP, u: UDP, l: listening, p: process, n: numeric

   Look for LISTEN state on expected ports (e.g., 80 for HTTP).

3. Check firewall rules with ufw (Debian/Ubuntu) or firewalld (RHEL):

   sudo ufw status  # Debian/Ubuntu
   sudo firewall-cmd --list-all  # RHEL/CentOS

   Ensure required ports (e.g., 443) are allowed.

4. Analyze traffic with tcpdump (advanced):

   sudo tcpdump -i eth0 port 80 -c 10  # Capture 10 HTTP packets on eth0

Resolution:

• Open blocked ports with sudo ufw allow 443/tcp.
• Fix DNS issues by editing /etc/resolv.conf or restarting systemd-resolved.

2.5 Service and Process Failures

Symptoms: Services not starting, systemctl status <service> shows “failed”.

Diagnosis Steps:

1. Check service status with systemctl:

   systemctl status nginx.service

   Output might show “Failed to start A high performance web server…“.

2. View service logs with journalctl:

   journalctl -u nginx.service --no-pager  # Show full logs for nginx

   Look for errors like “invalid PID file” or “configuration syntax error”.

3. Validate configuration files (critical for services like nginx, apache, or sshd):

   nginx -t  # Test nginx config syntax
   apachectl configtest  # Test Apache config

Resolution:

• Fix syntax errors in config files (e.g., missing ; in nginx.conf).
• Restore a working config from backup or version control.

3. Advanced Troubleshooting Techniques

3.1 Debugging with strace and ltrace

For processes behaving unexpectedly, trace system calls (strace) or library calls (ltrace):

• strace: Trace system calls (e.g., open(), read(), write()).

  # Trace a running process (PID=1234)
  strace -p 1234
  
  # Trace a command and log to a file
  strace -o curl_trace.txt curl https://example.com

  Look for ENOENT (file not found) or EACCES (permission denied) errors.

• ltrace: Trace dynamic library calls (e.g., printf(), malloc()).

  ltrace -p 1234  # Trace library calls of PID 1234

3.2 Analyzing System Calls with perf

The perf tool (part of linux-tools-common) profiles CPU usage at the kernel level:

# Monitor real-time CPU usage by function
perf top

# Record CPU events for a process, then generate a report
perf record -g -p <PID>  # -g: capture call graphs
perf report  # Analyze the recorded data

Use perf to identify bottlenecks in custom applications or kernel modules.

3.3 Kernel Logs and dmesg

The kernel ring buffer (dmesg) logs low-level events (hardware issues, driver failures):

dmesg | grep -i error  # Show kernel errors
dmesg -T  # Show human-readable timestamps (e.g., "2023-10-01 14:30:00")

Look for messages like “USB device not recognized” or “ata1: failed to read log page” (disk issues).

4. Best Practices for Efficient Troubleshooting

4.1 Documentation and Note-Taking

Always document:

• Symptoms, timestamps, and initial metrics (e.g., “14:00: Load avg 5.0, CPU 90%”).
• Commands run and their outputs (use script to log an entire session: script troubleshooting-20231001.log).
• Fixes applied and their outcomes (e.g., “Restarted nginx; CPU dropped to 15%“).

4.2 Scripting for Repetitive Tasks

Automate routine checks with bash scripts. Example: Disk space alert script:

#!/bin/bash
THRESHOLD=90
df -h | awk -v threshold="$THRESHOLD" '$5+0 > threshold {print "ALERT: " $0}'

Save as disk_alert.sh, make executable (chmod +x), and schedule with cron to run hourly.

4.3 Safety First: Avoiding Destructive Commands

• Test commands in a sandbox before running on production (e.g., use echo rm -rf /tmp/* to preview deletions).
• Avoid rm -rf as root—use rm -i (interactive) or move files to /tmp first.
• Use sudo sparingly: Run non-destructive commands as a regular user.

4.4 Version Control for Configuration Files

Track changes to /etc/ with git to revert mistakes:

cd /etc && sudo git init && sudo git add . && sudo git commit -m "Initial commit"

Commit changes before editing: sudo git commit -am "Update nginx.conf for SSL".

5. Conclusion

Linux CLI troubleshooting is a blend of art and science. By adopting a systematic mindset, mastering core tools like ps, top, and journalctl, and following best practices (documentation, scripting, safety), you can resolve issues faster and minimize downtime. Remember: the command line is your most powerful ally—invest time in learning its nuances, and you’ll become a more effective system administrator.

6. References

• Linux man pages (official documentation for commands).
• The Linux Command Line by William Shotts (free online book).
• Red Hat System Administrator’s Guide
• Ubuntu Server Troubleshooting Guide
• perf Wiki (kernel profiling with perf).