dotlinux guide

RealWorld Shell Scripting Applications and Use Cases: A Comprehensive Guide

Table of Contents

Understanding Shell Scripting Basics

Before diving into real-world use cases, let’s establish foundational concepts.

What is a Shell Script?

A shell script is a text file containing a series of commands that the shell (e.g., Bash, Zsh) executes sequentially. It eliminates the need to run commands manually, enabling automation of tasks like file manipulation, system monitoring, and application deployment.

Common Shells

The choice of shell impacts script behavior. Popular shells include:

  • Bash (Bourne-Again Shell): The most widely used shell, default on Linux and macOS.
  • Zsh: Extends Bash with features like auto-completion and themes (popular in developer workflows).
  • Sh (Bourne Shell): Legacy shell, often used for minimal environments (e.g., embedded systems).

We’ll focus on Bash in examples, as it’s the de facto standard for cross-platform scripting.

Core Syntax Fundamentals

Shebang Line

Every script should start with a “shebang” (#!) to specify the interpreter:

#!/bin/bash
# This script uses Bash

Variables

Store and manipulate data with variables (no spaces around =):

NAME="Alice"
echo "Hello, $NAME!"  # Output: Hello, Alice!

Loops and Conditionals

Control flow is critical for automation:

# For loop example
for i in {1..3}; do
  echo "Iteration $i"
done

# Conditional example
FILE="data.txt"
if [ -f "$FILE" ]; then
  echo "$FILE exists."
else
  echo "$FILE not found."
fi

Real-World Applications and Use Cases

Shell scripts shine in solving practical problems. Below are key use cases with actionable examples.

1. System Administration

System admins rely on shell scripts to manage infrastructure, automate maintenance, and ensure system health.

Example 1: Automated Log Rotation

Logs grow indefinitely, consuming disk space. A script to compress old logs and delete outdated ones:

#!/bin/bash
# log_rotator.sh: Compress logs older than 7 days and delete >30 days old

LOG_DIR="/var/log/myapp"
MAX_AGE_COMPRESS=7  # Days
MAX_AGE_DELETE=30    # Days

# Compress logs older than MAX_AGE_COMPRESS days
find "$LOG_DIR" -name "*.log" -type f -mtime +"$MAX_AGE_COMPRESS" -exec gzip {} \;

# Delete compressed logs older than MAX_AGE_DELETE days
find "$LOG_DIR" -name "*.log.gz" -type f -mtime +"$MAX_AGE_DELETE" -delete

echo "Log rotation completed at $(date)" >> "$LOG_DIR/rotation.log"

Why it works: Uses find to target files by age, gzip for compression, and logs activity for auditing.

Example 2: Backup Automation with Error Handling

Automate backups to a remote server (e.g., via rsync), with checks for failures:

#!/bin/bash
# backup_data.sh: Sync local data to remote server with error handling

SOURCE="/home/user/documents"
DEST="backup-server:/backups/user"
LOG_FILE="/var/log/backup.log"

# Exit on error, unset variables, or failed pipes
set -euo pipefail

# Log start time
echo "Backup started at $(date)" >> "$LOG_FILE"

# Perform backup with rsync (verbose, archive mode)
rsync -av --delete "$SOURCE" "$DEST" >> "$LOG_FILE" 2>&1

# Log success
echo "Backup completed successfully at $(date)" >> "$LOG_FILE"

Key practices: set -euo pipefail ensures the script exits on errors/unset variables, and 2>&1 redirects errors to the log.

2. DevOps and CI/CD

Shell scripts integrate seamlessly into CI/CD pipelines, automating testing, deployment, and infrastructure provisioning.

Example 3: CI/CD Pipeline for a Python App

A script to run tests, build, and deploy a Python application if tests pass:

#!/bin/bash
# deploy_app.sh: CI/CD workflow for a Python app

APP_DIR="/opt/my-python-app"
TEST_REPORT="test_results.txt"

# Step 1: Run unit tests
echo "Running tests..."
pytest "$APP_DIR/tests" > "$TEST_REPORT" 2>&1

# Step 2: Check test results (exit code 0 = success)
if [ $? -ne 0 ]; then
  echo "Tests failed! See $TEST_REPORT"
  exit 1
fi

# Step 3: Build and deploy
echo "Building app..."
python "$APP_DIR/setup.py" build >> "$TEST_REPORT" 2>&1

echo "Deploying to production..."
systemctl restart my-app-service

echo "Deployment successful at $(date)"

Use case: Trigger this script in GitHub Actions or GitLab CI to automate deployments on code pushes.

3. Data Processing and Log Analysis

Shell scripts are ideal for parsing logs, transforming data, and generating reports—no need for Python/R for simple tasks.

Example 4: Extract Errors from Logs and Alert

Parse a web server log to find 5xx errors and send an email alert:

#!/bin/bash
# error_alert.sh: Detect 5xx errors and alert via email

LOG_FILE="/var/log/nginx/access.log"
ERROR_THRESHOLD=5  # Alert if >5 errors in 1 hour
ALERT_EMAIL="[email protected]"

# Extract 5xx errors in the last hour
ERRORS=$(grep " $(date -d '1 hour ago' +%H):" "$LOG_FILE" | grep -c " 5[0-9][0-9] ")

if [ "$ERRORS" -gt "$ERROR_THRESHOLD" ]; then
  SUBJECT="ALERT: High 5xx Errors on Web Server"
  BODY="Found $ERRORS 5xx errors in the last hour. Check $LOG_FILE."
  echo "$BODY" | mail -s "$SUBJECT" "$ALERT_EMAIL"
  echo "Alert sent to $ALERT_EMAIL"
else
  echo "No critical errors detected."
fi

Tools used: grep for filtering, date for time-based parsing, and mail for alerts.

Example 5: CSV Data Filtering

Filter a CSV file to extract rows where a column meets a condition (e.g., “sales > 1000”):

#!/bin/bash
# filter_sales.sh: Extract high-value sales from a CSV

INPUT_CSV="sales_data.csv"
OUTPUT_CSV="high_sales.csv"
MIN_SALES=1000

# CSV format: date,product,sales (header row)
# Use awk to filter rows where sales > MIN_SALES
awk -v min="$MIN_SALES" 'BEGIN {FS=","} NR==1; NR>1 && $3 > min' "$INPUT_CSV" > "$OUTPUT_CSV"

echo "Filtered $INPUT_CSV to $OUTPUT_CSV (sales > $MIN_SALES)"

Why it works: awk efficiently processes structured data; NR==1 preserves the header.

4. Automation Workflows

Shell scripts automate end-to-end workflows, from file cleanup to cloud resource management.

Example 6: Scheduled Task with Cron

Use cron to run scripts at specific intervals. For example, schedule the backup_data.sh script to run daily at 2 AM:

  1. Open the crontab editor: crontab -e
  2. Add: 
    0 2 * * * /path/to/backup_data.sh  # Run daily at 2:00 AM

Example 7: Cloud Resource Cleanup (AWS)

Automatically terminate unused EC2 instances tagged “temporary” older than 7 days:

#!/bin/bash
# aws_cleanup.sh: Terminate old temporary EC2 instances

# Get instance IDs older than 7 days with tag "temporary=true"
INSTANCE_IDS=$(aws ec2 describe-instances \
  --filters "Name=tag:temporary,Values=true" \
  --query "Reservations[].Instances[?LaunchTime<='$(date -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ)'].InstanceId" \
  --output text)

# Terminate instances if any exist
if [ -n "$INSTANCE_IDS" ]; then
  echo "Terminating instances: $INSTANCE_IDS"
  aws ec2 terminate-instances --instance-ids "$INSTANCE_IDS"
else
  echo "No temporary instances older than 7 days found."
fi

Use case: Run this weekly to reduce cloud costs by cleaning up forgotten resources.

Common Practices for Effective Scripting

To write maintainable and reliable scripts, follow these practices:

Modularity with Functions

Break scripts into reusable functions to avoid repetition:

#!/bin/bash

# Reusable logging function
log() {
  local timestamp=$(date +"%Y-%m-%d %H:%M:%S")
  echo "[$timestamp] $1" >> "$LOG_FILE"
}

# Usage
LOG_FILE="app.log"
log "Starting backup..."
log "Backup failed!"  # Logs with timestamp

Error Handling

  • set -euo pipefail: Exit on errors (-e), unset variables (-u), and failed pipes (-o pipefail).
  • Trap signals: Clean up temporary files on script exit (success or failure): 
    TEMP_FILE=$(mktemp)
trap 'rm -f "$TEMP_FILE"' EXIT  # Delete TEMP_FILE on exit

Logging

Always log actions and errors for debugging:

# Log to file and stdout
log() {
  echo "[$(date)] $*" | tee -a "script.log"  # tee writes to both stdout and file
}

Input Validation

Check for required arguments or dependencies:

#!/bin/bash
# validate_input.sh

if [ $# -ne 1 ]; then
  echo "Usage: $0 <input-file>"
  exit 1
fi

INPUT_FILE="$1"
if [ ! -f "$INPUT_FILE" ]; then
  echo "Error: $INPUT_FILE does not exist."
  exit 1
fi

Best Practices for Production-Ready Scripts

To ensure scripts are secure, scalable, and easy to maintain:

  1. Use a Shebang: Always start with #!/bin/bash (not #!/bin/sh for portability).
  2. Avoid Hard-Coded Values: Use variables or command-line arguments instead: 
    # Bad: HARDCODED_DIR="/tmp"
# Good: DIR="${1:-/tmp}"  # Default to /tmp if no argument
  3. Use Absolute Paths: Avoid relative paths to prevent confusion: 
    # Bad: cp data.txt ../backup  # Depends on current directory
# Good: cp "/home/user/data.txt" "/mnt/backup"
  4. Test Thoroughly: Use bash -n script.sh to check syntax, and test edge cases (e.g., empty files, missing arguments).
  5. Security:
    • Avoid eval (risk of command injection).
    • Sanitize user input: read -r INPUT ( -r prevents backslash escapes).
    • Restrict permissions: chmod 700 script.sh (only owner can execute).

Conclusion

Shell scripting is a versatile tool for solving real-world automation challenges across system administration, DevOps, and data processing. By mastering core syntax, leveraging practical use cases (like log rotation, CI/CD, and cloud cleanup), and following best practices (modularity, error handling, security), you can write scripts that are efficient, maintainable, and scalable.

Whether you’re automating daily tasks or integrating into enterprise workflows, shell scripting remains an indispensable skill for modern tech professionals.

References