Automating System Tasks with Shell Scripts: A Step-by-Step Approach

Understanding Shell Scripts
Getting Started: Your First Shell Script
Core Concepts for Automation
Common Use Cases with Examples
Best Practices for Reliable Scripts
Advanced Tips: Scheduling and Scaling
Conclusion
References

1. Understanding Shell Scripts

What Are Shell Scripts?

A shell script is a plain text file containing a series of commands that the shell (e.g., Bash, Zsh, Sh) executes sequentially. It acts as a “glue” to combine system utilities (e.g., ls, grep, rsync) into automated workflows.

Why Use Shell Scripts for Automation?

Simplicity: No compilation required; write and run immediately.
Portability: Works on any Unix-like system (Linux, macOS, BSD) with minimal modifications.
Access to System Tools: Directly leverage powerful command-line utilities (e.g., awk, sed, find).
Low Overhead: Lightweight compared to scripting languages like Python or Ruby for simple tasks.

Common Shells

Bash (Bourne Again Shell): The most popular shell, default on most Linux systems and macOS (until macOS 10.15, which uses Zsh).
Zsh: Extends Bash with features like improved autocompletion and themes.
Sh (Bourne Shell): Older, simpler shell; scripts written for Sh are compatible with most shells.

This blog focuses on Bash (Bourne Again Shell), as it is the de facto standard for system automation.

2. Getting Started: Your First Shell Script

Let’s write a simple script to print a greeting and system information. Follow these steps:

Step 1: Create the Script File

Use a text editor (e.g., nano, vim) to create system-info.sh:

nano system-info.sh

Step 2: Add the Shebang Line

The first line of a shell script specifies the interpreter (shebang line). For Bash, use:

#!/bin/bash

This tells the system to run the script with /bin/bash.

Step 3: Write the Script Logic

Add commands to print a greeting and system details:

#!/bin/bash

# Print a welcome message
echo "Hello, $(whoami)! Welcome to your system info script."

# Print current date and time
echo "Current date: $(date)"

# Print OS version
echo "OS Version: $(uname -a)"

# Print disk usage
echo "Disk Usage:"
df -h /

Step 4: Make the Script Executable

By default, text files are not executable. Use chmod to set execute permissions:

chmod +x system-info.sh

Step 5: Run the Script

Execute the script with:

./system-info.sh

Output Example:

Hello, alice! Welcome to your system info script.
Current date: Wed Oct 11 14:30:00 UTC 2023
OS Version: Linux server 5.4.0-125-generic #141-Ubuntu SMP Wed Aug 24 16:10:11 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Disk Usage:
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       200G   80G  120G  40% /

Congratulations! You’ve written and executed your first shell script.

3. Core Concepts for Automation

3.1 Variables and Command Substitution

Variables store data for reuse. Use VAR=value to declare, and $VAR to access.

Example: Variables and Command Substitution

#!/bin/bash

# Declare a variable
NAME="Alice"
echo "Hello, $NAME!"  # Access with $

# Command substitution: capture output of a command
UPTIME=$(uptime)  # or `uptime` (backticks, older syntax)
echo "System uptime: $UPTIME"

# Environment variables (predefined)
echo "Home directory: $HOME"
echo "Path: $PATH"

Output:

Hello, Alice!
System uptime: 14:35:00 up 2 days, 3 hours, 10 minutes,  1 user,  load average: 0.50, 0.40, 0.30
Home directory: /home/alice
Path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

3.2 Loops: Iterating Over Tasks

Loops automate repetitive actions (e.g., processing files, checking services).

For Loops: Iterate Over Lists

#!/bin/bash

# Loop over files in the current directory
echo "Files in current directory:"
for file in *; do
  echo "- $file"
done

# Loop over numbers 1-5
echo "Counting to 5:"
for i in {1..5}; do
  echo $i
done

While Loops: Repeat Until a Condition Fails

#!/bin/bash

# Read lines from a file until EOF
echo "Reading lines from example.txt:"
while IFS= read -r line; do  # IFS= prevents trimming whitespace; -r preserves backslashes
  echo "Line: $line"
done < example.txt  # Input file

3.3 Conditionals: Making Decisions

Use if statements to execute code based on conditions (e.g., file existence, command success).

Basic If Statement

#!/bin/bash

FILE="data.txt"

# Check if file exists and is a regular file
if [ -f "$FILE" ]; then  # [ ] is the test command; -f checks for regular file
  echo "$FILE exists."
else
  echo "$FILE does NOT exist."
fi

Advanced Conditionals with `[[ ]]` (Bash-Specific)

[[ ]] supports more features than [ ], like pattern matching and logical operators:

#!/bin/bash

USER="alice"

# Check if user is "alice" or "bob"
if [[ "$USER" == "alice" || "$USER" == "bob" ]]; then  # || = logical OR
  echo "Welcome, $USER!"
elif [[ "$USER" == "root" ]]; then
  echo "Warning: Running as root!"
else
  echo "Access denied."
fi

Common Test Operators

Operator	Description	Example
`-f`	File exists and is regular	`[ -f "file.txt" ]`
`-d`	Directory exists	`[ -d "docs" ]`
`-e`	File/directory exists	`[ -e "path" ]`
`-r`	File is readable	`[ -r "data.txt" ]`
`-z`	String is empty	`[ -z "$VAR" ]`
`==`	String equality ([[ ]])	`[[ "$STR" == "hello" ]]`
`-eq`	Numeric equality	`[ $NUM -eq 10 ]`

3.4 Functions: Reusing Code

Functions group commands into reusable blocks, improving readability and reducing redundancy.

Define and Call a Function

#!/bin/bash

# Function to print a greeting
greet() {
  local name=$1  # Local variable (only accessible in the function)
  echo "Hello, $name!"
}

# Call the function with an argument
greet "Alice"  # Output: Hello, Alice!
greet "Bob"    # Output: Hello, Bob!

# Function with return value (via exit code or command substitution)
add() {
  local a=$1
  local b=$2
  echo $((a + b))  # Return sum via stdout
}

result=$(add 3 5)  # Capture output
echo "3 + 5 = $result"  # Output: 3 + 5 = 8

4. Common Use Cases with Examples

4.1 File Management Automation

Task: Batch rename all .txt files to add a “backup_” prefix.

#!/bin/bash

# Rename .txt files with "backup_" prefix
for file in *.txt; do
  if [ -f "$file" ]; then  # Ensure it's a regular file
    mv -- "$file" "backup_$file"  # -- handles filenames with leading hyphens
    echo "Renamed: $file → backup_$file"
  fi
done

Task: Delete log files older than 7 days.

#!/bin/bash

LOG_DIR="/var/log/myapp"
DAYS=7

# Delete .log files older than 7 days
find "$LOG_DIR" -name "*.log" -type f -mtime +"$DAYS" -delete
echo "Deleted logs older than $DAYS days in $LOG_DIR"

4.2 System Monitoring and Alerts

Task: Check disk usage and alert if root partition exceeds 90% usage.

#!/bin/bash

THRESHOLD=90
PARTITION="/"

# Get current usage percentage (e.g., "40" for 40%)
USAGE=$(df -h "$PARTITION" | awk 'NR==2 {print $5}' | sed 's/%//')

if [ "$USAGE" -gt "$THRESHOLD" ]; then
  echo "ALERT: Disk usage on $PARTITION is $USAGE% (exceeds $THRESHOLD%)" | mail -s "Disk Usage Alert" [email protected]
else
  echo "Disk usage on $PARTITION is $USAGE% (within threshold)"
fi

4.3 Backup Automation

Task: Backup a directory to a remote server using rsync.

#!/bin/bash

SOURCE="/home/alice/documents"
DEST="backupuser@backupserver:/backups/documents"
LOG_FILE="/var/log/backup.log"

# Add timestamp to log
echo "===== Backup started at $(date) =====" >> "$LOG_FILE"

# Run rsync with compression and verbose output
rsync -avz --delete "$SOURCE" "$DEST" >> "$LOG_FILE" 2>&1  # Redirect stdout/stderr to log

# Check if rsync succeeded
if [ $? -eq 0 ]; then  # $? is the exit code of the last command (0 = success)
  echo "Backup completed successfully at $(date)" >> "$LOG_FILE"
else
  echo "Backup FAILED at $(date)" >> "$LOG_FILE"
  exit 1  # Indicate failure to cron (if scheduled)
fi

4.4 Log Rotation

Task: Rotate a log file (move current log to .old, create new log, compress old logs monthly).

#!/bin/bash

LOG_FILE="/var/log/myapp/app.log"
OLD_LOG="${LOG_FILE}.old"
ARCHIVE_DIR="/var/log/myapp/archive"

# Create archive directory if it doesn't exist
mkdir -p "$ARCHIVE_DIR"

# If log file exists, rotate it
if [ -f "$LOG_FILE" ]; then
  # Compress and archive old log (e.g., app.log.2023-10-11.gz)
  tar -czf "${ARCHIVE_DIR}/app-$(date +%Y-%m-%d).log.gz" "$OLD_LOG" 2>/dev/null
  echo "Archived old log to ${ARCHIVE_DIR}/app-$(date +%Y-%m-%d).log.gz"
  
  # Move current log to .old
  mv -- "$LOG_FILE" "$OLD_LOG"
  echo "Rotated current log to $OLD_LOG"
  
  # Create new empty log file
  touch "$LOG_FILE"
  chmod 644 "$LOG_FILE"  # Set permissions
else
  echo "Log file $LOG_FILE not found; no rotation needed."
fi

5. Best Practices for Reliable Scripts

5.1 Error Handling

Prevent silent failures with these techniques:

set -e: Exit immediately if any command fails.
set -u: Treat unset variables as errors (avoids accidental $VAR typos).
set -o pipefail: Exit if any command in a pipeline fails (not just the last one).

#!/bin/bash
set -euo pipefail  # Enable strict error checking

# If "nonexistent_file.txt" doesn't exist, script exits here due to set -e
cat nonexistent_file.txt
echo "This line will NOT run"  # Script already exited

5.2 Readability and Maintainability

Comments: Explain why, not just what (e.g., # Clean up temp files to free disk space).
Indentation: Use 2–4 spaces for loops/conditionals/functions.
Meaningful Names: Avoid x or temp; use log_dir or backup_source.
Modularize with Functions: Break large scripts into functions (e.g., log_message(), validate_input()).

5.3 Security

Avoid Hardcoding Secrets: Use environment variables or secure vaults (e.g., export DB_PASSWORD="secret" instead of writing to the script).
Restrict Permissions: Make scripts executable only by the owner: chmod 700 script.sh.

Sanitize Input: Validate user input to prevent path traversal or command injection:

# Unsafe: user could pass "../etc/passwd" as $FILE
# cat "$FILE"

# Safe: restrict to files in /tmp
if [[ "$FILE" != /tmp/* ]]; then
  echo "Error: File must be in /tmp"
  exit 1
fi
cat "$FILE"

5.4 Testing

Use shellcheck: A linter for shell scripts to catch syntax errors and bad practices:

shellcheck myscript.sh  # Install with: sudo apt install shellcheck (Linux) or brew install shellcheck (macOS)

Test in Staging: Replicate production conditions before deploying.

Dry Runs: Add a --dry-run flag to simulate actions without making changes:

DRY_RUN=false
if [ "$1" == "--dry-run" ]; then DRY_RUN=true; fi

# Example: Dry-run safe deletion
if [ "$DRY_RUN" = true ]; then
  echo "Would delete: $file"
else
  rm -- "$file"
fi

5.5 Portability

Use POSIX-compliant syntax if targeting sh (avoid Bash-specific features like [[ ]] or arrays).
Avoid hardcoded paths (e.g., use #!/usr/bin/env bash instead of #!/bin/bash for systems where Bash is in /usr/local/bin).

Table of Contents