dotlinux guide

Automate File Management with Powerful Shell Scripts

Table of Contents

Understanding Shell Scripts for File Management

A shell script is a text file containing a sequence of command-line instructions executed by a shell (e.g., bash, zsh). For file management, shell scripts excel at:

  • Automating repetitive tasks (e.g., sorting, backups, cleanup).
  • Enforcing consistency (e.g., standardizing file naming conventions).
  • Reducing human error (e.g., avoiding accidental deletions).
  • Scaling workflows (e.g., processing thousands of files).

Unlike graphical tools, shell scripts are lightweight, scriptable, and work across Unix-like systems (Linux, macOS, BSD). They integrate seamlessly with core command-line utilities like cp, mv, rm, find, and rsync.

Core Concepts: Building Blocks of File Automation

To write effective file management scripts, you need to master these foundational concepts:

1. Variables: Store Paths and Configuration

Variables let you store filenames, directories, or settings, making scripts reusable. Use = to assign values (no spaces around =).

# Define source and destination directories
SOURCE_DIR="$HOME/Downloads"
DEST_DIR="$HOME/Documents/Organized"

# Store a file pattern
FILE_PATTERN="*.txt"

Use $VARIABLE to reference values (e.g., echo "Source: $SOURCE_DIR").

2. Loops: Iterate Over Files

Loops (e.g., for, while) process multiple files or directories.

Example: Loop through .txt files in a directory

for file in "$SOURCE_DIR"/"$FILE_PATTERN"; do
  echo "Processing: $file"
  # Add logic here (e.g., move, copy, rename)
done

3. Conditionals: Make Decisions

Use if statements to check file existence, type, or permissions.

Common file checks:

  • -f "$file": Is $file a regular file?
  • -d "$dir": Is $dir a directory?
  • -e "$path": Does $path exist?
  • -r "$file": Is $file readable?

Example: Check if a directory exists, create it if not

if [ ! -d "$DEST_DIR" ]; then
  echo "Creating destination directory: $DEST_DIR"
  mkdir -p "$DEST_DIR"  # -p creates parent dirs if needed
fi

4. File Operations: Copy, Move, Delete, and More

Use standard commands to manipulate files:

  • cp: Copy files (cp file1 file2)
  • mv: Move/rename files (mv oldname newname)
  • rm: Delete files (rm file; use -r for directories)
  • mkdir: Create directories (mkdir dir)
  • tar: Archive files (tar -czf archive.tar.gz files/)

Usage Methods: Writing and Executing Scripts

Step 1: Write the Script

Create a .sh file (e.g., organize_files.sh) with a shebang line to specify the shell:

#!/bin/bash
# organize_files.sh: Automatically sort files by type

SOURCE_DIR="$HOME/Downloads"
DEST_DIR="$HOME/Documents/Organized"

# Create destination if missing
mkdir -p "$DEST_DIR/Texts" "$DEST_DIR/Images"

# Move .txt files to Texts
for txt_file in "$SOURCE_DIR"/*.txt; do
  [ -f "$txt_file" ] && mv "$txt_file" "$DEST_DIR/Texts/"
done

# Move .jpg/.png to Images
for img_file in "$SOURCE_DIR"/*.{jpg,png}; do
  [ -f "$img_file" ] && mv "$img_file" "$DEST_DIR/Images/"
done

echo "Files organized successfully!"

Step 2: Make It Executable

Run chmod +x organize_files.sh to grant execution permissions.

Step 3: Execute the Script

Run with ./organize_files.sh (or absolute path: /home/user/organize_files.sh).

Debugging Tips

  • Use bash -x script.sh to trace execution (shows each command as it runs).
  • Add echo statements to print variables/debug info: echo "Moving $txt_file to $DEST_DIR/Texts".
  • Use shellcheck script.sh (install with sudo apt install shellcheck) to catch syntax errors.

Common File Management Tasks & Script Examples

1. Organize Files by Type

Goal: Sort downloads into folders by extension (e.g., .pdf → PDFs, .mp3 → Music).

#!/bin/bash
# organize_by_type.sh

SOURCE="$HOME/Downloads"
declare -A DESTINATIONS=(  # Associative array: extension → folder
  [txt]="Documents/Texts"
  [pdf]="Documents/PDFs"
  [jpg]="Pictures"
  [png]="Pictures"
  [mp3]="Music"
  [zip]="Archives"
)

for ext in "${!DESTINATIONS[@]}"; do
  dest="$HOME/${DESTINATIONS[$ext]}"
  mkdir -p "$dest"
  # Move all .ext files (skip if no files match)
  find "$SOURCE" -maxdepth 1 -type f -name "*.$ext" -exec mv {} "$dest/" \;
done

echo "Files organized by type!"

2. Automated Backups with Date Stamps

Goal: Back up a project folder to an external drive with a timestamp (e.g., backup_20240520).

#!/bin/bash
# backup_project.sh

PROJECT_DIR="$HOME/Projects/my_app"
BACKUP_DIR="/mnt/external_drive/backups"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)  # Format: YYYYMMDD_HHMMSS
BACKUP_NAME="my_app_backup_$TIMESTAMP.tar.gz"

# Check if project exists
if [ ! -d "$PROJECT_DIR" ]; then
  echo "Error: Project directory $PROJECT_DIR not found!"
  exit 1  # Exit with error code
fi

# Create backup (compress with gzip)
tar -czf "$BACKUP_DIR/$BACKUP_NAME" "$PROJECT_DIR"

# Verify backup success
if [ $? -eq 0 ]; then  # $? = exit code of last command (0 = success)
  echo "Backup created: $BACKUP_DIR/$BACKUP_NAME"
else
  echo "Backup failed!"
  exit 1
fi

3. Log Rotation: Compress Old Logs

Goal: Compress logs older than 7 days and delete logs older than 30 days.

#!/bin/bash
# rotate_logs.sh

LOG_DIR="/var/log/my_app"
MAX_AGE_DAYS=7  # Compress logs older than this
KEEP_DAYS=30    # Delete logs older than this

# Compress old logs (e.g., app.log.1 → app.log.1.gz)
find "$LOG_DIR" -name "*.log.*" -type f -mtime +"$MAX_AGE_DAYS" -exec gzip {} \;

# Delete logs older than 30 days
find "$LOG_DIR" -name "*.log.*.gz" -type f -mtime +"$KEEP_DAYS" -delete

echo "Log rotation complete."

4. Bulk Rename Files

Goal: Replace spaces with underscores in filenames (e.g., my file.txt → my_file.txt).

#!/bin/bash
# rename_replace_spaces.sh

TARGET_DIR="$HOME/Documents"

for file in "$TARGET_DIR"/*\ *; do  # Match files with spaces
  [ -f "$file" ] || continue  # Skip directories
  # Replace spaces with underscores
  new_name="${file// /_}"
  mv -v "$file" "$new_name"  # -v = verbose (show changes)
done

Best Practices for Reliable Scripts

1. Test in a Safe Environment

Always test scripts on non-critical files (e.g., a test_dir with dummy files) to avoid accidental data loss.

2. Use Absolute Paths

Avoid relative paths (e.g., ../docs), which depend on the working directory. Use absolute paths like /home/user/Documents.

3. Handle Errors Gracefully

  • set -e: Exit immediately if any command fails (add at the top of the script).
  • set -u: Treat unset variables as errors (avoids silent failures).
  • trap: Clean up temporary files on exit (e.g., trap 'rm -f temp.txt' EXIT).

Example: Robust error handling

#!/bin/bash
set -euo pipefail  # Exit on error, unset var, or failed pipe

# Cleanup temp files on exit
TEMP_FILE=$(mktemp)
trap 'rm -f "$TEMP_FILE"' EXIT

# Critical operation (will exit if it fails)
cp "important_file.txt" "$TEMP_FILE"

4. Log Output

Redirect script output to a log file for debugging:

# At the top of the script
LOG_FILE="$HOME/scripts/logs/organize_$(date +%Y%m%d).log"
exec > "$LOG_FILE" 2>&1  # Redirect stdout/stderr to log

# Now all commands log to $LOG_FILE
echo "Starting file organization at $(date)"

5. Comment and Document

Explain why (not just what) the code does. Example:

# Use find instead of glob (*.txt) to handle spaces in filenames
find "$SOURCE" -maxdepth 1 -type f -name "*.txt" -exec mv {} "$DEST/" \;

6. Avoid Hard-Coding

Use variables or command-line arguments for flexibility. Example:

#!/bin/bash
# backup.sh [source] [dest] (accepts arguments)
SOURCE="${1:-$HOME/Projects}"  # Default if no arg provided
DEST="${2:-/mnt/backup}"

Advanced Techniques: Scheduling and Complex Workflows

1. Schedule Scripts with cron

Use cron to run scripts automatically (e.g., daily backups, weekly log rotation).

Edit crontab: Run crontab -e and add:

# Run backup at 2 AM daily
0 2 * * * /home/user/scripts/backup_project.sh >> /home/user/scripts/backup_logs.txt 2>&1
  • Format: MIN HOUR DAY MONTH WEEKDAY command
  • >> log.txt 2>&1: Append output/errors to log.

2. Find Files Efficiently with find

The find command locates files by name, size, date, or type. Combine with exec to process results:

Example: Delete empty directories

find "$HOME/Documents" -type d -empty -delete

Example: Find large files (>100MB) and log them

find / -type f -size +100M -exec du -h {} \; > large_files.log 2>/dev/null

3. Check File Integrity with Checksums

Use md5sum or sha256sum to verify backups or detect corruption:

#!/bin/bash
# verify_backup.sh
BACKUP_FILE="/mnt/backup/my_app_backup_20240520.tar.gz"
CHECKSUM_FILE="$BACKUP_FILE.sha256"

# Generate checksum (run once after backup)
sha256sum "$BACKUP_FILE" > "$CHECKSUM_FILE"

# Verify later
if sha256sum --verify "$CHECKSUM_FILE"; then
  echo "Backup is valid!"
else
  echo "Backup corrupted!"
fi

Conclusion

Automating file management with shell scripts transforms tedious, error-prone tasks into efficient, repeatable workflows. By mastering variables, loops, conditionals, and core commands, you can build scripts to organize files, back up data, rotate logs, and more.

Start small: automate one task (e.g., sorting downloads), then layer in best practices like error handling and logging. As you gain confidence, explore advanced tools like find, cron, and rsync to tackle complex workflows.

With shell scripts, you’ll save time, reduce mistakes, and unlock the full power of the command line for file management.

References