dotlinux guide

How to Shell Script with Linux Command Line Effectively

Shell scripting is a cornerstone of Linux system administration, DevOps, and automation. By combining Linux commands into reusable scripts, you can automate repetitive tasks, streamline workflows, and solve complex problems with minimal effort. Whether you’re a system administrator managing servers, a developer automating build processes, or a power user simplifying daily tasks, mastering shell scripting unlocks efficiency and control over the Linux command line. This blog will guide you through the fundamentals of shell scripting, effective usage patterns, common practices, and best practices to write robust, maintainable, and efficient scripts. By the end, you’ll be equipped to create scripts that are not only functional but also secure, portable, and easy to debug.

Table of Contents

What is Shell Scripting?

A shell script is a text file containing a sequence of commands executed by a Unix/Linux shell (e.g., bash, sh, zsh). The shell acts as an interpreter, executing commands line-by-line, and supports programming constructs like variables, loops, and functions. Shell scripts bridge the gap between manual command-line execution and full-fledged programming, making them ideal for automation.

Fundamental Concepts

Shebang Line

The first line of a shell script, called the “shebang,” specifies the interpreter to use. For bash scripts, this is:

#!/bin/bash

For POSIX-compliant scripts (portable across shells like sh), use:

#!/bin/sh

Without a shebang, the script runs in the current shell, which may cause inconsistencies.

Variables and Data Types

Variables store data for reuse. They are declared without a type, and their values are strings by default.

Basic Variables

# Declaration (no spaces around =)
name="Alice"
age=30

# Access with $
echo "Name: $name, Age: $age"  # Output: Name: Alice, Age: 30

Arrays

Store multiple values in a single variable:

fruits=("apple" "banana" "cherry")
echo "First fruit: ${fruits[0]}"  # Output: apple
echo "All fruits: ${fruits[@]}"   # Output: apple banana cherry

Associative Arrays (Bash 4+)

Key-value pairs (not POSIX-compliant):

declare -A user
user["name"]="Bob"
user["age"]=25
echo "User: ${user[name]}, Age: ${user[age]}"  # Output: User: Bob, Age: 25

Control Structures

Control the flow of execution with conditionals.

if-else Statements

Check conditions (e.g., file existence, command success):

file="data.txt"

if [ -f "$file" ]; then  # -f checks if file exists and is regular
  echo "$file exists."
elif [ -d "$file" ]; then  # -d checks if directory
  echo "$file is a directory."
else
  echo "$file does not exist."
fi

case Statements

Simplify multi-condition checks:

day="Monday"

case $day in
  Monday|Tuesday|Wednesday|Thursday|Friday)
    echo "Weekday"
    ;;
  Saturday|Sunday)
    echo "Weekend"
    ;;
  *)
    echo "Invalid day"
    ;;
esac

Loops

Automate repetitive tasks with loops.

for Loop

Iterate over a list of items:

# Iterate over strings
for fruit in apple banana cherry; do
  echo "I like $fruit"
done

# Iterate over files (e.g., all .txt files)
for file in *.txt; do
  echo "Processing $file"
done

while Loop

Run until a condition is false:

count=1
while [ $count -le 5 ]; do  # -le = less than or equal
  echo "Count: $count"
  count=$((count + 1))  # Increment count
done

until Loop

Run until a condition is true (opposite of while):

count=5
until [ $count -lt 1 ]; do  # -lt = less than
  echo "Count: $count"
  count=$((count - 1))
done

Functions

Reusable blocks of code. Use them to avoid repetition and improve readability.

# Define function
greet() {
  local name=$1  # Local variable (scope limited to function)
  echo "Hello, $name!"
}

# Call function
greet "Alice"  # Output: Hello, Alice!

Parameters: Access with $1, $2, etc.
Return Values: Use return for exit codes (0-255) or echo to return strings (capture with command substitution).

Effective Usage Methods

Command Substitution

Embed the output of a command into a variable or another command using $(...) (preferred) or `...` (legacy).

# Get current date
current_date=$(date +"%Y-%m-%d")
echo "Today is $current_date"  # Output: Today is 2024-05-20

# Count lines in a file
line_count=$(wc -l < data.txt)
echo "Lines: $line_count"

Input/Output Redirection

Control where input comes from and output goes.

  • >: Overwrite file with output (e.g., echo "Hi" > file.txt).
  • >>: Append to file (e.g., echo "Again" >> file.txt).
  • <: Read input from file (e.g., grep "error" < log.txt).
  • 2>: Redirect errors (e.g., command_that_fails 2> errors.log).
  • &>: Redirect both output and errors (e.g., script.sh &> output.log).

Example:

# Save command output and errors to separate files
ls -l /nonexistent 1> files.txt 2> errors.txt

Pipes

Chain commands with | to pass output of one command as input to the next.

# Find .txt files, count lines, sort by line count (descending)
find . -name "*.txt" -exec wc -l {} + | sort -nr

Error Handling

Prevent silent failures and make scripts robust.

  • Exit Codes: Commands return 0 on success, non-zero on failure (check with $?).
  • set -e: Exit immediately if any command fails (add at the top of scripts).
  • trap: Catch signals (e.g., EXIT, ERR) to clean up resources.
  • Explicit Checks: Use if to verify command success.
set -e  # Exit on any error

# Check if a command succeeded
if grep "critical" log.txt; then
  echo "Critical error found!"
  exit 1  # Exit with non-zero code to indicate failure
fi

# Cleanup on exit (e.g., remove temp files)
trap 'rm -f temp.txt' EXIT

Common Practices

Script Structure

Organize scripts for clarity:

  1. Shebang line
  2. Comments (purpose, author, usage)
  3. Global variables
  4. Functions
  5. Main logic

Example:

#!/bin/bash  
# Purpose: Backup a directory to a compressed file  
# Usage: ./backup.sh <source_dir> <dest_dir>  

set -euo pipefail  # Strict error checking

# Global variables
SOURCE_DIR="$1"
DEST_DIR="$2"
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
BACKUP_FILE="${DEST_DIR}/backup_${TIMESTAMP}.tar.gz"

# Function: Validate input
validate_input() {
  if [ $# -ne 2 ]; then
    echo "Usage: $0 <source_dir> <dest_dir>"
    exit 1
  fi
  if [ ! -d "$SOURCE_DIR" ]; then
    echo "Error: Source directory $SOURCE_DIR does not exist." >&2; exit 1
  fi
}

# Main logic
validate_input "$@"  
echo "Backing up $SOURCE_DIR to $BACKUP_FILE..."  
tar -czf "$BACKUP_FILE" "$SOURCE_DIR"  
echo "Backup completed: $BACKUP_FILE"

Testing and Debugging

  • shellcheck: Static analysis tool to catch syntax errors (install with sudo apt install shellcheck). 
    shellcheck my_script.sh
  • Debug Mode: Use set -x to print commands as they execute (add at the top or run with bash -x my_script.sh).
  • Dry Runs: Test logic without modifying data (e.g., echo "tar -czf $BACKUP_FILE $SOURCE_DIR").

Best Practices

Security

  • Avoid eval: It executes arbitrary input, posing injection risks.
  • Sanitize Inputs: Validate user inputs (e.g., check file paths exist).
  • Use Absolute Paths: Avoid relying on $PATH (e.g., /usr/bin/grep instead of grep).
  • Limit Permissions: Make scripts executable only by the owner (chmod 700 script.sh).

Performance

  • Avoid Subshells: Use { ... } instead of (...) for grouping (no subshell overhead).
  • Efficient Loops: Minimize command calls inside loops (e.g., process files in batches).
  • Use Built-Ins: Prefer shell built-ins (e.g., [[ ]] over [ ], (( )) for arithmetic) over external tools like expr.

Portability

  • POSIX Compliance: Use sh syntax (e.g., [ ] instead of [[ ]], no arrays) if targeting non-bash shells.
  • Avoid Bashisms: Features like associative arrays, source (use . instead), or ** (globstar) may not work in sh.

Readability

  • Indentation: Use 2–4 spaces (no tabs).
  • Variable Names: Use descriptive names (user_count instead of uc).
  • Functions: Break logic into small, single-purpose functions.

Advanced Tips

  • Arrays: Process lists of data (e.g., files=(*.txt); echo "${files[@]}").
  • Here-Documents: Embed multi-line text (e.g., cat << EOF > config.txt ... EOF).
  • getopts: Parse command-line arguments (flags like -v, -f).
  • Process Substitution: Treat command output as a file (e.g., diff <(sort a.txt) <(sort b.txt)).

Conclusion

Shell scripting is a versatile skill that empowers you to automate, customize, and control Linux systems. By mastering fundamentals like variables, loops, and error handling, adopting best practices for security and readability, and leveraging tools like shellcheck, you can write scripts that are robust, efficient, and easy to maintain. Start small—automate a daily task—and gradually tackle more complex problems. With practice, shell scripting will become an indispensable part of your technical toolkit.

References