Dynamic Configuration Management with Shell Scripts: A Comprehensive Guide

Table of Contents

1. Fundamental Concepts
   • What is Dynamic Configuration Management?
   • Key Challenges in Configuration Management
   • Why Use Shell Scripts for Dynamic Configuration?
2. Usage Methods
   • Configuration Sources
   • Runtime Configuration Adjustment
   • Configuration Validation
3. Common Practices
   • Configuration Templates with Placeholders
   • Environment-Specific Configurations
   • Modular Configuration with Include Files
4. Best Practices
   • Security: Protecting Sensitive Data
   • Idempotency: Scripts That “Just Work”
   • Logging, Debugging, and Testing
   • Portability and Compatibility
5. Conclusion
6. References

Fundamental Concepts

What is Dynamic Configuration Management?

Dynamic Configuration Management (DCM) is the process of managing application or system settings that can be modified at runtime without requiring a service restart. This is critical for use cases like:

• Adjusting feature flags or rate limits in production.
• Updating database credentials or API endpoints during maintenance.
• Tailoring behavior to environment-specific requirements (e.g., development vs. production).

Unlike static configuration (fixed files deployed with the application), dynamic configurations are often fetched, parsed, or generated on-demand.

Key Challenges in Configuration Management

Effective DCM requires addressing:

• Environment Parity: Ensuring configurations work consistently across development, staging, and production.
• Runtime Adaptability: Updating settings without downtime.
• Consistency: Avoiding drift between nodes in distributed systems.
• Security: Protecting sensitive data (e.g., API keys, passwords) from exposure.
• Traceability: Tracking configuration changes for auditing and debugging.

Why Use Shell Scripts for Dynamic Configuration?

Shell scripts (Bash, POSIX sh, etc.) are uniquely suited for DCM in many scenarios:

• Ubiquity: Preinstalled on all Unix-like systems (Linux, macOS, BSD), requiring no additional dependencies.
• Flexibility: Integrate seamlessly with system tools (e.g., grep, awk, sed) and external services (APIs, databases).
• Simplicity: Lower learning curve compared to orchestration tools, making them accessible for small teams or ad-hoc tasks.
• Control: Fine-grained control over configuration logic, from parsing files to triggering reloads.

Usage Methods

1. Configuration Sources

Shell scripts can ingest configuration from diverse sources. Below are common examples:

Environment Variables

Environment variables are a lightweight way to pass dynamic values to scripts. They are ideal for sensitive data (e.g., API keys) or environment-specific settings (e.g., NODE_ENV).

Example: Generating a config file from environment variables
Suppose you need to generate an Nginx config with a dynamic $API_URL:

#!/bin/sh
# generate_nginx_config.sh

# Check if API_URL is set
if [ -z "$API_URL" ]; then
  echo "Error: API_URL environment variable not set" >&2
  exit 1
fi

# Generate config file with environment variable
cat > /etc/nginx/conf.d/app.conf << EOF
server {
  listen 80;
  location /api {
    proxy_pass $API_URL;
  }
}
EOF

# Reload Nginx to apply changes
nginx -s reload

Run with:
API_URL=https://api.example.com ./generate_nginx_config.sh

Configuration Files

Scripts can parse structured files (JSON, YAML, INI) to extract settings. Tools like jq (JSON), yq (YAML), or awk simplify parsing.

Example: Parsing a YAML config file with yq
Given config.yaml:

app:
  name: "myapp"
  port: 8080
database:
  host: "db.example.com"
  user: "admin"

Use yq (a YAML parser for the command line) to extract values:

#!/bin/bash
# load_config.sh

# Install yq if missing (example for Debian/Ubuntu)
if ! command -v yq &> /dev/null; then
  echo "Installing yq..."
  sudo apt-get install -y yq
fi

APP_NAME=$(yq eval '.app.name' config.yaml)
DB_HOST=$(yq eval '.database.host' config.yaml)

echo "Starting $APP_NAME on port $(yq eval '.app.port' config.yaml)"
echo "Connecting to database at $DB_HOST"

Command-Line Arguments

For ad-hoc adjustments, scripts can accept command-line arguments using getopts (POSIX) or argparse (Bash extensions).

Example: Using getopts to handle flags

#!/bin/sh
# deploy.sh - Deploys an app with dynamic config

ENV="dev"  # Default environment
PORT=8080  # Default port

# Parse arguments: -e (environment), -p (port)
while getopts "e:p:" opt; do
  case $opt in
    e) ENV="$OPTARG" ;;
    p) PORT="$OPTARG" ;;
    \?) echo "Invalid option: -$OPTARG" >&2; exit 1 ;;
    :) echo "Option -$OPTARG requires an argument." >&2; exit 1 ;;
  esac
done

echo "Deploying to $ENV environment on port $PORT"
# Deploy logic here...

Run with:
./deploy.sh -e prod -p 80

External Services

Scripts can fetch configs from APIs (e.g., Consul, etcd, or custom services) for centralized management.

Example: Fetching config from a REST API

#!/bin/bash
# fetch_config_from_api.sh

CONFIG_URL="https://config-service.example.com/prod/config"
CONFIG_FILE="/tmp/app_config.json"

# Fetch config with curl
if ! curl -sSL "$CONFIG_URL" -o "$CONFIG_FILE"; then
  echo "Failed to fetch config from $CONFIG_URL" >&2
  exit 1
fi

# Parse with jq and load into environment
export API_KEY=$(jq -r '.api_key' "$CONFIG_FILE")
export DB_CONN=$(jq -r '.database.connection_string' "$CONFIG_FILE")

echo "Config loaded from API"

2. Runtime Configuration Adjustment

To update configurations without restarting services, scripts can listen for signals (e.g., SIGUSR1) or poll for changes.

Example: Reloading config on signal

#!/bin/bash
# daemon_with_reload.sh

# Load initial config
load_config() {
  echo "Loading config..."
  CONFIG_VALUE=$(cat /etc/app/config)
}

# Handle SIGUSR1 to reload config
trap 'load_config; echo "Config reloaded: $CONFIG_VALUE"' SIGUSR1

# Initial load
load_config

# Simulate a long-running daemon
while true; do
  echo "Current config: $CONFIG_VALUE"
  sleep 5
done

Run the script, then trigger a reload with:
pkill -SIGUSR1 -f daemon_with_reload.sh

3. Configuration Validation

Invalid configurations can break systems. Scripts should validate settings before applying them.

Example: Validating a port number

#!/bin/sh
# validate_port.sh

PORT=$1

# Check if port is a number between 1 and 65535
if ! echo "$PORT" | grep -qE '^[1-9][0-9]{0,4}$' || [ "$PORT" -gt 65535 ]; then
  echo "Error: Invalid port number $PORT" >&2
  exit 1
fi

echo "Port $PORT is valid"

Common Practices

Configuration Templates with Placeholders

Use templates with placeholders (e.g., {{VAR}}) and replace them with dynamic values using envsubst, sed, or awk. This keeps configs readable and avoids duplication.

Example: Using envsubst for templates
Template file config.template.ini:

[app]
name={{APP_NAME}}
port={{APP_PORT}}

[database]
host={{DB_HOST}}

Script to generate config:

#!/bin/sh
# generate_config.sh

# Set variables (or load from environment)
APP_NAME="myapp"
APP_PORT=8080
DB_HOST="db.example.com"

# Export variables for envsubst
export APP_NAME APP_PORT DB_HOST

# Replace placeholders and output to config.ini
envsubst '{{APP_NAME}},{{APP_PORT}},{{DB_HOST}}' < config.template.ini > config.ini

envsubst is part of the gettext package and safely replaces environment variables in templates.

Environment-Specific Configurations

Maintain separate configs for environments (e.g., config.dev, config.prod) and symlink or copy the relevant file based on the environment.

Example: Selecting config by environment

#!/bin/sh
# set_env_config.sh

ENV="${1:-dev}"  # Default to 'dev' if no argument

# Validate environment
if [ ! -f "config.$ENV" ]; then
  echo "Error: config.$ENV not found" >&2
  exit 1
fi

# Symlink active config to config.current
ln -sf "config.$ENV" config.current
echo "Active config: $(readlink config.current)"

Modular Configuration with Include Files

Break large configs into smaller, reusable files and source them with . or source for modularity.

Example: Sourcing include files
config/base.sh:

LOG_LEVEL="info"
MAX_RETRIES=3

config/prod.sh:

. ./config/base.sh  # Source base config
LOG_LEVEL="warn"   # Override base value
DB_HOST="prod-db.example.com"

Main script:

#!/bin/bash
# load_modular_config.sh

ENV="prod"
source "config/$ENV.sh"

echo "Log level: $LOG_LEVEL"
echo "DB Host: $DB_HOST"

Best Practices

Security: Protect Sensitive Data

• Avoid hardcoding secrets: Never embed passwords, API keys, or tokens in scripts. Use environment variables, encrypted files, or secret managers (e.g., HashiCorp Vault).
  Example: Use vault read -field=token secret/myapp to fetch secrets at runtime.
• Restrict file permissions: Ensure config files have tight permissions (e.g., chmod 600 config.secret).
• Encrypt sensitive files: Use gpg or openssl to encrypt secrets at rest:
  openssl enc -aes-256-cbc -in config.secret -out config.secret.enc

Idempotency: Scripts That “Just Work”

Ensure scripts can run multiple times without side effects (e.g., avoid overwriting files if they already exist).

Example: Idempotent file creation

#!/bin/sh
# create_config_idempotent.sh

CONFIG_FILE="/etc/app/config"

# Only write if file doesn't exist or content differs
if ! grep -q "ENABLED=true" "$CONFIG_FILE" 2>/dev/null; then
  echo "ENABLED=true" >> "$CONFIG_FILE"
  echo "Updated config"
else
  echo "Config already up to date"
fi

Logging and Debugging

• Log actions: Write to stdout/stderr or log files for visibility:
  echo "[$(date)] Generated config: $CONFIG_FILE" >> /var/log/config_manager.log
• Enable debugging: Use set -x to trace execution or set -e to exit on errors:
  #!/bin/bash -eux ( -e exit on error, -u treat unset vars as error, -x debug trace)

Testing and Validation

• Lint scripts: Use shellcheck to catch syntax errors and bad practices:
  shellcheck my_script.sh
• Unit testing: Use frameworks like shunit2 or bats to test config logic.
  Example shunit2 test for port validation:

  test_valid_port() {
    ./validate_port.sh 8080
    assertEquals "Port 8080 should be valid" 0 $?
  }

• Integration testing: Validate end-to-end config deployment in staging environments.

Portability

• Use POSIX-compliant syntax (e.g., #!/bin/sh instead of #!/bin/bash) for wider compatibility.
• Avoid Bash-specific features (e.g., arrays, [[ ]]) if targeting minimal systems (e.g., Alpine Linux).
• Test scripts on target OSes (e.g., Ubuntu, CentOS, macOS).

Conclusion

Dynamic Configuration Management with shell scripts offers a pragmatic, lightweight solution for adapting systems to changing requirements. By leveraging environment variables, config files, templates, and external services, shell scripts can handle everything from simple app deployments to complex runtime adjustments.

While tools like Ansible or Kubernetes are better suited for large-scale infrastructure, shell scripts excel in scenarios where simplicity, speed, and control are prioritized. By following best practices—securing secrets, ensuring idempotency, testing rigorously—you can build robust, maintainable configuration workflows that scale with your needs.

References

• ShellCheck: Static analysis tool for shell scripts.
• shunit2: Unit testing framework for shell scripts.
• envsubst: Substitute environment variables in templates.
• yq: YAML parser for the command line.
• OWASP Secrets Management Cheat Sheet: Guidelines for secure secret handling.
• POSIX Shell Specification: Standards for portable shell scripting.