dotlinux guide

Shell Scripting with a Twist: Utilizing APIs and Web Services

Table of Contents

  1. Fundamentals: Shell Scripting + APIs
  2. Tools of the Trade
  3. Practical Usage: Step-by-Step Examples
  4. Common Use Cases
  5. Best Practices
  6. Conclusion
  7. References

1. Fundamentals: Shell Scripting + APIs

1.1 Shell Scripting Basics

A shell script is a text file containing a sequence of commands executed by a shell (e.g., Bash, Zsh). It automates tasks like file manipulation, process management, and system configuration. Key concepts include:

  • Shebang: #!/bin/bash (specifies the shell interpreter).
  • Variables: Store data (e.g., API_URL="https://api.example.com").
  • Control Flow: Conditionals (if/else), loops (for/while), and functions.
  • Command Execution: Run external tools (e.g., curl, grep) and capture output (e.g., response=$(curl ...)).

1.2 APIs and Web Services Basics

An API is a set of rules that allows one software application to interact with another. Web APIs (e.g., REST, GraphQL) enable communication over HTTP/HTTPS. Key terms:

  • Endpoint: A URL (e.g., https://api.example.com/users) that accepts requests.
  • HTTP Methods: Define actions (e.g., GET to retrieve data, POST to send data, PUT to update, DELETE to remove).
  • Status Codes: Indicate request success/failure (e.g., 200 OK, 404 Not Found, 500 Server Error).
  • Response Formats: Typically JSON (JavaScript Object Notation) or XML (eXtensible Markup Language); JSON is most common.

1.3 Why Combine Shell Scripts with APIs?

  • Extend Reach: Interact with cloud services (AWS, Slack), databases (Firebase), or third-party tools (GitHub, Twilio) from the command line.
  • Automate End-to-End Workflows: For example, a script that fetches data from an API, processes it locally, and sends results to another service.
  • Simplify Complex Tasks: Avoid building custom clients—use existing APIs to offload work (e.g., sending emails via SendGrid’s API).

2. Tools of the Trade

2.1 curl: The Swiss Army Knife for HTTP Requests

curl is a command-line tool for transferring data with URLs. It supports HTTP, HTTPS, FTP, and more, making it ideal for API interactions.

Basic Syntax:

curl [options] [URL]

Common Options:

  • -X <method>: Specify HTTP method (e.g., -X POST).
  • -H <header>: Add HTTP headers (e.g., -H "Content-Type: application/json").
  • -d <data>: Send data (e.g., JSON payloads for POST).
  • -f: Fail silently (no output) on HTTP errors (e.g., 4xx/5xx).
  • -s: Silent mode (suppress progress bars).
  • -o <file>: Save response to a file.

2.2 jq: JSON Parsing and Manipulation

APIs often return JSON, which is hard to parse with basic shell tools. jq is a lightweight command-line JSON processor that extracts, filters, and transforms JSON data.

Installation:

  • Linux: sudo apt install jq (Debian/Ubuntu) or sudo yum install jq (RHEL/CentOS).
  • macOS: brew install jq.

Basic Syntax:

jq '<filter>' [file.json]

Examples:

  • Extract a field: jq '.name' (from {"name":"Alice", "age":30} returns "Alice").
  • Filter an array: jq '.users[] | select(.age > 25)' (returns users over 25).
  • Get a nested value: jq '.data[0].id' (from {"data": [{"id":1}, {"id":2}]} returns 1).

2.3 Honorable Mentions

  • httpie: A user-friendly alternative to curl with color-coded output (e.g., http GET https://api.example.com).
  • openssl: Handle SSL/TLS (e.g., decode certificates for HTTPS endpoints).
  • base64: Encode credentials for Basic Auth (e.g., echo -n "user:pass" | base64).

3. Practical Usage: Step-by-Step Examples

3.1 Making GET Requests and Parsing JSON

Let’s fetch data from JSONPlaceholder (a free fake API for testing) and parse it with jq.

Example: Fetch a User Profile

#!/bin/bash
API_URL="https://jsonplaceholder.typicode.com/users/1"

# Fetch user data (silent mode, fail on error)
response=$(curl -s -f "$API_URL")

# Check if curl succeeded
if [ $? -ne 0 ]; then
  echo "Error: Failed to fetch user data"
  exit 1
fi

# Parse JSON with jq
name=$(echo "$response" | jq -r '.name')  # -r = raw output (no quotes)
email=$(echo "$response" | jq -r '.email')

echo "User: $name"
echo "Email: $email"

Output:

User: Leanne Graham
Email: [email protected]

3.2 Sending Data with POST Requests

Use POST to send data (e.g., create a new resource).

Example: Create a Post

#!/bin/bash
API_URL="https://jsonplaceholder.typicode.com/posts"
TITLE="My Shell Script Post"
BODY="Hello from curl!"
USER_ID=1

# Send JSON payload with POST
response=$(curl -s -f -X POST "$API_URL" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "'"$TITLE"'",
    "body": "'"$BODY"'",
    "userId": '"$USER_ID"'
  }')

# Parse and print the created post ID
post_id=$(echo "$response" | jq -r '.id')
echo "Created post with ID: $post_id"  # Output: Created post with ID: 101 (JSONPlaceholder fakes this)

3.3 Handling Authentication

APIs often require authentication. Common methods include:

API Key (in Header)

Many APIs use an API key in the Authorization header.

#!/bin/bash
API_KEY="your-secret-key"
API_URL="https://api.example.com/data"

response=$(curl -s -f -H "Authorization: Bearer $API_KEY" "$API_URL")

Basic Authentication

Encode username:password as base64 and send in the Authorization header.

#!/bin/bash
USER="admin"
PASS="secure-password"
API_URL="https://api.example.com/secret"

# Encode credentials (avoid echoing with -n to omit newline)
credentials=$(echo -n "$USER:$PASS" | base64)

response=$(curl -s -f -H "Authorization: Basic $credentials" "$API_URL")

4. Common Use Cases

4.1 API Monitoring and Health Checks

Monitor API uptime, response time, and status codes.

Example: Check API Health

#!/bin/bash
API_URL="https://api.example.com/health"
MAX_RESPONSE_TIME=2  # seconds

# Measure response time and status code
start_time=$(date +%s)
status_code=$(curl -s -o /dev/null -w "%{http_code}" -f "$API_URL" --max-time "$MAX_RESPONSE_TIME")
end_time=$(date +%s)
response_time=$((end_time - start_time))

# Evaluate results
if [ "$status_code" -eq 200 ] && [ "$response_time" -lt "$MAX_RESPONSE_TIME" ]; then
  echo "API is healthy (Status: $status_code, Time: ${response_time}s)"
else
  echo "API DOWN! (Status: $status_code, Time: ${response_time}s)"
  # Send alert (see 4.2)
fi

4.2 Automated Notifications

Send alerts via Slack, Email, or SMS APIs when scripts fail.

Example: Send Slack Notification
Slack’s Incoming Webhooks let you post messages to a channel.

#!/bin/bash
SLACK_WEBHOOK="https://hooks.slack.com/services/XXX/YYY/ZZZ"  # Replace with your webhook
MESSAGE="⚠️ API Health Check Failed! Status Code: 500"

curl -s -X POST "$SLACK_WEBHOOK" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "'"$MESSAGE"'"
  }'

4.3 Data Backup and Sync

Sync local files to cloud storage (e.g., AWS S3, Google Drive) via their APIs.

Example: Sync Logs to S3 (Simplified)
Using AWS CLI (which wraps S3’s API):

#!/bin/bash
LOG_DIR="/var/log/app"
S3_BUCKET="my-backup-bucket/logs"

# Sync logs to S3 (requires AWS CLI configured)
aws s3 sync "$LOG_DIR" "s3://$S3_BUCKET" --delete

5. Best Practices

5.1 Error Handling

  • Check Exit Codes: Use set -euo pipefail to exit on errors, unset variables, or failed pipeline commands. 
    #!/bin/bash
set -euo pipefail  # Strict error checking
  • Validate HTTP Status Codes: Use curl -w "%{http_code}" to capture status codes and check for 200-299 (success).
  • Handle Retries: Use curl --retry 3 --retry-delay 5 to retry failed requests.

5.2 Security

  • Avoid Hardcoding Secrets: Store credentials in environment variables or secure vaults (e.g., HashiCorp Vault). 
    # Bad: Hardcoded key
# API_KEY="secret"

# Good: Use environment variable
API_KEY="${API_KEY:-}"  # Exit if not set (thanks to set -u)
  • Use HTTPS: Always prefer https:// to encrypt data in transit.
  • Limit Permissions: Restrict API keys to minimal required scopes (e.g., “read-only”).

5.3 Efficiency and Reliability

  • Reuse Connections: Use curl --keepalive-time 60 to reuse TCP connections for multiple requests.
  • Respect Rate Limits: Avoid overwhelming APIs (check X-RateLimit-Remaining headers and add delays with sleep).
  • Cache Responses: Save frequent API responses to a file (e.g., curl -o cache.json ... and check timestamps before re-fetching).

5.4 Readability and Maintainability

  • Modularize with Functions: Break logic into reusable functions. 
    fetch_data() {
  local url="$1"
  curl -s -f "$url"
}
  • Add Comments: Explain why, not just what.
  • Use Consistent Formatting: Indentation and naming conventions (e.g., uppercase for constants: API_URL).

6. Conclusion

Shell scripting with APIs transforms humble command-line tools into powerful automation engines. By combining curl for HTTP requests, jq for JSON parsing, and shell logic, you can integrate with web services, monitor systems, send notifications, and automate cross-platform workflows.

To master this, start small (e.g., fetching public APIs), experiment with authentication, and adopt best practices like error handling and security. The possibilities are endless—from home automation to enterprise-grade DevOps pipelines.

7. References