How to Timeout HTTP Requests in Linux

Table of Contents#

1. Understanding HTTP Timeouts
2. Timeout with curl
3. Timeout with wget
4. Timeout with httpie
5. Using the timeout Command
6. Timeout in Python (with requests Library)
7. Bash Scripting with Timeouts
8. Network-Level Timeouts (iptables and tc)
9. Best Practices for HTTP Timeouts
10. Conclusion
11. References

Understanding HTTP Timeouts#

Before diving into tools, it’s critical to distinguish between two key timeout types:

• Connect Timeout: The time allowed to establish a TCP connection with the server (e.g., DNS resolution, TCP handshake).
• Total Timeout: The maximum time allowed for the entire request lifecycle, including connection, data transfer, and response processing.

Most tools let you configure both, but the syntax varies. Let’s explore how to implement these in Linux.

Timeout with curl#

curl is a versatile command-line tool for making HTTP/HTTPS requests. It offers granular control over timeouts via two primary flags:

Key Flags:#

• --connect-timeout <seconds>: Timeout for TCP connection establishment (default: 300 seconds).
• --max-time <seconds>: Total timeout for the entire request (including connection, transfer, and response).

Example 1: Basic Timeout#

Terminate the request if the connection isn’t established within 5 seconds, or if the total request takes longer than 10 seconds:

curl --connect-timeout 5 --max-time 10 https://example.com

Example 2: Timeout with Retries#

Retry the request up to 3 times (with 2-second delays) if it times out:

curl --connect-timeout 5 --max-time 10 --retry 3 --retry-delay 2 https://example.com

Notes:#

• --connect-timeout only affects the TCP handshake; it does not account for slow data transfer.
• --max-time is a hard limit for the entire request. Use this to prevent long-running transfers.

Timeout with wget#

wget is another popular tool for downloading files over HTTP/HTTPS. It uses the --timeout flag to control timeouts, with additional options for retries.

Key Flags:#

• --timeout=<seconds>: Timeout for connection and data transfer (applies to both connect and idle time).
• --tries=<num>: Number of retries (default: 20; use --tries=1 to disable retries).

Example: Timeout and Retry#

Set a 5-second timeout for connection/idle time, and retry once:

wget --timeout=5 --tries=1 https://example.com/file.zip

Notes:#

• Unlike curl, wget’s --timeout is a single value that applies to both connection establishment and periods of inactivity during transfer.
• To simulate a "total timeout," combine --timeout with --tries=1 (no retries) to limit the request to one attempt.

Timeout with httpie#

httpie is a modern, user-friendly alternative to curl with syntax that mirrors HTTP methods. It supports timeouts via the --timeout flag.

Key Flag:#

• --timeout=<seconds>: Total timeout for the request (includes connection and transfer).

Example:#

Set a 10-second total timeout for a GET request:

http --timeout=10 GET https://example.com/api/data

Notes:#

• httpie does not separate connect and total timeouts; --timeout applies to the entire request.

Using the timeout Command#

The timeout command (part of the coreutils package) is a generic tool to terminate any process after a specified duration. It works with any command-line tool, making it ideal for tools that lack built-in timeout options.

Syntax:#

timeout <duration> <command>

Example 1: Timeout curl with timeout#

Terminate curl after 10 seconds (regardless of curl’s internal timeouts):

timeout 10s curl https://example.com

Example 2: Timeout with Graceful Termination#

Send a SIGTERM after 10 seconds, and a SIGKILL if the process doesn’t exit within 2 more seconds:

timeout --kill-after=2s 10s wget https://example.com/large-file.iso

Notes:#

• Use suffixes like s (seconds), m (minutes), or h (hours) for duration (e.g., 5m = 5 minutes).
• timeout is useful for legacy tools or scripts that don’t support native timeouts.

Timeout in Python (with requests Library)#

For programmatic HTTP requests, Python’s requests library is widely used. It provides a timeout parameter to enforce timeouts.

Key Parameter:#

• timeout=<seconds>: Total timeout for the request (connect + read). Can also pass a tuple (connect_timeout, read_timeout) for granular control.

Example 1: Basic Timeout#

import requests
 
try:
    response = requests.get("https://example.com", timeout=5)  # 5-second total timeout
    print("Request succeeded:", response.status_code)
except requests.exceptions.Timeout:
    print("Request timed out")
except requests.exceptions.RequestException as e:
    print("Request failed:", e)

Example 2: Separate Connect and Read Timeouts#

# 3-second connect timeout, 7-second read timeout (total max 10 seconds)
response = requests.get("https://example.com", timeout=(3, 7))

Notes:#

• The timeout parameter in requests is mandatory for production code to avoid hanging processes.
• Always wrap requests in a try/except block to handle timeout exceptions.

Bash Scripting with Timeouts#

In Bash scripts, combine tools like curl or wget with timeout to build robust workflows. Here’s an example of a script that retries a request with timeouts:

Example: Retry with Timeout#

#!/bin/bash
 
URL="https://example.com/api/health"
MAX_RETRIES=3
TIMEOUT=5  # Seconds per attempt
 
for ((i=1; i<=MAX_RETRIES; i++)); do
    echo "Attempt $i/$MAX_RETRIES..."
    if timeout $TIMEOUT curl --silent --show-error $URL; then
        echo "Request succeeded!"
        exit 0
    fi
    sleep 2  # Wait before retrying
done
 
echo "All attempts failed."
exit 1

Notes:#

• Use --silent with curl to suppress output, and --show-error to display errors if they occur.
• Add delays (sleep) between retries to avoid overwhelming the server.

Network-Level Timeouts (iptables and tc)#

For system-wide or low-level control, use Linux network tools like iptables (firewall) and tc (traffic control) to enforce timeouts.

1. iptables: TCP Connection Timeouts#

iptables can set timeouts for established TCP connections using the --timeout flag in the state module.

Example: Close idle HTTP (port 80) connections after 60 seconds:

sudo iptables -A INPUT -p tcp --dport 80 -m state --state ESTABLISHED -j ACCEPT --timeout 60

2. tc: Simulate Timeouts (for Testing)#

The tc (traffic control) tool can introduce delays or packet loss to simulate network issues, effectively causing timeouts for testing.

Example: Add 2-second delay and 50% packet loss on eth0 (simulate a slow/unreliable network):

sudo tc qdisc add dev eth0 root netem delay 2000ms loss 50%

To reset:

sudo tc qdisc del dev eth0 root

Notes:#

• iptables timeouts apply system-wide and are useful for enforcing idle connection limits.
• tc is for testing/debugging; use it to validate how your application handles timeouts.

Best Practices for HTTP Timeouts#

1. Choose Appropriate Values:

   • Connect timeouts: 5–10 seconds (most connections establish quickly).
   • Total timeouts: 10–30 seconds (depends on expected response size; longer for large files).

2. Handle Retries Carefully:

   • Retry transient failures (e.g., 5xx errors) but avoid spamming servers. Use exponential backoff (e.g., 1s, 2s, 4s delays).

3. Log Timeouts:

   • Log timeout events for monitoring (e.g., with logger in Bash or Python’s logging module).

4. Test Edge Cases:

   • Use tc to simulate slow networks and validate timeout behavior.

5. Avoid Overriding System Limits:

   • If using iptables or sysctl to set network timeouts, ensure they don’t conflict with application-level timeouts.

Conclusion#

Timeout HTTP requests are critical for building resilient systems. Linux offers a range of tools to enforce timeouts, from command-line utilities like curl and wget to programming libraries like Python’s requests and network-level tools like iptables. By choosing the right method for your use case and following best practices, you can prevent resource leaks, improve reliability, and ensure your applications handle network issues gracefully.

References#

• curl Man Page
• wget Man Page
• HTTPie Documentation
• Python requests Timeout Docs
• Linux timeout Command
• iptables Man Page
• tc (Traffic Control) Man Page