dotlinux guide

Linux Command Line for Data Science: Tools and Techniques

In the realm of data science, efficiency, scalability, and automation are paramount. While graphical user interfaces (GUIs) have their place, the Linux command line (CLI) remains an indispensable tool for handling large datasets, automating workflows, and integrating with data science pipelines. Unlike GUIs, CLI tools are lightweight, scriptable, and designed to process data in streams—making them ideal for preprocessing raw data, managing files, and orchestrating complex tasks. Whether you’re cleaning a messy CSV, monitoring a long-running model training job, or automating data pipelines, proficiency with the Linux CLI can drastically reduce manual effort and accelerate your workflow. This blog will guide you through fundamental CLI concepts, essential tools for data manipulation, advanced techniques for large datasets, and best practices tailored to data science use cases.

Table of Contents

  1. Fundamentals of Linux Command Line
  2. Essential Data Manipulation Tools
  3. Text Processing and Data Cleaning Pipelines
  4. Handling Large Datasets Efficiently
  5. Process Management for Data Science Workflows
  6. Version Control with Git CLI
  7. Automation with Bash Scripting
  8. Best Practices
  9. Conclusion
  10. References

Fundamentals of Linux Command Line

Before diving into data-specific tools, mastering the basics of navigation and file management is critical. The Linux file system is hierarchical, with / (root) as the top-level directory. Below are key commands to orient yourself.

  • pwd (Print Working Directory): Shows your current directory.

    pwd  # Output: /home/data_scientist/projects

  • cd (Change Directory): Move between directories.

    cd projects  # Move to "projects" subdirectory
cd ..        # Move up one level (parent directory)
cd /tmp      # Absolute path: move to /tmp

  • ls (List Directory Contents): List files and folders. Use flags for details:

    • -l: Long format (permissions, size, modified time).
    • -h: Human-readable sizes (e.g., 1K, 2M).
    • -a: Show hidden files (start with .).
    ls -lha  # List all files with details and human-readable sizes

File and Directory Operations

  • mkdir (Make Directory): Create new directories.

    mkdir raw_data processed_data  # Create two directories

  • touch: Create empty files or update timestamps.

    touch raw_data/2023_sales.csv  # Create an empty CSV

  • cp (Copy): Copy files/directories. Use -r for directories.

    cp raw_data/2023_sales.csv processed_data/  # Copy file to processed_data
cp -r raw_data/ backup/                     # Copy entire directory

  • mv (Move/Rename): Move files or rename them.

    mv processed_data/2023_sales.csv processed_data/cleaned_sales.csv  # Rename
mv cleaned_sales.csv ../archive/                                   # Move

  • rm (Remove): Delete files/directories (use with caution!).

    rm old_logs.txt               # Delete a file
rm -r obsolete_data/          # Delete a directory (recursive)

Viewing File Contents

Quickly inspect files without opening a text editor:

  • cat: Print entire file content.

    cat cleaned_sales.csv  # Print CSV content to terminal

  • head/tail: View first/last N lines (default: 10).

    head -5 cleaned_sales.csv  # First 5 lines
tail -3 cleaned_sales.csv  # Last 3 lines
tail -f logs/training.log  # "Follow" a log file (updates in real time)

  • less: Scroll through large files interactively (press q to exit).

    less large_dataset.csv  # Navigate with arrow keys, search with /pattern

Essential Data Manipulation Tools

Data science often involves searching, filtering, and transforming text-based data (e.g., CSVs, logs). These CLI tools are workhorses for such tasks.

grep: Searching Text Patterns

grep searches for patterns in files or input streams. Use regular expressions (regex) for flexibility.

Common flags:

  • -i: Case-insensitive search.
  • -n: Show line numbers.
  • -E: Enable extended regex (supports |, (), etc.).

Examples:

# Search for "error" in a log file (case-insensitive)
grep -i "error" training.log

# Find lines with "warning" OR "critical" (extended regex)
grep -E "warning|critical" system.log

# Search for "NA" values in a CSV and show line numbers
grep -n "NA" raw_data/survey.csv

awk: Processing Structured Data

awk is a powerful language for processing structured text (e.g., CSVs, TSVs). It splits input into columns (default: whitespace) and applies actions to rows.

Basic syntax: awk 'pattern { action }' file

Examples:

  • Print the 3rd column of a CSV (use -F ',' to set comma as delimiter):

    awk -F ',' '{ print $3 }' sales.csv  # Print 3rd column

  • Filter rows where the 2nd column (e.g., “revenue”) is > 1000:

    awk -F ',' '$2 > 1000 { print $1, $2 }' sales.csv  # Print rows with revenue > 1000

  • Compute the average of the 4th column:

    awk -F ',' '{ sum += $4; count++ } END { print "Average:", sum/count }' metrics.csv

sed: Stream Editing

sed (stream editor) modifies text in a pipeline. Common uses: search-and-replace, deleting lines, or inserting text.

Basic syntax: sed 's/old/new/flags' file (substitute old with new).

Examples:

  • Replace “N/A” with “Missing” in a CSV:

    sed 's/N\/A/Missing/g' survey.csv > cleaned_survey.csv  # "g" = global (all occurrences)

  • Delete lines containing “irrelevant” (in-place edit with -i):

    sed -i '/irrelevant/d' raw_data/logs.txt  # "-i" edits the file directly

Text Processing and Data Cleaning Pipelines

The true power of CLI tools lies in combining them with pipes (|), which pass the output of one command as input to the next. This creates efficient data-cleaning pipelines.

cut: Extracting Columns

cut extracts specific columns from text. Use -f to specify columns and -d for delimiters.

Example: Extract columns 1 (ID) and 4 (age) from a TSV:

cut -d $'\t' -f 1,4 user_data.tsv  # TSV uses tab delimiter ($'\t')

sort and uniq: Organizing and Deduplicating Data

  • sort: Sort lines alphabetically/numerically. Use -n for numeric sort, -r for reverse.
  • uniq: Remove duplicate lines (requires sorted input). Use -c to count occurrences.

Examples:

# Sort a CSV by the 3rd column (numeric, reverse order)
sort -t ',' -k 3nr sales.csv  # -t ',' = delimiter, -k 3 = 3rd column, n = numeric, r = reverse

# Count unique values in the 2nd column (after sorting)
cut -d ',' -f 2 sales.csv | sort | uniq -c

Combining Tools with Pipes

Pipes (|) chain commands to build pipelines. For example, clean a messy CSV in one line:

Scenario: Extract user IDs, filter active users, sort, and count unique IDs.

# Step 1: Extract 1st column (ID) and 4th column (status)
# Step 2: Keep rows where status is "active"
# Step 3: Extract ID (1st column of the filtered output)
# Step 4: Sort IDs and count unique values
cut -d ',' -f 1,4 users.csv | awk -F ',' '$2 == "active" { print $1 }' | sort | uniq -c

Handling Large Datasets Efficiently

Data scientists often work with files too large for Excel or even pandas to load. CLI tools handle this gracefully.

split: Breaking Large Files

split divides large files into smaller chunks (e.g., 1GB each).

Example: Split a 10GB CSV into 1GB chunks named chunk_00, chunk_01, etc.:

split -b 1G large_data.csv chunk_  # -b: size in bytes (1G = 1 gigabyte)

pv: Monitoring Progress

pv (Pipe Viewer) shows progress bars for data transfers/pipelines. Install first: sudo apt install pv (Debian/Ubuntu) or brew install pv (macOS).

Example: Monitor a file copy:

pv large_file.csv > /backup/large_file.csv  # Shows speed, ETA, progress bar

Working with Compressed Files

Compressed files (e.g., .gz, .zip) save space. Use these tools to avoid uncompressing first:

  • zcat/gzcat: Read gzipped files (.gz) directly.

    zcat data.csv.gz | head  # View first lines without uncompressing

  • tar: Archive/extract multiple files (use -z for gzip compression).

    # Extract a .tar.gz archive
tar -xzf dataset.tar.gz

# Compress a directory into a .tar.gz
tar -czf backup.tar.gz raw_data/ processed_data/

Process Management for Data Science Workflows

Long-running tasks (e.g., model training, data scraping) require monitoring and background execution.

Monitoring Processes

  • ps: List running processes (use -ef for details).

    ps -ef | grep "python train.py"  # Find your Python training script

  • top/htop: Real-time system monitor (CPU, memory usage). htop is more user-friendly (install with sudo apt install htop).

Background Jobs and Long-Running Tasks

  • Run a command in the background: Append &.

    python train_model.py > training.log &  # Run script in background, log output

  • nohup (No Hangup): Keep processes running after closing the terminal.

    nohup python scrape_data.py > scrape.log 2>&1 &  # 2>&1: redirect errors to log

  • jobs: List background jobs.

  • fg %1: Bring job 1 to the foreground.

Version Control with Git CLI

Git is essential for tracking code and pipeline changes. Key commands:

# Initialize a repo
git init

# Track files and commit changes
git add preprocess.sh  # Stage a file
git commit -m "Add data cleaning script"  # Commit with message

# Push to remote (e.g., GitHub)
git remote add origin https://github.com/your/repo.git
git push -u origin main

Automation with Bash Scripting

Bash scripts automate repetitive tasks (e.g., daily data downloads, preprocessing).

Example: Data Preprocessing Script
Save as preprocess.sh:

#!/bin/bash
# Preprocess raw data: clean, filter, and save to processed_data/

# Input/output paths (passed as arguments)
RAW_DATA=$1
PROCESSED_DATA=$2

# Check if input file exists
if [ ! -f "$RAW_DATA" ]; then
  echo "Error: $RAW_DATA not found!"
  exit 1
fi

# Pipeline: clean NA values, filter rows, sort, and save
echo "Preprocessing $RAW_DATA..."
sed 's/NA/Missing/g' "$RAW_DATA" | \
awk -F ',' '$3 > 0 { print }' | \
sort -t ',' -k 2 > "$PROCESSED_DATA"

echo "Done! Saved to $PROCESSED_DATA"

Run the script:

chmod +x preprocess.sh  # Make executable
./preprocess.sh raw_data/sales.csv processed_data/clean_sales.csv

Best Practices

  1. Organize Directories: Use a consistent structure:

    project/
├── raw_data/         # Original, immutable data
├── processed_data/   # Cleaned/transformed data
├── scripts/          # Bash/Python/R scripts
└── logs/             # Process/output logs

  2. Document Commands: Save complex pipelines in a README.md (e.g., “To clean data: ./preprocess.sh ...”).

  3. Use Aliases: Shortcut common commands (add to ~/.bashrc or ~/.zshrc):

    alias ll='ls -lha'
alias clean_data='./scripts/preprocess.sh'

  4. Avoid rm -rf: Use rm -i (interactive) or trash-cli (trash-put) instead of irreversible deletions.

  5. Backup Data: Use rsync or cloud storage (e.g., AWS S3) to back up raw_data/.

Conclusion

The Linux command line is a data scientist’s Swiss Army knife—fast, flexible, and indispensable for handling large datasets and automating workflows. By mastering tools like awk, sed, and grep, and combining them with pipes, you can preprocess data, monitor jobs, and build robust pipelines without leaving the terminal. Paired with bash scripting and Git, CLI skills elevate productivity and reproducibility in data science projects.

Start small: next time you need to clean a CSV, try a CLI pipeline instead of Python. With practice, these tools will become second nature.

References