Backup Manager v1.0

Executive Summary

Backup Manager is a FastAPI service running in Docker on dellserver. It replaces cron-driven bash archival scripts with a Python-native scheduler, REST API, and web dashboard. It pulls home directories from all Linux hosts via SSH/rsync and produces three backup layers per host: rolling hourly hardlink snapshots (Time Machine-style), compressed tar archives, and monthly golden snapshots.

The scheduler fires every hour. All hosts are processed in parallel via ThreadPoolExecutor. Because rsync and gzip release the Python GIL, the real work runs as concurrent OS processes on separate cores.

Deployment Topology

Docker Container

Single service defined in docker-compose.yml. restart: always ensures survival across host reboots.

Volume Mounts

Port Mapping

All configuration injected via .env file. TZ=America/New_York set inline for consistent timestamp formatting.

FastAPI Application

Entry point: app/main.py. Wires three routers and manages the scheduler lifecycle through FastAPI's lifespan async context manager.

Scheduler

APScheduler BackgroundScheduler with IntervalTrigger. One job (run_backup_job) fires every SNAPSHOT_INTERVAL_HOURS (default: 1 hour).

Deduplication Guard

A threading.Lock (_job_lock) prevents concurrent runs. If the scheduler fires while a previous job is still in progress (e.g., slow network), the duplicate is dropped with a warning. Same guard applies to manual API triggers.

if not _job_lock.acquire(blocking=False):
    logger.warning("Backup job already running — ignoring duplicate trigger")
    return False

Backup Pipeline

The pipeline is split into two independent scheduler jobs that run concurrently. Both spawn one thread per host via ThreadPoolExecutor.

Job 1 — Hourly Snapshots (run_snapshot_job(), every 1 h)

The primary data-pull mechanism. Rsyncs directly from the remote host into a new snapshot directory, using --link-dest to hardlink unchanged files against the previous snapshot. Only deltas transfer over the wire.

After a successful snapshot, current/ is updated to be a symlink pointing to the latest complete snapshot. This means current/ is always a consistent, read-only view of the most recent data — no in-place modifications.

rsync -aHu --link-dest={prev_snapshot}/ {host}:~/ {snapshot_dir}/
# Then: current → symlink → hourly/latest-snapshot/

A .snapshot_complete marker is written on success (crash safety). Partial snapshots from crashes are cleaned up automatically. The latest and current symlinks are updated atomically via tmp-rename. Idempotent: if this hour's snapshot already exists with the marker, it is skipped.

Job 2 — Heavy Pipeline (run_heavy_job(), every 4 h)

Reads from current/ (symlink to latest snapshot). No rsync — data pull is handled entirely by Job 1.

■ Protected (never cancelled)   ■ Cancellable per-host

Tar archive

Compresses current/ to a .tar.gz (gzip level 6). Skipped if the most recent archive is less than TAR_INTERVAL_HOURS (4 h) old.

tar -czf backup_{host}_YYYY-MM-DD_HH00.tar.gz -C {current_dir} .

Monthly snapshot

On the 1st of each month (or immediately if no snapshots exist yet for this host), creates a hardlinked golden copy at monthly/YYYY-MM/.

Cleanup

Removes expired tar archives (older than TAR_RETENTION_DAYS), removes expired monthly snapshots (older than RETENTION_MONTHS).

Cancellation

Each host has an independent cancel flag (_cancel_flags[host]). Stages 2-5 check the flag before starting. Mid-stage subprocesses are terminated via proc.terminate() with a 2-second polling loop. Partial output (incomplete tar file, incomplete snapshot dir) is cleaned up immediately.

Storage — Directory Layout

/mnt/THE_BIG_DRIVE/________MAIN_LINUX_BACKUP/
  {host}/
    current/                      ← live rsync mirror of remote ~/
    latest -> hourly/SNAPSHOT     ← atomic symlink to most recent complete snapshot
    hourly/
      2026-03-06_0800/            ← hardlink snapshot (Time Machine-style)
        .snapshot_complete        ← crash-safety marker
        home/ ...                 ← files (hardlinked where unchanged)
      2026-03-06_0700/
      ...
    archives/
      backup_{host}_2026-03-06_0800.tar.gz
      backup_{host}_2026-03-05_2000.tar.gz
      ...
    monthly/
      2026-03/                    ← golden hardlink snapshot
      2026-02/
      ...

Disk Space Characteristics

Retention Policy

Hourly Snapshots — Tiered Retention

Single oldest-first pass. Each snapshot falls into exactly one tier:

Tar Archives — Rolling Window

Archives older than TAR_RETENTION_DAYS (5 days) are removed. New archives created only if TAR_INTERVAL_HOURS (4 h) have elapsed.

Monthly Snapshots — Age-Based

Directories named YYYY-MM older than RETENTION_MONTHS (24) are removed by lexicographic comparison to a cutoff string.

REST API

Status Response Schema

{
  "job_running": bool,
  "hosts": {
    "{host}": {
      "status":         "running" | "ok" | "failed" | "cancelled" | "dry_run" | "dry_run_ok",
      "stage":          "rsync" | "hourly" | "tar" | "monthly" | "cleanup" | "done",
      "stage_num":      1..5,
      "stage_total":    5,
      "started":        "ISO-8601",
      "finished":       "ISO-8601",   // absent while running
      "hourly":         {...} | null,
      "tar":            {...} | null,
      "snapshot":       str | null,
      "archives_removed": int
    }
  }
}

Dashboard

Served at / as server-side rendered HTML via Jinja2. Auto-refreshes every 10 seconds when a job is running.

/architecture serves this document as a static file (docs/architecture.html). /architecture-dummies serves the slide presentation.

PostgreSQL + PostgREST

Persistent job history, snapshot metadata, and anomaly detection state. All status was previously in-memory dicts (lost on restart). The database provides historical queries and feeds the dashboard's job history table.

Schema

Configuration

Hosts

Exclusion Patterns

Applied as --exclude flags to every rsync call. Targets ephemeral data, caches, build artifacts, and large binaries with no recovery value.