Graph View

Changelog

drop-cloudflared-plan

9 min read

Drop cloudflared: Direct Exposure Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Remove cloudflared; expose Immich/pad via nginx secret gate and SSH via hardened keys-only sshd, directly from Tower.

Architecture: Public 443 → nginx (wildcard LE cert, secret-gate map, return 444 otherwise) → Immich/pad. Public 22 → sshd (keys-only, GitHub AuthorizedKeysCommand trust root, fail2ban). Split-horizon DNS: Pi-hole resolves public hostnames to 10.0.0.100 on LAN. Spec: drop-cloudflared-direct-exposure.md.

Tech Stack: nginx, OpenSSH, certbot + python3-certbot-dns-cloudflare (DNS-01), fail2ban, unattended-upgrades. No other new dependencies.

Global Constraints

  • Lockout safety: cloudflared stays running until Task 4 verifies direct SSH from off-LAN. Keep one ssh lab session open across any sshd restart.
  • No secrets in repo: configs use __LAB_KEY__ / <CF_TOKEN> placeholders (matches existing <TUNNEL_UUID> pattern); real values live only on Tower/phones/password manager.
  • Port 80 is never opened; certs via DNS-01 only.
  • Tower user is ns; run Tower commands via ssh lab. GitHub user: Suputra.
  • USER marks steps only Saahas can do (router UI, CF dashboard, phones).

Task 1: Repo config files

Files:

  • Create: services/nginx/public.conf
  • Create: services/ssh/10-hardening.conf
  • Create: services/ssh/github-keys
  • Create: services/fail2ban/jail.local

Interfaces:

  • Produces: header X-Lab-Key, cookie lab_key, gate URL /gate/__LAB_KEY__ — Tasks 6–7 depend on these exact names; github-keys installs to /usr/local/bin/github-keys (Task 2).

  • Step 1: Write services/nginx/public.conf

# Public-facing ingress. Copy to /etc/nginx/sites-available/public, symlink into
# sites-enabled. Replace __LAB_KEY__ everywhere with `openssl rand -hex 32` output
# (same value goes in each Immich app's X-Lab-Key header).
 
# Request passes if the key appears in the X-Lab-Key header, the lab_key cookie,
# or the URI (which lets /gate/<key> through to set the cookie for browsers).
map "$http_x_lab_key$cookie_lab_key$uri" $lab_ok {
    default 0;
    "~__LAB_KEY__" 1;
}
 
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}
 
# Bare-IP / wrong-SNI scans: refuse the TLS handshake, reveal nothing.
server {
    listen 443 ssl default_server;
    ssl_reject_handshake on;
}
 
server {
    listen 443 ssl http2;
    server_name pics.saah.as;
    ssl_certificate     /etc/letsencrypt/live/saah.as/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/saah.as/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
 
    if ($lab_ok = 0) { return 444; }
 
    location = /gate/__LAB_KEY__ {
        add_header Set-Cookie "lab_key=__LAB_KEY__; Secure; HttpOnly; SameSite=Lax; Path=/; Max-Age=31536000";
        return 302 /;
    }
 
    location / {
        proxy_pass http://127.0.0.1:2283;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        client_max_body_size 0;          # Immich uploads
        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
    }
}
 
server {
    listen 443 ssl http2;
    server_name pad.saah.as;
    ssl_certificate     /etc/letsencrypt/live/saah.as/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/saah.as/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
 
    if ($lab_ok = 0) { return 444; }
 
    location = /gate/__LAB_KEY__ {
        add_header Set-Cookie "lab_key=__LAB_KEY__; Secure; HttpOnly; SameSite=Lax; Path=/; Max-Age=31536000";
        return 302 /;
    }
 
    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
  • Step 2: Write services/ssh/10-hardening.conf
# Copy to /etc/ssh/sshd_config.d/10-hardening.conf (lexically first => wins: sshd first-match).
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin no
AllowUsers ns
AuthorizedKeysCommand /usr/local/bin/github-keys %u
AuthorizedKeysCommandUser nobody
  • Step 3: Write services/ssh/github-keys
#!/bin/sh
# GitHub account = SSH trust root. Union with ~/.ssh/authorized_keys (offline break-glass):
# fetch fails closed, local file keeps working. Install root-owned 0755 at /usr/local/bin/github-keys.
[ "$1" = "ns" ] || exit 0
exec curl -sf --proto =https --max-time 5 https://github.com/Suputra.keys
  • Step 4: Write services/fail2ban/jail.local
# Copy to /etc/fail2ban/jail.local. Rate-limits pre-auth connections (mitigates
# regreSSHion-class race exploits) and quiets logs.
[sshd]
enabled = true
backend = systemd
maxretry = 5
findtime = 10m
bantime = 1h
  • Step 5: Commit
git add services/nginx services/ssh services/fail2ban
git commit -m "infra: nginx public gate + sshd hardening + fail2ban configs for direct exposure"

Task 2: Harden sshd on Tower (tunnel still primary)

Files:

  • Modify (Tower): /etc/ssh/sshd_config.d/10-hardening.conf, /usr/local/bin/github-keys

Interfaces:

  • Consumes: Task 1 files. Produces: keys-only sshd with GitHub key fetch, verified while cloudflared still carries traffic.

  • Step 1: Verify break-glass key exists (must pass BEFORE disabling passwords)

Run: ssh lab "grep -c . ~/.ssh/authorized_keys" Expected: 1 or more (the desktop key currently authenticating you).

  • Step 2: Install script and drop-in
scp services/ssh/github-keys lab:/tmp/ && scp services/ssh/10-hardening.conf lab:/tmp/
ssh -t lab 'sudo install -o root -g root -m 0755 /tmp/github-keys /usr/local/bin/github-keys \
  && sudo install -o root -g root -m 0644 /tmp/10-hardening.conf /etc/ssh/sshd_config.d/10-hardening.conf'
  • Step 3: Test the key fetch as the sshd would run it

Run: ssh lab "sudo -u nobody /usr/local/bin/github-keys ns" Expected: your public key lines (same keys as curl https://github.com/Suputra.keys). If empty: check Tower’s outbound HTTPS and that curl is installed.

  • Step 4: Syntax-check, then restart sshd — keep this ssh session open

Run: ssh -t lab 'sudo sshd -t && sudo systemctl restart ssh && systemctl is-active ssh' Expected: sshd -t silent; active. (Debian/Ubuntu unit is ssh; sshd is an alias.)

  • Step 5: Verify effective config + fresh login

Run: ssh lab "sudo sshd -T | grep -Ei 'passwordauthentication|kbdinteractiveauthentication|permitrootlogin|allowusers|authorizedkeyscommand '" Expected: passwordauthentication no, kbdinteractiveauthentication no, permitrootlogin no, allowusers ns, authorizedkeyscommand /usr/local/bin/github-keys %u. Then from a NEW terminal: ssh lab true && echo OK → OK (fresh login still works via tunnel).

Task 3: fail2ban + unattended-upgrades on Tower

  • Step 1: Install and configure
scp services/fail2ban/jail.local lab:/tmp/
ssh -t lab 'sudo apt-get install -y fail2ban unattended-upgrades \
  && sudo install -m 0644 /tmp/jail.local /etc/fail2ban/jail.local \
  && sudo systemctl enable --now fail2ban && sudo systemctl restart fail2ban \
  && echo "unattended-upgrades unattended-upgrades/enable_auto_updates boolean true" | sudo debconf-set-selections \
  && sudo dpkg-reconfigure -f noninteractive unattended-upgrades'
  • Step 2: Verify

Run: ssh lab "sudo fail2ban-client status sshd && apt-config dump APT::Periodic::Unattended-Upgrade" Expected: sshd jail status block (0 banned is fine); APT::Periodic::Unattended-Upgrade "1";

Task 4: Router forwards + direct SSH verification (USER)

  • Step 1 (USER): Port-forward on router (router.y): TCP 443 → 10.0.0.100:443, TCP 22 → 10.0.0.100:22.

  • Step 2: CGNAT check

Run: ssh lab "curl -s https://ifconfig.me" and compare with router WAN IP (router status page) and dig +short home.saah.as @1.1.1.1. Expected: all three match. If ifconfig.me ≠ router WAN → CGNAT; stop and reassess (ISP ticket or fallback to keeping the tunnel).

  • Step 3 (USER-assisted): Off-LAN direct SSH test — phone hotspot on laptop:

Run: ssh -o ProxyCommand=none [email protected] true && echo DIRECT-OK Expected: DIRECT-OK. This is the gate for decommissioning later; do not proceed to Task 8 without it.

Task 5: Wildcard cert via DNS-01

  • Step 1 (USER): Create CF API token — Cloudflare dashboard → My Profile → API Tokens → Create: permission Zone / DNS / Edit, zone saah.as. Paste into the Task 5 Step 2 command.

  • Step 2: Install certbot + store token on Tower

ssh -t lab 'sudo apt-get install -y certbot python3-certbot-dns-cloudflare \
  && sudo install -d -m 700 /root/.secrets \
  && sudo sh -c "umask 077; printf \"dns_cloudflare_api_token = %s\n\" \"<CF_TOKEN>\" > /root/.secrets/cloudflare.ini"'
  • Step 3: Issue *.saah.as

Run: ssh -t lab 'sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /root/.secrets/cloudflare.ini -d "*.saah.as" --non-interactive --agree-tos -m [email protected] --deploy-hook "systemctl reload nginx"' Expected: Successfully received certificate at /etc/letsencrypt/live/saah.as/. Renewal is automatic (systemd timer); deploy-hook reloads nginx.

Task 6: nginx gate live + DNS flip

Interfaces:

  • Consumes: cert from Task 5; public.conf from Task 1. Produces: gated HTTPS on pics/pad; the generated LAB_KEY (save to password manager — Task 7 needs it).

  • Step 1: Install config with generated secret

scp services/nginx/public.conf lab:/tmp/
ssh -t lab 'KEY=$(openssl rand -hex 32) && echo "LAB_KEY=$KEY" \
  && sudo sh -c "sed s/__LAB_KEY__/$KEY/g /tmp/public.conf > /etc/nginx/sites-available/public" \
  && sudo ln -sf /etc/nginx/sites-available/public /etc/nginx/sites-enabled/public \
  && sudo nginx -t && sudo systemctl reload nginx'

Expected: prints LAB_KEY=<64 hex chars> (record it), nginx -t ok. (http2 deprecation warning is fine on nginx ≥1.25.)

  • Step 2: Gate tests from LAN (before DNS changes), KEY from Step 1
curl -sv --resolve pics.saah.as:443:10.0.0.100 https://pics.saah.as/api/server/ping          # expect: curl error 52/reset — 444 closed
curl -s --resolve pics.saah.as:443:10.0.0.100 -H "X-Lab-Key: $KEY" https://pics.saah.as/api/server/ping   # expect: {"res":"pong"}
curl -s --resolve pad.saah.as:443:10.0.0.100 -b "lab_key=$KEY" -o /dev/null -w '%{http_code}' https://pad.saah.as/   # expect: 200

  • Step 3 (USER): Cloudflare DNS — delete tunnel CNAMEs for pics.saah.as and pad.saah.as; recreate both as CNAME → home.saah.as, DNS only (gray cloud). (lab.saah.as waits for Task 8.)

  • Step 4: Pi-hole split horizon — on the Pi (same /etc/hosts mechanism as the .y domains): add 10.0.0.100 pics.saah.as pad.saah.as home.saah.as, then pihole restartdns.

Verify from Mac on LAN: dig +short pics.saah.as → 10.0.0.100; from Tower: dig +short pics.saah.as @1.1.1.1 → public IP.

  • Step 5: Off-LAN gate test (phone hotspot): repeat Step 2 curls without --resolve. Expected: same results over the public path.

Task 7: Point devices at the gate (USER)

  • Step 1: Family phones + yours — Immich app → Settings → Advanced → Proxy Headers: remove the old CF Access header(s), add X-Lab-Key: <LAB_KEY>. Server URL stays https://pics.saah.as. Verify: app loads timeline + a photo upload syncs.
  • Step 2: Browsers that need pics/pad web — visit https://pics.saah.as/gate/<LAB_KEY> and https://pad.saah.as/gate/<LAB_KEY> once per browser (sets 1-year cookie); bookmark them.

Task 8: Decommission cloudflared

Files:

  • Delete: services/cloudflared/
  • Modify: dotfiles/ssh-config (Host lab block)

Gate: Task 4 Step 3 printed DIRECT-OK and Task 6 Step 5 passed.

  • Step 1: Stop tunnel on Tower

Run: ssh -t lab 'sudo systemctl disable --now cloudflared && cloudflared tunnel delete lab' Expected: service stopped; tunnel deleted (delete the lab.saah.as CNAME in CF dashboard first if deletion complains about DNS routes).

  • Step 2 (USER): Delete lab.saah.as CNAME in Cloudflare dashboard (if not already).

  • Step 3: Simplify dotfiles/ssh-config — replace the Host lab block:

Host lab
  HostName home.saah.as
  User ns
  ForwardX11 yes
  ForwardX11Trusted yes
  ServerAliveInterval 60
  ServerAliveCountMax 3

(Pi-hole resolves home.saah.as → 10.0.0.100 on LAN; public IP elsewhere. ProxyCommand and its nc LAN-check are gone.) Redeploy dotfiles the usual way; brew uninstall cloudflared on the Mac is now optional cleanup.

  • Step 4: Verify sync stack end-to-end

Run (on LAN): ssh lab true && echo LAN-OK && mutagen sync list Then off-LAN (hotspot): ssh lab true && echo WAN-OK Expected: LAN-OK, WAN-OK, mutagen session connected/watching.

  • Step 5: Remove repo dir + commit
git rm -r services/cloudflared
git commit -m "infra: drop cloudflared — direct nginx gate + hardened public ssh"

Task 9: Docs

  • Step 1: services/README.md — update Network Overview: tunnel → direct 443/22 ingress diagram from the spec; note the gate (X-Lab-Key / lab_key cookie / /gate/<key>), where the key lives, and split-horizon DNS on the Pi-hole.
  • Step 2: garden/private/notes/index.md — drop-cloudflared row → Shipped: direct exposure live; tunnel removed.
  • Step 3: Commit
git add services/README.md garden/private/notes/index.md
git commit -m "docs: network overview for direct exposure"

Recent Notes

Graph View