Deploy Backup Deduplication: 8280 MB to 96 MB, and the Set Difference to Run First

The server hit 86% disk with 3.4 GB left. The biggest directory was deploy backups: 278 directories, 8280 MB.

The instinct is “backups are big, that’s normal, delete the old ones”. Before deleting anything I did one thing first — worked out how much of those 8.3 GB was actually unique. The answer was 276 MB.

The other 8004 MB was one set of files copied 228 times.

Measure first, delete second

Every deploy snapshots the site’s current state. Looking at what one snapshot contains:

4.0K   404.php.before
28K    functions.php.before
500K   personal-site-experience.php.before
...
44M    public-downloads.before        <-- 44 of the 45 MB
8.0K   personal-site.conf.before

Of a 45 MB backup, 44 MB is public-downloads.before — a full copy of the static downloads directory. That directory only changes when I update lab materials, which is almost never during a normal deploy.

The blunter evidence is diffing two adjacent backups:

$ diff -rq 20260801-114501/ 20260801-122255/ | wc -l
3

Three files differ out of 241. The other 238 are byte-identical copies.

Grouping by day, the heaviest ones:

20260602   26 deploys   1170 MB
20260604   17 deploys    765 MB
20260801   13 deploys    588 MB

This is not “too many backups”. It is the deploy count during an intense debugging stretch, multiplied by a 44 MB constant.

Stripping the duplication out

total                 8280 MB
of which public-downloads  8004 MB (228 copies)
genuinely unique       276 MB

And the live public-downloads directory is itself a superset of those copies. Not one byte of those 8004 MB exists nowhere else.

That settles the archive design: exclude every public-downloads.before, and separately keep one current snapshot of it. Since the server had only 3.4 GB free and could not stage a tarball, the archive is streamed straight out without ever touching disk:

ssh server "cd /srv/backups && tar czf - --exclude=public-downloads.before ." \
  > backups-unique.tar.gz

Result: 8280 MB → 96 MB (56 MB deduplicated archive, 37 MB for one full snapshot, 3 MB of recovered extras).

The seven files I nearly lost

There is a hole in the reasoning so far. I kept the newest snapshot — but older snapshots may contain files that were later deleted and are therefore absent from the newest one.

Deletion is irreversible, so that hole had to be closed first. The way to close it is a set difference:

# every file path that ever appeared in any historical snapshot
find . -path "*/public-downloads.before/*" -type f \
  | sed "s#.*/public-downloads.before/##" | sort -u > all.txt
# newest snapshot + the live directory
cat new.txt live.txt | sort -u > keep.txt
comm -23 all.txt keep.txt          # present only in history

union of all history   220 files
newest snapshot        181
live directory         185
history only            35

Thirty-five. Going through them:

23 are macOS ._ metadata (AppleDouble resource forks, picked up when copying from a Mac) — worthless
5 had simply moved into a subdirectory — Iris.csv went from the root to iris-kmeans/Iris.csv; different path, same file, still live
7 genuinely existed only in historical backups: a shanhaijue-media/ set containing a hero image, a validation figure, a manifest and the script that generated them

Those seven are absent from both the live directory and the newest snapshot. Running “keep the last 10 backups, delete the rest” would have destroyed them permanently — silently, with no error, because nothing still referenced them.

Fetching them took under a minute. Skipping that step would have cost something irreversible that I would not have noticed for a long time.

Why the archive had to be streamed

There is a constraint here you cannot design around: the server has nowhere to stage a file.

The usual approach is to tar an archive on the server and then transfer it. But only 3.4 GB was free and the directory being archived was 8.3 GB — even on optimistic compression there is no safe place to put it. Worse, running out of space midway leaves both a truncated archive and a full disk.

The answer is to have tar write to stdout and pipe it over ssh, so nothing lands on the server at all:

ssh server "cd /srv/backups && tar czf - --exclude=public-downloads.before ." \
  > backups-unique.tar.gz

The pipe has a cost though: a dropped connection produces a plausible-looking but truncated gzip file. This link was already unreliable (ssh dropped a dozen times over one evening), so every transfer has to be verified and retried whole — there is no resume, because the output of tar czf - has no recoverable boundaries:

for i in 1 2 3 4 5 6; do
  ssh "$REMOTE" "$CMD" > "$OUT.part" \
    && gzip -t "$OUT.part" 2>/dev/null \
    && { mv "$OUT.part" "$OUT"; break; }
  echo "incomplete transfer, retrying"
  sleep 5
done

The gzip -t step is not optional. It is the only thing in this chain that distinguishes “finished” from “half-finished” — ssh frequently exits 0 when a connection is cut by something in the middle.

Verify, then delete

Having an archive is not permission to delete. Three checks ran first:

file list compared one by one   8957 / 8957   missing 0   size mismatch 0
random sha256 sample               25 / 25 match
gzip integrity                      all 3 archives pass

The sampling step initially returned one BAD. It turned out my script called ssh twice per file, and over a flaky link one of the two calls returned empty — a false positive from the verification tool itself. Batching all the hashes into a single ssh call gave 25/25.

That is worth recording on its own. Believing the BAD would have sent me chasing a transfer corruption that did not exist; ignoring it could have masked a real one. When a check fails, first establish whether what failed is the thing under test or the instrument.

The archive was destined for cloud storage, so two further independent checks ran on that side. First rclone check, comparing hashes file by file:

0 differences found
5 matching files

Then, not trusting that, the uploaded files were downloaded in full and re-hashed against the local originals:

rclone cat "$DEST/$f" | sha256sum

The two checks are not redundant. rclone check compares hash metadata the service reports; re-downloading hashes the bytes you can actually read back. The first proves the upload was correct, the second proves it still reads correctly. For an archive about to become the only copy, that distinction is worth the extra minutes.

Only once all three sha256 values matched did the delete run. Disk went from 86% to 53%, free space from 3.4 GB to 11 GB.

The retention choice was “keep the last 10”. There is no theory behind that number; it is simply enough to cover the most recent burst of deploys. Rollbacks essentially always happen within hours of a change, and with an archive as backstop, older snapshots have no reason to occupy local disk.

Closing the tap

Deleting only settles the account. Without a change it grows straight back. One line in the deploy script, skipping the large static assets when snapshotting:

rsync -a --exclude 'anime-matting' \
  "$SITE/public-downloads/" "$BACKUP/public-downloads.before/"

Per-deploy backup went from 99 MB to 45 MB. (It was 99 because a 53 MB model file had recently landed in that directory — same mechanism, double the cost.)

Three things worth keeping

A backup’s size tells you almost nothing about its value. 96.7% of those 8.3 GB were repeats of the same bytes. When auditing disk usage, group by content rather than by directory — du tells you who is occupying space, not who is a duplicate.

Compute the set difference before deleting history; do not trust intuition. “The newest one is enough” was wrong here, and wrong quietly. A single comm -23 falsifies it.

Duplication inside a backup is a fixed tax on every deploy. The thing to fix was never “too many backups” — it was a backup policy that re-snapshots large files that never change. A period of frequent deploys then multiplies that constant by a few dozen.

Only 276 MB of 8.3 GB Was Unique: Forensics Before a Delete