Commit 674c3b0b by PLN (Algolia)

gig-record: retention as a command, dry by default, never the live take

The black box writes ~72 MB/hour into a gitignored directory and nothing
deleted anything, so it only grew: a rehearsal week is >5 GB and the repo's
">500 MB is reported or deleted, never silent" line bites after one.

An automatic sweep is the wrong shape here. A timer or an ExecStopPost is the
one mechanism that could delete the take nobody meant to keep, which is the
entire reason the recorder exists. So: `prune [DAYS]`, dry by default. It
prints what it would remove and removes nothing until --yes.

Three things are never candidates, in the order they would hurt:

  * whatever the running recorder holds open, read from /proc/PID/fd rather
    than guessed from mtime — after a segment roll the newest file is not the
    one being written, and a long session's current segment can easily be
    older than the window. A held-back file is named BEFORE the
    nothing-to-do line, or the report would be a lie about the one file that
    matters most (the test that found this).
  * anything that is not a %Y-%m-%d_%H%M%S.opus segment, so a slice_*
    excerpt cut on purpose is not swept up with the raw capture.
  * anything inside KEEP_DAYS (14, overridable).

OUT_DIR is overridable with GIG_RECORD_DIR for one reason: a delete command
whose tests cannot run without pointing at real takes is a delete command
nobody tests. The six new tests assert what it REFUSES to delete, including
a real process holding a real fd.

The gate advises at 500 MB and names the command. It never prunes.
parent 3a24802c
...@@ -459,6 +459,27 @@ print(" ".join(str(p) for p in m.setlist_tracks()))')"""), ...@@ -459,6 +459,27 @@ print(" ".join(str(p) for p in m.setlist_tracks()))')"""),
"START, so it must be cycled AFTER Ardour is up and Master is " "START, so it must be cycled AFTER Ardour is up and Master is "
"wired where the room hears it"), "wired where the room hears it"),
# THE BLACK BOX ONLY GROWS. ~72 MB/hour, gitignored, and nothing deletes
# anything on a timer — deliberately, because an automatic sweep is the one
# mechanism that could delete the take nobody meant to keep, which is the
# whole reason the recorder exists. So the gate is what notices, at the
# repo's own ">500 MB is reported or deleted, never silent" line.
#
# ADVISE: a full disk is a real gig risk, a large directory is not.
Probe("recordings size", shell(r"""
d="$PWD/recordings"
[ -d "$d" ] || exit 0
kb=$(du -sk "$d" 2>/dev/null | cut -f1)
[ -n "$kb" ] || exit 0
[ "$kb" -lt 512000 ] && exit 0
echo "recordings/ is $(du -sh "$d" | cut -f1) — $(ls "$d"/*.opus 2>/dev/null | wc -l) segments"
exit 1"""),
kind=ADVISE,
fix="tools/gig_record.sh prune # lists what is older than 14 days "
"and deletes nothing; add --yes once you have read the list. It "
"never touches the segment the recorder holds open, nor a "
"slice_* excerpt you cut on purpose."),
# ---- the empirical gate, opt-in ---------------------------------------- # # ---- the empirical gate, opt-in ---------------------------------------- #
# Last, loud, and only on request: it makes sound and takes ~45s per track. # Last, loud, and only on request: it makes sound and takes ~45s per track.
# Booting the set to confirm what a typecheck just told you is ten wasted # Booting the set to confirm what a typecheck just told you is ten wasted
......
...@@ -9,15 +9,26 @@ ...@@ -9,15 +9,26 @@
# gig_record.sh stop stop the background recorder # gig_record.sh stop stop the background recorder
# gig_record.sh status is it running? which file? how big? pointed where? # gig_record.sh status is it running? which file? how big? pointed where?
# gig_record.sh check is what it hears still where Ardour Master goes? # gig_record.sh check is what it hears still where Ardour Master goes?
# gig_record.sh prune [DAYS] list segments older than DAYS (default 14)
# gig_record.sh prune [DAYS] --yes ...and actually delete them
# gig_record.sh slice FILE START_SEC MINUTES OUT fast lossless-cut a test excerpt # gig_record.sh slice FILE START_SEC MINUTES OUT fast lossless-cut a test excerpt
set -euo pipefail set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
OUT_DIR="$DIR/recordings" # Overridable ONLY so the prune path can be tested against fixtures instead of
# against the one directory that holds real takes. A delete command whose
# tests cannot run without pointing at live data is a delete command nobody
# tests.
OUT_DIR="${GIG_RECORD_DIR:-$DIR/recordings}"
PID_FILE="$OUT_DIR/.recorder.pid" PID_FILE="$OUT_DIR/.recorder.pid"
LOG_FILE="$OUT_DIR/recorder.log" LOG_FILE="$OUT_DIR/recorder.log"
BITRATE="${GIG_RECORD_BITRATE:-160k}" BITRATE="${GIG_RECORD_BITRATE:-160k}"
SEGMENT_SECS="${GIG_RECORD_SEGMENT:-3600}" SEGMENT_SECS="${GIG_RECORD_SEGMENT:-3600}"
# How long a black-box segment is worth keeping without anyone asking for it.
# Two weeks covers "was that Thursday any good?" and a rehearsal week, and at
# ~72 MB/hour that is the difference between a directory you forget about and
# one that trips the >500 MB rule. Nothing prunes on a timer — see cmd_prune.
KEEP_DAYS="${GIG_RECORD_KEEP_DAYS:-14}"
mkdir -p "$OUT_DIR" mkdir -p "$OUT_DIR"
...@@ -144,6 +155,82 @@ cmd_check() { ...@@ -144,6 +155,82 @@ cmd_check() {
return 3 return 3
} }
# RETENTION, AND WHY IT IS A COMMAND AND NOT A TIMER
# This writes ~72 MB/hour and is meant to run whenever the rig is up, so the
# directory only grows: a rehearsal week is >5 GB and the repo's own ">500 MB
# is reported or deleted, never silent" rule bites after one. But the whole
# POINT of a black box is that it still has the take nobody meant to keep, so
# an automatic sweep is the one mechanism that could delete the thing this
# tool exists to save. There is no ExecStopPost and no cron.
#
# So: a command, DRY BY DEFAULT. It prints exactly what it would remove and
# removes nothing until `--yes`. Three things are never candidates:
# * anything the running recorder holds open (read from /proc/PID/fd, not
# guessed from the name — the newest file by mtime is not always the one
# being written after a segment roll);
# * anything that is not a `%Y-%m-%d_%H%M%S.opus` segment, so a `slice_*`
# excerpt someone cut on purpose is not swept up with the raw capture;
# * anything inside KEEP_DAYS.
cmd_prune() {
local days="${1:-$KEEP_DAYS}" confirm="${2:-}"
case "$days" in
''|*[!0-9]*) echo "gig_record: prune DAYS must be a whole number, got '$days'" >&2; return 2 ;;
esac
# The file the live recorder is writing, by fd, so a segment roll cannot
# fool us. Empty when nothing is running.
local open="" pid
if [ -f "$PID_FILE" ] && pid="$(cat "$PID_FILE")" && kill -0 "$pid" 2>/dev/null; then
open="$(readlink -f /proc/"$pid"/fd/* 2>/dev/null | grep -F "$OUT_DIR/" || true)"
fi
local -a stale=() held=()
local f
while IFS= read -r -d '' f; do
case "$open" in
*"$f"*) held+=("$f"); continue ;;
esac
stale+=("$f")
done < <(find "$OUT_DIR" -maxdepth 1 -type f \
-regextype posix-extended \
-regex '.*/[0-9]{4}-[0-9]{2}-[0-9]{2}_[0-9]{6}\.opus' \
-mtime "+$days" -print0 2>/dev/null | sort -z)
local total
total="$(du -sh "$OUT_DIR" 2>/dev/null | cut -f1)"
echo "gig_record: $OUT_DIR holds ${total:-0} · keeping the last $days days"
# Say this BEFORE the nothing-to-do line. A segment old enough to prune and
# still open is the normal state of a long session, and "nothing older than
# 14 days" would then be a lie about the one file that matters most.
if [ "${#held[@]}" -gt 0 ]; then
echo "gig_record: ${#held[@]} file(s) held back — the recorder has them open:"
for f in "${held[@]}"; do echo " $(basename "$f")"; done
fi
if [ "${#stale[@]}" -eq 0 ]; then
echo "gig_record: nothing older than $days days to remove. Nothing to do."
return 0
fi
local bytes=0 sz
for f in "${stale[@]}"; do
sz="$(stat -c%s "$f" 2>/dev/null || echo 0)"
bytes=$((bytes + sz))
printf ' %s %s %s\n' "$(date -r "$f" '+%Y-%m-%d %H:%M')" \
"$(numfmt --to=iec --suffix=B "$sz" 2>/dev/null || echo "${sz}B")" \
"$(basename "$f")"
done
echo "gig_record: ${#stale[@]} segment(s), $(numfmt --to=iec --suffix=B "$bytes" 2>/dev/null || echo "${bytes}B")"
if [ "$confirm" != "--yes" ]; then
echo "gig_record: DRY RUN — nothing deleted. Re-run with --yes to delete these."
return 0
fi
for f in "${stale[@]}"; do rm -- "$f"; done
echo "gig_record: deleted ${#stale[@]} segment(s)"
}
# Fast, lossless (stream-copy) excerpt for auditioning / slop-visuals testing. # Fast, lossless (stream-copy) excerpt for auditioning / slop-visuals testing.
cmd_slice() { cmd_slice() {
local file="$1" start="$2" minutes="$3" out="${4:-}" local file="$1" start="$2" minutes="$3" out="${4:-}"
...@@ -158,6 +245,7 @@ case "${1:-}" in ...@@ -158,6 +245,7 @@ case "${1:-}" in
stop) cmd_stop ;; stop) cmd_stop ;;
status) cmd_status ;; status) cmd_status ;;
check) cmd_check ;; check) cmd_check ;;
prune) shift; cmd_prune "$@" ;;
slice) shift; cmd_slice "$@" ;; slice) shift; cmd_slice "$@" ;;
*) echo "usage: $0 {start|stop|status|check|slice FILE START_SEC MINUTES [OUT]}" >&2; exit 1 ;; *) echo "usage: $0 {start|stop|status|check|prune [DAYS] [--yes]|slice FILE START_SEC MINUTES [OUT]}" >&2; exit 1 ;;
esac esac
"""Retention for the black box: dry by default, and it never eats the take.
WHY THIS IS A COMMAND AND NOT A TIMER (2026-09-25)
`gig_record.sh` writes ~72 MB/hour and is meant to run whenever the rig is up,
so `recordings/` only ever grows; a rehearsal week is >5 GB and the repo's
">500 MB is reported or deleted, never silent" rule bites after one. But the
whole point of a black box is that it still has the take nobody meant to keep,
so an automatic sweep is the one mechanism that could delete the thing the tool
exists to save. Hence: a command, dry by default, `--yes` to delete.
Which makes these the tests that matter — not "does it delete" but "what does
it REFUSE to delete":
* the file the running recorder holds open, found through /proc/PID/fd
rather than guessed from mtime, because after a segment roll the newest
file is not the one being written;
* anything that is not a `%Y-%m-%d_%H%M%S.opus` segment, so a `slice_*`
excerpt cut on purpose is not swept up with the raw capture;
* anything inside the keep window.
Fixtures, in a tmp dir via `GIG_RECORD_DIR`. A delete command whose tests
cannot run without pointing at live data is a delete command nobody tests.
"""
from __future__ import annotations
import os
import pathlib
import subprocess
TOOLS = pathlib.Path(__file__).resolve().parent.parent
SCRIPT = TOOLS / "gig_record.sh"
def run(out_dir: pathlib.Path, *args: str) -> subprocess.CompletedProcess:
env = dict(os.environ, GIG_RECORD_DIR=str(out_dir))
return subprocess.run(["bash", str(SCRIPT), "prune", *args],
capture_output=True, text=True, env=env)
def seg(out_dir: pathlib.Path, name: str, days_old: float) -> pathlib.Path:
f = out_dir / name
f.write_bytes(b"\0" * 2048)
when = f"-{days_old} days" if days_old else "now"
subprocess.run(["touch", "-d", when, str(f)], check=True)
return f
def test_an_old_segment_is_listed_but_not_deleted_without_yes(tmp_path):
old = seg(tmp_path, "2026-08-01_180000.opus", 40)
r = run(tmp_path, "14")
assert r.returncode == 0, r.stderr
assert "2026-08-01_180000.opus" in r.stdout
assert "DRY RUN" in r.stdout
assert old.exists(), "a dry run deleted a file"
def test_yes_deletes_exactly_what_the_dry_run_listed(tmp_path):
old = seg(tmp_path, "2026-08-01_180000.opus", 40)
fresh = seg(tmp_path, "2026-09-25_120000.opus", 0)
r = run(tmp_path, "14", "--yes")
assert r.returncode == 0, r.stderr
assert not old.exists()
assert fresh.exists(), "the keep window was not honoured"
def test_a_file_inside_the_window_is_never_a_candidate(tmp_path):
keep = seg(tmp_path, "2026-09-20_180000.opus", 3)
r = run(tmp_path, "14", "--yes")
assert keep.exists()
assert "Nothing to do" in r.stdout
def test_a_slice_someone_cut_on_purpose_is_not_swept_up(tmp_path):
"""`slice_*.opus` is a derived excerpt, made deliberately. Only raw
`%Y-%m-%d_%H%M%S.opus` segments are the black box's own churn."""
slice_ = seg(tmp_path, "slice_2026-08-01_180000_30s_1m.opus", 60)
other = seg(tmp_path, "recorder.log", 60)
r = run(tmp_path, "14", "--yes")
assert slice_.exists(), "a hand-cut excerpt was deleted as churn"
assert other.exists()
assert "Nothing to do" in r.stdout
def test_the_file_the_recorder_holds_open_is_held_back(tmp_path):
"""The one that matters. A long-running recorder can easily have its
current segment older than the window — `-mtime` is the LAST WRITE, and
a file being appended to slowly still has an old mtime on some filesystems
— so 'is it open' has to be asked of the kernel, not of the clock."""
live = tmp_path / "2026-08-01_180000.opus"
live.write_bytes(b"\0" * 2048)
subprocess.run(["touch", "-d", "-40 days", str(live)], check=True)
pid_file = tmp_path / ".recorder.pid"
# A real process holding a real fd on that file: `sleep` with the file open
# on stdout would truncate it, so append instead.
with live.open("ab") as fh:
proc = subprocess.Popen(["sleep", "30"], stdout=fh)
try:
pid_file.write_text(str(proc.pid))
r = run(tmp_path, "14", "--yes")
assert live.exists(), "prune deleted the take being recorded"
assert "held back" in r.stdout
assert "2026-08-01_180000.opus" in r.stdout
finally:
proc.terminate()
proc.wait()
def test_a_nonsense_day_count_is_refused_rather_than_guessed(tmp_path):
old = seg(tmp_path, "2026-08-01_180000.opus", 40)
r = run(tmp_path, "abc", "--yes")
assert r.returncode == 2
assert old.exists()
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment