fix(preload): the gate asked the wrong question, and asked it in the wrong locale

gig-up runs check-preload --fix whenever the check fails, so a plan that was a
deliberate superset scored STALE and got silently narrowed — bass2 would have
been dropped on gig night and ete_a_mauerpark would have lazy-loaded mid-set.
Membership now uses comm -13/-23 (missing is a fault, extra is information),
counts join on shared banks, and --fix emits the union so it can never shrink.

Second hole, latent: under en_US.UTF-8 a whole-line sort orders 'bass2 5' before
'bass 4' while join compares field 1, so join dropped a pair (123 of 124 shared
banks) and 2>/dev/null ate its warning — a count check with a hole reports ok.
LC_ALL=C and sort -k1,1 on every set operation; join now equals comm -12.
Extras are also count-checked against disk: SuperDirt asserts the recorded count
at boot whether or not the bank is in the set.

tools/check-preload.sh

@@ -2,7 +2,7 @@
 # check-preload — is preload.scd warming the set we are actually going to play?
 #
 #     tools/check-preload.sh          # verify; exit 1 if stale
-#     tools/check-preload.sh --fix    # regenerate it from the setlist
+#     tools/check-preload.sh --fix    # regenerate it, NEVER shrinking coverage
 #
 # WHY (2026-08-01, found by ear). PLN heard crackles moving do_it_right ->
 # take_5_drops. Measured in that window: 0 SuperDirt lates, 0 pipewire xruns, and 17
@@ -30,6 +30,13 @@ cd "$(dirname "$0")/.." || exit 2
 FIX=0
 [ "${1:-}" = "--fix" ] && FIX=1
 
+# WHERE THE PLAN LIVES. Overridable for ONE reason: so this gate can be run against
+# fixtures without touching the live plan. A gate nobody ever exercised is the same
+# false confidence as a plan nobody compares to the set — and on 2026-09-22 this gate
+# was confidently wrong (see the comparison block below). tools/tests and the fixtures
+# in the commit that added this line are the callers; nothing in gig-up sets it.
+PLAN="${PV_PRELOAD_PLAN:-preload.scd}"
+
 # WHAT SET SHOULD THE PRELOAD WARM? Not the same answer as everyone else's.
 #
 # This used to borrow set-coherence.setlist_tracks() — "whatever gig-up and
@@ -101,51 +108,220 @@ fi
 # wrong in both directions: undercounting the plan AND inventing two bank names
 # (`rample`, `vocal`) that do not exist. Found 2026-09-07 by asking why the two numbers
 # never matched instead of assuming one of them was a rounding of the other.
-banks() { grep -oE '\[ \\[A-Za-z0-9_]+' "$1" 2>/dev/null | sed 's/.*\\//' | sort -u; }
+#
+# SORT IN THE C LOCALE, ON FIELD 1 (2026-09-22). `sort` and `join` have to agree on
+# collation or join and comm silently drop pairs — and this box runs LANG=en_US.UTF-8, where a
+# whole-line sort put `bass 4` after `bass2 5` while join's field-1 comparison expected
+# the opposite. Measured: 124 banks are in both plans, `comm -12` said 124 and `join`
+# said 123, losing one pair — and the `2>/dev/null` on the old join call was swallowing
+# join's own "input is not in sorted order" warning. A count check with a hole in it
+# reports ok. LC_ALL=C is set per command, not exported: exporting it would also set
+# LC_CTYPE and make the generator's em-dashes an encoding error, and `export LC_COLLATE=C`
+# is ignored whenever LC_ALL is already in the environment.
+banks() { grep -oE '\[ \\[A-Za-z0-9_]+' "$1" 2>/dev/null | sed 's/.*\\//' | LC_ALL=C sort -u; }
 bank_counts() {
   grep -oE '\[ \\[A-Za-z0-9_]+, [0-9]+' "$1" 2>/dev/null \
-    | sed -E 's/\[ \\([A-Za-z0-9_]+), ([0-9]+)/\1 \2/' | sort
+    | sed -E 's/\[ \\([A-Za-z0-9_]+), ([0-9]+)/\1 \2/' | LC_ALL=C sort -k1,1
+}
+nlines() { printf '%s\n' "$1" | grep -c . || true; }
+
+# disk_counts BANK...  ->  `name count` per line, sorted, read from DISK right now.
+#
+# For banks the current set does NOT ask for. Membership in the plan beyond the set is
+# fine (that is the whole over-cover design), but a WRONG COUNT never is: the emitted
+# plan asserts the count at boot, so `[ \bass2, 99, ... ]` makes SuperDirt print
+# "bass2: expected 99 files, got 5" and warm the bank short — for a bank the join-based
+# count check cannot see, because it is on one side only. Found 2026-09-22 while proving
+# --fix could not shrink: the fixture corrupted the count of the one extra bank and this
+# script said ok. Counting is delegated to setlist_samples.bank_file_count, never
+# re-implemented here (dotfiles and AppleDouble twins are its problem, and it solved it).
+disk_counts() {
+  [ "$#" -eq 0 ] && return 0
+  python3 - "$@" <<'PY' | LC_ALL=C sort -k1,1
+import importlib.util, sys
+spec = importlib.util.spec_from_file_location('ss', 'tools/setlist_samples.py')
+ss = importlib.util.module_from_spec(spec); spec.loader.exec_module(ss)
+idx = ss.build_index()
+for name in sys.argv[1:]:
+    folder = idx.get(name)
+    # No folder at all -> 0, which can never match a plan row and so reads as CHANGED.
+    print(name, ss.bank_file_count(folder) if folder and folder.is_dir() else 0)
+PY
 }
 
-new=$(mktemp); trap 'rm -f "$new"' EXIT
+# union_plan FRESH_PLAN BANK...  ->  FRESH_PLAN on stdout, with BANK... spliced back in.
+#
+# Used only by --fix, to make regeneration incapable of shrinking (see the long
+# comment at the fix site). The carried-over banks get a count read from DISK NOW
+# through the generator's own bank_file_count — carrying the old plan's number over
+# would smuggle a stale count past the CHANGED check for good.
+#
+# This edits the generated .scd by line, not by guessing at SuperCollider: it copies
+# every line verbatim and only rebuilds the `~pvPreload = [ ... ];` array, whose one
+# line shape is written three lines away in setlist_samples.emit_sc().
+union_plan() {
+  python3 - "$@" <<'PY'
+import importlib.util, re, sys
+from pathlib import Path
+
+spec = importlib.util.spec_from_file_location('ss', 'tools/setlist_samples.py')
+ss = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(ss)
+
+lines = Path(sys.argv[1]).read_text().split('\n')
+carry = sys.argv[2:]
+ROW = re.compile(r'^\t\[ \\([A-Za-z0-9_]+), (\d+), "(.*)" \],$')
+
+start = lines.index('~pvPreload = [')
+end = next(i for i in range(start + 1, len(lines)) if lines[i] == '];')
+rows = {}
+for ln in lines[start + 1:end]:
+    m = ROW.match(ln)
+    if not m:
+        sys.exit(f'! unrecognised preload row, refusing to merge: {ln!r}')
+    rows[m.group(1)] = ln
+
+idx = ss.build_index()
+kept, lost = [], []
+for name in carry:
+    folder = idx.get(name)
+    if folder is None or not folder.is_dir():
+        lost.append(name)          # a bank whose folder is GONE. Dropping it is the
+        continue                   # only option, so SAY SO — never in silence.
+    rows[name] = f'\t[ \\{name}, {ss.bank_file_count(folder)}, "{folder}" ],'
+    kept.append(name)
+
+body = [rows[n] for n in sorted(rows)]
+out = lines[:start + 1] + body + lines[end:]
+out = [re.sub(r'^// (\d+) track\(s\) scanned, \d+ banks',
+              lambda m: f'// {m.group(1)} track(s) scanned, {len(body)} banks', ln)
+       for ln in out]
+if kept:
+    hdr = next(i for i, ln in enumerate(out) if ln.startswith('// ') and 'banks,' in ln)
+    out.insert(hdr + 1, '// + %d bank(s) carried over from the previous plan '
+                        '(--fix never shrinks coverage): %s'
+                        % (len(kept), ' '.join(sorted(kept))))
+sys.stdout.write('\n'.join(out))
+for name in lost:
+    print(f'  carried-over bank {name!r} has no folder in the sample index — DROPPED',
+          file=sys.stderr)
+PY
+}
+
+new=$(mktemp); merged=""; trap 'rm -f "$new" "$merged"' EXIT
 # shellcheck disable=SC2086
 python3 tools/setlist_samples.py $tracks --emit-sc > "$new" 2>/dev/null || {
   echo "check-preload: setlist_samples.py failed" >&2; exit 2; }
 
-if [ ! -f preload.scd ]; then
-  echo "check-preload: preload.scd does NOT EXIST — every sample will lazy-load on first play."
-  missing=$(banks "$new" | wc -l); have=0; changed=0
+# THREE RELATIONS BETWEEN PLAN AND SET, and only TWO of them are faults (2026-09-22).
+#
+#   MISSING  the set needs the bank, the plan does not have it      -> FAULT.
+#            Read from disk on the audio thread, mid-transition, at the venue.
+#   CHANGED  bank is in BOTH and its file count moved               -> FAULT.
+#            SuperDirt says "expected N files, got M" and warms the bank short.
+#   EXTRA    the plan has it, THIS list does not ask for it         -> NOT A FAULT.
+#            Over-covering is the DESIGN, for exactly the asymmetry argued above.
+#
+# This used to compute CHANGED with `comm -3` over name+count pairs, which cannot
+# tell "the count moved" from "the bank is only on one side". So ANY plan that was a
+# SUPERSET of this list scored as changed. On 2026-09-22 the live 125-bank plan — the
+# 39-track computed list UNIONED with Thursday's 10, which is what pulled in `bass2`
+# for `ete_a_mauerpark` — reported
+#
+#     check-preload: STALE — preload.scd warms 125 bank(s), the set needs 124.
+#       1 bank(s) CHANGED SIZE — ...
+#     <empty table>
+#
+# The verdict and its own evidence disagreed: the table is built with `join`, and join
+# correctly found nothing, because bass2 is not in both. But gig-up believes the
+# VERDICT — it runs `--fix` on any failure — so a plain gig-up on gig night would have
+# regenerated from the 39-track list and DROPPED bass2. A false alarm that deletes
+# coverage is worse than no alarm at all.
+#
+# So: MEMBERSHIP is judged against the set (extras are information), while COUNTS are
+# judged for EVERY bank the plan contains — the shared ones against the fresh plan, the
+# extra ones straight off disk. Extra membership is free; an extra bank with a wrong
+# count is not, because the plan asserts that count at boot either way.
+if [ ! -f "$PLAN" ]; then
+  echo "check-preload: $PLAN does NOT EXIST — every sample will lazy-load on first play."
+  missing_list=$(banks "$new"); extra_list=""; drift=""; have=0
 else
-  missing=$(comm -13 <(banks preload.scd) <(banks "$new") | wc -l)
-  have=$(banks preload.scd | wc -l)
-  # Banks present in BOTH plans whose file COUNT moved. This is the check that
-  # was missing: a new pack dropped into an existing bank changes no bank name,
-  # so the name-set test above sees nothing and every new file lazy-loads.
-  changed=$(comm -3 <(bank_counts preload.scd) <(bank_counts "$new") \
-            | awk '{print $1}' | sort -u | grep -c . || true)
+  missing_list=$(LC_ALL=C comm -13 <(banks "$PLAN") <(banks "$new"))
+  extra_list=$(LC_ALL=C comm -23 <(banks "$PLAN") <(banks "$new"))
+  # One `name plan_count truth_count` table for the whole plan: shared banks get their
+  # truth from the fresh plan, extras from disk_counts. Kept as a table, not a bare name
+  # list, so the report below shows the same rows the verdict was computed from — the
+  # 2026-09-22 bug was precisely a verdict whose evidence table came out empty.
+  # shellcheck disable=SC2086
+  drift=$(LC_ALL=C join <(bank_counts "$PLAN") \
+            <(LC_ALL=C sort -k1,1 -m <(bank_counts "$new") <(disk_counts $extra_list)) \
+          | awk '$2 != $3')
+  have=$(banks "$PLAN" | wc -l)
 fi
-want=$(banks "$new" | wc -l)
+missing=$(nlines "$missing_list"); extra=$(nlines "$extra_list")
+changed=$(nlines "$drift"); want=$(banks "$new" | wc -l)
 
-if [ "$missing" -eq 0 ] && [ "${changed:-0}" -eq 0 ] && [ -f preload.scd ]; then
+if [ -f "$PLAN" ] && [ "$missing" -eq 0 ] && [ "$changed" -eq 0 ]; then
   echo "check-preload: ok — $have bank(s) warmed, covering all $(echo "$tracks" | wc -w) setlist track(s)."
+  if [ "$extra" -gt 0 ]; then
+    echo "  ($extra bank(s) warmed beyond this list — deliberate over-coverage, not staleness:"
+    echo "$extra_list" | paste -sd' ' - | fold -s -w 72 | sed -e 's/^/     /' -e '$s/$/)/'
+  fi
   exit 0
 fi
 
-echo "check-preload: STALE — preload.scd warms $have bank(s), the set needs $want."
+echo "check-preload: STALE — $PLAN warms $have bank(s), this set needs $want."
 if [ "$missing" -gt 0 ]; then
   echo "  $missing bank(s) MISSING — read from DISK on first play (crackle, mid-transition):"
-  comm -13 <(banks preload.scd) <(banks "$new") | head -20 | sed 's/^/    /'
+  echo "$missing_list" | head -20 | sed 's/^/    /'
 fi
-if [ "${changed:-0}" -gt 0 ] && [ -f preload.scd ]; then
+if [ "$changed" -gt 0 ]; then
   echo "  ${changed} bank(s) CHANGED SIZE — the plan's count no longer matches the folder,"
   echo "  so SuperDirt will report 'expected N files, got M' and warm the bank short:"
-  join <(bank_counts preload.scd) <(bank_counts "$new") 2>/dev/null \
-    | awk '$2 != $3 {printf "    %-24s plan %s -> disk %s\n", $1, $2, $3}' | head -20
+  echo "$drift" | awk 'NF {printf "    %-24s plan %s -> disk %s\n", $1, $2, $3}' | head -20
+fi
+if [ "$extra" -gt 0 ]; then
+  echo "  ($extra extra bank(s) in the plan — not a fault, and --fix will KEEP them.)"
 fi
 
 if (( FIX )); then
-  cp "$new" preload.scd
-  echo "  fixed: preload.scd regenerated ($want banks). Takes effect at the NEXT SuperDirt boot —"
+  # ANTI-SHRINK. This used to `cp` the fresh plan over the old one, full stop — so
+  # every regeneration narrowed coverage to whatever list THIS script computed today,
+  # silently. On 2026-09-22 that would have dropped `bass2` two days before the gig
+  # whose set needs it.
+  #
+  # CHOSEN BEHAVIOUR: **emit the UNION**. Not refuse. Why:
+  #   * Refusing leaves a genuine count drift unfixed, and gig-up's call site is
+  #     `check-preload.sh >/dev/null || check-preload.sh --fix >>$LOG && echo ok` —
+  #     a nonzero --fix prints NOTHING. Silence is exactly how #120 hid for a month.
+  #   * The asymmetry at the top of this file is total: a warmed unused bank costs
+  #     boot seconds and RAM, an unwarmed one costs a disk read on the audio thread
+  #     mid-set. So --fix is allowed to GROW the plan and never to shrink it.
+  #   * Carried-over banks are re-emitted with their count read from disk NOW, via the
+  #     generator's own bank_file_count (never a second counter — see the AppleDouble
+  #     note in setlist_samples.py), so a carried-over bank cannot smuggle a stale
+  #     count past the CHANGED check on the next run.
+  #
+  # CONSEQUENCE, stated plainly: `PV_PRELOAD_SETLIST=one_gig.txt --fix` can no longer
+  # produce a plan NARROWER than the current one. To narrow on purpose, delete the
+  # plan first — deleting is a deliberate act, and this script says loudly when the
+  # plan is absent.
+  merged=$(mktemp)
+  if [ "$extra" -eq 0 ]; then
+    cp "$new" "$merged"
+  # shellcheck disable=SC2086
+  elif ! union_plan "$new" $extra_list > "$merged"; then
+    echo "  REFUSED: could not merge the $extra carried-over bank(s) into the fresh plan."
+    echo "           $PLAN left untouched — it over-covers, which is the safe direction."
+    exit 1
+  fi
+  cp "$merged" "$PLAN"
+  final=$(banks "$PLAN" | wc -l)
+  # Report the delta MEASURED on the written file, not $extra — a carried-over bank whose
+  # folder has vanished is dropped by union_plan (loudly), and "= $want + $extra" would
+  # then be arithmetic that does not add up. Numbers in a gate must be readable as proof.
+  echo "  fixed: $PLAN regenerated ($final banks = $want from this set + $((final - want)) carried over)."
+  echo "         Takes effect at the NEXT SuperDirt boot —"
   echo "         systemctl --user restart parvagues-sc   (~15s of silence, safe when not playing)"
   exit 0
 fi