Commit 46b521d8 by PLN (Algolia)

fix(preload): make the freshness check trustable, and delete 198 fake wavs

Deleted 198 AppleDouble resource forks named zz._*.wav, on explicit owner
instruction. Every one confirmed by file(1) magic before removal rather than by
name pattern -- 198/198 candidates came back 'AppleDouble encoded Macintosh
file', zero false positives, 0.3 MB. 69 were in playable banks
(rhadamanthe_fx 227->189, _divers 314->293, _vocal 77->72, _melo 32->30); 129
were under Samples/baba/__MACOSX/, whose emptied dirs were rmdir-ed. Sample
indices were untouched: the zz. prefix sorted them after every real sample,
which is presumably why someone renamed rather than deleted. The boot is now
free of the 'WARNING: File reading failed' wall.

Then three faults in the freshness path, each hiding the next:

1. preload.scd's mismatch banner said 'whitelist is STALE, regenerate it' when
   the cause was unreadable files, and regenerating provably changed nothing.
   Now reports 'N bank(s) DID NOT FULLY LOAD', separates got==0 (whole bank) from
   0<got<expected (folder changed, or some files are not audio), and prints the
   file(1) one-liner that distinguishes them.

2. check-preload.sh compared bank NAME SETS and never counts. The plan said
   rhadamanthe_vocal=77 against a folder of 72 and the checker printed 'ok' while
   the boot said 'expected 77 files, got 72' -- and --fix refused to regenerate,
   because by the name-set test nothing had changed. The everyday way to hit this
   is dropping a new pack into an existing bank: every new file lazy-loads
   mid-set under a green check. Now compares name+count pairs and prints
   'plan N -> disk M' per drifted bank.

3. The bank-name regex was case-blind. [a-z0-9_]+ truncated rampleA0 to rample
   and collapsed every vocalOoh1/vocalScatJ into one 'vocal', which is why the
   checker said 111 banks while SuperDirt loaded 124. 124 was always the truth;
   the regex was wrong in both directions, undercounting the plan AND inventing
   two bank names that do not exist. bank_counts() had inherited it, so drift in
   a camelCase bank stayed invisible even after fault 2 was fixed.

Three numbers now agree: checker 124, boot '124/124 banks OK in 5.2 s',
independent parse 124/124/no gap. Drift tested both directions on a camelCase
bank, plan byte-identical afterwards.
parent 580e0000
...@@ -2155,8 +2155,56 @@ scattered through this file resolve locally. ...@@ -2155,8 +2155,56 @@ scattered through this file resolve locally.
available, i.e. 8%. Fine. Two unresolved track names, both harmless: available, i.e. 8%. Fine. Two unresolved track names, both harmless:
`mafia` is an alias for `mafia_sans_serif.tidal`, already in via opal, and `mafia` is an alias for `mafia_sans_serif.tidal`, already in via opal, and
`Outro Dub Siren` is a live improvisation with no file. `Outro Dub Siren` is a live improvisation with no file.
- **64 AppleDouble resource forks are masquerading as .wav in three banks PLN - ~~**AppleDouble resource forks masquerading as .wav**~~ **DELETED 2026-09-07
plays.** `rhadamanthe_fx` (38), `rhadamanthe_divers` (21), on PLN's explicit say-so** ("no reason to keep these imo"). **198 files, 0.3
MB**, every one confirmed by `file` magic before removal — the guard was
mechanical, not the name pattern, and 198/198 candidates came back
"AppleDouble encoded Macintosh file" with zero false positives.
69 were in playable banks: `rhadamanthe_fx` 227→189, `rhadamanthe_divers`
314→293, `rhadamanthe_vocal` 77→72, `rhadamanthe_melo` 32→30. The other 129
sat under `~/Work/Sound/Samples/baba/__MACOSX/`; the emptied dirs were
`rmdir`-ed (rmdir refuses a non-empty directory, so it cannot take data).
**Sample indices were NOT disturbed**: the `zz.` prefix sorted every one of
them after all real samples, which is presumably why someone renamed rather
than deleted. Only the counts moved. Boot is now free of the
`WARNING: File reading failed for path:` wall, and the preload reports
**124/124 banks OK in 5.2 s** where it used to report 122/124.
- ~~**`preload.scd`'s COUNT MISMATCH banner cries wolf**~~ **FIXED 2026-09-07,
and the real bug was underneath it.** Three separate faults, in order of
discovery:
1. The banner's headline said "whitelist is STALE, regenerate it" when the
cause was unreadable files, and regenerating provably changed nothing. It
now says "N bank(s) DID NOT FULLY LOAD", separates the two causes
(`got == 0` = whole bank failed; `0 < got < expected` = folder changed OR
some files are not audio) and prints the `file`-based one-liner that tells
them apart. *(Correction to an earlier claim of mine: it always DID name
the offending banks — `bad.do { ... }` prints them. My journal grep was
filtering those lines out. Only the headline was wrong.)*
2. **`check-preload.sh` compared bank NAME SETS, never counts.** So a bank
whose contents changed was invisible: the plan said `rhadamanthe_vocal, 77`
against a folder holding 72 and the checker printed **ok**, while
SuperDirt's boot said `expected 77 files, got 72`. Worse, `--fix` refused
to regenerate, because by the name-set test nothing had changed. The
everyday way to hit this is not deleting files — it is **dropping a new
sample pack into an existing bank**, which PLN does, and every new file
would then lazy-load mid-set under a green check. Now compares
`name count` pairs and reports `plan N -> disk M` per drifted bank.
3. **The bank-name regex was case-blind** — `[a-z0-9_]+` truncated `rampleA0`
to `rample` and collapsed every `vocalOoh1`/`vocalScatJ`/... into one
`vocal`. That is why the checker said **111 banks** while SuperDirt loaded
**124**: 124 was always the truth, and the shortfall was the regex, which
was wrong in both directions — undercounting the plan AND inventing two
bank names that do not exist. Found by asking why two numbers never
matched instead of assuming one rounded the other. Fixed to
`[A-Za-z0-9_]+`; and because `bank_counts()` inherited it, drift in any
camelCase bank had still been invisible after fault 2 was fixed.
**Now trustable, three ways agreeing**: `check-preload.sh` says 124, the boot
says `124/124 banks OK`, an independent Python parse says 124 names / 124
pairs / no gap. Drift detection tested in both directions on a camelCase bank
(`vocalOoh1` plan 9 vs disk 13 → STALE exit 1; restored → ok exit 0, plan
byte-identical after the test).
- (superseded) **64 AppleDouble resource forks are masquerading as .wav in
three banks PLN plays.** `rhadamanthe_fx` (38), `rhadamanthe_divers` (21),
`rhadamanthe_vocal` (5) — files named `zz._<original>.wav`, 4096 bytes each, `rhadamanthe_vocal` (5) — files named `zz._<original>.wav`, 4096 bytes each,
and `file` calls them exactly what they are: "AppleDouble encoded Macintosh and `file` calls them exactly what they are: "AppleDouble encoded Macintosh
file". They are macOS metadata, not audio. SuperCollider fails to read them file". They are macOS metadata, not audio. SuperCollider fails to read them
......
...@@ -77,9 +77,35 @@ print(' '.join(str(p) for p in m.setlist_tracks()))") || { ...@@ -77,9 +77,35 @@ print(' '.join(str(p) for p in m.setlist_tracks()))") || {
fi fi
fi fi
# Compare BANK SETS, not file bytes: the generator stamps a header and orders banks, and # Compare BANK NAME **AND FILE COUNT**, not file bytes: the generator stamps a header
# a diff on either would cry wolf on a rebuild that changed nothing that matters. # and orders banks, so a diff on either would cry wolf on a rebuild that changed nothing.
banks() { grep -oE '\[ \\[a-z0-9_]+' "$1" 2>/dev/null | sed 's/.*\\//' | sort -u; } #
# COUNTS ARE THE HALF THAT WAS MISSING (2026-09-07). This compared bank NAME SETS only,
# so a bank whose CONTENTS changed was invisible: the plan said `rhadamanthe_vocal, 77`
# while the folder held 72, and this script reported
#
# check-preload: ok — 111 bank(s) warmed, covering all 39 setlist track(s).
#
# while SuperDirt's own boot said `rhadamanthe_vocal: expected 77 files, got 72`. Worse,
# `--fix` refused to regenerate, because by the name-set test nothing had changed. The
# everyday way to hit this is not deleting files — it is DROPPING A NEW SAMPLE PACK into
# an existing bank, which is a thing PLN does, and every new file in it would then
# lazy-load off disk mid-set with this script saying ok.
#
# The plan already records the count per bank as `[ \name, N, "folder" ]`, so comparing
# `name N` pairs costs nothing and closes the whole class.
# BANK NAMES CONTAIN CAPITALS. `[a-z0-9_]+` truncated `rampleA0` to `rample` and
# collapsed rampleA0/A2/D2/D5 into one entry, and every `vocalOoh1`/`vocalScatJ`/...
# into a single `vocal` — so this reported 111 banks while SuperDirt's own boot loaded
# 124. 124 was always the truth; the shortfall was this regex, and it had been quietly
# wrong in both directions: undercounting the plan AND inventing two bank names
# (`rample`, `vocal`) that do not exist. Found 2026-09-07 by asking why the two numbers
# never matched instead of assuming one of them was a rounding of the other.
banks() { grep -oE '\[ \\[A-Za-z0-9_]+' "$1" 2>/dev/null | sed 's/.*\\//' | sort -u; }
bank_counts() {
grep -oE '\[ \\[A-Za-z0-9_]+, [0-9]+' "$1" 2>/dev/null \
| sed -E 's/\[ \\([A-Za-z0-9_]+), ([0-9]+)/\1 \2/' | sort
}
new=$(mktemp); trap 'rm -f "$new"' EXIT new=$(mktemp); trap 'rm -f "$new"' EXIT
# shellcheck disable=SC2086 # shellcheck disable=SC2086
...@@ -88,23 +114,34 @@ python3 tools/setlist_samples.py $tracks --emit-sc > "$new" 2>/dev/null || { ...@@ -88,23 +114,34 @@ python3 tools/setlist_samples.py $tracks --emit-sc > "$new" 2>/dev/null || {
if [ ! -f preload.scd ]; then if [ ! -f preload.scd ]; then
echo "check-preload: preload.scd does NOT EXIST — every sample will lazy-load on first play." echo "check-preload: preload.scd does NOT EXIST — every sample will lazy-load on first play."
missing=$(banks "$new" | wc -l); have=0 missing=$(banks "$new" | wc -l); have=0; changed=0
else else
missing=$(comm -13 <(banks preload.scd) <(banks "$new") | wc -l) missing=$(comm -13 <(banks preload.scd) <(banks "$new") | wc -l)
have=$(banks preload.scd | wc -l) have=$(banks preload.scd | wc -l)
# Banks present in BOTH plans whose file COUNT moved. This is the check that
# was missing: a new pack dropped into an existing bank changes no bank name,
# so the name-set test above sees nothing and every new file lazy-loads.
changed=$(comm -3 <(bank_counts preload.scd) <(bank_counts "$new") \
| awk '{print $1}' | sort -u | grep -c . || true)
fi fi
want=$(banks "$new" | wc -l) want=$(banks "$new" | wc -l)
if [ "$missing" -eq 0 ] && [ -f preload.scd ]; then if [ "$missing" -eq 0 ] && [ "${changed:-0}" -eq 0 ] && [ -f preload.scd ]; then
echo "check-preload: ok — $have bank(s) warmed, covering all $(echo "$tracks" | wc -w) setlist track(s)." echo "check-preload: ok — $have bank(s) warmed, covering all $(echo "$tracks" | wc -w) setlist track(s)."
exit 0 exit 0
fi fi
echo "check-preload: STALE — preload.scd warms $have bank(s), the set needs $want." echo "check-preload: STALE — preload.scd warms $have bank(s), the set needs $want."
echo " $missing bank(s) would be read from DISK on first play (crackle, mid-transition):" if [ "$missing" -gt 0 ]; then
if [ -f preload.scd ]; then echo " $missing bank(s) MISSING — read from DISK on first play (crackle, mid-transition):"
comm -13 <(banks preload.scd) <(banks "$new") | head -20 | sed 's/^/ /' comm -13 <(banks preload.scd) <(banks "$new") | head -20 | sed 's/^/ /'
fi fi
if [ "${changed:-0}" -gt 0 ] && [ -f preload.scd ]; then
echo " ${changed} bank(s) CHANGED SIZE — the plan's count no longer matches the folder,"
echo " so SuperDirt will report 'expected N files, got M' and warm the bank short:"
join <(bank_counts preload.scd) <(bank_counts "$new") 2>/dev/null \
| awk '$2 != $3 {printf " %-24s plan %s -> disk %s\n", $1, $2, $3}' | head -20
fi
if (( FIX )); then if (( FIX )); then
cp "$new" preload.scd cp "$new" preload.scd
......
...@@ -189,8 +189,30 @@ def emit_sc(resolved, unresolved, tracks): ...@@ -189,8 +189,30 @@ def emit_sc(resolved, unresolved, tracks):
"=== PRELOAD: %/% banks OK in % s ===".format( "=== PRELOAD: %/% banks OK in % s ===".format(
\tok, ~pvPreload.size, (Main.elapsedTime - t0).round(0.1)).postln; \tok, ~pvPreload.size, (Main.elapsedTime - t0).round(0.1)).postln;
if(bad.notEmpty) { if(bad.notEmpty) {
\t"=== PRELOAD COUNT MISMATCH — whitelist is STALE, regenerate it ===".postln; \t// SAY WHAT HAPPENED, NOT WHAT TO TYPE. This banner used to read
\t// "whitelist is STALE, regenerate it" — which was wrong twice on
\t// 2026-09-07: it fired on a plan generated sixty seconds earlier, and
\t// regenerating changed nothing, because the cause was 64 AppleDouble
\t// resource forks named zz._*.wav that the generator counted as files and
\t// scsynth could not read as audio. A banner that names the wrong remedy
\t// is worse than none: it teaches you to run a command that cannot help,
\t// and then to ignore the banner.
\t//
\t// The two causes are distinguishable, and the fix differs:
\t// got == 0 the whole bank failed (missing folder, or nothing
\t// in it is audio scsynth can open)
\t// 0 < got < expected the folder is there and SOME files failed. Either
\t// the folder changed since the plan was written
\t// (genuinely stale) or some files are not audio.
\t"=== PRELOAD: % bank(s) DID NOT FULLY LOAD ===".format(bad.size).postln;
\tbad.do { |m| (" " ++ m).postln }; \tbad.do { |m| (" " ++ m).postln };
\t(" Compare the counts against the folder as it is NOW:"
\t\t" tools/check-preload.sh").postln;
\t(" If the folder's count MATCHES 'expected', the plan is fine and those"
\t\t" files are not readable audio — find them with:").postln;
\t(" find <folder> -maxdepth 1 -type f -exec file {} + | grep -iv 'wave\\|aiff\\|flac'").postln;
\t(" Only if the count DIFFERS is the plan stale:"
\t\t" tools/check-preload.sh --fix").postln;
};""") };""")
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment