Commit 7232003f by PLN (Algolia)

docs: board + archive for the stem-pack batch work

#20 lands; #21 (auditioner SPA), #22 (acapellas + demucs catches), #23 (audit the existing
163 kits) opened with enough detail to start cold. The Fred pack's finish sequence is
written out step by step because the final run is still in flight.
parent 3ec21fc9
......@@ -1628,3 +1628,78 @@ cannot validate a stem-based lens — that gap should be closed before attempt f
**Deps.** Superseded in practice by the boundary lab (#B2), which made the ear call cheap
enough that the detector stopped being necessary.
## #20 — Batch the Foundry over producer stem packs
**Description.** Fred again.. dropped 36 GB of stems — 9 tracks split by the producer
himself into 6–13 named stems each (`MAREA MIX10 123BPM KIT STEM.wav`), plus 13
acapella/instrumental pairs. Cutting that into TidalCycles kits by hand in Audacity is the
drudgery the loop finder was built to kill. Goal: `fred_XXX` kit dirs, one per track, each
holding loops cut from that track's stems — and find out whether the finder actually works
on someone else's stems.
**Done.** Branch `claude/fred-kits`, 14 commits. `engine/stempack.py` (batch driver),
`engine/roles.py` (~40 role tokens → family, verified by measurement), `engine/kitcheck.py`
(the audit `grade` structurally cannot do), `kitgate.py` (the one gate before loading),
`restage.py` (gain-staging for kits already on disk), `--report-only` (rebuild views from
the manifest), `packs/` (provenance, not audio). Suite 86 → 112. Write-up:
`armada/tasks/027-the-label-and-the-lens.md`.
**Learnings.** The finder was good; everything that was wrong was found by RUNNING it and
measuring the output, never by reading code.
- **96 kHz silently broke it.** librosa's hop/window defaults are sample-rate-relative, so
at 96 k every analysis frame spans half the musical time. MAREA declares `123BPM` in its
filenames = free ground truth: native gave 123/123/124/**119**/**128.1**/**128.5** bpm at
a 0.661 top score; resampled to 44.1 k first, 123.0–123.1 on every candidate at 0.859, in
a third of the time. Rate is a correctness input, not a performance knob.
- **Bar length must come from the TRACK, not the window.** Candidate bpm is a per-window
mean — right for drift, wrong for a bar length: ME's grid read 129.2 while both its loops
were cut at 150.5, so no two loops in a kit layered. Use the MODE of the local-BPM
histogram plus a confidence (share of beats within ±5 %). The 35 % gate landed in a clean
empty gap: every track ≥46 % produced good loops, the one at 20 % correctly produced none.
- **Chops beat loops mechanically, always.** `grade` scores a sub-0.75 s clip as a one-shot
on `0.5·level + 0.3·dc + 0.2·zc` — no seam, no bar term — and export already removes DC
and snaps zero crossings, so chops land near 1.0 while loops pay 0.35 on seam. Measured:
chops 0.993–0.999, loops 0.841–0.881, and the vocal kit came out as three stabs with no
loops. Never rank two modes against one rubric.
- **The periodicity check measured loop LENGTH, then ROLE.** Autocorrelation at lag *k* has
only *N−k* overlapping frames; normalised by `ac[0]` it scored a *perfectly periodic*
click track 0.418/0.623/0.754 at 2/4/8 bars. Divide by the per-lag overlap count → flat.
Then the floor had to go per family (drums 0.754, fx 0.549, tonal 0.214, bass 0.173, vox
0.087): one threshold condemned 12 of 20 vocal loops for being vocal.
- **Dead bars grade S.** A 4-bar loop ending in a fade is silent-to-silent (perfect seam) at
an exact bar length. 10 of the first 161 cuts. An objective defect with an objective test
belongs in the gate, not a report.
- **`amp = gain^4` means kits must be staged.** MAREA's ALL DRUMS and KIT both peak at
**+0.2 dBFS**, so `# gain 1.2` is +7.6 dB and the orbit clips on contact. −6 dBFS ceiling,
as ONE gain per kit (per-file would flatten a sparse hat against a full drum bus) and
never boosting (a quiet stem is quiet on purpose).
- **One rotation pass leaves 1 loop in 6 opening on a weak beat** — the "timed ok but cut
BCDA/DABC" complaint from 2026-07-11. The roll point is zc-snapped up to ~10 ms against a
20 ms attack window, enough to flip near-tied slots. Check-and-repeat: best slot 29 → 46
of 60, worse half 10 → 1.
- **Dead end: HPSS for percussiveness.** It hears a modern kick as harmonic and calls a drum
bus 9 % percussive. Also slow. Frame-RMS crest gating onset density is the cheap win — but
crest is a *sparsity* measure (a single vocal phrase reads 17.98), so it may only gate.
- **The checks caught two bugs in my own work**, which is the whole argument for them:
`kitgate` re-grades the shipped files rather than trusting the manifest, and found that
staging applied its gain AFTER grading, so the record described audio never written. And a
report section never rendered — a `str.replace` whose anchor didn't match, in the one patch
where I hadn't asserted on it.
- **Nearly filed a false alarm:** the first "do loops start on the downbeat" measurement said
17 % and looked like a broken feature. It scored 16 slots with no full-band term while the
rotator scores `bars×4` WITH one — the disagreement was my instrument's. Factored the
scorer out so the two cannot drift, re-measured: 48 % best-slot, median miss 0.032. The bug
was real *and* the first number was wrong.
- **The deepest one is about authority.** "Never infer role from the name" is not "the
measurement always wins." Measurement called `ALL DRUMS` a bass (a modern kick carries the
mix's energy: lf 0.86, centroid 216 Hz), rhythmic `KEYS2` drums (5.1 onsets/s), sparse
`CLAP` tonal. Presence and bass are decided by measurement; everything else keeps the
producer's label, because a human's statement about their own stem is evidence, not
inference. A label WE invented (an unrecognised token) loses to measurement. Disagreements
get reported, never silently resolved. Banked as `feedback_label_and_lens`.
**Deps.** Unblocked #21 (the auditioner SPA PLN asked for), #22 (same driver at the
acapellas + the 27 demucs catches), #23 (kitcheck/restage over the 163 existing kits).
Runs 1 and 2 were discarded — each predated a fix; run 3 is the deliverable.
# The Foundry — TODO
Resume state for the URL→audio→stems→loops sampling tool. Snapshot **2026-06-29**.
Resume state for the URL→audio→stems→loops sampling tool. Snapshot **2026-08-21**.
Task IDs map to the session task board. Design: memory `project_foundry`.
## 🔵 In progress
......@@ -43,8 +43,89 @@ Task IDs map to the session task board. Design: memory `project_foundry`.
other top-3 BEFORE B/B/B → AFTER **S/S/S** (bad favorite @56.74 demoted out)
+2 rerank regression tests + 1 sparse-percussion test; suite 57→60 green.
## 🟢 Landed (2026-08-21) — #20 batch, via producer stem packs
`engine/stempack.py` + `engine/roles.py` + `engine/kitcheck.py` + `kitgate.py` +
`restage.py`, branch `claude/fred-kits`, 14 commits, suite 86 → 112.
Full write-up: `armada/tasks/027-the-label-and-the-lens.md`, memory
`reference_stempack` + `feedback_label_and_lens`.
```bash
cd tools/foundry
python3 -m engine.stempack "<pack>" --jobs 9 --no-link --out /tmp/cut
python3 kitgate.py /tmp/cut/fred_kits.json # then --link
```
## 🟡 In progress — finish the Fred pack (RESUME HERE)
The **3rd/final full run** was launched 17:32 (`pid 214097`, ~30 min, 9 parallel
children) writing to
`/tmp/claude-1000/-home-pln-Work-Sound-Tidal/51ee4fa8-7503-47a3-a4e5-cf19e930564b/scratchpad/final`
with log `…/final.log`. Runs 1 and 2 (161 and 153 samples) were DISCARDED — each
predated a fix. If that scratchpad is gone, just re-run the command above with
`--out <fresh dir>`; it takes ~30 min on 9 cores and is fully deterministic.
**Exact steps to finish, in order:**
1. `python3 restage.py --cuts <out>/fred_kits.json --dry-run` — expect **0 kits**
changed, because run 3 stages at export. Anything else means the export path
regressed.
2. `python3 kitgate.py <out>/fred_kits.json` — must be all-green on grade / kitcheck /
bank shadowing. `rights` is advisory and WILL warn (correctly) until step 3.
3. `python3 ../../armada/tide-table/register_fred_rights.py <out>/fred_kits.json`
— registers every `fred_*` bank as `third_party` **with provenance**, so a per-track
release blocks by design. Do this BEFORE linking.
4. `python3 kitgate.py <out>/fred_kits.json --link` — the Dirt-Samples symlinks
(PLN asked for this explicitly). Then `python3 ../bank-check.py --since 1` to confirm
0 shadowed.
5. Copy `fred_kits.{json,md,tidal}` + `kitcheck.md` into
`tools/foundry/packs/fred/` and commit (see `packs/README.md` — manifest, not audio).
6. Report to PLN: per-kit tier histogram, distinct-sound count, the "worth a second
listen" table, and the paste-ready `n` indices.
**Expected shape** (from run 1/2, so approximate): ~40 kits, ~150 samples, ~9 tracks.
Tempo confidence gates one track (PLEASEMAKEITBETTER, 20%) to chops-only — correct.
MAREA is the ground truth: grid must read **123.0 bpm vs its declared 123**.
## 🔴 Open
- [ ] **#20 — Batch-explore many sources** (Foundry over the full corpus + merge).
- [ ] **#21 — The kit auditioner SPA (asked for 2026-08-21): browse + loop + sequence.**
PLN: *"spawn a reusable SPA so i can listen to the samples, try each as a loop,
quickly sequence — could be a strudel playground, i'm open."* Design facts already
established, so this can start cold:
- **Where:** a new page in `armada/ui/` (Vite + React 19 + TS + Tailwind 4), which
ALREADY depends on `wavesurfer.js` + `@wavesurfer/react` and already ships
sibling pages this way (`bounds.html`, `judge.html`, `sextant`). Follow that
pattern; design language = the Ship's Bridge (`armada/DESIGN.md`), manage with
the `impeccable` skill from `armada/`.
- **Serving the audio:** add a `samples` prefix to `armada/ui/audio-mounts.json`
`/home/pln/Work/Sound/Samples`. That file is read by BOTH `vite.config.ts`
(dev) and `armada/serve.py` (prod), so `/audio/samples/<kit>/<file>.wav` works
identically in both. Same-origin, so no CORS work.
- **CANNOT be a published Artifact.** The Artifact CSP blocks external hosts and
the sandbox cannot reach a localhost audio server, so the page must be served by
`armada/serve.py`. Don't waste a session rediscovering this.
- **Sequencing = Strudel**, which is Tidal syntax in the browser, so a pattern
built there is nearly copy-pasteable into a `.tidal` file. `npm i @strudel/web`;
`initStrudel()`, then `evaluate("<pattern>")` / `hush()`. Load the kits with
`samples()` pointed at a generated `strudel.json` map — write a small generator
from `fred_kits.json` (it already carries kit, path, bars, bpm, tier, family,
stem_role, dur_s and CLAP tags, so the browser needs no analysis).
PLN's own research notes: `~/Downloads/Summarize how to integrate strudel js
music framew.md`.
- **The data is already browsable:** filter by track / family / tier, sort by
grade, show CLAP tags, and surface the report's "worth a second listen" rows.
- Suggested split: left = kit list, main = sample rows (waveform, tier badge,
bars@bpm, loop-toggle audition), bottom = Strudel pane with click-to-insert
(`s "fred_marea_drums" # n 3`). Confirm the layout with PLN before building deep.
- [ ] **#22 — Point stempack at the rest.** Same driver, no new code:
the 13 ACAPELLAS + 13 INSTRUMENTALS in `~/Work/Sound/Samples/Fred/` (already
44.1 kHz, so no resample; acapellas want chops mode), and the ~27 demucs catches in
`~/Downloads/separated/htdemucs/`. Each needs a `--prefix`.
- [ ] **#23 — Run `kitcheck`/`restage` over the whole existing corpus.** The dead-bar and
periodicity tests are general, and `Samples/` has 163 kits that have never been
audited; hand-cut kits also carry whatever level their source had.
`python3 restage.py --all-under ~/Work/Sound/Samples --dry-run` is the first look.
Expect "grades S, useless" to be non-rare.
- [ ] **#20b — Batch-explore the merged corpus** (the original #20 framing).
Workflow is one-catch-at-a-time; build batch exploration:
- Batch driver: `find_takes` over MANY source stems (the ~30 in
`~/Downloads/separated/htdemucs` + new fetches) → grade → surface the best
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment