Commit 67d82468 by PLN (Algolia)

docs(todo): persist open task state into project TODO.md files + archive #44

So the session task board survives a push to git.nech.pl (it is local harness state,
not in git). Two cold-readable TODOs next to their code:
- armada/api/TODO.md — Fourier audio-API remaining: heavy chain #30 /separate +
  #31 /loops/grade/correlate (need a GPU runner), optional #47 X-Accel + #46 Grafana,
  follow-ups (nech_api python client, hexa npm read token); EPIC #21 closes with the chain.
- tools/foundry/TODO.md — #19 auto-tune loop (in-progress, harness validated +10.5%,
  exact resume steps) + #20 batch-explore corpus.
Plus archived #44 (Verdaccio + @nech/api live) to completed-archive.md.
Root TODO.md left untouched (it is the paused Pulsar livecoding-perf session).
parent 851767ff
# Fourier (audio API) — TODO
Resume state for the `audio` sub-API of api.nech.pl (codename **Fourier**).
Snapshot **2026-06-29**. Task IDs map to the session task board; completed work
lives in [`../tasks/completed-archive.md`](../tasks/completed-archive.md).
## ✅ Live in production (api.nech.pl/audio/v1)
- 11 analysis engines (emotion/features/samples/grade/onsets/waveform/loudness/
spectrum/naming/…), torch-free light CPU. Central auth + hierarchical scopes.
- **Async jobs + worker** (#25), **artifact store** (#27), **`/sources`** yt-dlp
fetch (#29) — `fourier` + `fourier-worker` containers on erable.
- **`/docs` + `/openapi.json` gated** behind `api:docs` (#48).
- **`@nech/api`** generated TS client published to `npm.nech.pl` (Verdaccio, #44).
- Deployed via build-on-dev → `docker save|gzip|ssh erable docker load`. Recipe +
gotchas: memory `reference_fourier_deploy`, [`DEPLOY.md`](./DEPLOY.md),
[`VERDACCIO.md`](./VERDACCIO.md).
## 🔴 Open — the heavy chain (needs a GPU runner)
- [ ] **#30 — `POST /separate`** (demucs/roformer/merge as a heavy job). Wrap
`foundry.engine.separate` (pluggable REGISTRY + MultiModelBackend merge).
Stems → FLAC artifacts cached by `(content_id, backend, model)`. `GET
/separate/{content_id}?backend=` returns stem artifact refs or job status.
Surface `Backend.available()` so an absent backend → clean 4xx not 500. Keep
a cloud-GPU-runner adapter seam (no API change later). erable has **no GPU**
→ the route is a deliberate `503 compute_unavailable` until a runner exists.
Validate: pytest with a fake separator (submit→poll→stem refs; cache hit;
merge 4-stem contract; unavailable backend → 4xx) + one gated real clip.
*Blocked by: #29 ✓, #27 ✓ — so unblocked except for the GPU itself.*
- [ ] **#31 — `/loops` `/grade` `/correlate`** (full Foundry chain over the API).
`/loops {content_id, mode:loops|chops, bars?, top_n?}` over cached stems
(`?auto_separate=true` chains a separate job); `/grade` wraps
`engine.grade.grade`; `/correlate` locates a loop's provenance (NCC); `POST
/loops/export` applies B-rules naming → ZC-snapped FLAC. Validate with the
foundry fixtures (correlate ~1.0 on known slice, ~0.03 foreign).
*Blocked by: #30.* **Completing this closes EPIC #21.**
## 🟡 Optional / later
- [ ] **#47 — X-Accel artifact serving** (perf). Replace the app's FileResponse
with nginx `X-Accel-Redirect`: `FOURIER_X_ACCEL=1` + a per-API `/_artifacts/`
internal location (DEPLOY.md §5b) → means extending `render-gateway.py` to
emit per-API extra locations. Only worth it at high throughput / large stems;
FileResponse is correct + fine at current scale.
- [ ] **#46 — Grafana over Postgres** (`nechapi-db`: usage_log + health_checks).
Panels: calls/tenant/route, reject reasons + offender IPs, uptime%/latency.
Visual layer atop the existing `/admin/usage` `/admin/security` `/status`.
Add grafana.nech.pl (DNS+certbot) or serve under /admin/grafana; also a
nightly usage_monthly rollup. *Not urgent — /admin/* covers the need.*
- [ ] **`nech_api` Python client** — second generated client (openapi-generator
python) alongside `@nech/api`, same `npm.nech.pl`-style distribution.
- [ ] **Mint hexa an npm read token** for `@nech/api` (`@nech/*` reads are
auth-gated). Mint via the Verdaccio user API; hand off with their API bearer.
- [ ] Build the platform `/docs` Scalar portal (basic-auth RTFM/PLN) + fix the
gateway 404's dangling "See /docs" hint.
## EPIC
- [ ] **#21** closes when the heavy chain (#30 → #31) ships. The analysis MVP is
already live; this is the separation/loops half.
......@@ -374,3 +374,9 @@ memory `project_fourier_api`. Named after Joseph Fourier — the transform behin
- **Done:** deployed live 2026-06-29. `required_for()` special-cases `docs|openapi.json|redoc``{realm}:docs` (canonical `nechapi/_platform/scopes.py` commit on branch `claude/docs-scope`, rebased onto upstream + merged to main `0ab148b`; vendored byte-identical into `armada/api/scopes.py`, commit `d4d27a4`). Dropped docs/openapi from `audio.public_routes` (kept `/healthz`), re-rendered the vhost. Rebuilt + shipped `nechapi-platform:latest` (Alpine, Postgres-backed so no token loss), recreated the container on `nechapi-net`, installed the re-rendered vhost (diffed-first: only my 2 blocks removed), `nginx -t && reload`. VERIFIED LIVE: anon /docs+/openapi.json → 401 (was 200), /healthz → 200; a minted `api:docs` token → /docs 200 but POST /analyze/emotion → 403 (decoupling proven); test token revoked. `api:*` (hexa/pln) keeps docs access via subtree.
- **Learnings:** the cross-cutting scope (`api:docs`, not per-API `api:audio:docs`) decouples "read the specs" from "call the API" — a docs-reader token can't invoke anything, and `api:*` still covers both via the subtree wildcard. The drift guard reads the MAIN `~/Work/nechapi` checkout, so the canonical + vendored copies must merge in LOCKSTEP — committing the vendored copy ahead of the canonical merge turns the local suite red (CI skips it when the canonical is absent). Resolved by reverting the premature vendor, staging on a branch, and re-vendoring only after the nechapi merge. Also re-learned the shared-repo race: origin/main had advanced (a parallel SRE `bin/nechapi` drift-guard commit) — rebased onto it (no overlap), and ALWAYS diff the rendered vhost vs the LIVE one before `sudo tee` so a parallel session's work isn't clobbered.
- **Deps:** spun out of the #29/#27 deploy session; touched the security control plane (gated). Follow-up: build the platform `/docs` Scalar portal (basic-auth RTFM/PLN per registry) + fix the 404 "See /docs" dangling hint.
## #44 — Official generated clients + private npm (Verdaccio)
- **Description:** ship a real, never-drifts client for hexa + future seats, distributed from a private registry — replacing the interim hand-written `nech.ts` drop-in.
- **Done:** deployed live 2026-06-29. (a) Generated `@nech/api` TS client from the OpenAPI snapshot (openapi-generator typescript-fetch → tsup ESM bundle; clean operationIds → `audio.analyzeEmotion()`; subpath exports). (b) Stood up **Verdaccio 6** on erable (loopback `:4873`, `~/srv/verdaccio/{storage,conf}`, `unless-stopped`) behind an LE-TLS nginx vhost **`npm.nech.pl`**; seeded publisher `pln` via the user API, locked signups (`max_users:-1`), creds 0600 at `~/.config/nechapi/verdaccio.pln.{pw,token}`. (c) Published `@nech/api@0.1.0` and install-verified as a consumer (AudioApi/Configuration import clean). (d) Retired `nech.ts` (`git rm`), updated README + onboarding.html to the `@nech/api` install path. Commits: client `ed77ee9`/regens, retire `851767f`; VERDACCIO.md marked DEPLOYED.
- **Learnings:** three Verdaccio gotchas the bare runbook missed — `listen: 0.0.0.0:4873` (else binds localhost INSIDE the container, dead `-p` map), `chown -R 10001:65533` the bind-mount (verdaccio uid), and `ssl_protocols TLSv1.2;` ONLY (erable's old OpenSSL rejects TLSv1.3 and fails `nginx -t`). Non-interactive publish: create the user via `PUT /-/user/org.couchdb.user:<name>` to get a token, then `npm publish --userconfig <temp .npmrc with _authToken>` — no interactive `npm adduser`/`login` needed. KEY DECISION: `@nech/*` reads are `$authenticated` (not public) — consistent with the bearer-gated `/openapi.json` (#48); so consumers need an npm READ token, a deliberate consistency call over install-convenience.
- **Deps:** unblocked by the OpenAPI snapshot (#32); part of EPIC #21. Follow-ups (in `armada/api/TODO.md`): a `nech_api` Python client; mint hexa a read token.
# The Foundry — TODO
Resume state for the URL→audio→stems→loops sampling tool. Snapshot **2026-06-29**.
Task IDs map to the session task board. Design: memory `project_foundry`.
## 🔵 In progress
- [ ] **#19 — Auto-tune loop** (create samples → grade → iterate sampling settings).
PLN's closed-feedback main quest. **Harness BUILT & VALIDATED** (commit
`6f85012`): `engine/autotune.py` (evaluate/search), `autotune.py` CLI,
`loops.analyze_stem` `weights` override, 4 unit tests + a real run (baseline
0.690 → best 0.763, **+10.5%** on 2 drum stems). Finder defaults (`loops.W`)
deliberately UNCHANGED. **Exact next steps to resume cold:**
1. Get real fuel: `cd tools/foundry && python3 foundry.py demos --all --sep`
(fetches+separates the 10 sampling classics — SLOW, run as a durable bg
job). Existing fuel: `~/Downloads/separated/htdemucs/*/{drums,bass,vocals,
other}.wav` (27 tracks).
2. Add the anti-gaming 2nd objective: recall-vs-provenance-GT (reuse
`armada/tide-table/build_finder_eval.py` + `build_provenance.py`) so we
don't just amplify the seam/zc confound. Make objective multi-objective
(grade × recall) in `engine/autotune.py`.
3. Per-stem-ROLE tuning: vocals want chop mode + different weights than drums
(`--stems …/drums.wav` vs `…/vocals.wav`; analyze_chops path for vocals).
4. Full campaign: more stems (`--max-stems`↑) + more configs (`--n`↑); factor
window-feature extraction out of `analyze_stem` so weight sweeps are
instant (extract-once/score-many — the scarcity-minded speedup).
5. THEN adopt a VALIDATED profile as the new `loops.W` default, per role,
with the lift numbers.
Run: `python3 autotune.py --stems <glob> --n 24 --max-stems 4`.
Objective so far = mean full-rubric grade × coverage.
## 🔴 Open
- [ ] **#20 — Batch-explore many sources** (Foundry over the full corpus + merge).
Workflow is one-catch-at-a-time; build batch exploration:
- Batch driver: `find_takes` over MANY source stems (the ~30 in
`~/Downloads/separated/htdemucs` + new fetches) → grade → surface the best
loops/chops corpus-wide ("what's worth sampling across everything pulled").
- **Stem-role-aware**: skip near-silent stems, mode per role (chops for
vocals, loops for rhythmic), rank by grade. (e.g. Loituma has no drums —
bass + high voice are the interesting stems; grid-from-drums already falls
back drums→bass→first.)
- Surface in the Hall of Fame / a "fresh catches" view: best ungraded-yet
candidates corpus-wide, click to audition + forge.
- Ties to #19: once settings are tuned per role, batch-explore applies them
at scale. Relates #11 tierlist (showcase pattern), #9 merge.
> Note: the Foundry engines are also productized over HTTP by the **Fourier**
> audio API — `/separate` (#30) wraps `engine.separate`, `/loops /grade /correlate`
> (#31) wrap the rest. See `armada/api/TODO.md`.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment