Commit f140b0e8 by PLN (Algolia)

docs(tasks): archive #27 (artifact store) + #29 (/sources)

Rich entries for the documentary trail: the freebox-deferred simplification of
#27, the open-url-scope-with-apikey-trust decision on #29, the SSRF block set,
and how #25/#26 pre-built the seams (result_ref/gc, alias table, born-done path)
that made both mostly wiring.
parent c7e5392b
...@@ -356,3 +356,15 @@ memory `project_fourier_api`. Named after Joseph Fourier — the transform behin ...@@ -356,3 +356,15 @@ memory `project_fourier_api`. Named after Joseph Fourier — the transform behin
- **Done:** committed 17d3f46. Four design decisions taken with PLN (AskUserQuestion w/ sketches): HYBRID sync/async (cheap analyses stay sync, heavy = jobs), PER-ENGINE submit (202 {job_id}) + SHARED poll (GET /jobs/{id}) + cancel (DELETE), SINGLE serialized worker, X-Accel-Redirect for artifacts. Built: jobs table (db.py), jobs.py (enqueue + born-done idempotent fast path, atomic claim, progress/finish/fail-requeue-under-cap, crash recovery, cooperative cancel), worker.py (serialized recover→claim→dispatch→webhook loop, SIGTERM-graceful, @jobs.handler registry), typed JobStatus endpoints, deploy/fourier-worker.service (systemd --user). 11 new tests (67→78), incl. a real 6-thread×25-job no-double-claim concurrency test. OpenAPI refreshed (15 ops) + @nech/api regenerated (getJob/cancelJob). - **Done:** committed 17d3f46. Four design decisions taken with PLN (AskUserQuestion w/ sketches): HYBRID sync/async (cheap analyses stay sync, heavy = jobs), PER-ENGINE submit (202 {job_id}) + SHARED poll (GET /jobs/{id}) + cancel (DELETE), SINGLE serialized worker, X-Accel-Redirect for artifacts. Built: jobs table (db.py), jobs.py (enqueue + born-done idempotent fast path, atomic claim, progress/finish/fail-requeue-under-cap, crash recovery, cooperative cancel), worker.py (serialized recover→claim→dispatch→webhook loop, SIGTERM-graceful, @jobs.handler registry), typed JobStatus endpoints, deploy/fourier-worker.service (systemd --user). 11 new tests (67→78), incl. a real 6-thread×25-job no-double-claim concurrency test. OpenAPI refreshed (15 ops) + @nech/api regenerated (getJob/cancelJob).
- **Learnings:** SQLite WAL serializes writers, so the claim race reduces to candidate-select → `UPDATE ... WHERE id=? AND status='pending'` + rowcount-0 retry — no locks needed, provably exactly-once across threads. typescript-fetch's generated instanceOf guard indexes a camel/snake key union that trips TS7053 under noImplicitAny → relaxed noImplicitAny for the 100%-generated package (emitted .d.ts stays fully typed). MID-BUILD: the btrfs disk hit ENOSPC at 98% (20G "free" but unallocated, metadata-full) — writes failed intermittently; cleared ~5.7G of regenerable caches (npm + ~/.cache browser/HF, user-approved) to continue. Disk still 97% — flagged for real cleanup. - **Learnings:** SQLite WAL serializes writers, so the claim race reduces to candidate-select → `UPDATE ... WHERE id=? AND status='pending'` + rowcount-0 retry — no locks needed, provably exactly-once across threads. typescript-fetch's generated instanceOf guard indexes a camel/snake key union that trips TS7053 under noImplicitAny → relaxed noImplicitAny for the 100%-generated package (emitted .d.ts stays fully typed). MID-BUILD: the btrfs disk hit ENOSPC at 98% (20G "free" but unallocated, metadata-full) — writes failed intermittently; cleared ~5.7G of regenerable caches (npm + ~/.cache browser/HF, user-approved) to continue. Disk still 97% — flagged for real cleanup.
- **Deps:** was blocked by scaffold + central auth (done); unblocks #29 → #30 → #31. Worker prod container-vs-host wiring is a paved-road residual (SRE/#34). - **Deps:** was blocked by scaffold + central auth (done); unblocks #29 → #30 → #31. Worker prod container-vs-host wiring is a paved-road residual (SRE/#34).
## #27 — Artifact store (local disk on erable; freebox deferred)
- **Description:** somewhere to keep the big rebuildable binaries (fetched sources, later stems/loops) that don't belong inline in SQLite, served efficiently to hexa.
- **Done:** committed c7e5392. `artifacts.py`: content-addressed layout `ARTIFACTS_ROOT/<cid>/<name>`, tracked as ordinary `cache` rows (kind→result_ref→path, bytes) so the **existing LRU `cache.gc()` evicts them for free** under `FOURIER_ARTIFACTS_MAX_MB` (1 GB). `resolve()` refuses path traversal / escapes outside the root. Serving = zero-copy nginx `X-Accel-Redirect` (`FOURIER_X_ACCEL`), `FileResponse` fallback in dev. `GET /artifacts/{cid}/{name}` endpoint (auth-gated, any tenant holding the opaque ref). DEPLOY.md §5b documents the internal nginx location. The cache schema's `result_ref` column + `gc()` unlink were pre-built in #25/#26 *for* this — #27 was mostly wiring.
- **Learnings:** PLN dropped freebox-as-SSOT ("forget freebox storage for now") → the store collapsed from "signed Range URLs over a remote mount" to "local dir + nginx internal location", much simpler. Reusing the cache table for artifacts means one eviction policy, one stats surface, one GC — no separate artifact lifecycle. The `_meter` middleware had to switch from `request.url.path` to the **route template** (`request.scope["route"].path_format`) or `/artifacts/{cid}` + `/jobs/{id}` would each mint an unbounded Prometheus series.
- **Deps:** was blocked-by-nothing once #25 landed; unblocks #30 (stems need somewhere to live). Go-live (nginx location + FOURIER_X_ACCEL) tracked in #47.
## #29 — /sources endpoint (yt-dlp fetch as a job)
- **Description:** turn a URL into analysis-ready audio — the front door of the heavy chain (sources→separate→loops).
- **Done:** committed c7e5392. `engines/sources.py` registers the `sources` job handler: yt-dlp → `bestaudio` → 44.1k WAV (PLN's choice: analysis-ready PCM over keep-source-codec), content-address, store as a `source` artifact, `put_alias(url→cid)`. `POST /sources` returns `202 {job_id}`; a re-fetched URL (alias hit + artifact still on disk) is a **born-`done` job** — no re-download. `normalize_url()` validates: http(s) only, lowercases host, drops fragment (so trivially-different URLs dedupe), and hard-blocks SSRF footguns (localhost, RFC1918/loopback/link-local/reserved IP literals, incl. the 169.254.169.254 cloud-metadata classic). The fetch is a thin seam (`_fetch_audio_wav`) so tests mock the network with a synthetic WAV — 17 new tests (78→95), zero network. healthz reports yt-dlp presence honestly.
- **Learnings:** PLN chose OPEN url scope (any http(s)) over an allowlist, explicitly leaning on "our apikey + nechapi monitoring" as the protection layer — so the code accepts any public URL but still refuses the SSRF set in-process (defense the gateway can't do). yt-dlp+ffmpeg are **worker-only** deps (the API container stays torch/yt-dlp-free) → added to requirements-deploy, not requirements-test. The idempotent born-done path falls out of the #25 `enqueue(..., result=...)` fast path + #26 aliases — the URL→cid alias table was designed in #26 *for* exactly this.
- **Deps:** blocked-by #25 (done); unblocks #30 (separate consumes the fetched WAV). Go-live (yt-dlp install on erable) tracked in #47.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment