-
feat(audio-api): async job backbone + worker daemon (#25) · 17d3f468
The async spine for the heavy chain (sources/separate/loops): the API enqueues and returns a job_id; a worker runs it out-of-band so the API never blocks. Built to the four decisions taken with PLN: - HYBRID sync/async: the 11 cheap analyses stay synchronous (cache-backed); only heavy work becomes a job. The line is wall-clock cost, not endpoint kind. - PER-ENGINE submit + SHARED poll: engines return 202 {job_id} (wired in #29/#30/#31); GET /jobs/{id} polls, DELETE /jobs/{id} cancels. Typed JobStatus response_model so the generated client gets audio.getJob()/cancelJob(). - SINGLE serialized worker: one job at a time (erable is 2 GB / no GPU; two demucs runs would OOM). The atomic claim is still race-safe for N workers. - Identity = gateway tenant; a job is visible only to its owner (cross-tenant poll/cancel → 404, not 403, so existence doesn't leak). Pieces: - db.py: jobs table (status/priority/progress/result/attempts/webhook) + a claim-ordered index. - jobs.py: enqueue (incl. born-done idempotent fast path when the cache already has the result), atomic claim (candidate → guarded UPDATE WHERE status= 'pending'; rowcount-0 retry; WAL serializes writers so no double-claim), progress/finish/fail-with-requeue-under-cap, crash recovery (running→pending on boot), cooperative cancel + is_cancelled checkpoints. - worker.py: serialized loop — recover → claim → dispatch by type → finish/fail → optional webhook; SIGTERM-graceful; imports engine handler modules (none yet, idles politely); engines register via @jobs.handler. - deploy/douanier-worker.service: systemd --user unit (linger) — the durable run path (harness/nohup jobs die on teardown). Prod container-vs-host wiring is the paved-road call (SRE/#34). VALIDATION: tests/test_jobs.py — submit→claim→done; born-done fast path; finish/requeue-to-cap; crash recovery; NO double-claim across 6 threads × 25 jobs (each claimed exactly once); worker dispatch to done; missing-handler error; exception→requeue; cancel mid-flight; HTTP poll + ownership 404 + idempotent cancel. 11 new tests, full suite 67→78 green. OpenAPI snapshot refreshed (15 ops) + @nech/api regenerated (getJob/cancelJob); noImplicitAny relaxed for the 100%-generated client (typescript-fetch's camel/snake guard trips TS7053).PLN (Algolia) authored17d3f468
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| nech-api | Loading commit data... | |
| README.md | Loading commit data... | |
| nech.ts | Loading commit data... | |
| onboarding.html | Loading commit data... | |
| openapi.json | Loading commit data... | |
| openapitools.json | Loading commit data... | |
| refresh_openapi.py | Loading commit data... |