Commit 722daded by PLN (Algolia)

feat(douanier): CPU deploy image + conform to the nech.pl platform contract (#37)

The SRE edge is live (api.nech.pl returns 503 warming_up); the only thing
blocking a green healthz was a CPU-deployable image. This ships it, and folds
in the SRE's platform reframe that landed in the same letter.

Baseline image
- Dockerfile: python:3.12-slim + ffmpeg/libsndfile, the LIGHT torch-free stack
  only (requirements-deploy.txt: fastapi/uvicorn/librosa/numpy/soundfile). Builds
  to ~1 GB, runs well under the 2 GB-RAM erable budget. Default engine = light.
- Validated in-container end to end: healthz green, emotion miss->hit cache,
  401/403/200 auth gating, /metrics, /separate 503.

Platform contract (SRE update: api.nech.pl is a multi-API gateway; we're the
`audio` sub-API)
- Public path is /audio/v1/...; the gateway strips the prefix and proxies to us
  at root. Routes moved off the /v1 router to root; root_path=/audio/v1 so
  OpenAPI/docs advertise the real public paths (verified servers=[{/audio/v1}]).
- Central auth: dropped our own bearer verification in the request path. We now
  trust the gateway-injected X-Tenant / X-Scopes (loopback-only bind = only the
  gateway can reach us). Local-dev keeps a DOUANIER_DEV_TOKEN bearer fallback.
  Supersedes #23/#24 (the SQLite token store + CLI remain for dev only).
- /metrics: Prometheus text (douanier_up, requests_total{path,status},
  cache_rows/hits{kind}) for the erable scraper.
- /separate: deliberate 503 compute_unavailable (retriable) — no GPU path on
  erable; route exists so hexa can code against it now. GPU backend is env-selected later.

Docs + tests
- DEPLOY.md rewritten as the erable container contract (build, /data volume,
  env-file, loopback publish, cache cap, central-auth onboarding). README reframed
  to the platform shape. 32/32 tests pass (smoke retargeted to root paths +
  header auth; added /separate, /metrics, openapi-root-path coverage).
parent 09dc0bbe
# keep the build context (and image) lean — local state never belongs in the image
_cache/
*.db
*.db-wal
*.db-shm
__pycache__/
*.pyc
.pytest_cache/
.venv/
tests/
# docs are not needed at runtime
SRE.md
README.md
DEPLOY.md
# Deploying Douanier (the `audio` sub-API) on erable
The contract between **us** (the audio API) and **SRE** (who owns the platform
gateway + the seam to the internet).
`api.nech.pl` is a **multi-API platform**. Douanier is the internal codename for
its **`audio`** sub-API, served publicly at **`https://api.nech.pl/audio/v1/...`**.
The edge is already live: `curl https://api.nech.pl/audio/v1/healthz` returns
`503 {"status":"warming_up"}` until this container is running. The moment it's
up, that 503 becomes the real green `/healthz`.
This image is **CPU-only, torch-free** (the light V/A engine — see SRE.md). It is
~1 GB on disk and runs in well under 2 GB RAM.
## Two platform rules (SRE update 2026-06-25)
1. **Public path `/audio/v1/...`, app serves at root.** The gateway strips the
prefix (`location /audio/v1/ { proxy_pass http://127.0.0.1:9780/; }`), so the
container receives `/healthz`, `/analyze/emotion`, … at root. We set
`root_path=/audio/v1` (env `DOUANIER_ROOT_PATH`) so `/openapi.json` + docs
advertise the correct public paths.
2. **Central auth — no token store of ours.** The platform's `_platform` service
authenticates the bearer at the edge (`nginx auth_request`) and injects
**`X-Tenant`** + **`X-Scopes`**. We bind **loopback-only**, so only the gateway
can reach us, and we **trust those headers**. (Supersedes our old SQLite auth.)
Metering/quota is the platform's job too.
## 1. Build
```bash
docker build -t douanier:latest armada/api # ~1 GB image, light engine only
```
(Or build on the dev box and `docker save douanier:latest | ssh erable docker load`.)
## 2. State volume
One writable dir at `/data` holds the content-addressed **cache** (transient,
evictable, rebuildable — losing it is just a cache miss). With central auth there
is **no tokens DB to back up** anymore; `/data` is pure throwaway cache.
```bash
mkdir -p /srv/douanier/data # SRE-owned; cap at 2 GB (see §5)
```
## 3. Env file
`~/.config/douanier/douanier.env` (SRE-owned, NOT in git):
```env
# image default DOUANIER_EMOTION_ENGINE=light is correct for erable — leave unset
# image default DOUANIER_ROOT_PATH=/audio/v1 matches the gateway — leave unset
DOUANIER_MAX_UPLOAD_MB=40
# DOUANIER_TENANT_HEADER / DOUANIER_SCOPES_HEADER default to X-Tenant / X-Scopes;
# override only if the platform injects different header names.
# Do NOT set DOUANIER_DEV_TOKEN in prod (it's a local-dev-only bearer fallback).
```
## 4. Run
```bash
docker run -d --name douanier --restart unless-stopped \
--env-file ~/.config/douanier/douanier.env \
-v /srv/douanier/data:/data \
-p 127.0.0.1:9780:9780 \
douanier:latest
```
`-p 127.0.0.1:9780:9780` is the whole security model: loopback-only means the
gateway is the *only* way in, so the injected `X-Tenant`/`X-Scopes` headers can be
trusted. SRE wraps this in a keep-alive systemd unit.
Verify on the box, then from the internet:
```bash
curl -s http://127.0.0.1:9780/healthz # green, "selected":"light" (internal root path)
curl -s https://api.nech.pl/audio/v1/healthz # same JSON → edge green end-to-end
curl -s https://api.nech.pl/audio/v1/openapi.json # advertises /audio/v1/... public paths
```
## 5. Cache cap (transient storage)
The cache is rebuildable; keep it bounded — either cron the built-in GC or cap
the host dir:
```bash
docker exec douanier python douanier.py cache gc --max-mb 2048
```
## 6. Observability
`GET /metrics` (internal) / `https://api.nech.pl/audio/v1/metrics` exposes
Prometheus text — `douanier_up`, `douanier_requests_total{path,status}`,
`douanier_cache_rows{kind}`, `douanier_cache_hits_total{kind}`. Unauthenticated
by design (counts only, no secrets); reachable only via loopback/gateway. Point
the erable scraper at it. App logs → journald via the container.
## 7. Onboarding hexa
Token minting + scope grants are now the **platform's** job (central auth), not
ours — ask SRE to provision the `hexa` tenant with the `emotion` scope (and `*`
when more engines land). Hexa then calls
`https://api.nech.pl/audio/v1/analyze/emotion` with its platform bearer; the
gateway injects `X-Tenant: hexa` / `X-Scopes: emotion` for us.
## 8. What's NOT in this image (by design)
- **CLAP / torch emotion** — too heavy for erable. Behind a future `precise=true`
tier on a GPU runner; same response shape, transparent upgrade.
- **Separation (demucs/roformer)** — no CPU path. `/audio/v1/separate` returns
`503 compute_unavailable` (retriable) until a GPU-runner backend is wired
(env-selected). The route exists so hexa can code against it today.
- **Corpus emotion search** — precomputed offline on the GPU box, shipped as a
compact index; not part of the live container.
# Douanier — the CPU-deployable Audio-Intelligence API image (erable host).
#
# SRE reply (SRE.md): erable has NO GPU, ~2 GB RAM. The CLAP/torch stack does
# NOT fit, so this image ships ONLY the light V/A engine (librosa, no torch).
# One container, listens on :9780, SRE publishes it to 127.0.0.1:9780 behind
# the api.nech.pl nginx vhost + cert that's ALREADY live (returns 503 warming_up
# until this lands). The moment it's up, /v1/healthz goes green end-to-end.
#
# build: docker build -t douanier:latest armada/api
# run: see DEPLOY.md (env-file + /data volume + 127.0.0.1:9780 publish)
FROM python:3.12-slim AS base
# ffmpeg + libsndfile: librosa/soundfile decode uploaded mp3/m4a/wav/flac clips.
# Installed in one layer, apt lists removed, to keep the image lean.
RUN apt-get update && apt-get install -y --no-install-recommends \
ffmpeg libsndfile1 \
&& rm -rf /var/lib/apt/lists/*
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PIP_NO_CACHE_DIR=1 \
# the deployable default: the torch-free engine. Override only on a GPU box.
DOUANIER_EMOTION_ENGINE=light \
# cache + DB land on a mounted volume (transient cache; DB is the backup target)
DOUANIER_CACHE=/data/cache \
DOUANIER_DB=/data/douanier.db
WORKDIR /app
# deps first (cached layer) — only the light stack, see requirements-deploy.txt
COPY requirements-deploy.txt .
RUN pip install -r requirements-deploy.txt
# then the app source (flat-module layout: run from /app, import config/engines.*)
COPY . .
# /data is the only writable state (cache + tokens DB). SRE mounts a 2 GB-capped dir here.
VOLUME ["/data"]
EXPOSE 9780
# single uvicorn worker — heavy work is serialized anyway; SRE owns the keep-alive
# systemd wrapper + restart policy. Bind all interfaces; the publish is localhost-only.
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "9780"]
# Douanier — ParVagues Audio-Intelligence API 🛂 # Douanier — the `audio` sub-API of the nech.pl platform 🛂
The customs gate over the fleet's audio-intelligence engines: one self-hosted, The customs gate over the fleet's audio-intelligence engines: a self-hosted HTTP
token-authed, async HTTP API that turns a URL or a clip into **stems, loops, API that turns a URL or a clip into **stems, loops, emotion reads, sample
emotion reads, sample analysis, and features**. Named after Le Douanier Rousseau. analysis, and features**. Named after Le Douanier Rousseau. `api.nech.pl` is a
multi-API platform; Douanier is the internal codename for its **`audio`** sub-API,
served publicly at **`https://api.nech.pl/audio/v1/...`**.
Productizes the [Foundry](../../tools/foundry/) + the tide-table "ears" Productizes the [Foundry](../../tools/foundry/) + the tide-table "ears"
(`emotion_ontology`, `sample_semantics`, the feature stack) for **hexa** (Shipow) (`emotion_ontology`, `sample_semantics`, the feature stack) for **hexa** (Shipow)
and future seats. Full design + decisions: memory `project_douanier_api`, EPIC and future seats. Full design + decisions: memory `project_douanier_api`, EPIC
task **#21**, and [`SRE.md`](./SRE.md) for the hosting handoff. task **#21**, [`SRE.md`](./SRE.md) (hosting handoff) and [`DEPLOY.md`](./DEPLOY.md)
(the erable container contract).
## Layout ## Layout
``` ```
armada/api/ armada/api/
app.py FastAPI app — /v1 router, /v1/healthz, /v1/analyze/emotion app.py FastAPI app — routes at root (gateway owns /audio/v1), + /metrics
config.py paths · dev token · limits (env-overridable) config.py paths · root_path · gateway header names · limits (env-overridable)
engines/ thin adapters over the fleet libs (lazy heavy imports) engines/
ears.py emotion read = sample_semantics embed → emotion_ontology.score ears.py emotion read (CLAP) = sample_semantics embed → emotion_ontology.score
tests/ smoke tests (no CLAP/torch needed — engine call is mocked) ears_light.py torch-free V/A baseline (librosa) — the erable engine
requirements.txt API-layer deps only; ML stack inherited from system venv cache.py content-addressed result cache (the margin lever)
Dockerfile the CPU-only, torch-free deploy image (~1 GB)
tests/ smoke + engine tests (no CLAP/torch needed — heavy call mocked)
``` ```
## Two platform rules (SRE update 2026-06-25)
- **Public path `/audio/v1/...`; we serve at root.** The gateway strips the prefix
and proxies to us at `/`. We set `root_path=/audio/v1` so docs/OpenAPI advertise
the public paths.
- **Central auth — no token store of ours.** The platform authenticates the bearer
at the edge and injects `X-Tenant` / `X-Scopes`. We bind loopback-only and trust
them. (This supersedes the old SQLite bearer auth; metering is the platform's job.)
## Run (dev) ## Run (dev)
```bash ```bash
...@@ -28,49 +42,48 @@ python3 -m venv --system-site-packages ~/.virtualenvs/douanier # once ...@@ -28,49 +42,48 @@ python3 -m venv --system-site-packages ~/.virtualenvs/douanier # once
~/.virtualenvs/douanier/bin/pip install -r requirements.txt # once ~/.virtualenvs/douanier/bin/pip install -r requirements.txt # once
cd armada/api cd armada/api
DOUANIER_DEV_TOKEN=$(openssl rand -hex 16) \ # DEV bearer fallback (no gateway in front): any request with this bearer = wildcard
DOUANIER_DEV_TOKEN=devsecret DOUANIER_ROOT_PATH="" \
~/.virtualenvs/douanier/bin/uvicorn app:app --port 8780 ~/.virtualenvs/douanier/bin/uvicorn app:app --port 8780
``` ```
- `GET /v1/healthz` — status + engine availability Endpoints (internal root paths shown; public = `/audio/v1` + path):
- `GET /v1/me` — echo the calling token's account + scopes (onboarding check)
- `POST /v1/analyze/emotion``multipart` audio file → `{valence, arousal, top…}`
(Bearer token with the `emotion` or `*` scope)
- `GET /v1/docs` — Swagger UI
### Emotion engine selection
`DOUANIER_EMOTION_ENGINE` picks the V/A backend (same `/v1` response shape either way):
- `clap` (default, dev) — CLAP/torch, rich, ~4 GB, needs the GPU/dev box. - `GET /healthz` — status + engine availability (unauthenticated)
- `light` (prod / erable) — librosa-heuristic V/A, **no torch**, fits a ~2 GB-RAM - `GET /metrics` — Prometheus text for the erable scraper (unauthenticated)
container. Arousal from energy/tempo/brightness, valence from major/minor mode; - `GET /me` — echo the caller's gateway identity (any authenticated tenant)
mapped onto the same emotion anchors. A baseline (Essentia/CLAP-grade tier lands - `POST /analyze/emotion``multipart` audio → `{valence, arousal, top…}` (scope `emotion`)
later behind the same shape — see task #37). - `POST /separate` — stem separation; **503 `compute_unavailable`** until a GPU runner (scope `separate`)
- `GET /docs` — Swagger UI
```bash ```bash
DOUANIER_EMOTION_ENGINE=light ~/.virtualenvs/douanier/bin/uvicorn app:app --port 8780 # as the gateway calls us (injected identity):
curl -s -H "X-Tenant: hexa" -H "X-Scopes: emotion" \
-F file=@clip.wav http://127.0.0.1:8780/analyze/emotion | jq
# or, in pure local dev, the bearer fallback:
curl -s -H "Authorization: Bearer devsecret" \
-F file=@clip.wav http://127.0.0.1:8780/analyze/emotion | jq
``` ```
### Tokens (`douanier` CLI) ### Emotion engine selection
Real tokens live in SQLite (only their sha256 hash is stored). The dev one-liner `DOUANIER_EMOTION_ENGINE` picks the V/A backend (identical response shape):
above uses `DOUANIER_DEV_TOKEN` as a local-only wildcard bootstrap; for anything
real, mint:
```bash - `light` (default in the image / erable) — librosa-heuristic V/A, **no torch**,
~/.virtualenvs/douanier/bin/python douanier.py token mint --account hexa --scopes '*' --ttl 31 fits a ~2 GB-RAM container. Arousal from energy/tempo/brightness, valence from
~/.virtualenvs/douanier/bin/python douanier.py token list major/minor mode; mapped onto the emotion anchors. A baseline — a precise tier
~/.virtualenvs/douanier/bin/python douanier.py token revoke <id> (Essentia, or CLAP on a GPU runner) lands later behind the same shape.
``` - `clap` (dev box) — CLAP/torch, rich, ~4 GB, needs a GPU.
## Deploy
Scopes are space-separated engine names (`emotion features samples separate loops`) The torch-free image and the SRE handoff (build, volume, env-file, loopback
or `*`. `--ttl never` for no expiry; `--metered` to enforce quota (default publish, cache cap, onboarding) are in [`DEPLOY.md`](./DEPLOY.md):
unlimited; quota enforcement lands in #24).
```bash ```bash
curl -s -H "Authorization: Bearer $DOUANIER_DEV_TOKEN" \ docker build -t douanier:latest . # ~1 GB, light engine only
-F file=@clip.wav http://127.0.0.1:8780/v1/analyze/emotion | jq docker run -d -p 127.0.0.1:9780:9780 -v /srv/douanier/data:/data douanier:latest
curl -s https://api.nech.pl/audio/v1/healthz # green once SRE wires the keep-alive
``` ```
## Test ## Test
...@@ -81,16 +94,14 @@ cd armada/api && ~/.virtualenvs/douanier/bin/python -m pytest tests/ -q ...@@ -81,16 +94,14 @@ cd armada/api && ~/.virtualenvs/douanier/bin/python -m pytest tests/ -q
## Status ## Status
Done: scaffold + emotion endpoint (#22/#36), **SQLite bearer-token auth + scopes Done: scaffold + emotion endpoint (#22/#36), content-addressed cache (#26) — a
+ `douanier` CLI (#23)**, **content-addressed cache (#26)** — a repeat clip repeat clip returns in ~2 ms (measured 5663× on a real run; `X-Douanier-Cache:
returns in ~2ms instead of paying CLAP again (measured 5663× on a real run; hit`), **torch-free light V/A engine + CPU Docker image + platform contract
`X-Douanier-Cache: hit`). Queued, building *under* this without changing the (`/audio/v1`, central-auth headers, `/metrics`, `/separate` 503) (#37)**. The
public shape: per-token quota (#24), async jobs + worker (#25), signed artifact image is ~1 GB and the SRE edge is live and waiting.
URLs (#27), `/features`+`/samples` hardened (#28),
`/sources``/separate``/loops` heavy chain (#29–#31), OpenAPI client (#32), Superseded by the platform: our own SQLite bearer auth (#23) and per-token quota
ops/runbook (#34). (#24) — central auth/metering owns these now (the `auth.py` token store + CLI
remain for local dev only). Queued *under* the public shape without changing it:
> **Why a venv with `--system-site-packages`?** The ears/Foundry engines need the async jobs + worker (#25), signed artifact URLs (#27), `/features`+`/samples`
> multi-GB ML stack (torch/CLAP/librosa/demucs) already installed system-wide on (#28), `/sources``/separate``/loops` heavy chain (#29–#31), OpenAPI client (#32).
> the box; the venv reuses it and only adds FastAPI/uvicorn. Arch's PEP-668 lock
> is why we don't `pip install` into system Python.
...@@ -149,12 +149,54 @@ is the only thing that wants a backup** (it's tiny — nightly copy is plenty). ...@@ -149,12 +149,54 @@ is the only thing that wants a backup** (it's tiny — nightly copy is plenty).
9. **WAF/rate-limit** — coarse nginx `limit_req` at the edge on top of your per-token bucket. No Cloudflare (direct A record). 9. **WAF/rate-limit** — coarse nginx `limit_req` at the edge on top of your per-token bucket. No Cloudflare (direct A record).
10. **CORS** — enforce in-app; send hexa's Vercel origins when known. 10. **CORS** — enforce in-app; send hexa's Vercel origins when known.
### What SRE will build (independent of your engine choice) ### What SRE built (2026-06-25) — the edge is LIVE
- erable `api.nech.pl` nginx vhost → `127.0.0.1:9780` + certbot cert (the public edge; 502s until your container runs — expected). -`api.nech.pl` nginx vhost + valid Let's Encrypt cert (CN=api.nech.pl, auto-renew). HTTPS works, no cert warning.
- the Docker keep-alive wrapper (systemd unit) + the 2 GB cache dir + `--env-file`. - ✅ It proxies to `127.0.0.1:9780` and returns a clean `503 {"status":"warming_up"}` until your container is there. Try it: `curl https://api.nech.pl/v1/healthz`.
- vhost source is version-controlled in the SRE repo at `~/Work/SRE/erable/api.nech.pl.conf`.
- TODO (mine, when your image lands): Docker keep-alive systemd unit + 2 GB cache dir + `--env-file`.
**Blocking you on:** a CPU-deployable image (light engine, no CLAP/torch bloat) **The ONLY thing between us and a green healthz is you:** ship a CPU-deployable
that fits ~3 GB and runs in ~2 GB RAM. Ship that and `api.nech.pl/v1/healthz` image (light engine — Essentia/librosa, no CLAP/torch bloat) that runs in ~2 GB
goes green end-to-end. RAM and listens on a port I publish to `127.0.0.1:9780`. The moment it's up, the
503 turns into your real `/v1/healthz`. Disk is no longer a constraint (~15 GB free).
*SRE (the one who owns the seam to the internet)* 🛂 *SRE (the one who owns the seam to the internet)* 🛂
---
## ⚠️ Update (2026-06-25, later): `api.nech.pl` is a multi-API PLATFORM, and auth goes CENTRAL
PLN reframed the scope. `api.nech.pl` is **"Nech.PL APIs"**, a gateway hosting
many sub-APIs; **Douanier is the internal codename for the `audio` sub-API**.
Two things change for you:
1. **Public path = `/audio/v1/...`** (domain-first, independent per-API
versioning), **not** `/v1/...`. Run with **`root_path=/audio/v1`** so your
OpenAPI/docs advertise the right public paths; the gateway strips the prefix
(`location /audio/v1/ { proxy_pass http://127.0.0.1:9780/; }`). This reverses
my earlier Q1 answer.
2. **DROP your own auth (task #23).** The platform does **central** auth +
metering: one `_platform` service owns tokens/tenants/scopes/quota/usage, and
nginx `auth_request` validates the bearer once at the edge and injects
**`X-Tenant`** / **`X-Scopes`** headers. **Don't build a SQLite token store**
just (a) bind loopback-only so you're only reachable via the gateway, and
(b) trust those injected headers. This centralizes billing for "selling calls."
Keep your per-token *quota* logic out too — the platform meters.
Everything else (containerize, light CPU engine, `/healthz` + `/metrics` +
`/openapi.json`, 2 GB transient cache, X-Accel-Redirect for artifact serving)
stands. **The platform contract is now written** — read it before reshaping the
service:
- `~/Work/SRE/nechapi/RECIPE.md` — the contract you conform to (root_path,
/healthz + /metrics + /openapi.json, trust `X-Tenant`/`X-Scopes`, loopback
bind, X-Accel-Redirect for artifacts, the compliance checklist).
- `~/Work/SRE/nechapi/PLATFORM.md` — the full platform design + decisions log.
- `~/Work/SRE/nechapi/nech-apis.json` — the registry; your `audio`/v1 entry is
already in it (port 9780).
Net for you: `root_path=/audio/v1`, drop #23 (auth) and per-token quota, expose
the three standard endpoints, keep the engine CPU-light. Ping SRE when the
container's ready and I'll wire the gateway + auth_request + your docs tab.
*SRE* 🛂
...@@ -8,10 +8,11 @@ SQLite auth/quota/jobs/cache machinery (#23–#27) that hardens it later. ...@@ -8,10 +8,11 @@ SQLite auth/quota/jobs/cache machinery (#23–#27) that hardens it later.
run: cd armada/api && ~/.virtualenvs/douanier/bin/uvicorn app:app --port 8780 run: cd armada/api && ~/.virtualenvs/douanier/bin/uvicorn app:app --port 8780
""" """
import secrets
import tempfile import tempfile
from pathlib import Path from pathlib import Path
from fastapi import FastAPI, APIRouter, UploadFile, File, Header, HTTPException, Depends, Response from fastapi import FastAPI, APIRouter, UploadFile, File, Request, HTTPException, Depends, Response
from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.cors import CORSMiddleware
import auth import auth
...@@ -20,12 +21,16 @@ import config ...@@ -20,12 +21,16 @@ import config
from engines import ears from engines import ears
app = FastAPI( app = FastAPI(
title="Douanier — ParVagues Audio-Intelligence API", title="Douanier — the `audio` sub-API of the nech.pl platform",
version="0.1.0", version="0.1.0",
description="Customs gate over the fleet's audio-intelligence engines. " description="Customs gate over the fleet's audio-intelligence engines. Served "
"Versioned at /v1; SRE maps the public api.nech.pl path in. See SRE.md.", "publicly at /audio/v1/...; the platform gateway does central auth + "
docs_url=f"/{config.VERSION}/docs", "metering and injects X-Tenant/X-Scopes. See SRE.md + DEPLOY.md.",
openapi_url=f"/{config.VERSION}/openapi.json", # internal paths (gateway strips /audio/v1 and proxies here at root); root_path
# makes /openapi.json + docs advertise the PUBLIC /audio/v1/... paths to clients.
root_path=config.ROOT_PATH,
docs_url="/docs",
openapi_url="/openapi.json",
) )
# hexa lives in Shipow's Vercel world → allow cross-origin calls. Tighten to his # hexa lives in Shipow's Vercel world → allow cross-origin calls. Tighten to his
...@@ -34,24 +39,51 @@ app.add_middleware( ...@@ -34,24 +39,51 @@ app.add_middleware(
CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"], CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"],
) )
v1 = APIRouter(prefix=f"/{config.VERSION}") v1 = APIRouter() # routes mounted at root; the gateway owns the /audio/v1 prefix.
# ── observability (#37) — a counter the erable Prometheus scrapes ───────────────
_REQ_COUNTS: dict[tuple[str, int], int] = {}
@app.middleware("http")
async def _meter(request: Request, call_next):
resp = await call_next(request)
key = (request.url.path, resp.status_code)
_REQ_COUNTS[key] = _REQ_COUNTS.get(key, 0) + 1
return resp
# ── auth — TRUST the platform gateway's injected identity (supersedes #23/#24) ──
def _principal(request: Request) -> auth.Principal | None:
"""Resolve the caller from the gateway-injected headers. The platform's
`_platform` service authenticates the bearer at the edge (nginx auth_request)
and forwards X-Tenant / X-Scopes; we're loopback-only, so only the gateway
can reach us and those headers are trustworthy. No token store of our own.
Local-dev fallback (no gateway in front): a bearer matching DOUANIER_DEV_TOKEN
grants a wildcard, so `uvicorn` + curl works without the platform."""
tenant = request.headers.get(config.TENANT_HEADER)
if tenant:
scopes = request.headers.get(config.SCOPES_HEADER, "*")
return auth.Principal(tenant, frozenset(scopes.split()), None, None)
authz = request.headers.get("authorization", "")
kind, _, token = authz.partition(" ")
if (kind.lower() == "bearer" and token and config.DEV_TOKEN
and secrets.compare_digest(token, config.DEV_TOKEN)):
return auth.Principal("dev", frozenset("*"), None, None)
return None
# ── auth (#23) — SQLite bearer tokens + per-engine scopes ───────────────────────
def require_scope(scope: str | None): def require_scope(scope: str | None):
"""Dependency factory: resolve the bearer token to a Principal and (if `scope` """Dependency factory: resolve the gateway identity and (if `scope` is given)
is given) assert it carries that scope or the '*' wildcard. scope=None means assert it carries that scope or the '*' wildcard. scope=None means "any
"any valid token". 401 unknown/expired/revoked, 403 scoped out. The dev authenticated caller". 401 if no/unknown identity, 403 if scoped out."""
bootstrap (DOUANIER_DEV_TOKEN) is a local-only wildcard.""" def dep(request: Request) -> auth.Principal:
def dep(authorization: str = Header(default="")) -> auth.Principal: principal = _principal(request)
kind, _, token = authorization.partition(" ")
if kind.lower() != "bearer" or not token:
raise HTTPException(401, "missing bearer token")
principal = auth.verify(token)
if principal is None: if principal is None:
raise HTTPException(401, "invalid, expired, or revoked token") raise HTTPException(401, "no authenticated identity (gateway headers absent)")
if scope is not None and not principal.has_scope(scope): if scope is not None and not principal.has_scope(scope):
raise HTTPException(403, f"token lacks scope '{scope}'") raise HTTPException(403, f"caller lacks scope '{scope}'")
return principal return principal
return dep return dep
...@@ -62,17 +94,50 @@ def healthz(): ...@@ -62,17 +94,50 @@ def healthz():
ok, hint = ears.available() ok, hint = ears.available()
return { return {
"service": config.SERVICE, "service": config.SERVICE,
"api": "audio",
"version": config.VERSION, "version": config.VERSION,
"status": "ok", "status": "ok",
"dev_token_set": bool(config.DEV_TOKEN), "engines": {
"engines": {"emotion": {"selected": config.EMOTION_ENGINE, "emotion": {"selected": config.EMOTION_ENGINE, "available": ok, "hint": hint},
"available": ok, "hint": hint}}, "separate": {"available": False, "hint": "no CPU path on erable — GPU runner pending"},
},
} }
@app.get("/metrics", include_in_schema=False)
def metrics():
"""Prometheus text exposition (the erable scraper hits this). Counts only —
no secrets. Unauthenticated by design, reachable only on loopback/gateway."""
lines = [
"# HELP douanier_up 1 if the audio API is responding",
"# TYPE douanier_up gauge",
"douanier_up 1",
"# HELP douanier_requests_total HTTP requests by path and status",
"# TYPE douanier_requests_total counter",
]
for (path, status), n in sorted(_REQ_COUNTS.items()):
safe = path.replace("\\", "\\\\").replace('"', '\\"')
lines.append(f'douanier_requests_total{{path="{safe}",status="{status}"}} {n}')
try:
st = cache.stats() # one row per kind: {kind, n, bytes, hits}
lines += [
"# HELP douanier_cache_rows cached results by kind",
"# TYPE douanier_cache_rows gauge",
"# HELP douanier_cache_hits_total cache hits served by kind",
"# TYPE douanier_cache_hits_total counter",
]
for r in st:
k = str(r.get("kind", "")).replace('"', '\\"')
lines.append(f'douanier_cache_rows{{kind="{k}"}} {r.get("n", 0)}')
lines.append(f'douanier_cache_hits_total{{kind="{k}"}} {r.get("hits", 0)}')
except Exception:
pass
return Response("\n".join(lines) + "\n", media_type="text/plain; version=0.0.4")
@v1.get("/me") @v1.get("/me")
def whoami(principal: auth.Principal = Depends(require_scope("*"))): def whoami(principal: auth.Principal = Depends(require_scope(None))):
"""Echo the calling token's identity — handy for onboarding smoke tests.""" """Echo the caller's gateway-resolved identity — handy for onboarding smoke tests."""
return {"account": principal.account, "scopes": sorted(principal.scopes), return {"account": principal.account, "scopes": sorted(principal.scopes),
"expires_at": principal.expires_at} "expires_at": principal.expires_at}
...@@ -110,4 +175,18 @@ async def analyze_emotion(response: Response, file: UploadFile = File(...), ...@@ -110,4 +175,18 @@ async def analyze_emotion(response: Response, file: UploadFile = File(...),
return {"filename": file.filename, "content_id": cid, "emotion": read} return {"filename": file.filename, "content_id": cid, "emotion": read}
# ── #37 separation seam — no CPU path on erable; env-selected GPU backend later ──
@v1.post("/separate")
async def separate(_p: auth.Principal = Depends(require_scope("separate"))):
"""Stem separation (demucs/roformer). erable has no GPU and no CPU-viable path,
so this is a deliberate 503 until a GPU-runner backend is wired (env-selected,
same response shape). The route exists now so hexa can code against it and get
a clean, documented "compute pending" rather than a 404."""
raise HTTPException(503, detail={
"error": "compute_unavailable",
"detail": "stem separation needs a GPU runner; not available on this host yet.",
"retriable": True,
})
app.include_router(v1) app.include_router(v1)
...@@ -7,8 +7,22 @@ import os ...@@ -7,8 +7,22 @@ import os
from pathlib import Path from pathlib import Path
# ── identity ────────────────────────────────────────────────────────────────── # ── identity ──────────────────────────────────────────────────────────────────
SERVICE = "douanier" SERVICE = "douanier" # internal codename; the public sub-API is "audio".
VERSION = "v1" # forced route prefix; SRE maps the public path in. VERSION = "v1" # the audio sub-API's own version (independent).
# api.nech.pl is a multi-API PLATFORM (SRE update 2026-06-25): Douanier is the
# `audio` sub-API, served publicly at /audio/v1/... The gateway strips that prefix
# and proxies to us at root (so internally routes are /healthz, /analyze/emotion…).
# root_path makes FastAPI advertise the public paths in /openapi.json + docs.
ROOT_PATH = os.environ.get("DOUANIER_ROOT_PATH", "/audio/v1")
# ── central auth (platform-owned) ────────────────────────────────────────────
# The platform does CENTRAL auth + metering: nginx `auth_request` validates the
# bearer at the edge and injects the caller identity as these headers. We bind
# loopback-only (reachable only via the gateway) and TRUST them — no token store
# of our own (supersedes #23/#24). See SRE.md.
TENANT_HEADER = os.environ.get("DOUANIER_TENANT_HEADER", "X-Tenant")
SCOPES_HEADER = os.environ.get("DOUANIER_SCOPES_HEADER", "X-Scopes")
# ── repo-relative paths ───────────────────────────────────────────────────────── # ── repo-relative paths ─────────────────────────────────────────────────────────
HERE = Path(__file__).resolve().parent # armada/api HERE = Path(__file__).resolve().parent # armada/api
......
# Douanier — DEPLOY (erable container) requirements: the LIGHT, torch-free stack.
#
# Unlike requirements.txt (which assumes a --system-site-packages venv that
# inherits the dev box's ML stack), this file is SELF-CONTAINED: the erable
# host has no GPU and no shared ML packages, so the Docker image installs
# everything it needs and NOTHING it doesn't. No torch, no CLAP, no demucs —
# the light V/A engine (engines/ears_light.py) runs on librosa + numpy alone.
# See SRE.md + DEPLOY.md.
#
# Build target: ~1.3 GB image, < 2 GB RAM at runtime, CPU-only.
fastapi>=0.115
uvicorn[standard]>=0.30
python-multipart>=0.0.9 # UploadFile / multipart form parsing
# light audio-intelligence (CPU, no torch)
numpy>=1.26,<2.3
librosa>=0.11 # spectral/chroma/tempo features (pulls scipy, numba, sklearn)
soundfile>=0.12 # content_id PCM decode + librosa wav/flac backend
...@@ -49,7 +49,7 @@ def test_arousal_orders_energetic_above_calm(): ...@@ -49,7 +49,7 @@ def test_arousal_orders_energetic_above_calm():
def test_endpoint_light_path_is_torch_free(monkeypatch): def test_endpoint_light_path_is_torch_free(monkeypatch):
# Prove /v1/analyze/emotion works end-to-end with NO CLAP/torch, the erable path. # Prove /analyze/emotion works end-to-end with NO CLAP/torch, the erable path.
monkeypatch.setattr(config, "EMOTION_ENGINE", "light") monkeypatch.setattr(config, "EMOTION_ENGINE", "light")
from fastapi.testclient import TestClient from fastapi.testclient import TestClient
import app as APP import app as APP
...@@ -58,9 +58,9 @@ def test_endpoint_light_path_is_torch_free(monkeypatch): ...@@ -58,9 +58,9 @@ def test_endpoint_light_path_is_torch_free(monkeypatch):
wav = _tone([330, 660]) wav = _tone([330, 660])
with open(wav, "rb") as fh: with open(wav, "rb") as fh:
data = fh.read() data = fh.read()
r = client.post("/v1/analyze/emotion", r = client.post("/analyze/emotion",
files={"file": ("t.wav", data, "audio/wav")}, files={"file": ("t.wav", data, "audio/wav")},
headers={"Authorization": "Bearer test-secret"}) headers={"X-Tenant": "hexa", "X-Scopes": "emotion"})
assert r.status_code == 200 assert r.status_code == 200
body = r.json() body = r.json()
assert body["emotion"]["engine"] == "light" assert body["emotion"]["engine"] == "light"
......
"""Walking-skeleton + API smoke tests — no CLAP/torch required. """Walking-skeleton + API smoke tests — no CLAP/torch required.
Cover the seams that must hold: the app boots, /v1/healthz answers, /me echoes Cover the seams that must hold under the PLATFORM contract (SRE update): routes
identity, and auth fails closed / opens with a valid token. The actual emotion are served at root (the gateway owns the /audio/v1 prefix), and auth is the
inference is exercised separately (slow, needs CLAP) — here we only assert the gateway-injected X-Tenant / X-Scopes identity (with a local-dev bearer fallback).
route is wired & guarded, by monkeypatching the engine call. Env/DB/path setup The emotion inference itself is monkeypatched — here we assert routing, auth
is in conftest.py. gating, the cache, and the deploy seams (/separate 503, /metrics). Env/DB/path
setup is in conftest.py.
cd armada/api && ~/.virtualenvs/douanier/bin/python -m pytest tests/ -q cd armada/api && ~/.virtualenvs/douanier/bin/python -m pytest tests/ -q
""" """
...@@ -14,41 +15,60 @@ import app as APP ...@@ -14,41 +15,60 @@ import app as APP
from engines import ears from engines import ears
client = TestClient(APP.app) client = TestClient(APP.app)
AUTH = {"Authorization": "Bearer test-secret"}
# the local-dev fallback: a bearer matching DOUANIER_DEV_TOKEN (conftest) → wildcard.
DEV = {"Authorization": "Bearer test-secret"}
# how the platform gateway actually calls us: injected identity headers.
GW = {"X-Tenant": "hexa", "X-Scopes": "emotion"}
def test_healthz_ok(): def test_healthz_ok():
r = client.get("/v1/healthz") r = client.get("/healthz")
assert r.status_code == 200 assert r.status_code == 200
body = r.json() body = r.json()
assert body["service"] == "douanier" and body["version"] == "v1" assert body["service"] == "douanier" and body["api"] == "audio"
assert "emotion" in body["engines"] assert "emotion" in body["engines"] and "separate" in body["engines"]
def test_me_echoes_identity(): def test_me_via_gateway_headers():
r = client.get("/v1/me", headers=AUTH) r = client.get("/me", headers={"X-Tenant": "hexa", "X-Scopes": "emotion separate"})
assert r.status_code == 200 assert r.status_code == 200
assert r.json()["account"] == "dev" body = r.json()
assert body["account"] == "hexa"
assert set(body["scopes"]) == {"emotion", "separate"}
def test_me_via_dev_bearer_fallback():
r = client.get("/me", headers=DEV)
assert r.status_code == 200 and r.json()["account"] == "dev"
def test_me_requires_token(): def test_me_requires_identity():
assert client.get("/v1/me").status_code == 401 assert client.get("/me").status_code == 401
def test_emotion_requires_token(): def test_emotion_requires_identity():
r = client.post("/v1/analyze/emotion", files={"file": ("c.wav", b"RIFF....", "audio/wav")}) r = client.post("/analyze/emotion", files={"file": ("c.wav", b"RIFF....", "audio/wav")})
assert r.status_code == 401 assert r.status_code == 401
def test_emotion_rejects_bad_token(): def test_emotion_rejects_bad_bearer():
r = client.post("/v1/analyze/emotion", r = client.post("/analyze/emotion",
files={"file": ("c.wav", b"RIFF....", "audio/wav")}, files={"file": ("c.wav", b"RIFF....", "audio/wav")},
headers={"Authorization": "Bearer nope"}) headers={"Authorization": "Bearer nope"})
assert r.status_code == 401 assert r.status_code == 401
def test_emotion_scope_enforced():
# an authenticated tenant WITHOUT the emotion scope is 403, not 401.
r = client.post("/analyze/emotion",
files={"file": ("c.wav", b"RIFF....", "audio/wav")},
headers={"X-Tenant": "hexa", "X-Scopes": "features"})
assert r.status_code == 403
def test_emotion_happy_path_and_cache(monkeypatch): def test_emotion_happy_path_and_cache(monkeypatch):
# Stub the heavy CLAP call — we're testing the route/contract + cache, not the model. # Stub the heavy engine call — we're testing the route/contract + cache, not the model.
fake = {"valence": 0.42, "arousal": -0.10, "top": [["warm", 0.5]], "confidence": 0.2} fake = {"valence": 0.42, "arousal": -0.10, "top": [["warm", 0.5]], "confidence": 0.2}
calls = {"n": 0} calls = {"n": 0}
...@@ -60,19 +80,38 @@ def test_emotion_happy_path_and_cache(monkeypatch): ...@@ -60,19 +80,38 @@ def test_emotion_happy_path_and_cache(monkeypatch):
payload = b"UNIQUE-cache-probe-bytes-9f3a" # undecodable → raw_ content id payload = b"UNIQUE-cache-probe-bytes-9f3a" # undecodable → raw_ content id
files = {"file": ("c.wav", payload, "audio/wav")} files = {"file": ("c.wav", payload, "audio/wav")}
r1 = client.post("/v1/analyze/emotion", files=files, headers=AUTH) r1 = client.post("/analyze/emotion", files=files, headers=GW)
assert r1.status_code == 200 assert r1.status_code == 200
assert r1.headers["X-Douanier-Cache"] == "miss" assert r1.headers["X-Douanier-Cache"] == "miss"
assert r1.json()["emotion"]["valence"] == 0.42 assert r1.json()["emotion"]["valence"] == 0.42
assert r1.json()["content_id"].startswith(("cid_", "raw_")) assert r1.json()["content_id"].startswith(("cid_", "raw_"))
r2 = client.post("/v1/analyze/emotion", files=files, headers=AUTH) r2 = client.post("/analyze/emotion", files=files, headers=GW)
assert r2.headers["X-Douanier-Cache"] == "hit" assert r2.headers["X-Douanier-Cache"] == "hit"
assert calls["n"] == 1 # engine ran ONCE; 2nd served from cache assert calls["n"] == 1 # engine ran ONCE; 2nd served from cache
def test_emotion_too_large(monkeypatch): def test_emotion_too_large(monkeypatch):
monkeypatch.setattr(APP.config, "MAX_UPLOAD_MB", 0) # everything is too large monkeypatch.setattr(APP.config, "MAX_UPLOAD_MB", 0) # everything is too large
r = client.post("/v1/analyze/emotion", r = client.post("/analyze/emotion",
files={"file": ("c.wav", b"x" * 1024, "audio/wav")}, headers=AUTH) files={"file": ("c.wav", b"x" * 1024, "audio/wav")}, headers=GW)
assert r.status_code == 413 assert r.status_code == 413
def test_separate_is_503_until_gpu():
r = client.post("/separate", headers={"X-Tenant": "hexa", "X-Scopes": "separate"})
assert r.status_code == 503
assert r.json()["detail"]["error"] == "compute_unavailable"
def test_metrics_exposes_prometheus_text():
r = client.get("/metrics")
assert r.status_code == 200
assert "douanier_up 1" in r.text
assert "douanier_requests_total" in r.text
def test_openapi_advertises_public_root_path():
# root_path=/audio/v1 → clients see the real public paths in the spec.
spec = client.get("/openapi.json").json()
assert spec.get("servers") == [{"url": "/audio/v1"}]
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment