Skip to content

T
Tidal
Commit 5ad4ed0e by PLN (Algolia)
Options

rename(audio-api): codename Douanier → Fourier 

The audio API was internally codenamed "Douanier" (Le Douanier Rousseau — a
customs-officer pun on edge auth). Renamed to "Fourier": the FFT is literally
the transform behind /spectrum and most of the feature stack, and central auth
is the platform’s job now, not ours — so the name should point at the signal
work, not the gate.

Internal-codename-only: the public contract (/audio/v1, gateway headers) is
untouched. Mechanical case-aware sweep across armada/api + the completed-archive,
plus hand-rewritten prose (the Rousseau attribution → Joseph Fourier; dropped the
customs-gate / 🛂 metaphors). Renames: douanier.py → fourier.py,
deploy/douanier-worker.service → deploy/fourier-worker.service. Env prefix
DOUANIER_* → FOURIER_* (config contract; safe — nech.pl is pre-users), dev venv
~/.virtualenvs/douanier → fourier (shebangs fixed). 78 tests green after.
parent 3ed54057
Showing with 137 additions and 135 deletions
armada/api/DEPLOY.md
# Deploying Douanier (the `audio` sub-API) on erable
# Deploying Fourier (the `audio` sub-API) on erable
The contract between **us** (the audio API) and **SRE** (who owns the platform
gateway + the seam to the internet).
`api.nech.pl` is a **multi-API platform**. Douanier is the internal codename for
`api.nech.pl` is a **multi-API platform**. Fourier is the internal codename for
its **`audio`** sub-API, served publicly at **`https://api.nech.pl/audio/v1/...`**.
The edge is already live: `curl https://api.nech.pl/audio/v1/healthz` returns
`503 {"status":"warming_up"}` until this container is running. The moment it's
......@@ -17,7 +17,7 @@ This image is **CPU-only, torch-free** (the light V/A engine — see SRE.md). It
1. **Public path `/audio/v1/...`, app serves at root.** The gateway strips the
prefix (`location /audio/v1/ { proxy_pass http://127.0.0.1:9780/; }`), so the
container receives `/healthz`, `/analyze/emotion`, … at root. We set
`root_path=/audio/v1` (env `DOUANIER_ROOT_PATH`) so `/openapi.json` + docs
`root_path=/audio/v1` (env `FOURIER_ROOT_PATH`) so `/openapi.json` + docs
advertise the correct public paths.
2. **Central auth — no token store of ours.** The platform's `_platform` service
authenticates the bearer at the edge (`nginx auth_request`) and injects
......@@ -28,10 +28,10 @@ This image is **CPU-only, torch-free** (the light V/A engine — see SRE.md). It
## 1. Build
```bash
docker build -t douanier:latest armada/api # ~1 GB image, light engine only
docker build -t fourier:latest armada/api # ~1 GB image, light engine only
```
(Or build on the dev box and `docker save douanier:latest | ssh erable docker load`.)
(Or build on the dev box and `docker save fourier:latest | ssh erable docker load`.)
## 2. State volume
......@@ -40,36 +40,36 @@ evictable, rebuildable — losing it is just a cache miss). With central auth th
is **no tokens DB to back up** anymore; `/data` is pure throwaway cache.
```bash
mkdir -p /srv/douanier/data # SRE-owned; cap at 2 GB (see §5)
mkdir -p /srv/fourier/data # SRE-owned; cap at 2 GB (see §5)
```
## 3. Env file
`~/.config/douanier/douanier.env` (SRE-owned, NOT in git):
`~/.config/fourier/fourier.env` (SRE-owned, NOT in git):
```env
# image default DOUANIER_EMOTION_ENGINE=light is correct for erable — leave unset
# image default DOUANIER_ROOT_PATH=/audio/v1 matches the gateway — leave unset
DOUANIER_MAX_UPLOAD_MB=40
# image default FOURIER_EMOTION_ENGINE=light is correct for erable — leave unset
# image default FOURIER_ROOT_PATH=/audio/v1 matches the gateway — leave unset
FOURIER_MAX_UPLOAD_MB=40
# Single-thread BLAS on erable's old kernel (see §4 gotcha #2):
OPENBLAS_NUM_THREADS=1
OMP_NUM_THREADS=1
NUMEXPR_NUM_THREADS=1
MKL_NUM_THREADS=1
# DOUANIER_TENANT_HEADER / DOUANIER_SCOPES_HEADER default to X-Tenant / X-Scopes;
# FOURIER_TENANT_HEADER / FOURIER_SCOPES_HEADER default to X-Tenant / X-Scopes;
# override only if the platform injects different header names.
# Do NOT set DOUANIER_DEV_TOKEN in prod (it's a local-dev-only bearer fallback).
# Do NOT set FOURIER_DEV_TOKEN in prod (it's a local-dev-only bearer fallback).
```
## 4. Run
```bash
docker run -d --name douanier --restart unless-stopped \
docker run -d --name fourier --restart unless-stopped \
--security-opt seccomp=unconfined \
--env-file ~/.config/douanier/douanier.env \
-v /home/pln/srv/douanier/data:/data \
--env-file ~/.config/fourier/fourier.env \
-v /home/pln/srv/fourier/data:/data \
-p 127.0.0.1:9780:9780 \
douanier:latest
fourier:latest
```
`-p 127.0.0.1:9780:9780` is the whole security model: loopback-only means the
......@@ -90,7 +90,7 @@ trusted. SRE wraps this in a keep-alive systemd unit.
`OMP_NUM_THREADS=1`, `NUMEXPR_NUM_THREADS=1`, `MKL_NUM_THREADS=1`). Belt-and-
braces with #1 on the old kernel, and right-sized for a shared 4-vCPU / ~2 GB
box doing CPU-light work — avoids OpenBLAS spawning a pool that fights for RAM.
3. **Data volume** is `/home/pln/srv/douanier/data` (no sudo for `/srv`); it's
3. **Data volume** is `/home/pln/srv/fourier/data` (no sudo for `/srv`); it's
pure transient cache, so the path is immaterial — just keep it capped (§5).
Verify on the box, then from the internet:
......@@ -107,14 +107,14 @@ The cache is rebuildable; keep it bounded — either cron the built-in GC or cap
the host dir:
```bash
docker exec douanier python douanier.py cache gc --max-mb 2048
docker exec fourier python fourier.py cache gc --max-mb 2048
```
## 6. Observability
`GET /metrics` (internal) / `https://api.nech.pl/audio/v1/metrics` exposes
Prometheus text — `douanier_up`, `douanier_requests_total{path,status}`,
`douanier_cache_rows{kind}`, `douanier_cache_hits_total{kind}`. Unauthenticated
Prometheus text — `fourier_up`, `fourier_requests_total{path,status}`,
`fourier_cache_rows{kind}`, `fourier_cache_hits_total{kind}`. Unauthenticated
by design (counts only, no secrets); reachable only via loopback/gateway. Point
the erable scraper at it. App logs → journald via the container.
......
armada/api/Dockerfile
# Douanier — the CPU-deployable Audio-Intelligence API image (erable host).
# Fourier — the CPU-deployable Audio-Intelligence API image (erable host).
#
# SRE reply (SRE.md): erable has NO GPU, ~2 GB RAM. The CLAP/torch stack does
# NOT fit, so this image ships ONLY the light V/A engine (librosa, no torch).
......@@ -6,7 +6,7 @@
# the api.nech.pl nginx vhost + cert that's ALREADY live (returns 503 warming_up
# until this lands). The moment it's up, /v1/healthz goes green end-to-end.
#
# build: docker build -t douanier:latest armada/api
# build: docker build -t fourier:latest armada/api
# run: see DEPLOY.md (env-file + /data volume + 127.0.0.1:9780 publish)
FROM python:3.12-slim AS base
......@@ -21,10 +21,10 @@ ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PIP_NO_CACHE_DIR=1 \
# the deployable default: the torch-free engine. Override only on a GPU box.
DOUANIER_EMOTION_ENGINE=light \
FOURIER_EMOTION_ENGINE=light \
# cache + DB land on a mounted volume (transient cache; DB is the backup target)
DOUANIER_CACHE=/data/cache \
DOUANIER_DB=/data/douanier.db
FOURIER_CACHE=/data/cache \
FOURIER_DB=/data/fourier.db
WORKDIR /app
......
armada/api/README.md
# Douanier — the `audio` sub-API of the nech.pl platform 🛂
# Fourier — the `audio` sub-API of the nech.pl platform 🎛️
The customs gate over the fleet's audio-intelligence engines: a self-hosted HTTP
The transform over the fleet's audio-intelligence engines: a self-hosted HTTP
API that turns a URL or a clip into **stems, loops, emotion reads, sample
analysis, and features**. Named after Le Douanier Rousseau. `api.nech.pl` is a
multi-API platform; Douanier is the internal codename for its **`audio`** sub-API,
served publicly at **`https://api.nech.pl/audio/v1/...`**.
analysis, and features**. Named after Joseph Fourier — the FFT behind `/spectrum`
and most of the feature stack. `api.nech.pl` is a multi-API platform; Fourier is
the internal codename for its **`audio`** sub-API, served publicly at
**`https://api.nech.pl/audio/v1/...`**.
Productizes the [Foundry](../../tools/foundry/) + the tide-table "ears"
(`emotion_ontology`, `sample_semantics`, the feature stack) for **hexa** (Shipow)
and future seats. Full design + decisions: memory `project_douanier_api`, EPIC
and future seats. Full design + decisions: memory `project_fourier_api`, EPIC
task **#21**, [`SRE.md`](./SRE.md) (hosting handoff) and [`DEPLOY.md`](./DEPLOY.md)
(the erable container contract).
......@@ -38,13 +39,13 @@ armada/api/
## Run (dev)
```bash
python3 -m venv --system-site-packages ~/.virtualenvs/douanier # once
~/.virtualenvs/douanier/bin/pip install -r requirements.txt # once
python3 -m venv --system-site-packages ~/.virtualenvs/fourier # once
~/.virtualenvs/fourier/bin/pip install -r requirements.txt # once
cd armada/api
# DEV bearer fallback (no gateway in front): any request with this bearer = wildcard
DOUANIER_DEV_TOKEN=devsecret DOUANIER_ROOT_PATH="" \
~/.virtualenvs/douanier/bin/uvicorn app:app --port 8780
FOURIER_DEV_TOKEN=devsecret FOURIER_ROOT_PATH="" \
~/.virtualenvs/fourier/bin/uvicorn app:app --port 8780
```
Endpoints (internal root paths shown; public = `/audio/v1` + path):
......@@ -79,7 +80,7 @@ curl -s -H "Authorization: Bearer devsecret" \
### Emotion engine selection
`DOUANIER_EMOTION_ENGINE` picks the V/A backend (identical response shape):
`FOURIER_EMOTION_ENGINE` picks the V/A backend (identical response shape):
- `light` (default in the image / erable) — librosa-heuristic V/A, **no torch**,
fits a ~2 GB-RAM container. Arousal from energy/tempo/brightness, valence from
......@@ -93,8 +94,8 @@ The torch-free image and the SRE handoff (build, volume, env-file, loopback
publish, cache cap, onboarding) are in [`DEPLOY.md`](./DEPLOY.md):
```bash
docker build -t douanier:latest . # ~1 GB, light engine only
docker run -d -p 127.0.0.1:9780:9780 -v /srv/douanier/data:/data douanier:latest
docker build -t fourier:latest . # ~1 GB, light engine only
docker run -d -p 127.0.0.1:9780:9780 -v /srv/fourier/data:/data fourier:latest
curl -s https://api.nech.pl/audio/v1/healthz # green once SRE wires the keep-alive
```
......@@ -109,7 +110,7 @@ attempt cap. Identity is the gateway tenant; a job is only visible to its owner.
```bash
# run the worker (dev); prod = the systemd --user unit in deploy/
cd armada/api && ~/.virtualenvs/douanier/bin/python -m worker
cd armada/api && ~/.virtualenvs/fourier/bin/python -m worker
```
Engines register a handler with `@jobs.handler("separate")`; the worker is the
......@@ -118,7 +119,7 @@ only process that loads the heavy deps. See `jobs.py` / `worker.py`.
## Test
```bash
cd armada/api && ~/.virtualenvs/douanier/bin/python -m pytest tests/ -q
cd armada/api && ~/.virtualenvs/fourier/bin/python -m pytest tests/ -q
```
## Status
......
armada/api/SRE.md
# Dear SRE — re: hosting *Douanier* on `api.nech.pl` 🛂
# Dear SRE — re: hosting *Fourier* on `api.nech.pl` 🎛️
A letter, because you own the seam between this little service and the public
internet, and I'd rather ask than assume. Here's what Douanier is, what state
internet, and I'd rather ask than assume. Here's what Fourier is, what state
it's in, and the handful of decisions that are yours, not mine.
## What it is (context)
**Douanier** (after Le Douanier Rousseau) is the ParVagues **Audio-Intelligence
**Fourier** (after Joseph Fourier — the transform behind the spectral features)
is the ParVagues **Audio-Intelligence
API** — it productizes the fleet's engines (stem separation, loop/chop finding,
emotion analysis, sample analysis, the feature stack) behind one self-hosted,
token-authed, async HTTP API. First customer is **hexa** (Shipow's
......@@ -24,7 +25,7 @@ Design decisions already locked (so you know the shape):
accounts/tokens** is the one thing that genuinely wants a backup.
- **Versioned at `/v1`.** Every route is emitted under `/v1/...` and I'll bump
the segment for breaking changes — so you can route by path prefix safely.
- **Runs in a venv** at `~/.virtualenvs/douanier`, created with
- **Runs in a venv** at `~/.virtualenvs/fourier`, created with
`--system-site-packages` so it reuses the already-installed ML stack
(torch/CLAP/numpy/librosa/demucs) and only adds FastAPI/uvicorn on top. This
matters for the containerization question below.
......@@ -37,8 +38,8 @@ guarded by a hardcoded dev bearer. Run locally:
```bash
cd armada/api
DOUANIER_DEV_TOKEN=$(openssl rand -hex 16) \
~/.virtualenvs/douanier/bin/uvicorn app:app --port 8780
FOURIER_DEV_TOKEN=$(openssl rand -hex 16) \
~/.virtualenvs/fourier/bin/uvicorn app:app --port 8780
# → http://127.0.0.1:8780/v1/healthz /v1/docs
```
......@@ -51,8 +52,8 @@ incrementally *under* this skeleton without changing its public shape.
1. **Public mapping.** Path-based (`api.nech.pl/v1/sound/...` → proxy forwards to
the service) or host-based (`sound.api.nech.pl`)? I emit `/v1/...`; tell me if
you'll strip/rewrite a prefix so I can set `root_path` accordingly.
2. **Process model.** My default is **systemd --user** units (`douanier-api` +
`douanier-worker`, linger on), mirroring the Bridge. Do you prefer that, or a
2. **Process model.** My default is **systemd --user** units (`fourier-api` +
`fourier-worker`, linger on), mirroring the Bridge. Do you prefer that, or a
**Docker** container? ⚠️ Containerizing is awkward here: the venv leans on the
host's multi-GB GPU-coupled ML stack — a faithful image is huge and GPU-bound.
If you want Docker, let's talk about whether the heavy worker stays on the host.
......@@ -64,7 +65,7 @@ incrementally *under* this skeleton without changing its public shape.
*this* box (RTX 2060) or a separate runner? How do you want me to target cloud
GPU later — env-selected backend, or a dispatch URL?
6. **Mounts.** Is the **freebox** reachable by the service user (path? always
mounted?) so I can point `DOUANIER_CACHE` at it? It'll grow to ~1–2 TB at
mounted?) so I can point `FOURIER_CACHE` at it? It'll grow to ~1–2 TB at
"success" scale; it's evictable, so a size cap is fine — any cap you want?
7. **Secrets.** How should I inject the bearer tokens' DB and the **HMAC signing
key** for artifact URLs — a systemd `EnvironmentFile`, a secrets manager,
......@@ -90,13 +91,13 @@ incrementally *under* this skeleton without changing its public shape.
No rush on all ten — #1, #2, #3 unblock the first real deploy; the rest can come
as the heavier endpoints land. Reply inline in this file or ping me.
— *Douanier, the customs officer* (on behalf of the fleet) ⛵
— *Fourier* (on behalf of the fleet) ⛵
---
## — SRE reply (2026-06-25, uncommitted — relayed via the SRE session)
Dear Douanier — read this before you finish `requirements.txt`. **Your hosting
Dear Fourier — read this before you finish `requirements.txt`. **Your hosting
assumption is wrong**, and it changes the build. Verified facts:
### The two boxes, corrected
......@@ -132,7 +133,7 @@ not a mandate). Make `analyze/emotion` pick by **what hexa sends**:
seam now so flipping the backend is an env var, not a rewrite.
### Storage: none long-term (PLN)
No freebox on erable, no persistent store. `DOUANIER_CACHE` = a **local dir,
No freebox on erable, no persistent store. `FOURIER_CACHE` = a **local dir,
hard-capped at 2 GB**, content-addressed, aggressively evicted — purely transient
artifact serving back to hexa. Losing it is a cache miss. The **SQLite tokens DB
is the only thing that wants a backup** (it's tiny — nightly copy is plenty).
......@@ -144,7 +145,7 @@ is the only thing that wants a backup** (it's tiny — nightly copy is plenty).
4. **TLS** — yes, terminated at erable nginx + certbot; service stays plain HTTP on loopback. ✅
5. **GPU** — none here. CPU-light now (per above); cloud/homelab GPU later via env-selected dispatch backend.
6. **Mounts** — no freebox on erable. Cache = local dir, 2 GB cap (above).
7. **Secrets** — systemd/Docker `--env-file ~/.config/douanier/douanier.env` (0600): tokens-DB path + HMAC key. No secrets manager for one box.
7. **Secrets** — systemd/Docker `--env-file ~/.config/fourier/fourier.env` (0600): tokens-DB path + HMAC key. No secrets manager for one box.
8. **Observability** — erable already runs Prometheus + node/pg exporters (:9090/:9100/:9187). Expose **`/v1/metrics`**, I'll scrape it. journald is enough for app logs.
9. **WAF/rate-limit** — coarse nginx `limit_req` at the edge on top of your per-token bucket. No Cloudflare (direct A record).
10. **CORS** — enforce in-app; send hexa's Vercel origins when known.
......@@ -167,7 +168,7 @@ RAM and listens on a port I publish to `127.0.0.1:9780`. The moment it's up, the
## ⚠️ Update (2026-06-25, later): `api.nech.pl` is a multi-API PLATFORM, and auth goes CENTRAL
PLN reframed the scope. `api.nech.pl` is **"Nech.PL APIs"**, a gateway hosting
many sub-APIs; **Douanier is the internal codename for the `audio` sub-API**.
many sub-APIs; **Fourier is the internal codename for the `audio` sub-API**.
Two things change for you:
1. **Public path = `/audio/v1/...`** (domain-first, independent per-API
......@@ -207,8 +208,8 @@ container's ready and I'll wire the gateway + auth_request + your docs tab.
## ✅ Correction (2026-06-28): CLAP is RIGHT — used the canonical way (precompute → index)
My 2026-06-25 "CLAP is the wrong engine" line overstated it. Reconciled with the
canonical Douanier design (now in `git@git.nech.pl:pln/sre` →
`Raspi5/05-douanier-audio-intelligence.md` + `nechapi-douanier-onboarding.md`):
canonical Fourier design (now in `git@git.nech.pl:pln/sre` →
`Raspi5/05-fourier-audio-intelligence.md` + `nechapi-fourier-onboarding.md`):
- **CLAP stays the engine for the corpus/search features** (search-by-vibe,
similar, zero-shot auto-tags). What's wrong is *running CLAP at request time on
......@@ -224,7 +225,7 @@ canonical Douanier design (now in `git@git.nech.pl:pln/sre` →
Platform also moved since 06-25: there's a **golden base image (`nechapi/py`)** +
a **`nechapi ship`** CLI (the paved road), and a tenant (**geo/v1**) is already
live through it. **Onboard via the runbook `nechapi-douanier-onboarding.md`** in
live through it. **Onboard via the runbook `nechapi-fourier-onboarding.md`** in
the SRE repo (Dockerfile `FROM nechapi/py:1`, `ENV NECHAPI_ROOT_PATH=/audio/v1`,
`create_app(...)`, `require_scope(...)`, then `nechapi ship`). Net unchanged:
containerize, loopback `:9780`, trust `X-Tenant`/`X-Scopes`, CPU-light at request.
......
armada/api/app.py
"""Douanier — ParVagues Audio-Intelligence API (the customs gate).
"""Fourier — ParVagues Audio-Intelligence API.
Walking skeleton (#22 scaffold + #36 vertical): a versioned FastAPI app with a
health check and ONE real, synchronous endpoint — /v1/analyze/emotion — guarded
......@@ -6,7 +6,7 @@ by a hardcoded dev bearer. It proves the three risky seams end-to-end (engines
import · service runs · a client can auth & call) before we invest in the
SQLite auth/quota/jobs/cache machinery (#23–#27) that hardens it later.
run: cd armada/api && ~/.virtualenvs/douanier/bin/uvicorn app:app --port 8780
run: cd armada/api && ~/.virtualenvs/fourier/bin/uvicorn app:app --port 8780
"""
import secrets
import tempfile
......@@ -68,7 +68,7 @@ def _principal(request: Request) -> auth.Principal | None:
and forwards X-Tenant / X-Scopes; we're loopback-only, so only the gateway
can reach us and those headers are trustworthy. No token store of our own.
Local-dev fallback (no gateway in front): a bearer matching DOUANIER_DEV_TOKEN
Local-dev fallback (no gateway in front): a bearer matching FOURIER_DEV_TOKEN
grants a wildcard, so `uvicorn` + curl works without the platform."""
tenant = request.headers.get(config.TENANT_HEADER)
if tenant:
......@@ -145,27 +145,27 @@ def metrics():
"""Prometheus text exposition (the erable scraper hits this). Counts only —
no secrets. Unauthenticated by design, reachable only on loopback/gateway."""
lines = [
"# HELP douanier_up 1 if the audio API is responding",
"# TYPE douanier_up gauge",
"douanier_up 1",
"# HELP douanier_requests_total HTTP requests by path and status",
"# TYPE douanier_requests_total counter",
"# HELP fourier_up 1 if the audio API is responding",
"# TYPE fourier_up gauge",
"fourier_up 1",
"# HELP fourier_requests_total HTTP requests by path and status",
"# TYPE fourier_requests_total counter",
]
for (path, status), n in sorted(_REQ_COUNTS.items()):
safe = path.replace("\\", "\\\\").replace('"', '\\"')
lines.append(f'douanier_requests_total{{path="{safe}",status="{status}"}} {n}')
lines.append(f'fourier_requests_total{{path="{safe}",status="{status}"}} {n}')
try:
st = cache.stats() # one row per kind: {kind, n, bytes, hits}
lines += [
"# HELP douanier_cache_rows cached results by kind",
"# TYPE douanier_cache_rows gauge",
"# HELP douanier_cache_hits_total cache hits served by kind",
"# TYPE douanier_cache_hits_total counter",
"# HELP fourier_cache_rows cached results by kind",
"# TYPE fourier_cache_rows gauge",
"# HELP fourier_cache_hits_total cache hits served by kind",
"# TYPE fourier_cache_hits_total counter",
]
for r in st:
k = str(r.get("kind", "")).replace('"', '\\"')
lines.append(f'douanier_cache_rows{{kind="{k}"}} {r.get("n", 0)}')
lines.append(f'douanier_cache_hits_total{{kind="{k}"}} {r.get("hits", 0)}')
lines.append(f'fourier_cache_rows{{kind="{k}"}} {r.get("n", 0)}')
lines.append(f'fourier_cache_hits_total{{kind="{k}"}} {r.get("hits", 0)}')
except Exception:
pass
return Response("\n".join(lines) + "\n", media_type="text/plain; version=0.0.4")
......
armada/api/auth.py
"""Auth — opaque bearer tokens in SQLite. The customs gate's papers check.
"""Auth — opaque bearer tokens in SQLite. The papers check at the door.
A token is a random URL-safe string handed out ONCE at mint time; we store only
its sha256 hash, so a DB leak can't be replayed as credentials. Verification is
......@@ -38,7 +38,7 @@ class Principal:
return _scopes.match_any(self.scopes, scope)
# ── minting / management (used by the douanier CLI) ─────────────────────────────
# ── minting / management (used by the fourier CLI) ─────────────────────────────
def ensure_account(name: str, notes: str = "") -> int:
with db.cursor() as c:
row = c.execute("SELECT id FROM accounts WHERE name=?", (name,)).fetchone()
......
armada/api/clients/README.md
......@@ -97,7 +97,7 @@ from it. Both are reproducible:
```bash
# 1. refresh the snapshot from the app (adds the public server block)
~/.virtualenvs/douanier/bin/python clients/refresh_openapi.py
~/.virtualenvs/fourier/bin/python clients/refresh_openapi.py
# 2. regenerate + bundle the TS client
cd clients/nech-api && ./codegen.sh
```
......
armada/api/clients/nech-api/README.md
......@@ -38,7 +38,7 @@ checked-in OpenAPI snapshot, then bundled with tsup.
```bash
# 1. refresh the snapshot from the live app (run from ../, in the venv)
~/.virtualenvs/douanier/bin/python ../refresh_openapi.py
~/.virtualenvs/fourier/bin/python ../refresh_openapi.py
# 2. regenerate + bundle every domain
./codegen.sh
```
......
armada/api/clients/refresh_openapi.py
......@@ -6,7 +6,7 @@ generated clients build against (#32, #44).
served behind the gateway's root_path), so we add it here. test_openapi_snapshot
guards both the path set and this server URL — keep them in sync via THIS script.
~/.virtualenvs/douanier/bin/python clients/refresh_openapi.py
~/.virtualenvs/fourier/bin/python clients/refresh_openapi.py
"""
import json
import sys
......
armada/api/config.py
"""Douanier config — paths, dev token, version. Env-overridable, sane defaults.
"""Fourier config — paths, dev token, version. Env-overridable, sane defaults.
Flat-module layout (run from this dir): the service imports `config` and
`engines.*` directly, matching the tide-table analyzers' sibling-import style.
......@@ -7,22 +7,22 @@ import os
from pathlib import Path
# ── identity ──────────────────────────────────────────────────────────────────
SERVICE = "douanier" # internal codename; the public sub-API is "audio".
SERVICE = "fourier" # internal codename; the public sub-API is "audio".
VERSION = "v1" # the audio sub-API's own version (independent).
# api.nech.pl is a multi-API PLATFORM (SRE update 2026-06-25): Douanier is the
# api.nech.pl is a multi-API PLATFORM (SRE update 2026-06-25): Fourier is the
# `audio` sub-API, served publicly at /audio/v1/... The gateway strips that prefix
# and proxies to us at root (so internally routes are /healthz, /analyze/emotion…).
# root_path makes FastAPI advertise the public paths in /openapi.json + docs.
ROOT_PATH = os.environ.get("DOUANIER_ROOT_PATH", "/audio/v1")
ROOT_PATH = os.environ.get("FOURIER_ROOT_PATH", "/audio/v1")
# ── central auth (platform-owned) ────────────────────────────────────────────
# The platform does CENTRAL auth + metering: nginx `auth_request` validates the
# bearer at the edge and injects the caller identity as these headers. We bind
# loopback-only (reachable only via the gateway) and TRUST them — no token store
# of our own (supersedes #23/#24). See SRE.md.
TENANT_HEADER = os.environ.get("DOUANIER_TENANT_HEADER", "X-Tenant")
SCOPES_HEADER = os.environ.get("DOUANIER_SCOPES_HEADER", "X-Scopes")
TENANT_HEADER = os.environ.get("FOURIER_TENANT_HEADER", "X-Tenant")
SCOPES_HEADER = os.environ.get("FOURIER_SCOPES_HEADER", "X-Scopes")
# ── repo-relative paths ─────────────────────────────────────────────────────────
HERE = Path(__file__).resolve().parent # armada/api
......@@ -32,19 +32,19 @@ FOUNDRY = ARMADA.parent / "tools" / "foundry" # the Foundry engines
# ── cache / data roots (transient — freebox is the SSOT later) ───────────────────
# Default to a local dir for dev; point CACHE_ROOT at the freebox mount in prod.
CACHE_ROOT = Path(os.environ.get("DOUANIER_CACHE", HERE / "_cache"))
DB_PATH = Path(os.environ.get("DOUANIER_DB", HERE / "_cache" / "douanier.db"))
CACHE_ROOT = Path(os.environ.get("FOURIER_CACHE", HERE / "_cache"))
DB_PATH = Path(os.environ.get("FOURIER_DB", HERE / "_cache" / "fourier.db"))
# ── auth ─────────────────────────────────────────────────────────────────────
# WALKING-SKELETON stub (#36): one hardcoded bearer from the env, swapped for the
# SQLite token store in #23. No token set → the guarded routes return 503 (not
# wide open) so we never accidentally ship an unauthenticated endpoint.
DEV_TOKEN = os.environ.get("DOUANIER_DEV_TOKEN", "")
DEV_TOKEN = os.environ.get("FOURIER_DEV_TOKEN", "")
# ── limits ─────────────────────────────────────────────────────────────────────
MAX_UPLOAD_MB = int(os.environ.get("DOUANIER_MAX_UPLOAD_MB", "40"))
MAX_UPLOAD_MB = int(os.environ.get("FOURIER_MAX_UPLOAD_MB", "40"))
# ── engine selection ─────────────────────────────────────────────────────────
# "clap" = CLAP/torch (rich, ~4 GB, needs the GPU/dev box) — default for local dev.
# "light" = librosa-heuristic V/A (no torch, fits the erable container) — set in prod.
EMOTION_ENGINE = os.environ.get("DOUANIER_EMOTION_ENGINE", "clap").lower()
EMOTION_ENGINE = os.environ.get("FOURIER_EMOTION_ENGINE", "clap").lower()
armada/api/deploy/douanier-worker.service → armada/api/deploy/fourier-worker.service
# Douanier job worker (#25) — systemd --user unit.
# Fourier job worker (#25) — systemd --user unit.
#
# The worker is a SEPARATE long-lived process from the API container: it claims
# pending jobs and runs the heavy engines (separate/sources/loops). Runs as a
......@@ -8,26 +8,26 @@
#
# Install on the deploy host (the box that runs the audio service):
# loginctl enable-linger $USER
# cp deploy/douanier-worker.service ~/.config/systemd/user/
# cp deploy/fourier-worker.service ~/.config/systemd/user/
# systemctl --user daemon-reload
# systemctl --user enable --now douanier-worker
# journalctl --user -u douanier-worker -f
# systemctl --user enable --now fourier-worker
# journalctl --user -u fourier-worker -f
#
# NOTE: the worker needs the SAME env as the API (DOUANIER_DB, DOUANIER_CACHE,
# NOTE: the worker needs the SAME env as the API (FOURIER_DB, FOURIER_CACHE,
# engine venv). Point WorkingDirectory + EnvironmentFile at the deployed copy.
# If the API runs containerized via `nechapi ship`, either run this worker on the
# host against a shared DB/cache volume, or add it as a second process in the
# image — a paved-road decision (see SRE.md / #34).
[Unit]
Description=Douanier audio-API job worker
Description=Fourier audio-API job worker
After=network.target
[Service]
Type=simple
WorkingDirectory=%h/srv/douanier/api
EnvironmentFile=%h/.config/douanier/douanier.env
ExecStart=%h/.virtualenvs/douanier/bin/python -m worker
WorkingDirectory=%h/srv/fourier/api
EnvironmentFile=%h/.config/fourier/fourier.env
ExecStart=%h/.virtualenvs/fourier/bin/python -m worker
Restart=on-failure
RestartSec=3
# graceful: the worker traps SIGTERM, finishes the current job's bookkeeping, exits
......
armada/api/douanier.py → armada/api/fourier.py
#!/usr/bin/env python3
"""douanier — admin CLI for the Audio-Intelligence API.
"""fourier — admin CLI for the Audio-Intelligence API.
The ONLY way tokens are created (no public signup in MVP). Run with the service
venv so it shares config/db:
~/.virtualenvs/douanier/bin/python douanier.py token mint --account hexa --scopes '*' --ttl 31
~/.virtualenvs/douanier/bin/python douanier.py token list
~/.virtualenvs/douanier/bin/python douanier.py token revoke 3
~/.virtualenvs/douanier/bin/python douanier.py db init
~/.virtualenvs/fourier/bin/python fourier.py token mint --account hexa --scopes '*' --ttl 31
~/.virtualenvs/fourier/bin/python fourier.py token list
~/.virtualenvs/fourier/bin/python fourier.py token revoke 3
~/.virtualenvs/fourier/bin/python fourier.py db init
"""
import argparse
import datetime as dt
......@@ -74,7 +74,7 @@ def cmd_cache_gc(a):
def main(argv=None):
p = argparse.ArgumentParser(prog="douanier", description=__doc__)
p = argparse.ArgumentParser(prog="fourier", description=__doc__)
sub = p.add_subparsers(dest="group", required=True)
tok = sub.add_parser("token", help="manage API tokens").add_subparsers(
......
armada/api/requirements-deploy.txt
# Douanier — DEPLOY (erable container) requirements: the LIGHT, torch-free stack.
# Fourier — DEPLOY (erable container) requirements: the LIGHT, torch-free stack.
#
# Unlike requirements.txt (which assumes a --system-site-packages venv that
# inherits the dev box's ML stack), this file is SELF-CONTAINED: the erable
......
armada/api/requirements-test.txt
# Douanier API — reproducible TEST deps (CI + a clean local run).
# Fourier API — reproducible TEST deps (CI + a clean local run).
#
# Deliberately NOT --system-site-packages: the suite must install standalone so
# CI is hermetic and a host python upgrade can't silently break it (it did once:
......
armada/api/requirements.txt
# Douanier API — the API layer only. Heavy audio deps (numpy, pydantic, torch,
# Fourier API — the API layer only. Heavy audio deps (numpy, pydantic, torch,
# CLAP, librosa, demucs) are INHERITED from system site-packages via a
# --system-site-packages venv, so we don't reinstall the multi-GB ML stack:
#
# python3 -m venv --system-site-packages ~/.virtualenvs/douanier
# ~/.virtualenvs/douanier/bin/pip install -r requirements.txt
# python3 -m venv --system-site-packages ~/.virtualenvs/fourier
# ~/.virtualenvs/fourier/bin/pip install -r requirements.txt
#
fastapi>=0.115
uvicorn[standard]>=0.30
......
armada/api/tests/conftest.py
"""Test bootstrap — runs before any test module imports config.
Points the SQLite DB at a throwaway temp file and sets a known dev token, so
tests never touch the real _cache/douanier.db. Starts from a clean slate.
tests never touch the real _cache/fourier.db. Starts from a clean slate.
"""
import os
import sys
import tempfile
from pathlib import Path
_DB = Path(tempfile.gettempdir()) / "douanier_test.db"
os.environ["DOUANIER_DB"] = str(_DB)
os.environ.setdefault("DOUANIER_DEV_TOKEN", "test-secret")
_DB = Path(tempfile.gettempdir()) / "fourier_test.db"
os.environ["FOURIER_DB"] = str(_DB)
os.environ.setdefault("FOURIER_DEV_TOKEN", "test-secret")
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
......
armada/api/tests/test_auth.py
......@@ -53,6 +53,6 @@ def test_only_hash_is_stored():
def test_dev_bootstrap_token():
# conftest sets DOUANIER_DEV_TOKEN=test-secret → recognized as wildcard
# conftest sets FOURIER_DEV_TOKEN=test-secret → recognized as wildcard
p = auth.verify("test-secret")
assert p is not None and p.account == "dev" and p.has_scope("emotion")
armada/api/tests/test_extra.py
......@@ -3,7 +3,7 @@
Torch-free, real DSP on synth clips. The naming engine is a vendored copy of the
Foundry namer; test_naming_vendored_copy_has_not_drifted cross-checks it.
cd armada/api && ~/.virtualenvs/douanier/bin/python -m pytest tests/test_extra.py -q
cd armada/api && ~/.virtualenvs/fourier/bin/python -m pytest tests/test_extra.py -q
"""
import importlib.util
import io
......
armada/api/tests/test_feats.py
......@@ -4,7 +4,7 @@ Synth clips with known properties so the asserts are about MEASUREMENT, not a
fixture: a low sine reads bass register, a bright noise reads high register, the
feature stack carries the L0 floor + a correct tempo. conftest sets up paths.
cd armada/api && ~/.virtualenvs/douanier/bin/python -m pytest tests/test_feats.py -q
cd armada/api && ~/.virtualenvs/fourier/bin/python -m pytest tests/test_feats.py -q
"""
import sys
import tempfile
......
armada/api/tests/test_grade_endpoint.py
......@@ -5,7 +5,7 @@ container has no Foundry). This locks the endpoint contract AND mechanically
proves the copy still grades IDENTICALLY to the canonical — so a future edit to
the Foundry grader that isn't re-vendored fails CI here, not silently in prod.
cd armada/api && ~/.virtualenvs/douanier/bin/python -m pytest tests/test_grade_endpoint.py -q
cd armada/api && ~/.virtualenvs/fourier/bin/python -m pytest tests/test_grade_endpoint.py -q
"""
import importlib.util
import io
......
armada/api/tests/test_signal.py
......@@ -2,7 +2,7 @@
Torch-free, real DSP on synth clips. conftest sets paths + the dev fallback.
cd armada/api && ~/.virtualenvs/douanier/bin/python -m pytest tests/test_signal.py -q
cd armada/api && ~/.virtualenvs/fourier/bin/python -m pytest tests/test_signal.py -q
"""
import io
import sys
......
armada/api/tests/test_smoke.py
......@@ -7,7 +7,7 @@ The emotion inference itself is monkeypatched — here we assert routing, auth
gating, the cache, and the deploy seams (/separate 503, /metrics). Env/DB/path
setup is in conftest.py.
cd armada/api && ~/.virtualenvs/douanier/bin/python -m pytest tests/ -q
cd armada/api && ~/.virtualenvs/fourier/bin/python -m pytest tests/ -q
"""
from fastapi.testclient import TestClient
......@@ -16,7 +16,7 @@ from engines import ears
client = TestClient(APP.app)
# the local-dev fallback: a bearer matching DOUANIER_DEV_TOKEN (conftest) → wildcard.
# the local-dev fallback: a bearer matching FOURIER_DEV_TOKEN (conftest) → wildcard.
DEV = {"Authorization": "Bearer test-secret"}
# how the platform gateway actually calls us: injected identity headers. A broad
# api:audio:* grant authorizes every audio route (scopes are path-derived now).
......@@ -149,8 +149,8 @@ def test_separate_is_503_until_gpu():
def test_metrics_exposes_prometheus_text():
r = client.get("/metrics")
assert r.status_code == 200
assert "douanier_up 1" in r.text
assert "douanier_requests_total" in r.text
assert "fourier_up 1" in r.text
assert "fourier_requests_total" in r.text
def test_openapi_advertises_public_root_path():
......
armada/api/worker.py
#!/usr/bin/env python3
"""The job worker daemon (#25) — a single serialized loop.
~/.virtualenvs/douanier/bin/python -m worker # or: python worker.py
~/.virtualenvs/fourier/bin/python -m worker # or: python worker.py
Boots, recovers crashed in-flight jobs, then loops: claim one pending job → run
its registered handler → write result / error → optional webhook. One job at a
......
armada/tasks/completed-archive.md
......@@ -144,12 +144,12 @@ below is measured; honest negatives kept honest.
---
# Douanier — birth of the Audio-Intelligence API (2026-06-25)
# Fourier — birth of the Audio-Intelligence API (2026-06-25)
Designed live with PLN via a long AskUserQuestion co-design (5 rounds), then built lean. The
product: a self-hosted, token-authed, async API in `armada/api/` exposing the fleet's engines
(emotion, samples, features, separation, loops) to hexa (Shipow) + future seats. Full design in
memory `project_douanier_api`. Named after Le Douanier Rousseau (the customs gate).
memory `project_fourier_api`. Named after Joseph Fourier — the transform behind `/spectrum` and the feature stack. [Codename renamed 2026-06-29 from the original "Douanier" (Le Douanier Rousseau).]
## #22 / #36 — Scaffold + emotion walking skeleton
- **Description:** stand up the service shell + ONE real endpoint to de-risk the three seams
......@@ -157,19 +157,19 @@ memory `project_douanier_api`. Named after Le Douanier Rousseau (the customs gat
- **Done:** FastAPI app (`app.py`) versioned `/v1`; `/v1/healthz` + synchronous
`POST /v1/analyze/emotion` (upload → valence/arousal via the tide-table CLAP seam
sample_semantics.embed_audios → emotion_ontology.score); `engines/` lazy-import adapter;
runs in `~/.virtualenvs/douanier` (--system-site-packages). Commit 7f558a7.
runs in `~/.virtualenvs/fourier` (--system-site-packages). Commit 7f558a7.
- **Learnings:** lean rewire — the cheap "ears" don't need the job queue/quota for one trusted
customer, so a walking skeleton (sync + stub token) reaches value in ~1 session vs 6 tasks.
Arch PEP-668 blocks system pip → venv that INHERITS the multi-GB ML stack, adds only FastAPI.
CLAP cold-load is ~34 s (→ later #28 warms it once in the worker). Real pipeline proven on a
synthetic clip end-to-end.
- **Deps:** root of the whole Douanier tree.
- **Deps:** root of the whole Fourier tree.
## #23 — Bearer-token auth + scopes + the `douanier` CLI
## #23 — Bearer-token auth + scopes + the `fourier` CLI
- **Description:** replace the dev-token stub with a real self-hosted token store.
- **Done:** `db.py` (stdlib sqlite3, WAL for API+worker concurrency); `auth.py` — mint()
returns plaintext ONCE, stores only sha256 (DB leak unreplayable), verify() → Principal with
scope/expiry checks, `require_scope()` dependency (401/403); `douanier.py` admin CLI
scope/expiry checks, `require_scope()` dependency (401/403); `fourier.py` admin CLI
(token mint/list/revoke). `/v1/me` echoes identity. Commit fe9ca02.
- **Learnings:** scope='*' wildcard for hexa; dev-token kept as a documented local-only
bootstrap (prod never sets it) so curl works without minting. 15/15 tests; mint expiry landed
......@@ -181,7 +181,7 @@ memory `project_douanier_api`. Named after Le Douanier Rousseau (the customs gat
- **Done:** `cache.py` — content_id = sha256 of DECODED PCM (re-encodes dedupe) + raw-bytes
fallback; params_hash folds engine settings; get/put (JSON inline or result_ref path);
yt-id/url→content_id aliases; LRU/size-cap gc() that unlinks evicted artifacts. Wired into
`/v1/analyze/emotion` (`X-Douanier-Cache: hit|miss`). Commit 394bec0.
`/v1/analyze/emotion` (`X-Fourier-Cache: hit|miss`). Commit 394bec0.
- **Learnings:** **measured 5663× on a real CLAP run** (10.6 s miss → 0.002 s hit). Cache keys
on the engine so clap/light don't collide. 22/22 tests. This is the margin story vs cloud
egress that justified self-hosting.
......@@ -249,14 +249,14 @@ memory `project_douanier_api`. Named after Le Douanier Rousseau (the customs gat
## #32 — Make it callable: OpenAPI snapshot + zero-dep TS client
- **Done:** `clients/openapi.json` (3.1 snapshot, servers pinned to api.nech.pl/audio/v1, +
drift-guard test vs the live app), `clients/douanier.ts` (typed, zero-dep, one method/engine,
`_cache` surfaced, DouanierError; tsc --strict clean), `clients/README.md` (Vercel example +
drift-guard test vs the live app), `clients/fourier.ts` (typed, zero-dep, one method/engine,
`_cache` surfaced, FourierError; tsc --strict clean), `clients/README.md` (Vercel example +
scope table + codegen one-liner). Commit 11c8f9d.
- **Learnings:** a hand-written zero-dep client beats generated bloat for ~11 simple multipart
routes; the snapshot-drift test keeps hexa's spec honest as routes evolve. 60/60 tests.
- **Deps:** unblocks #33 (onboard hexa); needs #42 deploy to be live-callable.
## #42 — Deploy douanier:latest to erable → green /audio/v1/healthz
## #42 — Deploy fourier:latest to erable → green /audio/v1/healthz
- **Done:** First real deploy of the CPU image to erable (pln@nech.pl). Image transferred via
`docker save | ssh erable docker load`, run loopback-only on 127.0.0.1:9780. Registered
/openapi.json + /docs as public gateway routes (nechapi 89586eb). DEPLOY.md gotchas captured
......@@ -266,7 +266,7 @@ memory `project_douanier_api`. Named after Le Douanier Rousseau (the customs gat
glibc 2.36 uses for pthread_create → container boots uvicorn then SIGSEGVs (exit 139) /
"OpenBLAS pthread_create … Operation not permitted". (2) pin OPENBLAS/OMP/NUMEXPR/MKL
_NUM_THREADS=1. The _platform service sidesteps #1 by being Alpine/musl on purpose. Data
volume = ~/srv/douanier/data (no sudo for /srv; pure transient cache).
volume = ~/srv/fourier/data (no sudo for /srv; pure transient cache).
- **Deps:** unblocked #33; the keystone for the whole live API.
## #43 — Hierarchical scope convention (realm:domain:path) + central enforcement + CLI/docs
......@@ -293,7 +293,7 @@ memory `project_douanier_api`. Named after Le Douanier Rousseau (the customs gat
instead), so the old "31-day" idea didn't apply. Token plaintext is shown ONCE at mint (only
the sha256 is stored). Onboarding artifact is transitional — PLN's stance: Scalar /docs +
landing will replace the PDF; "only token distribution matters." Client SDK renamed
Douanier→NechAPI (umbrella + .audio namespace) + header X-Douanier-Cache→X-Nech-Cache so the
Fourier→NechAPI (umbrella + .audio namespace) + header X-Fourier-Cache→X-Nech-Cache so the
internal codename never reaches a consumer.
- **Deps:** was blocked by #42 (deploy) + #43 (scopes); closes the EPIC's MVP.
......@@ -353,6 +353,6 @@ memory `project_douanier_api`. Named after Le Douanier Rousseau (the customs gat
## #25 — Async job model + worker daemon (SQLite)
- **Description:** the async backbone so heavy work (sources/separate/loops) runs out-of-band and the API stays responsive.
- **Done:** committed 17d3f46. Four design decisions taken with PLN (AskUserQuestion w/ sketches): HYBRID sync/async (cheap analyses stay sync, heavy = jobs), PER-ENGINE submit (202 {job_id}) + SHARED poll (GET /jobs/{id}) + cancel (DELETE), SINGLE serialized worker, X-Accel-Redirect for artifacts. Built: jobs table (db.py), jobs.py (enqueue + born-done idempotent fast path, atomic claim, progress/finish/fail-requeue-under-cap, crash recovery, cooperative cancel), worker.py (serialized recover→claim→dispatch→webhook loop, SIGTERM-graceful, @jobs.handler registry), typed JobStatus endpoints, deploy/douanier-worker.service (systemd --user). 11 new tests (67→78), incl. a real 6-thread×25-job no-double-claim concurrency test. OpenAPI refreshed (15 ops) + @nech/api regenerated (getJob/cancelJob).
- **Done:** committed 17d3f46. Four design decisions taken with PLN (AskUserQuestion w/ sketches): HYBRID sync/async (cheap analyses stay sync, heavy = jobs), PER-ENGINE submit (202 {job_id}) + SHARED poll (GET /jobs/{id}) + cancel (DELETE), SINGLE serialized worker, X-Accel-Redirect for artifacts. Built: jobs table (db.py), jobs.py (enqueue + born-done idempotent fast path, atomic claim, progress/finish/fail-requeue-under-cap, crash recovery, cooperative cancel), worker.py (serialized recover→claim→dispatch→webhook loop, SIGTERM-graceful, @jobs.handler registry), typed JobStatus endpoints, deploy/fourier-worker.service (systemd --user). 11 new tests (67→78), incl. a real 6-thread×25-job no-double-claim concurrency test. OpenAPI refreshed (15 ops) + @nech/api regenerated (getJob/cancelJob).
- **Learnings:** SQLite WAL serializes writers, so the claim race reduces to candidate-select → `UPDATE ... WHERE id=? AND status='pending'` + rowcount-0 retry — no locks needed, provably exactly-once across threads. typescript-fetch's generated instanceOf guard indexes a camel/snake key union that trips TS7053 under noImplicitAny → relaxed noImplicitAny for the 100%-generated package (emitted .d.ts stays fully typed). MID-BUILD: the btrfs disk hit ENOSPC at 98% (20G "free" but unallocated, metadata-full) — writes failed intermittently; cleared ~5.7G of regenerable caches (npm + ~/.cache browser/HF, user-approved) to continue. Disk still 97% — flagged for real cleanup.
- **Deps:** was blocked by scaffold + central auth (done); unblocks #29 → #30 → #31. Worker prod container-vs-host wiring is a paved-road residual (SRE/#34).
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment