Commit a828d5f0 by PLN (Algolia)

perf(protect): the guard was spending a tenth of a core to discover nothing had changed

219.1ms per 2s tick, 6 forks, with NOTHING running to protect. None of it was
protection: three pgrep -x at 59ms each (pgrep reads cmdline for every process
on the box, and it ran once per target), plus cat, two chrt -p reads and an
unconditional prlimit per pid, each in its own command substitution.

Now zero forks in the steady state: one bash pass over /proc/*/comm, sched from
/proc/<pid>/stat fields 40/41, prlimit only when /proc/<pid>/limits says so.
28.5ms/tick, 1.42% of a core at the unchanged 2s interval.

The interval is deliberately untouched -- the 7.7x came from forks alone, so the
responsiveness that re-protects scsynth before its first sound was not traded
for battery.

Verified differentially against the old pgrep path with a decoy fleet (ardour9,
ArdourGUI, ardour-8.6 must match; ardour-decoy, sclang-notreally, scsynthx and a
bash whose path contains 'ardour' must not) and against chrt -p / cat on live
scsynth+sclang. Needs sudo tools/install-protect.sh to take effect.

Also logs two findings from the same measurement pass: sc-watchdog restarts
SuperDirt on every clean start (6s patience vs 8s boot) and two starts in five
minutes hit StartLimitBurst, leaving the rig unstartable; and the other two
reconcile loops cost 5.3% between them with real event sources available.
parent 9150a200
......@@ -4,6 +4,58 @@ Sprint entries, newest first. Player-facing: what changed about *playing*, not
about the code. Task IDs reference the L'Armada board; `n/a` where the work was
unplanned (which, on a gig night, is most of it).
## Sprint 4 — 2026-09-06 (later) · the guard stops charging rent
Sprint 3's guard finally guarded. Then we asked what it cost to do so, and the
answer was a tenth of a core, forever, on a battery, whether or not anything was
playing. This sprint is about the difference between watching and *fidgeting*.
### Fixed
- **`parvagues-protect` costs 1.4% of a core instead of 11%** (n/a) — 219.1 ms
per 2 s tick became 28.5 ms, and 6 forks became **zero**. None of that 219 ms
was protection: three `pgrep -x` at 59 ms each (pgrep reads every process's
cmdline, and it was called once per target), then `cat`, two `chrt -p` reads
and an unconditional `prlimit` per pid, each in its own subshell. Everything
now comes from `/proc` through bash builtins. **The poll interval is
unchanged at 2 s on purpose** — the whole 7.7× came from forks, so the
responsiveness that re-protects scsynth before it makes its first sound was
not sold for battery. On stage nothing about it behaves differently; the fan
just has less to say.
### Found, not yet fixed
- **Starting SuperDirt twice inside five minutes bricks it until
`reset-failed`** (n/a). `parvagues-sc` reports `active` the moment sclang
execs, but scsynth appears ~8 s later — and `sc-watchdog` gives up waiting at
6 s, so it restarts a unit that was booting perfectly well. Each clean start
therefore spends two of systemd's three allowed starts, and the second start
in a five-minute window lands on `start-limit-hit`: SuperDirt refuses to come
up at all, which on stage is indistinguishable from dead. Reproduced tonight
from a fresh start with nothing wrong, cleared, and written up with the fix
(a boot grace read from the unit's own `ActiveEnterTimestamp`) in
`backlog.md`. **This is the next thing to fix, ahead of any polish.**
- **The idle rig costs ~14.5% of a core with nothing playing** (n/a), across
three reconcile loops that re-discover an unchanged graph every 2 s. Protect
was the biggest but only 60% of it; `midi-autoconnect` (2.74%) and
`tidal-ardour-autoroute` (2.53%) both have a real event source to block on
instead — `pw-link --monitor`, and `/proc/asound/seq/clients` for ALSA.
### Doctrine
- **Cheap prefilter, expensive test last.** Bash's regex engine is the slowest
thing in reach: testing one anchored alternation against every process's
`comm` costs 149 ms/tick — *worse than the `pgrep` it replaced*. A `[sSaA]*`
character class rejects 606 of 641 processes first, and the same regex then
costs nothing. Measure the replacement, not just the thing being replaced.
- **`case "$x" in $pattern)` does not do alternation.** `|` is recognised when
`case` is parsed, so a `s*|S*|a*|A*` arriving from a variable is one pattern
containing literal pipes, and matches nothing. Every part was individually
correct and the scan came back empty. A bracket class survives expansion; a
differential test against the code being replaced is what caught it, and
reading it again would not have.
- **Don't cache pid → name.** It would save 25 ms a tick and it is a stale
binding by construction — the kernel recycles pids, and a cached "not a
target" leaves scsynth naked. This is the same bug shape as the one the file
was written to fix; the file's own header says so.
## Sprint 3 — 2026-09-06 (late) · the box plays alone, and the guard now guards
Sprint 2 made the surface tell the truth. This one makes the *laptop* self-
......
......@@ -2129,19 +2129,72 @@ scattered through this file resolve locally.
## Housekeeping
- **`rig-doctor` does not check LV2 plugins** — the same gap class as the PySide6 miss.
- **`parvagues-protect` burns 10.7% of a core, forever, on a battery laptop.**
Measured 2026-09-06: 35.7 s CPU over 333 s uptime. It is a system unit enabled
at boot, so this runs whether or not anything is playing — which cuts against
the whole reason rig units are on-demand here (see the box's boot policy in
`tools/rig_units.py`). Cause is fork count, not work: every 2 s tick forks
`pgrep` per target, then **two** `chrt -p` calls per pid merely to READ the
policy and priority, plus an unconditional `prlimit`. Roughly 12 processes
every two seconds to discover that nothing changed.
Fixes, cheapest first: read policy/priority from `/proc/<pid>/stat` fields 18
and 19 instead of forking `chrt -p` twice; call `prlimit` once per pid rather
than every tick; raise the interval (2 s is far below anything a human or an
OOM event notices — 10 s would be 5× cheaper on its own). Do not "fix" it by
disabling the unit: the protection has to be up BEFORE the audio starts.
- ~~**`parvagues-protect` burns 10.7% of a core, forever, on a battery laptop.**~~
**FIXED 2026-09-06 (late), pending a root install.** Re-measured to 219.1 ms
per 2 s tick = **10.96% of a core with nothing running to protect**, so the
original figure was if anything generous. It was all discovery, none of it
protection: three `pgrep -x` at 59 ms each (pgrep reads cmdline for every
process on the box, and it was called once per target), plus `cat`, two
`chrt -p` reads and an unconditional `prlimit` per pid, each inside a command
substitution. Now answered entirely with bash builtins — one pass over
`/proc/*/comm`, sched read from `/proc/<pid>/stat` fields 40/41, `prlimit`
only when `/proc/<pid>/limits` says it is needed:
| | ms/tick | forks/tick | % of a core @2 s |
|---|---|---|---|
| before | 219.1 | 6 | 10.96% |
| after | 28.5 | **0** | **1.42%** |
The interval is untouched at 2 s deliberately — the 7.7× came from forks
alone, so the responsiveness that catches a restart before first sound was
not traded away. `PARVAGUES_PROTECT_INTERVAL=5` takes it to 0.57% if the
battery ever matters more than a 5 s window of non-RT audio after a restart.
What made it cheap was the *prefilter*: a `[sSaA]*` character class rejects
606 of 641 processes before bash's regex engine — by far the slowest thing in
reach — is consulted. Testing the regex on every comm costs 149 ms/tick,
**worse than the pgrep it replaced**. Verified differentially against the old
`pgrep` path with a decoy fleet (`ardour9`, `ArdourGUI`, `ardour-8.6` must
match; `ardour-decoy`, `sclang-notreally`, `scsynthx` and a `bash` whose path
contains "ardour" must not) and against `chrt -p`/`cat` on live scsynth+sclang.
**Still to do: `sudo tools/install-protect.sh`** — the repo has the fix, the
running daemon does not.
- **`sc-watchdog` restarts SuperDirt on every clean start, and two starts in
five minutes leave the rig `failed` and unstartable.** Found 2026-09-06 by
starting `parvagues-sc` for an unrelated test. `parvagues-sc.service` is
`Type=simple`, so it reports `active` the instant `sclang` execs — but
`scsynth` only appears ~8 s later, when SuperDirt boots the server. The
watchdog's main loop counts a miss every `POLL_SECS=2` while the unit is
active and scsynth is absent, and acts at `MISSES_TO_ACT=3`. **6 s < 8 s**, so
a perfectly healthy start always trips it. Journal, verbatim, from a fresh
start with nothing wrong: `23:17:12 scsynth GONE (3 polls) while
parvagues-sc.service is active — restarting (0 prior in window)` → `23:17:21
scsynth up after 8s` → `recovered`.
The consequence is the gig-night one: each start burns two of systemd's
`StartLimitBurst=3` (PLN's, then the watchdog's), so a *second* start inside
`StartLimitIntervalUSec=5min` hits the limit and the unit goes
`failed (result: start-limit-hit)` — SuperDirt then refuses to start at all
until `systemctl --user reset-failed parvagues-sc`. On stage that reads as
"the rig is dead and will not come back". Reproduced end to end tonight and
cleared with `reset-failed`; the box is back to `inactive/linked` as found.
Fix: give the watchdog a boot grace keyed to the unit's own
`ActiveEnterTimestamp` — do not count misses until the unit has been active
longer than SuperDirt's measured boot (~8-12 s; 25 s is a safe floor). It
already shells `systemctl --user is-active` every poll, so the timestamp is
one field on a call it is making anyway. `await_scsynth`'s
`BOOT_WAIT_SECS=100` covers the *post-restart* wait and was never wired to the
*initial* start, which is the whole bug.
- **The other two reconcile loops cost 5.3% of a core between them, and both can
be event-driven.** Measured 2026-09-06 on the same idle box:
`midi-autoconnect` **2.74%**, `tidal-ardour-autoroute` **2.53%**,
`parvagues-sc-watchdog` 0.48%, `parvagues-bridge` 0.07%, `midiviz` 0.09%.
Neither of the big two is slow per call (`aconnect -l` ~10 ms, `pw-link -l`
~11 ms); they just make 3-5 of them plus awk every 2 s to re-discover an
unchanged graph. Both have a real event source: `pw-link -m/--monitor` blocks
and prints on link/port change, and ALSA-seq exposes
`/proc/asound/seq/clients`, which a bash `read` can compare between ticks for
free. Reconcile on change instead of on a timer and each drops to ~0.1%.
Total idle rig cost today: **~14.5% of a core with nothing playing**; protect's
fix takes that to ~5.8%, and these two would take it under 1%.
- **`rig-doctor`'s `parvagues-protect: PASS` tests for files, not for function.**
It checks that `bin` and `unit` exist — the exact pair that stayed true all
through 2026-09-06 while the daemon could not write a single `oom_score_adj`
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment