Commit b227c7aa by PLN (Algolia)

gig: the launch that printed green had already had Ardour killed

The desktop icon brought Ardour up at 23:59 and the shell reported it Killed
ten seconds later, three lines before gig-up said 'gig-up done'. One root
cause under three complaints:

parvagues-protect's ardour:80 target promotes Ardour's GUI thread to
SCHED_FIFO/80. Its audio thread already gets RT from the backend the right
way (AudioEngine 1, FIFO|RESET_ON_FORK at 83, above the GUI), so the
promotion buys nothing and costs two things. RLIMIT_RTTIME is inherited and
differs by launcher -- gnome-shell hands children 200000us, a systemd user
unit hands them unlimited -- so an Ardour started from the icon carries a
200ms realtime budget, and loading the session blows it: SIGXCPU then
SIGKILL, with no kernel log line, which is why the journal looked clean.
And a GUI thread at 40-85% of a core on a realtime policy misses deadlines:
xrun_by.ardour went 0->46 in six minutes, and it is chronic, not new --
5172, 4852, 1114 and 987 ardour xruns in the Sep 10/18/20/22 sessions.

d1 alone on the speakers was never a second bug; it is what WirePlumber
does with SuperCollider:out_1/2 when there is no Ardour to route through.

Also: the DJF overlay's third row was an empty string, which is a row paying
no rent. It now carries travel from the bypass detent as a signed percent,
each side normalised against its own throw so 100% means end stop. The
cutoff stays on row 2 -- row 3 is the bezel row.

Findings in TODO_GIG.md with Thursday reordered; story in armada/tasks/046.
parent 8dbc068b
......@@ -541,3 +541,109 @@ control per second in corpus numbering — so "minute 23:30 the bass is too
saturated" resolves to which knob, how far, how many times. That is the input
for code/effect/DAW changes, not just a mastering fix. Then
`docs/2026-09-25-post-gig-cleanup.md`.
## Thu 24, 00:15 — one root cause, three symptoms: pv-protect puts the Ardour GUI on a realtime policy
PLN pressed the ParVagues desktop icon at 23:59, heard **kick on the laptop
speakers**, saw **no Ardour**, and later **crackles**. Three complaints, one
mechanism. The launcher is NOT stale: the symlink resolves to
`tools/parvagues.desktop` → `tools/gig-up-window.sh` → the repo-root `gig-up.sh`,
and the log shows it doing every step in order.
**What the launch log says** (`~/.cache/parvagues/gig-up.log`):
```
• launching Ardour (ardour) via pw-jack…
• launching Pulsar (pulsar)…
gig-up.sh: line 619: 540676 Killed setsid pw-jack ardour "…/Tidal Live.ardour"
✓ gig-up done. SuperDirt owns the LaunchControl; Ardour + Pulsar are coming up.
```
Ardour was launched, lived ~10 s, and was **SIGKILLed**. Then the script said
green. `pv-protect` had touched it 2 s in:
```
23:59:55 protected: ardour[540676] oom:100->-1000 sched:SCHED_OTHER/0->FIFO/80
```
### The mechanism (measured, not guessed)
`parvagues-protect`'s third target is `ardour:80` — it puts **Ardour's GUI
thread** on `SCHED_FIFO/80`. Measured on the Ardour PLN opened by hand at 00:04
(PID 615285): the `ArdourGUI` main thread is FIFO/80, and gig-log has it burning
**40–85 % of a core** while the session sits idle. Meanwhile Ardour's audio
thread gets its own RT *correctly* from the backend — `AudioEngine 1` is
`SCHED_FIFO|SCHED_RESET_ON_FORK` at **83**, i.e. above the GUI, granted by
PipeWire, needing nothing from us. So the FIFO on the GUI thread buys zero and
costs three things:
1. **The SIGKILL.** `RLIMIT_RTTIME` is inherited, and it differs by who
launched you: `gnome-shell` runs with **200000 µs**, a systemd user unit with
**unlimited** (measured: gnome-shell 200000, pv-protect's own shell
unlimited). The desktop icon is a gnome-shell descendant, so Ardour launched
from it carries a **200 ms realtime budget** — and a GUI thread loading a
355 KB session with plugins blows 200 ms of uninterrupted CPU easily. The
kernel then sends SIGXCPU and, soft == hard, SIGKILL. **No kernel log line,
which is why the journal looked empty.** The Ardour PLN started by hand at
00:04 shows `Max realtime timeout: unlimited` and survived — the
discriminator holds.
2. **The crackles.** A FIFO/80 thread at 85 % of a core, on a CPU capped at
`max_perf_pct=65` with governor `powersave`, misses deadlines. gig-log's
attribution is unambiguous: in the 6 minutes after Ardour opened,
`xrun_by.ardour` went **0 → 46** while SuperCollider added ~9. And it is
**chronic, not new** — `ardour` xruns per session: 5172 (Sep 10), 4852
(Sep 18), 1114 (Sep 20), 987 (Sep 22). Nobody had ever attributed them.
3. **d1 straight to the speakers.** Not a second bug — the consequence. With no
Ardour, WirePlumber auto-links `SuperCollider:out_1/2` to the internal sink
and d2–d14 reach nothing. The gate said exactly that ("headphones: 2/28 fold
links present") and then *graded the rig as the headphones rig*, because it
infers the mode from Ardour's absence. With Ardour back up the graph is
correct again: 61 orbit links into `ardour:Tidal NN`, `ardour:Master` → the
sinks, no direct SC → speaker link left over.
### Thu morning — in this order
- [ ] **1. Make Ardour survive its own launch.** Blocks every Ardour-side item
below. Two candidate fixes, both one line, PLN's call (the file is
root-owned — `sudoedit /usr/local/bin/parvagues-protect`, then
`sudo systemctl restart parvagues-protect`):
- **(a) Drop the FIFO, keep the OOM shield.** `ardour:80` → protect
`oom_score_adj=-1000` only. Justification: `AudioEngine 1` already runs
FIFO/83 from the backend, so the GUI promotion protects nothing that
makes sound. This also fixes symptom 2. **Recommended.**
- **(b) Lift the budget before promoting.** `prlimit --pid $pid
--rttime=unlimited` before the `chrt`, exactly the shape the script
already uses for `memlock_unlimited`. Fixes symptom 1 only; leaves an
85 %-of-a-core GUI thread at FIFO/80 in the graph.
AC for either: launch from the **desktop icon** (the path that failed —
launching from a terminal or the Bridge does not reproduce it), and
Ardour is still alive 60 s later with `xrun_by.ardour` flat.
Check whether `perf-audio` does the same `chrt` on Ardour; if so it needs
the same edit or it will re-arm the bug on the next perf pass.
- [ ] **2. The CPU cap, still unpaid** (third session running). Measured now:
`max_perf_pct=65`, governor `powersave`, EPP `balance_power`, and gig-up's
own line claimed "thermal-mode performance holding" while the gate read
the opposite — that perf line is **stale and must not be trusted**.
`echo 100 | sudo tee /sys/devices/system/cpu/intel_pstate/max_perf_pct`
plus `sudo cpupower set --epp performance`. Until this is paid, no xrun
number from the rehearsal means anything.
- [ ] **3. `systemctl --user restart parvagues-sc` at the START of the
rehearsal** — `s.latency` 0.1 is still only in the file; the running
process holds 0.2. AC after 10 min of play:
`journalctl --user -u parvagues-sc -b | grep -cE '\blate [0-9]'` == 0.
- [ ] **4. Master fader, then Ctrl+S.** `check-mix` reads the SAVED session and
`use-osc=0`, so nothing outside Ardour can see the live desk. Enable
Ardour OSC while you are in there and the blind spot closes for good.
- [ ] **5. Quantum 512 listen test**, only after items 1 and 2.
- [ ] **6. Full-set run**, late-count + xrun attribution from gig-log.
### The gate bug behind the false green (post-gig, write it down now)
In stage mode, an Ardour that **gig-up itself just spawned** and that is dead at
gate time must **FAIL**, not `SKIP`. Tonight the gate reasoned "Ardour is not
open → this must be the headphones rig" and graded a mode PLN was not playing,
then `gig-up.sh` printed `✓ gig-up done … Ardour + Pulsar are coming up` about a
process it had already been told was `Killed`. The launcher knows the PID and
knows whether the spawn survived; it must say so. "A clean exit is not a launch"
is already written in `gig-up-window.sh` — this is the same lesson one layer in:
**a spawn is not a process.**
---
log: 046
title: "The bodyguard was the assassin"
date: 2026-09-24
task: "n/a — gig-eve rig triage (TODO_GIG.md)"
tags: [rig, audio, realtime, observability, gig]
shareable: true
---
## Cap (what & why)
Midnight, gig day. PLN presses the ParVagues icon and reports three things at
once: kick coming out of the **laptop speakers**, **no Ardour on screen**, and a
minute later, **crackles**. The launcher's own log ends with a green tick. Three
complaints, one hour before the only rehearsal the set will ever get.
## Manœuvre (how)
Read the launch log before touching anything. It said, in order: launching
Ardour, launching Pulsar, then —
```
gig-up.sh: line 619: 540676 Killed setsid pw-jack ardour "…/Tidal Live.ardour"
✓ gig-up done. SuperDirt owns the LaunchControl; Ardour + Pulsar are coming up.
```
A SIGKILL and a green tick, three lines apart. The journal had no OOM, no
segfault, nothing — which is itself the clue: exactly one kill mechanism leaves
no kernel line. Two seconds before the death, `parvagues-protect` had logged
`protected: ardour[540676] sched:SCHED_OTHER/0->FIFO/80`.
Then the discriminator, measured on the Ardour PLN opened by hand afterwards:
`RLIMIT_RTTIME` is **inherited**, and it differs by parentage — `gnome-shell`
runs at **200 ms**, a systemd user unit at **unlimited**. The desktop icon is a
gnome-shell descendant. So: promote the GUI thread to a realtime policy, hand it
a 200 ms realtime budget, then ask it to load a 355 KB session with plugins.
SIGXCPU, then SIGKILL, silently.
## Prise (findings / artifacts)
- **One root cause, three symptoms.** `parvagues-protect`'s `ardour:80` target
puts Ardour's **GUI** thread on `SCHED_FIFO/80`. Its *audio* thread already
gets RT correctly from the backend — `AudioEngine 1`, `FIFO|RESET_ON_FORK` at
**83**, above the GUI, granted by PipeWire, needing nothing from us. The GUI
promotion buys zero.
- **Symptom 2, the crackles**, attributed by gig-log rather than guessed: in the
6 minutes after Ardour opened, `xrun_by.ardour` went **0 → 46** while
SuperCollider added 9. And it is chronic, not new — `ardour` xruns per
session: **5172** (Sep 10), **4852** (Sep 18), **1114** (Sep 20), **987**
(Sep 22). Four sessions of evidence nobody had ever read by name.
- **Symptom 3 was never a bug.** With no Ardour, WirePlumber auto-links
`SuperCollider:out_1/2` to the internal sink: d1 on the speakers, d2–d14
nowhere. The gate even printed it — "headphones: 2/28 fold links" — and then
graded the rig *as* the headphones rig, because it infers the mode from
Ardour's absence.
- Context that made it bite: `max_perf_pct=65`, governor `powersave`, EPP
`balance_power`, while gig-up's own line claimed "thermal-mode performance
holding". Third session that cap has gone unpaid.
- Observability verdict: **it worked.** `xrun_by` named the culprit in one query
and four sessions of history were already on disk. The gap was never
instrumentation, it was nobody asking the log a question.
## Sel (the shareable learning)
**A process's realtime budget is inherited from whoever launched it, so the same
binary is safe from one launcher and fatal from another.** `gnome-shell` hands
its children `RLIMIT_RTTIME=200000`; systemd hands them `unlimited`. Nothing in
the process tells you which you got. So "it works when I start it from the
terminal" is not the user being unreliable — it is a real, measurable
difference, and the bug lives in the difference.
And the shape above it: a protection whose job is to keep the sound alive was
the thing killing it, because it promoted the thread that *draws* instead of the
thread that *plays*, and the one that plays never needed it. Every guard is a
screw. This one had no nut — and it also had a blade.
## Hameçon (hook)
The log said `Killed` and `✓ gig-up done` three lines apart. Two hours before
the gig, that was the whole story: the rig's bodyguard had been shooting the
band for two weeks, and nobody had read the ballistics report already sitting on
disk.
......@@ -212,7 +212,11 @@ def touch_lines(v2cc: int, value: int,
role, who = G.CC_ROLE.get(v2cc, ("?", 0))
lab = ctx or G.label(v2cc)
if role == "family_filter":
return (f"DJF {FAMILY_NAME.get(who, '?')}", djf_readout(value), "")
# Row 3 is the bezel row (see the ergonomics note above), so it carries
# the reading you only need to GLANCE at: how far from the detent, as a
# percentage. The row that matters — the cutoff — stays second.
return (f"DJF {FAMILY_NAME.get(who, '?')}", djf_readout(value),
djf_percent(value))
if role == "family_mute":
return (f"MUTE {MUTE_NAME.get(who, '?')}", str(value), "")
title = f"d{who} {role}" if isinstance(who, int) and who else lab
......@@ -388,6 +392,27 @@ def djf_readout(value: int) -> str:
return s
def djf_percent(value: int) -> str:
"""Travel from the bypass detent, as a signed percentage of the throw.
The Hz readout answers "what is it filtering"; this answers "how far have I
pushed it", which is the question a hand on the knob actually has — Hz is
logarithmic and the detent band is 7 wide, so "HPF 2.1kHz" gives no sense of
remaining travel. Two knobs at "+50%" have moved the same distance whatever
their cutoffs say.
The two sides have different throws (0-60 below the band, 68-127 above), so
each is normalised against its OWN, and 100% means "at the end stop" on
either side rather than "at wire value 127".
"""
v = max(0, min(127, value))
if DJF_LO <= v <= DJF_HI:
return "0%"
if v < DJF_LO:
return f"-{round(100 * (DJF_LO - v) / DJF_LO)}%"
return f"+{round(100 * (v - DJF_HI) / (127 - DJF_HI))}%"
def home_lines(track: str, detail: str = "",
footer: str = "ParVagues") -> tuple[str, str, str]:
"""The stationary track HUD, rows by PLN's chair ergonomics: the name, then
......
......@@ -157,6 +157,38 @@ def test_control_label_family_filter_and_role_naming():
"labels must fit the panel")
def test_djf_percent_is_travel_from_the_detent_not_wire_value():
"""Each side is normalised against its OWN throw, so 100% means end stop.
The naive `value/127` would put the bypass detent at 48% and the bottom end
stop at 0% — two readings that both contradict the knob in the hand.
"""
for v in range(lang.DJF_LO, lang.DJF_HI + 1):
assert lang.djf_percent(v) == "0%", f"the whole detent band reads zero ({v})"
assert lang.djf_percent(0) == "-100%", "bottom end stop is a full sweep down"
assert lang.djf_percent(127) == "+100%", "top end stop is a full sweep up"
assert lang.djf_percent(60) == "-2%", "just below the band barely moved"
assert lang.djf_percent(68) == "+2%", "just above the band barely moved"
# Out-of-range wire values must not produce >100% — the OLED has 16 columns
# and a "-190%" would be a lie about a knob that cannot go there.
assert lang.djf_percent(-40) == "-100%" and lang.djf_percent(300) == "+100%"
def test_djf_overlay_fills_all_three_rows():
"""Row 3 used to be "", and PLN asked for the percent there (2026-09-24).
The bezel note above touch_lines still holds — the cutoff stays on row 2 —
but an empty third field was a row paying no rent.
"""
c1 = grid.CELL_TO_CC[("C", 1)]
title, value, pct = lang.touch_lines(c1, 100)
assert title == "DJF percs"
assert value.startswith("HPF"), f"row 2 keeps the cutoff: {value!r}"
assert pct == lang.djf_percent(100), f"row 3 is the travel: {pct!r}"
for line in (title, value, pct):
assert len(line) <= 16, f"every row must fit the panel: {line!r}"
# --------------------------------------------------------------------------- #
# parse_context: the trailing stage note beats the Haskell
# --------------------------------------------------------------------------- #
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment