Commit b227c7aa by PLN (Algolia)

gig: the launch that printed green had already had Ardour killed

The desktop icon brought Ardour up at 23:59 and the shell reported it Killed
ten seconds later, three lines before gig-up said 'gig-up done'. One root
cause under three complaints:

parvagues-protect's ardour:80 target promotes Ardour's GUI thread to
SCHED_FIFO/80. Its audio thread already gets RT from the backend the right
way (AudioEngine 1, FIFO|RESET_ON_FORK at 83, above the GUI), so the
promotion buys nothing and costs two things. RLIMIT_RTTIME is inherited and
differs by launcher -- gnome-shell hands children 200000us, a systemd user
unit hands them unlimited -- so an Ardour started from the icon carries a
200ms realtime budget, and loading the session blows it: SIGXCPU then
SIGKILL, with no kernel log line, which is why the journal looked clean.
And a GUI thread at 40-85% of a core on a realtime policy misses deadlines:
xrun_by.ardour went 0->46 in six minutes, and it is chronic, not new --
5172, 4852, 1114 and 987 ardour xruns in the Sep 10/18/20/22 sessions.

d1 alone on the speakers was never a second bug; it is what WirePlumber
does with SuperCollider:out_1/2 when there is no Ardour to route through.

Also: the DJF overlay's third row was an empty string, which is a row paying
no rent. It now carries travel from the bypass detent as a signed percent,
each side normalised against its own throw so 100% means end stop. The
cutoff stays on row 2 -- row 3 is the bezel row.

Findings in TODO_GIG.md with Thursday reordered; story in armada/tasks/046.
parent 8dbc068b
...@@ -541,3 +541,109 @@ control per second in corpus numbering — so "minute 23:30 the bass is too ...@@ -541,3 +541,109 @@ control per second in corpus numbering — so "minute 23:30 the bass is too
saturated" resolves to which knob, how far, how many times. That is the input saturated" resolves to which knob, how far, how many times. That is the input
for code/effect/DAW changes, not just a mastering fix. Then for code/effect/DAW changes, not just a mastering fix. Then
`docs/2026-09-25-post-gig-cleanup.md`. `docs/2026-09-25-post-gig-cleanup.md`.
## Thu 24, 00:15 — one root cause, three symptoms: pv-protect puts the Ardour GUI on a realtime policy
PLN pressed the ParVagues desktop icon at 23:59, heard **kick on the laptop
speakers**, saw **no Ardour**, and later **crackles**. Three complaints, one
mechanism. The launcher is NOT stale: the symlink resolves to
`tools/parvagues.desktop` → `tools/gig-up-window.sh` → the repo-root `gig-up.sh`,
and the log shows it doing every step in order.
**What the launch log says** (`~/.cache/parvagues/gig-up.log`):
```
• launching Ardour (ardour) via pw-jack…
• launching Pulsar (pulsar)…
gig-up.sh: line 619: 540676 Killed setsid pw-jack ardour "…/Tidal Live.ardour"
✓ gig-up done. SuperDirt owns the LaunchControl; Ardour + Pulsar are coming up.
```
Ardour was launched, lived ~10 s, and was **SIGKILLed**. Then the script said
green. `pv-protect` had touched it 2 s in:
```
23:59:55 protected: ardour[540676] oom:100->-1000 sched:SCHED_OTHER/0->FIFO/80
```
### The mechanism (measured, not guessed)
`parvagues-protect`'s third target is `ardour:80` — it puts **Ardour's GUI
thread** on `SCHED_FIFO/80`. Measured on the Ardour PLN opened by hand at 00:04
(PID 615285): the `ArdourGUI` main thread is FIFO/80, and gig-log has it burning
**40–85 % of a core** while the session sits idle. Meanwhile Ardour's audio
thread gets its own RT *correctly* from the backend — `AudioEngine 1` is
`SCHED_FIFO|SCHED_RESET_ON_FORK` at **83**, i.e. above the GUI, granted by
PipeWire, needing nothing from us. So the FIFO on the GUI thread buys zero and
costs three things:
1. **The SIGKILL.** `RLIMIT_RTTIME` is inherited, and it differs by who
launched you: `gnome-shell` runs with **200000 µs**, a systemd user unit with
**unlimited** (measured: gnome-shell 200000, pv-protect's own shell
unlimited). The desktop icon is a gnome-shell descendant, so Ardour launched
from it carries a **200 ms realtime budget** — and a GUI thread loading a
355 KB session with plugins blows 200 ms of uninterrupted CPU easily. The
kernel then sends SIGXCPU and, soft == hard, SIGKILL. **No kernel log line,
which is why the journal looked empty.** The Ardour PLN started by hand at
00:04 shows `Max realtime timeout: unlimited` and survived — the
discriminator holds.
2. **The crackles.** A FIFO/80 thread at 85 % of a core, on a CPU capped at
`max_perf_pct=65` with governor `powersave`, misses deadlines. gig-log's
attribution is unambiguous: in the 6 minutes after Ardour opened,
`xrun_by.ardour` went **0 → 46** while SuperCollider added ~9. And it is
**chronic, not new** — `ardour` xruns per session: 5172 (Sep 10), 4852
(Sep 18), 1114 (Sep 20), 987 (Sep 22). Nobody had ever attributed them.
3. **d1 straight to the speakers.** Not a second bug — the consequence. With no
Ardour, WirePlumber auto-links `SuperCollider:out_1/2` to the internal sink
and d2–d14 reach nothing. The gate said exactly that ("headphones: 2/28 fold
links present") and then *graded the rig as the headphones rig*, because it
infers the mode from Ardour's absence. With Ardour back up the graph is
correct again: 61 orbit links into `ardour:Tidal NN`, `ardour:Master` → the
sinks, no direct SC → speaker link left over.
### Thu morning — in this order
- [ ] **1. Make Ardour survive its own launch.** Blocks every Ardour-side item
below. Two candidate fixes, both one line, PLN's call (the file is
root-owned — `sudoedit /usr/local/bin/parvagues-protect`, then
`sudo systemctl restart parvagues-protect`):
- **(a) Drop the FIFO, keep the OOM shield.** `ardour:80` → protect
`oom_score_adj=-1000` only. Justification: `AudioEngine 1` already runs
FIFO/83 from the backend, so the GUI promotion protects nothing that
makes sound. This also fixes symptom 2. **Recommended.**
- **(b) Lift the budget before promoting.** `prlimit --pid $pid
--rttime=unlimited` before the `chrt`, exactly the shape the script
already uses for `memlock_unlimited`. Fixes symptom 1 only; leaves an
85 %-of-a-core GUI thread at FIFO/80 in the graph.
AC for either: launch from the **desktop icon** (the path that failed —
launching from a terminal or the Bridge does not reproduce it), and
Ardour is still alive 60 s later with `xrun_by.ardour` flat.
Check whether `perf-audio` does the same `chrt` on Ardour; if so it needs
the same edit or it will re-arm the bug on the next perf pass.
- [ ] **2. The CPU cap, still unpaid** (third session running). Measured now:
`max_perf_pct=65`, governor `powersave`, EPP `balance_power`, and gig-up's
own line claimed "thermal-mode performance holding" while the gate read
the opposite — that perf line is **stale and must not be trusted**.
`echo 100 | sudo tee /sys/devices/system/cpu/intel_pstate/max_perf_pct`
plus `sudo cpupower set --epp performance`. Until this is paid, no xrun
number from the rehearsal means anything.
- [ ] **3. `systemctl --user restart parvagues-sc` at the START of the
rehearsal** — `s.latency` 0.1 is still only in the file; the running
process holds 0.2. AC after 10 min of play:
`journalctl --user -u parvagues-sc -b | grep -cE '\blate [0-9]'` == 0.
- [ ] **4. Master fader, then Ctrl+S.** `check-mix` reads the SAVED session and
`use-osc=0`, so nothing outside Ardour can see the live desk. Enable
Ardour OSC while you are in there and the blind spot closes for good.
- [ ] **5. Quantum 512 listen test**, only after items 1 and 2.
- [ ] **6. Full-set run**, late-count + xrun attribution from gig-log.
### The gate bug behind the false green (post-gig, write it down now)
In stage mode, an Ardour that **gig-up itself just spawned** and that is dead at
gate time must **FAIL**, not `SKIP`. Tonight the gate reasoned "Ardour is not
open → this must be the headphones rig" and graded a mode PLN was not playing,
then `gig-up.sh` printed `✓ gig-up done … Ardour + Pulsar are coming up` about a
process it had already been told was `Killed`. The launcher knows the PID and
knows whether the spawn survived; it must say so. "A clean exit is not a launch"
is already written in `gig-up-window.sh` — this is the same lesson one layer in:
**a spawn is not a process.**
---
log: 046
title: "The bodyguard was the assassin"
date: 2026-09-24
task: "n/a — gig-eve rig triage (TODO_GIG.md)"
tags: [rig, audio, realtime, observability, gig]
shareable: true
---
## Cap (what & why)
Midnight, gig day. PLN presses the ParVagues icon and reports three things at
once: kick coming out of the **laptop speakers**, **no Ardour on screen**, and a
minute later, **crackles**. The launcher's own log ends with a green tick. Three
complaints, one hour before the only rehearsal the set will ever get.
## Manœuvre (how)
Read the launch log before touching anything. It said, in order: launching
Ardour, launching Pulsar, then —
```
gig-up.sh: line 619: 540676 Killed setsid pw-jack ardour "…/Tidal Live.ardour"
✓ gig-up done. SuperDirt owns the LaunchControl; Ardour + Pulsar are coming up.
```
A SIGKILL and a green tick, three lines apart. The journal had no OOM, no
segfault, nothing — which is itself the clue: exactly one kill mechanism leaves
no kernel line. Two seconds before the death, `parvagues-protect` had logged
`protected: ardour[540676] sched:SCHED_OTHER/0->FIFO/80`.
Then the discriminator, measured on the Ardour PLN opened by hand afterwards:
`RLIMIT_RTTIME` is **inherited**, and it differs by parentage — `gnome-shell`
runs at **200 ms**, a systemd user unit at **unlimited**. The desktop icon is a
gnome-shell descendant. So: promote the GUI thread to a realtime policy, hand it
a 200 ms realtime budget, then ask it to load a 355 KB session with plugins.
SIGXCPU, then SIGKILL, silently.
## Prise (findings / artifacts)
- **One root cause, three symptoms.** `parvagues-protect`'s `ardour:80` target
puts Ardour's **GUI** thread on `SCHED_FIFO/80`. Its *audio* thread already
gets RT correctly from the backend — `AudioEngine 1`, `FIFO|RESET_ON_FORK` at
**83**, above the GUI, granted by PipeWire, needing nothing from us. The GUI
promotion buys zero.
- **Symptom 2, the crackles**, attributed by gig-log rather than guessed: in the
6 minutes after Ardour opened, `xrun_by.ardour` went **0 → 46** while
SuperCollider added 9. And it is chronic, not new — `ardour` xruns per
session: **5172** (Sep 10), **4852** (Sep 18), **1114** (Sep 20), **987**
(Sep 22). Four sessions of evidence nobody had ever read by name.
- **Symptom 3 was never a bug.** With no Ardour, WirePlumber auto-links
`SuperCollider:out_1/2` to the internal sink: d1 on the speakers, d2–d14
nowhere. The gate even printed it — "headphones: 2/28 fold links" — and then
graded the rig *as* the headphones rig, because it infers the mode from
Ardour's absence.
- Context that made it bite: `max_perf_pct=65`, governor `powersave`, EPP
`balance_power`, while gig-up's own line claimed "thermal-mode performance
holding". Third session that cap has gone unpaid.
- Observability verdict: **it worked.** `xrun_by` named the culprit in one query
and four sessions of history were already on disk. The gap was never
instrumentation, it was nobody asking the log a question.
## Sel (the shareable learning)
**A process's realtime budget is inherited from whoever launched it, so the same
binary is safe from one launcher and fatal from another.** `gnome-shell` hands
its children `RLIMIT_RTTIME=200000`; systemd hands them `unlimited`. Nothing in
the process tells you which you got. So "it works when I start it from the
terminal" is not the user being unreliable — it is a real, measurable
difference, and the bug lives in the difference.
And the shape above it: a protection whose job is to keep the sound alive was
the thing killing it, because it promoted the thread that *draws* instead of the
thread that *plays*, and the one that plays never needed it. Every guard is a
screw. This one had no nut — and it also had a blade.
## Hameçon (hook)
The log said `Killed` and `✓ gig-up done` three lines apart. Two hours before
the gig, that was the whole story: the rig's bodyguard had been shooting the
band for two weeks, and nobody had read the ballistics report already sitting on
disk.
...@@ -212,7 +212,11 @@ def touch_lines(v2cc: int, value: int, ...@@ -212,7 +212,11 @@ def touch_lines(v2cc: int, value: int,
role, who = G.CC_ROLE.get(v2cc, ("?", 0)) role, who = G.CC_ROLE.get(v2cc, ("?", 0))
lab = ctx or G.label(v2cc) lab = ctx or G.label(v2cc)
if role == "family_filter": if role == "family_filter":
return (f"DJF {FAMILY_NAME.get(who, '?')}", djf_readout(value), "") # Row 3 is the bezel row (see the ergonomics note above), so it carries
# the reading you only need to GLANCE at: how far from the detent, as a
# percentage. The row that matters — the cutoff — stays second.
return (f"DJF {FAMILY_NAME.get(who, '?')}", djf_readout(value),
djf_percent(value))
if role == "family_mute": if role == "family_mute":
return (f"MUTE {MUTE_NAME.get(who, '?')}", str(value), "") return (f"MUTE {MUTE_NAME.get(who, '?')}", str(value), "")
title = f"d{who} {role}" if isinstance(who, int) and who else lab title = f"d{who} {role}" if isinstance(who, int) and who else lab
...@@ -388,6 +392,27 @@ def djf_readout(value: int) -> str: ...@@ -388,6 +392,27 @@ def djf_readout(value: int) -> str:
return s return s
def djf_percent(value: int) -> str:
"""Travel from the bypass detent, as a signed percentage of the throw.
The Hz readout answers "what is it filtering"; this answers "how far have I
pushed it", which is the question a hand on the knob actually has — Hz is
logarithmic and the detent band is 7 wide, so "HPF 2.1kHz" gives no sense of
remaining travel. Two knobs at "+50%" have moved the same distance whatever
their cutoffs say.
The two sides have different throws (0-60 below the band, 68-127 above), so
each is normalised against its OWN, and 100% means "at the end stop" on
either side rather than "at wire value 127".
"""
v = max(0, min(127, value))
if DJF_LO <= v <= DJF_HI:
return "0%"
if v < DJF_LO:
return f"-{round(100 * (DJF_LO - v) / DJF_LO)}%"
return f"+{round(100 * (v - DJF_HI) / (127 - DJF_HI))}%"
def home_lines(track: str, detail: str = "", def home_lines(track: str, detail: str = "",
footer: str = "ParVagues") -> tuple[str, str, str]: footer: str = "ParVagues") -> tuple[str, str, str]:
"""The stationary track HUD, rows by PLN's chair ergonomics: the name, then """The stationary track HUD, rows by PLN's chair ergonomics: the name, then
......
...@@ -157,6 +157,38 @@ def test_control_label_family_filter_and_role_naming(): ...@@ -157,6 +157,38 @@ def test_control_label_family_filter_and_role_naming():
"labels must fit the panel") "labels must fit the panel")
def test_djf_percent_is_travel_from_the_detent_not_wire_value():
"""Each side is normalised against its OWN throw, so 100% means end stop.
The naive `value/127` would put the bypass detent at 48% and the bottom end
stop at 0% — two readings that both contradict the knob in the hand.
"""
for v in range(lang.DJF_LO, lang.DJF_HI + 1):
assert lang.djf_percent(v) == "0%", f"the whole detent band reads zero ({v})"
assert lang.djf_percent(0) == "-100%", "bottom end stop is a full sweep down"
assert lang.djf_percent(127) == "+100%", "top end stop is a full sweep up"
assert lang.djf_percent(60) == "-2%", "just below the band barely moved"
assert lang.djf_percent(68) == "+2%", "just above the band barely moved"
# Out-of-range wire values must not produce >100% — the OLED has 16 columns
# and a "-190%" would be a lie about a knob that cannot go there.
assert lang.djf_percent(-40) == "-100%" and lang.djf_percent(300) == "+100%"
def test_djf_overlay_fills_all_three_rows():
"""Row 3 used to be "", and PLN asked for the percent there (2026-09-24).
The bezel note above touch_lines still holds — the cutoff stays on row 2 —
but an empty third field was a row paying no rent.
"""
c1 = grid.CELL_TO_CC[("C", 1)]
title, value, pct = lang.touch_lines(c1, 100)
assert title == "DJF percs"
assert value.startswith("HPF"), f"row 2 keeps the cutoff: {value!r}"
assert pct == lang.djf_percent(100), f"row 3 is the travel: {pct!r}"
for line in (title, value, pct):
assert len(line) <= 16, f"every row must fit the panel: {line!r}"
# --------------------------------------------------------------------------- # # --------------------------------------------------------------------------- #
# parse_context: the trailing stage note beats the Haskell # parse_context: the trailing stage note beats the Haskell
# --------------------------------------------------------------------------- # # --------------------------------------------------------------------------- #
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment