Commit 774087a0 by PLN (Algolia)

feat(tools): gig-preflight — assert the rig is ready BEFORE you play

The rig is heavily instrumented and nothing ever looked. gig-log has been
sampling freq, per-core cpu, throttle deltas, fan rpm, per-process cpu/RSS and
xruns at 1 Hz for weeks (512k samples in the current file), yet every failure
this session was discovered by feel, mid-session, at the worst moment. This is
the looking.

Every check exists because it silently cost a real session on 2026-09-04/05:

  * Composed an hour at 800 MHz because the Bridge's perf-watch reasserted
    `silent` every 30 s, undoing each fix within half a minute. Cost 2671
    xruns/hour. Every throttle counter read zero — with HWP, epp=power makes the
    CPU *choose* its floor, so the failure has no error message.
      -> check_perf, check_regime_agreement
  * `gluck` refused to load for 90 minutes. The symlink was perfect; sclang had
    been up 6 days and loadSoundFiles scans folder NAMES once, at startup, so a
    bank added later is invisible. Pulsar's "restart SuperDirt" button is a no-op
    here (SuperDirt is a systemd service, not Pulsar's child) and honestly
    reported 142 h uptime while looking fine.
      -> check_superdirt_fresh
  * Samples/crutch was a symlink pointing at ITSELF since Oct 2025. Dead bank,
    no complaint from anything.
      -> check_sample_integrity
  * A new LCXL3 showed default LEDs because the resolver name-matches the mk2
    product string, while the watcher logged "-> 27 controls" and looked healthy.
      -> check_midi_surface

Three design rules, all learned the hard way:

  1. Green is not evidence. `active`, exit 0 and silence all lied during this
     session, so every check reads the LIVE value from sysfs/procfs/the device
     and prints the number it saw. No check trusts a status.
  2. Every FAIL carries its fix as a runnable command. A preflight that reports
     a problem without the remedy just adds a step between you and playing.
  3. It never mutates the rig. Read-only by construction — a preflight that
     writes is one more un-arbitrated writer, the exact bug class it detects.

16 checks. First run found 1 fail + 3 warns, including a genuine xrun rate of
2224/hour with every CPU line green — a problem we had not noticed and which the
clock fix does not explain.

Known over-sensitivity: check_xruns uses the raw total, and two of the top three
sources (input.hydra_in, the built-in Speaker sink) may not be in the live
monitoring path at all. Xruns on an idle sink inflate the count without being
audible; it should weight by nodes actually in the path.

  gig-preflight.py            human report, exit 1 on any FAIL
  gig-preflight.py --quiet    only WARN/FAIL, for gig-up.sh
  gig-preflight.py --json     for the tray/Bridge
parent f7d38407
#!/usr/bin/env python3
"""gig-preflight — assert the rig is actually ready, BEFORE you play.
Every check here exists because it silently cost a real session. The rig is
heavily instrumented (gig-log samples freq, cpu, xruns, thermals at 1 Hz) but
nothing ever *looked*, so failures were discovered by feel — mid-set, at the
worst possible moment. This is the looking.
The 2026-09-05 session, which is why this file exists:
* Composed for an hour at **800 MHz** because the Bridge's perf-watch was
reasserting `silent` every 30 s. Cost: 2671 xruns/hour. Nothing warned.
Every number needed to catch it was already in gig-log. -> check_perf,
check_regime_agreement
* `gluck` would not load for 90 minutes. The symlink was perfect; sclang had
been up for 6 days and its sample dictionary predated the folder. The GUI's
"restart SuperDirt" is a no-op (SuperDirt is a systemd service, not Pulsar's
child), so it reported 142 h uptime and looked fine. -> check_superdirt_fresh
* `Samples/crutch` was a symlink pointing at ITSELF since Oct 2025. That bank
could never load, and nothing said so. -> check_sample_integrity
* A new LCXL3 showed default LEDs. Not autostart, not wiring: the resolver
matches the name "Launch Control XL" and the mk3 enumerates as "LCXL3 1",
so the watcher painted into the void while logging "-> 27 controls" and
looking healthy. -> check_midi_surface
DESIGN — three rules, learned the hard way:
1. **Green is not evidence.** `active`, exit 0 and silence all lied during that
session. So every check reads the LIVE value it cares about (sysfs, procfs,
the device itself) and prints the number it saw. No check trusts a status.
2. **Every FAIL carries its fix**, as a command you can run. A preflight that
tells you something is wrong without telling you what to do just adds a
step between you and playing.
3. **It never changes anything.** Read-only by construction. A preflight that
mutates the rig is one more un-arbitrated writer, which is the exact bug
class it exists to catch.
Usage:
gig-preflight.py # human report, exits 1 if any FAIL
gig-preflight.py --quiet # only WARN/FAIL lines (for gig-up.sh)
gig-preflight.py --json # machine-readable, for the tray/Bridge
"""
from __future__ import annotations
import argparse
import glob
import json
import os
import re
import subprocess
import sys
import time
from pathlib import Path
HOME = Path.home()
SAMPLE_ROOTS = [
HOME / ".local/share/SuperCollider/downloaded-quarks/Dirt-Samples",
HOME / "Work/Sound/Samples/extra",
HOME / "Work/Sound/Samples",
]
GIG_LOG_DIR = Path(os.environ.get(
"GIG_LOG_DIR", HOME / ".local/share/parvagues/gig-log"))
PERF_DESIRED = Path(os.environ.get(
"PERF_DESIRED_FILE", HOME / ".cache/parvagues/perf-desired"))
THERMAL_CONF = Path("/etc/thermal-mode.conf")
OK, WARN, FAIL = "OK", "WARN", "FAIL"
results: list[dict] = []
def add(name: str, state: str, detail: str, fix: str = "") -> None:
results.append({"check": name, "state": state, "detail": detail, "fix": fix})
def read(path, default=None):
try:
return Path(path).read_text().strip()
except OSError:
return default
def read_int(path, default=None):
v = read(path)
try:
return int(v)
except (TypeError, ValueError):
return default
def run(cmd: list[str], timeout=5) -> str:
try:
return subprocess.run(cmd, capture_output=True, text=True,
timeout=timeout).stdout
except Exception:
return ""
# --------------------------------------------------------------------------- #
# 1. the perf regime — the one that cost 2671 xruns/hour
# --------------------------------------------------------------------------- #
def check_perf() -> None:
"""Clock ceiling, power bias and live frequency, read from sysfs.
EPP is the subtle one. With HWP, `epp=power` makes the CPU *choose* its
800 MHz floor under load while every throttle counter stays at zero and
nothing looks wrong — the failure has no error message. It is checked
first because it is the one that hurts most and shows least.
"""
cpu = "/sys/devices/system/cpu"
gov = read(f"{cpu}/cpu0/cpufreq/scaling_governor", "?")
epp = read(f"{cpu}/cpu0/cpufreq/energy_performance_preference", "?")
maxp = read_int(f"{cpu}/intel_pstate/max_perf_pct", 100)
cur = read_int(f"{cpu}/cpu0/cpufreq/scaling_cur_freq", 0) // 1000
ceil = read_int(f"{cpu}/cpu0/cpufreq/scaling_max_freq", 0) // 1000
if epp in ("power", "balance_power"):
add("cpu energy bias", FAIL, f"epp={epp} — HWP will sit at the floor",
"sudo cpupower set --epp performance # or: bridge.py perf standard")
else:
add("cpu energy bias", OK, f"epp={epp}")
if gov == "powersave":
add("cpu governor", WARN, f"gov={gov}",
"cd ~/Work/Sound/Tidal/tools/bridge && python3 bridge.py perf standard")
else:
add("cpu governor", OK, f"gov={gov}")
if maxp is not None and maxp < 100:
eff = int(ceil * maxp / 100) if ceil else 0
add("clock ceiling", FAIL,
f"max_perf_pct={maxp}% — capped at ~{eff}MHz of {ceil}MHz",
"sudo sh -c 'echo 100 > /sys/devices/system/cpu/intel_pstate/max_perf_pct'")
else:
add("clock ceiling", OK, f"max_perf_pct={maxp}% of {ceil}MHz")
# A live freq pinned at the floor while the ceiling is high is the exact
# fingerprint of the silent-mode reassert. Report it as its own line so the
# symptom and the cause are never confused again.
floor = read_int(f"{cpu}/cpu0/cpufreq/scaling_min_freq", 0) // 1000
if cur and floor and cur <= floor * 1.15:
add("live frequency", WARN,
f"{cur}MHz — at the {floor}MHz floor (idle, or something is capping)")
else:
add("live frequency", OK, f"{cur}MHz")
for z in sorted(glob.glob("/sys/class/powercap/intel-rapl:[0-9]")):
if read(f"{z}/name") != "package-0":
continue
pl1 = read_int(f"{z}/constraint_0_power_limit_uw")
if pl1:
w = pl1 // 1_000_000
state = FAIL if w < 20 else (WARN if w < 35 else OK)
add("package power (PL1)", state, f"{w}W",
"" if state == OK else
"sudo /usr/local/sbin/thermal-mode-apply aggressive")
def check_regime_agreement() -> None:
"""Two independent stores decide the rig's power state. They can disagree.
`/etc/thermal-mode.conf` (GEAR/TRIM, written by thermal-mode-apply) and
`~/.cache/parvagues/perf-desired` (written by bridge/tray, reasserted every
30 s by the Bridge's perf-watch). On 2026-09-05 the first said
`GEAR=aggressive` while the second said `silent`, and the reasserter won —
silently undoing every fix within 30 seconds. Divergence is the bug.
"""
desired = read(PERF_DESIRED)
conf = read(THERMAL_CONF, "") or ""
gear = next((l.split("=", 1)[1] for l in conf.splitlines()
if l.startswith("GEAR=")), None)
if desired is None:
add("perf regime", WARN, "no desired mode — perf-watch maintenance is off")
return
quiet = {"silent", "cool"}
fast = {"standard", "extreme"}
if desired in quiet:
add("perf regime", FAIL,
f"desired={desired!r} — the Bridge will reassert this every 30s",
"cd ~/Work/Sound/Tidal/tools/bridge && python3 bridge.py perf standard")
elif gear and desired in fast and gear in quiet:
add("perf regime", WARN,
f"stores disagree: thermal GEAR={gear!r} vs desired={desired!r}")
else:
add("perf regime", OK, f"desired={desired}" + (f", gear={gear}" if gear else ""))
# --------------------------------------------------------------------------- #
# 2. the audio chain
# --------------------------------------------------------------------------- #
def _pid_of(name: str) -> str | None:
out = run(["pgrep", "-x", name]).split()
return out[0] if out else None
def check_audio_rt() -> None:
"""SCHED_FIFO on the audio chain.
This matters more than any nice value: a realtime policy preempts EVERY
ordinary process, which is why the editor can safely run at nice -5 without
stealing a cycle from SuperDirt. If the chain has fallen back to
SCHED_OTHER, xruns follow under any desktop load.
"""
for proc in ("scsynth", "sclang", "pipewire"):
pid = _pid_of(proc)
if not pid:
add(f"{proc} running", FAIL, "not found",
"systemctl --user restart parvagues-sc.service")
continue
pol = run(["chrt", "-p", pid])
m = re.search(r"policy:\s*(\S+)", pol)
policy = m.group(1) if m else "?"
if "FIFO" in policy or "RR" in policy:
prio = re.search(r"priority:\s*(\d+)", pol)
add(f"{proc} realtime", OK,
f"{policy} prio {prio.group(1) if prio else '?'}")
else:
add(f"{proc} realtime", WARN, f"policy={policy} (not realtime)",
"sudo /usr/local/sbin/perf-audio --optimize")
def check_superdirt_fresh() -> None:
"""Is sclang's sample dictionary older than the samples on disk?
`loadSoundFiles` scans folder NAMES once, at sclang start. Lazy loading
defers reading the files, not discovering them — so a bank added later is
invisible until sclang restarts, and the only symptom is
"no synth or sample named 'x'". This is the gluck bug, and it is impossible
to guess from inside Pulsar because the GUI's restart button cannot restart
a systemd-owned sclang (it reported 142 h uptime and was telling the truth).
"""
pid = _pid_of("sclang")
if not pid:
add("superdirt dictionary", FAIL, "sclang not running",
"systemctl --user restart parvagues-sc.service")
return
try:
started = Path(f"/proc/{pid}").stat().st_mtime
except OSError:
add("superdirt dictionary", WARN, "cannot read sclang start time")
return
newest, newest_age = 0.0, None
for root in SAMPLE_ROOTS:
if not root.is_dir():
continue
try:
for entry in os.scandir(root):
st = entry.stat(follow_symlinks=False)
if st.st_mtime > newest:
newest = st.st_mtime
except OSError:
continue
age_h = (time.time() - started) / 3600
if newest > started:
newest_age = (newest - started) / 3600
add("superdirt dictionary", FAIL,
f"sclang up {age_h:.1f}h; a sample folder changed {newest_age:.1f}h "
f"AFTER it started — those banks will report 'not found'",
"systemctl --user restart parvagues-sc.service # NOT the GUI button")
else:
add("superdirt dictionary", OK, f"current (sclang up {age_h:.1f}h)")
def check_sample_integrity() -> None:
"""Broken symlinks in the sample roots — counts only, never names.
Per this estate's PII contract the assistant sees counts and class names;
anything filename-bearing goes to a file the human opens. `crutch -> crutch`
(self-referential, unused, dead since Oct 2025) is why this check exists.
"""
broken = 0
for root in SAMPLE_ROOTS:
if not root.is_dir():
continue
try:
for entry in os.scandir(root):
if entry.is_symlink() and not entry.is_dir(follow_symlinks=True) \
and not entry.is_file(follow_symlinks=True):
broken += 1
except OSError:
continue
if broken:
add("sample symlinks", WARN, f"{broken} broken",
"find ~/Work/Sound/Samples -maxdepth 2 -xtype l # inspect, then unlink")
else:
add("sample symlinks", OK, "none broken")
# --------------------------------------------------------------------------- #
# 3. the control surface
# --------------------------------------------------------------------------- #
def check_midi_surface() -> None:
"""Is a surface present, and does our LED tooling speak its dialect?
The generation matters, not just the presence. mk2 ("Launch Control XL")
takes bicolor SysEx `F0 00 20 29 02 11 78 ...`; mk3 ("LCXL3 1") takes RGB
`F0 00 20 29 02 15 01 53 <idx> <R> <G> <B> F7`. A mk3 on mk2 tooling shows
factory-default LEDs while the watcher cheerfully logs "-> 27 controls".
"""
clients = run(["aconnect", "-l"])
if not clients:
add("midi surface", WARN, "aconnect unavailable")
return
names = re.findall(r"^client \d+: '([^']*)'", clients, re.M)
surface = next((n for n in names if re.search(r"launch\s*control\s*xl|lcxl",
n, re.I)), None)
if not surface:
add("midi surface", WARN, "no Launch Control XL / LCXL client present")
return
mk3 = bool(re.search(r"lcxl\s*3|xl\s*3", surface, re.I))
if mk3:
add("midi surface", WARN,
f"{surface!r} is mk3 (RGB, device id 0x15, cmd 01 53); "
f"lcxl-leds.py sends mk2 (0x11/0x78, bicolor) — LEDs will stay default",
"mk3 dialect support is unimplemented; see TODO.d")
else:
add("midi surface", OK, f"{surface!r} (mk2 dialect matches tooling)")
# Wiring: the surface must reach Midi Through, which is what feeds the
# aseqdump taps and SuperCollider.
if re.search(r"Connecting To:.*\b14:0", clients):
add("midi wiring", OK, "surface -> Midi Through connected")
else:
add("midi wiring", WARN, "surface not wired to Midi Through",
"systemctl --user restart midi-autoconnect.service")
# --------------------------------------------------------------------------- #
# 4. the editor, and what else is eating the machine
# --------------------------------------------------------------------------- #
def check_editor() -> None:
"""Pulsar's scheduling priority and the highlighter's state.
The editor's renderer is a single JS thread, so its responsiveness is set by
single-thread clock and by whether it wins the CPU promptly — not by core
count (the box sat 91% idle while feeling unusable).
"""
pids = run(["pgrep", "-x", "pulsar"]).split()
if not pids:
add("editor", WARN, "pulsar not running")
return
nices = []
for pid in pids:
stat = read(f"/proc/{pid}/stat")
if stat:
try:
nices.append(int(stat.rsplit(")", 1)[1].split()[16]))
except (IndexError, ValueError):
pass
if nices:
worst = max(nices)
if worst > 0:
add("editor priority", WARN, f"nice={worst} (deprioritized)",
"sudo install -m 755 ~/Work/Sound/Tidal/perf.sh /usr/local/sbin/perf-audio"
" # then: bridge.py perf standard")
else:
add("editor priority", OK, f"nice={worst}")
cfg = read(HOME / ".pulsar/config.cson", "") or ""
m = re.search(r"eventHighlighting:\s*\n\s*enable:\s*(\w+)", cfg)
if m:
add("event highlighting", OK, f"enable={m.group(1)}")
def check_background_load() -> None:
"""Services that compete for the machine during a session.
Not failures — choices. But an idle database and a container runtime held
RAM and CPU through a whole session nobody meant to share.
"""
noisy = []
for unit in ("mariadb.service", "docker.service", "ollama.service",
"gitlab-runner.service"):
scope = ["--user"] if unit == "gitlab-runner.service" else []
state = run(["systemctl", *scope, "is-active", unit]).strip()
if state == "active":
noisy.append(unit.replace(".service", ""))
if noisy:
add("background load", WARN, f"active: {', '.join(noisy)}",
"sudo systemctl stop " + " ".join(n + ".service" for n in noisy))
else:
add("background load", OK, "no competing services")
def check_xruns() -> None:
"""Recent xrun rate, straight from gig-log's own samples.
The number that made the 800 MHz session obvious in hindsight: 2671/hour
while capped, 313/hour after. If this is climbing, something above is wrong.
"""
files = sorted(glob.glob(str(GIG_LOG_DIR / "gig-*.jsonl")),
key=os.path.getmtime)
if not files:
add("xrun rate", WARN, "no gig-log data")
return
tail = []
try:
with open(files[-1], errors="replace") as fh:
for line in fh:
if '"k":"s"' in line:
tail.append(line)
if len(tail) > 600:
tail.pop(0)
except OSError:
add("xrun rate", WARN, "gig-log unreadable")
return
pts = []
for line in tail:
try:
d = json.loads(line)
except Exception:
continue
if d.get("xrun") is not None and d.get("t"):
pts.append((d["t"], d["xrun"]))
if len(pts) < 2:
add("xrun rate", WARN, "not enough samples")
return
dt = pts[-1][0] - pts[0][0]
dx = pts[-1][1] - pts[0][1]
rate = (dx / dt * 3600) if dt > 0 else 0
state = OK if rate < 60 else (WARN if rate < 600 else FAIL)
add("xrun rate", state, f"{rate:.0f}/hour over the last {dt/60:.0f} min",
"" if state == OK else "check the perf lines above first — they cause this")
# --------------------------------------------------------------------------- #
CHECKS = (check_perf, check_regime_agreement, check_audio_rt,
check_superdirt_fresh, check_sample_integrity, check_midi_surface,
check_editor, check_background_load, check_xruns)
COLOR = {OK: "\033[32m", WARN: "\033[33m", FAIL: "\033[31m"}
RESET = "\033[0m"
def main(argv=None) -> int:
ap = argparse.ArgumentParser(prog="gig-preflight", description=__doc__.split("\n")[0])
ap.add_argument("--json", action="store_true", help="machine-readable output")
ap.add_argument("-q", "--quiet", action="store_true", help="only WARN/FAIL")
a = ap.parse_args(argv)
for check in CHECKS:
try:
check()
except Exception as e: # a broken check must never block a session
add(check.__name__, WARN, f"check itself failed: {e}")
if a.json:
print(json.dumps(results, indent=2))
else:
tty = sys.stdout.isatty()
width = max(len(r["check"]) for r in results)
for r in results:
if a.quiet and r["state"] == OK:
continue
tag = r["state"]
if tty:
tag = f"{COLOR[r['state']]}{r['state']:<4}{RESET}"
else:
tag = f"{r['state']:<4}"
print(f"{tag} {r['check']:<{width}} {r['detail']}")
if r["fix"] and r["state"] != OK:
print(f" {'':<{width}} -> {r['fix']}")
n_fail = sum(1 for r in results if r["state"] == FAIL)
n_warn = sum(1 for r in results if r["state"] == WARN)
if not a.quiet or n_fail or n_warn:
print(f"\n{len(results)} checks: {n_fail} fail, {n_warn} warn, "
f"{len(results) - n_fail - n_warn} ok")
return 1 if any(r["state"] == FAIL for r in results) else 0
if __name__ == "__main__":
sys.exit(main())
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment