# RUNBOOK — when it breaks mid-set

**Task #15.** Symptom-first, because that's how you read it in a panic: you don't know the
cause, you know *the sound stopped*. Written 2026-07-25 (J-10 to OPAL).

> **Rule zero: the room forgives silence, not flailing.** Everything below is ordered
> cheapest-first. Do not reboot SuperCollider as a first move — it costs 30–120s of dead air
> and loses every pattern. Almost nothing here needs it.

---

## 0. The 10-second triage

Something's wrong. Ask in this order:

1. **Is it only ONE stream?** → §1 (a track problem)
2. **Is it ALL sound, but Pulsar still responds?** → §2 (a routing/eval problem)
3. **Is Pulsar frozen/unresponsive?** → §3 (an editor problem — audio may be fine!)
4. **Did the laptop just wake / did the screen just do something weird?** → §4 (the display
   stack takes the audio stack down with it)

**Silence-with-a-working-editor is the most recoverable state there is.** Keep a hand on the
LCXL and keep the last working track's code in front of you.

---

## 1. One stream died

### 1a. You pushed a fader and the sound vanished — THE MUTE-BOMB
**This is the #1 historical footgun.** Old `gF1/2/3` sent values into `djfbus`, a phantom
SuperDirt effect; faders rest at 0, so gF-wrapped streams muted with no recovery.

- **Fixed permanently 2026-07-25** (commit `98d073f`) — `BootTidal.hs` now uses safe-at-rest
  `hpf`/`lpf` where fader-rest = bypass. A fresh boot is immune.
- **But if you're on a STALE boot** (an old ghci that survived, a machine that didn't pick up
  the new file): symptom is *push fader 49 → sound MUTES instead of sweeping*.
- **Rescue, no reboot:** open `live/lib/prelude.tidal`, Ctrl-Enter inside each `:{ … :}`
  block. It shadows the boot defs in the live session. Patterns keep playing.
- **Instant mitigation while you find it:** pull the fader back to 0. Rest = bypass on the
  new defs; on the old broken defs, 0 is the *mute* position, so if pulling to 0 makes it
  worse you have confirmed you're on a stale boot.

### 1b. A stream is just gone / wrong
- `d5 $ silence` then re-eval the block. With default-xfade `dN` (#13) this blends over 4
  cycles rather than cutting.
- Need it back NOW with no blend: `jumpTo 5 $ …` (orbit-correct hard cut) or `p 5 $ …`.
- Muted by a global mask? Check `gM1/2/3` — buttons **73 / 74 / 75**. Press to un-toggle.
- Panic chord stuck on? Buttons **73+74+91+92** together toggles `^93`. If `gPanic` is wired
  into that stream, `^93=1` means gain 0. Hold the chord again to release (LEDs show state).

---

## 2. All sound gone, Pulsar still works

### 2a. Check SuperDirt is actually alive — VERIFIED diagnostic
```sh
pgrep -x scsynth && ss -uln | grep 57120
```
- **Both OK** → SuperDirt is up, so it's a Tidal-side problem → §2b.
- **Nothing** → scsynth died → §5 (the only real reboot case).

### 2b. Tidal booted but nothing reaches SuperCollider — the ZOMBIE PORT
**Observed live 2026-07-25.** A leftover `ghc` process holds Tidal's control port 6010; the
new Tidal boots, fails to bind, and prints *"Can't handshake with SuperCollider without
control port"*. Pulsar looks completely normal. You eval, and nothing happens — **no error**.

```sh
ss -lunp | grep 6010          # who holds it
```
Expected healthy output — exactly one `ghc` process, and it's *your current* Tidal:
```
UNCONN 0 0 127.0.0.1:6010 0.0.0.0:* users:(("ghc-9.4.7",pid=NNNN,fd=22))
```
If that pid is a stale one, kill it (`kill NNNN`) and restart Tidal **in Pulsar only**
(Ctrl-Alt-B / Tidal: Boot) — SuperCollider is untouched, samples stay warm.

> Worth doing as part of soundcheck, not just as a fix: this failure is *silent*.

### 2c. Ardour stole the LaunchControl XL
Whoever opens the LCXL first wins it. `gig-up.sh` exists precisely to force
SuperDirt-first ordering, but if something restarted out of order:
```sh
aconnect -l | grep -iE "client [0-9]+:"
```
You want a `client 128: 'SuperCollider' [type=user,...]` present. If SuperCollider isn't
listed as a MIDI client, it never claimed the controller — knobs will do nothing while
audio still plays.

### 2d. Master path
- Ardour master fader / limiter — is the record-bus limiter clamping everything?
- `# gain` global: `^77` is the global gain fader (`midiGGlobal = "^77" * 1.3`). **If fader
  77 is down, everything is silent and nothing looks broken.** Check it early — it's a
  one-move fix that's easy to overlook.

---

## 3. Pulsar frozen or crawling

**Audio usually keeps playing — do not panic-kill it.** Tidal patterns live in the ghci
process, not the editor; a wedged renderer doesn't stop the music.

- **`Window: Reload`** (Ctrl-Alt-Ctrl-L / command palette) — reloads the renderer, keeps the
  ghci session and therefore the playing patterns. This is the cheap fix and it resolves the
  known highlighter marker-leak renderer burn.
- If the reload loses the Tidal session, re-boot Tidal in Pulsar and re-eval the current
  track from the file — SuperDirt and your samples are still warm.
- Known standing issue: the highlighter marker pool leaks and pegs a core (#7). Reload is the
  mitigation until that's fixed.

---

## 4. Post-suspend / display-stack death

**Established root cause** (memory: `project_resume_audio_crash`): on resume, kwin can abort,
and because sclang was Qt-linked it got cascade-killed along with scsynth and Ardour —
"music death" after an overnight suspend.

- **Mitigated** by headless sclang (`QT_QPA_PLATFORM=offscreen`, commit `d834667`) — SuperDirt
  no longer shares fate with the display stack.
- **Don't suspend the gig laptop between soundcheck and set.** The mitigation reduces the
  trigger; it doesn't make resume free.
- dGPU off (`./gpu-mode.sh`) further reduces the trigger.

---

## 5. Full SuperCollider restart — the LAST resort

Only when scsynth is genuinely dead (§2a found nothing).

```sh
./gig-up.sh
```
Idempotent: it converges the rig rather than spawning duplicates, brings SuperDirt up
**first**, waits for a real readiness gate (scsynth alive + `:57120` listening + a
SuperCollider MIDI client), and only then touches Ardour/Pulsar. Preloads the last 20 edited
tracks' samples so the first plays don't crack.

**Expect up to ~120s** (`READY_TIMEOUT`). That is a long time on stage. Have something to say,
or a track already playing off another source.

---

## 6. Hardware

### LCXL lights are dark but knobs still work
**Diagnosed pattern** (memory: `reference_lcxl_led_stall`): the USB **OUT** endpoint stalls
(urb -32) while IN keeps working — so input is alive and LED feedback is dead.
**Fix: physically replug the LCXL.** It re-enumerates and LEDs come back. Input bindings
survive because the CC→OSC bridge re-claims it.

### Knobs do nothing at all
Not the same failure — that's §2c (SuperCollider never claimed the controller).

---

## 7. Always-on recording — check it's still going

The set is being captured continuously to rotating opus segments, independent of Ardour.

```sh
bash tools/gig_record.sh status
```
Healthy output is `gig_record: RUNNING (pid NNNN)` plus recent segment files in
`recordings/`. A brand-new segment reading `0` bytes is **normal** — it just rotated (they
rotate hourly).

If it died mid-set: `bash tools/gig_record.sh start`. It's a separate process from Ardour
on purpose — if Ardour dies you still have the set.

---

## Pre-set checklist (the 2 minutes that prevent §2b and §2d)

1. `./gig-up.sh` — wait for the readiness gate, don't race it.
2. `ss -lunp | grep 6010` → exactly one ghc, and it's the current one.
3. `aconnect -l | grep SuperCollider` → present.
4. **Fader 77 up.** **Knobs C1/C2/C3 (the DJ filters, CC 49/50/51) at their CENTRE
   detent — centre is bypass, NOT down.** Corrected 2026-07-28: this checklist used
   to say "faders 49/50 down (= bypass)", which is now WRONG and audibly harmful.
   `gDJF` is built as `lpf` and `hpf` sections that are both wide open at ch=0.5 and
   close as you move off centre in either direction (`BootTidal.hs` ~line 358), so
   a knob at 0 is a hard low-pass — almost only subbass. Down is the opposite of
   bypass.
5. **`tools/check-mix.py`** → no orbit at `-inf` or "very low". Do this AFTER setting
   the LCXL D-row, and **save the Ardour session**, because this reads the file as
   saved, not the live desk (they have been observed disagreeing in both directions).
6. **`tools/check-tracks.sh`** → "every declared orbit of every track made sound".
   This is the compile check as well as the smoke test: a `.tidal` file cannot be
   typechecked statically, and ALL-orbits-silent means the track did not compile.
   Takes ~45 s per track, so run it the day before, not at doors.
7. `tools/lcxl-leds.py --map <the first track>` → the surface lights up, and the dark
   controls are the ones that do nothing on that track.
8. Eval one track, hear it, then `hush`.
9. `bash tools/gig_record.sh status` → RUNNING.
10. Do **not** suspend the laptop after this point.

**After ANY edit to `BootTidal.hs`, re-run step 6.** A broken helper there silences
the entire corpus at once — one nested `#` in `gDJF` took down every `g*` helper on
2026-07-27, and a single undefined name or misplaced `.` takes down a whole track
because these files have almost no blank lines, and a blank line is Tidal's block
separator. Blank lines between orbits are free insurance.

---

## 8. "I evaluated it and heard nothing" — check it COMPILED before blaming audio

The single most common cause of silence on this rig is not audio at all. Three
separate times in the week to 2026-07-28 a "dead rig" was a compile error:
an undefined name (`crackles`), a nested `#` in a `gDJF` helper, and
`|+ note 12 . (|* gain 0.6)` where `.` binds tighter than the section.

**The rule that identifies it instantly:**

| symptom | meaning |
|---|---|
| **ALL** declared orbits silent | it **did not compile** — one block, one error |
| **SOME** orbits silent | a real audio problem (fader, bank, routing) |

A Tidal block compiles entirely or not at all, and these tracks have almost no blank
lines, so the "block" is usually the whole file.

What to do:
1. Look at the Pulsar notification — it now says `EVAL ERROR (block line N, col C):`
   with the position intact. Before 2026-07-28 the editor was *deleting* the word
   "error" and the position from GHC's output, which is why this class of failure went
   unrecognised for so long. If you don't see notifications, do `Window: Reload`.
2. Bisect the way PLN does: **insert a blank line** after a working orbit. That splits
   the block, so the healthy half survives and the broken half is isolated.
3. Remember Pulsar evaluates the **editor buffer**, not the file on disk — a `grep` of
   the file can disagree with what is playing, in both directions. Save before trusting
   any file-based check.
4. `tools/check-tracks.sh <track>` for the machine's answer.

## Status of these procedures

- **§1a, §2a, §2b, §2c, §7** — commands verified by running them on 2026-07-25.
- **§4, §6** — from documented prior incidents with established root causes; the *fixes*
  (headless sclang, replug) are known-good, but not re-tested tonight.
- **§5** — `gig-up.sh` behaviour read from source, not exercised end-to-end tonight. Worth a
  real rehearsal run (#14), because a runbook you've never executed is a hypothesis.
- **§8, and pre-set steps 5-7** — added 2026-07-28 from three real incidents that week;
  `check-mix.py`, `check-tracks.sh` and `lcxl-leds.py --map` were all run against the live
  rig that day. The LED paint is the one item **not** confirmed by eye — a clean exit is
  not a lit LED.
- **Pre-set step 4 was WRONG until 2026-07-28** and said to put the DJ-filter knobs down
  for bypass. Centre is bypass; down is a hard low-pass. If you have muscle memory from
  the old line, unlearn it.
