Commit e7961f57 by PLN (Algolia)

fix(lcxl): the tap went deaf for six hours and every signal stayed green

PLN, mid-session: "why atm i see no midi color on lcxl reacting to my touches?"

The LED watcher was enabled, active, 6h uptime, re-parsing the loaded track
into the journal every few minutes. `systemctl status` was green. And it had
decoded exactly zero MIDI events the entire time, because its aseqdump child
was bound to `20:0` while the LCXL had moved to `28:0`.

WHY THE EXISTING DEFENCE DIDN'T FIRE

cmd_watch already re-resolved the port by name on every respawn -- the comment
on that line even says so, and it is true. It is also useless, because respawn
is triggered by the child EXITING, and an aseqdump whose sequencer client has
vanished does not exit. It blocks on a dead subscription indefinitely. The
read loop never ends, so the re-resolution is never reached.

A re-resolution that is never reached is indistinguishable from a correct one.
Same shape as the two lenses that shipped with thresholds that could never
fire: the code was right, the trigger was wrong.

THE FIX

Detect the drift from OUTSIDE the read loop. A 10s poll compares the live
name resolution against what the current child is actually bound to; on a
change it invalidates the send-side port cache and terminates the child, and
the existing main loop respawns and re-resolves through the path it already
had. No new resolution logic -- find_seq_port() was correct all along and
returns 28:0 today. Only the re-check was missing.

Deliberately NOT keyed on "no events for N seconds". A board nobody is
touching is also silent, so that check would fire through every quiet passage
and respawn the tap mid-set. Port identity changes when the binding is
genuinely stale and at no other time.

VALIDATION

  find_seq_port() -> 28:0                   (resolution was never the bug)
  before restart:  aseqdump -p 20:0         6h stale, 0 events decoded
  after restart:   aseqdump -p 28:0         re-parsed rose_rouge -> 26 controls

The restart also reaped its own orphan. Two unrelated zombie taps remain
(6.4h on the dead 20:0, 7.7h with no -p at all) -- that is #129/#155, logged
separately, not addressed here.

NOT PROVEN: the drift watchdog itself has not fired yet, because that needs
the port to actually move. Unplug and replug the LCXL and the journal should
show "LCXL moved 28:0 -> NN:0; respawning the tap". Until someone sees that
line, this fix is reasoned, not demonstrated.
parent f584a1c2
...@@ -1103,6 +1103,45 @@ def cmd_watch(s: Sender, track: str | None, reassert: float = 30.0, ...@@ -1103,6 +1103,45 @@ def cmd_watch(s: Sender, track: str | None, reassert: float = 30.0,
if follow: if follow:
threading.Thread(target=follow_loop, daemon=True).start() threading.Thread(target=follow_loop, daemon=True).start()
# The port can move UNDER a healthy child. `find_seq_port()` below is only
# reached on respawn, and respawn only happens when aseqdump EXITS — but an
# aseqdump whose client has vanished does NOT exit. It blocks on a dead
# subscription forever, decoding nothing, while `systemctl status` reports the
# unit active (running) and this loop reports nothing at all. Measured
# 2026-08-14: the daemon sat on `aseqdump -p 20:0` for six hours after the LCXL
# had moved to 28:0 — board dark, PLN asking why, every other signal green. The
# "re-resolved on every respawn" comment below was true and useless: a
# re-resolution that is never reached is indistinguishable from a correct one.
#
# So the drift has to be seen from OUTSIDE the read loop. Poll the name
# resolution on a slow timer and kill the child when the answer changes; the
# main loop then re-resolves by itself, through the path it already had.
#
# Deliberately NOT "no events for N seconds". A board nobody is touching is
# also silent, so that check would fire through every quiet passage and
# respawn the tap mid-set. Port identity is level-independent: it changes when
# the binding is genuinely stale and at no other time.
PORT_DRIFT_POLL = 10.0
child: dict = {"proc": None, "port": None}
def port_drift_loop():
while True:
time.sleep(PORT_DRIFT_POLL)
try:
proc, bound = child.get("proc"), child.get("port")
if proc is None or bound is None or proc.poll() is not None:
continue # between spawns; the main loop has it
now_port = find_seq_port()
if now_port and now_port != bound:
print(f"lcxl-leds --watch: LCXL moved {bound} -> {now_port}; "
"respawning the tap", file=sys.stderr)
invalidate_ports() # the SEND side resolved the old address too
proc.terminate()
except Exception as e:
print(f"lcxl-leds --watch: port watchdog: {e}", file=sys.stderr)
threading.Thread(target=port_drift_loop, daemon=True).start()
while True: while True:
port = find_seq_port() # re-resolved on every (re)spawn, never cached port = find_seq_port() # re-resolved on every (re)spawn, never cached
if port is None: if port is None:
...@@ -1123,6 +1162,10 @@ def cmd_watch(s: Sender, track: str | None, reassert: float = 30.0, ...@@ -1123,6 +1162,10 @@ def cmd_watch(s: Sender, track: str | None, reassert: float = 30.0,
time.sleep(backoff); backoff = min(backoff * 2, 30.0) time.sleep(backoff); backoff = min(backoff * 2, 30.0)
continue continue
# Publish what this child is actually bound to, so the drift watchdog
# compares against the live binding rather than a re-resolution of its own.
child["proc"], child["port"] = proc, port
started = time.time() started = time.time()
try: try:
for line in proc.stdout: # type: ignore[union-attr] for line in proc.stdout: # type: ignore[union-attr]
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment