Commit fdd1418b by PLN (Algolia)

fix(log 038): retract the nohup diagnosis — two false negatives, and two renders racing

I wrote in 038 that `nohup … &` inside a tool call does not survive, on the
evidence of an empty logfile and no matching process. Both were artefacts, the
diagnosis was wrong, and acting on it cost considerably more than the dead job
I thought I was recovering from.

The log was empty because Python block-buffers stdout to a file. No TTY means a
~8 KB buffer, and PYTHONUNBUFFERED=1 was set in the systemd unit but not on the
nohup line, so forty minutes of genuine work produced a zero-byte log. The process
appeared missing because rtk silently dropped the grep; `rtk proxy ps` showed it
immediately, reparented to PID 1553 — the user systemd manager — and well into its
render. RTK.md says to fall back to `rtk proxy` whenever output looks suspiciously
empty, and I skipped that on the one check the entire conclusion rested on.

So I started the same 63-minute render a second time. For roughly forty minutes two
processes wrote the same output paths, racing on Cosmic26_v1_streaming.flac and
master_report.json. Nothing warned me; it surfaced only because two ffmpeg
processes turned up at different pipeline stages, which is luck rather than a
designed tell. Killed the unsupervised duplicate by PID, removed its 102 MB partial
FLAC so cosmic-finish could not split a truncated file, and let the journaled unit
carry on. The unit has since reported attempt 1 cleanly: pre-gain +11.6 dB giving
I=-15.94, TP=-1.00, LRA 8.30 down from 10.60 — dynamics substantially intact.

Two rules earned the hard way, both recorded in
reference_durable_background_jobs alongside the retraction. An empty logfile is
not evidence of no progress: use `python -u` or PYTHONUNBUFFERED on anything whose
log you intend to read. And before starting a replacement for a job you believe is
gone, PROVE it is gone — via `rtk proxy ps` or the unit's own cgroup.procs —
because a duplicate writing the same paths is a corruption risk, not merely wasted
CPU. feedback_check_the_instrument_first and feedback_absence_needs_proof both
applied here and I honoured neither.

What the episode does not undermine is the house rule, which stands on its own
evidence from the same night: systemd --user units outlive the session. Every
harness background waiter and monitor I had armed was killed at a between-turn
teardown, while all three cosmic-* units kept running untouched. That is the reason
to use them. The nohup story was never the reason.
parent f87c8ead
...@@ -226,10 +226,33 @@ already records that defect for OPAL's `take-five-drops`. ...@@ -226,10 +226,33 @@ already records that defect for OPAL's `take-five-drops`.
## Two lessons about running work overnight ## Two lessons about running work overnight
`nohup … &` inside a Bash tool call **does not survive**, and it fails silently: **Retracted, same night.** I first wrote here that `nohup … &` inside a tool call
no process, an empty log, no output, nothing errored. I read "still running" off a does not survive, on the evidence of an empty log and no matching process. Both
90-second poll and lost ~25 minutes. Everything longer than one tool call now were false negatives, and the wrong diagnosis cost more than a dead job would have.
goes in a `systemd --user` unit — which is what my own memory already said.
The log was empty because **Python block-buffers stdout to a file** — no TTY, ~8 KB
buffer, and I had set `PYTHONUNBUFFERED=1` only in the systemd unit, not on the
`nohup` line. The process was missing because **rtk silently dropped the grep**;
`rtk proxy ps` showed it plainly, reparented to the user systemd manager and forty
minutes into its work. RTK.md says to fall back to `rtk proxy` when output looks
suspiciously empty, and I skipped that on the single check the whole conclusion
rested on.
So I started a second copy of the same 63-minute render. For ~40 minutes **two
processes wrote the same output paths**, racing on `Cosmic26_v1_streaming.flac` and
`master_report.json`. It surfaced only because two `ffmpeg` processes appeared at
different pipeline stages, which is a lucky tell rather than a designed one. Killed
the unsupervised duplicate by PID, deleted its 102 MB partial FLAC so the finisher
could not split a truncated file, and let the journaled unit continue.
Two rules earned the hard way: **an empty logfile is not evidence of no progress**
(use `python -u`), and **before starting a replacement, prove the original is
gone** — a duplicate writing the same paths is a corruption risk, not just waste.
The house rule survives intact and for its own reasons: systemd units outlive the
session. Verified the same night — every harness background waiter and monitor I
had armed was killed at a between-turn teardown, while all three `cosmic-*` units
kept running untouched.
And the steps *after* the long jobs were sitting in my session, which would have And the steps *after* the long jobs were sitting in my session, which would have
left PLN with masters and nothing else. So `cosmic-finish.service` is ordered left PLN with masters and nothing else. So `cosmic-finish.service` is ordered
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment