Skip to content

T
Tidal
Commit 5468507b by PLN (Algolia)
Options

perf(ardour-sweep): derive the archive path instead of searching 25 G for it 

The tool worked but could not answer its own question. Proving a missing source
is gone meant `rglob(name)` per file across the Freebox — 24 full traversals of a
25 G network mount for one session. It ran 8m15s without finishing, and on the
600s budget it would have returned "not found" purely because it ran out of time.
An unfinished search reported as an absence is how you drop the last reference to
a take that was actually recoverable.

The fix is not a faster search, it is not searching. The Freebox is a converged
mirror of $HOME (fbk), so an archived file's location is COMPUTABLE: mirror root +
the path relative to $HOME. One stat per file instead of a walk. Runtime went from
"8m15s, unfinished" to 0.40s, and the answer is now exact rather than best-effort.
It still checks the session's own dead/ and audiofiles dirs, since a file can be
moved aside rather than removed, and it refuses outright if no mirror is mounted —
"gone" is not a claim you get to make about audio when the SSOT is offline.

It also now reports HOW CURRENT the mirror is, because that changes what the
absence means. Here: the mirror last changed 2026-08-29 22:41, so it holds up to
Take100. Take101 is therefore absent locally AND absent from the mirror, but was
also never eligible for that backup — so the honest statement is "it is not
anywhere I can reach", not "it was archived and then lost". The tool now says that
distinction out loud rather than letting the operator read absence as proof.

Verdict for Tidal Live: 294 sources, 24 missing, all Take101, 12 orbits x L/R.
Would drop 24 sources and 24 regions. That take's .mid files survive; only the
audio is gone.

Two bugs of my own, fixed here and worth the note: the mirror loop bound `root`,
shadowing the XML tree and killing the region sweep with "'PosixPath' object has
no attribute 'iter'" — a local rebinding eating an outer name, which is this
repo's recurring self-inflicted wound. And an ElementTree truth test that Python
3.14 deprecates.

Verified the write refusal fires: with Ardour running it stops at "Refusing to
write: Ardour is running (pid [2708769])" and the session file's mtime is
unchanged. Applying it needs Ardour closed, by design.
parent 9bb6c294
Showing with 69 additions and 40 deletions
tools/ardour-drop-missing-sources.py
......@@ -43,7 +43,7 @@ import time
import xml.etree.ElementTree as ET
from pathlib import Path
FREEBOX = Path("/mnt/freebox/PLN")
MIRRORS = [Path("/mnt/freebox/PLN")] # converged $HOME mirrors; SSOT for audio
def ardour_running() -> list[int]:
......@@ -75,33 +75,49 @@ def source_path(session: Path, src: ET.Element) -> Path:
return session.parent / "interchange" / session.stem / sub / name
def find_elsewhere(names: set[str], session: Path, budget: float) -> dict[str, str]:
"""{name: first path} for every one of `names` that survives elsewhere.
def mirror_candidates(path: Path) -> list[Path]:
"""Where a mirrored backup would keep `path`, by DERIVATION not search.
ONE walk per root, matching against a set — not one rglob per file. The first
version did the latter and spent 24 full traversals of the Freebox over the
network for a single session; it never finished. Cheap to write, and the kind
of thing that only shows up on the real corpus.
The Freebox is a converged mirror of $HOME (see fbk), so an archived file's
location is computable: mirror root + the path relative to $HOME. That makes
the question one stat instead of a filesystem walk.
The deadline is checked per DIRECTORY, not per hit, so a fruitless search
still stops on time. A search that ran out of budget is reported as such by
the caller — "I did not find it" and "I did not finish looking" are not the
same answer, and only one of them justifies dropping a reference.
The first version of this tool searched instead — `rglob(name)` per missing
file, 24 full traversals of a 25 G network mount. It ran for over eight
minutes without finishing and would have reported "not found" purely because
it ran out of budget. When the archive's layout is known, derive the path;
only fall back to searching when it is not.
"""
out = []
try:
rel = path.relative_to(Path.home())
except ValueError:
return out
for root in MIRRORS:
if root.exists():
out.append(root / rel)
return out
def find_elsewhere(missing: list[tuple[str, Path]]) -> dict[str, str]:
"""{name: path} for every missing source that survives in a mirror.
`missing` is [(name, where Ardour looked)]. Also checks the session's own
tree, since a file can be moved aside into dead/ rather than deleted.
"""
found: dict[str, str] = {}
deadline = time.monotonic() + budget
for root in (session.parent, FREEBOX):
if not root.exists():
continue
for dirpath, dirnames, filenames in os.walk(root, onerror=lambda e: None):
if time.monotonic() > deadline:
return found
# The session's own audiofiles dir is where we already looked.
for fn in filenames:
if fn in names and fn not in found:
found[fn] = str(Path(dirpath) / fn)
if len(found) == len(names):
return found
for name, wanted in missing:
for cand in mirror_candidates(wanted):
if cand.exists():
found[name] = str(cand)
break
else:
# Moved aside rather than removed? Cheap, local, bounded.
for alt in (wanted.parent.parent / "dead", wanted.parent):
hit = alt / name
if hit.exists() and hit != wanted:
found[name] = str(hit)
break
return found
......@@ -113,8 +129,6 @@ def main(argv=None) -> int:
help="write the session (default is a dry run)")
ap.add_argument("--allow-archived", action="store_true",
help="drop references even if the file survives elsewhere")
ap.add_argument("--search-budget", type=float, default=60.0,
help="seconds to spend proving a file is gone (default 60)")
a = ap.parse_args(argv)
session = a.session.expanduser()
......@@ -141,29 +155,44 @@ def main(argv=None) -> int:
print(f" {take}: {len(ss)} source(s)")
# Prove they are gone everywhere before touching a reference.
names = {(s.get("name") or "") for s in missing}
t0 = time.monotonic()
recoverable = find_elsewhere(names, session, a.search_budget)
spent = time.monotonic() - t0
print(f"\nsearched the session tree and {FREEBOX} in {spent:.0f}s")
pairs = [((s.get("name") or ""), source_path(session, s)) for s in missing]
recoverable = find_elsewhere(pairs)
if recoverable:
print(f"{len(recoverable)} missing file(s) SURVIVE elsewhere:", file=sys.stderr)
print(f"\n{len(recoverable)} missing file(s) SURVIVE elsewhere:", file=sys.stderr)
for name, hit in list(recoverable.items())[:5]:
print(f" {name} -> {hit}", file=sys.stderr)
if not a.allow_archived:
print("\nRefusing: restore these instead of dropping the reference "
"(or pass --allow-archived).", file=sys.stderr)
return 1
elif spent >= a.search_budget - 1:
print("the search ran OUT OF BUDGET before finishing — 'not found' here "
"means 'not yet found'.", file=sys.stderr)
if not a.allow_archived:
print("Refusing on an unfinished search: raise --search-budget, or "
"pass --allow-archived if you know these are gone.",
else:
# NB: not `root` — that name holds the XML tree here, and shadowing it
# made the region sweep below fail with "'PosixPath' has no attribute
# 'iter'". A local rebinding quietly eating an outer name is this repo's
# recurring self-inflicted wound.
for mroot in MIRRORS:
if not mroot.exists():
print(f"\nWARNING: {mroot} is not mounted — 'gone' is unproven.",
file=sys.stderr)
if not a.allow_archived:
print("Refusing on an unverifiable archive: mount it, or pass "
"--allow-archived.", file=sys.stderr)
return 1
else:
print("confirmed gone: nothing survives in the session tree or the Freebox.")
print("\nconfirmed gone: absent locally and absent from every mounted mirror.")
# Say how current the mirror is. An empty answer from a STALE mirror is a
# weaker fact than it looks, and the operator should see which they have.
for cand in mirror_candidates(source_path(session, missing[0])):
d = cand.parent
if not d.exists():
continue
try:
newest = max((f.stat().st_mtime for f in d.iterdir()), default=0)
except OSError:
continue
print(f" mirror {d} last changed "
f"{time.strftime('%Y-%m-%d %H:%M', time.localtime(newest))} — if "
"that predates the take, its absence there proves nothing about "
"whether it was ever archived, only that it is not there now.")
ids = {s.get("id") for s in missing}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment