Skip to content

T
Tidal
Commit 446d7a44 by PLN (Algolia)
Options

feat(samples): drop a pack in, keep playing — the rig picks it up in a minute 

PLN, mid-compose: "can we build a parvagues samples watcher tool. watches every
min the samples folder. on new found, loads in current service running sc the
sample pack?"

Three pieces, and the interesting part is what each one REFUSES to do.

sample_watch.scd — an OSC responder (/pv/ping, /pv/loadBank) so a new bank can
be registered into the RUNNING rig. Sourced from start_and_midi.scd, and safe
to evaluate standalone mid-session since OSCdef replaces its own key. Loading
is cheap because lazy mode reads WAV headers only.

tools/pvbanks.py — ONE definition of "what banks exist", mirroring the boot's
three loaders. A watcher and a checker that disagreed about what a bank is
would be the worst bug available here: the checker says clean, the watcher
loads something else, and the rig plays a sound nobody chose.

tools/sample-watcher.py — the loop, plus a systemd --user unit that is
deliberately NOT PartOf the SuperCollider unit, because a supervisor coupled
to its supervisee launders its own restart limit.

WHAT IT REFUSES

Shadowing. `~dirt.loadSoundFiles` cannot append even if you ask it to —
SuperDirt.sc:132 passes `appendToExisting = false` as an ASSIGNMENT rather than
forwarding the argument, so every load replaces. A pack shipping a folder
called `kick` would therefore free the existing bank's buffers and take the
name, changing what an old track plays with no error anywhere. Any new name
claimed by two folders is reported and SKIPPED for a human.

Claiming success at a silent rig. It pings first, and a load that gets no
reply is NOT written to the state file, so the next pass retries instead of
believing itself. Verified against a mock rig: no-reply pass loads 0 and marks
nothing known.

TWO BUGS CAUGHT BEFORE THEY SHIPPED

PathName.folderName returns the PARENT for a directory path — it is built for
file paths. The responder would have looked up the wrong bank and reported a
wrong file count on every success. It now diffs soundLibrary.buffers.keys
before and after, so whatever key APPEARS is the bank name by construction,
and no assumption about basename-with-trailing-separator is needed.

My own protocol test wrote to the real state file, which then made --dry-run
report all 1425 banks as new. Fixed the test isolation and, separately, made
the baseline branch honour --dry-run instead of saving.

Also verified against the real tree: 1425 banks, 19396 playable files, zero
shadowed names. File counts follow SuperDirt's pathMatch("*"), which skips
dotfiles — the AppleDouble `._X.wav` trap that once made a folder look clean
to `ls` while a naive glob double-counted it.

NOT YET VERIFIED: the .scd against a live SuperDirt. PLN is composing and the
responder needs one eval in his session; the Python half is proven end to end
against a mock.
parent c4e70571
Showing with 515 additions and 0 deletions
sample_watch.scd 0 → 100644
// ParVagues sample watcher — the SuperCollider half.
//
// Installs an OSC responder so `tools/sample-watcher.py` can register a newly
// dropped sample pack into the RUNNING rig. No reboot, no interruption: with
// lazy loading on, registering a bank reads WAV headers only, and the audio is
// read on first hit like any other bank.
//
// Loaded from start_and_midi.scd at boot, and safe to evaluate on its own mid
// session (OSCdef replaces its own key, so re-evaluating never stacks
// responders).
//
// /pv/ping -> replies /pv/pong <nBanks>
// /pv/loadBank <path> [<naming>] -> loads one folder as a bank
// naming: "basename" (default) | "nodash"
//
// SAFETY, in order of how badly each one bites:
//
// 1. doNotReadYet is FORCED BACK TO TRUE around every load. It is the only
// gate on the on-demand read path — with it false, a freshly registered
// bank is never read at all. Not slow: permanently silent. That cost two
// evenings in July 2026.
// 2. Paths are refused unless they sit under a known sample root, so a stray
// OSC packet cannot make the rig read arbitrary parts of the disk.
// 3. The responder REPLIES. A watcher that fires into the void and reports
// success is worse than no watcher, so every load is acknowledged with the
// file count SuperCollider actually saw — which is the only number that
// proves the bank is playable.
(
~pvSampleRoots = [
"/home/pln/.local/share/SuperCollider/downloaded-quarks/Dirt-Samples",
"/home/pln/Work/Sound/Samples/extra",
"/home/pln/Work/Sound/Samples/tidal-drum-machines/machines"
];
~pvBankCount = { ~dirt.soundLibrary.buffers.size };
OSCdef(\pvPing, { |msg, time, addr|
addr.sendMsg('/pv/pong', ~pvBankCount.value);
}, '/pv/ping');
OSCdef(\pvLoadBank, { |msg, time, addr|
var path = msg[1].asString;
var naming = if(msg.size > 2) { msg[2].asString } { "basename" };
var name, ok, before, after;
ok = ~pvSampleRoots.any { |r| path.beginsWith(r) };
if(ok.not) {
"pv: REFUSED (outside the sample roots): %".format(path).warn;
addr.sendMsg('/pv/loaded', path, -1, "outside sample roots");
} {
if(File.exists(path).not) {
"pv: REFUSED (no such folder): %".format(path).warn;
addr.sendMsg('/pv/loaded', path, -1, "no such folder");
} {
// Do NOT try to derive the bank name from the path. PathName's
// folderName returns the PARENT for a directory path, and the
// default namingFunction is the `basename` PRIMITIVE applied to a
// path with a trailing separator — behaviour we would be guessing
// at. Ask the library what changed instead: whatever key appears
// IS the bank name, by construction.
before = ~dirt.soundLibrary.buffers.keys.copy;
// Force lazy mode for the duration. See safety note 1.
~dirt.doNotReadYet = true;
if(naming == "nodash") {
~dirt.loadSoundFiles(path,
namingFunction: { |x| x.basename.replace("-", "") });
} {
~dirt.loadSoundFiles(path);
};
~dirt.doNotReadYet = true;
after = ~dirt.soundLibrary.buffers.keys.select { |k|
before.includes(k).not
};
if(after.size == 1) {
name = after.asArray[0];
~pvLast = ~dirt.soundLibrary.buffers.at(name).size;
"pv: loaded bank '%' — % files".format(name, ~pvLast).postln;
addr.sendMsg('/pv/loaded', path, ~pvLast, name.asString);
} {
// Zero means the name already existed and this REPLACED it —
// which the watcher refuses to do on purpose, so say so rather
// than reporting a success nobody asked for.
"pv: % added % new bank name(s) — expected exactly 1".format(
path, after.size).warn;
addr.sendMsg('/pv/loaded', path, -1,
"added % names, expected 1".format(after.size));
};
};
};
}, '/pv/loadBank');
"pv: sample watcher responder armed (/pv/ping, /pv/loadBank) — % banks".format(
~pvBankCount.value).postln;
)
start_and_midi.scd
...@@ -293,6 +293,15 @@ s.waitForBoot { ...@@ -293,6 +293,15 @@ s.waitForBoot {
x.globalEffects = x.globalEffects.addFirst(verb).addFirst(clouds).addFirst(ripples); x.globalEffects = x.globalEffects.addFirst(verb).addFirst(clouds).addFirst(ripples);
x.initNodeTree; x.initNodeTree;
}; };
// Sample watcher responder: lets tools/sample-watcher.py register a newly
// dropped pack into THIS running rig, no reboot. Non-essential — if the
// file is missing the rig boots exactly as before.
if(File.exists("/home/pln/Work/Sound/Tidal/sample_watch.scd")) {
"/home/pln/Work/Sound/Tidal/sample_watch.scd".load;
} {
"pv: sample_watch.scd not found — watcher responder not armed.".postln;
};
}; };
s.volume = 2; // +6dB — SuperDirt defaults are conservative s.volume = 2; // +6dB — SuperDirt defaults are conservative
......
tools/bank-check.py 0 → 100644
#!/usr/bin/env python3
"""What sample banks does SuperDirt actually have, and which ones shadow which?
Run this after dropping new samples in, BEFORE reloading. It resolves bank
names exactly the way `start_and_midi.scd`'s three loaders do, so it can answer
the two questions that bite:
1. Did my new folder QUIETLY REPLACE an existing bank? `loadSoundFiles`
defaults to appendToExisting = false, so a later loader with the same
basename frees the earlier bank's buffers and takes the name. A pack with
a folder called `kick` or `hh` will silently shadow whatever was there,
and the only symptom is that an old track now plays the wrong sound.
2. How many files will SuperDirt see? NOT what `ls` shows. SuperDirt globs
with pathMatch("*"), which skips dotfiles — and some packs ship an
AppleDouble `._X.wav` beside every real `X.wav`. `ls` hides those too, so
a folder looks clean while a naive glob double-counts it.
Usage:
tools/bank-check.py # full report
tools/bank-check.py --since 3 # only banks touched in the last 3 days
tools/bank-check.py --reload # print the sclang to paste into SC
"""
from __future__ import annotations
import argparse
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from pvbanks import LOADERS, scan, fmt_time # one definition of "a bank"
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--since", type=float, help="only banks with files newer than N days")
ap.add_argument("--reload", action="store_true", help="print the sclang to paste")
a = ap.parse_args()
banks = scan()
total_files = sum(e[2] for v in banks.values() for e in v)
print(f"{len(banks)} bank names, {total_files} playable files\n")
shadowed = {k: v for k, v in banks.items() if len(v) > 1}
if shadowed:
print(f"SHADOWED — {len(shadowed)} name(s) claimed by more than one folder.")
print("The LAST loader wins; the earlier bank's buffers are freed.\n")
for k, v in sorted(shadowed.items()):
print(f" {k!r}")
for i, (label, f, n, mt) in enumerate(v):
mark = " <- WINS" if i == len(v) - 1 else ""
print(f" {label:<14} {n:>5} files "
f"{fmt_time(mt)[:10]}{mark}")
print()
else:
print("No shadowed bank names. Every `s \"name\"` is unambiguous.\n")
if a.since:
cut = time.time() - a.since * 86400
fresh = sorted((max(e[3] for e in v), k, v) for k, v in banks.items()
if max(e[3] for e in v) > cut)
print(f"Banks with files newer than {a.since:g} day(s): {len(fresh)}")
for mt, k, v in fresh:
loaders = "+".join(e[0] for e in v)
print(f" {fmt_time(mt)} "
f"{k:<28} {sum(e[2] for e in v):>5} files ({loaders})")
print()
if a.reload:
print("Paste into SuperCollider to pick up new banks WITHOUT rebooting:\n")
print("(")
print("~dirt.doNotReadYet = true; // keep lazy loading ON — see below")
for label, root, depth, _ in LOADERS:
if depth == 1:
print(f'~dirt.loadSoundFiles("{root}/*");')
else:
print(f'PathName.new("{root}").folders.do {{ |m|')
print(' m.folders.do { |f| ~dirt.loadSoundFiles(f.fullPath,')
print(' namingFunction: { |x| x.basename.replace("-", "") }) };')
print("};")
print(")")
print("\n~dirt.doNotReadYet MUST stay true. It is the ONLY gate on the")
print("on-demand read path, so with it false a freshly registered bank")
print("is never read at all — permanently silent, not merely slow.")
if __name__ == "__main__":
main()
tools/parvagues-samples.service 0 → 100644
[Unit]
Description=ParVagues sample watcher — register new packs into the running SuperCollider
Documentation=file:///home/pln/Work/Sound/Tidal/tools/sample-watcher.py
[Service]
Type=simple
ExecStart=/usr/bin/python3 /home/pln/Work/Sound/Tidal/tools/sample-watcher.py --interval 60
Restart=always
RestartSec=30
# Deliberately NOT PartOf/BindsTo the SuperCollider unit. The watcher exits
# when SC is not answering, systemd restarts it 30 s later, and it picks up
# again once the rig is back. Coupling a supervisor to its supervisee is how a
# watchdog launders its own rate limit into a dead service.
Nice=10
IOSchedulingClass=idle
[Install]
WantedBy=default.target
tools/pvbanks.py 0 → 100644
"""Where ParVagues' sample banks come from, resolved the way SuperDirt does.
ONE definition of "what banks exist", imported by both `bank-check.py` and
`sample-watcher.py`. A watcher and a checker that disagree about what a bank is
would be the worst kind of bug here: the checker says clean, the watcher loads
something else, and the rig plays a sound nobody chose.
Mirrors the three loaders in `start_and_midi.scd`, in the order they run.
Later loaders WIN — `loadSoundFiles` defaults to appendToExisting = false, so a
same-named folder frees the earlier bank's buffers and takes the name.
"""
from __future__ import annotations
import time
from pathlib import Path
AUDIO = {".wav", ".aif", ".aiff", ".flac", ".ogg", ".mp3",
".WAV", ".AIF", ".AIFF", ".FLAC"}
LOADERS = [
# (label, root, folder depth below root, bank-name function)
("Dirt-Samples",
Path("~/.local/share/SuperCollider/downloaded-quarks/Dirt-Samples").expanduser(),
1, lambda n: n),
("extra",
Path("~/Work/Sound/Samples/extra").expanduser(),
1, lambda n: n),
("drum-machines",
Path("~/Work/Sound/Samples/tidal-drum-machines/machines").expanduser(),
2, lambda n: n.replace("-", "")), # the namingFunction in the boot
]
ROOTS = [str(r) for _, r, _, _ in LOADERS]
def playable(folder: Path) -> list[Path]:
"""Files SuperDirt will actually load.
It globs with pathMatch("*"), which SKIPS DOTFILES. Some packs ship an
AppleDouble `._X.wav` beside every real `X.wav`; `ls` hides those too, so
the folder looks clean while a naive glob double-counts it. That mismatch
once produced "jbk_kick expected 508, got 254" and the tool was the one
that was wrong.
"""
try:
return sorted(p for p in folder.iterdir()
if p.is_file() and not p.name.startswith(".")
and p.suffix in AUDIO)
except OSError:
return []
def scan() -> dict[str, list[tuple]]:
"""bank name -> [(loader_label, folder_path, n_files, newest_mtime), ...]
More than one entry means the name is SHADOWED; the last one wins.
"""
banks: dict[str, list[tuple]] = {}
for label, root, depth, name_of in LOADERS:
if not root.is_dir():
continue
try:
if depth == 1:
folders = sorted(p for p in root.iterdir() if p.is_dir())
else:
folders = [g for f in sorted(root.iterdir()) if f.is_dir()
for g in sorted(f.iterdir()) if g.is_dir()]
except OSError:
continue
for f in folders:
files = playable(f)
if not files:
continue
mt = max(p.stat().st_mtime for p in files)
banks.setdefault(name_of(f.name), []).append(
(label, f, len(files), mt))
return banks
def under_a_root(path: str) -> bool:
p = str(Path(path).expanduser().resolve())
return any(p == r or p.startswith(r.rstrip("/") + "/") for r in ROOTS)
def fmt_time(mt: float) -> str:
return time.strftime("%Y-%m-%d %H:%M", time.localtime(mt))
tools/sample-watcher.py 0 → 100644
#!/usr/bin/env python3
"""Watch the sample folders; register new packs into the RUNNING SuperCollider.
Drop a pack in, keep playing, `s "newbank"` works about a minute later. No
reboot — the SC half (`sample_watch.scd`) answers an OSC message and registers
the bank lazily, which reads WAV headers only.
tools/sample-watcher.py --once one pass, then exit
tools/sample-watcher.py loop (default every 60 s)
tools/sample-watcher.py --dry-run report what it WOULD load
Two refusals it makes on purpose:
SHADOWING. `loadSoundFiles` defaults to appendToExisting = false, so a folder
whose basename matches an existing bank frees that bank's buffers and takes
the name. Auto-loading one mid-session would change what an old track plays,
with no error anywhere. A pack that would shadow is reported and SKIPPED
until a human decides.
A SILENT RIG. It pings first and refuses to claim anything if the responder
does not answer, then verifies each load against the file count SC reports
back. A watcher that fires into the void and prints "loaded" is worse than
no watcher at all.
"""
from __future__ import annotations
import argparse
import json
import socket
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import pvbanks # noqa: E402
STATE = Path("~/.cache/parvagues/sample-watcher.json").expanduser()
SC_ADDR = ("127.0.0.1", 57120)
# ------------------------------------------------------------------ tiny OSC
def _pad(b: bytes) -> bytes:
return b + b"\0" * ((4 - len(b) % 4) % 4)
def osc_msg(addr: str, *args: str | int) -> bytes:
tags, body = ",", b""
for a in args:
if isinstance(a, int):
tags += "i"
body += a.to_bytes(4, "big", signed=True)
else:
tags += "s"
body += _pad(str(a).encode() + b"\0")
return _pad(addr.encode() + b"\0") + _pad(tags.encode() + b"\0") + body
def osc_parse(data: bytes) -> tuple[str, list]:
def take_str(b, i):
e = b.index(b"\0", i)
s = b[i:e].decode("utf-8", "replace")
return s, i + (((e - i) + 4) // 4) * 4
addr, i = take_str(data, 0)
if i >= len(data):
return addr, []
tags, i = take_str(data, i)
out = []
for t in tags[1:]:
if t == "i":
out.append(int.from_bytes(data[i:i + 4], "big", signed=True)); i += 4
elif t in "fd":
i += 4 if t == "f" else 8
elif t == "s":
s, i = take_str(data, i); out.append(s)
return addr, out
class Rig:
def __init__(self, timeout=2.0):
self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.s.bind(("127.0.0.1", 0))
self.s.settimeout(timeout)
def call(self, addr, *args, expect=None):
self.s.sendto(osc_msg(addr, *args), SC_ADDR)
deadline = time.time() + self.s.gettimeout()
while time.time() < deadline:
try:
data, _ = self.s.recvfrom(65535)
except socket.timeout:
return None
a, vals = osc_parse(data)
if expect is None or a == expect:
return vals
return None
def ping(self):
return self.call("/pv/ping", expect="/pv/pong")
def load(self, path: str, naming: str):
return self.call("/pv/loadBank", path, naming, expect="/pv/loaded")
# --------------------------------------------------------------------- watch
def load_state() -> dict:
try:
return json.loads(STATE.read_text())
except Exception:
return {"known": [], "skipped": {}}
def save_state(st: dict):
STATE.parent.mkdir(parents=True, exist_ok=True)
STATE.write_text(json.dumps(st, indent=1))
def naming_for(label: str) -> str:
return "nodash" if label == "drum-machines" else "basename"
def one_pass(rig: Rig | None, st: dict, dry: bool, verbose: bool) -> int:
banks = pvbanks.scan()
known = set(st.get("known", []))
new = {k: v for k, v in banks.items() if k not in known}
if not known: # first run: adopt, never bulk-load
st["known"] = sorted(banks)
if not dry:
save_state(st)
print(f" baseline: {len(banks)} banks adopted, nothing loaded"
f"{' (dry-run: not saved)' if dry else ''}")
return 0
if not new:
if verbose:
print(f" no new banks ({len(banks)} known)")
return 0
loaded = 0
for name, entries in sorted(new.items()):
if len(entries) > 1:
print(f" SKIP {name!r}: {len(entries)} folders claim this name — "
f"would shadow. Resolve by hand.")
st.setdefault("skipped", {})[name] = "ambiguous"
continue
label, folder, n_files, mt = entries[0]
# A NEW name cannot shadow, but re-check against the live rig anyway:
# the state file can be stale if a bank was added outside the watcher.
print(f" NEW {name!r} {n_files} files ({label}, {pvbanks.fmt_time(mt)})")
if dry:
continue
reply = rig.load(str(folder), naming_for(label))
if reply is None:
print(" ! no reply from SuperCollider — NOT marking as known")
continue
got = reply[1] if len(reply) > 1 else -1
if got < 0:
print(f" ! refused by SC: {reply[2] if len(reply) > 2 else '?'}")
continue
if got != n_files:
# Not fatal — SC counts what it can decode — but say so plainly.
print(f" ~ SC registered {got} files, I counted {n_files}")
else:
print(f" loaded, {got} files — `s \"{name}\"` is live")
known.add(name)
loaded += 1
if not dry:
st["known"] = sorted(known)
save_state(st)
return loaded
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--interval", type=float, default=60.0)
ap.add_argument("--once", action="store_true")
ap.add_argument("--dry-run", action="store_true")
ap.add_argument("--verbose", action="store_true")
ap.add_argument("--reset", action="store_true",
help="forget the baseline and re-adopt everything")
a = ap.parse_args()
if a.reset and STATE.exists():
STATE.unlink()
print(" state reset")
st = load_state()
rig = None
if not a.dry_run:
rig = Rig()
pong = rig.ping()
if pong is None:
sys.exit(
"SuperCollider is not answering /pv/ping.\n"
"Load the responder once (it survives until sclang restarts):\n"
' in SC: "/home/pln/Work/Sound/Tidal/sample_watch.scd".load\n'
"It is also sourced from start_and_midi.scd, so a normal boot "
"arms it automatically.")
print(f" SuperCollider answered: {pong[0]} banks registered")
while True:
try:
one_pass(rig, st, a.dry_run, a.verbose or a.once)
except Exception as e: # never die on one bad pass
print(f" pass failed: {type(e).__name__}: {e}", file=sys.stderr)
if a.once:
break
time.sleep(a.interval)
if __name__ == "__main__":
main()
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment