Commit fa7aef0b by PLN (Algolia)

add(tools): sample-pack.py — bundle a set's must-travel sample banks

Answers: for a chosen set of tracks, which Dirt-Samples banks does it
actually need, and can that be handed off over swisstransfer to play the
set from a different laptop? Selectors: --tracks (explicit/bare names),
--set NAME (armada/setlist_*.txt or a gig's tracks.json under
../../Web/www/content/lives/{year}/{slug}/, --set list to enumerate),
--all (every .tidal under live/+copycat/, the default). Dry run by
default; --pack DEST copies, --manifest-only writes just the manifest
(the one meant to be committed, so a remote box can verify/reproduce
the bundle without re-scanning).

Reuses tools/setlist_samples.py (extract_names/build_index/resolve_track/
read_setlist/bank_file_count) as the ONLY .tidal parser — no new regex
over .tidal text. A second ad-hoc parser is the specific mistake this
tool exists to avoid (a prior grep false-positived "gfunk" prose against
the gfunk_* banks).

THE MAPPED-ONLY FILTER: only banks actually mapped into the Dirt-Samples
root travel (a real dir there, or a symlink into ~/Work/Sound/Samples).
Classified as stock (tracked in the Dirt-Samples git checkout — ships
free with a fresh install, excluded unless --include-stock), mapped
(the symlink farm + local additions), extra (~/Work/Sound/Samples/extra),
or unmapped (e.g. tidal-drum-machines, excluded unless --include-unmapped).
The raw ~/Work/Sound/Samples tree (36 GB, mostly work-in-progress) is
never walked directly — only through the same mapped index
setlist_samples.py already resolves against, which is what keeps that
36 GB out of the bundle.

BANK-INDEX SAFETY: `s "bank:N"` indexes files in the order SuperDirt's
pathMatch("*") glob returns them, which skips dotfiles. So packing copies
EVERY non-dotfile in a bank (never subsets), follows symlinks to copy
real content, and preserves the bank's directory name and file names
exactly — dropping even one real file renumbers every index after it and
changes what plays on the receiving machine. --force on a re-pack rmtrees
the destination bank first rather than merging, for the same reason: a
merge leaves stale files from a previous pack sitting next to fresh ones.

Measured against --all today (714 .tidal files, live/+copycat/): 400
banks resolve; 119 are stock (~166 MB) and excluded by default; 265
mapped + 16 extra = 281 must travel, ~15.1 GB total. rhadamanthe_divers
alone is a symlink to 314 files / ~4.3 GB, roughly 30% of the bundle.

Tokens that resolve to no bank (--unresolved) are NOT reported as
missing/broken: ParVagues' `#` is structure-left/values-right
(`$ s "k" # s "jazz"` — k is a rhythm skeleton, jazz is what plays), so
most unresolved tokens are structural placeholders or built-in SuperDirt
synth names (superpiano, moogBass, supersaw...), never dropped samples.
setlist_samples.py's own docstring treats the unresolved set the same
way — as a non-error remainder, not an alarm channel.
parent 747f5be2
#!/usr/bin/env python3
"""sample-pack — which sample banks does a set need, and can I hand them off?
PLN plays from a borrowed/second laptop sometimes (see the "portable rig"
project). This answers: for a chosen set of tracks, which sample banks do they
actually reference, which of those already ship with a fresh SuperDirt
install (skip them), which are ParVagues-specific and must travel, and how
big is that bundle — then, on request, copies it into a folder that can go
over swisstransfer and get dropped into Dirt-Samples/ on the other machine.
WHAT BITES
1. THE MAPPED-ONLY FILTER. `~/Work/Sound/Samples` is 36 GB of work-in-progress
— everything that's actually usable is MAPPED into the Dirt-Samples root
(a real folder there, or a symlink pointing back into that raw tree).
This tool only ever enumerates banks through that mapped view (the same
`build_index()` `tools/setlist_samples.py` uses to resolve `s "name"`
tokens for preload). It never walks the raw Samples/ tree looking for
more — that would silently pull in unfinished/unmixed material and blow
the bundle size past reason. "Mapped" IS "done", by PLN's own rule.
2. BANK-INDEX SAFETY. `s "bank:N"` indexes files in the order SuperDirt's
`pathMatch("*")` glob returns them — which SKIPS DOTFILES (so an
AppleDouble `._X.wav` sidecar never shifts anything; see
`tools/bank-check.py`'s docstring for the exact mechanism and the
508-vs-254 file count that this cost once). Dropping even one *real*
file when copying a bank renumbers every index after it, and the
receiving laptop plays a different sound than the one PLN chose on
stage. So packing NEVER subsets a bank: every non-dotfile travels,
original names and directory name preserved, nothing sorted or renamed.
3. THE PARSER. `.tidal` files are ParVagues' own dialect — bare-string
sample names mixed with notes/structure/MIDI-ctrl strings in the same
quotes, comments that look like code. `tools/setlist_samples.py` is the
one sanctioned parser (extract_names/build_index/resolve_track/
read_setlist/bank_file_count); this tool imports it rather than growing
a second regex, because the second regex is exactly how `gfunk` in prose
once false-positived against the `gfunk_*` banks.
4. AN UNRESOLVED TOKEN IS NOT A BROKEN REFERENCE. In ParVagues' dialect
`#` is `|>` — structure from the LEFT, values from the RIGHT:
$ s "k" -- rhythm skeleton, never resolved as a sample
# s "jazz" -- the sound that actually plays
`k` never needed to resolve to a folder; it gets overridden by the `#
s "jazz"` downstream. Most tokens that fail to resolve are exactly this
(a structural placeholder) or a built-in SuperDirt synth name
(`superpiano`, `supersaw`...), never a missing sample. This is also WHY
`setlist_samples.py` extracts every token from every quoted string and
lets the on-disk index be the sole filter, silently: the unresolved set
is not an error channel, and reporting it as one would make PLN think
the set is broken when it never was. This tool only surfaces it behind
`--unresolved`, worded as what it usually is.
MEASURED (2026-09-06, this repo, `--all`, 714 .tidal files under live/+copycat/):
400 banks resolve; 119 are stock (~166 MB, upstream Dirt-Samples git);
281 are mapped+extra and must travel (~15.1 GB). `rhadamanthe_divers` is
a symlink to a 314-file/4.3 GB folder — ~30% of the bundle alone.
Usage:
tools/sample-pack.py --set list # what sets exist
tools/sample-pack.py --set opal2026 # dry run for a set
tools/sample-pack.py --tracks desire.tidal wap.tidal # dry run, explicit
tools/sample-pack.py # dry run, --all (default)
tools/sample-pack.py --all --json > report.json
tools/sample-pack.py --set opal2026 --pack /mnt/freebox/staging/opal2026
tools/sample-pack.py --all --manifest-only armada/tide-table/bundle.json
"""
from __future__ import annotations
import argparse
import json
import shutil
import subprocess
import sys
import time
from collections import Counter
from pathlib import Path
TOOLS_DIR = Path(__file__).resolve().parent
REPO_ROOT = TOOLS_DIR.parent
sys.path.insert(0, str(TOOLS_DIR))
import setlist_samples as ss # noqa: E402 — THE sanctioned .tidal parser
# setlist_samples.py hardcodes its repo/live paths to the canonical checkout
# (~/Work/Sound/Tidal). We are told to work ONLY in this worktree and never
# touch that shared checkout — including read-only globs racing another
# party's uncommitted edits — so repoint its root constants at wherever this
# script actually lives before calling any of its resolution helpers.
ss.REPO = REPO_ROOT
ss.LIVE_DIR = REPO_ROOT / "live"
LIVES_ROOT = Path("/home/pln/Work/Web/www/content/lives") # gig metadata SSOT
DIRT_SAMPLES = ss.SAMPLE_ROOTS[0]
EXTRA_ROOT = ss.SAMPLE_ROOTS[1]
DRUM_MACHINES = ss.DRUM_MACHINES
# ── selection ─────────────────────────────────────────────────────────────
def all_tracks() -> list[Path]:
"""Every .tidal under live/ and copycat/ — the --all default."""
return sorted(REPO_ROOT.glob("live/**/*.tidal")) + \
sorted(REPO_ROOT.glob("copycat/**/*.tidal"))
def find_sets() -> dict[str, list[tuple[str, Path]]]:
"""name -> [(kind, path), ...]. Two sources, unioned:
* `armada/setlist_*.txt` — the gig-up preload whitelists (today: just
opal2026). name = the part between "setlist_" and ".txt".
* `.../content/lives/{year}/{slug}/tracks.json` — the canonical gig
catalog in the www repo (per CLAUDE.md: gig metadata lives there, not
here). name = the slug, PLUS "{slug}-{year}" always, because slugs
collide across years (e.g. "la-french-stack" exists in both 2024 and
2025) and the qualified form is how an ambiguous NAME gets resolved.
"""
sets: dict[str, list[tuple[str, Path]]] = {}
for f in sorted((REPO_ROOT / "armada").glob("setlist_*.txt")):
name = f.stem[len("setlist_"):]
sets.setdefault(name, []).append(("setlist", f))
if LIVES_ROOT.is_dir():
for tj in sorted(LIVES_ROOT.glob("*/*/tracks.json")):
slug, year = tj.parent.name, tj.parent.parent.name
sets.setdefault(slug, []).append(("gig", tj))
if not slug.endswith(f"-{year}"): # e.g. "cosmicfest-2026" already has it
sets.setdefault(f"{slug}-{year}", []).append(("gig", tj))
return sets
def resolve_gig_track(file_field: str) -> list[Path]:
"""A tracks.json entry's "file" is repo-relative; fall back to resolve_track
(bare-name search) if the literal path has moved."""
p = REPO_ROOT / file_field
if p.is_file():
return [p]
return ss.resolve_track(file_field)
def resolve_set(name: str, sets: dict[str, list[tuple[str, Path]]]) -> list[Path]:
if name not in sets:
avail = "\n".join(f" {n}" for n in sorted(sets))
sys.exit(f"! no set named {name!r}. Available:\n{avail}")
entries = sets[name]
if len(entries) > 1:
kinds = ", ".join(f"{k} ({p})" for k, p in entries)
sys.exit(f"! {name!r} is ambiguous ({len(entries)} sources: {kinds}). "
f"Qualify it, e.g. try one of the '-<year>' aliases from --set list.")
kind, path = entries[0]
if kind == "setlist":
return ss.read_setlist(path)
data = json.loads(path.read_text())
tracks: list[Path] = []
for entry in data.get("tracks", []):
f = entry.get("file")
if f:
tracks += resolve_gig_track(f)
return list(dict.fromkeys(tracks))
def gather_tracks(args) -> tuple[dict, list[Path]]:
if args.tracks:
tracks = [p for arg in args.tracks for p in ss.resolve_track(arg)]
selector = {"kind": "tracks", "args": args.tracks}
elif args.set:
sets = find_sets()
if args.set == "list":
print("available --set names:")
for n in sorted(sets):
kinds = {k for k, _ in sets[n]}
flag = " (ambiguous)" if len(sets[n]) > 1 else ""
print(f" {n:30s} [{','.join(sorted(kinds))}]{flag}")
sys.exit(0)
tracks = resolve_set(args.set, sets)
selector = {"kind": "set", "name": args.set}
else:
tracks = all_tracks()
selector = {"kind": "all"}
tracks = list(dict.fromkeys(tracks))
if not tracks:
sys.exit("! selection resolved to zero tracks — nothing to pack")
return selector, tracks
# ── classification ───────────────────────────────────────────────────────
def upstream_tracked() -> set[str] | None:
"""Bank names that ship in a fresh Dirt-Samples git checkout (= 'stock',
excluded from the bundle by default). None means "couldn't tell" — the
caller must then treat every real folder as travel-worthy rather than
guess it away."""
try:
out = subprocess.run(
["git", "-C", str(DIRT_SAMPLES), "ls-tree", "-d", "--name-only", "HEAD"],
capture_output=True, text=True, check=True, timeout=15)
return set(out.stdout.split())
except Exception as e:
print(f"! could not read Dirt-Samples' git history ({e}) — cannot tell stock "
f"from local; treating every real (non-symlink) folder there as "
f"must-travel to be safe. Fix: `git -C {DIRT_SAMPLES} status`.",
file=sys.stderr)
return None
def _under(path: Path, root: Path) -> bool:
try:
path.relative_to(root)
return True
except ValueError:
return False
def classify(name: str, folder: Path, tracked: set[str] | None) -> str:
"""stock | mapped | extra | unmapped — see the module docstring's rule 1."""
if _under(folder, DIRT_SAMPLES):
if folder.is_symlink():
return "mapped" # the symlink farm — points into raw Samples/
if tracked is not None and name in tracked:
return "stock" # a fresh SuperDirt install already has it
return "mapped" # real dir, NOT upstream (e.g. a local addition)
if _under(folder, EXTRA_ROOT):
return "extra"
return "unmapped" # drum-machines, or anything else build_index found
# ── bank contents (the index-safety-critical part) ──────────────────────
def walk_bank(folder: Path) -> list[tuple[str, Path, int]]:
"""(relpath, real file, size) for every non-dotfile in the bank, recursive,
FOLLOWING symlinks (most mapped banks are one big symlink to raw content).
Dotfiles/dot-dirs are skipped, and ONLY dotfiles — SuperDirt's own
pathMatch("*") glob already skips them (see bank-check.py), so omitting
them cannot shift bank:N. Nothing else is filtered: no extension check,
no subsetting. That's the whole point — see docstring rule 2.
"""
out: list[tuple[str, Path, int]] = []
stack = [(folder, "")]
while stack:
d, prefix = stack.pop()
try:
entries = sorted(d.iterdir())
except OSError:
continue
for e in entries:
if e.name.startswith("."):
continue
rel = prefix + e.name
if e.is_dir():
stack.append((e, rel + "/"))
elif e.is_file():
try:
out.append((rel, e, e.stat().st_size))
except OSError:
pass
return out
# ── report ────────────────────────────────────────────────────────────────
def human(n: float) -> str:
if n >= 1e9:
return f"{n/1e9:.2f} GB"
if n >= 1e6:
return f"{n/1e6:.1f} MB"
if n >= 1e3:
return f"{n/1e3:.1f} KB"
return f"{n:.0f} B"
def build_report(selector: dict, tracks: list[Path]) -> dict:
idx = ss.build_index()
tracked = upstream_tracked()
names: set[str] = set()
unresolved_files: Counter = Counter() # unresolved name -> how many tracks used it
for t in tracks:
try:
found = ss.extract_names(t.read_text(errors="replace"))
except OSError as e:
print(f"! skip {t}: {e}", file=sys.stderr)
continue
names |= found
for n in found:
if n not in idx and n not in ss.STOP:
unresolved_files[n] += 1
resolved = {n: idx[n] for n in names if n in idx and n not in ss.STOP}
banks: dict[str, list[dict]] = {"stock": [], "mapped": [], "extra": [], "unmapped": []}
for name in sorted(resolved):
folder = resolved[name]
cls = classify(name, folder, tracked)
files = walk_bank(folder.resolve() if folder.is_symlink() else folder)
total_bytes = sum(sz for _, _, sz in files)
banks[cls].append({
"name": name,
"files": len(files), # every non-dotfile copied, recursive
# top-level audio files SuperDirt's own pathMatch("*") will register —
# what a receiver actually checks bank:N against (setlist_samples.py's
# bank_file_count, reused rather than reimplemented).
"superdirt_files": ss.bank_file_count(folder),
"bytes": total_bytes,
"source": str(folder),
})
return {
"selector": selector,
"tracks": [str(t) for t in tracks],
"banks": banks,
# NOT an error list — see docstring rule 4 (`s "k" # s "jazz"`). Most
# entries here are structural placeholders or built-in synth names,
# not broken sample references. Only surfaced behind --unresolved.
"unresolved": [{"name": n, "file_count": c}
for n, c in sorted(unresolved_files.items(), key=lambda kv: -kv[1])],
"tracked_upstream_known": tracked is not None,
}
def bundle_classes(report: dict, include_stock: bool, include_unmapped: bool) -> list[str]:
classes = ["mapped", "extra"]
if include_stock:
classes.append("stock")
if include_unmapped:
classes.append("unmapped")
return classes
def print_table(report: dict, include_stock: bool, include_unmapped: bool,
show_unresolved: bool) -> None:
sel = report["selector"]
print(f"selector: {sel}")
print(f"tracks scanned: {len(report['tracks'])}")
print()
order = ["stock", "mapped", "extra", "unmapped"]
grand_bytes = 0
for cls in order:
entries = report["banks"][cls]
n_bytes = sum(e["bytes"] for e in entries)
n_files = sum(e["files"] for e in entries)
in_bundle = cls in bundle_classes(report, include_stock, include_unmapped)
if in_bundle:
grand_bytes += n_bytes
tag = "-> BUNDLE" if in_bundle else "(excluded by default)" if cls in ("stock", "unmapped") else ""
print(f"{cls:10s} {len(entries):4d} banks {n_files:6d} files {human(n_bytes):>10s} {tag}")
print()
extras = " ".join(f for f in
("+stock" if include_stock else "", "+unmapped" if include_unmapped else "")
if f)
print(f"BUNDLE TOTAL: {human(grand_bytes)}" + (f" ({extras})" if extras else ""))
if not report["tracked_upstream_known"]:
print("\n! stock/mapped split for Dirt-Samples could not be verified via git "
"(see stderr) — everything not a symlink was counted as mapped/must-travel.")
if show_unresolved:
unresolved = report["unresolved"]
print(f"\ntokens that resolved to no bank: {len(unresolved)} distinct — this is "
f"EXPECTED, not an error. `#` is structure-left/values-right "
f"(`$ s \"k\" # s \"jazz\"` — `k` is a rhythm skeleton, `jazz` is what "
f"plays), so most of these are structural placeholders or built-in synth "
f"names, not missing samples:")
for m in unresolved[:20]:
print(f" {m['file_count']:4d} files {m['name']}")
if len(unresolved) > 20:
print(f" ... and {len(unresolved) - 20} more (see --json)")
# ── pack ──────────────────────────────────────────────────────────────────
def build_manifest(report: dict, include_stock: bool, include_unmapped: bool) -> dict:
classes = bundle_classes(report, include_stock, include_unmapped)
packed = []
for cls in classes:
for e in report["banks"][cls]:
folder = Path(e["source"])
real = folder.resolve() if folder.is_symlink() else folder
files = sorted(rel for rel, _, _ in walk_bank(real))
packed.append({**e, "class": cls, "files_listing": files})
return {
"generated": time.strftime("%Y-%m-%dT%H:%M:%S"),
"selector": report["selector"],
"tracks": report["tracks"],
"include_stock": include_stock,
"include_unmapped": include_unmapped,
"banks": packed,
"bundle_bytes": sum(b["bytes"] for b in packed),
"bundle_banks": len(packed),
}
README_TEMPLATE = """\
# sample-pack bundle
Generated {generated} by tools/sample-pack.py.
Selector: {selector}
{n_banks} bank(s), {total} total.
## Install on the receiving machine
Drop each of these folders straight into your Dirt-Samples root
(`~/.local/share/SuperCollider/downloaded-quarks/Dirt-Samples/`) — or symlink
them there if you'd rather keep this bundle separate:
for d in */; do ln -s "$PWD/${{d%/}}" ~/.local/share/SuperCollider/downloaded-quarks/Dirt-Samples/; done
Then reboot SuperDirt (or `~dirt.loadSoundFiles` per folder) so it re-globs.
## DO NOT edit a bank folder's contents after installing
`s "bank:N"` indexes files by the order SuperDirt's directory glob returns
them. Adding, removing, or renaming ANY file inside one of these folders
changes what index N plays — silently, no error, just a different sound at
that index next time the track runs. If you need to fix a sample, replace the
file in place with the exact same name.
See MANIFEST.json for the exact file count/size/listing of every bank, to
verify nothing shifted in transit.
"""
def do_pack(report: dict, dest: Path, include_stock: bool, include_unmapped: bool,
force: bool) -> None:
if dest.exists() and any(dest.iterdir()) and not force:
sys.exit(f"! {dest} exists and is non-empty — pass --force to pack into it anyway")
manifest = build_manifest(report, include_stock, include_unmapped)
need = manifest["bundle_bytes"]
probe = dest if dest.exists() else next(
(p for p in [dest, *dest.parents] if p.exists()), Path("/"))
free = shutil.disk_usage(probe).free
margin = int(need * 1.02) # small safety margin for filesystem overhead
if margin > free:
sys.exit(f"! not enough free space at {probe}: need ~{human(margin)}, "
f"have {human(free)} free. Consider /mnt/freebox (~3.6 TB) for a "
f"bundle this size.")
dest.mkdir(parents=True, exist_ok=True)
running = 0
classes = bundle_classes(report, include_stock, include_unmapped)
banks = [e for cls in classes for e in report["banks"][cls]]
print(f"packing {len(banks)} bank(s), {human(need)} -> {dest}")
for i, e in enumerate(banks, 1):
name, folder = e["name"], Path(e["source"])
real = folder.resolve() if folder.is_symlink() else folder
dst = dest / name
if dst.exists():
# --force re-pack: replace wholesale, never merge. A merge would leave
# stale files from a previous pack sitting alongside fresh ones, which
# is exactly the silent bank:N reshuffle rule 2 exists to prevent.
shutil.rmtree(dst)
shutil.copytree(real, dst, symlinks=False,
ignore=shutil.ignore_patterns(".*"))
running += e["bytes"]
print(f" [{i}/{len(banks)}] {name:30s} {human(e['bytes']):>10s} "
f"(running: {human(running)}/{human(need)})")
(dest / "MANIFEST.json").write_text(json.dumps(manifest, indent=2) + "\n")
(dest / "README.md").write_text(README_TEMPLATE.format(
generated=manifest["generated"], selector=manifest["selector"],
n_banks=manifest["bundle_banks"], total=human(manifest["bundle_bytes"])))
print(f"done. wrote MANIFEST.json + README.md into {dest}")
# ── cli ──────────────────────────────────────────────────────────────────
def main():
ap = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sel = ap.add_mutually_exclusive_group()
sel.add_argument("--tracks", nargs="+", metavar="TRACK",
help="explicit tracks (path or bare name, resolved like "
"setlist_samples.py's resolve_track)")
sel.add_argument("--set", metavar="NAME",
help="a named set/gig (--set list to see what's available)")
sel.add_argument("--all", action="store_true",
help="every .tidal under live/ and copycat/ (the default)")
ap.add_argument("--include-stock", action="store_true",
help="also pack banks that ship with a fresh Dirt-Samples install")
ap.add_argument("--include-unmapped", action="store_true",
help="also pack banks that resolve outside Dirt-Samples/extra "
"(e.g. tidal-drum-machines)")
ap.add_argument("--pack", metavar="DEST", type=Path,
help="actually copy the bundle here (default is a dry run)")
ap.add_argument("--manifest-only", metavar="PATH", type=Path,
help="write just the MANIFEST.json content to PATH, no copying — "
"this is what gets committed to git")
ap.add_argument("--force", action="store_true",
help="allow --pack into a non-empty DEST")
ap.add_argument("--json", action="store_true", help="machine-readable report")
ap.add_argument("--unresolved", action="store_true",
help="also show tokens that resolved to no bank — usually "
"structural placeholders or synth names, not missing "
"samples (see docstring rule 4)")
args = ap.parse_args()
selector, tracks = gather_tracks(args)
report = build_report(selector, tracks)
if args.manifest_only:
manifest = build_manifest(report, args.include_stock, args.include_unmapped)
args.manifest_only.parent.mkdir(parents=True, exist_ok=True)
args.manifest_only.write_text(json.dumps(manifest, indent=2) + "\n")
print(f"wrote manifest ({manifest['bundle_banks']} banks, "
f"{human(manifest['bundle_bytes'])}) -> {args.manifest_only}")
return
if args.pack:
do_pack(report, args.pack, args.include_stock, args.include_unmapped, args.force)
return
if args.json:
out = {
**report,
"include_stock": args.include_stock,
"include_unmapped": args.include_unmapped,
"bundle_bytes": sum(
e["bytes"] for cls in bundle_classes(report, args.include_stock, args.include_unmapped)
for e in report["banks"][cls]),
}
if not args.unresolved:
out.pop("unresolved", None)
print(json.dumps(out, indent=2))
else:
print_table(report, args.include_stock, args.include_unmapped, args.unresolved)
if __name__ == "__main__":
main()
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment