Skip to content

T
Tidal
Commit 22a9fc46 by PLN (Algolia)
Options

feat(tide-table): the first pass measured loudness, not similarity — three fixes and a reframe 

The first scan reported raw hash collisions and I then analysed the noise. Every
bank occupied 375 of 378 time bins. Three things were wrong, each traceable to
prior art rather than to tuning:

Offset consistency was missing entirely, and it is half the algorithm. Wang's
2003 Shazam paper is explicit: real matches agree on a time offset, so the
(query_t, ref_t) scatterplot shows a diagonal and the offset histogram spikes.
Counting collisions without that step counts coincidences.

Fixed offset is wrong anyway for this material. Sonnleitner, Arzt & Widmer
(ISMIR 2016, DJ mix monitoring) allow the offset line a free slope, because a DJ
plays material at a different rate — and a livecoder changing setcps does the
same, across 80 to 170 BPM in one set.

Raw counts measure loudness. Measured, not assumed: 8 of 18 banks had their
global peak in the final minute, at 1.3-2.5x density. The finale is loud and
broadband, so it makes more spectral peaks, so it matches everything better.
Scores are now normalised by each window's own query-hash count and by each
bank's reference size.

But the real problem was the reference unit, and PLN named the fix: "comparing
to prior recs, of each track of the opal songs for example (almost the same
lineup in the end!)". A 0.4s dry one-shot cannot survive a PA and a 320 kbps
encode — too few hashes, below the detection floor by construction. Meanwhile
Prod/Opal26_master/tracks holds FIFTEEN ear-verified per-track renders, 154 to
524 seconds each, same performer, same sample library, same patches.

That changes the task from sample detection to VERSION identification, where the
literature is unambiguous: when audio is related but not identical you stop
matching exact time-frequency points and match sequences of features instead
(Serra et al. on covers, McFee & Ellis on structure). So `crossmatch` embeds
15s windows as MFCC mean+std, chroma and spectral contrast, matches by cosine
against every reference window, and median-filters over 45s because a track
lasts minutes and a one-window flicker is noise by definition.

Standardising before the cosine is not cosmetic: raw MFCC coefficients differ in
scale by orders of magnitude, so without it the first two — level and spectral
tilt — decide everything, and level is exactly what a room mic gets wrong.

The docstring says what it cannot do. The three cafe_* tracks are the same
instruments at nearly the same tempo and no timbral profile will separate them,
so the output reports a runner-up margin and flags thin ones: a win by 0.01 is
not a win.

Also included, from the custom-mode thread: the generated Components entry table
(40 of 48 controls need changing, all of row A already matches), and the sniff
that caught Components driving the device with documented feature controls —
9F 0B 7F then CC 30 = 6 — plus an undocumented cmd 0x05 whose read/write forms
match the audiocontrol project's reverse-engineered protocol.
parent 470cc882
Showing with 370 additions and 0 deletions
armada/tide-table/lcxl3_custom_mode_v2map.md 0 → 100644
# ParVagues v2 map for a Launch Control XL 3 Custom Mode
Generated from `tools/lcxl_grid.py` — the one authored table. Do not hand-edit;
regenerate. Enter these CC numbers in Novation Components, per control,
left to right. Channel 1 for everything (SuperCollider's MIDIFunc.cc binds
with no channel filter, so only the number matters).
| physical row | col | factory CC | **set CC to** | changed? |
|---|---|---|---|---|
| Encoder row 1 (top) | 1 | 13 | **13** | |
| Encoder row 1 (top) | 2 | 14 | **14** | |
| Encoder row 1 (top) | 3 | 15 | **15** | |
| Encoder row 1 (top) | 4 | 16 | **16** | |
| Encoder row 1 (top) | 5 | 17 | **17** | |
| Encoder row 1 (top) | 6 | 18 | **18** | |
| Encoder row 1 (top) | 7 | 19 | **19** | |
| Encoder row 1 (top) | 8 | 20 | **20** | |
| Encoder row 2 (middle) | 1 | 21 | **29** | **yes** |
| Encoder row 2 (middle) | 2 | 22 | **30** | **yes** |
| Encoder row 2 (middle) | 3 | 23 | **31** | **yes** |
| Encoder row 2 (middle) | 4 | 24 | **32** | **yes** |
| Encoder row 2 (middle) | 5 | 25 | **33** | **yes** |
| Encoder row 2 (middle) | 6 | 26 | **34** | **yes** |
| Encoder row 2 (middle) | 7 | 27 | **35** | **yes** |
| Encoder row 2 (middle) | 8 | 28 | **36** | **yes** |
| Encoder row 3 (bottom) | 1 | 29 | **49** | **yes** |
| Encoder row 3 (bottom) | 2 | 30 | **50** | **yes** |
| Encoder row 3 (bottom) | 3 | 31 | **51** | **yes** |
| Encoder row 3 (bottom) | 4 | 32 | **52** | **yes** |
| Encoder row 3 (bottom) | 5 | 33 | **53** | **yes** |
| Encoder row 3 (bottom) | 6 | 34 | **54** | **yes** |
| Encoder row 3 (bottom) | 7 | 35 | **55** | **yes** |
| Encoder row 3 (bottom) | 8 | 36 | **56** | **yes** |
| Faders | 1 | 5 | **77** | **yes** |
| Faders | 2 | 6 | **78** | **yes** |
| Faders | 3 | 7 | **79** | **yes** |
| Faders | 4 | 8 | **80** | **yes** |
| Faders | 5 | 9 | **81** | **yes** |
| Faders | 6 | 10 | **82** | **yes** |
| Faders | 7 | 11 | **83** | **yes** |
| Faders | 8 | 12 | **84** | **yes** |
| Button row E (upper) | 1 | 37 | **41** | **yes** |
| Button row E (upper) | 2 | 38 | **42** | **yes** |
| Button row E (upper) | 3 | 39 | **43** | **yes** |
| Button row E (upper) | 4 | 40 | **44** | **yes** |
| Button row E (upper) | 5 | 41 | **57** | **yes** |
| Button row E (upper) | 6 | 42 | **58** | **yes** |
| Button row E (upper) | 7 | 43 | **59** | **yes** |
| Button row E (upper) | 8 | 44 | **60** | **yes** |
| Button row F (lower) | 1 | 45 | **73** | **yes** |
| Button row F (lower) | 2 | 46 | **74** | **yes** |
| Button row F (lower) | 3 | 47 | **75** | **yes** |
| Button row F (lower) | 4 | 48 | **76** | **yes** |
| Button row F (lower) | 5 | 49 | **89** | **yes** |
| Button row F (lower) | 6 | 50 | **90** | **yes** |
| Button row F (lower) | 7 | 51 | **91** | **yes** |
| Button row F (lower) | 8 | 52 | **92** | **yes** |
40 of 48 controls need changing; 8 already match.
## Why bother, when the driver also works
Two different jobs. A custom mode plays the corpus with NOTHING running —
no process to die mid-set — but cannot paint: colour and the OLED exist only
in DAW mode (tested). The driver gets paint but is one more moving part.
Firmware 1.1 makes them complementary rather than exclusive: *"The Mode
button can now be held to function as a momentary button, allowing for faster
Custom Mode switching."* So the custom mode is the fallback you reach for when
the instrumented path fails, and it costs one Components session to have.
armada/tide-table/lcxl3_sniff.txt 0 → 100644
# lcxl3 custom-mode sniff
[DAW] note_on channel=15 note=11 velocity=127 time=0
[DAW] control_change channel=6 control=30 value=6 time=0
[DAW] note_on channel=15 note=11 velocity=0 time=0
[MIDI] SYSEX len=10 cmd=0x05 00 20 29 02 15 05 00 15 00 06
[MIDI] SYSEX len=10 cmd=0x05 00 20 29 02 15 05 00 15 03 06
armada/tide-table/set_finder.py
...@@ -306,6 +306,126 @@ def cmd_scan(args) -> int: ...@@ -306,6 +306,126 @@ def cmd_scan(args) -> int:
return 0 return 0
# ------------------------------------------------------------------- locate ----
#
# `scan` counted raw hash collisions, and that is not the algorithm. Three fixes,
# each traceable to prior art rather than to tuning:
#
# 1. OFFSET CONSISTENCY (Wang 2003, the Shazam paper). Real matches agree on a
# time offset: the (query_t, ref_t) scatterplot shows a DIAGONAL, and the
# histogram of (query_t - ref_t) spikes. Without this step you are counting
# coincidences, which is exactly what the first pass reported.
#
# 2. FREE SLOPE (Sonnleitner, Arzt & Widmer, ISMIR 2016, "Landmark-based audio
# fingerprinting for DJ mix monitoring"). Do not require slope 1: a DJ — or a
# livecoder changing setcps — plays material at a different rate. This set spans
# 80 to 170 BPM, so a fixed-offset assumption is wrong on its face. We allow the
# offset to drift across the window instead of demanding one exact value.
#
# 3. DENSITY NORMALISATION. Measured, not assumed: 8 of 18 banks had their global
# peak in the final minute at 1.3-2.5x density. The finale is loud and broadband,
# so it makes more spectral peaks, so it matches EVERYTHING better. Raw counts
# therefore measure loudness. Each window is normalised by its own query-hash
# count and each bank by its reference size, so the number means similarity.
#
# What it still cannot do: rescue a reference too short to carry hashes. A 0.4s
# one-shot through a PA is below the detection floor by construction, which is why
# `--min-ref-dur` exists and why loops beat one-shots as references.
def cmd_locate(args) -> int:
librosa = _lib()
if not DB_OUT.exists():
sys.exit("set_finder: run `fingerprint` first")
db = np.load(DB_OUT, allow_pickle=False)
bank_names = [str(x) for x in db["bank_names"]]
track_banks = json.loads(str(db["track_banks"]))
order = np.argsort(db["hashes"], kind="stable")
h_sorted = db["hashes"][order]
b_sorted = db["banks"][order]
a_sorted = db["anchors"][order]
ref_size = collections.Counter(int(b) for b in b_sorted)
rec = pathlib.Path(args.recording)
dur = float(subprocess.run(
["ffprobe", "-v", "error", "-show_entries", "format=duration",
"-of", "csv=p=0", str(rec)], capture_output=True, text=True).stdout.strip())
W, HOPW = args.window, args.hop_s
OFF_BIN = args.offset_bin_frames
print(f"locating in {rec.name}: {dur/60:.1f} min, {W}s windows every {HOPW}s")
print(f"offset bin = {OFF_BIN} frames ({OFF_BIN*HOP/SR*1000:.0f} ms)\n")
results = [] # (t_start, {bank: (score, consistency, peak, nmatch)})
t = 0.0
t0 = time.time()
while t + W <= dur:
y, _ = librosa.load(str(rec), sr=SR, mono=True, offset=t, duration=W)
if len(y) < N_FFT:
break
S = np.abs(librosa.stft(y, n_fft=N_FFT, hop_length=HOP))
qh, qa = hashes_of(peaks_of(librosa.amplitude_to_db(S, ref=np.max)))
nq = max(1, len(qh))
per_bank = collections.defaultdict(list) # bank -> [offset]
if len(qh):
li = np.searchsorted(h_sorted, qh, side="left")
ri = np.searchsorted(h_sorted, qh, side="right")
for k in range(len(qh)):
lo, hi = li[k], ri[k]
if hi <= lo or hi - lo > args.max_ambiguity:
continue
for m in range(lo, hi):
per_bank[int(b_sorted[m])].append(int(qa[k]) - int(a_sorted[m]))
scored = {}
for bank, offs in per_bank.items():
if len(offs) < args.min_matches:
continue
o = np.array(offs)
binned = o // OFF_BIN
vals, counts = np.unique(binned, return_counts=True)
peak = int(counts.max())
consistency = peak / len(o)
# normalise by query density AND reference size: a cosine-ish score, so
# a big bank and a loud window cannot buy their way up the ranking.
norm = peak / np.sqrt(nq * ref_size[bank])
scored[bank_names[bank]] = (float(norm), float(consistency), peak, len(o))
results.append((t, scored))
pct = 100.0 * (t + W) / dur
print(f"\r {pct:5.1f}%", end="", flush=True)
t += HOPW
print(f"\n located in {time.time()-t0:.0f}s\n")
out = {
"recording": str(rec), "duration_s": dur,
"window_s": W, "hop_s": HOPW,
"windows": [
{"t": tt, "banks": {k: {"score": v[0], "consistency": v[1],
"peak": v[2], "matches": v[3]}
for k, v in sc.items()}}
for tt, sc in results
],
"track_banks": track_banks,
"method": "offset-consistency + density-normalised (Wang 2003; Sonnleitner 2016)",
}
p_out = REPO / "armada" / "tide-table" / "set_finder_locate.json"
p_out.write_text(json.dumps(out, indent=1))
print(f"wrote {p_out.name}")
print("\nbest window per bank (score = offset-peak / sqrt(query x ref), "
"consistency = peak/matches):")
best = {}
for tt, sc in results:
for bank, (score, cons, peak, nm) in sc.items():
if bank not in best or score > best[bank][0]:
best[bank] = (score, cons, peak, nm, tt)
for bank, (score, cons, peak, nm, tt) in sorted(best.items(), key=lambda kv: -kv[1][0]):
mm = f"{int(tt)//60}:{int(tt)%60:02d}"
print(f" {bank:<20} {mm:>7} score={score:.4f} cons={cons:.3f} "
f"peak={peak:<5} of {nm}")
return 0
def main(argv=None) -> int: def main(argv=None) -> int:
p = argparse.ArgumentParser(prog="set_finder", description=__doc__.splitlines()[0]) p = argparse.ArgumentParser(prog="set_finder", description=__doc__.splitlines()[0])
sub = p.add_subparsers(dest="cmd", required=True) sub = p.add_subparsers(dest="cmd", required=True)
...@@ -319,9 +439,183 @@ def main(argv=None) -> int: ...@@ -319,9 +439,183 @@ def main(argv=None) -> int:
help="skip hashes matching more than this many DB entries — a hash " help="skip hashes matching more than this many DB entries — a hash "
"everything shares carries no information") "everything shares carries no information")
q.set_defaults(func=cmd_scan) q.set_defaults(func=cmd_scan)
q = sub.add_parser("locate", help="offset-consistent, density-normalised detection")
q.add_argument("recording")
q.add_argument("--window", type=float, default=15.0)
q.add_argument("--hop-s", type=float, default=5.0)
q.add_argument("--offset-bin-frames", type=int, default=8)
q.add_argument("--min-matches", type=int, default=12)
q.add_argument("--max-ambiguity", type=int, default=20)
q.set_defaults(func=cmd_locate)
q = sub.add_parser("crossmatch", help="match against PRIOR RENDERS of the same tracks")
q.add_argument("recording")
q.add_argument("--refdir", default="/home/pln/Work/Sound/Prod/Opal26_master/tracks")
q.add_argument("--window", type=float, default=15.0)
q.add_argument("--hop-s", type=float, default=5.0)
q.add_argument("--smooth-s", type=float, default=45.0)
q.add_argument("--min-seg-s", type=float, default=40.0)
q.add_argument("--margin-warn", type=float, default=0.03)
q.set_defaults(func=cmd_crossmatch)
a = p.parse_args(argv) a = p.parse_args(argv)
return a.func(a) return a.func(a)
if __name__ == "__main__": if __name__ == "__main__":
sys.exit(main()) sys.exit(main())
# --------------------------------------------------------------- crossmatch ----
#
# PLN's idea, and it is the one that reframes the problem: "comparing to prior recs,
# of each track of the opal songs for example (almost the same lineup in the end!)".
#
# WHY THIS BEATS SAMPLE FINGERPRINTING HERE
# A 0.4s dry one-shot is below the detection floor of any landmark method — too
# few hashes to survive a PA, a room and a 320 kbps encode. But we hold FIFTEEN
# ear-verified per-track renders from OPAL-26 (154-524s each), played by the same
# performer from the same sample library with the same patches. That is minutes
# of reference per track instead of fractions of a second.
#
# WHY NOT JUST FINGERPRINT THOSE
# Because two performances of a livecoded track are not the same audio. The
# samples are identical but the arrangement, the gates and the tempo are not, so
# landmark matching (which demands near-identity) will find only the passages
# that happen to coincide. That is worth having as corroboration, and it is what
# `locate` now does — but it cannot be the primary lens.
#
# The primary lens is the one the VERSION-IDENTIFICATION literature uses when
# audio is related but not identical (Serra et al. on cover detection; McFee &
# Ellis on structure): compare SEQUENCES OF FEATURES, not exact time-frequency
# points. Here: window embeddings of MFCC + chroma + spectral contrast, matched
# by cosine, then smoothed over time because a track lasts minutes and a
# one-window flicker is noise by definition.
#
# WHAT IT CANNOT TELL YOU
# Tracks that share a sample palette will confuse it — the three cafe_* tracks
# are the same instruments at nearly the same tempo, and no timbral profile will
# separate them cleanly. Expect a family, not always a name, and read the
# runner-up margin: a win by 0.01 is not a win.
FEAT_DIM_NOTE = "20 MFCC (mean+std) + 12 chroma (mean) + 7 spectral contrast (mean)"
def _window_profiles(path, win, hop_s, sr=SR):
"""[(t_start, feature_vector)] for one audio file."""
librosa = _lib()
y, _ = librosa.load(str(path), sr=sr, mono=True)
if len(y) < sr * 2:
return []
hop = 512
mfcc = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=20, hop_length=hop)
chroma = librosa.feature.chroma_cqt(y=y, sr=sr, hop_length=hop)
contrast = librosa.feature.spectral_contrast(y=y, sr=sr, hop_length=hop)
fps = sr / hop
wf, hf = int(win * fps), int(hop_s * fps)
out = []
for s in range(0, max(1, mfcc.shape[1] - wf), hf):
e = s + wf
v = np.concatenate([
mfcc[:, s:e].mean(axis=1), mfcc[:, s:e].std(axis=1),
chroma[:, s:e].mean(axis=1), contrast[:, s:e].mean(axis=1),
])
out.append((s / fps, v.astype(np.float32)))
return out
def cmd_crossmatch(args) -> int:
refdir = pathlib.Path(args.refdir)
refs = sorted(p for p in refdir.iterdir()
if p.suffix.lower() in (".flac", ".wav", ".mp3", ".aiff"))
if not refs:
sys.exit(f"set_finder: no audio in {refdir}")
print(f"references: {len(refs)} tracks from {refdir.name}")
print(f"features: {FEAT_DIM_NOTE}")
print(f"windows: {args.window}s every {args.hop_s}s\n")
t0 = time.time()
R_vecs, R_track = [], []
for p in refs:
prof = _window_profiles(p, args.window, args.hop_s)
for _, v in prof:
R_vecs.append(v); R_track.append(p.stem)
print(f" {p.stem[:44]:<46} {len(prof):>4} windows")
R = np.stack(R_vecs)
print(f"\n{len(R)} reference windows in {time.time()-t0:.0f}s")
print("\nprofiling the query …")
q = _window_profiles(args.recording, args.window, args.hop_s)
Q = np.stack([v for _, v in q]); Qt = [t for t, _ in q]
print(f" {len(Q)} query windows")
# z-score on the REFERENCE distribution, then cosine. Standardising matters:
# raw MFCC coefficients differ in scale by orders of magnitude, so without it
# the first two coefficients (level and spectral tilt) decide everything, and
# level is precisely what a room mic gets wrong.
mu, sd = R.mean(axis=0), R.std(axis=0) + 1e-8
Rz = (R - mu) / sd
Qz = (Q - mu) / sd
Rn = Rz / (np.linalg.norm(Rz, axis=1, keepdims=True) + 1e-8)
Qn = Qz / (np.linalg.norm(Qz, axis=1, keepdims=True) + 1e-8)
S = Qn @ Rn.T # [query, ref_window]
tracks = sorted(set(R_track))
tidx = {t: i for i, t in enumerate(tracks)}
ref_of = np.array([tidx[t] for t in R_track])
# per-track score for a query window = best-matching window of that track
per_track = np.full((len(Q), len(tracks)), -1.0, dtype=np.float32)
for i in range(len(tracks)):
cols = ref_of == i
per_track[:, i] = S[:, cols].max(axis=1)
# temporal smoothing: a track lasts minutes, so a single-window flicker is noise
k = max(1, int(args.smooth_s / args.hop_s) | 1)
if k > 1:
from scipy.ndimage import median_filter # noqa: PLC0415
per_track = median_filter(per_track, size=(k, 1), mode="nearest")
best = per_track.argmax(axis=1)
bestv = per_track.max(axis=1)
second = np.partition(per_track, -2, axis=1)[:, -2]
margin = bestv - second
# collapse consecutive equal labels into segments
segs = []
for i, b in enumerate(best):
if segs and segs[-1][2] == b:
segs[-1][1] = Qt[i] + args.window
segs[-1][3].append(float(bestv[i])); segs[-1][4].append(float(margin[i]))
else:
segs.append([Qt[i], Qt[i] + args.window, int(b), [float(bestv[i])],
[float(margin[i])]])
segs = [s for s in segs if s[1] - s[0] >= args.min_seg_s]
print(f"\n{len(segs)} segments >= {args.min_seg_s}s "
f"(smoothing {k} windows = {k*args.hop_s:.0f}s)\n")
print(f"{'start':>7} {'end':>7} {'len':>6} {'sim':>5} {'margin':>6} track")
print("-" * 88)
for s0, s1, b, sims, margs in segs:
mm = f"{int(s0)//60}:{int(s0)%60:02d}"
ee = f"{int(s1)//60}:{int(s1)%60:02d}"
flag = "" if np.mean(margs) > args.margin_warn else " <- thin margin"
print(f"{mm:>7} {ee:>7} {s1-s0:>5.0f}s {np.mean(sims):>5.3f} "
f"{np.mean(margs):>6.3f} {tracks[b][:40]}{flag}")
out = REPO / "armada" / "tide-table" / "set_crossmatch.json"
out.write_text(json.dumps({
"recording": str(args.recording), "refdir": str(refdir),
"window_s": args.window, "hop_s": args.hop_s, "smooth_windows": k,
"features": FEAT_DIM_NOTE,
"tracks": tracks,
"segments": [{"start": s0, "end": s1, "track": tracks[b],
"sim_mean": float(np.mean(sims)),
"margin_mean": float(np.mean(margs))}
for s0, s1, b, sims, margs in segs],
"caveat": ("similarity is to a DIFFERENT performance of the same track. "
"A thin margin means the palette matched but the track did not; "
"tracks sharing instruments (the cafe_* family) will not separate."),
}, indent=1))
print(f"\nwrote {out.name}")
return 0
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment