Commit 37eb401f by PLN (Algolia)

fix(demucs): the OOM killer ended it after 2 of 14 sections — one bounded path for both modes

The first run died at 8.2 GB peak, killed by the kernel, having finished two
sections in 32 minutes of wall clock and nearly five hours of CPU. The arithmetic
was there to be done beforehand and I did not do it: a 532-second section is
188 MB of input, 750 MB of output across four sources, and apply_model's overlap
accumulator wants that again — call it 2 GB, times four workers, on a machine
already holding 18 GB of someone else's Gradle builds.

The fix was already written, in the other half of the same file. `global_pass`
chunked its work specifically because a 63-minute tensor would need ~12 GB, and
sections needed exactly the same treatment for exactly the same reason. So the
two modes now share one applier and differ only in which normalisation statistics
they are handed — which is the entire local/global distinction anyway, since
htdemucs' receptive field is 7.8 s and neither mode can give the model more
musical context than that. Sections pass their own stats; the global pass passes
the whole recording's.

Chunking turned out to be faster as well as smaller: four sections landed in the
time the unchunked version took to finish two, presumably from allocating a few
hundred MB repeatedly instead of gigabytes once. Memory peak is now 5.45 GB.

The unit also grew MemoryMax=10G / MemoryHigh=8G. A bounded algorithm should not
need it, but a cap turns any future regression into a failed unit rather than the
kernel choosing a victim on PLN's desktop while he sleeps.

Two details kept from the global implementation because they are load-bearing:
the chunk seam is joined with an equal-power ramp, since it sums two estimates of
the same audio and a linear fade would dip 3 dB right at the join; and the output
is length-exact, verified at 3969000 frames on all four stems for a 90 s input,
zero drift. That second one matters beyond tidiness — the EDA pass seeks into the
global stems by absolute time to compare them against each section, so any
accumulated drift would silently misalign every later track.

Also discarded the two sections the old code had produced. They were valid
separations, but they were made by a different code path, and a local-versus-
global comparison built on a mixed dataset would have a confound in it that no
amount of care downstream could remove.
parent b8059f81
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment