CoreML was 2.3× slower than the CPU
karanow separates vocals in real time on a Mac, which means an Apple Silicon machine with a Neural Engine sitting idle next to a busy CPU. Moving the model to it looked like the obvious win. It was measured, and it lost by more than a factor of two.
The numbers
The model is HTDemucs, re-exported to ONNX for a 1.5-second window. Running it through ONNX Runtime on the CPU execution provider takes 268 ms per window on an M2 Max, all cores. Running the identical graph through the CoreML execution provider takes 622 ms.
That is 2.3× slower on hardware specifically built for this shape of work, so the interesting question is not which is faster but where the time goes.
Partitioning is the whole story
The exported graph has 1291 operators. The CoreML execution provider claims 1072 of them. The remaining 219 have to run on the CPU, and because they are scattered through the graph rather than clustered at one end, the runtime cannot hand over the model once and take back a result. It has to split.
The split produces 98 partitions. Every boundary between them is a handoff: tensors marshalled out of CoreML’s representation, run on the CPU, marshalled back. Ninety-eight times per window, on a window that has 1.125 seconds to finish.
The arithmetic saved inside the CoreML partitions is real. It is simply smaller than the cost of crossing the boundaries around them.
And then there is the compile
CoreML compiles the model when the session is created. On this graph that takes 8.8 seconds. For a batch job that is amortised into nothing. For an app where a user presses a button and expects sound, it is 8.8 seconds of nothing before the first window has even been attempted.
It could be hidden — compile at launch, cache the result, warm the session in the background. But hiding a cost is only worth doing when paying it buys something, and here it buys a graph that then runs 2.3× slower.
What this does not say
It does not say CoreML is slow. It says this graph partitions badly on this execution provider, and a graph that partitions badly will lose to a homogeneous one regardless of how fast the accelerator is. A model authored for CoreML from the start, or exported with operator coverage in mind, would be a different measurement entirely.
It also does not say the CPU path is comfortable. 268 ms against a 1125 ms budget is roughly a quarter, which is four times the headroom needed — on this machine. That margin is the subject of another note.
Reproducing it
The benchmark is scripts/benchmark-separation.py in the karanow repository. It uses the same runtime and the same session options as the app, because a benchmark that configures the runtime differently from the product measures something the product never does.