The integer dot that won, and the core farm that did not

Dictation on this machine is a port of whisper.cpp running on a core of its own. It worked and it was too slow to feel live: about 4.7 times real time with the plain scalar maths, which is a machine you talk to and then wait for. The encoder is roughly ninety per cent of that, so that is where the work went.

Measuring the same sentence twice

The first thing built was not an optimisation. Comparing transcribe times across builds is meaningless if the input changes, and live dictation changes every time - a different sentence, a different length, a different amount of silence at the front.

remote.sh wbench transcribes a baked 11.0 second clip instead, the JFK sample whisper.cpp ships, 16 kHz mono, loaded from the boot medium. It needs no microphone, so a soft reboot is enough to run it, and it reports the transcript alongside the millisecond timing. Measuring the code rather than the sentence is the whole point of it.

What worked

SSE2 first. The four-accumulator dot product was written to auto-vectorise and LLVM left it scalar, emitting mulss and addss. Rewritten with explicit intrinsics, four wide, with the lanes laid out to accumulate the same terms in the same order as the scalar path - so the result is bit-identical and the transcript does not change, it just runs about four times faster on the inner loop.

Then AVX2 with FMA, eight floats per instruction and the multiply-add folded into one. Kaby Lake has AVX2 and not AVX-512, so that is the end of the width available here. That took it to 2.18 times real time.

Then the integer dot, which is what the port was always meant to use: the weights stay in their quantised Q5_1 form, the activations are quantised to Q8_1, and the dot is int8 against int8 with no weight ever widened to a float. Measured on the X1 with base.en, that is 1.38 times real time, transcript unchanged. So 4.68 down to 1.38, with the numerical result checked against the float path rather than checked by listening to it.

Two optimisations that lost

Getting the integer dot to actually win took two goes, and both failures are more instructive than the success.

The first version finalised each block on its own: a scalar horizontal sum per 32-element block, because every block carries its own scale and offset. That per-block overhead swamped the integer gain, badly in the decoder's vocabulary projection, which is 51,864 rows. It measured 2.47 times real time - correct, bit for bit, and slower than the float path it replaced. The fix is to accumulate each block's scaled sum into a SIMD float accumulator across blocks and do one horizontal sum at the end, which is what whisper.cpp does and which I had read without understanding why.

That got it to 2.34, still losing. The second fault was amortisation. The encoder's matrix multiply calls the dot once per token, and the dot unpacks the weight row's nibbles inside itself - so for every weight row the same bytes were unpacked again for all roughly 1,500 tokens. The float path never had this problem because its row-block structure dequantises each row once and reuses it. The integer path had thrown that away: 1,500 times the necessary work, hidden inside a function that was faster at the thing it was measured on.

Unpack each row once, dot it against all the tokens, and it lands at 1.38.

The core farm that did not pay

There are eight cores, so running the encoder across several of them should get close to another multiple. That was built, merged, and measured with this same benchmark, and it did not improve wall-clock time at all. The decoder is autoregressive, so it cannot be parallelised that way, and the fork-join and allocation overhead on the encoder ate the gain.

It was dropped rather than carried as complexity that does not pay for itself. The banked win is the integer dot.

What is not built

1.38 times real time is not real time. It is close enough that a short phrase feels responsive and a long one does not, and the remaining gap is not going to close with another SIMD trick on this part.

One clip mis-transcribed "fleece was white as" as something unrepeatable. The numerical tests say the integer path tracks the float path to within quantisation error, so that is almost certainly base.en itself rather than the port, but I have not proved it on that phrase and should not claim it.

AVX2 is enabled on the speech core by setting CR4.OSXSAVE and the relevant bits of XCR0 on that core at startup. Nothing yet re-establishes that after a resume from S3 or S4, which means the first dictation after a sleep is running on a core whose state was set up by a boot that no longer happened. That is a known hole and it is written down.