Dictation while the key is still down

Dictation has worked for a while by recording and then transcribing: hold Caps Lock, say the thing, let go, wait, read the result. That is a usable arrangement and it does not feel like talking to a machine, because nothing happens while you are speaking.

There is now a second way to do it, and it is a different key on purpose.

Two keys, two behaviours

Holding AltGr streams. The speech core decodes every 300 ms of captured audio as it arrives and the words appear in the transcript strip while the key is still down. Letting go finishes the session.

Caps Lock is unchanged: record, then transcribe the whole recording with base.en. The two share the microphone and the speech core, so pressing one while the other is listening is ignored, with a line in the log saying so.

The words that appear while you talk are shown in two states. Text that can no longer change is in the ordinary ink; the last one or two tokens that the next chunk may still revise are dimmed behind it. That distinction is not decoration - a streaming decoder genuinely does retract its last couple of tokens, and showing them as settled and then changing them is worse than admitting they are provisional.

Nothing is interpreted or delivered before the key is released. The live words are display only and are cleared when the final text arrives, which then goes through exactly the same grammar session, preflight and dispatch as a Caps Lock transcript. One path for acting on speech, two paths for capturing it.

The Caps Lock lamp lights while AltGr is streaming, which is a small thing and the only feedback available on a machine where the microphone has no light of its own.

Matching the reference, token for token

The streaming decoder is a different model, not a different loop over the same one: the causally fine-tuned WhisperRT weights, with the encoder's keys and values and the cross-attention cache appended per chunk.

It follows the authors' greedy procedure exactly, including the parts that look like mistakes - a 201-sample first pad, a running-maximum mel, the retraction of the last two tokens - because the model was trained and measured that way, and a procedure that looks tidier is a procedure the weights have never seen.

It was developed and proved in a fork beside this tree rather than on the machine, which is the only way this class of thing can be checked. On the JFK sample and 44 LibriSpeech clips, clean and at 10 dB of noise, every chunk's tokens are identical to the authors' Python, 91 retractions included, with the mel agreeing to within 2.6e-5 and the encoder to within 6.4e-3.

Two things it needed on the way. An exact erf GELU for the streaming path, because the batch path's tanh approximation - which is whisper.cpp's - moved the encoder two per cent against PyTorch. And the multilingual control-token ids, which were simply wrong for two of them and had never been exercised, because the English-only models never use them.

What is not built

The streaming model is a second set of weights kept resident alongside base.en, read off the boot medium on the first AltGr press. A machine that has used both has two large blocks of memory it did not have at boot, and that has consequences elsewhere that are not this post's subject.

Interpretation of intent is still a grammar rather than anything cleverer. The plan is for the words to drive the machine through paths the person using it can shape themselves, which is what would give each machine its own character, and none of that exists yet. What exists is a transcript strip that fills in while you talk.