Patrick Lidstone
Self-hosted

CW — native Morse decoder (soft decisions + duration HMM)

Machine reading of hand- and machine-sent Morse, Bell-1977 style: a Goertzel tone detector with log-domain AGC feeds per-5-ms soft mark probabilities into a semi-Markov Viterbi decoder whose states are the Morse elements themselves — dit, dah, element/character/word space, idle — with log-normal duration priors scaled by a continuously tracked dit period. Pure Python stdlib, real time, and deliberately paranoid about emitting text that isn't really Morse.

Rafe project · app/radio/cwdecoder.py, test_cw.py · RX only (transmit is the radio's own keyer via CI-V cw_send) · approach credits: soft-decision statistical decoding after RSCW (PA3FWM); duration-model Viterbi after Bell, MIT 1977 · supersedes the "current decoder" described in cw-decoder-roadmap.md — this is that roadmap's phases 1–2 implemented

Abstract

CW is the hardest "easy" mode in the catalogue. There is no framing, no sync word, no FEC, no fixed speed — just a keyed tone whose durations carry the information, timed by a human wrist through QSB. A threshold-and-run-length decoder (the classic approach) collapses the moment fading shortens a dah or bridges a gap. This decoder keeps two pieces of information such decoders throw away: how confident each 5 ms slice is (a soft probability, not a hard mark/space), and how long Morse elements are allowed to be (explicit duration priors). A Viterbi search over the element grammar then finds the globally best reading of the last few seconds — healing fade-shortened dahs and bridged gaps that no local decision could — before text is committed.

The other half of the design is restraint: an SNR + spectral-purity gate decides when a CW signal exists at all, a two-means duration clustering confirms the signal has genuine Morse structure, and all decoded text is quarantined until that confirmation arrives — so speech, data modes and noise produce silence, not gibberish.

1. Front end

Constant Value Meaning
input 48 kHz s16le mono from the radio audio chain
DECIM 4 → 12 kHz boxcar-by-4 decimation
FRAME 60 samples = 5 ms hop analysis granularity
PHIST 900 hops = 4.5 s level history for floor/peak statistics
GATE_ON / GATE_OFF 9 dB / 5 dB activation hysteresis on peak−floor
K_LOGI 0.8 /dB logistic slope of the soft decision
MIN_SEP 9 dB minimum floor-to-threshold separation
EM_CLAMP −6 per-hop log-likelihood clamp
DELAY 500 hops = 2.5 s decode delay before commitment
COMMIT_EVERY 100 hops traceback cadence
IDLE_PEN 11 log-cost of entering the idle state

Each hop: a Goertzel single-bin energy at the tracked tone (a 200 Hz-wide 5 ms filter), averaged with the previous hop (a symmetric 2-hop boxcar — no IIR release tail to smear mark edges), and converted to dB. Every 10 hops the level history yields the noise floor (15th percentile) and peak (95th percentile); their gap is the SNR statistic driving the gate.

Log-domain AGC. A fast-attack/slow-decay peak tracker (decay 0.03 dB/hop ≈ 6 dB/s) rides the QSB envelope in dB, where multiplicative fading becomes a slowly varying additive offset. The decision threshold is the midpoint between the floor and the tracked peak (never closer than floor + 2·MIN_SEP), so both the threshold and the soft-decision slope follow fades automatically.

Soft decision. Each hop's dB level maps through a logistic to \(p(\text{mark})\) with slope 0.8/dB about the threshold; the decoder stores \(\log p(\text{mark})\) and \(\log p(\text{space})\), clamped at −6 so no single noise spike can veto a path (the same LLR-clamping logic as every soft decoder in the catalogue — maths guide §10.3).

2. Tone tracking

A separate 256-sample buffer at 6 kHz (~43 ms — short enough to fit inside a dah at 40 wpm, so a "keyed" sweep sees pure tone, not keying splatter) is swept 300–1000 Hz in 12.5 Hz steps with Goertzel:

  • Keyed sweeps fire when a mark run is in progress (≥ 8 hops, rate-limited to one per 80 hops; before activation ≥ 6 hops / 30, so a lone K can still be caught); timer sweeps fill the gaps every 130 hops. The two run on separate rate limits so a timer sweep landing in a gap can't starve the keyed ones.
  • A sweep reports the best bin, whether the line dominates the spectrum (> 6× mean; > 15× ⇒ counts as keyed even without a mark run) and whether it is pure (best > 3× the strongest peak more than 75 Hz away — keying sidebands live within ~75 Hz, a second station or speech formant rarely does).
  • Retune happens on a consistent, clearly stronger, pure peak away from the current tone (≥ 3× the current tone's power while active and agreeing with the previous sweep; ≥ 10× when idle). A small step (slow drift) is transparent to the 200 Hz hop filter; a big jump (> 75 Hz) invalidates history — an established session is flushed (station change), a young one is restarted, and pre-start buffered emissions are recomputed from raw samples at the new tone so the catch-up decode sees the signal's real first characters.

The gate. Activation needs two consecutive pure keyed sweeps within 30 Hz of each other (or one overwhelming sweep, > 20× mean) and ≥ 9 dB SNR. Deactivation: SNR < 5 dB, or two consecutive strong-but-impure sweeps — the latter aborts an unconfirmed session (it was never CW) but flushes a confirmed one (the CW ended or was steamrollered).

3. Speed bootstrap — two-means duration clustering

Threshold-run mark lengths (runs of 3–90 hops whose peak cleared the floor by ≥ 13 dB — keeps borderline noise pops out of the statistics) accumulate in a deque, and _cluster_tb fits two clusters in log-duration space with an outlier-shedding budget. The model: dit marks measure \(T+B\) hops, dahs \(3T+B\) — one shared dit period \(T\) and one keying bias \(B\) (marks stretched, spaces shortened, by the same amount — the physical signature of edge timing). From cluster centres \(c_1, c_2\):

\[T = \frac{c_2 - c_1}{2},\qquad B = c_1 - T\]

Accepted only when the structure is genuinely Morse-like: dah/dit ratio \(c_2/c_1 \in [1.8, 3.8]\), log-domain RMS deviation ≤ 0.22, \(T \in [3, 42]\) hops (≈ 6–68 wpm; wpm = 240/T at 5 ms hops), \(B\) small relative to \(T\). A tighter fit (RMS ≤ 0.17, ≥ 3 members per cluster, ratio 2.0–3.6) sets the strict flag used for text confirmation (§6). Degenerate cases have fallbacks: a long single tight cluster is read as dits; an over too short to cluster at all (a lone K, E E) gets _fallback_tb's dah/dit split or nearest-to-20 wpm single-length reading, so even two-mark overs decode.

The HMM starts once the gate is open, a speed estimate exists, and ≥ 20 mark hops are buffered (or the over already ended with ≥ 3 mark hops — decode the buffer with the fallback speed). At start, a trial traceback re-fits \((T, B)\) on the lattice's own segmentation of the buffered opening — the threshold bootstrap is often a couple of hops off — and rebuilds before anything is committed.

4. The semi-Markov Viterbi

Six states with an alternation grammar — a mark must be followed by a space and vice versa:

        ┌──────────── dit ◄──┐             mark states: dit, dah
        ▼                    │
  element-space        char-space        word-space        idle
        │                    ▲                                 (geometric, entry
        └──────────── dah ◄──┘                                  penalty 11; exits
                                                                to a mark)

Each element's duration is scored by a log-normal prior around its Morse-grammar mean, scaled by the tracked \((T, B)\):

Element Mean (hops) σ (log) Support
dit \(T + B\) (≥ 2.5) 0.25 \([\max(2, 0.45\,\mu_{dit}),\ 1.15\,m]\)
dah \(3T + B\) 0.22 \([0.85\,m,\ 1.9\,\mu_{dah}]\)
element space \(T - B\) (≥ 1.5) 0.35 \([\max(2, 0.4\,\mu_{esp}),\ 1.2\,s]\)
char space \(3T - B\) (≥ 3) 0.30 \([0.8\,s,\ 0.55(\mu_{csp}{+}\mu_{wsp})]\)
word space \(7T - B\) (≥ 6) 0.40 \([0.45(\mu_{csp}{+}\mu_{wsp}),\ 1.9\,\mu_{wsp}]\)

(\(m\) = dit/dah mean midpoint, \(s\) = esp/csp mean midpoint; prior cost \(= -\tfrac{z^2}{2} - \ln d - \ln\sigma - \ln\sqrt{2\pi}\) with \(z = \ln(d/\mu)/\sigma\), weight PW = 1.)

The lattice keeps cumulative mark/space log-likelihood sums, so scoring "a dah occupying hops \(j..i\)" is two array lookups + the prior — each hop's update scans the allowed duration window per state (a semi-Markov, i.e. explicit-duration, Viterbi rather than a per-hop HMM). The idle state has geometric duration with an 11-unit entry penalty: long dead air costs something to enter, then nothing — so the lattice neither invents structure in silence nor resists returning from it.

Delayed commitment. Every 100 hops, the best path is traced back and everything older than 2.5 s is committed — young enough to feel live, old enough that path merges have collapsed the ambiguity (the survivor-merge argument of Viterbi decoding — maths guide §10.2). Committed lattice history is trimmed; an over ends (flush) when silence exceeds \(\max(12T, 240)\) hops or the gate closes.

5. Speed tracking while decoding

Committed mark durations feed a second clustering (deque of 24, re-fit every 4 new marks). Small drift blends in at 0.3/update; a fit differing by > 12 % in \(T\) (or 1.5 hops in \(B\)) snaps and schedules a lattice rebuild, and if the tracked speed moved > 4 % since the priors were built they are rebuilt — so a mid-over speed change (or a bad bootstrap) retroactively re-decodes the uncommitted window. The threshold-run clusters keep running in parallel as a scale-lock check: firm disagreement > 25 % forces the correction.

6. Quarantine: only Morse gets out

All output is buffered until the session confirms:

  • the committed marks form a strict two-cluster fit (§3), and
  • the committed mark hops are near-saturated — geometric-mean \(p(\text{mark}) \ge 0.955\) over ≥ 40 mark hops (real CW measures ≥ 0.967 even at 40 wpm / 4 dB / deep QSB; speech-like envelopes top out ≈ 0.95 — the 5 ms edges of real keying are what saturate the hops), and
  • ≥ 5 characters, of which ≤ 60 % are E/T/I/# (the letters noise makes).

Confirmed sessions stream text live from then on. Sessions that end unconfirmed release only if short (≤ 10 marks with the contrast bar met, or ≤ 34 marks with contrast + a valid cluster + ≥ 2 characters) — a long unconfirmed session is structureless noise and is discarded. Sustained cluster failure (3 consecutive failed fits with ≥ 20 marks banked, without a crisp two-level contrast excuse) aborts: everything unconfirmed is dropped, and a 30 s distrust window suppresses the short-over release rules. One more nicety: a lone dit/dah at session start followed by a ≥ 9 dit-period gap is dropped as a lead-in noise blip rather than emitted as a one-letter word.

7. The Morse table

Decoded element strings look up directly; unknown patterns emit #:

A .-      B -...    C -.-.    D -..     E .       F ..-.    G --.
H ....    I ..      J .---    K -.-     L .-..    M --      N -.
O ---     P .--.    Q --.-    R .-.     S ...     T -       U ..-
V ...-    W .--     X -..-    Y -.--    Z --..
0 -----   1 .----   2 ..---   3 ...--   4 ....-   5 .....
6 -....   7 --...   8 ---..   9 ----.
. .-.-.-   , --..--   ? ..--..   / -..-.    = -...-    + .-.-.
( -.--.    ) -.--.-   @ .--.-.   : ---...   - -....-   " .-..-.
_ ..--.-   ' .----.   ! -.-.--
<SK> ...-.-   <AS> .-...   <KA> -.-.-   <ERR> ........

8. Validation

test_cw.py synthesises keyed CW at 48 kHz with 5 ms raised-cosine edges, calibrated noise, and optional QSB, then requires exact text:

Text wpm Tone SNR Stress
CQ CQ CQ DE M0ABC M0ABC K 20 650 25 dB baseline
CQ TEST 5NN 73 12 700 20 dB slow
THE QUICK BROWN FOX 599 28 500 15 dB fast, all letters
R TU 73 E E 35 800 20 dB very fast, single-dit words
HELLO WORLD 20 650 8 dB weak
CQ DX DE G4XYZ 24 600 12 dB weak-ish
CQ CQ DE F5ABC K 16 500 18 dB QSB 0.55 deep fading

9. Limitations and pointers

  • One signal at a time per audio channel: the tracked tone follows the strongest pure line in 300–1000 Hz; a second CW signal is competition, not a second decode. (Multi-signal decoding is the natural next phase in cw-decoder-roadmap.md.)
  • Text lags real time by the 2.5 s commitment delay by design.
  • No linguistic priors: the decoder is deliberately language-agnostic (a character model would help marginal copy and hurt callsigns/serials).
  • Transmit is not an audio modem here — keying is delegated to the radio's own CW keyer over CI-V (cw_send).

Related: Feld Hell and ISM/OOK (the other on-off-keyed modes, with fixed machine timing where CW has a human wrist), the Beginner's Tour stage 1, and the maths guide on detection and Viterbi.