Patrick Lidstone
Self-hosted

RVQ-Voice: a native residual-vector-quantized digital voice codec for ham radio

A self-contained 1.6–3.2 kbps speech codec that marries a classical source–filter vocoder to a SoundStream-style residual vector quantizer.

Rafe project · app/radio/rvqvoice.py, rvqvoice_stream.py · experimental

Abstract

RVQ-Voice is a low-bitrate digital-voice mode built entirely in-repo: no external binaries, no neural-network framework, no downloaded model weights — pure NumPy at runtime. It analyses 8 kHz speech into a compact per-frame parameter vector using classical linear-predictive source–filter analysis, then compresses that vector with a residual vector quantizer (RVQ) — the same quantization structure used by modern neural codecs (SoundStream, EnCodec) — whose codebooks are the only learned component and are trained by k-means on a small in-repo speech corpus. The resulting 1.6–3.2 kbps bitstream is protected by a rate-½ convolutional code with interleaving and a CRC, and is carried over the project's existing audio data modems exactly as the text digimodes are. On held-out speech the quantizer is near-transparent to the spectral envelope (log-spectral distortion ≈ 2.1 dB at all three bitrates, versus a 2.0 dB copy-synthesis ceiling), and the channel code corrects to roughly 5 % raw bit-error rate. The novelty for amateur radio is the pairing: RVQ bitrate-scalable compression — normally the preserve of GPU neural codecs — running on a Raspberry-Pi-class box with zero dependencies.

1. Motivation

Amateur digital voice today means one of a few external stacks: FreeDV/Codec2, M17's Codec2, or the proprietary AMBE/IMBE families behind DMR/D-STAR/P25. Rafe already bridges to these through digivoice.py, but every one of them is an outside binary. The rest of this project has moved the digital text modes in-house (native FT8, the WSJT-X family, the fldigi family) for a reason: no build dependencies, full control, and something genuinely ours. RVQ-Voice does the same for voice.

The interesting question is whether the modern idea in speech coding — residual vector quantization, the engine inside neural codecs — can be divorced from the neural network and the GPU, and applied to a lightweight classical front-end that runs anywhere. It can. The learned intelligence in a neural codec lives largely in its quantizer's codebooks; the convolutional encoder/decoder mostly provide a good feature space. If we replace that learned feature space with a classical one that is already known to be an excellent low-dimensional description of speech — the source–filter model — then the RVQ can do its job on top with nothing but k-means-trained codebooks.

2. Background

2.1 The source–filter model

Human speech is well modelled as an excitation (the glottal source: a quasi-periodic pulse train when voiced, turbulent noise when unvoiced) passed through a slowly time-varying vocal-tract filter (the resonances/formants). If we can estimate, per short frame, (a) the filter and (b) a few excitation parameters (pitch, voicing, gain), we can reconstruct intelligible speech from a tiny number of values. This is the basis of every sub-3-kbps vocoder from LPC-10 (1976) through MELP and Codec2. The filter is captured by linear prediction (LPC): each sample is predicted from a weighted sum of the previous p samples; the prediction-error filter A(z) is the inverse of the vocal-tract response.

2.2 Residual vector quantization

Vector quantization (VQ) replaces a continuous vector with the index of the nearest entry in a trained codebook. A single codebook large enough for transparent speech would be astronomically large, so residual VQ stages it: the first codebook gives a coarse approximation, the second quantizes what the first missed, the third quantizes what the first two missed, and so on:

x ≈ c₁[i₁] + c₂[i₂] + … + c_N[i_N]
r₀ = x ;  iₖ = argminⱼ ‖rₖ₋₁ − cₖ[j]‖² ;  rₖ = rₖ₋₁ − cₖ[iₖ]

Two properties matter here. It is exponentially efficient: N codebooks of K entries span Kᴺ points with only N·K stored vectors. And it is bitrate-scalable by construction: because stage k only refines, decoding the first n < N stages still yields a valid (coarser) reconstruction — so one trained model serves several bitrates just by sending fewer stages. SoundStream and EnCodec use exactly this structure (on a neural latent, trained with straight-through gradients); we use it on a classical feature vector, trained with plain k-means — which is the original way RVQ codebooks were built and needs no back-propagation.

3. System overview

                         ANALYSIS (encoder)
  speech 8k ─▶ frame 20ms ─▶ LPC(10) ─▶ LSF(10) ┐
                          └▶ pitch / voicing ────┼─▶ feature v  ─▶ RVQ ─▶ indices
                          └▶ gain ───────────────┘   (13-dim)     (N stages)
                                                                      │
   indices ─▶ pack ─▶ conv-FEC(½,K7) ─▶ interleave ─▶ [UW|hdr|·|CRC] ─▶ bits
                                                                      │
                                                                  [ modem ] ── air ──▶
                         SYNTHESIS (decoder)  ── the mirror of the above ──

The codec proper (§4–6) is rvqvoice.py; the channel layer (§7) is rvqvoice_stream.py. Everything downstream of "bits" is an ordinary audio data modem — the same layer the text modes ride on — so RVQ-Voice is deliberately modem-agnostic.

Key parameters: 8 kHz sampling, 20 ms frames (50 frames/s), LPC order 10, feature dimension 13, RVQ 8 × 256-entry codebooks (nested).

4. Analysis front-end

Each 20 ms frame (160 samples) produces a 13-element feature vector [ LSF₁…₁₀ , log₂F0 , voicing , log-gain ].

4.1 Linear prediction (LPC)

The windowed frame's autocorrelation r[0..10] is computed and a mild lag-window (≈ 60 Hz bandwidth expansion) plus a white-noise floor are applied for numerical stability. Levinson–Durbin recursion then solves the Toeplitz normal equations for the order-10 LPC polynomial A(z) = 1 + a₁z⁻¹ + … + a₁₀z⁻¹⁰ in O(p²):

kᵢ = −(rᵢ + Σⱼ aⱼ rᵢ₋ⱼ) / Eᵢ₋₁ ;  aᵢ = kᵢ ;  aⱼ ← aⱼ + kᵢ a_{i−j} ;  Eᵢ = Eᵢ₋₁(1 − kᵢ²)

4.2 Line spectral frequencies (LSF)

Raw LPC coefficients quantize and interpolate badly (small changes can make the filter unstable). We convert to line spectral frequencies, the standard representation for quantized speech, which are ordered, bounded in (0, π), and interpolate stably. LSFs are the angles of the roots of the symmetric and antisymmetric polynomials

P(z) = A(z) + z⁻⁽ᵖ⁺¹⁾A(z⁻¹)      Q(z) = A(z) − z⁻⁽ᵖ⁺¹⁾A(z⁻¹)

whose roots lie exactly on the unit circle and interlace. The trivial roots (z = −1 in P, z = +1 in Q) are discarded; the remaining p angles are the LSFs. The inverse rebuilds P and Q as products of second-order factors 1 − 2cos(ωᵢ)z⁻¹ + z⁻² and forms A(z) = ½(P(z) + Q(z)). The implementation round-trips LPC→LSF→LPC to 1 × 10⁻¹⁵ (a unit test); getting there required care with NumPy's descending-power polynomial convention versus our ascending z⁻¹ arrays.

4.3 Pitch and voicing

Estimating the glottal period robustly is the hardest classical sub-problem (octave errors — mistaking a harmonic for the fundamental — are endemic). We:

  1. low-pass the signal (windowed-sinc, ~900 Hz) over a longer 480-sample window than the coding frame, so several pitch periods are present for a stable autocorrelation and the formant structure that biases short lags is removed;
  2. take the autocorrelation peak in the 54–364 Hz lag range;
  3. apply iterative octave-down correction: while a peak one octave lower (twice the lag) is ≥ 80 % as strong, adopt it — defeating the harmonic-locking that pushes estimates too high;
  4. after all frames, median-smooth the voiced-F0 track (5-tap) to remove isolated octave jumps without blurring real pitch movement.

Voicing is the normalized autocorrelation peak height (0…1); below 0.35 the frame is treated as unvoiced (F0 = 0).

4.4 Gain

The frame RMS, stored as log-gain; synthesis rescales the reconstructed frame to match it exactly, which decouples loudness from the (approximate) excitation model.

5. Quantization: the RVQ

The 13-dim features span wildly different ranges (LSFs 0…π, log₂F0 up to ~8, voicing 0…1, log-gain ≈ −10…0). Before quantization each dimension is z-scored (the training mean/σ are stored with the codebooks) so k-means and the nearest-neighbour search weight all dimensions commensurately.

The RVQ is 8 stages of 256 entries (8 bits each). At 50 frames/s, each stage contributes 8 × 50 = 400 bps, so:

stages n bits/frame bitrate residual RMS (z-space, training)
4 32 1600 bps 0.130
6 48 2400 bps 0.064
8 64 3200 bps 0.032

Because the codebooks are nested, all three bitrates come from one trained model — you simply transmit fewer stages. Encoding is the greedy residual search of §2.2; decoding sums the selected codebook vectors and de-normalizes.

5.1 Training

train_rvq.py (run once, offline; the only place SciPy is used) analyses the corpus into ~22 000 feature frames, z-scores them, and trains each stage by k-means (k-means++ init) on the running residual. The output is a 202 KB _tables/rvq_codebooks.npz (8 × 256 × 13 floats + mean/σ), committed to the repo like any other data table. Corpus: the Open Speech Repository Harvard sentences at 8 kHz — ~8 minutes, 11 speakers (male and female) — with list 0010 held out for evaluation.

6. Synthesis back-end

Per frame the decoder:

  1. interpolates the LSFs from the previous frame across 4 sub-frames (40 samples each) and converts each to LPC — smooth spectral motion, no clicks;
  2. builds the excitation: voiced frames use a phase-continuous pulse train at F0 (a running phase accumulator places pulses, so there are no period-boundary discontinuities) mixed with noise scaled by (1 − voicing); unvoiced frames use white noise;
  3. filters the excitation through the all-pole synthesis filter 1/A(z), carrying the filter state across sub-frames and frames;
  4. rescales the frame to the decoded gain.

Objective quality (held-out male speaker, re-analysis log-spectral distortion of the reconstructed vs original envelopes):

condition envelope LSD
copy-synthesis (no quantization) — the ceiling 2.04 dB
RVQ 1600 bps 2.14 dB
RVQ 2400 bps 2.07 dB
RVQ 3200 bps 2.09 dB

The striking result is that the quantizer is near-transparent to the spectral envelope even at 1600 bps — the LSF sub-vector is compressed with almost no envelope penalty; the remaining gap to natural speech is in the excitation model (§9), not the spectral quantization. (LSD measures the envelope, which is what LPC codes and what dominates intelligibility; it does not capture excitation naturalness, so treat it as a correctness/where-the-bits-go metric, not a MOS.)

7. Channel layer — "your own FEC/modem"

rvqvoice_stream.py turns RVQ indices into a robust, self-describing bitstream:

  • Packing — each stage index is one byte; a frame of voice-frames is packed row-major with a 3-byte header (n_stages, frame count).
  • CRC-16 (CCITT 0x1021) over header + payload for whole-frame integrity.
  • FEC — the rate-½, constraint-length-7 convolutional code (generator polynomials 0x6D / 0x4F, the same NASA-standard code the MFSK text modes use), decoded by a hard-decision Viterbi with full traceback.
  • Interleaving — an 8-column block interleaver spreads the coded bits so a fading burst is scattered into isolated errors the convolutional code can fix.
  • Framing — a 16-bit unique word (0x2B47) precedes each frame for the modem's sync search.

Measured error performance (rate-½, hard decision):

raw channel BER decoded BER
1 % 0.00 %
3 % 0.06 %
5 % 0.04 %
8 % 2.7 %

A 40-bit contiguous burst dropped into a frame is fully corrected (interleaver + FEC), CRC-verified. The FEC roughly doubles the rate (a 1600 bps voice stream → ~3.2 kbps on the channel); this is the usual robustness trade for HF/VHF voice and is tunable (puncturing for less overhead, or a stronger code for more).

7.1 The audio modem (rvqvoice_modem.py)

MFSK (as the text modes use) is far too slow for a ~3 kbps voice stream, so RVQ-Voice ships a differential QPSK burst modem: 2 bits/symbol, and differential so no absolute carrier-phase recovery is needed — each symbol is decoded from its phase step. Each RVQ super-frame is one self-contained burst:

[ 32-symbol preamble (sharp autocorrelation) | framed payload | guard ]

The receiver matched-filters against the known preamble to find each burst, then does energy-locked fine timing (the sample offset maximizing per-symbol energy — sharp for rectangular pulses) before differentially demodulating. Defaults: 8 kHz, 1600 baud, 1600 Hz carrier → 3200 bit/s in ~0–3.2 kHz (an FM channel, or clean SSB). Measured raw modem BER: 0 % clean, ~0.0004 at 6 dB SNR, robust to arbitrary sample-timing offsets. Two subtleties that had to be right: the preamble must be non-periodic (a periodic one gives the matched filter ambiguous peaks and silently loses half the bursts), and the streaming demod must only decode once a whole burst is resident (so bursts straddling audio-chunk boundaries aren't dropped).

The codec/framing are still deliberately modem-agnostic — the same bitstream could ride any of the project's data modems — but the DQPSK modem makes RVQ-Voice a complete, self-contained mode out of the box.

7.2 App integration (rvqvoice_app.py)

NativeVoice is a drop-in for the external-decoder DigiVoice class, so the app drives it identically (RX feed(pcm48k), TX feed_tx(pcm48k), callbacks for decoded speech / modem audio), but in-process — no subprocess. It bundles superframe voice frames (20 ms each) per DQPSK burst, handles the 48 k↔︎8 k rate change with a stateful decimator (per-chunk resampling would inject edge transients), and carries the vocoder synth state across bursts. It registers as the rvqvoice (1600 bps) and rvqvoice-hi (2400 bps) modes in VOICE_MODES alongside FreeDV/M17/P25. Full loopback — real speech → mic path → DQPSK → noisy channel → speaker path — recovers 16/16 bursts down to 8 dB channel SNR.

8. Results summary

  • End-to-end: speech → analysis → RVQ → FEC frame → DQPSK modem → channel → demod → Viterbi → RVQ⁻¹ → synthesis, validated through the app's streaming mic/ speaker path — real speech recovers 16/16 bursts down to 8 dB channel SNR.
  • Compression: 1.6 / 2.4 / 3.2 kbps from one nested model; envelope quantization near-transparent (§6).
  • Robustness: corrects to ~5 % channel BER; burst-tolerant; CRC-guarded.
  • Footprint: pure NumPy at runtime; 202 KB of codebooks; runs on the prod box.
  • Tests: 8 dedicated tests (LPC↔︎LSF exactness, vocoder stability, RVQ nesting/bitrates, FEC correction, frame + burst round-trip, full chain) within the project's 71-test suite.

9. Limitations and future work

This is an honest first cut. Known gaps, roughly in order of perceptual impact:

  • Excitation naturalness. The simple mixed pulse/noise excitation is the main quality limiter — it sounds "vocoder-ish" (a touch buzzy/mechanical). The classical remedies are well known: multi-band voicing (MELP-style per-band pulse/noise mixing), pulse dispersion, adaptive spectral enhancement, and a postfilter. Each is a bounded addition.
  • Pitch robustness. Octave errors are much reduced but not eliminated; a proper tracker (YIN/RAPT-style dynamic programming) would firm up voiced quality.
  • Perceptual weighting. RVQ uses plain Euclidean distance on z-scored features. Weighting LSF distance by spectral sensitivity (as in perceptual VQ) would spend bits where the ear notices.
  • No packet-loss concealment yet — a dropped burst is a ~0.5 s gap; feature interpolation/repeat would hide short losses gracefully.
  • Latency: bundling superframe frames per burst adds ~0.5 s each way. Smaller super-frames cut latency at the cost of more per-burst framing overhead.
  • SSB efficiency: the DQPSK modem fits ~3.2 kHz (fine for FM/VHF, marginal for a crowded SSB channel). An OFDM waveform (à la FreeDV) would be more HF-spectrum-efficient — the codec/FEC layers are unchanged by that choice.
  • Training corpus is small (~8 min, English, clean). More/broader speech would generalize the codebooks; the pipeline supports any 8 kHz corpus.
  • Objective metrics only — no listening-test MOS. WAVs are produced for subjective judgement.

None of these change the architecture; they are refinements on a working base.

10. Implementation & reproduction

file role
app/radio/rvqvoice.py vocoder (analysis/synthesis) + RVQ encode/decode
app/radio/rvqvoice_stream.py FEC + interleave + framing + Viterbi
app/radio/rvqvoice_modem.py DQPSK burst modem (bits ↔︎ audio)
app/radio/rvqvoice_app.py NativeVoice — the app-facing streaming mode
app/radio/_tables/rvq_codebooks.npz trained codebooks + normalization (202 KB)
train_rvq.py offline k-means trainer (SciPy, offline only)
test_rvqvoice.py test suite

Runtime API:

from app.radio import rvqvoice as V, rvqvoice_stream as S
feats, idx = V.encode(pcm8k, n_stages=4)   # analyse + quantize (1600 bps)
bits = S.build_frame(idx)                   # FEC + framing -> hand to a modem
idx2, ok = S.parse_frame(bits)              # after the modem, on RX
speech = V.decode(idx2)                      # RVQ^-1 + synthesis -> 8k float

Retrain the codebooks on any 8 kHz mono corpus:

python train_rvq.py "/path/to/corpus/*.wav" --stages 8 --k 256

Everything at runtime is NumPy; the only learned artefact is the committed codebook file, trained offline exactly as the LoRa/Meshtastic constant tables were transcribed once and committed.

11. References

  1. N. Zeghidour et al., "SoundStream: An End-to-End Neural Audio Codec," 2021 — the RVQ structure and bitrate-scalable nesting.
  2. A. Défossez et al., "High Fidelity Neural Audio Compression (EnCodec)," 2022.
  3. D. Rowe, Codec2 — the reference open-source low-bitrate speech codec and the source–filter / LSP design lineage.
  4. F. Itakura, "Line spectrum representation of linear predictor coefficients," 1975 — LSFs.
  5. A. McCree, T. Barnwell, "A mixed excitation LPC vocoder model (MELP)," 1995 — the excitation improvements in §9.
  6. Open Speech Repository — the public-domain Harvard-sentence training corpus.