Refactoring plan: a shared DSP/coding toolkit for the native protocols¶
Status: implemented (2026-07-08) on the refactor branch. Author aid: Claude.
Date: 2026-07-07.
Implementation note (2026-07-08). Delivered on
refactor, one tested commit per library/migration. All twelve primitives landed asapp/radio/dsp/(bits,crc,sync,galois,reedsolomon,convolution,blockcodes,filters,scramble,interleave,modems/{fsk,gmsk,psk},framing/hdlc), each with atest_dsp_<name>.pyasserting the named presets reproduce their origin modules bit-for-bit, and ~35 protocol modules migrated onto them with the fulltest_*.pysuite green at every step. Scope held to the plan: LDPC codes, the vocoder stack, and mode-specific data tables (DAB/DVB symbol/cell/time interleavers, the DMR BPTC 181-step, the DVB-S2 physical-layer Gold-code scrambler, RNG-seeded whiteners) were left in place as non-goals. Stateful modem timing/carrier recovery (Gardner/Costas loops, AIS/C4FM open-eye search) stays in the protocol modules; only the deterministic modem DSP was unified. Compatibility shims (e.g.p25/fec.py) are kept while importers remain. Seeapp/radio/dsp/README.md.
1. Context and motivation¶
app/radio/ now holds ~75 top-level protocol modules plus subpackages
(sat/, tetra/, dab/, datv/, p25/, hamdrm/, mercury/, ftx/,
jtx/, js8/, lorasdr/, rtl433/). Each was landed clean-room, one at a
time, and each rolled its own copy of the same handful of DSP and
forward-error-correction primitives. A survey of the package (counts are
regex matches, so approximate but directionally exact) shows the duplication:
| Primitive | Distinct implementations | Representative files |
|---|---|---|
| Root-raised-cosine filter | 4 near-identical | p25/demod.py, tetra/dsp.py, sat/qpsk_rx.py, datv/dvbs.py |
| Convolutional encoder + Viterbi | ~9 (K=3/5/7, various rates) | sat/viterbi.py, tetra/coding.py, m17.py, ysf.py, dstar.py, dab/fec.py, hamdrm/fec.py, datv/dvbs.py, rvqvoice_stream.py |
| GF(2^m) exp/log field build | ~9 | p25/fec.py (2^6), sat/rs.py (0x187), vdl2.py/dmr.py (0x11D), hamdrm/rs.py, dab/audio.py, ardop_native.py, datv/dvbs.py |
| Reed–Solomon codec | ~4 | sat/rs.py, hamdrm/rs.py, vdl2.py, dmr.py (RS12,9), dab/audio.py |
| CRC / FCS | ~15 variants across 26 files | aisdecode._fcs, acars.bcs, p25/trunk.crc16, tetra/coding.crc16_fcs, m17.crc_m17, nxdn.crc6/crc12, adsb.crc24, js8/crc, mercury/crc |
| Bit pack / unpack helpers | duplicated in ~every module | _to_bits/_from_bits in imbe, ambe, dmr, dstar, ysf, nxdn, vdl2, p25/voice, … |
| Interleaver / deinterleaver | ~15 | inmarsat, tetra/coding, m17, mt63, mercury/interleaver, dstar, ysf, datv/dvbs |
| Scrambler / PN / LFSR | ~19 | sat/ccsds.pn_sequence, inmarsat._pn, p25/voice._pn, tetra.scramble, m17._randomize, dstar/ysf._prbs, vdl2._scramble, datv/dvbs |
| Sync-word correlator (Hamming-tolerant) | ~8 identical | p25/frame.find_sync, sat/ccsds.find_asm, inmarsat._find_uw, tetra/burst.find_sync, dmr.find_sync, dstar._find |
| Golay / Hamming block codes | 1 shared + inline | p25/fec.py (already imported by ambe,dmr,ysf,vdl2) |
| GMSK / MSK modem | 2 | aisdecode (GMSK), acars (MSK) |
| PSK modem (BPSK…D8PSK) | ~4 | tetra/dsp (π/4-DQPSK), sat/qpsk_rx (BPSK/QPSK + Gardner/Costas), inmarsat (BPSK), vdl2 (D8PSK) |
| LDPC | ~5 mode-specific codes | datv/dvbs2, ftx/ldpc, js8/ldpc, mercury/ldpc, hamdrm |
The cost of this is real: a correctness fix or a performance improvement to, say, the Viterbi ACS loop has to be made in nine places; each new protocol re-derives (and can re-introduce bugs into) code that already exists; and the subtle protocol-specific conventions (bit order, polynomial endianness, normalization) are invisible because they're buried in near-duplicate bodies instead of being explicit parameters.
The pattern already works. The recent DV rungs (ambe.py, dmr.py,
ysf.py, vdl2.py) import Golay/Hamming from p25/fec.py, the RRC from
tetra/dsp.py, and HDLC+FCS+Gaussian-pulse from aisdecode.py rather than
copying them. That is the proof of concept. This plan generalizes that one
shared-library habit into a proper, documented toolkit and migrates the
existing modules onto it.
2. Goals and non-goals¶
Goals
- One correct, unit-tested, documented implementation of each shared primitive.
- Every primitive parameterized by its defining constants (poly, field,
constraint length, roll-off, symbol rate, …), with named presets for the
known protocols so call sites read declaratively.
- A standalone package with no dependency on the app (only NumPy), so the
toolkit can be imported, tested and reasoned about in isolation.
- Backward-compatible migration: every existing test_*.py keeps passing,
byte-for-byte, at every step.
Non-goals
- Not unifying the LDPC codes themselves (DVB-S2 / FT8 / JS8 / Mercury use
different parity-check matrices). A shared min-sum decoder core is worth
extracting; the codes stay separate.
- Not touching mode-specific tables (varicodes, DAB constants, JT/FT parity
tables) — those are data, not duplicated logic.
- Not changing on-air behaviour, module public APIs, or the app's mode
dispatch. This is an internal, mechanical consolidation.
- Not a rewrite of the vocoder stack (mbe/imbe/ambe) — it is already
layered well; it only consumes the new toolkit (e.g. shared bit helpers).
3. Proposed package layout¶
A new leaf package app/radio/dsp/ (pure NumPy, no imports from app.*). The
name dsp reads well at call sites (from ..dsp import crc, conv, rs). Layers
are ordered so each only depends on the ones above it:
app/radio/dsp/
__init__.py # curated re-exports of the stable surface
bits.py # bit<->int<->bytes, MSB/LSB, Gray code, packing
crc.py # parameterized CRC engine + named presets
galois.py # GF(2^m) field object (parameterized by primitive poly)
filters.py # RRC, raised-cosine, Gaussian pulse, matched filter
scramble.py # LFSR/PN generators (Fibonacci & Galois), additive scrambler
interleave.py # block / convolutional / quadratic (de)interleavers
sync.py # Hamming-tolerant sync-word correlator (bit & soft)
blockcodes.py # Golay(23/24,12), generic Hamming(n,k), BCH (seeded from p25/fec.py)
convolution.py # conv encoder + Viterbi (K, polys, puncture, soft/erasure, terminate)
reedsolomon.py # RS(n,k) over a galois.Field (nroots, fcr, prim, optional dual basis)
modems/
__init__.py
fsk.py # M-FSK / C4FM (levels, symrate) on discriminator audio
psk.py # differential M-PSK (BPSK/QPSK/DQPSK/D8PSK), Gray map, Costas/Gardner
gmsk.py # GMSK + plain MSK (IQ and discriminator-audio variants)
afsk.py # AFSK tone-pair modem (DSC, POCSAG-style, Bell-202)
framing/
__init__.py
hdlc.py # bit-stuffing + FCS framing (AX.25 / AVLC)
Optionally, a sibling grouping for the voice codecs (lower priority, purely
organizational): app/radio/vocoder/{mbe,imbe,ambe}.py re-exporting the
current modules. Not required for this refactor; listed for completeness.
4. Module interfaces (sketch)¶
Signatures below are the intended public surface. Conventions (bit order, sample-rate expectation, dtype) are stated in each module docstring and are the single most important thing to get right — most "duplicates" differ only in convention, so the parameters must make convention explicit.
4.1 bits.py (foundational, no deps)¶
to_bits(value: int, n: int, msb_first=True) -> list[int]
from_bits(bits, msb_first=True) -> int
bytes_to_bits(data: bytes, msb_first=True) -> np.ndarray
bits_to_bytes(bits, msb_first=True) -> bytes
gray_encode(x: int) -> int # x ^ (x >> 1)
gray_decode(g: int) -> int
_to_bits/_from_bits pairs and the ad-hoc
np.packbits/unpackbits calls. msb_first captures the one axis on which
the copies actually differ (P25 is MSB-first; AIS/ACARS characters are
LSB-first).
4.2 crc.py¶
class Crc:
def __init__(self, width, poly, init=0, reflect=False, xorout=0): ...
def __call__(self, data: bytes | Iterable[int]) -> int
def of_bits(self, bits) -> int
CCITT_FALSE = Crc(16, 0x1021, 0xFFFF) # p25, ysf, dmr
X25 = Crc(16, 0x1021, 0xFFFF, reflect=True, xorout=0xFFFF) # AIS FCS, ACARS BCS
CRC24_ADSB = Crc(24, 0xFFF409) # (present convention preserved)
CRC6_NXDN = Crc(6, 0x27)
CRC12_NXDN = Crc(12, 0x80F)
# … one preset per protocol, defined once, named.
4.3 galois.py¶
class Field: # GF(2^m)
def __init__(self, m: int, prim_poly: int): ... # builds exp/log tables
def mul(a,b) / inv(a) / pow(a,n) / div(a,b)
def poly_eval(coeffs, x) / poly_mul(p,q) / poly_add(p,q)
GF256_0x11D = Field(8, 0x11D) # DMR RS(12,9), VDL2 RS(255,249), QR/Ethernet
GF256_0x187 = Field(8, 0x187) # CCSDS (sat/rs.py)
GF64_0x43 = Field(6, 0x43) # P25 BCH(63,16)
reedsolomon and blockcodes.bch
consume a Field.
4.4 reedsolomon.py¶
class ReedSolomon:
def __init__(self, field: Field, nroots: int, fcr=0, prim=1, dual_basis=False): ...
def encode(self, data: bytes) -> bytes # systematic, supports shortening
def decode(self, codeword: bytes) -> tuple[bytes, bool] # (data, corrected_ok)
RS255_223_CCSDS = ReedSolomon(GF256_0x187, 32, fcr=112, dual_basis=True) # sat
RS255_249_VDL2 = ReedSolomon(GF256_0x11D, 6, fcr=0) # vdl2
RS12_9_DMR = ReedSolomon(GF256_0x11D, 3, fcr=0) # dmr LC
sat/rs.py structure is the
reference), parameterized. The CCSDS dual-basis transform becomes an optional
wrap. Collapses sat/rs.py, hamdrm/rs.py, vdl2.rs_*, dmr._rs12_9_*.
4.5 convolution.py¶
class ConvCode:
def __init__(self, constraint_len, polys, puncture=None, terminate=True): ...
def encode(self, bits) -> list[int]
def viterbi(self, symbols, soft=False, erasure_value=None) -> list[int]
K7_CCSDS = ConvCode(7, (0o171, 0o133)) # sat, inmarsat, acars AERO
K5_G23G35 = ConvCode(5, (0o23, 0o35)) # ysf, nxdn (with puncture)
K3_G7G5 = ConvCode(3, (0o7, 0o5)) # dstar header
conv_encode copies. This is the single
highest-value extraction: it is the most-copied and the most bug-prone
primitive.
4.6 blockcodes.py¶
Seeded by moving p25/fec.py here essentially verbatim (it is already the
shared, well-tested implementation), then adding the small generalizations the
DV rungs needed:
golay23_encode/decode, golay24_encode/decode
hamming_encode(d, n, k) / hamming_decode(cw, n, k) # generic (already 13,9 / 15,11 / 10,6)
bch(field, n, k) # BCH(63,16) generalized
p25/fec.py becomes a thin shim: from ..dsp.blockcodes import *.
4.7 filters.py¶
rrc(sps, alpha=0.35, span=8) -> np.ndarray # one root-raised-cosine
rcos(sps, alpha, span) -> np.ndarray
gaussian_pulse(sps, bt=0.5, span=4) -> np.ndarray # from aisdecode._gauss
_rrc,
rrc_taps, _gauss become one-line aliases during migration.
4.8 scramble.py, interleave.py, sync.py¶
# scramble.py
class Lfsr: # Fibonacci or Galois LFSR
def __init__(self, poly, seed, taps=None, galois=False): ...
def sequence(self, n) -> list[int]
def additive_scramble(bits, lfsr) -> list[int] # XOR, involutive
ccsds_pn(n) -> np.ndarray
# interleave.py
block_interleave(bits, rows, cols) / block_deinterleave(...)
conv_interleave(...) ; quadratic_interleave(...) # m17-style
# sync.py
def find_sync(bits, pattern_bits, start=0, max_err=4) -> int
def find_sync_all(bits, pattern_bits, max_err) -> list[int]
4.9 modems/¶
Each modem is the corresponding existing implementation, generalized:
- fsk.C4fm(symrate=4800, levels=4) — from p25/demod.py (now already
symrate-parameterized); serves P25, DMR, YSF, NXDN.
- psk.DiffPsk(order=8, alpha=0.35, gray=True) — from tetra/dsp.py widened;
serves TETRA (DQPSK), VDL2 (D8PSK). psk.Coherent (BPSK/QPSK) from
sat/qpsk_rx.py with its Gardner + Costas loops; serves Inmarsat, LRPT.
- gmsk.Gmsk(bt=0.5) and gmsk.Msk() — from aisdecode + acars; IQ and
discriminator-audio front ends; serves AIS, D-STAR, VHF-ACARS.
- afsk.Afsk(mark, space, baud) — serves DSC, NAVTEX, POCSAG, APRS.
4.10 framing/hdlc.py¶
hdlc_frame(payload_bits, fcs=X25) / hdlc_deframe(bits, fcs=X25) lifted from
aisdecode.py, parameterized on the FCS. Serves AIS, APRS/AX.25, VDL2/AVLC.
5. Migration strategy (incremental, test-guarded)¶
The invariant at every step: the full test_*.py suite stays green. The
existing tests are round-trip + noise tests, so any convention drift in an
extracted primitive fails a test immediately. Work proceeds leaf-first so each
new library is fully covered before anything depends on it.
Phase 0 — scaffold + proof. Create dsp/, move p25/fec.py → dsp/blockcodes.py, leave p25/fec.py as a re-export shim. Run the whole suite: it must pass unchanged. This validates the shim approach with zero behavioural risk (the DV modules already import p25/fec, so it exercises the shim immediately).
Phase 1 — foundational leaves (bits, crc, galois, filters, scramble, interleave, sync). Each: write the library + a dedicated test_dsp_<name>.py that (a) tests the primitive directly and (b) asserts the named presets reproduce the current outputs of 2–3 origin modules bit-for-bit. Then migrate those 2–3 consumers to import from dsp/, deleting their local copy, and run their tests.
Phase 2 — composite coders (reedsolomon on galois; convolution). Port the proven sat/rs.py and a reference Viterbi, parameterize, and cover with unit tests that reproduce each origin (CCSDS RS, VDL2 RS, DMR RS; K=7/K=5/K=3 Viterbi). Migrate consumers one module per commit.
Phase 3 — modems + framing. Extract fsk, psk, gmsk/msk, afsk, hdlc. These carry the most convention risk (timing recovery, normalization), so migrate the simplest consumer first (e.g. P25 C4FM, which is already the shared modem) and expand outward.
Phase 4 — sweep. Migrate every remaining protocol module to the toolkit, deleting local duplicates, one protocol per commit with that protocol's test run each time. Modules touched: essentially all of section 1's table.
Phase 5 — cleanup. Remove the compatibility shims (p25/fec.py etc.) once no importer remains; update docs/native-protocols.md and NOTICE.md provenance to point at the toolkit; add a short dsp/README.md documenting the toolkit surface and conventions.
Commit granularity mirrors the existing ladder: one focused commit per library or per migrated module, each with its tests, each independently revertible.
6. Risks and mitigations¶
- Silent convention drift (an extracted primitive is "the same" but for bit
order / poly endianness / normalization). Mitigation: every preset is
asserted bit-exact against the origin module's current output in a unit test
before the call site is switched; the shim layer keeps each migration
independently revertible; the round-trip+noise
test_*.pysuite is the backstop. - Over-generalization (a parameterized API so flexible it's unreadable).
Mitigation: the public surface is the named presets, not the raw
constructors; call sites read
conv.K7_CCSDS.viterbi(...), not a soup of polynomials. Keep constructors minimal; add parameters only when a real second consumer needs them. - Performance regression from indirection. Mitigation: the hot paths
(Viterbi ACS, RS syndrome/Chien, RRC convolve) are unchanged NumPy inner
loops, just relocated; benchmark the two or three heaviest (
bench_cw.pyexists as a model) before/after on a representative signal. - Scope creep into LDPC / vocoder. Mitigation: explicitly out of scope (section 2); revisit as a separate effort.
- Churn across ~40 modules. Mitigation: phased, leaf-first, one module per commit; the toolkit lands and is proven (phases 0–3) before the bulk sweep (phase 4), so the sweep is mechanical.
7. Payoff¶
- ~9 Viterbi + ~11 conv encoders → 1 parameterized coder.
- ~9 GF fields → 1; ~4 Reed–Solomon → 1; ~15 CRC variants → 1 engine with named presets; ~19 scramblers, ~15 interleavers, ~8 sync correlators, ~4 RRC filters each → 1.
- A new protocol becomes framing + presets, not framing + re-deriving FEC — matching how the last DV rungs already worked, but for the whole package.
- One place to fix a bug or optimize a hot loop; conventions become explicit parameters instead of hidden differences between near-duplicates.
8. Suggested first slice (if approved)¶
Phase 0 + dsp/bits.py + dsp/crc.py + dsp/sync.py in one short series:
they are pure, dependency-free, cover the most call sites, and carry the least
risk — a concrete, low-stakes proof of the whole approach before the heavier
convolution/reedsolomon/modems extractions.