FlowMachine

FlowMachine (aka Flow-er, hence the logo) - is an OS designed for long-lived workflows with human intervention. Jobs may be started and last years before being completed. Traditional architectures don't handle this well. As a human workflow engine, precidence was given to making it easy to define and deploy workflows - hence FlowBASIC. The execution model of durable tasks pairs well with an interpreted language. There are some subtleties around verisoning - each dehydrated process has all the data it needs to execute on a self-describing basis, so it is agnostic to any increment to application that triggered the workflow - versioning is essentially free with this scheme. Rehydration is triggered by events, which are in themselves delivered via channels. These range from webhooks to mqtt to polled sources. All of this infrastructure is underpinned by role based access control - as coarse as a tenant, as fine as a user and specific action. The bare-metal x86-64 microkernel in Rust that runs FlowBASIC p-code as durable, memory- and fault-isolated Ring-3 jobs. It has its own object store, its own network stack running in user mode, TLS 1.3 in both directions, and no operating system underneath it - it is the operating system. Two nodes are serving on the house LAN as I write this; performance tests show speeds approaching that of apache on the same hardware, which isn't too shabby. Whilst not actually running conventional workflows perse, acting as the backend server (BES) for my DIY Psion project exercises both the basic language interpreter and channels over a long range test. Pretty lights are always a welcome addition, so FlowMachine has a VGA display which summarises activity over 15 second slots, and gives basis status indication. The operator's console can be accessed over serial or ssh. The code had 255 commits between 24 July and 3 August 2026, many focused on performance and multithreading aspects. That is the panel of the node at 192.168.42.135: four cores, no faults, serving, sixteen channels and thirty-three FlowBASIC processes, most of which are sitting dehydrated in the store waiting for something to happen. Fetching from the associated channel resumes one of them.

Why a language would want its own kernel

The p-code VM builds no_std. Once that is true, what sits underneath becomes a choice rather than a given, and the workflow model wants three things that a general-purpose OS makes awkward. A job should be a first-class durable object, not a file that a process happens to write. On FlowMachine a dehydrated job is stored by the kernel in the same store as the p-code it runs and the records it touches, made durable by the same commit. Durability should have one boundary. A store whose commit is the atomic point removes the question of what happens if the process dies between the database write and the file write, because there is one write. I leverage this atomicity in the replication engine - one can run as many parallel instances as one wishes and they all act in synchrony. Isolation should be per job, not per process. Each Ring-3 job gets its own address space behind W^X and NX and its own recovery point; a fault in one long-jumps to that core's handler and the node carries on serving.

What the kernel does

Boots via multiboot2 into long mode and sets up GDT, TSS and IDT. User code runs at Ring 3 behind a 4 KiB W^X remap with NX enforced, entering the kernel through an int 0x80 syscall gate. SMP brings up the application processors with a per-CPU CR3, which is required rather than elegant: the user binary is linked at a fixed virtual address, so two cores that both need 0x4000_0000 need different physical pages behind it. Only the spine that must diverge is cloned; the low kernel mapping stays shared. On top of that: virtio-blk and virtio-net drivers, smoltcp, TLS 1.3 as client and server, HTTP on :80 and :443, SSH on :22, SMTP on :25, MQTT ingest, an outbox for outbound delivery, UDP-multicast replication between nodes, a Telegram driver, an Ollama channel, and hardware crypto through AES-NI, AVX2 and PCLMUL.

The four crates

Crate	What it is	Target
interp/	The language: lexer, parser, tree-walker, p-code compiler and VM, record codec. Builds for the host CLI and for the kernel from the same source.	host + no_std
flowuser/	The Ring-3 user binary. Links the language, runs p-code jobs at CPL 3, talks to the kernel through a shared arena and int 0x80.	x86_64-unknown-none
flowsvc/	The Ring-3 I/O-plane service. Native Rust, no language at all: a virtio-net virtqueue driver and smoltcp running at CPL 3.	x86_64-unknown-none
kernel/	FlowMachine. Embeds both Ring-3 binaries, builds to a staticlib and links to an ISO.	x86_64-unknown-none (staticlib)

Plus crypto_hw/, a small hard-float staticlib built for its own custom target and linked into the soft-float kernel, because AVX2 cannot be compiled into a kernel built with soft float. Only byte slices and fixed arrays cross that boundary, so the ABI mix is safe. The network stack living in flowsvc at Ring 3 is the load-bearing decision of the whole design. A blocking or misbehaving network operation is contained in an isolated user-mode service that the kernel enters once per tick and drains; it cannot take the kernel down and it cannot stall the Flow workers on the other cores. smoltcp compiles for x86_64-unknown-none with no allocator at all, given borrowed static storage for its sockets and buffers. Each Ring-3 binary is a matched pair: the flat .bin and a generated _layout.rs giving the section boundaries the kernel uses to set W^X page permissions. Regenerate the binary without the layout and the kernel maps the grown .text non-executable, so the job page-faults on an instruction fetch the moment it runs past the old boundary. One line to fix, an afternoon to find the first time, which is why the build script now regenerates both or neither.

flowSTORE

The store is checksummed and copy-on-write, written straight onto a raw block device. It is not a filesystem: no inodes, no directories, no POSIX semantics. It holds exactly what the kernel needs - named p-code packages, parked jobs, durable records and their indexes. Every 4 KiB block ends with a CRC32C over its first 4092 bytes, so a torn or bit-rotted block fails its checksum and is treated as absent rather than as data. Nothing is rewritten in place: a change writes fresh blocks, flushes them, then flips a root pointer in the superblock. Atomicity comes from that single checksummed superblock commit, and the superblock is kept in four copies, two at the start of the device and two mirrored at the very end, so wiping either end still leaves a valid root. Copy-on-write turned out to pay for itself twice. Because a block written at a given sector is immutable until the compactor reclaims it, a sector-keyed read cache is coherent by construction - no invalidation protocol, just an invariant that every write populates or invalidates its own entry.

The Psion backend

The largest single body of work is a per-user messaging backend, built against endpoint contracts frozen before any code was written. Eight phases:

• P6.0 - a TLS 1.3 server on :443, the inverse of the existing client. Mandatory: the device verifies certificates and there is no plain-HTTP fallback, so no endpoint could ship before it.
• P6.1 - a per-user mailbox and GET /receive, flattening MIME to plain text on read, with an opaque monotonic cursor that survives a device restart.
• P6.2 - GET /catalogue and one shared sizes contract for every channel: ids, byte counts and kinds, no bodies. Mail thresholds at 2 KB and 8 KB, images at 56 KB and 64 KB, any response at 32 KB and 64 KB. Over the threshold a message is withheld but still returns a stub, because a message must never silently vanish.
• P6.3 - POST /send with device-generated idempotency ids, so a device that loses the reply can retry safely.
• P6.4 - a signed UDP poll on :9001. PSN1 POLL <device-id> <nonce> <channels> <hmac> returns PSN1 STAT <nonce> mail=<count> mail.gen=<n> <hmac>, HMAC-SHA256 truncated to 16 bytes in both directions, reply capped at 512 bytes, never carrying content.
• P6.5 - GET /image: decode, scale to fit 560x800 preserving aspect, dither to the device's fixed 16-colour palette, and emit exactly ((w+1)/2)*h bytes of 4bpp packed pixels with no header. Any other length is a hard reject at the device. PNG first, then baseline JPEG, then progressive JPEG, each new decoder feeding the same scale, dither and pack back-end.
• P6.6 - calendar, /read and /delete.
• P6.7 - fetching a real Fastmail mailbox over JMAP: HTTPS, JSON and a bearer token, no IMAP, with three Fastmail hosts SPKI-pinned. Fetched mail flattens through the identical path as locally-delivered mail.

One deviation from the frozen contract is worth recording. The contract keys the notify HMAC with the user's bearer token, but the server stores an API key as SHA-256 of the secret only and never holds the secret, so it cannot reproduce that HMAC. Storing the key in plaintext would have made the contract satisfiable at the cost of the property that makes the key store worth having. The notify channel therefore uses a dedicated per-user secret, provisioned node-local and looked up by device id. A documented divergence, not a silent one. Real-hardware proof for the heaviest case: twelve consecutive emails of roughly 5 MB each, delivered back to back, all stored, node stayed serving with zero faults, and everything survived a reboot. Getting there needed three separate sizing fixes and a store that assembles a multi-megabyte blob in 256 KiB pieces rather than one contiguous allocation.

How it connects to the other projects

The DIY Psion. FlowMachine is the per-user backend for the ESP32-S3 Psion, explicitly on the BlackBerry BES/NOC model. The server holds the mailbox connection and does all the heavy work - MIME flattening, attachments, image decode and dither - and feeds a thin, authenticated, bandwidth-constrained device over a lightweight "what changed" poll with fetch-on-demand. The device is dumb on purpose. It never sees a MIME tree or a JPEG; it gets plain text and pre-dithered pixels, and the real computer does the work. That framing settled almost every design question in the backend on its own, including which end owns the sizes contract. The endpoints are FlowBASIC, not kernel Rust. A route is a SUBSCRIPTION record dispatched to a FlowBASIC handler, and the whole mailbox - cursors, sizes contract, idempotency bookkeeping, JSON - is 959 lines of apps/psionmail.flow. A handler is a process that waits:

PROCESS psreceive
  WHILE TRUE
    DIM req = WAIT FOR http WHERE "/receive"
    DIM owner = LOWER(req.user)
    ...
  WEND
END PROCESS

The kernel adds native primitives only for what p-code cannot do efficiently. The rule of thumb is flat: records, strings, routing and protocol shape are FlowBASIC; a cipher or an image codec is a native primitive that FlowBASIC calls. In practice that meant base64, quoted-printable, charset and RFC 2047 decoders, the image pipeline, and the TLS and HMAC crypto. Everything above them is the language, which was the point of building the language. Path OS. FlowMachine's kernel is the kernel Path OS is built on. It is cloned into that project's upstream/ directory as a read-only source and never modified in place, contributing the part that already boots: long mode, Ring 3 with W^X and NX, the syscall gate, separate user-mode binaries, SMP, virtio block and net, smoltcp and TLS. Path OS supplies its own window server on top and ports the Psion UI toolkit to it.

Verification, and why QEMU is not enough

The kernel cannot cargo test - x86 assembly, no_std, staticlib. Two things stand in. Host harnesses compile the real kernel source against an in-memory block device or an in-memory socket pair, so flowSTORE, the TLS client-to-server handshake, the notify protocol, the image pipeline and the crypto backends all get a fast gate with no VM involved. The image gate is byte-exact against a Python reference for PNG and a bounded per-pixel tolerance against PIL for JPEG. Then it gets booted, and where it gets booted matters. The dev VM runs QEMU with SLIRP user networking, and SLIRP has twice hidden a real bug: once when a hardcoded IP passed because SLIRP's guest is always 10.0.2.15 regardless of DHCP, and once when a polling loop that starved the inbound path looked fine on SLIRP and was dead on a real bridge. The dev VM is therefore a compile-and-boot smoke test only, and anything touching the network is verified on throwaway Proxmox VMs on a real bridge, each destroyed afterwards.

The EFER.NXE triple fault

Bringing up the application processors, the machine started silently resetting. No panic, no serial output, nothing to read. With EFER.NXE clear, a set NX bit is not ignored - it is a reserved-bit page fault. An application processor that has not yet loaded a TSS cannot deliver that fault, which escalates to a double fault, which it also cannot deliver, which is a triple fault and a silent reset. The kernel's hardening pass marks everything outside .text as NX, so the first AP to walk a hardened page after hardening ran would reset the box. The fix is one line in the right place: every AP enables NXE first in its entry path, before it touches any page. Finding it took considerably longer than writing it.

The 3.6 ms that turned out to be a serial port

A keep-alive HTTP request on a real LAN measured about 3.7 ms, of which the FlowBASIC handler was 117 microseconds - 3%. The obvious reading is that the other 97% is the network. The obvious reading was wrong. Instrumenting a server-internal span inside the Ring-3 service - from the moment a socket first delivers request data to the moment the first response byte is sent, excluding the virtio DMA wait and the wire - gave 9.3 ms per request. That span is the decisive one: milliseconds there means the fault is above the wire and is therefore mine to fix. The culprit was a 75-byte trace line written to COM1 on every request. The UART spin-waits per byte and QEMU models the baud rate, and the boot divisor is 38400 baud, so one trace line stalls the serve loop for milliseconds. The log had 1244 trace lines for 1244 requests, which is as clean a smoking gun as measurement produces. Rate-limiting the trace to roughly one per second took the internal span from 9.3 ms to 1.8 ms. The residual needed one more instrument. On a single-core VM a TSC delta keeps advancing while the host runs the test harness on the same core, so wall-clock time inflates without any work being done. Counting serve passes per request instead - a counter that only advances when the guest actually runs - gave 1.0 passes per request, about 70 to 85 microseconds of real work. The serve loop is already at its floor: request read in one tick, handler runs, response sent in the next. The honest conclusion is that if the 3.6 ms survives on real hardware it lives below smoltcp, in virtio or the bridge, and "here is the one-pass evidence that it is not us" is the correct place to stop.

The store lock that was measured and left alone

The task was to narrow a coarse store lock so that concurrent HTTP requests would stop serialising, with the gate set in advance: if a provably safe narrowing is not achievable in one clean pass, stop and report. Measurement killed it twice over. The copy-on-write B-tree had no resident node cache, so every read went to the single non-reentrant virtio-blk queue - and that queue takes the same lock at the device. Concurrent readers were physically impossible regardless of what the outer lock did. And the only candidate narrowing, releasing the lock across the Ring-3 handler excursion, was worth nothing: the excursion averaged 183.6 microseconds against a hold that was essentially all device round-trips, and lock contention under sequential HTTP measured 0.0 microseconds. It was also unsafe, because releasing mid-delivery breaks the load-run-store atomicity around a job's snapshot. Nothing was committed. The instrumentation was reverted, the worktree left pristine, the version not bumped. What went in was the finding: the prerequisite is a resident node cache, not a smaller lock. That cache landed next. Two versions later the remaining per-request cost was traced to a snapshot write that a stateless handler did not need - and the reason a naive "skip if unchanged" would not have fired is that the delivered request stayed bound to a dead variable across the next WAIT, so the snapshot differed every request by exactly the last request. Clearing that provably-dead slot at the dehydration boundary took the write from 5486 to 112.7 microseconds per request, and store commits over 400 requests from about 401 to about 1. Neither fix was in the original plan.

Where the code lives

The repository is basic_alike, private and unpublished. The kernel part of it:

• kernel/ - the microkernel. lib.rs is the bulk, with flowstore.rs, recstore.rs, net.rs, tls.rs, ssh.rs, image.rs, notify.rs, cpu.rs and chw.rs around it.
• flowuser/, flowsvc/, crypto_hw/ - the Ring-3 binaries and the hard-float crypto staticlib.
• kernel/hosttest/ - the host gates that compile the real kernel source.
• kernel/docs/ - as-built architecture references: flowSTORE, the Ring-3 architecture, the I/O plane, replication, TLS, hardware crypto, the Psion backend and its frozen contracts.
• apps/ - the FlowBASIC applications the node serves, including psionmail.flow.
• DEPLOYMENT.md - the build, deploy and verification process. The file to read first if you ever have to touch any of it.

The language is described separately, in FlowBASIC.