The build that hung every core, and the gate that does not catch it

The scheduler work of the last few days ran into something more instructive than a scheduling result.

The morning it hung

On 5 October a kernel built from the scheduler branch went onto the X1's internal disk, hung every core before the network came up, and left nothing at all to read. Getting the machine back took a USB stick and somebody standing in front of it.

The same build boots happily under full-emulation QEMU on the Mac, which is the whole problem. That is not the machine, and a test that passes a build the laptop cannot survive is worse than no test, because it is believed.

Eighty seconds in front of the laptop

There is now a gate between building and flashing. It takes the same EFI tree that gets deployed, writes it onto a Proxmox guest's disk as a real EFI system partition, boots it, reads the serial console, and exits zero only if the machine reached a desktop and stayed there. One run is about eighty seconds, twenty of them sending the tree to the node.

The guest is shaped like the X1 on purpose: the node's own processor so that CPUID, AVX2 and the timer calibration answer the way they do on metal; eight logical processors arranged as four cores of two threads, which is the laptop's shape; a codec attached to the HD Audio controller that is already there, so the dedicated audio core is real on this machine; no VMware backdoor, so the pointer comes up on PS/2 as it does on a laptop; UEFI with Secure Boot off, booting an EFI partition off a disk through the firmware's own GPT and FAT rather than PXE or an ISO; the same driver family as the X1's wired NIC. And a serial port, which is the instrument - the X1 has no serial port, which is exactly why its hang left nothing behind.

The verdict needs positive evidence only. A boot that produces no serial output at all is reported as a failure in those words, because silence is what the X1 did and it is the one thing that must never pass. It wants a build stamp matching the ELF that was just built, so yesterday's disk cannot pass as today's; a line saying how many cores there are to account for, rather than assuming eight; at least two loop-rate reports with the last one non-zero and every core named alive; and the scheduler saying which process is on which core. And none of the wedge or panic markers anywhere in the log.

One detail of that is not fussiness. Every pattern is searched for rather than anchored at the start of a line, because the serial lock is taken per print and a few places build a line out of several prints without holding it across them. One perfectly healthy eight-core boot came out as:

smp   cpu   6 carries speech recognition
…
8 of 8 cores online

An anchored pattern failed that boot. A gate that calls a good build bad is worse than no gate, because the next person stops believing it.

It does not catch the build it was written for

This is the first thing to say about it, because the honest answer is that the VM boots the bad build.

Twelve boots, six of each, same guest and same configuration. Main's tip - what the X1 is running - booted healthily five times out of six. Main plus the three scheduler commits, the ones that went on the X1 and hung it, also booted healthily five times out of six.

The one failure in each six has the same shape, and main has it too: a page fault on a core whose instruction pointer is above the user base with the user bit set, so it is one of the freshly spawned processes faulting moments after it reaches ring 3. The core that took it was parked while holding the kernel lock, and the other seven then wedged behind it. It appears on both builds at the same rate, so it is a pre-existing defect in main rather than anything the scheduler commits did - and it is a real one, now reproducible in about a minute without a laptop being involved.

Two rules follow. Run the gate more than once, because one trial is not evidence. And a build that fails the same way as main is a pre-existing fault rather than a regression, which is the comparison the kept logs exist for.

What is not built

The gate proves nothing about the X1's own hardware. The guest has no wireless, no Bluetooth, no NVMe, no Synaptics touchpad, a different audio codec, 27 KB of firmware tables against the laptop's 238 KB with no S3 in them, no thermal sensor, 4 GiB of memory against 16, and an EFI partition and nothing else - so no store partition and no hibernation area. A build that wedges in the wireless driver, or that cannot reset the audio controller after a warm reboot, will pass this gate and still cost an afternoon.

It is also a gate on one boot rather than on every boot. It catches a kernel that cannot start, cannot find its memory, cannot start its cores or wedges its lock, none of which anything else here catches without a laptop in the room. It is not a licence to stop reading the X1's log after a flash, and the build that prompted it is the proof of that.

As for the scheduler itself: the measurements are built, the hold-off has been made, and none of the policy has changed. The acceptance criteria for that phase are unmet - the changes are host-verified and no hold has been measured since. Blocking idle applications is the strongest next move and it has not been made.