S3 sleep, and the three bugs it found

What S3 actually is

S3 is a deal with the firmware. RAM stays powered and refreshed, everything else loses power - the processor forgets every register, devices reset, the interrupt controllers reset. On the way back the firmware starts from reset, re-trains memory, and then reads one 32-bit word out of an ACPI table called the FACS. That word is an address, and it jumps there in 16-bit real mode.

The whole contract is therefore: leave an address in the FACS, and be ready to be re-entered as though the machine had just been switched on, with your RAM intact.

Going down

Inside sleep::enter():

1. Check the description. The value that means "S3" is not a constant - the board's firmware publishes it in AML as \_S3, and the value next to it is S5 (power off, no return). Read at every boot, never guessed.
2. Park every other core. This is the part that interacts hardest with the kernel, and it is covered below.
3. Copy the resume trampoline into the low page the loader reserved (shared with the SMP trampoline, at a different offset), and fill in a small data block: the GDT to climb through, CR3, the entry address, a landing stack.
4. Write the waking vector into the FACS.
5. Save the processor - the callee-saved registers, RSP and the return address (a hand-written setjmp), plus CR0/CR3/CR4, EFER and the FS/GS/KERNEL_GS base MSRs, which no calling convention describes.
6. wbinvd, so all of that is actually in RAM and not in a cache line the resumed processor will never see.
7. Clear the wake status bits, arm the power button, write SLP_TYP | SLP_EN to the PM1a control register. The machine stops mid-instruction-stream.

Coming back

The firmware jumps to our page in real mode. From there:

1. A far jump to itself, pinning CS. The spec does not say how firmware splits the address between CS and IP, and everything below addresses data through a fixed segment base.
2. Protected mode, then PAE, then kernel page tables, then EFER.LME, then paging, then a far jump to 64-bit. This is nearly identical to the SMP trampoline, because it is the same problem.
3. Land on a dedicated stack and call into Rust. MSRs first - GS is how a core finds out which core it is, and anything that takes the kernel lock asks.
4. Reload the descriptor tables. They survived in RAM; what was lost is the registers pointing at them.
5. Relight the 8259, the PIT and the local APIC.
6. Restore the saved registers and stack and ret, so enter() returns a second time, from the other side of the sleep, as if the call had returned normally.

Where it rubs against the kernel

Three places, and all three were the actual bugs.

The kernel lock is a ticket lock. A core that disappears holding a ticket is a number never served, so everyone behind it waits forever. A suspend stops all cores at once, so every started core must first be stopped at the top of its own idle loop, where it holds nothing. proc::choose_next answers "nothing" to a started core while a park is wanted, which is what lets a core running a process come home at all. If they do not all stop, the suspend is refused.

The GDT is a static, so it remembers too much. ltr set the TSS's busy bit at boot; RAM preserved it; ltr on a busy descriptor faults, before there is an IDT to deliver the fault to. Triple fault.

Interrupt handlers take the kernel lock, so anything that waits on the tick while waiting for the lock waits forever. The park watchdog uses the TSC instead.

The deliberate limit

cpu::reload_descriptors is shared by boot and resume, but no driver is re-initialised. The NIC comes back reading link=DOWN. Bringing devices back is walking the collation graph in reverse: the graph is stage 2's, the work is stage 5's.