Run 10403: under a CI job container's non-reaping PID 1 the script child the deadline killed stays a zombie, and kill -0 still found it. The check now reads the process state from /proc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BdKQBQkJkq4zE7iPFMkAhS
The Device Simulator
A QEMU virtual machine that boots the real forward-ported kernel and real
userspace: the 86 Panel (init, daemons, networking, OTA, watchdog, display)
developed and tested with no board attached. The third simulator in the stack
(three-way split: root README), deliberately not named "sim": it runs the
whole machine above the kernel entry point on real binaries: bring your own,
or drop prebuilt payloads in payload/. Decision record: ADR-0006.
The Boundary
There is no RV1106 machine model in QEMU and everything below the kernel is
closed rkbin blobs plus mask ROM, so the VM enters at -kernel zImage on
-M virt,highmem=off (single Cortex-A7, 256M: the RV1106G3's shape).
| Emulated / substituted | Not emulated (stays bench / sim/ territory) |
|---|---|
| Kernel boot, init ordering, switch_root | BootROM, idblock/DDR-init, SPL, U-Boot |
A/B outcome (warden.slot= cmdline) |
Real BCB A/B selection, bootcount auto-revert |
Storage: virtio-blk with the device's exact blkdevparts= layout + /dev/block/by-name/ contract |
eMMC controller itself |
| Network: virtio-net (slirp, hostfwd 22/80/28443) | GMAC, AIC8800 wifi, usb0 gadget |
| Display: virtio-gpu 720x720 via fbdev emulation | VOP/RGB666 pipeline, CH32V003 panel init, RGA blits |
Touch: virtio-tablet (QMP input-send-event) |
GT911 on I2C3 |
Watchdog: i6300esb (PCI), -action watchdog=reset |
DW watchdog @0xff5a0000, HPMCU supervisor |
RS485: pci-serial chardev bridged to sim/'s ModbusSlave |
Real UART4 timing/electrical behavior |
RTC: PL031 (--rtc reproduces the no-RTC 2021-clock incident class) |
The unpopulated backup-cell reality |
"Boots/works under emulation" is never evidence of "works on silicon." The VM narrows which claims need a panel; on-device claims still need on-device evidence. Conversely, the VM is the first environment that runs production binaries on a non-RV1106 memory map: it found flare-edge #106 (fatal SIGBUS in flared's HPMCU probe) and #107 (Y2038 time_t truncation) on its first two boots of real userspace.
Documented guest deviations from production, set by stage-2 init:
WARDEN_FLARE_INSECURE=1 (the desk mock portal is plain HTTP) and
WARDEN_HPMCU=0 (no mailbox SRAM on virt; flared >= flare-edge#106 fix
required, or the daemon dies of SIGBUS).
The VM's kernel is ahead of the panels' in one way that shows in the UI.
It is built from build/warden_defconfig plus configs/virt.fragment, so it
has CONFIG_BRIDGE; the vendor 5.10 kernel most panels still run does not.
A Client-mode access point is bridged to the wired uplink and needs that, so
the Wi-Fi page offers it in the VM and says "Needs Router mode: this kernel
cannot bridge" on a 5.10 panel. Both are correct for the kernel underneath.
The difference disappears as panels move to the 6.18 build, but until then a
screenshot from here is not evidence about a 5.10 panel's Wi-Fi page.
Quick Start
# 1. kernel: canonical build boots the VM as-is; the fragment variant adds
# the scenario devices (PCI serial, watchdog, WireGuard, virtio-gpu/input)
WORK=$HOME/kbuild-out CROSS_COMPILE=arm-linux-gnueabihf- \
WARDEN_KCONFIG_FRAGMENT=qemu/configs/virt.fragment bash build/build-kernel.sh
# 2. initramfs (sha256-pinned static busybox + qemu/rootfs/) and A/B disk;
# RIG_BUSYBOX=panel builds the image's own BusyBox instead (qemu/busybox/)
bash qemu/mkinitramfs.sh
bash qemu/mkimage.sh # options: --portal-url --state K=V --fw-version
# 3. run (see run.sh header for all flags)
bash qemu/run.sh --kernel $HOME/kbuild-out/linux-6.18.46/arch/arm/boot/zImage --shell
Payload: drop static musl armv7 binaries into qemu/payload/ (see its
README). Stage-2 init starts every panel daemon it finds, in the panel's own
S-number order (warden-ai, warden-automation, warden-modbus,
warden-mikrotik, warden-asic, warden-starlink, warden-stratum,
warden-flared), then warden-ui (the LVGL fbdev+evdev build from
flare-edge tools/build-ui-vm.sh). The quickest faithful payload is the
daemons a bench panel already runs: flare-edge tools/rig-pull-daemons.sh --host <panel> copies them in. The RS485 UART is always present (a null
chardev without --rs485), so warden-modbus polls a port with nothing on
it, as on a panel with an empty header, and never the control channel.
A mosquitto in the payload starts the panel's device bus first (socket
/run/warden/bus/bus.sock, one credential file per bus user under
/run/warden/bus/cred/), as S10bus does before every bus client on a panel;
see payload/README.md. The rig broker takes any user and loads no plugin.
With a test-flavor warden-ui and wthctl (flare-edge tools/wthctl.c, the
test hook client) in the payload as well, a flow's override step pins a
state document inside the UI: qmp.py hands the request to the control bridge
(@hookdoc pieces, then @hook override), which sends it to test/req/ui
as bus user testhook and answers the hook's reply. The run's first hook
request waits until the hook answers a ping (retried while the bridge
reports no answer, up to HOOK_READY_S): right after boot the UI is still
taking the broker's retained replay, and an override sent then was lost
(flare-edge #2602).
The VT cursor is kept off (vt.global_cursor_default=0): fbcon shares the
virtio-gpu framebuffer with warden-ui and its blinking cursor would otherwise
show up in screendumps at random (issue #18).
The guest carries the panel's own /etc/passwd, /etc/shadow and
/etc/group (root's md5-crypt of the documented default password), so a
screen that verifies the root password against /etc/shadow behaves as it
does on a panel instead of rejecting every attempt. qemu_stage_rootfs()
(lib.sh) forces etc/shadow to mode 0600 on every stage: git tracks only
the executable bit, so the checked-out source file's own mode depends on the
checking-out umask and cannot be trusted to arrive non-world-readable.
--append warden.initchain boots the UI the way a panel does (flare-edge
#2210). The caller lays the panel's own scripts over the rootfs with
ROOTFS_OVERLAY=<dir> mkimage.sh (only etc/ and usr/ at its top, no
symlinks). Stage-2 init then starts /etc/init.d/S00-S09 with start
before the bus (S03warden-accounts creates the platform accounts, and the
bus hands cred/ui to warden-ui), and S10-S99 after the payload
daemons in place of its direct root start of warden-ui. S99lvgl then runs
warden-ui-run, which drops the UI to its account and respawns it. Rig
differences: the splash handover is marked done (no S09splash) and
warden-ui-run passes -b fbdev, not the panel's -b drm. A chain boot
with no script prints WARDEN-QEMU-INITCHAIN-EMPTY and starts no UI.
tests/ui-drive.sh --init-chain <dir> does both steps, and the drive verb
assert_ui_chain judges the result: one warden-ui and one warden-ui-run,
the UI's four uids all warden-ui's, no safe-mode flag, and no exit or
refusal in /tmp/warden-ui.log. tests/scripts/init-chain-ui.txt runs the
check at about T+120 s.
A flow seed may contain qemu-flow-setup.sh. Stage-2 init runs it inside the
disposable guest after mounting userdata and before starting daemons or the
UI. A nonzero exit stops the boot. Use it only for test devices and commands
that the virt machine does not provide.
Scenarios
All take the virt-fragment <zImage>; FLARE_EDGE=<checkout> where noted.
| Scenario | Needs | Proves |
|---|---|---|
stage-rootfs-perms.sh |
- | offline, no VM: qemu_stage_rootfs() always lands etc/shadow at 0600, even staged from a source copy deliberately left 0644 |
seed-dir.sh |
- | offline, no VM: mkimage.sh's SEED_DIR hook lands every seeded file under userdata/warden with its original mode (a 0600 secret included), a seeded file beats a same-named --state value, a SEED_DIR that isn't a directory fails closed, and so does an individual entry that is a symlink, a subdirectory, or named outside [A-Za-z0-9_.-]+ -- a hyphenated key such as gas-plant.devices still seeds cleanly |
payload-dir.sh |
- | offline, no VM: mkimage.sh's PAYLOAD_DIR hook puts the run's own payload set (symlinked entries included, as regular 0755 files; its README.md skipped) into /usr/bin of both slots and nothing from qemu/payload, the default still reads qemu/payload, a PAYLOAD_DIR that is not a directory fails closed, and ui-drive.sh looks for warden-ui in the same directory |
boot-smoke.sh |
- | sentinel-asserting boot; runs in CI inside kernel-build |
portal-scenario.sh |
FLARE_EDGE |
real flared against the desk mock portal: authenticated check-in, desired-state pull, signed tier-1 .wfw download; verify/stage/APPLYING as a dry run (no WARDEN_FW_ALLOW_APPLY) |
ota-apply.sh |
FLARE_EDGE |
the FULL apply: the .wfw's bootable rootfs payload is written to rootfs_b (run.sh --allow-apply gates it per boot), the AvbABData in misc flips, and slot _b boots the applied version |
ui-shot.sh |
- | display+touch, headless: QMP-screendumps the 720x720 UI, taps the Metrics tab via input-send-event, asserts the frame changed (qmp.py is the QMP client) |
ui-drive.sh <script> |
- | the same rig for a SEQUENCE: boots once, runs a qmp.py drive script of taps/swipes/screenshots in panel pixels, and FAILS if warden-ui died on the way (stage-2 init announces the exit on the console). tests/scripts/nav-stress.txt is the navigation regression: it reproduces the s_row_left overflow that segfaulted the UI on returning to Settings > Apps; tests/scripts/home-leaves-fullscreen.txt proves the home verb clears dashboard fullscreen, the state a hardware wake tap leaves behind (flare-edge #176); tests/scripts/fullscreen-toggle-tracks-real-state.txt proves a fullscreen toggle right after home (or a real tap) reads the dashboard's actual state rather than a belief home bypassed -- test_qmp_drive.py's FullscreenToggleTracksRealState pins the same contract offline |
real-image-boot.sh |
matched rootfs.img + oem.img |
an ACTUAL flare-edge build (placed by mkimage.sh --rootfs-image/--oem-image) boots its own init chain to getty; binaries predating known fixes reproduce their bugs faithfully, a time machine for field issues |
test-ui-drive-rs485.sh |
- | offline, no VM: a fake run.sh and (for one case) a fake socat stand in so ui-drive.sh --rs485-devices's startup fails closed instead of printing a false "== rs485 simulator: ..." over a bus nothing is serving, for each of a qemu rs.sock that never appears, a socat that never links rs.pty, and an mbsim.py that dies before opening rs.ctl; also proves cleanup() escalates to SIGKILL for a sim process that ignores SIGTERM |
rootfs-overlay.sh |
- | offline, no VM: mkimage.sh's ROOTFS_OVERLAY hook lays a directory of panel files (flare-edge's S03warden-accounts, S99lvgl, warden-ui-run) over the rootfs of both slots with their modes kept, beside the skeleton's own files; without it there is no /etc/init.d; an overlay that is not a directory, names anything but etc/ and usr/ at its top, or holds a symlink fails closed and writes no image; ui-drive.sh --init-chain refuses a missing directory or one without etc/init.d/S99lvgl before boot |
init-chain-test.sh |
- | offline, no VM: warden_init_chain (rootfs/etc/warden-lib.sh) starts the executable S<NN>* scripts of a number range in name order, names a failing one and goes on, and counts them; warden_rig_bus_owners hands a bus credential to its account only when the rootfs has it; sbin/init runs S00-S09 before the bus and S10-S99 instead of the direct UI start on a warden.initchain boot (flare-edge #2210) |
run-sh-args-test.sh |
- | offline, no VM: a fake qemu-system-arm captures run.sh's own argv and pins the ctl/rs485 pci-serial ordering contract rootfs/sbin/init's ttyS0-vs-ttyS1 alias depends on -- the ctl device always enumerates before rs485, warden.ctl only lands on the cmdline when --ctl is given, and an rs485 pci-serial device (real or null-backed) is always present so the port count never shifts |
busybox-panel-test.sh |
- | offline, no VM: the panel BusyBox inputs (qemu/busybox/). panel.config carries every setting of the image's applet patch, rig.fragment changes only STATIC, USE_BB_PWD_GRP and USE_BB_SHADOW, a Kconfig rewrite of any setting and an applet the panel lacks fail the build, applets.txt and help.txt agree, and RIG_BUSYBOX refuses an unknown value, and accepts a prebuilt panel BUSYBOX only when its .inputs match this tree (flare-edge#2210) |
busybox-panel-help.sh [BIN] |
an ARM runner (binfmt or QEMU_USER=qemu-arm) |
the built panel BusyBox against the bench capture: its applet list and every applet's --help equal the panel's own binary's. CI runs it in qemu-tools after build-busybox.sh |
console-shell-test.sh |
- | offline, no VM: warden_console_shell (warden.shell) runs setsid cttyhack sh where cttyhack exists, else setsid -c sh on the console tty read from sysfs (the panel BusyBox has no cttyhack), else a plain sh. CI runs it under bash's sh and busybox sh |
ctl-bridge-test.sh |
- | offline, no VM: runs the control bridge (warden_ctl_bridge in rootfs/etc/warden-lib.sh) against a fake warden-ui that answers some commands late. A reply slower than the old 5 s bound still arrives (the bound is now 30 s, WARDEN_CTL_WAIT_S), a timeout answers bridge: no reply from warden-ui ... (qmp.py raises it as a rig fault), and a timed-out command's late reply is drained instead of being read as the next command's answer (flare-edge #1448). A fake wthctl stands in for the hook client behind @hookdoc/@hook (flare-edge #2054): the pieces join into one request whose args are used once, only plain words reach the client, and a missing client or a request with no reply answers a bridge: line. CI runs it under bash's sh and busybox sh |
init-rig-bus-test.sh |
- | offline, no VM: warden_rig_bus in rootfs/etc/warden-lib.sh against stub brokers. No broker in the payload creates nothing; a broker that listens gets the panel's layout (the bus.sock link, a 0400 64-hex credential per bus user, a config without password file, ACL or plugin); a broker that exits or never listens is reported within the bound; init starts the bus before the payload daemons. WARDEN_TEST_MOSQUITTO_DIR adds a real mosquitto carrying one message between two users logged in by credential file (flare-edge #1754) |
agent-test.sh |
- | offline, no VM: the rig agent (warden_agent in rootfs/etc/warden-lib.sh, armed by run.sh --agent and warden.agent) answers @run ID SECS B64 with <<BEGIN ID>>, the base64 of the script's output and <<END ID RC>>; @chunk pieces join into one script used once; a script past its deadline is killed with its process group and answers 124; a malformed request answers one <<ERR ...>> line and runs nothing; init puts the agent's port after ctl and RS485. It is the guest shell of flare-edge tools/soak/rig-qemu.py (M9 soak-lite). CI runs it under bash's sh and busybox sh |
watchdog (run.sh --watchdog) |
- | arm /dev/watchdog, don't pet: the VM resets ~30 s later (verified) |
Scenario fine print:
- OTA: the BCB slot CHOICE and the physical reset stay emulated by the
harness (ADR-0006 boundary); the VM exports
WARDEN_HARD_RESET=0so flared's post-apply reset surfaces as a reported error, not a /dev/mem fault. - Touch injection holds 200 ms: an instantaneous press+release lands inside one LVGL poll and never clicks.
- Watchdog + a flared payload don't mix: flared pets only while the UI heartbeat is fresh.
Gotchas
- AF_UNIX socket paths cap at ~108 chars; keep
--rs485/--qmppaths short. - A serial port that is closed discards incoming bytes: hold ONE fd open across write and read when scripting the guest side of the RS485 bridge.
highmem=offand-global virtio-mmio.force-legacy=falseare required (32-bit ECAM reach; virtio-1-only gpu/input); both live ONLY in run.sh, which every script (boot smoke included) delegates to.- Never pass
earlyprintk: DEBUG_UART_PHYS is the RV1106's 0xff4c0000.
Requirements
qemu-system-arm (Debian 13 ships QEMU 10), curl, cpio, mkfs.ext4,
debugfs (both ship in e2fsprogs; seed-dir.sh reads a built partition
back with it), gcc-arm-linux-gnueabihf (kernel build), python3
(+cryptography for the portal scenario's .wfw signing). CI: the hosted
qemu-tools job builds the tooling; the boot smoke runs inside the (also
hosted, dispatch-only) kernel-build job, which apt-installs its own
toolchain and qemu (ADR-0007).