Stage-2 init started only warden-flared and warden-modbus from the
payload, so every daemon-backed screen (Automation, MikroTik, Mining,
RS485, Starlink, Services) sat in its "not running" state on the rig
while the bench panel showed the daemons up with nothing attached. It now
starts every warden-* daemon it finds in the panel's own S-number order
(ai, automation, modbus, mikrotik, asic, starlink, stratum, flared), each
with no arguments as start-stop-daemon runs them on the panel. flare-edge
tools/rig-pull-daemons.sh copies a panel's static binaries in.
Doing that exposed a latent alias: init pointed warden-modbus's /dev/ttyS4
at /dev/ttyS0 whenever ttyS0 existed, and with --ctl that port IS the
control channel, so Modbus polls arrived inside the wake reply and the
bridge hung. run.sh now always attaches an RS485 pci-serial (a null
chardev without --rs485), and init aliases ttyS4 to ttyS1 when warden.ctl
is on the cmdline, ttyS0 otherwise.
Verified on a boot with all eight daemons: status json reports
automation.engine, mikrotik.backend, mining.available and rs485.available
true with the Modbus scan sweeping the empty UART; six daemon-backed pages
matched screenshots of the bench panel structurally (0/256 cells differ).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013aHKWzT5EF86RFKRMtAv9n
- rootfs/sbin/init: the control bridge answers `@cat PATH` locally so the
driver can read /tmp/warden-web-status.json out of the guest. That file
has no trailing newline; the bridge adds one so the sentinel stays on
its own line and the line-based reader never blocks.
- tests/qmp.py: wait_json/assert_json (dotted paths, eq/ne/contains/
len_eq/len_ge/gt/lt), assert_stat off the FIFO's stats reply,
capture_region, assert_region NAME [TOLERANCE] and assert_ocr. A
tolerance other than the captured one, a reference box that does not
fit the screendump, a missing reference or a missing tesseract is FATAL
for that step and the run continues (flare-edge #147).
- tests/imgtools.py: P6 reader, crop, perceptual and structural hashes,
compare, with a self-test.
- tests/test_qmp_drive.py: drive() with QMP and the control channel
faked, pinning the per-step ok/fail/fatal contract.
- mkimage.sh SEED_DIR and ui-drive.sh --seed/--refs: settings fixtures
staged into userdata before warden-ui starts, and a reference store
handed to the driver.
- ci: the driver tests and the imgtools self-test run in qemu-tools.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013aHKWzT5EF86RFKRMtAv9n
The rig could drive the UI and detect one outcome: the process died. Nothing
could ask the UI what page it was on or what a tap would land on, because
that channel is a FIFO inside the guest and the initramfs is busybox-only
with no sshd. Scenarios therefore asserted nothing and screenshots went
unread.
run.sh --ctl exposes a second pci-serial port as a unix socket, the same
device the RS485 bridge already rides, listed first so it is always ttyS0.
It also puts warden.ctl on the kernel command line, and init bridges only
when that marker is present: a VM launched with --rs485 alone has a ttyS0
too, and that one is the Modbus wire. The bridge relays one command line in
and the FIFO's reply out, then a sentinel so the reader needs no timeout.
qmp.py gains the channel verbs (nav, page, stats, hit, assert_page,
assert_hit), records every step to results.jsonl as ok/fail/fatal, continues
past an assertion mismatch so one run reports every broken expectation, and
checks the console after EVERY step for the stage-2 init's EXITED line so a
crash is pinned to the step that caused it. assert_hit matches the widget's
bounding box: an icon has no usable caption and two list rows share a class,
but the geometry the UI itself resolved is exact.
The vocabulary is what tools/warden-ctl already speaks over SSH to a real
panel, so a script that runs here runs there. Verified end to end on the rig
(11/11 verbs round-tripped) and against the bench panel, where the same
commands returned byte-identical results.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013aHKWzT5EF86RFKRMtAv9n
A scenario driving the VM from outside can only see the console, and the UI's
own log was going to a file inside the guest -- reachable only after the
process exited. That made a whole class of question unanswerable while the UI
was still running: what the UI thought it was doing at the moment a screenshot
looked wrong.
It paid for itself immediately: a dashboard fullscreen that kept reverting was
a full UI rebuild firing 553 ms after the tap, which the console showed
outright (flare-edge: the applier's first-tick rebuild).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T2D2KtdgwbhbF6Mo64eUrn
ui-shot.sh proves touch reaches the UI in one tap. Verifying a UI change needs
a SEQUENCE -- swipe through the app rows, open a submenu, tap a tab, bring up
the keyboard -- and booting per step costs about a minute under TCG, so:
- qmp.py gains a `drive` mode: one connection, one boot, a script of
tap/swipe/fling/shot/sleep steps in PANEL PIXELS rather than the raw
0..32767 tablet axis. Swipes interpolate their motion, because LVGL decides
a gesture from the movement between indev polls and a press-then-release
with nothing in between is a click, not a scroll.
- ui-drive.sh runs such a script against a booted VM and collects the
screenshots.
It also FAILS on a UI that died mid-script. warden-ui crashing leaves its last
frame in the framebuffer, so screendumps keep returning a plausible picture of
a program that no longer exists; stage-2 init now announces the exit and its
status on the console, and ui-drive.sh greps for that after the run. This is
what caught the SIGSEGV behind flare-edge#125.
Stage-2 init also mounts devpts. The UI's Terminal page opens a PTY, so
without it that page could only ever report "no PTY available" -- it rendered,
which made a screenshot scenario look fine while the one thing the page does
was untestable.
tests/scripts/nav-stress.txt is the first committed drive script: the
navigation sequence that reproduces flare-edge#125.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T2D2KtdgwbhbF6Mo64eUrn
Replace every em dash with real punctuation (rewrites, not hyphen swaps)
in docs, code comments, scripts, configs, and the port records; convert
en dashes, curly quotes, ellipsis glyphs, arrows, and section signs to
ASCII; drop machine-writing tell phrases from living docs. ADR titles
now use a colon. The M2 bring-up DTS model string carried an em dash
into the patch series and its record echoes; fixed at both, and the full
series re-verified to apply cleanly onto pristine 6.18.46. One comment
in freshness.h deliberately names the em dash glyph the UI renders as
the unknown mark; that is data, kept as prose naming it.
Verified: cargo tests (sim, config-lint, rs485-bridge), shellcheck,
both driver MC/DC harnesses, patches-apply.
mkimage gains --rootfs-image/--oem-image (raw ext4 matched pair placed
into slot A, size-checked fail-closed; slot B keeps the skeleton as a
known-good fallback), and stage-1 rc now hands the live devtmpfs to the
new root across switch_root — busybox switch_root moves nothing, our
skeleton init remounted defensively, but a REAL image's getty opens
/dev/console immediately and looped on ENOENT without it.
Verified with the 2026-08-24 flare-edge build (rootfs 192M + oem 26M):
the image's own rcS chain runs on the 6.18 virt kernel, real daemons
start (warden-modbus confirmed running via root shell; the shipped
0.2.4 flared reproduces flare-edge#106 exactly, and S99hciinit reports
the issue-#4 btlpm symptom — the VM doubles as a faithful reproducer of
known field bugs), and a getty answers on the console.
qemu/tests/real-image-boot.sh asserts switch_root + daemon starts +
login prompt, credential-free. REAL-IMAGE-BOOT-PASS.
Regression sweep after the shared-path changes: OTA-APPLY-PASS,
PORTAL-SCENARIO-PASS, UI-SHOT-PASS, boot smoke OK.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018HUayid7W5w7jBdb9Rrj1K
Closes the loop every prior harness stopped short of, entirely
off-hardware: the real flared (WARDEN_HARD_RESET-gated build) pulls a
real signed tier-1 .wfw whose payload is a BOOTABLE rootfs stamped
0.0.2, verifies it, writes /dev/block/by-name/rootfs_b inside disk.img,
and flips the AvbABData (slot B: priority 15, 3 tries, unsuccessful —
the exact pre-first-boot arming state, round-tripped through a portal
check-in). The harness then boots slot _b and asserts the applied
version + marker are what runs. OTA-APPLY-PASS verified end to end.
- mkimage: the misc partition now carries REAL provisioned AvbABData
(bytes mirror flare-edge's provisioning defaults) — slotctl fail-closes
on bad AB magic before writing, which a zeroed misc tripped.
- run.sh --allow-apply / cmdline warden.fwapply: per-boot opt-in that
makes stage-2 init export WARDEN_FW_ALLOW_APPLY=1; never the default.
- stage-2 init also exports WARDEN_HARD_RESET=0 (the CRU poke is fatal on
virt, same class as the HPMCU probe); the harness performs the reboot.
- ADR-0006 boundary documented in the scenario and README: BCB slot
CHOICE and the physical reset remain emulated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018HUayid7W5w7jBdb9Rrj1K
ubuntu-latest ships shellcheck 0.9.0, which warns that ash scripts are
checked as dash; 0.10 (local) does not. shell=dash is the accurate dialect
for busybox ash. Verified clean under BOTH versions; boot smoke re-run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018HUayid7W5w7jBdb9Rrj1K
- build-kernel.sh fragment assertion: survives a missing trailing newline
(the read footgun, reproduced) and treats an absent symbol on a disable
line as FATAL, symmetric with the enable arm.
- fetch-kernel-tarball.sh checks the pin before downloading; both fetchers
add --retry-connrefused.
- mkimage rejects '.'/'..' state keys.
- ui-shot: VM liveness checked before every QMP call, console.log preserved
as evidence on every failure path, repaint deadline widened to 90s with
the contended-runner rationale documented.
- rs485-bridge: overflow discards back off one gap and rate-limit their log
line, mirroring the accept-loop fix; clippy nit fixed.
- .gitignore ignores *.elf/*.map so the untracked artifacts cannot silently
return; CI shellcheck glob now covers build/ and the rootfs boot scripts
(directives added for the deliberate in-guest source paths).
- Docs: NPU parity row matches its sibling verification docs; line-pinned
audit cross-references unpinned; CROSS_COMPILE documented in the build
header; payload README lists warden-ui; ci-cd tense settled.
Verified: guards negative-tested (bad state keys, no-newline fragment);
boot smoke, portal scenario, ui-shot all PASS; 53 tests green; shellcheck
clean across the widened glob; clippy zero.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018HUayid7W5w7jBdb9Rrj1K
CI/pipeline:
- KERNEL_TARBALL passed as a YAML env literal '~' was never tilde-expanded
and would have failed every hosted kernel-build dispatch; the path is now
exported from the shell. Verified reproducible before the fix.
- Every job gets timeout-minutes; boot smoke uses timeout -k so a wedged
qemu is SIGKILLed instead of holding the job.
- Tarball fetch + fail-closed sha256 verification deduplicated into
build/fetch-kernel-tarball.sh (with curl retries), used by build-kernel.sh
and both CI jobs. busybox fetch gains retries too.
- ccache layer for kernel-build (cache keyed on defconfig+patches) recovers
the incremental-compile speed the ephemeral-runner move cost.
- build-kernel.sh now asserts every fragment option survived olddefconfig —
merge_config -m pastes text and Kconfig silently drops unmet symbols.
rs485-bridge:
- pending-buffer cap (2x max RTU ADU) instead of unbounded growth;
explicit accept-loop error handling with backoff instead of .flatten();
per-arm inline bounds instead of the string-keyed lookup whose default
would have mis-bounded a future get-input; control-socket cleanup errors
surfaced; flag-shaped values rejected in arg parsing; doc example uses a
private mktemp dir. Test timing margins widened for contended runners
(gap 25->120ms, 60x margin on the split-frame test).
VM harness:
- stage-1/stage-2 boot scripts share one validated slot parser and one
by-name populator (qemu/rootfs/etc/warden-lib.sh) — the duplicated
parser had already diverged on validation; userdata/oem mount failures
now fail fast with a greppable sentinel; udhcpc fallback keys off the
interface actually having an address; switch_root applet guarded.
- boot-smoke delegates the qemu invocation to run.sh (machine shape lives
in ONE place); run.sh port 0 disables a hostfwd.
- mkimage: unknown partition names fail at build time; DISK_END is a max,
not last-entry; --state keys validated as filenames.
- portal-scenario: mock readiness is asserted (no silent fall-through),
hostfwd port collisions retried, mount-failure sentinel fails fast.
- ui-shot: fixed sleeps replaced with bounded screendump polling; the
repaint assertion is real and documented as such. qmp.py loses its
module-global and gains argv validation.
Docs/scrub: bench-host paths and the site AP name removed from six more
port docs and two evidence tables; path-bearing build artifacts (.elf,
.map) untracked (the 154-byte firmware .bin is path-free and stays);
ADR-0003 marked visibility-superseded by ADR-0007; stale section
cross-reference fixed; flare-edge noted as private for outside readers;
stale root-level review report removed per the new workspace rule.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018HUayid7W5w7jBdb9Rrj1K
Phase 4+5 of the device sim:
- Display + touch verified end-to-end: virtio-gpu at 720x720 (fbdev
emulation) renders the real WardenOS dashboard from the static LVGL
fbdev+evdev UI build (flare-edge qemu-vm-support tools/build-ui-vm.sh);
QMP input-send-event taps the Metrics tab and qemu/tests/ui-shot.sh
asserts the repaint from screendumps. Two load-bearing QEMU flags found
and documented: -global virtio-mmio.force-legacy=false (gpu/input are
VERSION_1-only) and the 200ms press hold (an instantaneous press+release
lands inside one LVGL indev poll and never clicks).
- qemu/tests/qmp.py: minimal QMP client (screendump, tap, quit).
- stage-2 init starts warden-ui when present and fb0 exists.
- docs/decisions/0006-qemu-device-sim.md: virt-not-custom-board, the
enters-at-kernel boundary, fragment policy, naming, consequences.
- docs/architecture.md: new section 7 (device emulation), order-of-work
item 7; modbus cross-reference to the bridge.
- qemu/README.md: emulated-vs-not table, scenarios, gotchas, host/runner
requirements. docs/ci-cd.md: runner needs one-time qemu-system-arm
install (fail-closed smoke until then, [maintainer]-gated). Repo README updated.
Final sweep on this commit: shellcheck clean, bridge 7/7 tests, boot smoke
PASS, portal scenario PASS (check-in + fw pull + signed .wfw download),
ui-shot PASS (touch navigates to Metrics) — all under the final flags.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018HUayid7W5w7jBdb9Rrj1K
Phase 3 of the device sim, all verified on QEMU 10.0.11:
- qemu/rs485-bridge/: std-only crate bridging a QEMU serial chardev (unix
socket) to warden_sim::ModbusSlave — gap-based RTU framing (CRC failures
degrade to real-slave silence), line-protocol control socket for register
seeding and fault injection (drop/exception/clear), bounds-checked so a
scenario typo answers err instead of panicking the bus. 7 unit tests,
bench in the sim_bench pattern. Verified end-to-end: guest master frame
on /dev/ttyS4 (pci-serial) answered from the sim slave, CRC-correct.
- virt machine gains highmem=off: the 32-bit non-LPAE kernel cannot reach
virt's default 40-bit PCIe ECAM (pci-host-generic EOVERFLOW); with it the
full PCI set probes (16550A ttyS0, i6300esb).
- Watchdog scenario verified: guest arms /dev/watchdog, no petting, i6300esb
resets the VM ~30s later (first environment where this arm is testable).
- qemu/tests/portal-scenario.sh: the real static-musl warden-flared inside
the VM against flare-edge's mock portal on the host — authenticated
check-in, firmware desired-state pull, and download of a real signed
tier-1 .wfw offer, asserted from the portal log. Found and filed
flare-edge#106 (fatal SIGBUS in the HPMCU boot-loaded probe on
non-RV1106 memory maps); runs against a flared built from the
qemu-vm-support fix branch.
- stage-2 init: WARDEN_FLARE_INSECURE=1 + WARDEN_HPMCU=0 (documented VM
deviations), firmware-version stamp, newline-terminated state seeds.
- CI: rs485-bridge joins the test loop and bench job; new qemu-tools job
(shellcheck + initramfs + disk image on hosted runners); kernel-build
gains a fail-closed qemu boot-smoke step. All qemu scripts shellcheck-clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018HUayid7W5w7jBdb9Rrj1K
The third simulator (deliberately not named "sim"): a QEMU -M virt VM that
boots the real 6.18.46 kernel and enters at -kernel zImage — everything below
(BootROM/idblock/U-Boot/real BCB A/B selection) is closed blobs + mask ROM
and is explicitly out of scope.
- qemu/mkinitramfs.sh: pinned static busybox (sha256 fail-closed) + rootfs/
- qemu/mkimage.sh: unprivileged sparse disk image with the device's canonical
12-partition blkdevparts A/B layout (vda == mmcblk0 mapping)
- qemu/rootfs/: stage-1 init (by-name symlinks from PARTNAME uevents,
whole-token warden.slot= parse, switch_root) + stage-2 init (userdata/oem
mounts, slirp networking, payload daemon start)
- qemu/run.sh: runner with --slot/--rtc/--watchdog/--rs485/--qmp/--display
- qemu/configs/virt.fragment + WARDEN_KCONFIG_FRAGMENT hook in
build/build-kernel.sh (canonical RV1106 build untouched when unset):
adds PCI, pci-serial, i6300esb watchdog, WireGuard, virtio-gpu/input
- qemu/tests/boot-smoke.sh: sentinel-asserting boot test
Verified on QEMU 10.0.11: canonical zImage boots -M virt unmodified (the
feared DEBUG_UNCOMPRESS decompressor hang does not exist in 6.18); full
stack boots both slots; 12 by-name symlinks; userdata persists across
reboot; -rtc base=2021-01-01 reproduces the no-RTC wrong-clock class.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018HUayid7W5w7jBdb9Rrj1K
