From c756622c960d8807f9238c3abff05333354bff38 Mon Sep 17 00:00:00 2001 From: BFE Engineering Date: Sun, 30 Aug 2026 22:19:12 -0600 Subject: [PATCH] docs: reposition as the 86 Panel development environment The repo's documentation framed it as a support repo for one product (WardenOS). Since going public the real audience is anyone with a Luckfox Pico 86 Panel: a maintained 6.18 kernel, an off-device development loop, and a device simulator that exist nowhere else for this board. Reframe the README and top-level docs board-first, with WardenOS documented as the downstream consumer it is (ADR-0008). Also an editorial pass over the whole doc set: - every H1/H2 is now a short title, not a sentence (ADRs, qemu/, patches/, drivers/, architecture, NPU feasibility, config-lint, payload); workflow flowchart titles fixed at the source in tools/flowgen.py and regenerated with fresh bench numbers - README Quick Start commands verified against the scripts; requirements corrected (curl, bare python, gcc >= 14) and the MC/DC gate added as a step (run green locally on gcc 14.2) - dropped the 'needs python (not python3)' vendor dig: build-kernel.sh inherited the same requirement (filed #10 to remove it) - glossed MC/DC and HPMCU on first use; marked the tests/uboot-ab reference as flare-edge; deduplicated the three-simulator list into the root README table --- README.md | 204 +++++++++--------- docs/architecture.md | 28 +-- docs/ci-cd.md | 2 +- docs/decisions/0001-kernel-base.md | 2 +- docs/decisions/0002-mcdc-tiering.md | 2 +- docs/decisions/0003-standalone-repo.md | 2 +- docs/decisions/0004-ci-runner.md | 2 +- docs/decisions/0005-driver-source-of-truth.md | 2 +- docs/decisions/0006-qemu-device-sim.md | 2 +- .../0007-public-repo-hosted-kernel-build.md | 2 +- .../0008-panel-devenv-positioning.md | 28 +++ docs/npu-graphics-feasibility.md | 34 +-- docs/workflows/README.md | 14 +- docs/workflows/cru-reset-ladder.md | 4 +- docs/workflows/freshness-contract.md | 2 +- docs/workflows/hpmcu-watchdog.md | 2 +- docs/workflows/modbus-read-holding.md | 4 +- docs/workflows/relay-drive.md | 2 +- docs/workflows/rga-offload.md | 4 +- drivers/README.md | 6 +- patches/README.md | 4 +- qemu/README.md | 27 ++- qemu/payload/README.md | 5 +- sim/bench.json | 7 + tools/config-lint/Cargo.toml | 2 +- tools/config-lint/README.md | 8 +- tools/flowgen.py | 14 +- 27 files changed, 221 insertions(+), 194 deletions(-) create mode 100644 docs/decisions/0008-panel-devenv-positioning.md create mode 100644 sim/bench.json diff --git a/README.md b/README.md index cbbe3d2..2f5baf3 100644 --- a/README.md +++ b/README.md @@ -5,133 +5,125 @@ ![Tests](.github/badges/tests.svg) ![Coverage](.github/badges/coverage.svg) -The build, driver, and simulation SDK for WardenOS (the Luckfox Pico 86-Panel / -RV1106 HMI). A from-scratch replacement for the twice-ported vendor stack -(Rockchip SDK → Luckfox SDK → our patched fork), built to the same standard as -the rest of the firmware: tested, benchmarked, reproducible, and honest about -what runs on real silicon versus what we simulate. +A modern, open development environment for the **Luckfox Pico 86 Panel** +(Rockchip RV1106): a current Linux kernel as a reviewable patch series, a +hermetic build, a QEMU device simulator, register-level hardware models, and +MC/DC-hardened drivers (Modified Condition/Decision Coverage — the +avionics-grade test bar). It replaces the vendor stack — a ~2 GB, twice-forked +SDK pinned to Linux 5.10 — with tooling that is tested, benchmarked, +reproducible, and honest about what runs on real silicon versus what is +simulated. -> Status: **bringup.** The hardware **simulator** and its tests, the RV1106 kernel -> forward-port as a reviewable `patches/` series, the hermetic kernel build, two -> Tier-1 drivers at 100% MC/DC, and the **QEMU device sim** (`qemu/`, ADR-0006: -> boots the real kernel + real userspace on `-M virt` — check-in/OTA against the -> mock portal, watchdog, RS485-to-sim bridge, 720x720 display + touch, all -> emulation-verified) are in. What remains before this is on the production build -> path: having flare-edge consume warden-sdk as a dependency (maintainer-gated). -> Until then, flare-edge still builds firmware from the vendored SDK + -> `sdk-patches/`. +Originally built as the SDK for WardenOS (BlueFlare Energy's wall-panel +firmware), but nothing here requires it: if you have an 86 Panel, this repo +gives you a modern kernel and a way to develop for the board without flashing +it on every change. -## Why a new SDK +## Why -The vendored SDK is a ~2 GB opaque fork of a fork. Our real changes to it lived, -until recently, as uncommitted edits in one working copy (`flare-edge/sdk-patches/` -is the tracked form). It bakes absolute paths, needs `python` (not python3), -silently drops Kconfig options, and — the failure that motivated this repo — gives -us **no way to test hardware-dependent code off the device.** Every driver change -had to be validated by flashing a panel. That is slow, and it is dangerous: it is -how a boot-loaded-watchdog change bricked a bench unit (the load address collided -with unreserved kernel RAM — a mistake a target-config check or a memory-map model -would have caught before any flash). +The vendor SDK bakes in absolute paths, silently drops Kconfig options, and +offers **no way to test hardware-dependent +code off the device** — every change means flashing a panel. That is slow and +occasionally destructive: a coprocessor load address that collided with +unreserved kernel RAM bricked a bench unit, a mistake a static memory-map +check would have caught before any flash (`tools/config-lint` now is that +check). This SDK exists so the board is buildable, testable, and hardenable +**without a panel in the loop**, on a maintained kernel. -The SDK's job is to make the firmware **buildable, testable, and hardenable -without a panel in the loop**, and to move us onto a modern, maintained kernel. +## What Works -## Goals (from future-features) +A self-built **Linux 6.18.46**, forward-ported from the vendor 5.10.160 tree +as a subsystem-split patch series (`patches/`) and hardware-verified on a +bench panel: clk, pinctrl, eMMC, GMAC, TRNG, OTP, SARADC/TSADC, RTC, USB +host, PWM/backlight, VOP display, GT911 touch, AIC8800 wifi, RGA, I2S audio, +the HPMCU (RISC-V watchdog coprocessor) mailbox, the open NPU driver, and +PVTM. Mainline alone was not +viable (no RV1106 device tree, clock, display, RGA, NPU, or flash-boot support +upstream); see `docs/decisions/0001-kernel-base.md`. -1. **Modern kernel.** A self-built **Linux 6.18.46**, forward-ported directly from - the vendor 5.10.160 tree (no plan44/OpenWrt code) on our current Buildroot LTS - (2025.02.x). This is **done and hardware-verified on `warden-c8a3`**: essentially - every RV1106 block the 86-Panel uses boots and works — clk, pinctrl, eMMC, GMAC, - TRNG, OTP, SARADC/TSADC, RTC, USB host, PWM/backlight, **VOP display**, **GT911 - touch**, **AIC8800 wifi**, **RGA**, **I2S audio**, **HPMCU mailbox**, the **open - NPU driver**, and **PVTM**. Mainline was not viable (no DT/clk/display/RGA/NPU/ - flash-boot upstream for RV1106); the direct 5.10→6.18 forward-port reuses the - already-in-mainline rv1126 register data where it matches and carries our deltas - as a reviewable patch series (`patches/`). -2. **Ported, hardened drivers → 100% MC/DC on the code we own.** "100% MC/DC on - 100% of drivers" is infeasible as literally stated: ~97% of driver LOC is - vendor blobs (the AIC8800 wifi driver alone is 88.5K lines). So the target is - **tiered**: real MC/DC on *our* hardware code (modbus master, relays, RGA - wrapper, HPMCU supervisor, devmem/reset ladder); fault-injection + branch - hardening for the vendor blobs behind a stable seam. -3. **A proper simulator.** Simulate the hardware the vendor SDK cannot: **RGA** - (2D blitter), the **RISC-V HPMCU** coprocessor, and the **NPU** — plus the - register/SRAM (`/dev/mem`) and sysfs surfaces the drivers touch — so driver and - supervisor logic runs and is tested on the host, in CI, with no panel. -4. **Its own repo, held to firmware standards.** Tests, benchmarks, reproducible - builds, CI. This repo. +## Quick Start -## Architecture — one seam, two backends +Requirements: `gcc-arm-linux-gnueabihf`, `qemu-system-arm`, `curl`, `cpio`, +`mkfs.ext4`, a bare `python` on PATH (Debian/Ubuntu: `python-is-python3`), +gcc >= 14 (for the MC/DC gate), and Rust (for the simulators' tests). -The organizing idea is a thin **Hardware Abstraction Seam** per hardware block. -Firmware code talks to the seam (a trait in Rust, a function table in C); the seam -has two backends: +```sh +# 1. Build the kernel: fetch pinned pristine 6.18.46, apply patches/, emit +# zImage + rv1106-warden.dtb. WORK must sit outside any git checkout. +WORK=$HOME/kbuild-out CROSS_COMPILE=arm-linux-gnueabihf- bash build/build-kernel.sh -``` - firmware / driver logic - │ - Hardware Abstraction Seam (devmem, hpmcu, rga, npu, modbus, gpio) - ┌────┴────┐ - real backend sim backend - (/dev/mem, ioctl, (software model, - /proc, serial) host-testable) +# 2. Boot it in the QEMU device simulator (no hardware needed): +bash qemu/mkinitramfs.sh +bash qemu/mkimage.sh +bash qemu/run.sh --kernel $HOME/kbuild-out/linux-6.18.46/arch/arm/boot/zImage --shell + +# 3. Run the test suites: +for d in sim tools/config-lint qemu/rs485-bridge; do + (cd "$d" && cargo test) +done +for d in drivers/*/test; do make -C "$d" check; done # 100% MC/DC gate (gcc >= 14) ``` -- **On-device**, the seam binds the real backend (mmap `/dev/mem`, `librga` - ioctls, the serial port, `/proc/rknpu`). -- **On the host**, it binds the **sim backend** — a faithful software model of the - block. The HPMCU sim, for example, runs the SCR1 watchdog firmware's exact state - machine (boot-grace, heartbeat-timeout, fire) against an in-memory mailbox, so - the flared supervisor's arm/beat protocol is exercised end-to-end in a unit test. - -The seam is the same object the driver-hardening effort measures MC/DC against, -and the same object the simulator implements — so the two goals reinforce rather -than duplicate each other. +The kernel variant with the simulator's extra devices (PCI serial, watchdog, +WireGuard, display) adds one env var to step 1: +`WARDEN_KCONFIG_FRAGMENT=qemu/configs/virt.fragment`. See `qemu/README.md` +for the scenario tests (portal, OTA apply, display + touch, watchdog). ## Layout ``` -sim/ the hardware simulator (Rust): membus/devmem, HPMCU, CRU, Modbus, RGA, NPU. -qemu/ the device simulator (ADR-0006): QEMU -M virt boots the real kernel and - real userspace; A/B disk layout, RS485 bridge into sim/, scenario tests. -drivers/ our own hardened drivers + their seams (relays, freshness; more migrate in). -patches/ the RV1106 kernel forward-port delta onto pristine linux-6.18.46 (subsystem-split). -kernel/ forward-port docs + provenance (rv1106-enablement/, PROVENANCE.md). -build/ the hermetic kernel build (fetch pristine → apply patches → zImage + dtb). -docs/ architecture + ADRs (decisions/) + ci-cd + generated workflow flowcharts. -tools/ dev tooling. config-lint: static target-config gates (MCU-load-vs-reserved-memory, - the 0x40000 brick class); flowgen: the workflow-flowchart generator. -.github/ CI (workflows/ci.yml): patches-apply, host tests, coverage, MC/DC, benchmarks, badges. +patches/ the RV1106 forward-port onto pristine linux-6.18.46 (subsystem-split) +build/ the hermetic kernel build (fetch pinned source -> apply patches -> zImage + dtb) +qemu/ the device simulator: QEMU -M virt boots the real kernel and real userspace; + A/B disk layout, RS485 bridge into sim/, scenario tests +sim/ register-level hardware models (Rust): membus, HPMCU, CRU, Modbus, RGA, NPU +drivers/ hardened hardware-facing drivers with HAL seams and 100% MC/DC harnesses +kernel/ forward-port provenance and bring-up records (point-in-time; patches/ is canonical) +tools/ config-lint (static memory-map gates) and dev tooling +docs/ architecture, ADRs (decisions/), CI/CD ``` +## Architecture + +One thin **hardware abstraction seam** per block (a trait in Rust, a function +table in C). Firmware logic talks to the seam; the seam binds a real backend +on the device (`/dev/mem`, ioctls, serial, `/proc`) or a simulated backend on +the host. The same seam is what the driver-hardening effort measures MC/DC +against and what the simulator implements, so the two reinforce rather than +duplicate each other. Full detail: `docs/architecture.md`. + +Three simulators, by design not one: + +| Simulator | What it runs | What it proves | +|---|---|---| +| `sim/` | register-level Rust models | driver and supervisor logic, with fault injection | +| `qemu/` | the real kernel + real userspace on `-M virt` | boot, init, daemons, networking, OTA, watchdog, display + touch | +| `lvglsim` (downstream) | the LVGL UI on SDL | rendering and UI flows | + +Boots and passes under emulation are never treated as on-silicon evidence; +the simulators narrow which claims need a panel, they do not replace it. + ## Principles -Evaluated against the stack philosophy — **openness, hardness, modernness**: +- **Open**: open tools over closed ones (`rkdeveloptool`, source-built + components, an open simulator); GPL-2.0-only. +- **Hard**: every seam has a fault-injection path — recovery code is tested + against failure, not just success. +- **Modern**: the newest kernel the hardware can run, current toolchains, + Rust for new host-testable code, reproducible builds. -- **Open** over closed where we can: `rkdeveloptool` over the closed `upgrade_tool`; - source-buildable `librga` over blobs where a source path exists; the simulator is - fully open and ours. -- **Hard**: every seam has a fault-injection path (a wedged SDIO link, a stalled - MCU, an RGA timeout) so recovery code is tested against failure, not just success. - On-device claims still need on-device evidence; the sim narrows *which* claims - need a panel, it does not replace that rule. -- **Modern**: newest kernel we can actually run; current Buildroot LTS; Rust for new - host-testable code; reproducible builds. +## Downstream -## Relationship to flare-edge - -flare-edge (WardenOS: the LVGL UI + the `flared` daemon) is the product; warden-sdk -is what builds and tests it. flare-edge is BlueFlare's private companion repo — -not publicly available — so flare-edge issue references and checkout paths in -this repo's docs are context, not reachable links. During bootstrap, flare-edge consumes warden-sdk piece -by piece: first the simulator (as a dev/test dependency), later the image build. -No flare-edge code moves here — only the SDK/build/sim/driver-seam layer. +WardenOS (the 86 Panel firmware this SDK was born for) consumes warden-sdk +from its own private repo, flare-edge; issue references and checkout paths +pointing there are context, not reachable links. The QEMU simulator runs its +production binaries unmodified — including real over-the-air updates against +a mock portal. ## License -**GPL-2.0-only**, repo-wide (see `LICENSE`; a per-file SPDX identifier governs -where one is present, e.g. a few GPL-2.0-or-later kernel files). The kernel -material in `patches/` and `kernel/rv1106-enablement/` is derivative of the -Linux kernel and of GPL-2.0 vendor code either way — per-driver origin and -license are tracked in `kernel/rv1106-enablement/PROVENANCE.md`. Contributions -are accepted under the same license (inbound = outbound). +**GPL-2.0-only**, repo-wide (see `LICENSE`; a per-file SPDX identifier +governs where present). The kernel material in `patches/` and `kernel/` is +derivative of the Linux kernel and GPL-2.0 vendor code; per-driver origin is +tracked in `kernel/rv1106-enablement/PROVENANCE.md`. Contributions are +accepted under the same license (inbound = outbound). diff --git a/docs/architecture.md b/docs/architecture.md index 812773d..9fa1d0d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,10 +1,12 @@ -# warden-sdk architecture +# Architecture -How the SDK makes WardenOS buildable, testable, and hardenable without a panel in -the loop. Grounded in a full survey of the current flare-edge firmware (the seam -inventory below is from that survey, not aspiration). +How the SDK makes the 86 Panel buildable, testable, and hardenable without a +board in the loop. The seam inventory below comes from a full survey of the +downstream WardenOS firmware — the SDK's first consumer — not aspiration; the +file paths in it point into that (private) codebase and serve as engineering +context. -## 1. The problem the seams solve +## 1. The Problem The firmware touches RV1106 hardware through a *grab-bag* of mechanisms, each tested (or not) differently. Today: @@ -26,7 +28,7 @@ socket, misc dev, cmdline, hpmcu fw), and **fails-soft-because-the-path-is-absen "relay 1 is ON" or "NPU at 80%", only "absent". The SDK's job is to turn all of these into **one deliberate seam per block** with a real backend and a sim backend. -## 2. The seam taxonomy +## 2. Seam Taxonomy Two seam kinds cover everything above: @@ -47,7 +49,7 @@ backend is "a fake `improcess` that records the blits it was asked to do", swapp behind the same `#if`, so the offload *dispatch* logic gets tested even though the blit itself is modelled. -## 3. The simulator (`sim/`) +## 3. The Register Simulator A host Rust library modelling the hardware the vendor SDK cannot, so driver and supervisor logic runs in CI with no panel. @@ -92,12 +94,12 @@ which is a **maintainer go-ahead item** (credential/remote creation). Until then firmware-side seam and a local test double land in flare-edge, unified with `sim/` once the dependency exists. No duplication of *logic* — only the tiny trait. -## 4. Driver hardening (the "port + harden to MC/DC" goal) +## 4. Driver Hardening "100% MC/DC on 100% of drivers" is infeasible literally: ~97% of driver LOC is vendor blobs (AIC8800 wifi = 88.5K lines). Tiered target: -- **Tier 1 — our own hardware code → real MC/DC.** Method: the proven `tests/uboot-ab` +- **Tier 1 — our own hardware code → real MC/DC.** Method: the proven flare-edge `tests/uboot-ab` pattern — extract the unit behind a small injectable seam, mock its world, build `-fcondition-coverage`, enforce with the shared `drivers/enforce-mcdc.sh` (gcc-14 `gcov --conditions`) in the CI `mcdc` job. **Done here now:** `relays.c` (40/40 @@ -116,7 +118,7 @@ Every seam gets a fault-injection mode (a wedged SDIO link, a stalled MCU heartbeat, an RGA timeout, a GPIO write EIO) so recovery code is tested against failure, not just the happy path. -## 5. Target-config checks (a class the sim cannot cover) +## 5. Target-Config Checks The brick was a *memory-map* fault: the boot-loaded MCU's load address (`0x40000`) is a reserved carve-out on Thunder-Boot boards but plain kernel RAM on ours. No @@ -136,7 +138,7 @@ fails against a DT with no `rtos@40000` node and passes once the reservation is added. **Next** target-config checks: partition-table-vs-image-size and vermagic-vs-kernel. -## 6. Kernel forward-port (done — see ADR-0001) +## 6. Kernel Forward-Port A self-built **Linux 6.18.46**, forward-ported directly from the vendor 5.10.160 tree onto our Buildroot LTS/uClibc base — **not** the plan44/OpenWrt 6.6 fork this section @@ -154,7 +156,7 @@ kernel move as one matched boot+oem image, never a partial reflash. series honest against pristine 6.18.46; provenance is in `patches/README.md` and `kernel/rv1106-enablement/`. -## 7. Device emulation (`qemu/`) — see ADR-0006 +## 7. Device Emulation The third simulator, deliberately not named "sim": a QEMU VM (`-M virt,highmem=off`, one Cortex-A7, 256M — the RV1106G3's shape) that boots the real forward-ported @@ -178,7 +180,7 @@ development* stays `lvglsim`; the VM is where processes, the kernel, and the network meet. §5 still applies — no behavioural sim, this one included, catches memory-map faults; and "boots under emulation" is never on-silicon evidence. -## 8. Order of work +## 8. Order of Work 1. **Simulator core** — `membus`, `hpmcu`, the `cru` reset ladder, `modbus`, plus the `rga`/`npu` models. **Done.** diff --git a/docs/ci-cd.md b/docs/ci-cd.md index 0873833..b21fc62 100644 --- a/docs/ci-cd.md +++ b/docs/ci-cd.md @@ -18,7 +18,7 @@ execute code on private infrastructure (ADR-0007). | `prune-artifacts` | ubuntu-latest, dispatch-only | Deletes `kernel-rv1106` artifacts beyond the newest 3. | | `badges` | ubuntu-latest | Renders loc/tests/coverage shields on push to `main` (`[skip ci]` + `paths-ignore` loop guard). | -## History: the self-hosted runner (retired) +## Runner History `kernel-build` originally ran on a repo-scoped self-hosted runner (ADR-0004, 2026-08-25, verified end-to-end) because hosted minutes were metered on the diff --git a/docs/decisions/0001-kernel-base.md b/docs/decisions/0001-kernel-base.md index 0112322..9306f37 100644 --- a/docs/decisions/0001-kernel-base.md +++ b/docs/decisions/0001-kernel-base.md @@ -1,4 +1,4 @@ -# ADR 0001 — Kernel base: direct 5.10→6.18 forward-port +# ADR 0001 — Kernel Forward-Port **Status:** Accepted (2026-08-25). Supersedes the README's original plan44/6.6 goal. diff --git a/docs/decisions/0002-mcdc-tiering.md b/docs/decisions/0002-mcdc-tiering.md index a97ff3f..1e08c7c 100644 --- a/docs/decisions/0002-mcdc-tiering.md +++ b/docs/decisions/0002-mcdc-tiering.md @@ -1,4 +1,4 @@ -# ADR 0002 — Tiered MC/DC coverage policy +# ADR 0002 — Tiered MC/DC **Status:** Accepted (2026-08-25). diff --git a/docs/decisions/0003-standalone-repo.md b/docs/decisions/0003-standalone-repo.md index bc592e5..5b118dc 100644 --- a/docs/decisions/0003-standalone-repo.md +++ b/docs/decisions/0003-standalone-repo.md @@ -1,4 +1,4 @@ -# ADR 0003 — warden-sdk is a standalone repo +# ADR 0003 — Standalone Repo **Status:** Accepted (2026-08-25). Repo-visibility half superseded by ADR-0007 (2026-08-30) — warden-sdk went public; the "private for now" consequence below diff --git a/docs/decisions/0004-ci-runner.md b/docs/decisions/0004-ci-runner.md index 2c2d831..ffe161f 100644 --- a/docs/decisions/0004-ci-runner.md +++ b/docs/decisions/0004-ci-runner.md @@ -1,4 +1,4 @@ -# ADR 0004 — CI/CD runner: 3rd repo-scoped self-hosted runner on 0640 +# ADR 0004 — Self-Hosted Runner **Status:** Superseded in part by ADR-0007 (2026-08-30) — `kernel-build` moved to GitHub-hosted runners for the public repo and the self-hosted registration diff --git a/docs/decisions/0005-driver-source-of-truth.md b/docs/decisions/0005-driver-source-of-truth.md index 8ac83bc..be6d10b 100644 --- a/docs/decisions/0005-driver-source-of-truth.md +++ b/docs/decisions/0005-driver-source-of-truth.md @@ -1,4 +1,4 @@ -# ADR 0005 — Hardened drivers live in warden-sdk/drivers/ +# ADR 0005 — Driver Source of Truth **Status:** Accepted (2026-08-25). diff --git a/docs/decisions/0006-qemu-device-sim.md b/docs/decisions/0006-qemu-device-sim.md index cd23ce9..23f69dd 100644 --- a/docs/decisions/0006-qemu-device-sim.md +++ b/docs/decisions/0006-qemu-device-sim.md @@ -1,4 +1,4 @@ -# ADR 0006 — QEMU device sim: generic `-M virt`, entered at the kernel +# ADR 0006 — QEMU Device Simulator **Status:** Accepted (2026-08-29). diff --git a/docs/decisions/0007-public-repo-hosted-kernel-build.md b/docs/decisions/0007-public-repo-hosted-kernel-build.md index f568587..504b112 100644 --- a/docs/decisions/0007-public-repo-hosted-kernel-build.md +++ b/docs/decisions/0007-public-repo-hosted-kernel-build.md @@ -1,4 +1,4 @@ -# ADR 0007 — Public repo: kernel-build moves to GitHub-hosted runners +# ADR 0007 — Hosted-Only CI **Status:** Accepted (2026-08-30). Supersedes the runner half of ADR-0004. diff --git a/docs/decisions/0008-panel-devenv-positioning.md b/docs/decisions/0008-panel-devenv-positioning.md new file mode 100644 index 0000000..c37c2fd --- /dev/null +++ b/docs/decisions/0008-panel-devenv-positioning.md @@ -0,0 +1,28 @@ +# ADR 0008 — Panel Development Environment + +**Status:** Accepted (2026-08-30). + +## Context +warden-sdk was written as the SDK for WardenOS, and its documentation framed +it that way: a support repo for one product. Since going public (ADR-0007, +GPL-2.0-only), the actual audience is wider — anyone with a Luckfox Pico 86 +Panel gets a maintained 6.18 kernel, an off-device development loop, and a +device simulator out of this repo, none of which exists elsewhere for this +board. The product-first framing undersold that and confused the entry point +for outside readers. + +## Decision +Position warden-sdk as **a modern, open development environment for the +Luckfox Pico 86 Panel (RV1106)**. WardenOS is documented as the downstream +consumer it is, not the purpose. Documentation follows three rules: lead with +the board, not the product; keep private-repo references clearly marked as +context; keep titles short — a heading names a section, it does not summarize +it. + +## Consequences +- README and top-level docs lead with the hardware and the developer loop + (build, simulate, test), with verified quick-start commands. +- WardenOS/flare-edge specifics stay where they are engineering truth (the + seam inventory, scenario payloads) but read as one consumer's usage. +- The honesty rule is unchanged: emulation results are never on-silicon + claims. diff --git a/docs/npu-graphics-feasibility.md b/docs/npu-graphics-feasibility.md index 071f292..7dc2c1c 100644 --- a/docs/npu-graphics-feasibility.md +++ b/docs/npu-graphics-feasibility.md @@ -1,4 +1,4 @@ -# Can the RV1106 NPU be used for 3D graphics or other graphical tasks? +# NPU Graphics Feasibility **Bottom line: no, not for 3D rendering — not "slower," but genuinely not how the hardware works past the first pipeline stage. The RKNPU on RV1106 is a @@ -24,9 +24,9 @@ or flagged as general knowledge / needing TRM confirmation. --- -## 1. What the RV1106 NPU actually is +## 1. What the NPU Is -### Identity and generation +### Identity and Generation - It is the **RKNPU** — Rockchip's 4th-generation NPU IP, exposed to tooling as the "RKNPU2" software generation (same toolchain family as RK3566/68/88), but @@ -56,7 +56,7 @@ or flagged as general knowledge / needing TRM confirmation. assigns **420MHz**. No datasheet states a default/rated frequency. (`luckfox-pico-86-panel/npu.md:11-13`) -### Architecture: fixed-function command-stream engine, not a programmable core +### Architecture This is the load-bearing fact for everything downstream. Reading the vendored kernel driver source directly (`flare-edge/sdk/sysdrv/source/kernel/drivers/rknpu/`): @@ -106,7 +106,7 @@ kernel driver source directly (`flare-edge/sdk/sysdrv/source/kernel/drivers/rknp primitive operations the hardware's fixed-function units implement underneath. It is not a route to arbitrary per-element or per-pixel code. -### Data types, memory, DDR sharing +### Data Types and Memory - **INT8-only quantization tier** for RV1106/RV1103 (`quantize=8` mandatory at conversion time) — inputs/outputs must be int8 and strictly 4-D. @@ -129,7 +129,7 @@ kernel driver source directly (`flare-edge/sdk/sysdrv/source/kernel/drivers/rknp the 128–256MB shared-DDR budget is the practical ceiling well before compute is. (`luckfox-pico-86-panel/npu.md:38`) -### Software stack — how it's actually driven +### Software Stack 1. **Kernel driver** (`rknpu.ko`, currently v0.9.2 on our shipped firmware) — exposes `/dev/rknpu` (a DRM device or a misc device, selectable at build time @@ -159,7 +159,7 @@ footgun on this board (no regulator wired) — never poll it. --- -## 2. 3D rendering feasibility, stage by stage +## 2. 3D Rendering, Stage by Stage A conventional 3D pipeline: **vertex transform → primitive assembly → rasterization → depth test → texture sampling → per-pixel shading → @@ -175,7 +175,7 @@ framebuffer write.** Verdict per stage, given everything in §1: | **Per-pixel shading** (arbitrary per-fragment program) | **No** | The NPU executes one fixed, precompiled graph over a whole tensor — it cannot run per-pixel conditional/arbitrary code. You could contrive a *specific* visual effect that literally is a small CNN (see §3), but that's not "shading" in the pipeline sense — it's a different, narrower thing wearing the name. | | **Framebuffer write** (write final pixels to the display's scanout buffer) | **No** | The NPU has no display/scanout connection at all — no DRM plane, no VOP link. Its only output path is writing tensor data to a DDR buffer, which is not a display pixel format. Something else (CPU or RGA) has to dequantize (`int8 → float → pixel`) and repack it into an actual framebuffer format before it's visible — and even RGA doesn't consume NPU tensor layouts directly (see §4). | -### Verdict: full or hybrid 3D pipeline +### Verdict **No full pipeline is possible on this hardware — five of six stages have no mapping at all, not a slow one.** A "hybrid" design where only vertex transform @@ -214,7 +214,7 @@ they are the wrong tool, full stop. --- -## 3. Other graphical tasks that might fit a CNN accelerator +## 3. CNN-Shaped Image Tasks Setting 3D aside — a CNN accelerator's real strength is convolution, which *does* map to some classic image-processing tasks. Evaluated against this @@ -228,7 +228,7 @@ specific 0.5–1 TOPS-class, 128–256MB-shared-DDR, no-camera product: | **Segmentation-driven UI effects** | Yes, in principle | **Moot — no camera, no visual input of any kind to segment.** | | **2D affine transforms** (rotate/scale/skew as matrix math) | Yes, technically a small matmul | **No — RGA already does this natively, in fixed-function hardware, cheaper.** RGA2-Enhance on this board already does scale (bicubic up / averaging down, to 16× either direction), rotate (90/180/270° on input windows), crop, and color/format conversion as dedicated blit-engine operations — no model compile, no INT8 quantization, no job-submit-and-IRQ-wait round trip, just a register-programmed blit. It is already wired into LVGL (the Monitor-page double-buffer-sync offload, verified 20%→8% CPU on real hardware) and proven in production. (`luckfox-pico-86-panel/rga.md`) | -### A concrete, on-record precedent: this was already considered and rejected once +### Prior Rejection The product wiki records that the keyboard's touch-bias correction (snapping an ambiguous tap to the nearest key) was **explicitly evaluated for NPU @@ -241,7 +241,7 @@ problem has no business going through a tensor accelerator's compile-and-submit pipeline. Nothing in this research changes that conclusion — if anything it generalizes it. -### Realistic verdict for §3 +### Realistic Verdict None of the CNN-shaped graphical tasks clear the bar for this specific product. Where a hardware assist genuinely helps (2D blit/scale/rotate/blend, classic @@ -255,9 +255,9 @@ detection, touch-gesture-pattern classification) — not graphics of any kind. --- -## 4. The driver-porting reality +## 4. Driver Porting -### What porting `rknpu.ko` to 6.18 actually involves +### Porting Scope - **This is a forward-port of Rockchip's out-of-tree vendor driver, not a from-scratch write.** The driver already carries version-gated compatibility @@ -299,7 +299,7 @@ detection, touch-gesture-pattern classification) — not graphics of any kind. driver forward against a newer kernel ABI" — bounded, evidence-backed, but real engineering, not a version-string bump. (`luckfox-pico-86-panel/rga.md:86`) -### Does "graphics" use need the full RKNN toolchain, or is there a more direct compute submit? +### Direct Submit Path **It needs the full toolchain.** As established in §1, the raw `DRM_IOCTL_RKNPU_SUBMIT` path exists at the kernel-ioctl level, but the @@ -313,7 +313,7 @@ slower-to-iterate loop than driving RGA (which is a direct, synchronous `im2d`-style C API call with no offline compile step at all) or writing plain CPU code. -### RGA is the existing, already-adequate 2D accelerator +### RGA Comparison Worth restating plainly since it's the thing the NPU would be compared against: **RGA2-Enhance already does everything this panel's UI plausibly needs from 2D @@ -332,7 +332,7 @@ silicon to work in, and it lines up with everything else in this document: RGA --- -## 5. Bottom-line recommendation +## 5. Recommendation | | Worth prototyping? | Why | |---|---|---| @@ -343,7 +343,7 @@ silicon to work in, and it lines up with everything else in this document: RGA | **Porting `rknpu.ko` to 6.18** | **Conditionally yes — but scope it for inference, not graphics** | Same bounded, evidence-backed effort class as the RGA port; keeps the door open for the platform wiki's actual identified NPU opportunity (a small non-visual classifier: audio, RS-485/sensor anomaly detection, touch-gesture patterns). Do not justify or scope the port around a graphics capability — it doesn't unlock one. | | **A first NPU spike, if one is wanted for team familiarity** | **Only the already-identified real use case** | A tiny non-visual model (e.g. an RS-485 anomaly classifier) — not a graphics stunt. This is the same conclusion the platform wiki already reached independent of this research. | -### What would need to happen for this document to be wrong +### Falsifiers Flagging explicitly, per the instruction to distinguish settled facts from things needing verification: diff --git a/docs/workflows/README.md b/docs/workflows/README.md index 6c4ddd3..c6b0e3a 100644 --- a/docs/workflows/README.md +++ b/docs/workflows/README.md @@ -1,11 +1,11 @@ -# Workflow flowcharts +# Workflow Flowcharts Generated by `tools/flowgen.py` from the modelled decision paths. Each is an outcome-first flowchart of a workflow the SDK tests, with its benchmark or MC/DC metric. -- [HPMCU watchdog: arm → beat → fire](hpmcu-watchdog.md) -- [Modbus RTU: read-holding-registers round trip](modbus-read-holding.md) -- [CRU reset ladder: cause + boot-mode survival](cru-reset-ladder.md) -- [RGA 2D offload dispatch](rga-offload.md) -- [Relay drive (Tier-1, 100% MC/DC)](relay-drive.md) -- [UI freshness contract (Tier-1, 100% MC/DC)](freshness-contract.md) +- [HPMCU Watchdog](hpmcu-watchdog.md) +- [Modbus RTU Round Trip](modbus-read-holding.md) +- [CRU Reset Ladder](cru-reset-ladder.md) +- [RGA Offload](rga-offload.md) +- [Relay Drive](relay-drive.md) +- [UI Freshness Contract](freshness-contract.md) diff --git a/docs/workflows/cru-reset-ladder.md b/docs/workflows/cru-reset-ladder.md index 0752226..081885d 100644 --- a/docs/workflows/cru-reset-ladder.md +++ b/docs/workflows/cru-reset-ladder.md @@ -1,8 +1,8 @@ -# CRU reset ladder: cause + boot-mode survival +# CRU Reset Ladder > **Outcome tested:** Reset cause is attributable and the boot-mode register survives a warm reset. -**Benchmark** (`cru_poll`): 23.0 ns/op +**Benchmark** (`cru_poll`): 21.4 ns/op ```mermaid flowchart TD diff --git a/docs/workflows/freshness-contract.md b/docs/workflows/freshness-contract.md index 2467cbd..c747afd 100644 --- a/docs/workflows/freshness-contract.md +++ b/docs/workflows/freshness-contract.md @@ -1,4 +1,4 @@ -# UI freshness contract (Tier-1, 100% MC/DC) +# UI Freshness Contract > **Outcome tested:** The UI never shows a stale number: it holds briefly, then marks unknown. diff --git a/docs/workflows/hpmcu-watchdog.md b/docs/workflows/hpmcu-watchdog.md index be7721b..a0f0f74 100644 --- a/docs/workflows/hpmcu-watchdog.md +++ b/docs/workflows/hpmcu-watchdog.md @@ -1,4 +1,4 @@ -# HPMCU watchdog: arm → beat → fire +# HPMCU Watchdog > **Outcome tested:** A hung A7/flared ends in a counted reset, not a dark panel. diff --git a/docs/workflows/modbus-read-holding.md b/docs/workflows/modbus-read-holding.md index 127c781..80e8ab7 100644 --- a/docs/workflows/modbus-read-holding.md +++ b/docs/workflows/modbus-read-holding.md @@ -1,8 +1,8 @@ -# Modbus RTU: read-holding-registers round trip +# Modbus RTU Round Trip > **Outcome tested:** A well-formed request yields the right registers; a bad one a defined fault. -**Benchmark** (`modbus_read_holding`): 88.0 ns/op +**Benchmark** (`modbus_read_holding`): 73.7 ns/op ```mermaid flowchart TD diff --git a/docs/workflows/relay-drive.md b/docs/workflows/relay-drive.md index e45cb62..d22a295 100644 --- a/docs/workflows/relay-drive.md +++ b/docs/workflows/relay-drive.md @@ -1,4 +1,4 @@ -# Relay drive (Tier-1, 100% MC/DC) +# Relay Drive > **Outcome tested:** A relay is exported transparently and driven without disturbing a held contact. diff --git a/docs/workflows/rga-offload.md b/docs/workflows/rga-offload.md index 727ae17..a0cd5a4 100644 --- a/docs/workflows/rga-offload.md +++ b/docs/workflows/rga-offload.md @@ -1,8 +1,8 @@ -# RGA 2D offload dispatch +# RGA Offload > **Outcome tested:** Blits go to the RGA when it succeeds, and fall back to the CPU when it doesn't. -**Benchmark** (`rga_improcess`): 7.2 ns/op +**Benchmark** (`rga_improcess`): 6.7 ns/op ```mermaid flowchart TD diff --git a/drivers/README.md b/drivers/README.md index 2cee72b..29b7133 100644 --- a/drivers/README.md +++ b/drivers/README.md @@ -1,11 +1,11 @@ -# drivers/ — our own hardened, hardware-facing drivers +# Hardened Drivers Per **ADR-0002** (tiered MC/DC) and **ADR-0005** (source-of-truth), our own hardware-facing code migrates here behind a HAL seam and is hardened. "100% MC/DC on 100% of drivers" is infeasible (≈97% of kernel-driver LOC is vendor blobs — AIC8800 alone is 88.5K lines); the realistic, honest target is tiered. -## Tier 1 — real 100% MC/DC (here now, CI-enforced) +## Tier 1 — 100% MC/DC Self-contained logic with a clean seam, measured to **100% MC/DC** (gcc-14 `-fcondition-coverage`) by the CI `mcdc` job (`make -C drivers/*/test check`): @@ -23,7 +23,7 @@ build/test.rc` (it derives the driver name from the `.gcov` file, so there is no per-driver copy to keep in sync). The CI `mcdc` job picks up any `drivers/*/test/Makefile` automatically. -## Tier 2 — serious testing + fault-injection + benchmarks +## Tier 2 — Fault Injection Drivers too large or too vendor/UI-coupled for literal MC/DC get fault-injection, branch coverage, and benchmarks against the simulator instead. Their **hardware side diff --git a/patches/README.md b/patches/README.md index 6195373..d8dfcd4 100644 --- a/patches/README.md +++ b/patches/README.md @@ -1,4 +1,4 @@ -# RV1106 kernel patch series (pristine linux-6.18.46 → WardenOS) +# The Kernel Patch Series The delta that forward-ports the vendor Rockchip 5.10.160 RV1106 support onto a pristine upstream **linux-6.18.46**, as a reviewable, subsystem-split series. Every @@ -9,7 +9,7 @@ reproduces the 223-file / ~136K-line delta that boots and is hardware-verified o Build it with `../build/build-kernel.sh` (fetch pristine → apply these → configure with `../build/warden_defconfig` → `zImage` + `rv1106-warden.dtb`). -## The series +## The Series | Patch | Scope | |---|---| diff --git a/qemu/README.md b/qemu/README.md index 02af94b..a0d201b 100644 --- a/qemu/README.md +++ b/qemu/README.md @@ -1,19 +1,16 @@ -# qemu/ — the WardenOS device simulator +# The Device Simulator A QEMU virtual machine that boots the real forward-ported kernel (`build/` + -`patches/`) and real userspace, so the *device* — init, daemons, networking, -OTA, watchdog, display — can be tested off-hardware. The third simulator in -the stack, deliberately not named "sim": - -- `lvglsim` (flare-edge) — SDL desktop build of the UI. Rendering only. -- `sim/` (this repo) — register-level Rust models of RV1106 blocks behind - driver seams. -- `qemu/` (this) — the whole machine above the kernel entry point, running - the real binaries. +`patches/`) and real userspace, so the 86 Panel — init, daemons, networking, +OTA, watchdog, display — can be developed and tested with no board attached. +It is the third simulator in the stack (the root README's table has the +three-way split), deliberately not named "sim": where `sim/` models +registers, this runs the whole machine above the kernel entry point on real +binaries — bring your own, or drop prebuilt payloads in `payload/`. Decision record: `docs/decisions/0006-qemu-device-sim.md`. -## The boundary (read this before trusting a green run) +## The Boundary There is no RV1106 machine model in QEMU and everything below the kernel is closed rkbin blobs plus mask ROM, so the VM **enters at `-kernel zImage`** on @@ -43,7 +40,7 @@ Documented guest deviations from production, set by stage-2 init: `WARDEN_HPMCU=0` (no mailbox SRAM on virt; flared >= flare-edge#106 fix required, or the daemon dies of SIGBUS). -## Quick start +## Quick Start ```sh # 1. kernel: canonical build boots the VM as-is; the fragment variant adds @@ -64,7 +61,7 @@ README) — `warden-flared`, `warden-modbus`, and `warden-ui` (the LVGL fbdev+evdev build from flare-edge `tools/build-ui-vm.sh`) are started by stage-2 init when present. -## Scenario tests (`qemu/tests/`) +## Scenarios - `boot-smoke.sh ` — sentinel-asserting boot; runs in CI inside the kernel-build job. @@ -83,7 +80,7 @@ stage-2 init when present. the VM resets ~30 s later (verified). Do NOT combine with a flared payload expecting survival: flared pets only while the UI heartbeat is fresh. -## Gotchas that cost time (so they cost it once) +## Gotchas - AF_UNIX socket paths cap at ~108 chars — keep `--rs485`/`--qmp` paths short. - A serial port that is closed discards incoming bytes: hold ONE fd open @@ -93,7 +90,7 @@ stage-2 init when present. which every script (boot smoke included) delegates to. - Never pass `earlyprintk`: DEBUG_UART_PHYS is the RV1106's 0xff4c0000. -## Host requirements +## Requirements `qemu-system-arm` (Debian 13 ships QEMU 10), `curl`, `cpio`, `mkfs.ext4`, `gcc-arm-linux-gnueabihf` (kernel build), `python3` (+`cryptography` for the diff --git a/qemu/payload/README.md b/qemu/payload/README.md index f5554a8..753d01e 100644 --- a/qemu/payload/README.md +++ b/qemu/payload/README.md @@ -1,6 +1,7 @@ -# qemu/payload/ — guest binaries (never committed) +# Guest Payloads -Drop **static musl armv7** binaries here; `qemu/mkimage.sh` copies everything +Drop **static musl armv7** binaries in this directory (contents are +gitignored — binaries are never committed); `qemu/mkimage.sh` copies everything in this directory (except this README) into `/usr/bin/` of both rootfs slots. Static musl is the same target the device uses for its Rust daemons, so the exact production binaries run unmodified in the VM. diff --git a/sim/bench.json b/sim/bench.json new file mode 100644 index 0000000..4e9bf4b --- /dev/null +++ b/sim/bench.json @@ -0,0 +1,7 @@ + Finished `bench` profile [optimized] target(s) in 0.08s + Running benches/sim_bench.rs (target/release/deps/sim_bench-57835de7b7b68615) +{"bench":"hpmcu_tick","ns_per_op":1.8,"iters":1000000} +{"bench":"cru_poll","ns_per_op":21.4,"iters":1000000} +{"bench":"modbus_read_holding","ns_per_op":73.7,"iters":1000000} +{"bench":"rga_improcess","ns_per_op":6.7,"iters":1000000} +{"bench":"membus_poke_peek","ns_per_op":27.0,"iters":1000000} diff --git a/tools/config-lint/Cargo.toml b/tools/config-lint/Cargo.toml index 578fd08..5f8711b 100644 --- a/tools/config-lint/Cargo.toml +++ b/tools/config-lint/Cargo.toml @@ -2,7 +2,7 @@ name = "warden-config-lint" version = "0.1.0" edition = "2021" -description = "Static target-config checks for WardenOS: catch memory-map faults (the 0x40000 MCU-load brick class) and other flash-time config mistakes before a flash, not on the bench." +description = "Static target-config checks for the Luckfox Pico 86 Panel: catch memory-map faults (the 0x40000 MCU-load brick class) and other flash-time config mistakes before a flash, not on the bench." license = "GPL-2.0-only" [[bin]] diff --git a/tools/config-lint/README.md b/tools/config-lint/README.md index c2bf3f6..86904e2 100644 --- a/tools/config-lint/README.md +++ b/tools/config-lint/README.md @@ -1,9 +1,9 @@ -# config-lint — static target-config gates +# config-lint -Catches flash-time config faults the behavioural sim cannot: mistakes in the +Static target-config gates: catches flash-time config faults the behavioural sim cannot: mistakes in the *memory map*, not the logic. The first check is the one that would have caught the **c8a3 brick** — a boot-loaded coprocessor firmware dropped at `0x40000`, which is -a `reserved-memory` carve-out on Thunder-Boot boards but plain kernel RAM on ours, +a `reserved-memory` carve-out on Thunder-Boot boards but plain kernel RAM on the 86 Panel, so the MCU and the kernel fought over the same DRAM and the board hung before eth0. ## The check @@ -37,6 +37,6 @@ found; `2` = usage/IO error. The suite encodes the brick as a regression: the real Thunder-Boot `.ini` (Hpmcu @ `0x40000`) *fails* against a DT with no `rtos@40000` node and *passes* -once the reservation is added — and our board's non-TB loader (no boot-loaded MCU) +once the reservation is added — and the 86 Panel's non-TB loader (no boot-loaded MCU) always passes. See `../../docs/architecture.md` §5 and, for the hardware hazard, the `boot-loaded-mcu-0x40000-hazard` note. diff --git a/tools/flowgen.py b/tools/flowgen.py index 330a466..1e2d219 100644 --- a/tools/flowgen.py +++ b/tools/flowgen.py @@ -27,7 +27,7 @@ OUT = os.path.join(REPO, "docs", "workflows") WORKFLOWS = [ { "name": "hpmcu-watchdog", - "title": "HPMCU watchdog: arm → beat → fire", + "title": "HPMCU Watchdog", "outcome": "A hung A7/flared ends in a counted reset, not a dark panel.", "metric": ("bench", "hpmcu_tick"), "mermaid": """flowchart TD @@ -42,7 +42,7 @@ WORKFLOWS = [ }, { "name": "modbus-read-holding", - "title": "Modbus RTU: read-holding-registers round trip", + "title": "Modbus RTU Round Trip", "outcome": "A well-formed request yields the right registers; a bad one a defined fault.", "metric": ("bench", "modbus_read_holding"), "mermaid": """flowchart TD @@ -58,7 +58,7 @@ WORKFLOWS = [ }, { "name": "cru-reset-ladder", - "title": "CRU reset ladder: cause + boot-mode survival", + "title": "CRU Reset Ladder", "outcome": "Reset cause is attributable and the boot-mode register survives a warm reset.", "metric": ("bench", "cru_poll"), "mermaid": """flowchart TD @@ -71,7 +71,7 @@ WORKFLOWS = [ }, { "name": "rga-offload", - "title": "RGA 2D offload dispatch", + "title": "RGA Offload", "outcome": "Blits go to the RGA when it succeeds, and fall back to the CPU when it doesn't.", "metric": ("bench", "rga_improcess"), "mermaid": """flowchart TD @@ -84,7 +84,7 @@ WORKFLOWS = [ }, { "name": "relay-drive", - "title": "Relay drive (Tier-1, 100% MC/DC)", + "title": "Relay Drive", "outcome": "A relay is exported transparently and driven without disturbing a held contact.", "metric": ("mcdc", "relays.c — 40/40 conditions, 100% MC/DC (CI-enforced)"), "mermaid": """flowchart TD @@ -102,7 +102,7 @@ WORKFLOWS = [ }, { "name": "freshness-contract", - "title": "UI freshness contract (Tier-1, 100% MC/DC)", + "title": "UI Freshness Contract", "outcome": "The UI never shows a stale number: it holds briefly, then marks unknown.", "metric": ("mcdc", "freshness.c — 66/66 conditions, 100% MC/DC (CI-enforced)"), "mermaid": """flowchart TD @@ -148,7 +148,7 @@ def metric_line(metric, bench): def main(): bench = load_bench(sys.argv) os.makedirs(OUT, exist_ok=True) - index = ["# Workflow flowcharts", "", + index = ["# Workflow Flowcharts", "", "Generated by `tools/flowgen.py` from the modelled decision paths.", "Each is an outcome-first flowchart of a workflow the SDK tests, with its" " benchmark or MC/DC metric.", ""]