qemu: json/stat/region channels, seeding, driver tests

- rootfs/sbin/init: the control bridge answers `@cat PATH` locally so the
  driver can read /tmp/warden-web-status.json out of the guest. That file
  has no trailing newline; the bridge adds one so the sentinel stays on
  its own line and the line-based reader never blocks.
- tests/qmp.py: wait_json/assert_json (dotted paths, eq/ne/contains/
  len_eq/len_ge/gt/lt), assert_stat off the FIFO's stats reply,
  capture_region, assert_region NAME [TOLERANCE] and assert_ocr. A
  tolerance other than the captured one, a reference box that does not
  fit the screendump, a missing reference or a missing tesseract is FATAL
  for that step and the run continues (flare-edge #147).
- tests/imgtools.py: P6 reader, crop, perceptual and structural hashes,
  compare, with a self-test.
- tests/test_qmp_drive.py: drive() with QMP and the control channel
  faked, pinning the per-step ok/fail/fatal contract.
- mkimage.sh SEED_DIR and ui-drive.sh --seed/--refs: settings fixtures
  staged into userdata before warden-ui starts, and a reference store
  handed to the driver.
- ci: the driver tests and the imgtools self-test run in qemu-tools.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013aHKWzT5EF86RFKRMtAv9n
This commit is contained in:
Noah
2026-09-07 23:26:01 -06:00
co-authored by Claude Fable 5.1
parent 8a57057053
commit 86a9544dcc
7 changed files with 1140 additions and 27 deletions
+392 -3
View File
@@ -35,6 +35,52 @@ a real panel, so these verbs mean the same thing on the rig and on hardware:
reports a MOVED button as such, not as broken
behaviour
Two more channels reach behavioural state instead of the widget tree.
`--refs FILE` (default <outdir>/refs.json) is the reference store the pixel
ones read and write; imgtools.py in this directory does the actual image
math, this file only drives it.
assert_json PATH OP VALUE
wait_json PATH OP VALUE TIMEOUT_S
PATH is dotted with optional [N] indexes (e.g.
network.scan.ranges[0]) into the guest's
/tmp/warden-web-status.json, fetched fresh on
every check with `@cat` over the SAME control
channel as nav/hit (see the bridge added to
rootfs/sbin/init). OP is one of eq, ne, contains,
len_eq, len_ge, gt, lt. `wait_json` polls every
0.5s until OP holds or TIMEOUT_S elapses and
`assert_json` reads once. A PATH the document
doesn't have is a `fail`, not a crash -- that
document is a live snapshot and the key might
simply not be built yet.
assert_stat FIELD OP VALUE
FIELD is cpu, fps or render, read off a fresh
`stats` reply (warden_debug.c). Same OP vocabulary
as assert_json.
capture_region NAME X Y W H TOLERANCE
screendump now, crop to X,Y,WxH, and (over)write
NAME in the refs file with both a phash and a
structural hash plus TOLERANCE (exact, loose or
structural). Run this once, by hand, to author a
reference -- the same way targets are authored,
see flows/README.md.
assert_region NAME [TOLERANCE]
fresh screendump, crop to NAME's stored box,
compare against its stored hash at its stored
tolerance. TOLERANCE, when given, must be the one
NAME was captured under (flowc.py always emits it).
assert_ocr NAME REGEX same crop, run through `tesseract`, REGEX searched
against the extracted text (spaces in REGEX are
fine, same as assert_hit's TEXT).
assert_region/assert_ocr against a NAME with no captured reference, and
assert_ocr with no `tesseract` on PATH, are FATAL for that one step: a
silently-skipped check reads as coverage that was never actually there. So
are a TOLERANCE that differs from the captured one and a stored box that does
not fit the screendump: both are the reference's fault, not the UI's, and one
bad reference must not take the rest of the script with it.
Every step is recorded to <outdir>/results.jsonl as ok / fail / fatal. An
assertion mismatch is a `fail` and the script continues, so one run reports
every broken expectation. With `--console LOG` the console is checked after
@@ -46,13 +92,38 @@ A step that names an unknown command is a FATAL error rather than a skip: a
silently-ignored line in a scenario is a test that proves nothing.
"""
import json
import os
import re
import shutil
import socket
import subprocess
import sys
import time
# imgtools.py lives beside this file and does the actual pixel math (phash,
# structural hash, crop, compare) for the region/ocr verbs. Imported at
# module load but never let a missing/broken imgtools take down the verbs
# that don't need it: nav/tap/page/... must keep working while it is being
# written, so the failure is deferred to need_imgtools() at first use.
try:
import imgtools
except ImportError:
imgtools = None
# Pillow, likewise, is only needed to hand tesseract an image file (it has no
# raw-RGB stdin mode); a host without it still runs every other verb.
try:
from PIL import Image as _PILImage
except ImportError:
_PILImage = None
AXIS_MAX = 32767
# Where webstatus.c (ui-src/src/warden/webstatus.c) publishes its atomic
# snapshot inside the guest. assert_json/wait_json `@cat` this path over the
# control channel; see the bridge in rootfs/sbin/init.
STATUS_JSON_PATH = "/tmp/warden-web-status.json"
def rpc(sock, sock_file, obj):
sock.sendall((json.dumps(obj) + "\n").encode())
@@ -198,8 +269,168 @@ def ui_exited(console_path):
return data[i:i + 80].split(b"\n", 1)[0].decode("utf-8", "replace")
def drive(s, f, script_path, outdir, size, ctl_path=None, console_path=None):
import os
PATH_TOKEN_RE = re.compile(r'^([^\[\]]*)((?:\[\d+\])*)$')
IDX_RE = re.compile(r'\[(\d+)\]')
def resolve_path(doc, path):
"""Walk a dotted PATH (keys and [N] indexes, e.g. 'a.b[0].c') over an
already-parsed JSON document. -> (value, None) or (None, 'no such path').
Pure and I/O-free on purpose (contract: testable on its own) -- fetching
the document is a separate step (fetch_status_json) so this function can
be unit-tested against a plain dict/list literal with nothing running.
"""
cur = doc
for part in path.split("."):
m = PATH_TOKEN_RE.match(part)
if not m:
return None, "no such path"
key, idxs = m.group(1), m.group(2)
if key:
if not isinstance(cur, dict) or key not in cur:
return None, "no such path"
cur = cur[key]
for idx_s in IDX_RE.findall(idxs):
idx = int(idx_s)
if not isinstance(cur, list) or idx >= len(cur) or idx < 0:
return None, "no such path"
cur = cur[idx]
return cur, None
def resolve_token(raw):
"""A verb's literal OP argument -> a comparable Python value. A JSON
literal (a number, true/false/null, or a quoted string) parses as
itself; anything else -- most values in practice, e.g. `connected` --
compares as the raw string. This is what lets both
`assert_json state.net eq connected` and `assert_json state.count eq 3`
work from one unquoted token, with no escaping convention of its own."""
try:
return json.loads(raw)
except ValueError:
return raw
def _len_of(v):
try:
return len(v)
except TypeError:
raise TypeError(f"{v!r} has no length") from None
def apply_op(op, actual, raw_value):
"""The comparison vocabulary shared by assert_json/wait_json/assert_stat:
eq, ne, contains, len_eq, len_ge, gt, lt. Raises on a combination that
cannot be judged (contains on a number, gt across incompatible types)
so the caller turns that into a `fail` detail instead of a driver crash --
a spec that asks a nonsensical question should be reported, not silently
True or False."""
value = resolve_token(raw_value)
if op == "eq":
return actual == value
if op == "ne":
return actual != value
if op == "contains":
if isinstance(actual, str):
return str(value) in actual
if isinstance(actual, (list, tuple, dict)):
return value in actual
raise TypeError(f"contains: {actual!r} is not a string or list")
if op == "len_eq":
return _len_of(actual) == int(value)
if op == "len_ge":
return _len_of(actual) >= int(value)
if op == "gt":
return actual > value
if op == "lt":
return actual < value
raise ValueError(f"unknown op {op!r}")
def fetch_status_json(ctl):
"""`@cat` the guest's webstatus.c snapshot over the control channel and
parse it. -> (doc, None) or (None, detail); never raises, so a torn read
or a not-yet-written file is an ordinary retry/fail for the caller, not a
crash of the whole driver."""
reply = ctl.send(f"@cat {STATUS_JSON_PATH}")
if reply.startswith("bridge: no such file"):
return None, reply
try:
return json.loads(reply), None
except ValueError as e:
return None, f"status json unparsable: {e}"
def eval_json(ctl, path, op, value):
"""One fetch + path-resolve + op-apply round for assert_json/wait_json.
-> (ok, detail); detail is empty on success, otherwise the reason."""
doc, err = fetch_status_json(ctl)
if err is not None:
return False, err
actual, perr = resolve_path(doc, path)
if perr is not None:
return False, perr
try:
ok = apply_op(op, actual, value)
except (TypeError, ValueError) as e:
return False, f"{op} {value!r} vs {actual!r}: {e}"
return ok, "" if ok else f"{path} is {actual!r}"
STATS_FIELD_RE = {
"cpu": re.compile(r'^cpu:\s*(-?\d+(?:\.\d+)?)%?\s*$'),
"fps": re.compile(r'^fps:\s*(-?\d+(?:\.\d+)?)\s*$'),
"render": re.compile(r'^render:\s*(-?\d+(?:\.\d+)?)\s*ms/frame\s*$'),
}
def parse_stats(reply):
"""The `stats` reply (warden_debug.c: 'page: X\\ncpu: N%\\nfps: N\\n
render: A.BB ms/frame\\nrga: N%') -> {'cpu'|'fps'|'render': float} for
whichever lines are present. A field the reply lacks is simply absent
from the result -- the caller reports that as 'no such field', the same
shape as resolve_path's 'no such path' for the json channel."""
out = {}
for ln in reply.splitlines():
ln = ln.strip()
for field, rx in STATS_FIELD_RE.items():
m = rx.match(ln)
if m:
out[field] = float(m.group(1))
return out
def load_refs(path):
"""The region/ocr reference store (see qmp.py's own docstring for the
schema). Missing file -> {}, same as an empty COVERAGE.yaml: capture_region
is how one gets created, not a prerequisite to have one already."""
if not path or not os.path.exists(path):
return {}
with open(path) as fh:
return json.load(fh)
def save_refs(path, refs):
# Written after every capture_region, not batched at the end: a script
# that dies three steps later must not lose a reference that was already
# good.
with open(path, "w") as fh:
json.dump(refs, fh, indent=2, sort_keys=True)
fh.write("\n")
def write_png(img, path):
"""imgtools' (w, h, rgb_bytes) tuple -> a PNG file, because tesseract has
no raw-RGB input mode. Only assert_ocr needs this; assert_region and
capture_region work on imgtools' own tuples end to end."""
if _PILImage is None:
raise RuntimeError("Pillow (PIL) is not installed")
w, h, data = img
_PILImage.frombytes("RGB", (w, h), data).save(path)
def drive(s, f, script_path, outdir, size, ctl_path=None, console_path=None, refs_path=None):
os.makedirs(outdir, exist_ok=True)
with open(script_path) as fh:
lines = fh.readlines()
@@ -221,6 +452,36 @@ def drive(s, f, script_path, outdir, size, ctl_path=None, console_path=None):
sys.exit(f"FATAL: {script_path}:{lineno}: '{cmd}' needs the control "
f"channel; run with --ctl (ui-drive.sh passes it)")
def need_imgtools(lineno, cmd):
# A whole-script infrastructure gap (imgtools.py absent), same class
# as a missing --ctl: no region/ocr verb can do anything without it,
# so this exits the process rather than recording a per-step fail.
if imgtools is None:
sys.exit(f"FATAL: {script_path}:{lineno}: '{cmd}' needs imgtools.py "
f"next to qmp.py (vendor/warden-sdk/qemu/tests/imgtools.py)")
refs_path = refs_path or os.path.join(outdir, "refs.json")
refs = load_refs(refs_path)
def fresh_region(name, tag):
"""screendump NOW (never reuse an earlier `shot`) and crop to NAME's
stored box. Fails closed on an uncaptured NAME: assert_region and
assert_ocr both need this, capture_region does not."""
ref = refs.get(name)
if ref is None:
return None, None, f"no reference for {name!r} (run capture_region first)"
path = os.path.join(os.path.abspath(outdir), f"{tag}_{name}.ppm")
rpc(s, f, {"execute": "screendump", "arguments": {"filename": path}})
try:
img = imgtools.load_ppm(path)
cropped = imgtools.crop(img, ref["x"], ref["y"], ref["w"], ref["h"])
except (ValueError, OSError, KeyError, TypeError) as e:
# A box outside this screendump (captured at another --size, or
# mistyped in the store) is a per-step fatal, never a traceback
# that ends the run: every later step still gets judged.
return None, None, f"reference {name!r} unusable: {e}"
return ref, cropped, None
for lineno, raw in enumerate(lines, 1):
line = raw.split("#", 1)[0].strip()
if not line:
@@ -296,7 +557,134 @@ def drive(s, f, script_path, outdir, size, ctl_path=None, console_path=None):
record(lineno, line, "fail", f"{cls} text={got[1]!r}")
else:
record(lineno, line, "ok")
elif cmd == "assert_json":
# assert_json PATH OP VALUE: one read of the guest's status JSON
# (see eval_json). PATH not (yet) in the doc is a `fail`, since
# the doc is a live snapshot, not a schema.
need_ctl(lineno, cmd)
ok, detail = eval_json(ctl, args[0], args[1], args[2])
record(lineno, line, "ok" if ok else "fail", detail)
elif cmd == "wait_json":
# wait_json PATH OP VALUE TIMEOUT_S: poll eval_json every 0.5s
# until it holds or the deadline passes. The status file is
# written by a 2s guest timer (webstatus.c), so this exists
# because a value the doc will reach shortly is not the same
# fact as a value it never reaches -- assert_json alone cannot
# tell those apart.
need_ctl(lineno, cmd)
timeout_s = float(args[3])
start = time.monotonic()
deadline = start + timeout_s
while True:
ok, detail = eval_json(ctl, args[0], args[1], args[2])
if ok or time.monotonic() >= deadline:
break
time.sleep(0.5)
waited = time.monotonic() - start
record(lineno, line, "ok" if ok else "fail",
f"waited {waited:.1f}s" if ok else detail)
elif cmd == "assert_stat":
# assert_stat FIELD OP VALUE: FIELD off a fresh `stats` reply.
need_ctl(lineno, cmd)
field, op, value = args[0], args[1], args[2]
if field not in ("cpu", "fps", "render"):
record(lineno, line, "fail", f"unknown stat field {field!r} "
f"(known: cpu, fps, render)")
else:
stats = parse_stats(ctl.send("stats"))
if field not in stats:
record(lineno, line, "fail", "no such field")
else:
try:
ok = apply_op(op, stats[field], value)
except (TypeError, ValueError) as e:
record(lineno, line, "fail", str(e))
else:
record(lineno, line, "ok" if ok else "fail",
"" if ok else f"{field} is {stats[field]}")
elif cmd == "capture_region":
# capture_region NAME X Y W H TOLERANCE: (re)writes NAME in the
# refs file from a fresh screendump. Both hashes are stored
# regardless of TOLERANCE so a later spec edit can change
# tolerance class without a recapture.
need_imgtools(lineno, cmd)
name = args[0]
x, y, w, h = int(args[1]), int(args[2]), int(args[3]), int(args[4])
tolerance = args[5]
if tolerance not in ("exact", "loose", "structural"):
record(lineno, line, "fail", f"unknown tolerance {tolerance!r} "
f"(known: exact, loose, structural)")
else:
path = os.path.join(os.path.abspath(outdir), f"capture_{name}.ppm")
rpc(s, f, {"execute": "screendump", "arguments": {"filename": path}})
try:
cropped = imgtools.crop(imgtools.load_ppm(path), x, y, w, h)
except (ValueError, OSError) as e:
record(lineno, line, "fatal", f"cannot capture {name!r}: {e}")
else:
refs[name] = {
"x": x, "y": y, "w": w, "h": h, "tolerance": tolerance,
"phash": f"{imgtools.phash(cropped):016x}",
"structural": imgtools.structural(cropped).hex(),
}
save_refs(refs_path, refs)
record(lineno, line, "ok", f"captured box={x},{y},{w}x{h}")
elif cmd == "assert_region":
# assert_region NAME [TOLERANCE]: fresh screendump, crop to
# NAME's stored box, compare at NAME's stored tolerance. A
# missing NAME is FATAL for this step: an uncaptured reference
# is a spec/authoring gap, not a UI defect the run should merely
# `fail` on. So is a TOLERANCE other than the one NAME was
# captured under: the script and the reference would be two
# claims about the same pixels, and judging by either alone
# would hide that.
need_imgtools(lineno, cmd)
name = args[0]
want_tol = args[1] if len(args) > 1 else None
ref = refs.get(name)
if ref is not None and want_tol and want_tol != ref.get("tolerance"):
record(lineno, line, "fatal",
f"reference {name!r} was captured as {ref.get('tolerance')}, "
f"script expects {want_tol}: recapture or fix the spec")
else:
ref, cropped, err = fresh_region(name, "assert")
if err:
record(lineno, line, "fatal", err)
else:
ok, detail = imgtools.compare(ref, cropped, ref["tolerance"])
record(lineno, line, "ok" if ok else "fail", detail)
elif cmd == "assert_ocr":
# assert_ocr NAME REGEX: same crop as assert_region, OCR'd
# through `tesseract`, REGEX searched (re.search, spaces allowed
# like assert_hit's TEXT) against the extracted text. No
# tesseract on PATH, or no reference for NAME, is FATAL: pixels
# were never actually checked, so this must never look like a
# skipped-but-passing step.
need_imgtools(lineno, cmd)
name, pattern = args[0], " ".join(args[1:])
if shutil.which("tesseract") is None:
record(lineno, line, "fatal", "tesseract not installed")
else:
ref, cropped, err = fresh_region(name, "ocr")
if err:
record(lineno, line, "fatal", err)
else:
png_path = os.path.join(os.path.abspath(outdir), f"ocr_{name}.png")
try:
write_png(cropped, png_path)
text = subprocess.run(
["tesseract", png_path, "stdout"],
capture_output=True, text=True, timeout=20, check=True,
).stdout
except Exception as e:
record(lineno, line, "fatal", f"ocr failed: {e}")
else:
if re.search(pattern, text):
record(lineno, line, "ok")
else:
record(lineno, line, "fail", f"text was {text.strip()!r}")
else:
results.close() # the rows before this one are still evidence
sys.exit(f"FATAL: {script_path}:{lineno}: unknown command '{cmd}'")
crashed = ui_exited(console_path)
@@ -326,6 +714,7 @@ def main():
size = int(sys.argv[sys.argv.index("--size") + 1])
ctl_path = sys.argv[sys.argv.index("--ctl") + 1] if "--ctl" in sys.argv else None
console_path = sys.argv[sys.argv.index("--console") + 1] if "--console" in sys.argv else None
refs_path = sys.argv[sys.argv.index("--refs") + 1] if "--refs" in sys.argv else None
s = socket.socket(socket.AF_UNIX)
s.connect(path)
@@ -338,7 +727,7 @@ def main():
elif cmd == "tap":
do_tap(s, f, int(sys.argv[3]), int(sys.argv[4]))
elif cmd == "drive":
drive(s, f, sys.argv[3], sys.argv[4], size, ctl_path, console_path)
drive(s, f, sys.argv[3], sys.argv[4], size, ctl_path, console_path, refs_path)
elif cmd == "quit":
s.sendall(b'{"execute":"quit"}\n')