qemu: json/stat/region channels, seeding, driver tests
- rootfs/sbin/init: the control bridge answers `@cat PATH` locally so the driver can read /tmp/warden-web-status.json out of the guest. That file has no trailing newline; the bridge adds one so the sentinel stays on its own line and the line-based reader never blocks. - tests/qmp.py: wait_json/assert_json (dotted paths, eq/ne/contains/ len_eq/len_ge/gt/lt), assert_stat off the FIFO's stats reply, capture_region, assert_region NAME [TOLERANCE] and assert_ocr. A tolerance other than the captured one, a reference box that does not fit the screendump, a missing reference or a missing tesseract is FATAL for that step and the run continues (flare-edge #147). - tests/imgtools.py: P6 reader, crop, perceptual and structural hashes, compare, with a self-test. - tests/test_qmp_drive.py: drive() with QMP and the control channel faked, pinning the per-step ok/fail/fatal contract. - mkimage.sh SEED_DIR and ui-drive.sh --seed/--refs: settings fixtures staged into userdata before warden-ui starts, and a reference store handed to the driver. - ci: the driver tests and the imgtools self-test run in qemu-tools. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013aHKWzT5EF86RFKRMtAv9n
This commit is contained in:
+392
-3
@@ -35,6 +35,52 @@ a real panel, so these verbs mean the same thing on the rig and on hardware:
|
||||
reports a MOVED button as such, not as broken
|
||||
behaviour
|
||||
|
||||
Two more channels reach behavioural state instead of the widget tree.
|
||||
`--refs FILE` (default <outdir>/refs.json) is the reference store the pixel
|
||||
ones read and write; imgtools.py in this directory does the actual image
|
||||
math, this file only drives it.
|
||||
|
||||
assert_json PATH OP VALUE
|
||||
wait_json PATH OP VALUE TIMEOUT_S
|
||||
PATH is dotted with optional [N] indexes (e.g.
|
||||
network.scan.ranges[0]) into the guest's
|
||||
/tmp/warden-web-status.json, fetched fresh on
|
||||
every check with `@cat` over the SAME control
|
||||
channel as nav/hit (see the bridge added to
|
||||
rootfs/sbin/init). OP is one of eq, ne, contains,
|
||||
len_eq, len_ge, gt, lt. `wait_json` polls every
|
||||
0.5s until OP holds or TIMEOUT_S elapses and
|
||||
`assert_json` reads once. A PATH the document
|
||||
doesn't have is a `fail`, not a crash -- that
|
||||
document is a live snapshot and the key might
|
||||
simply not be built yet.
|
||||
assert_stat FIELD OP VALUE
|
||||
FIELD is cpu, fps or render, read off a fresh
|
||||
`stats` reply (warden_debug.c). Same OP vocabulary
|
||||
as assert_json.
|
||||
capture_region NAME X Y W H TOLERANCE
|
||||
screendump now, crop to X,Y,WxH, and (over)write
|
||||
NAME in the refs file with both a phash and a
|
||||
structural hash plus TOLERANCE (exact, loose or
|
||||
structural). Run this once, by hand, to author a
|
||||
reference -- the same way targets are authored,
|
||||
see flows/README.md.
|
||||
assert_region NAME [TOLERANCE]
|
||||
fresh screendump, crop to NAME's stored box,
|
||||
compare against its stored hash at its stored
|
||||
tolerance. TOLERANCE, when given, must be the one
|
||||
NAME was captured under (flowc.py always emits it).
|
||||
assert_ocr NAME REGEX same crop, run through `tesseract`, REGEX searched
|
||||
against the extracted text (spaces in REGEX are
|
||||
fine, same as assert_hit's TEXT).
|
||||
|
||||
assert_region/assert_ocr against a NAME with no captured reference, and
|
||||
assert_ocr with no `tesseract` on PATH, are FATAL for that one step: a
|
||||
silently-skipped check reads as coverage that was never actually there. So
|
||||
are a TOLERANCE that differs from the captured one and a stored box that does
|
||||
not fit the screendump: both are the reference's fault, not the UI's, and one
|
||||
bad reference must not take the rest of the script with it.
|
||||
|
||||
Every step is recorded to <outdir>/results.jsonl as ok / fail / fatal. An
|
||||
assertion mismatch is a `fail` and the script continues, so one run reports
|
||||
every broken expectation. With `--console LOG` the console is checked after
|
||||
@@ -46,13 +92,38 @@ A step that names an unknown command is a FATAL error rather than a skip: a
|
||||
silently-ignored line in a scenario is a test that proves nothing.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
# imgtools.py lives beside this file and does the actual pixel math (phash,
|
||||
# structural hash, crop, compare) for the region/ocr verbs. Imported at
|
||||
# module load but never let a missing/broken imgtools take down the verbs
|
||||
# that don't need it: nav/tap/page/... must keep working while it is being
|
||||
# written, so the failure is deferred to need_imgtools() at first use.
|
||||
try:
|
||||
import imgtools
|
||||
except ImportError:
|
||||
imgtools = None
|
||||
|
||||
# Pillow, likewise, is only needed to hand tesseract an image file (it has no
|
||||
# raw-RGB stdin mode); a host without it still runs every other verb.
|
||||
try:
|
||||
from PIL import Image as _PILImage
|
||||
except ImportError:
|
||||
_PILImage = None
|
||||
|
||||
AXIS_MAX = 32767
|
||||
|
||||
# Where webstatus.c (ui-src/src/warden/webstatus.c) publishes its atomic
|
||||
# snapshot inside the guest. assert_json/wait_json `@cat` this path over the
|
||||
# control channel; see the bridge in rootfs/sbin/init.
|
||||
STATUS_JSON_PATH = "/tmp/warden-web-status.json"
|
||||
|
||||
|
||||
def rpc(sock, sock_file, obj):
|
||||
sock.sendall((json.dumps(obj) + "\n").encode())
|
||||
@@ -198,8 +269,168 @@ def ui_exited(console_path):
|
||||
return data[i:i + 80].split(b"\n", 1)[0].decode("utf-8", "replace")
|
||||
|
||||
|
||||
def drive(s, f, script_path, outdir, size, ctl_path=None, console_path=None):
|
||||
import os
|
||||
PATH_TOKEN_RE = re.compile(r'^([^\[\]]*)((?:\[\d+\])*)$')
|
||||
IDX_RE = re.compile(r'\[(\d+)\]')
|
||||
|
||||
|
||||
def resolve_path(doc, path):
|
||||
"""Walk a dotted PATH (keys and [N] indexes, e.g. 'a.b[0].c') over an
|
||||
already-parsed JSON document. -> (value, None) or (None, 'no such path').
|
||||
|
||||
Pure and I/O-free on purpose (contract: testable on its own) -- fetching
|
||||
the document is a separate step (fetch_status_json) so this function can
|
||||
be unit-tested against a plain dict/list literal with nothing running.
|
||||
"""
|
||||
cur = doc
|
||||
for part in path.split("."):
|
||||
m = PATH_TOKEN_RE.match(part)
|
||||
if not m:
|
||||
return None, "no such path"
|
||||
key, idxs = m.group(1), m.group(2)
|
||||
if key:
|
||||
if not isinstance(cur, dict) or key not in cur:
|
||||
return None, "no such path"
|
||||
cur = cur[key]
|
||||
for idx_s in IDX_RE.findall(idxs):
|
||||
idx = int(idx_s)
|
||||
if not isinstance(cur, list) or idx >= len(cur) or idx < 0:
|
||||
return None, "no such path"
|
||||
cur = cur[idx]
|
||||
return cur, None
|
||||
|
||||
|
||||
def resolve_token(raw):
|
||||
"""A verb's literal OP argument -> a comparable Python value. A JSON
|
||||
literal (a number, true/false/null, or a quoted string) parses as
|
||||
itself; anything else -- most values in practice, e.g. `connected` --
|
||||
compares as the raw string. This is what lets both
|
||||
`assert_json state.net eq connected` and `assert_json state.count eq 3`
|
||||
work from one unquoted token, with no escaping convention of its own."""
|
||||
try:
|
||||
return json.loads(raw)
|
||||
except ValueError:
|
||||
return raw
|
||||
|
||||
|
||||
def _len_of(v):
|
||||
try:
|
||||
return len(v)
|
||||
except TypeError:
|
||||
raise TypeError(f"{v!r} has no length") from None
|
||||
|
||||
|
||||
def apply_op(op, actual, raw_value):
|
||||
"""The comparison vocabulary shared by assert_json/wait_json/assert_stat:
|
||||
eq, ne, contains, len_eq, len_ge, gt, lt. Raises on a combination that
|
||||
cannot be judged (contains on a number, gt across incompatible types)
|
||||
so the caller turns that into a `fail` detail instead of a driver crash --
|
||||
a spec that asks a nonsensical question should be reported, not silently
|
||||
True or False."""
|
||||
value = resolve_token(raw_value)
|
||||
if op == "eq":
|
||||
return actual == value
|
||||
if op == "ne":
|
||||
return actual != value
|
||||
if op == "contains":
|
||||
if isinstance(actual, str):
|
||||
return str(value) in actual
|
||||
if isinstance(actual, (list, tuple, dict)):
|
||||
return value in actual
|
||||
raise TypeError(f"contains: {actual!r} is not a string or list")
|
||||
if op == "len_eq":
|
||||
return _len_of(actual) == int(value)
|
||||
if op == "len_ge":
|
||||
return _len_of(actual) >= int(value)
|
||||
if op == "gt":
|
||||
return actual > value
|
||||
if op == "lt":
|
||||
return actual < value
|
||||
raise ValueError(f"unknown op {op!r}")
|
||||
|
||||
|
||||
def fetch_status_json(ctl):
|
||||
"""`@cat` the guest's webstatus.c snapshot over the control channel and
|
||||
parse it. -> (doc, None) or (None, detail); never raises, so a torn read
|
||||
or a not-yet-written file is an ordinary retry/fail for the caller, not a
|
||||
crash of the whole driver."""
|
||||
reply = ctl.send(f"@cat {STATUS_JSON_PATH}")
|
||||
if reply.startswith("bridge: no such file"):
|
||||
return None, reply
|
||||
try:
|
||||
return json.loads(reply), None
|
||||
except ValueError as e:
|
||||
return None, f"status json unparsable: {e}"
|
||||
|
||||
|
||||
def eval_json(ctl, path, op, value):
|
||||
"""One fetch + path-resolve + op-apply round for assert_json/wait_json.
|
||||
-> (ok, detail); detail is empty on success, otherwise the reason."""
|
||||
doc, err = fetch_status_json(ctl)
|
||||
if err is not None:
|
||||
return False, err
|
||||
actual, perr = resolve_path(doc, path)
|
||||
if perr is not None:
|
||||
return False, perr
|
||||
try:
|
||||
ok = apply_op(op, actual, value)
|
||||
except (TypeError, ValueError) as e:
|
||||
return False, f"{op} {value!r} vs {actual!r}: {e}"
|
||||
return ok, "" if ok else f"{path} is {actual!r}"
|
||||
|
||||
|
||||
STATS_FIELD_RE = {
|
||||
"cpu": re.compile(r'^cpu:\s*(-?\d+(?:\.\d+)?)%?\s*$'),
|
||||
"fps": re.compile(r'^fps:\s*(-?\d+(?:\.\d+)?)\s*$'),
|
||||
"render": re.compile(r'^render:\s*(-?\d+(?:\.\d+)?)\s*ms/frame\s*$'),
|
||||
}
|
||||
|
||||
|
||||
def parse_stats(reply):
|
||||
"""The `stats` reply (warden_debug.c: 'page: X\\ncpu: N%\\nfps: N\\n
|
||||
render: A.BB ms/frame\\nrga: N%') -> {'cpu'|'fps'|'render': float} for
|
||||
whichever lines are present. A field the reply lacks is simply absent
|
||||
from the result -- the caller reports that as 'no such field', the same
|
||||
shape as resolve_path's 'no such path' for the json channel."""
|
||||
out = {}
|
||||
for ln in reply.splitlines():
|
||||
ln = ln.strip()
|
||||
for field, rx in STATS_FIELD_RE.items():
|
||||
m = rx.match(ln)
|
||||
if m:
|
||||
out[field] = float(m.group(1))
|
||||
return out
|
||||
|
||||
|
||||
def load_refs(path):
|
||||
"""The region/ocr reference store (see qmp.py's own docstring for the
|
||||
schema). Missing file -> {}, same as an empty COVERAGE.yaml: capture_region
|
||||
is how one gets created, not a prerequisite to have one already."""
|
||||
if not path or not os.path.exists(path):
|
||||
return {}
|
||||
with open(path) as fh:
|
||||
return json.load(fh)
|
||||
|
||||
|
||||
def save_refs(path, refs):
|
||||
# Written after every capture_region, not batched at the end: a script
|
||||
# that dies three steps later must not lose a reference that was already
|
||||
# good.
|
||||
with open(path, "w") as fh:
|
||||
json.dump(refs, fh, indent=2, sort_keys=True)
|
||||
fh.write("\n")
|
||||
|
||||
|
||||
def write_png(img, path):
|
||||
"""imgtools' (w, h, rgb_bytes) tuple -> a PNG file, because tesseract has
|
||||
no raw-RGB input mode. Only assert_ocr needs this; assert_region and
|
||||
capture_region work on imgtools' own tuples end to end."""
|
||||
if _PILImage is None:
|
||||
raise RuntimeError("Pillow (PIL) is not installed")
|
||||
w, h, data = img
|
||||
_PILImage.frombytes("RGB", (w, h), data).save(path)
|
||||
|
||||
|
||||
def drive(s, f, script_path, outdir, size, ctl_path=None, console_path=None, refs_path=None):
|
||||
os.makedirs(outdir, exist_ok=True)
|
||||
with open(script_path) as fh:
|
||||
lines = fh.readlines()
|
||||
@@ -221,6 +452,36 @@ def drive(s, f, script_path, outdir, size, ctl_path=None, console_path=None):
|
||||
sys.exit(f"FATAL: {script_path}:{lineno}: '{cmd}' needs the control "
|
||||
f"channel; run with --ctl (ui-drive.sh passes it)")
|
||||
|
||||
def need_imgtools(lineno, cmd):
|
||||
# A whole-script infrastructure gap (imgtools.py absent), same class
|
||||
# as a missing --ctl: no region/ocr verb can do anything without it,
|
||||
# so this exits the process rather than recording a per-step fail.
|
||||
if imgtools is None:
|
||||
sys.exit(f"FATAL: {script_path}:{lineno}: '{cmd}' needs imgtools.py "
|
||||
f"next to qmp.py (vendor/warden-sdk/qemu/tests/imgtools.py)")
|
||||
|
||||
refs_path = refs_path or os.path.join(outdir, "refs.json")
|
||||
refs = load_refs(refs_path)
|
||||
|
||||
def fresh_region(name, tag):
|
||||
"""screendump NOW (never reuse an earlier `shot`) and crop to NAME's
|
||||
stored box. Fails closed on an uncaptured NAME: assert_region and
|
||||
assert_ocr both need this, capture_region does not."""
|
||||
ref = refs.get(name)
|
||||
if ref is None:
|
||||
return None, None, f"no reference for {name!r} (run capture_region first)"
|
||||
path = os.path.join(os.path.abspath(outdir), f"{tag}_{name}.ppm")
|
||||
rpc(s, f, {"execute": "screendump", "arguments": {"filename": path}})
|
||||
try:
|
||||
img = imgtools.load_ppm(path)
|
||||
cropped = imgtools.crop(img, ref["x"], ref["y"], ref["w"], ref["h"])
|
||||
except (ValueError, OSError, KeyError, TypeError) as e:
|
||||
# A box outside this screendump (captured at another --size, or
|
||||
# mistyped in the store) is a per-step fatal, never a traceback
|
||||
# that ends the run: every later step still gets judged.
|
||||
return None, None, f"reference {name!r} unusable: {e}"
|
||||
return ref, cropped, None
|
||||
|
||||
for lineno, raw in enumerate(lines, 1):
|
||||
line = raw.split("#", 1)[0].strip()
|
||||
if not line:
|
||||
@@ -296,7 +557,134 @@ def drive(s, f, script_path, outdir, size, ctl_path=None, console_path=None):
|
||||
record(lineno, line, "fail", f"{cls} text={got[1]!r}")
|
||||
else:
|
||||
record(lineno, line, "ok")
|
||||
elif cmd == "assert_json":
|
||||
# assert_json PATH OP VALUE: one read of the guest's status JSON
|
||||
# (see eval_json). PATH not (yet) in the doc is a `fail`, since
|
||||
# the doc is a live snapshot, not a schema.
|
||||
need_ctl(lineno, cmd)
|
||||
ok, detail = eval_json(ctl, args[0], args[1], args[2])
|
||||
record(lineno, line, "ok" if ok else "fail", detail)
|
||||
elif cmd == "wait_json":
|
||||
# wait_json PATH OP VALUE TIMEOUT_S: poll eval_json every 0.5s
|
||||
# until it holds or the deadline passes. The status file is
|
||||
# written by a 2s guest timer (webstatus.c), so this exists
|
||||
# because a value the doc will reach shortly is not the same
|
||||
# fact as a value it never reaches -- assert_json alone cannot
|
||||
# tell those apart.
|
||||
need_ctl(lineno, cmd)
|
||||
timeout_s = float(args[3])
|
||||
start = time.monotonic()
|
||||
deadline = start + timeout_s
|
||||
while True:
|
||||
ok, detail = eval_json(ctl, args[0], args[1], args[2])
|
||||
if ok or time.monotonic() >= deadline:
|
||||
break
|
||||
time.sleep(0.5)
|
||||
waited = time.monotonic() - start
|
||||
record(lineno, line, "ok" if ok else "fail",
|
||||
f"waited {waited:.1f}s" if ok else detail)
|
||||
elif cmd == "assert_stat":
|
||||
# assert_stat FIELD OP VALUE: FIELD off a fresh `stats` reply.
|
||||
need_ctl(lineno, cmd)
|
||||
field, op, value = args[0], args[1], args[2]
|
||||
if field not in ("cpu", "fps", "render"):
|
||||
record(lineno, line, "fail", f"unknown stat field {field!r} "
|
||||
f"(known: cpu, fps, render)")
|
||||
else:
|
||||
stats = parse_stats(ctl.send("stats"))
|
||||
if field not in stats:
|
||||
record(lineno, line, "fail", "no such field")
|
||||
else:
|
||||
try:
|
||||
ok = apply_op(op, stats[field], value)
|
||||
except (TypeError, ValueError) as e:
|
||||
record(lineno, line, "fail", str(e))
|
||||
else:
|
||||
record(lineno, line, "ok" if ok else "fail",
|
||||
"" if ok else f"{field} is {stats[field]}")
|
||||
elif cmd == "capture_region":
|
||||
# capture_region NAME X Y W H TOLERANCE: (re)writes NAME in the
|
||||
# refs file from a fresh screendump. Both hashes are stored
|
||||
# regardless of TOLERANCE so a later spec edit can change
|
||||
# tolerance class without a recapture.
|
||||
need_imgtools(lineno, cmd)
|
||||
name = args[0]
|
||||
x, y, w, h = int(args[1]), int(args[2]), int(args[3]), int(args[4])
|
||||
tolerance = args[5]
|
||||
if tolerance not in ("exact", "loose", "structural"):
|
||||
record(lineno, line, "fail", f"unknown tolerance {tolerance!r} "
|
||||
f"(known: exact, loose, structural)")
|
||||
else:
|
||||
path = os.path.join(os.path.abspath(outdir), f"capture_{name}.ppm")
|
||||
rpc(s, f, {"execute": "screendump", "arguments": {"filename": path}})
|
||||
try:
|
||||
cropped = imgtools.crop(imgtools.load_ppm(path), x, y, w, h)
|
||||
except (ValueError, OSError) as e:
|
||||
record(lineno, line, "fatal", f"cannot capture {name!r}: {e}")
|
||||
else:
|
||||
refs[name] = {
|
||||
"x": x, "y": y, "w": w, "h": h, "tolerance": tolerance,
|
||||
"phash": f"{imgtools.phash(cropped):016x}",
|
||||
"structural": imgtools.structural(cropped).hex(),
|
||||
}
|
||||
save_refs(refs_path, refs)
|
||||
record(lineno, line, "ok", f"captured box={x},{y},{w}x{h}")
|
||||
elif cmd == "assert_region":
|
||||
# assert_region NAME [TOLERANCE]: fresh screendump, crop to
|
||||
# NAME's stored box, compare at NAME's stored tolerance. A
|
||||
# missing NAME is FATAL for this step: an uncaptured reference
|
||||
# is a spec/authoring gap, not a UI defect the run should merely
|
||||
# `fail` on. So is a TOLERANCE other than the one NAME was
|
||||
# captured under: the script and the reference would be two
|
||||
# claims about the same pixels, and judging by either alone
|
||||
# would hide that.
|
||||
need_imgtools(lineno, cmd)
|
||||
name = args[0]
|
||||
want_tol = args[1] if len(args) > 1 else None
|
||||
ref = refs.get(name)
|
||||
if ref is not None and want_tol and want_tol != ref.get("tolerance"):
|
||||
record(lineno, line, "fatal",
|
||||
f"reference {name!r} was captured as {ref.get('tolerance')}, "
|
||||
f"script expects {want_tol}: recapture or fix the spec")
|
||||
else:
|
||||
ref, cropped, err = fresh_region(name, "assert")
|
||||
if err:
|
||||
record(lineno, line, "fatal", err)
|
||||
else:
|
||||
ok, detail = imgtools.compare(ref, cropped, ref["tolerance"])
|
||||
record(lineno, line, "ok" if ok else "fail", detail)
|
||||
elif cmd == "assert_ocr":
|
||||
# assert_ocr NAME REGEX: same crop as assert_region, OCR'd
|
||||
# through `tesseract`, REGEX searched (re.search, spaces allowed
|
||||
# like assert_hit's TEXT) against the extracted text. No
|
||||
# tesseract on PATH, or no reference for NAME, is FATAL: pixels
|
||||
# were never actually checked, so this must never look like a
|
||||
# skipped-but-passing step.
|
||||
need_imgtools(lineno, cmd)
|
||||
name, pattern = args[0], " ".join(args[1:])
|
||||
if shutil.which("tesseract") is None:
|
||||
record(lineno, line, "fatal", "tesseract not installed")
|
||||
else:
|
||||
ref, cropped, err = fresh_region(name, "ocr")
|
||||
if err:
|
||||
record(lineno, line, "fatal", err)
|
||||
else:
|
||||
png_path = os.path.join(os.path.abspath(outdir), f"ocr_{name}.png")
|
||||
try:
|
||||
write_png(cropped, png_path)
|
||||
text = subprocess.run(
|
||||
["tesseract", png_path, "stdout"],
|
||||
capture_output=True, text=True, timeout=20, check=True,
|
||||
).stdout
|
||||
except Exception as e:
|
||||
record(lineno, line, "fatal", f"ocr failed: {e}")
|
||||
else:
|
||||
if re.search(pattern, text):
|
||||
record(lineno, line, "ok")
|
||||
else:
|
||||
record(lineno, line, "fail", f"text was {text.strip()!r}")
|
||||
else:
|
||||
results.close() # the rows before this one are still evidence
|
||||
sys.exit(f"FATAL: {script_path}:{lineno}: unknown command '{cmd}'")
|
||||
|
||||
crashed = ui_exited(console_path)
|
||||
@@ -326,6 +714,7 @@ def main():
|
||||
size = int(sys.argv[sys.argv.index("--size") + 1])
|
||||
ctl_path = sys.argv[sys.argv.index("--ctl") + 1] if "--ctl" in sys.argv else None
|
||||
console_path = sys.argv[sys.argv.index("--console") + 1] if "--console" in sys.argv else None
|
||||
refs_path = sys.argv[sys.argv.index("--refs") + 1] if "--refs" in sys.argv else None
|
||||
|
||||
s = socket.socket(socket.AF_UNIX)
|
||||
s.connect(path)
|
||||
@@ -338,7 +727,7 @@ def main():
|
||||
elif cmd == "tap":
|
||||
do_tap(s, f, int(sys.argv[3]), int(sys.argv[4]))
|
||||
elif cmd == "drive":
|
||||
drive(s, f, sys.argv[3], sys.argv[4], size, ctl_path, console_path)
|
||||
drive(s, f, sys.argv[3], sys.argv[4], size, ctl_path, console_path, refs_path)
|
||||
elif cmd == "quit":
|
||||
s.sendall(b'{"execute":"quit"}\n')
|
||||
|
||||
|
||||
Reference in New Issue
Block a user