qemu: wait for the UI test hook to answer a ping before the first override (flare-edge#2602)
ci / quality (pull_request) Successful in 1m43s
ci / badges (pull_request) Skipped
ci / test (push) Successful in 11s
ci / prune-artifacts (push) Skipped
ci / quality (push) Successful in 26s
ci / qemu-tools (push) Successful in 4m32s
ci / kernel-symbols (push) Successful in 5m22s
ci / bench (push) Successful in 11s
ci / mcdc (push) Successful in 21s
ci / kernel-build (push) Skipped
ci / badges (push) Skipped
ci / patches-apply (push) Successful in 54s
ci / mcdc (pull_request) Successful in 18s
ci / bench (pull_request) Successful in 15s
ci / prune-artifacts (pull_request) Skipped
ci / kernel-build (pull_request) Skipped
ci / patches-apply (pull_request) Successful in 40s
ci / qemu-tools (pull_request) Successful in 2m3s
ci / kernel-symbols (pull_request) Successful in 3m59s
ci / test (pull_request) Successful in 26s
ci / quality (pull_request) Successful in 1m43s
ci / badges (pull_request) Skipped
ci / test (push) Successful in 11s
ci / prune-artifacts (push) Skipped
ci / quality (push) Successful in 26s
ci / qemu-tools (push) Successful in 4m32s
ci / kernel-symbols (push) Successful in 5m22s
ci / bench (push) Successful in 11s
ci / mcdc (push) Successful in 21s
ci / kernel-build (push) Skipped
ci / badges (push) Skipped
ci / patches-apply (push) Successful in 54s
ci / mcdc (pull_request) Successful in 18s
ci / bench (pull_request) Successful in 15s
ci / prune-artifacts (pull_request) Skipped
ci / kernel-build (pull_request) Skipped
ci / patches-apply (pull_request) Successful in 40s
ci / qemu-tools (pull_request) Successful in 2m3s
ci / kernel-symbols (pull_request) Successful in 3m59s
ci / test (pull_request) Successful in 26s
Right after boot the UI is still taking the broker's retained replay under its per-tick bus budget, so a hook request queued behind it missed wthctl's 5 s bound and its own 5 s message expiry: the failsafe-ack flow's first override was lost, or applied with its reply lost. qmp.py now sends a run's first hook request only after the hook has answered a ping. ping changes nothing, so it is retried while the bridge reports no answer, up to HOOK_READY_S (90 s); the override itself goes out once. A hook that never answers stays a fatal step with its own message. Three offline tests in test_qmp_drive.py; removing the wait fails them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KerYPVh72hhNgKi3wS5ijG
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
ff3d1de088
commit
61ecfac844
+5
-1
@@ -82,7 +82,11 @@ With a test-flavor `warden-ui` and `wthctl` (flare-edge `tools/wthctl.c`, the
|
||||
test hook client) in the payload as well, a flow's `override` step pins a
|
||||
state document inside the UI: qmp.py hands the request to the control bridge
|
||||
(`@hookdoc` pieces, then `@hook override`), which sends it to `test/req/ui`
|
||||
as bus user `testhook` and answers the hook's reply.
|
||||
as bus user `testhook` and answers the hook's reply. The run's first hook
|
||||
request waits until the hook answers a `ping` (retried while the bridge
|
||||
reports no answer, up to `HOOK_READY_S`): right after boot the UI is still
|
||||
taking the broker's retained replay, and an override sent then was lost
|
||||
(flare-edge #2602).
|
||||
|
||||
The VT cursor is kept off (`vt.global_cursor_default=0`): fbcon shares the
|
||||
virtio-gpu framebuffer with warden-ui and its blinking cursor would otherwise
|
||||
|
||||
@@ -969,6 +969,8 @@ class Ctx:
|
||||
self.input_wait_s = 0.0
|
||||
# Stall probes run so far (STALL_REPORTS_MAX bounds them).
|
||||
self.stall_reports = 0
|
||||
# The UI's test hook answered a ping (hook_wait_ready).
|
||||
self.hook_ready = False
|
||||
|
||||
def need_ctl(self, lineno, cmd):
|
||||
if self.ctl is None:
|
||||
@@ -1200,6 +1202,49 @@ def hook_request(ctl, words, args):
|
||||
return ctl.send(f"@hook {words}")
|
||||
|
||||
|
||||
# The bridge's line for a hook request whose client got no reply (_wcb_hook
|
||||
# in rootfs/etc/warden-lib.sh).
|
||||
HOOK_NO_ANSWER = "bridge: hook request got no answer"
|
||||
# How long the run's first hook request waits for the UI's hook to answer a
|
||||
# ping, and the pause between two pings. Each ping is bounded by the hook
|
||||
# client itself (wthctl: 5 s for the UI).
|
||||
HOOK_READY_S = 90.0
|
||||
HOOK_READY_POLL_S = 1.0
|
||||
|
||||
|
||||
def hook_wait_ready(ctx):
|
||||
"""Wait, once per run, until the UI's test hook answers a `ping` -> None,
|
||||
or the fatal detail when it never did.
|
||||
|
||||
The hook takes requests from the device bus, which the UI drains under a
|
||||
per-tick budget (libwardenbus WBUS_TICK_PACKETS / WBUS_TICK_BUDGET_US).
|
||||
Right after boot it is still taking the broker's retained replay, and a
|
||||
request queued behind it misses both the client's 5 s bound and its own
|
||||
5 s message expiry: an override sent then was lost, or applied with its
|
||||
reply lost (flare-edge #2602). ping changes nothing, so it is the one
|
||||
request that is safe to repeat; the step's own request is sent once,
|
||||
after a ping came back. Replies come back in request order, so a late
|
||||
answer to an earlier ping is gone before that request goes out.
|
||||
|
||||
Anything other than "no answer" ends the wait: a missing client or a
|
||||
reply that is not JSON is reported by the step's own request."""
|
||||
if ctx.hook_ready:
|
||||
return None
|
||||
deadline = time.monotonic() + HOOK_READY_S
|
||||
pings = 0
|
||||
while True:
|
||||
pings += 1
|
||||
reply = hook_request(ctx.ctl, "ping", {})
|
||||
if not reply.startswith(HOOK_NO_ANSWER):
|
||||
break
|
||||
if time.monotonic() >= deadline:
|
||||
return (f"the UI test hook answered no ping in {HOOK_READY_S:.0f} s "
|
||||
f"({pings} sent): {reply}")
|
||||
time.sleep(HOOK_READY_POLL_S)
|
||||
ctx.hook_ready = judge_hook_reply(reply)[0] == "ok"
|
||||
return None
|
||||
|
||||
|
||||
def judge_hook_reply(reply):
|
||||
"""(status, detail) for a hook reply: ok:true passes, ok:false fails with
|
||||
the hook's err, and anything else (a bridge line, no JSON) is fatal: the
|
||||
@@ -1230,6 +1275,9 @@ def verb_override(ctx, lineno, cmd, args, line):
|
||||
return "fatal", f"override document is not JSON: {doc_text!r}"
|
||||
if doc is not None and not isinstance(doc, (dict, list)):
|
||||
return "fatal", "override document must be an object, an array or null"
|
||||
not_ready = hook_wait_ready(ctx)
|
||||
if not_ready:
|
||||
return "fatal", not_ready
|
||||
return judge_hook_reply(hook_request(ctx.ctl, "override", {"topic": topic, "doc": doc}))
|
||||
|
||||
|
||||
|
||||
@@ -892,6 +892,11 @@ class OverrideVerb(unittest.TestCase):
|
||||
HookCtl.sent = []
|
||||
HookCtl.answer = staticmethod(lambda words, args: '{"v":1,"ok":true,"data":{}}')
|
||||
HookCtl.refuse_pieces = False
|
||||
self.ready = (qmp.HOOK_READY_S, qmp.HOOK_READY_POLL_S)
|
||||
qmp.HOOK_READY_S, qmp.HOOK_READY_POLL_S = 0.0, 0.0
|
||||
|
||||
def tearDown(self):
|
||||
qmp.HOOK_READY_S, qmp.HOOK_READY_POLL_S = self.ready
|
||||
|
||||
def test_pin_and_unpin_reach_the_hook_as_one_request_each(self):
|
||||
seen = []
|
||||
@@ -906,6 +911,7 @@ class OverrideVerb(unittest.TestCase):
|
||||
self.assertIsNone(rc, rows)
|
||||
self.assertEqual([r["status"] for r in rows], ["ok", "ok"])
|
||||
self.assertEqual(seen, [
|
||||
("ping", {}),
|
||||
("override", {"topic": "io/state/failsafe",
|
||||
"doc": {"last_trip": {"why": "two words"}, "ack": False}}),
|
||||
("override", {"topic": "io/state/failsafe", "doc": None}),
|
||||
@@ -915,7 +921,8 @@ class OverrideVerb(unittest.TestCase):
|
||||
seen = []
|
||||
|
||||
def answer(words, args):
|
||||
seen.append(args)
|
||||
if words != "ping":
|
||||
seen.append(args)
|
||||
return '{"v":1,"ok":true}'
|
||||
HookCtl.answer = staticmethod(answer)
|
||||
doc = {"rows": ["x" * 100] * 120}
|
||||
@@ -924,7 +931,9 @@ class OverrideVerb(unittest.TestCase):
|
||||
ctl_cls=HookCtl)
|
||||
self.assertEqual(rows[0]["status"], "ok", rows)
|
||||
self.assertEqual(seen, [{"topic": "cfg/state/big", "doc": doc}])
|
||||
pieces = [c for c in HookCtl.sent if c.startswith("@hookdoc")]
|
||||
# The pieces after the readiness ping (which sends {} in one piece).
|
||||
after_ping = HookCtl.sent[HookCtl.sent.index("@hook ping") + 1:]
|
||||
pieces = [c for c in after_ping if c.startswith("@hookdoc")]
|
||||
self.assertGreater(len(pieces), 5)
|
||||
self.assertTrue(pieces[0].startswith("@hookdoc {"))
|
||||
self.assertTrue(all(c.startswith("@hookdoc+ ") for c in pieces[1:]))
|
||||
@@ -947,6 +956,53 @@ class OverrideVerb(unittest.TestCase):
|
||||
rc, by, rows = run_script('override io/state/x {"a":1}\n', ctl_cls=HookCtl)
|
||||
self.assertEqual(rows[0]["status"], "fatal", reply)
|
||||
|
||||
def test_the_first_override_waits_for_the_hook_to_answer_a_ping(self):
|
||||
# flare-edge #2602: right after boot the UI's hook answered nothing
|
||||
# for a while; the override must be sent once, after a ping came back.
|
||||
qmp.HOOK_READY_S = 60.0
|
||||
seen = []
|
||||
lost = [2]
|
||||
|
||||
def answer(words, args):
|
||||
seen.append(words)
|
||||
if words == "ping" and lost[0]:
|
||||
lost[0] -= 1
|
||||
return "bridge: hook request got no answer (client exit 2): wthctl: no reply (-5)"
|
||||
return '{"v":1,"ok":true,"data":{}}'
|
||||
HookCtl.answer = staticmethod(answer)
|
||||
rc, by, rows = run_script(
|
||||
'override io/state/failsafe {"a":1}\n'
|
||||
"override io/state/failsafe null\n", ctl_cls=HookCtl)
|
||||
self.assertIsNone(rc, rows)
|
||||
self.assertEqual([r["status"] for r in rows], ["ok", "ok"])
|
||||
self.assertEqual(seen, ["ping", "ping", "ping", "override", "override"])
|
||||
|
||||
def test_a_hook_that_never_answers_a_ping_is_fatal_and_gets_no_override(self):
|
||||
HookCtl.answer = staticmethod(lambda words, args: (
|
||||
"bridge: hook request got no answer (client exit 2): wthctl: no reply (-5)"))
|
||||
rc, by, rows = run_script('override io/state/x {"a":1}\nwake\n', ctl_cls=HookCtl)
|
||||
self.assertEqual([r["status"] for r in rows], ["fatal", "ok"])
|
||||
self.assertIn("answered no ping", rows[0]["detail"])
|
||||
self.assertIn("@hook ping", HookCtl.sent)
|
||||
self.assertNotIn("@hook override", HookCtl.sent)
|
||||
|
||||
def test_a_ping_that_is_not_ok_leaves_the_verdict_to_the_override(self):
|
||||
# A hook that answers but refuses ping (an older hook) is still a
|
||||
# hook: the override goes out and its own reply is judged, and the
|
||||
# next override pings again rather than trusting it.
|
||||
seen = []
|
||||
|
||||
def answer(words, args):
|
||||
seen.append(words)
|
||||
if words == "ping":
|
||||
return '{"v":1,"ok":false,"err":{"code":"E_UNKNOWN_VERB"}}'
|
||||
return '{"v":1,"ok":true}'
|
||||
HookCtl.answer = staticmethod(answer)
|
||||
rc, by, rows = run_script(
|
||||
'override io/state/x {"a":1}\noverride io/state/x null\n', ctl_cls=HookCtl)
|
||||
self.assertEqual([r["status"] for r in rows], ["ok", "ok"])
|
||||
self.assertEqual(seen, ["ping", "override", "ping", "override"])
|
||||
|
||||
def test_a_refused_piece_is_fatal_for_the_step_only(self):
|
||||
HookCtl.refuse_pieces = True
|
||||
rc, by, rows = run_script('override io/state/x {"a":1}\nwake\n', ctl_cls=HookCtl)
|
||||
|
||||
Reference in New Issue
Block a user