qemu: rig agent, a framed guest shell on its own serial port (flare-edge M9 soak-lite)
ci / test (push) Successful in 10s
ci / patches-apply (push) Successful in 25s
ci / quality (push) Successful in 36s
ci / badges (push) Skipped
ci / kernel-symbols (push) Successful in 4m38s
ci / mcdc (push) Successful in 37s
ci / bench (push) Successful in 6s
ci / qemu-tools (push) Failing after 15s
ci / prune-artifacts (push) Skipped
ci / kernel-build (push) Skipped
ci / test (pull_request) Successful in 9s
ci / prune-artifacts (pull_request) Skipped
ci / mcdc (pull_request) Successful in 34s
ci / bench (pull_request) Successful in 20s
ci / kernel-build (pull_request) Skipped
ci / qemu-tools (pull_request) Failing after 57s
ci / patches-apply (pull_request) Successful in 1m16s
ci / kernel-symbols (pull_request) Successful in 2m14s
ci / badges (pull_request) Skipped
ci / quality (pull_request) Successful in 2m51s

run.sh --agent SOCK adds a third pci-serial after ctl and rs485 (their
ttyS order is unchanged) and warden.agent on the command line. Stage-2
init then runs warden_agent (rootfs/etc/warden-lib.sh) on that port with
echo off: `@run ID SECS B64` runs a script and answers <<BEGIN ID>>, the
base64 of its output and <<END ID RC>>; `@chunk` pieces get past the
4095-byte cooked line; a script past its deadline is killed with its
process group and answers 124 (the panel BusyBox has no timeout). It is
the guest channel of flare-edge tools/soak/rig-qemu.py, which needs bus
reads and writes, the clock and /proc inside the gate image. Rig rootfs
only: no panel image carries it.

Tests: qemu/tests/agent-test.sh (sh and busybox sh, in CI) and new
run-sh-args-test.sh cases (agent last, warden.agent only with the flag).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdKQBQkJkq4zE7iPFMkAhS
This commit is contained in:
BFE Engineering
2026-10-31 17:05:50 -06:00
co-authored by Claude Opus 5.5
parent 0232823855
commit 579ac260ca
7 changed files with 302 additions and 1 deletions
+8
View File
@@ -170,6 +170,14 @@ jobs:
if command -v busybox >/dev/null; then
WARDEN_TEST_SH="busybox sh" bash qemu/tests/ctl-bridge-test.sh
fi
- name: rig agent runs framed scripts under a deadline (flare-edge M9 soak-lite)
# warden_agent against request lines on stdin; once more under
# busybox sh, the rig's own shell.
run: |
bash qemu/tests/agent-test.sh
if command -v busybox >/dev/null; then
WARDEN_TEST_SH="busybox sh" bash qemu/tests/agent-test.sh
fi
- name: rig init boots through the shipped init chain (flare-edge#2210)
# warden_init_chain against stub scripts and the warden.initchain
# wiring in sbin/init; once more under busybox sh, the rig's own shell.
+1
View File
@@ -146,6 +146,7 @@ All take the virt-fragment `<zImage>`; `FLARE_EDGE=<checkout>` where noted.
| `console-shell-test.sh` | - | offline, no VM: `warden_console_shell` (warden.shell) runs `setsid cttyhack sh` where cttyhack exists, else `setsid -c sh` on the console tty read from sysfs (the panel BusyBox has no cttyhack), else a plain sh. CI runs it under bash's `sh` and busybox `sh` |
| `ctl-bridge-test.sh` | - | offline, no VM: runs the control bridge (`warden_ctl_bridge` in rootfs/etc/warden-lib.sh) against a fake warden-ui that answers some commands late. A reply slower than the old 5 s bound still arrives (the bound is now 30 s, `WARDEN_CTL_WAIT_S`), a timeout answers `bridge: no reply from warden-ui ...` (qmp.py raises it as a rig fault), and a timed-out command's late reply is drained instead of being read as the next command's answer (flare-edge #1448). A fake `wthctl` stands in for the hook client behind `@hookdoc`/`@hook` (flare-edge #2054): the pieces join into one request whose args are used once, only plain words reach the client, and a missing client or a request with no reply answers a `bridge:` line. CI runs it under bash's `sh` and busybox `sh` |
| `init-rig-bus-test.sh` | - | offline, no VM: `warden_rig_bus` in rootfs/etc/warden-lib.sh against stub brokers. No broker in the payload creates nothing; a broker that listens gets the panel's layout (the `bus.sock` link, a 0400 64-hex credential per bus user, a config without password file, ACL or plugin); a broker that exits or never listens is reported within the bound; init starts the bus before the payload daemons. `WARDEN_TEST_MOSQUITTO_DIR` adds a real mosquitto carrying one message between two users logged in by credential file (flare-edge #1754) |
| `agent-test.sh` | - | offline, no VM: the rig agent (`warden_agent` in rootfs/etc/warden-lib.sh, armed by `run.sh --agent` and `warden.agent`) answers `@run ID SECS B64` with `<<BEGIN ID>>`, the base64 of the script's output and `<<END ID RC>>`; `@chunk` pieces join into one script used once; a script past its deadline is killed with its process group and answers 124; a malformed request answers one `<<ERR ...>>` line and runs nothing; init puts the agent's port after ctl and RS485. It is the guest shell of flare-edge `tools/soak/rig-qemu.py` (M9 soak-lite). CI runs it under bash's `sh` and busybox `sh` |
| watchdog (`run.sh --watchdog`) | - | arm `/dev/watchdog`, don't pet: the VM resets ~30 s later (verified) |
Scenario fine print:
+110
View File
@@ -387,3 +387,113 @@ warden_init_chain() {
done
echo "$_wic_ran"
}
# The rig agent (flare-edge M9 soak-lite, d10 6.5): a shell inside the guest
# for a host-side rig that must read state and change it (tools/soak/rig-qemu.py:
# bus reads and writes, the clock, /proc for metrics). /sbin/init runs it on
# its own pci-serial port, only on a boot with warden.agent on the command
# line (run.sh --agent). It is part of the rig rootfs, which no panel image
# carries; the flow channels (ctl bridge, test hook) are untouched by it.
#
# One request per line on stdin, every answer framed so the host never needs
# a timeout to know a reply is complete:
#
# @ping -> <<PONG>>
# @chunk ID B64 -> <<OK ID BYTES>> appends B64 to ID's script text
# @run ID SECS [B64] -> <<BEGIN ID>>, the base64 of the script's stdout
# and stderr, <<END ID RC>>
# anything else -> <<ERR reason>>
#
# The script is the base64 text of the @chunk lines for ID, then the @run
# line's own B64, decoded and run by sh with stdin from /dev/null. The agent
# port is a tty in cooked mode, which cuts a line at 4095 bytes, hence the
# chunks. ID is [A-Za-z0-9_-] (1 to 32 characters), SECS 1 to 3600. A script
# still running after SECS seconds is killed with its whole process group and
# answers RC 124, as timeout(1) does; the panel BusyBox has no timeout applet.
# WARDEN_AGENT_DIR exists for the host test only.
warden_agent() {
_wa_dir="${WARDEN_AGENT_DIR:-/tmp/warden-agent}"
mkdir -p "$_wa_dir"
while IFS= read -r _wa_line; do
# A cooked tty may hand the line over with a trailing CR.
_wa_line="${_wa_line%"$(printf '\r')"}"
[ -n "$_wa_line" ] || continue
set -f
# shellcheck disable=SC2086 # split into words on purpose, globbing off
set -- $_wa_line
set +f
case "$1" in
@ping) echo "<<PONG>>" ;;
@chunk)
if [ "$#" -ne 3 ] || ! _wa_id_ok "$2" || ! _wa_b64_ok "$3"; then
echo "<<ERR bad chunk>>"
continue
fi
printf '%s' "$3" >> "$_wa_dir/$2.b64"
echo "<<OK $2 $(wc -c < "$_wa_dir/$2.b64" | tr -d ' ')>>"
;;
@run)
if [ "$#" -lt 3 ] || [ "$#" -gt 4 ] || ! _wa_id_ok "$2" ||
! _wa_secs_ok "$3" || { [ "$#" -eq 4 ] && ! _wa_b64_ok "$4"; }; then
echo "<<ERR bad run>>"
continue
fi
[ "$#" -eq 4 ] && printf '%s' "$4" >> "$_wa_dir/$2.b64"
_wa_run "$2" "$3"
;;
*) echo "<<ERR unknown request>>" ;;
esac
done
}
_wa_id_ok() {
case "$1" in ""|*[!A-Za-z0-9_-]*) return 1 ;; esac
[ "${#1}" -le 32 ]
}
_wa_b64_ok() {
case "$1" in ""|*[!A-Za-z0-9+/=]*) return 1 ;; esac
return 0
}
_wa_secs_ok() {
case "$1" in ""|*[!0-9]*) return 1 ;; esac
[ "${#1}" -le 4 ] && [ "$1" -ge 1 ] && [ "$1" -le 3600 ]
}
# _wa_run ID SECS: decode, run under the deadline, answer, forget ID.
_wa_run() {
_wr_b64="$_wa_dir/$1.b64"
_wr_sh="$_wa_dir/$1.sh"
_wr_out="$_wa_dir/$1.out"
if [ ! -s "$_wr_b64" ] || ! base64 -d < "$_wr_b64" > "$_wr_sh" 2>/dev/null ||
[ ! -s "$_wr_sh" ]; then
rm -f "$_wr_b64" "$_wr_sh"
echo "<<ERR bad script $1>>"
return 0
fi
rm -f "$_wr_b64"
# setsid: the script leads its own process group, so a timeout kills
# every process it started, not only its shell.
setsid sh "$_wr_sh" < /dev/null > "$_wr_out" 2>&1 &
_wr_pid=$!
_wr_n=0
_wr_limit=$(($2 * 10))
_wr_late=0
while kill -0 "$_wr_pid" 2>/dev/null; do
if [ "$_wr_n" -ge "$_wr_limit" ]; then
kill -KILL "-$_wr_pid" 2>/dev/null || kill -KILL "$_wr_pid" 2>/dev/null
_wr_late=1
break
fi
sleep 0.1
_wr_n=$((_wr_n + 1))
done
wait "$_wr_pid" 2>/dev/null
_wr_rc=$?
[ "$_wr_late" = 1 ] && _wr_rc=124
echo "<<BEGIN $1>>"
[ -s "$_wr_out" ] && base64 < "$_wr_out"
echo "<<END $1 $_wr_rc>>"
rm -f "$_wr_sh" "$_wr_out"
}
+22
View File
@@ -229,6 +229,28 @@ if grep -qw warden.ctl /proc/cmdline && [ -c /dev/ttyS0 ]; then
) &
fi
# Rig agent (run.sh --agent, warden.agent on the command line): a shell for a
# host-side rig that reads and changes guest state, such as the QEMU soak rig
# (flare-edge tools/soak/rig-qemu.py). Its pci-serial port comes after the
# ctl port (when there is one) and the always-present RS485 port, so it is
# ttyS2 with warden.ctl and ttyS1 without; the ctl/RS485 order above is
# unchanged. Echo is turned off so the host reads only the agent's frames.
# The protocol is warden_agent (/etc/warden-lib.sh). Rig rootfs only: no
# panel image carries it.
if grep -qw warden.agent /proc/cmdline; then
if grep -qw warden.ctl /proc/cmdline; then agent=/dev/ttyS2; else agent=/dev/ttyS1; fi
if [ -c "$agent" ]; then
echo "init: rig agent on $agent"
(
exec 4<> "$agent"
stty -echo <&4 2>/dev/null || echo "init: rig agent: stty -echo failed"
warden_agent <&4 >&4 2>/dev/null
) &
else
echo "init: rig agent: $agent missing (needs the virt.fragment kernel)"
fi
fi
if grep -qw warden.shell /proc/cmdline; then
echo "warden.shell: interactive shell (exit powers off)"
warden_console_shell
+14 -1
View File
@@ -17,6 +17,10 @@
# --ctl SOCK unix socket to warden-ui's debug channel (nav/page/hit),
# bridged by init from a pci-serial port; the rig-side
# twin of tools/warden-ctl over SSH on a real panel
# --agent SOCK unix socket to the rig agent (warden_agent in
# rootfs/etc/warden-lib.sh): a guest shell for host-side
# rigs such as flare-edge tools/soak/rig-qemu.py; a third
# pci-serial after ctl and rs485, so their order holds
# --watchdog add i6300esb watchdog, reset on expiry (fragment kernel)
# --qmp SOCK QMP unix socket (screendump, input-send-event, quit)
# --display MODE off (default, -nographic) | on (gtk window) | headless
@@ -38,7 +42,7 @@ QEMU_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
OUT="${OUT:-$QEMU_DIR/out}"
KERNEL="" INITRD="$OUT/initramfs.cpio.gz" DISK="" NO_DISK=0 SLOT="_a"
RTC="" RS485="" CTL="" WATCHDOG=0 QMP="" DISPLAY_MODE="off" SHELL_FLAG=0 ALLOW_APPLY=0
RTC="" RS485="" CTL="" AGENT="" WATCHDOG=0 QMP="" DISPLAY_MODE="off" SHELL_FLAG=0 ALLOW_APPLY=0
SSH_PORT=2222 HTTP_PORT=8080 API_PORT=28443
EXTRA=()
EXTRA_APPEND=""
@@ -53,6 +57,7 @@ while [ $# -gt 0 ]; do
--rtc) RTC="${2:?}"; shift 2 ;;
--rs485) RS485="${2:?}"; shift 2 ;;
--ctl) CTL="${2:?}"; shift 2 ;;
--agent) AGENT="${2:?}"; shift 2 ;;
--watchdog) WATCHDOG=1; shift ;;
--qmp) QMP="${2:?}"; shift 2 ;;
--display) DISPLAY_MODE="${2:?}"; shift 2 ;;
@@ -143,6 +148,14 @@ if [ -n "$RS485" ]; then
else
ARGS+=( -chardev "null,id=rs485" -device "pci-serial,chardev=rs485" )
fi
# The rig agent comes LAST of the pci-serial ports, so ctl stays ttyS0 and
# rs485 stays where init looks for it; init finds the agent at ttyS1 (no
# ctl) or ttyS2 (with ctl), and only with warden.agent on the cmdline.
if [ -n "$AGENT" ]; then
APPEND="$APPEND warden.agent"
ARGS+=( -chardev "socket,id=agent,path=$AGENT,server=on,wait=off"
-device "pci-serial,chardev=agent" )
fi
[ -n "$QMP" ] && ARGS+=( -qmp "unix:$QMP,server=on,wait=off" )
case "$DISPLAY_MODE" in
+117
View File
@@ -0,0 +1,117 @@
#!/usr/bin/env bash
# Host test for the rig agent, warden_agent in qemu/rootfs/etc/warden-lib.sh
# (flare-edge M9 soak-lite). Feeds request lines on stdin, as the host rig
# does over the agent's serial port, and checks the framed answers: a script
# runs and answers its output in base64 with its exit code, chunks join into
# one script, a script past its deadline is killed with its children and
# answers 124, and malformed requests answer one <<ERR ...>> line and run
# nothing.
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
LIB="$HERE/../rootfs/etc/warden-lib.sh"
# Unquoted where it runs: WARDEN_TEST_SH may be "busybox sh", the rig's shell.
SH="${WARDEN_TEST_SH:-sh}"
WORK="$(mktemp -d)" || exit 1
trap 'rm -rf "$WORK"' EXIT
FAIL=0
CHECKS=0
check() { # check <label> <command...>
CHECKS=$((CHECKS + 1))
if "${@:2}"; then
echo "[PASS] $1"
else
echo "[FAIL] $1"
FAIL=1
fi
}
b64() { printf '%s' "$1" | base64 -w0; }
agent() { # agent <name> <lines...>: run the agent over the lines
local name="$1"
shift
# $SH unquoted (it may be "busybox sh"); the quoted script is meant literally.
# shellcheck disable=SC2086,SC2016
printf '%s\n' "$@" |
env WARDEN_AGENT_DIR="$WORK/agent-$name" \
$SH -c '. "$1"; warden_agent' sh "$LIB" > "$WORK/$name.out" 2> "$WORK/$name.err"
}
same() { # same <file> <expected lines...>
local file="$1"
shift
diff <(printf '%s\n' "$@") "$file" > "$WORK/diff" || { cat "$WORK/diff"; return 1; }
}
# The body between <<BEGIN ID>> and <<END ID ...>>, decoded.
body() { # body <file> <id>
awk -v b="<<BEGIN $2>>" -v e="<<END $2 " '$0 == b {on = 1; next} index($0, e) == 1 {on = 0} on' "$1" |
base64 -d
}
agent basic @ping "@run a1 10 $(b64 'echo hello; echo oops >&2; exit 3')"
check "ping and a run answer framed lines" \
grep -qx '<<PONG>>' "$WORK/basic.out"
check "the run ends with its exit code" grep -qx '<<END a1 3>>' "$WORK/basic.out"
check "stdout and stderr come back in base64" \
same <(body "$WORK/basic.out" a1) hello oops
# Chunks: the script text arrives in pieces, then @run with no B64 of its own
# and once more with a last piece on the run line.
script='i=0; while [ $i -lt 3 ]; do echo line$i; i=$((i + 1)); done'
whole="$(b64 "$script")"
agent chunks "@chunk c1 ${whole:0:20}" "@chunk c1 ${whole:20}" "@run c1 10" \
"@chunk c2 ${whole:0:12}" "@run c2 10 ${whole:12}"
check "chunks acknowledge the bytes so far" \
grep -qx "<<OK c1 20>>" "$WORK/chunks.out"
check "chunked script runs whole" same <(body "$WORK/chunks.out" c1) line0 line1 line2
check "a last piece on the run line joins the chunks" \
same <(body "$WORK/chunks.out" c2) line0 line1 line2
check "a script's text is used once" \
test -z "$(ls -A "$WORK/agent-chunks" 2>/dev/null)"
# Deadline: the script and the child it started both die; RC 124.
start=$(date +%s)
agent late "@run t1 1 $(b64 "sleep 30 & echo \$! > $WORK/child.pid; sleep 30")" @ping
took=$(( $(date +%s) - start ))
check "a script past its deadline answers 124" grep -qx '<<END t1 124>>' "$WORK/late.out"
check "the deadline holds (took ${took}s)" test "$took" -le 5
child="$(cat "$WORK/child.pid" 2>/dev/null)"
check "the script's child is killed too" bash -c "[ -n '$child' ] && ! kill -0 '$child' 2>/dev/null"
check "the agent answers after a timeout" grep -qx '<<PONG>>' "$WORK/late.out"
# Malformed requests: one error line each, nothing runs.
touch_b64="$(b64 "touch $WORK/ran")"
agent bad "@run bad/id 10 $touch_b64" "@run b1 0 $touch_b64" "@run b1 99999 $touch_b64" \
"@run b1 10 not*base64" "@chunk b2" "@run b3 10" "frobnicate" "@run b4 10 $(b64 'x') extra" \
"@run b5 10 ===="
check "malformed requests answer one error line each" \
same "$WORK/bad.out" '<<ERR bad run>>' '<<ERR bad run>>' '<<ERR bad run>>' '<<ERR bad run>>' \
'<<ERR bad chunk>>' '<<ERR bad script b3>>' '<<ERR unknown request>>' '<<ERR bad run>>' \
'<<ERR bad script b5>>'
check "no malformed request ran anything" test ! -e "$WORK/ran"
# A trailing CR (a cooked tty line) is not part of the last word.
agent cr "$(printf '@ping\r')" "$(printf '@run r1 10 %s\r' "$(b64 'echo ok')")"
check "a trailing CR is ignored" bash -c "grep -qx '<<PONG>>' '$WORK/cr.out' && grep -qx '<<END r1 0>>' '$WORK/cr.out'"
# A request with glob characters in a word does not expand them.
agent glob "@run * 10 $(b64 'echo no')"
check "a glob word is not expanded" same "$WORK/glob.out" '<<ERR bad run>>'
# sbin/init arms the agent only on warden.agent, on the port after ctl and
# RS485 (ttyS2 with warden.ctl, ttyS1 without), with echo off.
INIT="$HERE/../rootfs/sbin/init"
check "init arms the agent only with warden.agent" grep -q 'grep -qw warden.agent /proc/cmdline' "$INIT"
check "init picks ttyS2 with ctl and ttyS1 without" \
grep -q 'if grep -qw warden.ctl /proc/cmdline; then agent=/dev/ttyS2; else agent=/dev/ttyS1; fi' "$INIT"
check "init turns echo off on the agent port" grep -q 'stty -echo <&4' "$INIT"
if [ "$FAIL" = 0 ]; then
echo "agent tests: OK ($CHECKS checks)"
else
echo "agent tests: FAILED"
exit 1
fi
+30
View File
@@ -140,6 +140,36 @@ if run_case "$argv" --ctl "$SCRATCH/ctl.sock" --rs485 "$SCRATCH/rs.sock"; then
fi
fi
# --- case: --agent (the rig agent, flare-edge M9 soak-lite) -> its
# pci-serial comes LAST, after ctl and rs485, so neither moves, and
# warden.agent lands on the cmdline only with the flag -------------------
if ! grep -qw 'warden.agent' "$SCRATCH/argv-both.txt" && ! grep -qF 'id=agent' "$SCRATCH/argv-both.txt"; then
pass "no --agent: no agent chardev and no warden.agent"
else
fail "no --agent: an agent chardev or warden.agent appeared unrequested"
fi
for combo in "agent-ctl:--ctl $SCRATCH/ctl.sock" "agent-alone:"; do
name="${combo%%:*}" extra="${combo#*:}"
argv="$SCRATCH/argv-$name.txt"
# shellcheck disable=SC2086 # extra is a word list on purpose
if run_case "$argv" $extra --rs485 "$SCRATCH/rs.sock" --agent "$SCRATCH/agent.sock"; then
ag_ln="$(chardev_line "$argv" "socket,id=agent,path=$SCRATCH/agent.sock,")"
rs_ln="$(chardev_line "$argv" "socket,id=rs485,path=$SCRATCH/rs.sock,")"
ctl_ln="$(chardev_line "$argv" "socket,id=ctl,")"
if [ -n "$ag_ln" ] && [ -n "$rs_ln" ] && [ "$rs_ln" -lt "$ag_ln" ] &&
{ [ -z "$ctl_ln" ] || [ "$ctl_ln" -lt "$rs_ln" ]; }; then
pass "$name: agent chardev (line $ag_ln) comes after rs485 (line $rs_ln) and ctl (${ctl_ln:-none})"
else
fail "$name: expected ctl < rs485 < agent, got ctl=$ctl_ln rs485=$rs_ln agent=$ag_ln"
fi
if grep -qF 'pci-serial,chardev=agent' "$argv" && grep -qw 'warden.agent' "$argv"; then
pass "$name: agent pci-serial device and warden.agent present"
else
fail "$name: agent pci-serial device or warden.agent missing"
fi
fi
done
if [ "$FAIL" -eq 0 ]; then
echo "ALL RUN.SH ARGV TESTS PASSED"
exit 0