7 Commits
Author SHA1 Message Date
noah a2ef500560 Gate benchmark releases on live Pages privacy
next-site-candidate / validate-and-candidate (push) Successful in 5s
main-site-gate / validate-and-release (push) Successful in 10s
2026-10-17 03:04:38 -06:00
noah 35c38eaf89 Tighten public benchmark repository guide
main-site-gate / validate-and-release (push) Successful in 31s
next-site-candidate / validate-and-candidate (push) Successful in 5s
2026-10-17 01:26:10 -06:00
Alkane CI ea160545c4 Expose prepublication benchmark page guard
next-site-candidate / validate-and-candidate (push) Successful in 5s
2026-10-16 08:44:49 -06:00
Alkane CI 334b993d52 Restrict public benchmark pages to generated HTML
next-site-candidate / validate-and-candidate (push) Canceled after 0s
2026-10-16 08:41:57 -06:00
Alkane CI 78f1f82b38 Add honest badges for public benchmark repository
next-site-candidate / validate-and-candidate (push) Canceled after 0s
2026-10-16 08:38:26 -06:00
Alkane CI 1c04bd6ff7 Exclude generated Python bytecode from site source
next-site-candidate / validate-and-candidate (push) Canceled after 0s
2026-10-16 08:34:32 -06:00
Alkane CI a78438fed0 Validate public benchmark Pages and gate site releases
next-site-candidate / validate-and-candidate (push) Canceled after 0s
2026-10-16 08:34:23 -06:00
18 changed files with 932 additions and 9 deletions
+26
View File
@@ -0,0 +1,26 @@
name: completed-site-ci-badge
on:
schedule:
- cron: '*/15 * * * *'
workflow_dispatch:
concurrency:
group: alkane-benchmarks-completed-badge
cancel-in-progress: false
jobs:
publish:
runs-on: alkane-ci
permissions:
actions: read
contents: write
steps:
- uses: actions/checkout@v4
- name: Publish last completed main and next attempts
env:
GITEA_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: python3 ci/last-attempt-badge.py
- name: Clean generated Python work
if: always()
run: python3 ci/cleanup.py
+40
View File
@@ -0,0 +1,40 @@
name: main-site-gate
on:
push:
branches: [main]
concurrency:
group: alkane-benchmarks-main
cancel-in-progress: false
jobs:
validate-and-release:
runs-on: alkane-ci
permissions:
contents: write
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Test site and version guards
run: python3 -m unittest discover -s ci -p 'test_*.py'
- name: Validate published Pages snapshot
run: python3 ci/check_published.py
- name: Verify live Pages and source isolation
run: python3 ci/check_live_pages.py
- name: Publish measured repository badges
env:
GITEA_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: python3 ci/metrics.py
- name: Select release version
id: version
run: python3 ci/version.py main >> "$GITHUB_OUTPUT"
- name: Release validated site configuration
if: steps.version.outputs.release == 'true'
env:
GITEA_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: python3 ci/release.py "${{ steps.version.outputs.tag }}" main
- name: Clean generated Python work
if: always()
run: python3 ci/cleanup.py
+32
View File
@@ -0,0 +1,32 @@
name: next-site-candidate
on:
push:
branches: [next]
concurrency:
group: alkane-benchmarks-next
cancel-in-progress: false
jobs:
validate-and-candidate:
runs-on: alkane-ci
permissions:
contents: write
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Validate published Pages snapshot
run: python3 ci/check_published.py
- name: Select candidate version
id: version
run: python3 ci/version.py next >> "$GITHUB_OUTPUT"
- name: Release candidate
if: steps.version.outputs.release == 'true'
env:
GITEA_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: python3 ci/release.py "${{ steps.version.outputs.tag }}" next
- name: Clean generated Python work
if: always()
run: python3 ci/cleanup.py
+2
View File
@@ -0,0 +1,2 @@
__pycache__/
*.pyc
+21 -9
View File
@@ -1,19 +1,31 @@
# Alkane benchmarks
Public, generated microbenchmark history for Alkane SDK, OS, and Toolkit.
[![CI main](https://gitea.bluefla.re/blueflare/alkane-benchmarks/raw/branch/ci-badges/main.svg)](https://gitea.bluefla.re/blueflare/alkane-benchmarks/actions)
[![CI next](https://gitea.bluefla.re/blueflare/alkane-benchmarks/raw/branch/ci-badges/next.svg)](https://gitea.bluefla.re/blueflare/alkane-benchmarks/actions)
![tests](https://gitea.bluefla.re/blueflare/alkane-benchmarks/raw/branch/ci-badges/badges/tests.svg)
![coverage](https://gitea.bluefla.re/blueflare/alkane-benchmarks/raw/branch/ci-badges/badges/coverage.svg)
![bench](https://gitea.bluefla.re/blueflare/alkane-benchmarks/raw/branch/ci-badges/badges/bench.svg)
![bench coverage](https://gitea.bluefla.re/blueflare/alkane-benchmarks/raw/branch/ci-badges/badges/bench-coverage.svg)
![lines of code](https://gitea.bluefla.re/blueflare/alkane-benchmarks/raw/branch/ci-badges/badges/loc.svg)
![dependencies](https://gitea.bluefla.re/blueflare/alkane-benchmarks/raw/branch/ci-badges/badges/dependencies.svg)
![code quality](https://gitea.bluefla.re/blueflare/alkane-benchmarks/raw/branch/ci-badges/badges/quality.svg)
![license](https://gitea.bluefla.re/blueflare/alkane-benchmarks/raw/branch/ci-badges/badges/license.svg)
[View the benchmark site](https://blueflare.gitea.bluefla.re/alkane-benchmarks/).
No Gitea login is required. The `gh-pages` branch contains only static pages.
The source repositories remain private; their main CI workflows update the
respective site paths after measurement.
Public [benchmark history](https://blueflare.gitea.bluefla.re/alkane-benchmarks/)
for Alkane SDK, OS, and Toolkit. Scheduled publishers copy measured host data
after CI. Source repositories remain private. These results do not measure
miner boards or qualify a release.
These are host microbenchmarks. They do not show miner board performance or release qualification.
This site has no microbenchmarks of its own, so its bench badges read `n/a`.
Its coverage badge measures the site validator and version selector.
## Branches
`main` holds this description. `next` is reserved for site changes before
they reach `main`. The `gh-pages` branch is generated output.
`main` runs the site guards and publishes `vX.Y.Z` releases. `next` checks
the snapshot and publishes `-rc.N` candidates. `gh-pages` holds generated
pages only. Edit `ci/version-x.txt` to change X.
Validator changes increment Y; documentation and CI changes increment Z.
## License
The site layout is [MIT licensed](LICENSE). Benchmark measurements are factual data from CI.
The site layout is [MIT licensed](LICENSE).
+54
View File
@@ -0,0 +1,54 @@
# SPDX-License-Identifier: MIT
"""Write small badge artifacts to the public Gitea CI badge branch."""
import base64
import json
from urllib.error import HTTPError
from urllib.parse import quote
from urllib.request import Request, urlopen
BADGE_BRANCH = "ci-badges"
def request_json(url: str, token: str, method: str = "GET",
payload: dict | None = None) -> dict:
body = None if payload is None else json.dumps(payload).encode()
request = Request(url, data=body, method=method,
headers={"Authorization": f"token {token}",
"Content-Type": "application/json"})
with urlopen(request, timeout=20) as response:
return json.load(response)
def publish_file(api: str, repo: str, filename: str, data: bytes,
token: str, message: str, max_bytes: int = 65_536) -> None:
if not filename or filename.startswith("/") or any(
part in ("", ".", "..") for part in filename.split("/")):
raise ValueError("invalid badge path")
if len(data) > max_bytes:
raise ValueError("artifact exceeds size limit")
path = f"{api}/repos/{repo}/contents/{quote(filename, safe='/')}"
encoded = base64.b64encode(data).decode("ascii")
for attempt in range(3):
try:
current = request_json(path + "?ref=" + BADGE_BRANCH, token)
if base64.b64decode(current["content"]) == data:
return
sha = current["sha"]
method = "PUT"
except HTTPError as error:
if error.code != 404:
raise
sha = None
method = "POST"
payload = {"branch": BADGE_BRANCH, "content": encoded,
"message": message + " [skip ci]"}
if sha is not None:
payload["sha"] = sha
try:
request_json(path, token, method, payload)
return
except HTTPError as error:
if error.code not in (409, 422) or attempt == 2:
raise
raise RuntimeError("badge write did not complete")
+53
View File
@@ -0,0 +1,53 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: MIT
"""Check that benchmark Pages are public and source repository Pages stay closed."""
from urllib.error import HTTPError, URLError
from urllib.request import HTTPRedirectHandler, Request, build_opener
BASE = "https://blueflare.gitea.bluefla.re"
PUBLIC = (
"/alkane-benchmarks/",
"/alkane-benchmarks/sdk/",
"/alkane-benchmarks/os/",
"/alkane-benchmarks/toolkit/",
)
PRIVATE = ("/alkane-sdk/", "/alkane-os/", "/alkane-toolkit/")
class NoRedirect(HTTPRedirectHandler):
def redirect_request(self, request, response, code, message, headers, url):
return None
def status(opener, path: str) -> int:
request = Request(BASE + path, headers={"User-Agent": "alkane-pages-gate/1"})
try:
with opener.open(request, timeout=15) as response:
return response.status
except HTTPError as error:
return error.code
def main() -> int:
opener = build_opener(NoRedirect)
failed = False
for path in PUBLIC + PRIVATE:
expected = 200 if path in PUBLIC else 404
try:
actual = status(opener, path)
except URLError as error:
print(f"FAIL {path}: {error.reason}")
failed = True
continue
if actual != expected:
print(f"FAIL {path}: HTTP {actual}; expected {expected}")
failed = True
else:
print(f"PASS {path}: HTTP {actual}")
return int(failed)
if __name__ == "__main__":
raise SystemExit(main())
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env python3
"""Validate the current public Pages branch in a disposable directory."""
import io
from pathlib import Path
import subprocess
import tarfile
from tempfile import TemporaryDirectory
from validate_site import validate
def main() -> int:
subprocess.run(["git", "fetch", "--quiet", "origin", "gh-pages"], check=True)
archive = subprocess.check_output(["git", "archive", "FETCH_HEAD"])
with TemporaryDirectory(prefix="alkane-benchmarks-site-") as directory:
with tarfile.open(fileobj=io.BytesIO(archive)) as source:
source.extractall(directory, filter="data")
errors = validate(Path(directory))
for error in errors:
print(error)
if errors:
return 1
print("Published benchmark pages are data-only and structurally valid")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env python3
"""Remove generated Python work from this checkout."""
from pathlib import Path
import shutil
cache = Path(__file__).resolve().parent / "__pycache__"
if cache.exists():
if cache.is_symlink() or cache.name != "__pycache__":
raise SystemExit("unsafe Python cache path")
shutil.rmtree(cache)
print("Removed site CI Python cache")
+77
View File
@@ -0,0 +1,77 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: MIT
"""Publish the last completed main/next CI result to a public badge branch."""
import os
from urllib.parse import quote, urlencode
from badge_store import publish_file, request_json
WORKFLOWS = {"main.yml": "main", "next.yml": "next"}
def badge(branch: str, conclusion: str, run_id: int) -> bytes:
passed = conclusion == "success"
word = "pass" if passed else "fail"
color = "#2ea043" if passed else "#d1242f"
title = f"CI {branch}: last completed attempt {word} (run {run_id})"
return (f'<svg xmlns="http://www.w3.org/2000/svg" width="102" height="20" '
f'role="img" aria-label="{title}"><title>{title}</title>'
'<rect width="102" height="20" rx="3" fill="#555"/>'
f'<path d="M39 0h60a3 3 0 0 1 3 3v14a3 3 0 0 1-3 3H39z" fill="{color}"/>'
'<text x="20" y="14" fill="#fff" text-anchor="middle" '
'font-family="sans-serif" font-size="11">CI</text>'
f'<text x="70" y="14" fill="#fff" text-anchor="middle" '
f'font-family="sans-serif" font-size="11">{word}</text></svg>\n').encode()
def latest_completed(api: str, repo: str, workflow: str, branch: str,
token: str) -> tuple[int, str]:
path = quote(workflow, safe="")
latest = None
for page in range(1, 21):
query = urlencode({"branch": branch, "limit": 50, "page": page})
url = f"{api}/repos/{repo}/actions/workflows/{path}/runs?{query}"
runs = request_json(url, token).get("workflow_runs", [])
for run in runs:
if run.get("head_branch") != branch or run.get("status") != "completed":
continue
if run.get("conclusion") not in ("success", "failure"):
continue
candidate = (int(run["id"]), run["conclusion"])
if latest is None or candidate[0] > latest[0]:
latest = candidate
if len(runs) < 50:
break
if latest is None:
raise RuntimeError(f"no completed {branch} run found for {workflow}")
return latest
def publish(api: str, repo: str, branch: str, data: bytes, token: str) -> None:
publish_file(api, repo, f"{branch}.svg", data, token,
f"Record completed {branch} CI attempt")
def main() -> None:
if os.environ["GITHUB_EVENT_NAME"] not in ("schedule", "workflow_dispatch"):
raise RuntimeError("badge publisher must run on a timer or manual dispatch")
if os.environ["GITHUB_REF"] != "refs/heads/main":
raise RuntimeError("badge publisher must use the main branch")
repo = os.environ["GITHUB_REPOSITORY"]
api = os.environ["GITHUB_API_URL"].rstrip("/")
token = os.environ["GITEA_TOKEN"]
completed = {}
for workflow, branch in WORKFLOWS.items():
try:
completed[branch] = latest_completed(api, repo, workflow, branch, token)
except RuntimeError:
print(f"No completed {branch} attempt yet")
for branch, (run_id, conclusion) in completed.items():
publish(api, repo, branch, badge(branch, conclusion, run_id), token)
print(f"{branch} badge records completed run {run_id}: {conclusion}")
if __name__ == "__main__":
main()
+126
View File
@@ -0,0 +1,126 @@
#!/usr/bin/env python3
"""Measure site guard tests and publish honest repository badges."""
import argparse
import ast
import html
import json
from pathlib import Path
import trace
import unittest
from badge_store import publish_file
ROOT = Path(__file__).resolve().parents[1]
OWNED = (ROOT / "ci/validate_site.py", ROOT / "ci/version.py")
def statement_lines(path: Path) -> set[int]:
tree = ast.parse(path.read_text())
return {node.lineno for node in ast.walk(tree)
if isinstance(node, ast.stmt)
and not isinstance(node, (ast.FunctionDef, ast.ClassDef,
ast.Import, ast.ImportFrom))
and not (isinstance(node, ast.Expr)
and isinstance(node.value, ast.Constant)
and isinstance(node.value.value, str))}
def tests_and_coverage() -> tuple[int, float]:
suite = unittest.TestLoader().discover(str(ROOT / "ci"), pattern="test_*.py")
count = suite.countTestCases()
tracer = trace.Trace(count=True, trace=False)
result = tracer.runfunc(unittest.TextTestRunner(verbosity=0).run, suite)
if not result.wasSuccessful():
raise RuntimeError("site guard tests failed")
hit = {(Path(path).resolve(), line) for (path, line), times
in tracer.results().counts.items() if times > 0}
executable = {(path, line) for path in OWNED for line in statement_lines(path)}
covered = sum(item in hit for item in executable)
return count, 100 * covered / len(executable)
def compact_lines(count: int) -> str:
if count < 1000:
return str(count)
for scale, suffix in ((1_000_000_000, "G"), (1_000_000, "M"), (1000, "K")):
if count >= scale:
number = count / scale
return f"{number:.1f}{suffix}" if number < 10 else f"{number:.0f}{suffix}"
raise AssertionError("unreachable")
def svg(label: str, value: str, color: str) -> bytes:
label = html.escape(label)
value = html.escape(value)
left = max(40, len(label) * 7 + 14)
right = max(36, len(value) * 7 + 14)
total = left + right
title = f"{label}: {value}"
return (f'<svg xmlns="http://www.w3.org/2000/svg" width="{total}" height="20" '
f'role="img" aria-label="{title}"><title>{title}</title>'
f'<rect width="{total}" height="20" rx="3" fill="#555"/>'
f'<path d="M{left} 0h{right - 3}a3 3 0 0 1 3 3v14a3 3 0 0 1-3 3H{left}z" fill="{color}"/>'
f'<text x="{left // 2}" y="14" fill="#fff" text-anchor="middle" '
f'font-family="sans-serif" font-size="11">{label}</text>'
f'<text x="{left + right // 2}" y="14" fill="#fff" text-anchor="middle" '
f'font-family="sans-serif" font-size="11">{value}</text></svg>\n').encode()
def generate() -> tuple[dict[str, bytes], dict]:
tests, coverage = tests_and_coverage()
sources = list((ROOT / "ci").glob("*.py"))
lines = sum(1 for path in sources if not path.name.startswith("test_")
for line in path.read_text().splitlines()
if line.strip() and not line.lstrip().startswith("#"))
if "MIT License" not in (ROOT / "LICENSE").read_text():
raise RuntimeError("site license is not MIT")
values = {
"tests": (str(tests), "#2ea043"),
"coverage": (f"{coverage:.1f}%", "#2ea043"),
"bench": ("n/a", "#777"),
"bench-coverage": ("n/a", "#777"),
"loc": (compact_lines(lines), "#0969da"),
"dependencies": ("0", "#2ea043"),
"quality": ("A", "#2ea043"),
"license": ("MIT", "#0969da"),
}
labels = {"bench-coverage": "bench coverage", "loc": "LOC",
"quality": "code quality"}
badges = {name: svg(labels.get(name, name), value, color)
for name, (value, color) in values.items()}
metrics = {"schema": 1, "test_cases": tests,
"statement_coverage_percent": round(coverage, 1),
"source_lines": lines, "dependencies": 0,
"benchmarks": None, "benchmark_coverage_percent": None,
"quality": "A", "license": "MIT"}
return badges, metrics
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--preview", type=Path)
args = parser.parse_args()
badges, metrics = generate()
data = json.dumps(metrics, indent=2).encode() + b"\n"
if args.preview:
args.preview.mkdir(parents=True, exist_ok=True)
for name, content in badges.items():
(args.preview / f"{name}.svg").write_bytes(content)
(args.preview / "metrics.json").write_bytes(data)
else:
import os
api = os.environ["GITHUB_API_URL"].rstrip("/")
repo = os.environ["GITHUB_REPOSITORY"]
token = os.environ["GITEA_TOKEN"]
for name, content in badges.items():
publish_file(api, repo, f"badges/{name}.svg", content, token,
f"Measure {name} badge")
publish_file(api, repo, "badges/metrics.json", data, token,
"Measure site repository")
print(f"Measured {metrics['test_cases']} tests and {metrics['statement_coverage_percent']}% statement coverage")
if __name__ == "__main__":
main()
+43
View File
@@ -0,0 +1,43 @@
#!/usr/bin/env python3
"""Create a scoped Gitea release for validated site configuration."""
import json
import os
import sys
import urllib.error
import urllib.request
def main() -> int:
if len(sys.argv) != 3:
raise SystemExit("usage: release.py TAG main|next")
tag, branch = sys.argv[1:]
if branch not in {"main", "next"}:
raise SystemExit("invalid branch")
token = os.environ["GITEA_TOKEN"]
api = os.environ.get("GITHUB_API_URL", "https://gitea.bluefla.re/api/v1").rstrip("/")
repo = os.environ["GITHUB_REPOSITORY"]
sha = os.environ["GITHUB_SHA"]
payload = json.dumps({
"tag_name": tag,
"target_commitish": sha,
"name": tag,
"body": f"Validated public benchmark site configuration from {sha}.",
"prerelease": branch == "next",
}).encode()
request = urllib.request.Request(
f"{api}/repos/{repo}/releases", data=payload, method="POST",
headers={"Authorization": "token " + token, "Content-Type": "application/json"},
)
try:
with urllib.request.urlopen(request, timeout=60) as response:
result = json.load(response)
except urllib.error.HTTPError as error:
print(f"release creation failed: HTTP {error.code}", file=sys.stderr)
return 1
print(f"Created {result['tag_name']}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+72
View File
@@ -0,0 +1,72 @@
#!/usr/bin/env python3
"""Protect branch badges from queued runs and recursive events."""
import importlib.util
import os
from pathlib import Path
import unittest
from unittest.mock import patch
import xml.etree.ElementTree as ET
from badge_store import publish_file
path = Path(__file__).with_name("last-attempt-badge.py")
spec = importlib.util.spec_from_file_location("last_attempt_badge", path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
ENVIRONMENT = {
"GITHUB_EVENT_NAME": "schedule",
"GITHUB_REF": "refs/heads/main",
"GITHUB_REPOSITORY": "owner/repo",
"GITHUB_API_URL": "https://example.invalid/api/v1",
"GITEA_TOKEN": "test-token",
}
class BadgeTests(unittest.TestCase):
def test_queued_run_does_not_displace_last_failure(self):
runs = {"workflow_runs": [
{"id": 105, "head_branch": "main", "status": "completed", "conclusion": "cancelled"},
{"id": 104, "head_branch": "main", "status": "queued", "conclusion": None},
{"id": 103, "head_branch": "main", "status": "completed", "conclusion": "failure"},
{"id": 102, "head_branch": "main", "status": "completed", "conclusion": "success"},
]}
with patch.object(module, "request_json", return_value=runs):
self.assertEqual(module.latest_completed("https://example.invalid/api/v1",
"owner/repo", "main.yml", "main", "token"),
(103, "failure"))
def test_badge_distinguishes_pass_and_fail(self):
badge = module.badge("main", "failure", 103)
root = ET.fromstring(badge)
self.assertEqual(root.attrib["aria-label"],
"CI main: last completed attempt fail (run 103)")
self.assertIn(b">fail</text>", badge)
self.assertIn(b">pass</text>", module.badge("next", "success", 108))
def test_timer_updates_both_branches(self):
with patch.dict(os.environ, ENVIRONMENT), \
patch.object(module, "latest_completed", side_effect=[(201, "success"),
(202, "failure")]), \
patch.object(module, "publish") as published, \
patch("builtins.print"):
module.main()
self.assertEqual([call.args[2] for call in published.call_args_list],
["main", "next"])
def test_workflow_run_event_cannot_trigger_publisher(self):
with patch.dict(os.environ, {**ENVIRONMENT, "GITHUB_EVENT_NAME": "workflow_run"}):
with self.assertRaisesRegex(RuntimeError, "timer or manual dispatch"):
module.main()
def test_unsafe_badge_paths_fail(self):
for name in ("", "/main.svg", "../main.svg", "badges//main.svg"):
with self.assertRaises(ValueError):
publish_file("https://example.invalid/api/v1", "owner/repo", name,
b"<svg/>", "token", "test")
if __name__ == "__main__":
unittest.main()
+117
View File
@@ -0,0 +1,117 @@
#!/usr/bin/env python3
"""Privacy and shape checks for the generated Pages validator."""
import tempfile
import unittest
from pathlib import Path
import subprocess
import sys
from validate_site import validate
INDEX = """<!doctype html><html><head><title>Alkane benchmarks</title></head>
<body><h1>Alkane benchmarks</h1><a href="sdk/">SDK</a>
<a href="os/">OS</a><a href="toolkit/">Toolkit</a></body></html>"""
PAGE = """<!doctype html><html><head><title>Alkane benchmarks</title></head>
<body><h1>Alkane benchmarks</h1><table><tr><td>10 ns/op</td></tr></table></body></html>"""
class SiteValidationTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
(self.root / "index.html").write_text(INDEX)
for repo in ("sdk", "os", "toolkit"):
(self.root / repo).mkdir()
(self.root / repo / "index.html").write_text(PAGE)
def test_valid_data_only_site(self):
self.assertEqual(validate(self.root), [])
def test_single_page_prepublication_cli(self):
command = [sys.executable, str(Path(__file__).with_name("validate_site.py")),
"--page", "os", str(self.root / "os/index.html")]
self.assertEqual(subprocess.run(command, capture_output=True).returncode, 0)
(self.root / "os/index.html").write_text(PAGE + '<meta http-equiv="refresh" content="0">')
self.assertEqual(subprocess.run(command, capture_output=True).returncode, 1)
def test_rejects_active_content(self):
(self.root / "os/index.html").write_text(PAGE + "<script>alert(1)</script>")
self.assertTrue(any("forbidden tag" in error for error in validate(self.root)))
def test_rejects_external_source_and_credential_marker(self):
(self.root / "sdk/index.html").write_text(
PAGE + '<img src="https://example.com/a.png">GITEA_TOKEN'
)
errors = validate(self.root)
self.assertTrue(any("forbidden tag: img" in error for error in errors))
self.assertTrue(any("credential marker" in error for error in errors))
def test_allows_only_pinned_source_commit_links(self):
commit = "a" * 40
(self.root / "os/index.html").write_text(
PAGE + f'<a href="https://gitea.bluefla.re/blueflare/alkane-os/commit/{commit}">commit</a>'
)
self.assertEqual(validate(self.root), [])
(self.root / "os/index.html").write_text(
PAGE + '<a href="https://example.com/commit/secret">commit</a>'
)
self.assertTrue(any("external or active reference" in error for error in validate(self.root)))
def test_rejects_extra_files(self):
(self.root / "source.tar.gz").write_bytes(b"private source")
self.assertTrue(any("unexpected published files" in error for error in validate(self.root)))
def test_requires_all_three_reports(self):
(self.root / "toolkit/index.html").unlink()
self.assertTrue(any("missing regular page" in error for error in validate(self.root)))
def test_rejects_css_import(self):
(self.root / "os/index.html").write_text(PAGE + "<style>@import 'https://example.com/x.css';</style>")
self.assertTrue(any("external CSS reference" in error for error in validate(self.root)))
def test_rejects_css_escaped_import(self):
(self.root / "os/index.html").write_text(
PAGE + r"<style>@im\70ort 'https://example.com/x.css';</style>"
)
self.assertTrue(any("external CSS reference" in error for error in validate(self.root)))
def test_rejects_meta_refresh(self):
(self.root / "os/index.html").write_text(
PAGE + '<meta http-equiv="refresh" content="0;url=https://example.com/">'
)
self.assertTrue(any("unexpected meta directive" in error for error in validate(self.root)))
def test_rejects_base_and_unexpected_attributes(self):
(self.root / "os/index.html").write_text(
PAGE + '<base href="https://example.com/"><a href="../" ping="https://example.com/">Back</a>'
)
errors = validate(self.root)
self.assertTrue(any("forbidden tag: base" in error for error in errors))
self.assertTrue(any("forbidden attribute on a: ping" in error for error in errors))
def test_rejects_unlisted_local_link(self):
(self.root / "os/index.html").write_text(
PAGE + '<a href="../../private/">Source</a>'
)
self.assertTrue(any("external or active reference" in error for error in validate(self.root)))
def test_rejects_symlink(self):
(self.root / "private").symlink_to(self.root / "os/index.html")
self.assertTrue(any("contains a symlink" in error for error in validate(self.root)))
def test_requires_index_navigation(self):
(self.root / "index.html").write_text(INDEX.replace('<a href="toolkit/">Toolkit</a>', ""))
self.assertTrue(any("missing project links" in error for error in validate(self.root)))
def test_requires_measurements_or_pending_notice(self):
(self.root / "sdk/index.html").write_text(
"<!doctype html><html><head><title>Alkane</title></head><body><h1>Alkane</h1></body></html>"
)
self.assertTrue(any("no measurements" in error for error in validate(self.root)))
if __name__ == "__main__":
unittest.main()
+38
View File
@@ -0,0 +1,38 @@
#!/usr/bin/env python3
"""Check the site release schedule before tag creation."""
import unittest
from version import release_tag
class VersionTests(unittest.TestCase):
def test_first_release_and_candidate(self):
self.assertEqual(release_tag("main", [], ["initial"], 0), "v0.1.0")
self.assertEqual(release_tag("next", [], ["initial"], 0), "v0.1.0-rc.1")
def test_candidate_increments_and_merges_to_final(self):
tags = ["v0.1.0", "v0.1.1-rc.1", "v0.1.1-rc.2"]
self.assertEqual(release_tag("next", tags, ["README.md"], 0), "v0.1.1-rc.3")
self.assertEqual(release_tag("main", tags, ["README.md"], 0), "v0.1.1")
def test_validator_change_bumps_minor(self):
self.assertEqual(
release_tag("main", ["v0.1.2"], ["ci/validate_site.py"], 0),
"v0.2.0",
)
def test_no_change_skips_release(self):
self.assertIsNone(release_tag("main", ["v0.1.0"], [], 0))
def test_major_cannot_move_backwards(self):
with self.assertRaises(ValueError):
release_tag("main", ["v2.0.0"], ["README.md"], 1)
def test_hand_edited_major_resets_minor_and_patch(self):
self.assertEqual(release_tag("main", ["v0.8.7"], ["ci/version-x.txt"], 1),
"v1.0.0")
if __name__ == "__main__":
unittest.main()
+128
View File
@@ -0,0 +1,128 @@
#!/usr/bin/env python3
"""Validate the generated public benchmark pages before publication."""
import argparse
from html.parser import HTMLParser
from pathlib import Path
import re
PAGES = ("index.html", "sdk/index.html", "os/index.html", "toolkit/index.html")
ALLOWED_TAGS = {"html", "head", "meta", "title", "style", "body", "h1",
"p", "a", "ul", "li", "div", "small", "table", "thead",
"tbody", "tr", "th", "td"}
ALLOWED_ATTRIBUTES = {"html": {"lang"}, "a": {"href"},
"div": {"class"}, "th": {"scope"}}
LOCAL_LINKS = {"sdk/", "os/", "toolkit/", "../"}
FORBIDDEN_TEXT = ("BEGIN PRIVATE KEY", "GITEA_TOKEN", "ghp_", "glpat-")
COMMIT_LINK = re.compile(
r"https://gitea\.bluefla\.re/blueflare/alkane-(?:sdk|os|toolkit)/commit/[0-9a-f]{40}"
)
class PageParser(HTMLParser):
def __init__(self):
super().__init__(convert_charrefs=True)
self.title = False
self.heading = False
self.tables = 0
self.links = set()
self.errors = []
self.in_style = False
def handle_starttag(self, tag, attrs):
if tag not in ALLOWED_TAGS:
self.errors.append(f"forbidden tag: {tag}")
if tag == "title":
self.title = True
if tag == "h1":
self.heading = True
if tag == "table":
self.tables += 1
if tag == "style":
self.in_style = True
names = [name for name, _ in attrs]
if len(names) != len(set(names)):
self.errors.append(f"duplicate attribute on {tag}")
if tag == "meta" and attrs not in (
[("charset", "utf-8")],
[("name", "viewport"), ("content", "width=device-width, initial-scale=1")],
):
self.errors.append("unexpected meta directive")
for name, value in attrs:
value = value or ""
if tag != "meta" and name not in ALLOWED_ATTRIBUTES.get(tag, set()):
self.errors.append(f"forbidden attribute on {tag}: {name}")
if tag == "a" and name == "href":
if value not in LOCAL_LINKS and not COMMIT_LINK.fullmatch(value):
self.errors.append(f"external or active reference: {value}")
self.links.add(value)
def handle_endtag(self, tag):
if tag == "style":
self.in_style = False
def handle_data(self, data):
if self.in_style and (re.search(r"url\s*\(|@import", data, re.IGNORECASE)
or "\\" in data):
self.errors.append("external CSS reference is not allowed")
def validate_page(path: Path, relative: str) -> list[str]:
if not path.is_file() or path.is_symlink():
return [f"{relative}: missing regular page"]
if path.stat().st_size > 2_000_000:
return [f"{relative}: page exceeds 2 MB"]
text = path.read_text(encoding="utf-8")
parser = PageParser()
parser.feed(text)
errors = [f"{relative}: {item}" for item in parser.errors]
if not text.lstrip().lower().startswith("<!doctype html>"):
errors.append(f"{relative}: missing HTML doctype")
if not parser.title or not parser.heading:
errors.append(f"{relative}: missing title or heading")
if any(token in text for token in FORBIDDEN_TEXT):
errors.append(f"{relative}: credential marker in public output")
if re.search(r"url\s*\(|@import", text, re.IGNORECASE):
errors.append(f"{relative}: external CSS reference is not allowed")
if relative == "index.html":
if not {"sdk/", "os/", "toolkit/"}.issubset(parser.links):
errors.append("index.html: missing project links")
elif not parser.tables and "Waiting for a completed measured CI run." not in text:
errors.append(f"{relative}: no measurements or pending notice")
return errors
def validate(root: Path) -> list[str]:
if not root.is_dir() or root.is_symlink():
return ["site root is not a directory"]
paths = {p.relative_to(root).as_posix() for p in root.rglob("*") if p.is_file()}
errors = []
if any(p.is_symlink() for p in root.rglob("*")):
errors.append("published site contains a symlink")
for relative in PAGES:
errors.extend(validate_page(root / relative, relative))
unexpected = paths - set(PAGES)
if unexpected:
errors.append("unexpected published files: " + ", ".join(sorted(unexpected)))
return errors
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("site", type=Path)
parser.add_argument("--page", choices=("sdk", "os", "toolkit"),
help="validate one generated report before publication")
args = parser.parse_args()
errors = (validate_page(args.site, f"{args.page}/index.html")
if args.page else validate(args.site))
for error in errors:
print(error)
if errors:
return 1
print(f"Validated {args.page or len(PAGES)} public benchmark pages")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+1
View File
@@ -0,0 +1 @@
0
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env python3
"""Select site configuration release tags; version X is edited by hand."""
import argparse
from pathlib import Path
import re
import subprocess
FINAL = re.compile(r"v(\d+)\.(\d+)\.(\d+)$")
RC = re.compile(r"v(\d+)\.(\d+)\.(\d+)-rc\.(\d+)$")
def git(*args: str) -> str:
return subprocess.check_output(["git", *args], text=True).strip()
def release_tag(branch: str, tags: list[str], changed: list[str], major: int) -> str | None:
finals = sorted((tuple(map(int, match.groups())), tag)
for tag in tags if (match := FINAL.fullmatch(tag)))
latest = finals[-1] if finals else None
if latest and latest[0][0] > major:
raise ValueError("manual major version is behind the latest release")
if not changed:
return None
if latest and latest[0][0] == major:
_, minor, patch = latest[0]
if "ci/validate_site.py" in changed:
minor, patch = minor + 1, 0
else:
patch += 1
else:
minor, patch = (1, 0) if major == 0 else (0, 0)
base = f"v{major}.{minor}.{patch}"
if branch == "main":
return base
numbers = [int(match.group(4)) for tag in tags
if (match := RC.fullmatch(tag)) and tag.startswith(base + "-rc.")]
return f"{base}-rc.{max(numbers, default=0) + 1}"
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("branch", choices=("main", "next"))
args = parser.parse_args()
major = int(Path(__file__).with_name("version-x.txt").read_text().strip())
if major < 0:
raise ValueError("major version must be nonnegative")
tags = git("tag", "-l", "v*").splitlines()
finals = sorted((tuple(map(int, match.groups())), tag)
for tag in tags if (match := FINAL.fullmatch(tag)))
changed = git("diff", "--name-only", finals[-1][1], "HEAD").splitlines() if finals else ["initial"]
tag = release_tag(args.branch, tags, changed, major)
print(f"release={'true' if tag else 'false'}")
if tag:
print(f"tag={tag}")
if __name__ == "__main__":
main()