Testing

The backend has a pytest suite that runs offline in about five seconds: 100 tests collected from 14 files under tests/, with no network, no API key and no trained checkpoint required. The frontend has a strict typecheck and a three-test Playwright smoke run against a mock backend. There are no live Jev tests and no frontend unit tests.

Running

uv run pytest                                   # the whole suite
uv run pytest tests/test_shield.py -q           # one file
uv run pytest -k "escalation or replay"         # by name
uv run ruff format backend tests && uv run ruff check backend tests
cd frontend && pnpm typecheck
cd frontend && pnpm test:e2e

pyproject.toml sets testpaths = ["tests"] and relaxes the ANN rules for tests only, so test functions need no annotations.

Suite map

File	Covers
test_env.py	Action names match highway-env, the demo config and overrides, seeded reset reproducibility
test_policy.py	Ranking covers every action in Q order, the top rank equals SB3's greedy prediction, finite Q-values, the promoted checkpoint when present
test_facts.py	Lane clamping, the facts schema from live simulator state, gaps against the simulator geometry, the lane-to-the-right block, zone facts, snapshots and sentences, edge lane changes as no-ops
test_spike_facts.py	The spike's hand-written fact builder, the speed grid, positive question phrasing with the rule verbatim, shape A state keys
test_constraints.py	Preset checker inference including typed variants, ids and toggling, empty text rejection, zone drop and retirement
test_checkers.py	Each checker on crafted facts, free-text rules label None, label_all covers every pair
test_jev_client.py	Question text, cache key dependencies, mock-mode misses at 0.5, caching and replay of live answers, outage after a warm cache, no constraints means no request, make_client modes, the judge record
test_shield.py	Every branch of decide, the three fallback tiers with the no-op exclusion and rank tie-break, the no-compliant-action escalation, resolve for approve, deny, override and a bad override, inclusive band edges, threshold validation
test_logger.py	Run id format, the record has exactly the contract keys, JSONL round trip
test_episode.py	Mock shield-on escalates every step as no compliant action and logs, the simulated operator overrides conflicts with the fewest-violation candidate and denies true violations when uncertain, shield off in shadow, no rules means allow, a mid-episode zone drop appears in the facts, a passed zone retires its rule
test_metrics.py	Episode counters and rates, the all-blocked flag counted over old and new records, conflicts, pooled aggregates, reliability bins and ECE, labels filling in free-text rules with their labelers as the label source
test_baseline_llm.py	Collecting labeled checks with their facts once per state, hand labels for fuzzy rules, stratified sampling with top-up, the prompt carrying the judge's question verbatim and the facts JSON, the reply cache, a fake chat through run_models with failures and prices, and the dry run against a temporary eval directory
test_label.py	Content keys, calibration groups, collecting unlabeled pairs by id, preset id or text, sampling across probability bins while skipping labeled keys, the session's write, skip, quit and re-prompt paths, reading a log or an eval directory
test_autolabel.py	Both standards on crafted facts, the skip band around the loose thresholds, and a run that labels each state once, writes nothing on a dry run, adds nothing on a rerun and reaches the calibration with its label source
test_evaluate.py	Configuration labels and a full run_config plus build_report in a temp directory
test_twin.py	Finding the newest eval off run before a runs/ log and ignoring shielded logs, the compact message scored from the logged facts against the rules given (a fuzzy or disabled rule counts nothing, a second checker rule adds its steps, no scored rule keeps the scene with a zero count), the scoring key, and the empty twin for no log, a shielded log or another seed
test_server.py	Connect greeting, the twin after a reset, the twin re-scored and re-sent when a scored rule is added or switched off and not when a rule without a checker is added, a replay's twin scored against the replayed record's rules, history for late joiners, the escalation round trip with the mock judge, an override that executes a named candidate and an error for one outside the candidates, play and pause, shield and threshold and speed and reset commands, constraint broadcasts, a step record over the socket, replay streaming, reset during an open escalation, /eval/latest, CORS with and without CONSOLE_ORIGINS
test_cli.py	Subcommand dispatch, module flags, --help, usage errors, console flags to environment, child process supervision and reaping

Shared patterns

An untrained policy. Module-scoped fixtures build Policy(build_model(make_env(), seed=0)) from backend.train. The Q-values are arbitrary but deterministic, which is all the shield, the logger and the server need. Only test_default_checkpoint_loads_and_ranks_when_present touches models/dqn_highway.zip, and it calls pytest.skip("no promoted checkpoint in models/") when the file is absent.

An offline judge. JevClient(live=None, cache=JevCache(None)) is mock mode with an in-memory cache: every question misses and comes back at p = 0.5 with source unavailable, so every shielded step escalates. test_jev_client.py adds a FakeJudge that stands in for LiveJudge: it returns scripted probabilities per action, records every (state, questions) call, and returns None when constructed with no table, which is how an outage after a warm cache is tested without a network.

The server through TestClient. tests/test_server.py builds Session(policy, JevClient(...), escalation_timeout_s=None) and wraps create_app(session) in FastAPI's TestClient. The fixture monkeypatches server.RUNS_DIR, backend.logger.LOGS_DIR and the twin's RUNS_DIR and EVAL_DIR to tmp_path so runs never land in backend/logs/ and the twin lookup sees only what the test wrote. Two helpers keep the tests readable: drain_greeting(ws) consumes hello, state, history, metrics and twin in order and returns the hello; recv_until(ws, kind, **fields) reads frames until one matches. The CORS tests set CONSOLE_ORIGINS with monkeypatch.setenv, build a fresh create_app(session) so the lifespan reads it, and check the access-control-allow-origin header on /health for a listed origin and for a stranger.

The record contract guard. tests/test_logger.py defines PLAN_KEYS, the exact top-level key set of a log record. test_record_has_exactly_the_plan_keys_and_roundtrips and the episode tests assert set(record) == PLAN_KEYS, so adding or renaming a field fails here first. See Log record.

Process supervision. tests/test_cli.py tests run_console without Vite or uvicorn by monkeypatching VITE_BIN, server_command and ui_command to small python -c children: one sleeps, one exits with code 3 or kills itself with SIGTERM. The tests check that the survivor is stopped, that the exit code is passed through shell-style (128 + 15 for a signalled child), and that reap() escalates from SIGTERM to SIGKILL for a child that ignores the first.

The smoke test

frontend/e2e/smoke.spec.ts runs under frontend/playwright.config.ts, which starts its own stack so an open dev console on 8000 and 5173 is untouched:

• uv run uvicorn backend.server:app --host 127.0.0.1 --port 8001 from the repo root with JEV_MODE=mock, ESCALATION_TIMEOUT_S=0, SEED=0 and JEV_CACHE_PATH=frontend/e2e/.cache/jev_cache.jsonl, a cache path that holds nothing, so every question misses.
• pnpm exec vite --port 5174 --strictPort with SHIELD_BACKEND=http://127.0.0.1:8001.

The first test connects, sees jev mock and the empty rules message, adds the speed limit preset, steps once, waits for waiting on operator and the judge unavailable badge, denies, opens step 0 in the inspector, confirms operator deny, presses Escape, drops a work zone and checks the zone row and its lane_closed checker appear. The second test switches to the metrics view and accepts either the report's first tile or the "No evaluation yet" message, then returns to the console. Viewport is 1280 by 800, the narrowest window the console is designed for, so the header's one-row layout is checked on every run; Chromium only, traces kept on failure under frontend/test-results/.

What is not covered

• No test calls Jev. test_live_judge_constructs_and_closes_without_network only builds and closes a LiveJudge. Measured behaviour comes from interlock spike and interlock eval, which are tools, not tests.
• No frontend unit tests. The store, the selectors and apply() are exercised only through the smoke test and pnpm typecheck.
• The canvas in road-view.tsx is not asserted anywhere; the smoke test checks the panel title only.
• Training is not tested beyond build_model; interlock train is run by hand.