Metrics tab¶

The "metrics" tab in the status strip replaces the console with the evaluation report: the measured effect of the shield over seeded episodes. The numbers are not from the episode you are watching. They come from interlock eval, see Evaluation and results. While the tab is open the strip shows the report's generation time, seed count, thresholds and judge mode in place of the source menu and transport controls.

The metrics tab

What it loads¶

The tab fetches /eval/latest from the backend once when opened. The server returns the newest report written by the evaluation, internal/eval/latest.json. If none exists the tab says "No evaluation yet" and names the command that produces one:

uv run interlock eval --seeds 0-19 --all

The backend container ships with the report baked in, so the hosted console always has one.

The five tiles¶

Each tile leads with the difference the shield makes and shows the shield-off value in orange and the shield-on value in blue under it, both at the default thresholds.

Tile	Headline	Meaning
violations per episode	`30× fewer`, or `none with the shield`	executed actions the ground-truth checkers labeled as violations, averaged over seeds
task return	`-9.4% return`	the environment's own reward summed per episode, averaged: the cost of safety
crash rate	`40% → 0%`	share of episodes that ended in a collision
steps escalated	`16.5% of steps`	share of steps sent to the operator (always 0 with the shield off)
cost per episode	`$0.0005 / episode`	Jev input tokens priced per episode

The four charts¶

Violations per seed. Grouped bars, one pair per seed, shield off versus on. The spread shows that the reduction holds on every seed and not only on average.

Task return per seed. The same layout for return. Expect the shielded bars to be a little lower: the speed limit costs speed reward.

Choosing the thresholds. One point per threshold pair in the sweep, plotted as violations per episode against the share of steps escalated, with a table beside it that names each pair and marks the default. Moving left along the curve asks the operator less; moving down violates less. The knee is where to operate when operator attention is the scarce resource.

Is p(violation) calibrated? A reliability diagram. Jev's probabilities are bucketed into ten bins; for each bin the chart plots the mean predicted probability against the observed violation rate from the checkers. The grey diagonal is perfect calibration. One dashed line for all rules together and one solid line per preset rule. The table beside the chart gives each line's expected calibration error, how many checks it rests on and where its labels came from (the checker, or the labeling of the two fuzzy rules).

The floor table¶

The physics floor. Below the charts, when a threshold pair was run with more than one floor, one row per pair and floor shows crash rate, violations per episode, task return, the share of steps escalated and the share of steps on which the floor removed at least one candidate. "off" is a run with no floor. Rows are banded by pair and the default pair and floor is marked.

Jev against chat models. When interlock baseline has run, a second reliability diagram puts Jev's line next to one line per chat model on the same sampled checks. The table beside it gives ECE, Brier, accuracy at 0.5, median latency and cost per judge, and how many checks a model failed to answer.

Reading a reliability diagram¶

A point on the diagonal means that when Jev said 0.7, about 70% of those checks really were violations.
Points below the diagonal are overconfident: Jev said 0.7 and fewer were violations.
Points above are underconfident.
ECE, expected calibration error, is the average gap between the line and the diagonal weighted by how many checks fall in each bin. Lower is better; 0 is perfect.
Brier in the subtitle is the mean squared error of the probabilities against the labels. It rewards both calibration and sharpness.

A rule with a flat or wobbly line between 0.3 and 0.6 is one the judge finds fuzzy; those are the probabilities that land in the escalation band.

The last line gives when the report was generated, the p50 and p95 Jev latency per shielded step over live requests, and how many escalations the simulated operator approved and denied during the eval. The simulated operator answers from the checkers, so it is always right; a person will not be.

Back to the console¶

Press "console" in the strip. The live episode kept running while you were on the metrics tab.