Model evaluation
CURRENT: none. Humans can read pastes in /admin/. Agents cannot write a score.
PLANNED: structured ratings + automated evaluators + per-project weights.
Human in the loop
Users (and staff) should be able to:
- thumbs up/down
- rate 1-10
- compare A/B candidates
- mark production-ready
- reject with a required reason code
- tag defects (overlap, contrast, overflow, wrong brand, broken form)
- select a winner
Store these as ratings rows: rater id, scale, value, comment, created_at. Do not bury them in paste bodies.
Automated evaluators
| Family | Examples | Failure mode |
|---|---|---|
| LLM-as-judge | Rubric JSON | Halo bias, prompt injection from candidate text |
| Vision | EE VIZN, screenshot rubric | Cost, lighting, dynamic data |
| Deterministic | schema, PHP lint, SQL parse | Misses visual bugs |
| A11y | axe, contrast | Incomplete coverage |
| Lighthouse | perf, SEO | Lab vs field |
| Pixel diff | baseline vs candidate | False positives on ads/time |
| Tests | unit, Playwright | Brittle selectors |
| Economics | latency_ms, cost_cents | Not quality |
| Product | conversion, bounce | Lagging, confounders |
No single metric elects a winner.
Weighted score
Example default (configurable per project, basis points):
Overall = Visual 3000 + Functionality 2500 + Brand 1500 + Accessibility 1000 + Performance 1000 + Human 1000 (sum 10000)
Payment, auth, and ledger paths should raise Functionality to a veto: if deterministic tests fail, overall is 0 regardless of visual score.
Leaderboard
GET /experiments/{id}/leaderboard returns runs sorted by overall, with metric breakdown, cost, latency, and whether a human overrode the rank.
RESEARCH
Calibrating LLM judges against EE staff ratings. Until calibration error is measured, auto-promote stays off.