The test-gap audit for nondeterminism — golden sets, regression evals, judge quality, and the drift between eval and production.
Free & open · no signup · read-only — it ends by asking · nothing leaves your machine
You are working inside this repo. Mission: map what agent behavior is actually evaluated against what the product actually does, and produce the eval plan that catches regressions before users do.
Read-only pass. Your only write is the report file.
Phase 2 sweeps the codebase through every one of these, citing file and line for each finding.
task types × eval existence: the matrix, with the uncovered high-volume cells circled
size, age, whether cases came from real traffic or imagination
LLM judges never validated against human labels; position, length, and self-preference bias unchecked
prompt and model changes in git log that shipped with no eval run attached
sanitized eval inputs vs the messy inputs production actually receives
pass rates hiding severity; no cost or latency tracked alongside quality
single-sample pass/fail on stochastic outputs; no repeat runs, no variance reported
production failures becoming new eval cases, or evaporating
One structured report at the repo root — or in reports/, if you keep one — the same shape every time, ready for a teammate — or the optional Studio — to act on.
Every Goal Prompt follows the same four steps, so results are consistent and repeatable — no matter which one you run.
What exists.
Coverage vs reality — task types × eval existence.
Rank gaps by volume × blast radius of an unnoticed regression.
Create EVALS.md at repo root.
Copy it and paste it into your agent inside the repo you want checked.
Install the goal plugin once — two commands — then just type /goal:eval-coverage-audit.
/plugin marketplace add GhostlyGawd/goal-prompts/plugin install goal@goal-promptsOr install only this Goal Prompt as /goal-eval-coverage-audit:
curl -fsSL https://goal-prompts.vercel.app/install | BRIEF=34 shLet an agent fetch it mid-conversation, or pull the raw Goal Prompt by URL.
https://goal-prompts.vercel.app/raw/34.mdNothing hidden — this is the whole Goal Prompt, verbatim. Read it in a minute, edit it, or copy it as-is.
# Goal: Eval Coverage Audit You are working inside this repo. Mission: map what agent behavior is actually evaluated against what the product actually does, and produce the eval plan that catches regressions before users do. Read-only pass. Your only write is the report file. ## Phase 1 — Learn the eval reality - What exists: golden datasets, assertion suites, LLM judges, human review, dashboards? Where do they live, when do they run — CI, ad hoc, never? - List the product's real task types (from code and traces), with rough volume per type. - What raw material exists for building evals — traces, logs, user corrections? ## Phase 2 — Audit through 8 lenses 1. **Coverage vs reality** — task types × eval existence: the matrix, with the uncovered high-volume cells circled 2. **Golden-set health** — size, age, whether cases came from real traffic or imagination 3. **Judge quality** — LLM judges never validated against human labels; position, length, and self-preference bias unchecked 4. **Regression protection** — prompt and model changes in git log that shipped with no eval run attached 5. **Eval–prod drift** — sanitized eval inputs vs the messy inputs production actually receives 6. **Metric honesty** — pass rates hiding severity; no cost or latency tracked alongside quality 7. **Nondeterminism handling** — single-sample pass/fail on stochastic outputs; no repeat runs, no variance reported 8. **Failure feedback** — production failures becoming new eval cases, or evaporating ## Phase 3 — Curate - Rank gaps by volume × blast radius of an unnoticed regression - Prefer small, real-traffic golden sets over large synthetic ones ## Phase 4 — Report Create `EVALS.md` at repo root: 1. **Coverage matrix** — task type · volume · eval today · gap 2. **Health verdicts** — golden sets, judges, CI wiring 3. **Eval plan** — each: capability · dataset source (mine the traces) · grader (assertion/judge/human) · size · run trigger 4. **The first eval** — spelled out completely, buildable today Start the report with today's date. If `EVALS.md` already exists from a previous run, read it first and lead with what changed since. ## Rules - An eval that never blocks a ship is a dashboard, not a gate - Judge the judges: no LLM grader without a human-agreement check - No LLM behavior to evaluate in this repo? Say so in a one-paragraph null report and stop — a null result is a valid finding. - If a `reports/` directory exists at the repo root, write the report there instead of the root. - Before asking, present the top findings as a ranked list in plain words - Report only — end by asking which evals to build
reports/ directory exists at the repo root, write the report there instead of the root.Curated neighbors — briefs that answer the adjacent question, worth running in the same session.
Every system prompt in the repo: contradictions, dead instructions, bloat, and injection surface — with a consolidation plan.
Your tools are an API whose consumer is a model — audit names, descriptions, error contracts, and guards on destructive actions.
How runs end — iteration caps, stop conditions, stuck-state detection, and the true cost of one pathological run.
Reconstruct what actually enters the model's window each step — stale payloads, token hogs, retrieval junk, and buried instructions.
Runs inside these playbooks — curated sequences you can launch with one paste: