Whether the product's AI outputs are grounded in real sources or confidently invented — and whether it cites, hedges, and refuses when it should.
Free & open · no signup · read-only — it ends by asking · nothing leaves your machine
You are working inside this repo. Mission: judge whether the product's AI tells the truth about what it knows — whether outputs are grounded in real data, attributed so users can verify, and honest about uncertainty instead of confidently making things up.
Read-only pass. Read the prompts, retrieval, and output-handling code; change nothing but the report file.
Phase 2 sweeps the codebase through every one of these, citing file and line for each finding.
are outputs tied to retrieved sources and data, or generated unmoored
does it show where a claim came from so a user can verify
does it hedge when unsure, or state everything with equal certainty
will it decline when it lacks grounds, or always answer
where it invents facts, figures, citations, or APIs
checks that catch ungrounded claims before they reach the user
extra care where a wrong answer causes real harm (medical, legal, financial)
One structured report at the repo root — or in reports/, if you keep one — the same shape every time, ready for a teammate — or the optional Studio — to act on.
Every Goal Prompt follows the same four steps, so results are consistent and repeatable — no matter which one you run.
Find where the product generates content or answers with a model.
Grounding — are outputs tied to retrieved sources and data, or generated unmoored.
Rank by harm of a confident wrong answer × how often that surface is used.
Create GROUNDING.md at repo root.
Copy it and paste it into your agent inside the repo you want checked.
Install the goal plugin once — two commands — then just type /goal:hallucination-grounding.
/plugin marketplace add GhostlyGawd/goal-prompts/plugin install goal@goal-promptsOr install only this Goal Prompt as /goal-hallucination-grounding:
curl -fsSL https://goal-prompts.vercel.app/install | BRIEF=117 shLet an agent fetch it mid-conversation, or pull the raw Goal Prompt by URL.
https://goal-prompts.vercel.app/raw/117.mdNothing hidden — this is the whole Goal Prompt, verbatim. Read it in a minute, edit it, or copy it as-is.
# Goal: Hallucination & Grounding Audit You are working inside this repo. Mission: judge whether the product's AI tells the truth about what it knows — whether outputs are grounded in real data, attributed so users can verify, and honest about uncertainty instead of confidently making things up. Read-only pass. Read the prompts, retrieval, and output-handling code; change nothing but the report file. ## Phase 1 — Trace where answers come from - Find where the product generates content or answers with a model. - For each, see whether the output is grounded in retrieved sources or produced from the model's memory. - Note what the user is shown about sources and confidence. ## Phase 2 — Audit through 7 lenses 1. **Grounding** — are outputs tied to retrieved sources and data, or generated unmoored 2. **Citation & attribution** — does it show where a claim came from so a user can verify 3. **Confidence calibration** — does it hedge when unsure, or state everything with equal certainty 4. **Refusal & "I don't know"** — will it decline when it lacks grounds, or always answer 5. **Fabrication surfaces** — where it invents facts, figures, citations, or APIs 6. **Verification guardrails** — checks that catch ungrounded claims before they reach the user 7. **High-stakes handling** — extra care where a wrong answer causes real harm (medical, legal, financial) ## Phase 3 — Curate - Rank by harm of a confident wrong answer × how often that surface is used. - For each, name the fix — ground it in retrieval, require citations, add a refusal path, gate high-stakes output. - Separate "sometimes wrong" from "wrong with authority"; the second erodes trust fastest. ## Phase 4 — Report Create `GROUNDING.md` at repo root: 1. **Grounding map** — where outputs are grounded, and where they are free-generated 2. **Fabrication-prone surfaces** — the places most likely to invent, ranked by harm 3. **Findings** — each: surface · the risk · the grounding, citation, or refusal fix 4. **High-stakes gates** — the outputs that need verification before a user sees them Start the report with today's date. If `GROUNDING.md` already exists from a previous run, read it first and lead with what changed since. ## Rules - A confident hallucination is more dangerous than an honest "I don't know" - Grounding without visible citation still leaves the user unable to verify - No model-generated content in this repo? Say so in a one-paragraph null report and stop — a null result is a valid finding. - If a `reports/` directory exists at the repo root, write the report there instead of the root. - Before asking, present the top findings as a ranked list in plain words - Report only — end by asking which grounding gaps to close first
reports/ directory exists at the repo root, write the report there instead of the root.Find the code that ranks, scores, or filters people, list the exact fields it consumes, and check whether any eval ever sliced results by group — suspicions labeled.
Attack the product's own AI like an adversary — make it ignore instructions, leak secrets, or misuse tools — and find where untrusted input can hijack it.
Diff what the UI says about the AI against what the code does — disclosure strings quoted, dropped confidence traced, retention claims checked against real payloads.
Whether the data used to train, fine-tune, or retrieve is licensed, clean, and free of the PII and contamination that create legal and quality risk.
Runs inside these playbooks — curated sequences you can launch with one paste: