ai-output-verifier
Check AI output before you trust or use it — where it's likely wrong, what to verify, and how to catch confident-sounding errors. Use when asked can I trust this AI answer, how do I verify what AI told me, fact-check this AI output, or is this AI response reliable. Produces a risk read on the specific output (the claims most likely to be wrong or made up), the parts that need independent verification vs the parts that are low-risk, how to actually verify each, the tells of AI hallucination and overconfidence, and a habit for building verification into your AI use — because AI is confidently wrong often enough that unchecked trust is a real risk.
git clone --depth 1 https://github.com/mohitagw15856/pm-claude-skills /tmp/ai-output-verifier && cp -r /tmp/ai-output-verifier/exports/openclaw/ai-output-verifier ~/.claude/skills/ai-output-verifierSKILL.md
# AI-Output Verifier AI is fluent, confident, and sometimes completely wrong — inventing facts, citations, and details in the same authoritative tone as the correct ones. That confidence is exactly what makes unverified trust dangerous. This checks a specific output: which claims are most likely wrong or fabricated, what genuinely needs independent verification, how to verify it, and the tells of hallucination — so you use AI's speed without inheriting its errors. ## What This Skill Produces - **A risk read of the output** — which specific claims are most likely to be wrong, outdated, or made up (facts, numbers, citations, names, recent events, specifics) - **Verify vs. low-risk split** — what genuinely needs independent checking vs. what's low-stakes or self-evident, so you spend effort where it counts - **How to verify each** — the concrete way to check the high-risk claims (a primary source, a second tool, a domain expert, testing it) - **The hallucination tells** — the signs AI is likely fabricating (oddly specific citations, confident claims about recent/niche facts, plausible-but-unverifiable details) - **A verification habit** — how to build appropriate checking into your AI use by default, scaled to the stakes (trust more for low-stakes, verify hard for high-stakes) ## Required Inputs Ask for these if not provided: - **The output** — the AI response to check (paste it) - **What it's for** — the stakes (a casual question vs. something you'll publish, decide on, or act on) - **The domain** — factual/technical/legal/medical/current-events (some are far higher-risk for AI) - **What you'd do with it** — trust it, act on it, share it, build on it ## Framework: Risk-Rate The Claims, Verify What Matters 1. **Scan for the high-risk claim types.** Specific facts, numbers, dates, names, citations, recent events, and niche/technical specifics are where AI most often invents — flag these. 2. **Split by risk and stakes.** Separate the claims that genuinely need verification (high-risk × high-stakes) from the low-risk or low-stakes ones you can reasonably accept — don't verify everything equally. 3. **Verify against real sources.** For the high-risk claims, check a primary source, a second independent tool, an expert, or by testing — not by asking the same AI "are you sure?" (it'll often just re-confirm). 4. **Watch the hallucination tells.** Oddly precise citations, confident answers about very recent or obscure things, and unverifiable specifics are red flags — treat them as unverified until checked. 5. **Scale trust to stakes.** For low-stakes uses, light verification is fine; for anything you'll publish, decide on, or that could harm if wrong, verify hard. Build this reflex in. ## Output Format ### Verifying: [the output] · for [use/stakes] **High-risk claims (verify these):** [specific facts/numbers/citations/recent/niche → most likely wrong]. **Low-risk (reasonable to accept):** [self-evident / low-stakes parts]. **How to verify each:** [primary source / second tool / expert / test — not re-asking the same AI]. **Hallucination tells present:** [odd-specific citations · confident on recent/niche · unverifiable specifics]. **Trust level for your use:** [light check for low-stakes / verify hard because it's high-stakes]. ## Quality Checks - [ ] Flags the specific high-risk claim types in the output - [ ] Splits what needs verification from what's low-risk, by stakes - [ ] Gives concrete verification methods (not "ask the AI again") - [ ] Names the hallucination/overconfidence tells present - [ ] Scales the recommended trust to the actual stakes ## Anti-Patterns - **"Verify everything"** equally, ignoring stakes. - **Re-asking the same AI** "are you sure?" as verification. - **Trusting confident tone** as a signal of correctness. - **Missing the high-risk claim types** (citations, recent facts, numbers). - **No stakes-based scaling** of how hard to check. ## Example Trigger Phrases - "Can I trust this answer the AI gave me?" - "How do I verify what ChatGPT told me before I use it?" - "Fact-check this AI output — I'm about to publish it." - "Is this AI response reliable enough to act on?" - "What in this AI answer should I double-check?"
Conduct a structured ethical review of an AI or ML feature, model, or product. Use when preparing to deploy an AI system, assessing algorithmic risk, auditing a model for bias, or producing a responsible AI impact assessment. Produces a structured ethics review covering fairness, transparency, privacy, safety, accountability, and societal impact with a risk tier score, pre-deployment checklist, and prioritised mitigations.
Structure AI and ML product decisions with the rigour of any product decision. Use when building AI-powered features, evaluating LLM integrations, designing AI products, or assessing AI readiness. Produces a complete AI product canvas covering problem definition, model approach, data requirements, evaluation framework, UX design, responsible AI checklist, and launch monitoring plan.
Transform feature briefs into structured design briefs that give designers the context they need before opening Figma. Use when asked to write a design brief, create a design handoff, brief a designer on a new feature, or translate a PRD into design requirements. Produces a brief with user goal, emotional context, success criteria, constraints, edge cases, and out-of-scope boundaries.
Design statistically rigorous A/B tests and interpret experiment results. Use when asked to design an experiment, run an A/B test, calculate sample size, interpret test results, or assess whether an experiment was successful. Produces a complete experiment design with hypothesis, sample size, run time, success criteria, and risk flags — or a results interpretation with ship/iterate/kill recommendation.
Synthesises user signals from multiple research sources into a unified, weighted insight brief. Use when you have data from interviews, support tickets, NPS verbatims, app reviews, or sales calls and need to reconcile contradictions, surface the underlying need behind requests, or answer 'what are users really telling us'. Produces ranked insights with confidence ratings, source weighting rationale, divergent signal analysis by user segment, and a research gap identification section.
Structure a product data analysis, metric deep-dive, funnel analysis, or cohort study. Use when asked to analyse product metrics, investigate a drop in conversion, explain a data change to stakeholders, or find the root cause of a metric movement. Produces a structured analysis with question, root cause, confidence level, and recommended action.
Interpret product metrics against goals and surface actionable signals. Use when asked to analyse product health, review key metrics, investigate a performance issue, produce a health report, or assess product-market fit signals. Produces a structured health report with RAG status, trend analysis, root cause hypotheses, and prioritised actions.
Structure a retention analysis, churn investigation, or engagement deep-dive for any product team. Use when asked to analyse user retention, investigate churn, measure DAU/MAU, or build a retention improvement plan. Produces a retention snapshot with root cause hypotheses, aha-moment correlation, and prioritised interventions.