AI Eval Plan Skill

You can't improve an AI feature you can't measure, and "it looks good in the demo" is not measurement. This skill produces an evaluation plan that turns a fuzzy quality goal into a repeatable, gated test — so a prompt change that quietly makes outputs worse can't ship.

Required Inputs

Ask for these only if they aren't already provided:

• The feature & task — what the model does and what "good output" means to a user.
• Failure modes that matter — what bad looks like (hallucination, wrong format, unsafe, off-tone, too slow).
• Available data — any real examples, logs, or labelled cases; or note there are none yet.
• Who judges quality — automated checks, an LLM judge, human raters, or a mix.
• The decision this gates — ship/no-ship, model selection, or prompt iteration.

Output Format

Eval Plan: [feature]

1. What we're measuring — the task, and a one-line definition of a good vs. bad response.

2. Eval dataset

• Cases: how many, where they come from (real logs > synthetic), and how they're split (smoke set vs. full set).
• Coverage: the slices/scenarios that must be represented (edge cases, adversarial, each major input type).
• Golden answers / references: present or not, and how they were created.

3. Metrics & rubric

• Per-dimension scores — define each dimension (e.g. correctness, grounding, format, safety, tone) on an explicit 1–5 rubric with anchor descriptions, not vibes.
• — deterministic assertions first (valid JSON, contains required fields, no PII, latency budget).