Prompt Regression Suite Skill

Every prompt tweak, model upgrade, and context change is a deploy. This skill designs the suite that runs on each one and answers a single question: did anything that used to work stop working?

What This Skill Produces

• A golden case set: curated inputs with per-case pass criteria
• Scoring methods per case class (exact, rubric-judge, property checks)
• CI gate thresholds — what blocks a merge vs. what warns
• A failure triage protocol — flaky vs. regressed vs. golden-set-wrong

Required Inputs

Ask for (if not already provided):

• The feature and its contract — what the LLM step receives and must produce
• What has broken before (or nearly) — past incidents seed the best cases
• Real traffic examples — 10-20 representative inputs, including ugly ones
• What triggers a run — prompt edits, model bumps, retrieval changes, all of the above?

Building the Golden Set

Compose the set from four deliberate classes — not a random sample:

Class	Purpose	Share
Core paths	The 5-10 inputs that represent most real traffic	~40%
Past failures	Every input that caused a bug, complaint, or incident — permanently	~25%
Edge & adversarial	Empty/huge inputs, wrong language, injection attempts, off-topic	~25%
Canaries	Cases pinned to behaviours you never want to change (refusals, format, tone)	~10%

Keep it small enough to run on every change (30-80 cases beats 500 nobody runs). Version it in git next to the prompt.

Scoring Per Case

Choose the cheapest check that catches the regression: