Golden outputs for a skill whose output is never identical twice
Model output varies run to run, so exact-match snapshots fail. Check what must hold instead: structure, required elements, banned patterns, token use.
Snapshot tests assume the same input gives the same output. Skills do not work that way. Two runs can both be right and share few identical lines. So test what must be true, not what the text says.
Properties instead of snapshots
| For a design skill, check | How |
|---|---|
| Uses the tokens, not raw colours | No hex values outside the token file |
| One accent per screen | Count uses of the accent token |
| Numbers are tabular | tabular-nums present where numbers render |
| Nothing on the banned list | No gradient text on body copy, no emoji icons |
Keep each case small
A golden case is a task prompt plus five or so assertions. Write them from the skill's hard rules, which is one more reason to keep those rules checkable, as in how long SKILL.md should be. For judgement calls a script cannot make, use a short human checklist instead: a rubric for skill output.
Questions
Why do snapshot tests fail for skill output?
Because generated text and code vary in wording and order between runs even when they are equally correct. Exact matching turns normal variation into failures.
What should a golden test check instead?
Properties that must always hold: required files or sections exist, specific values appear, banned patterns do not, and the result parses or builds.
How many golden cases do I need?
A handful of representative tasks is enough to start, each with a short list of properties. Add a case when a real failure slips through.