Claude updated and my skill changed: regression testing skills
A skill that worked last month can behave differently after Claude or the model updates. How to notice early, confirm the cause, and fix the rules.
Skills sit on top of a system that changes. A skill that fired reliably and followed its rules last month can drift after an update, with no change to its files. That is the maintenance buyers are actually paying for, and it needs a routine.
Notice early
- Keep the prompt set and last results next to the skill.
- Rerun it after any Claude Code update you install, or on a schedule.
- Compare both trigger numbers with the previous run, as in measuring trigger accuracy.
Confirm the cause
Hold everything else still: same installed skills, same project, same prompts. If the numbers moved and nothing but the version changed, you have a regression to fix rather than a flaky run. If results vary between identical runs, read flaky skill results first.
Fix and record
Adjust the description when triggering shifted, or the specific rule when behaviour did. Rerun, and write the version, date and change into the skill's changelog so the next person knows why the wording is what it is.
Questions
Can a Claude update change how my skill behaves?
Yes. Updates to Claude Code or the underlying model can change how descriptions are matched and how instructions are followed. Your skill file did not change, but its results can.
How do I know an update is the cause?
Rerun the same prompt set you ran before, with the same skills installed, and compare. If nothing else changed and the numbers moved, the update is the likely cause.
What do I usually need to fix?
Often the description, when triggering shifts, or a rule that is now followed too literally or too loosely. Small wording changes usually restore it.