Testing Claude skills in CI on every change
Structural checks are free and belong on every commit; trigger runs cost tokens and suit a nightly job. How to split skill testing across CI sensibly.
Skill testing splits into two kinds of check with very different costs. Put each in the right place and CI stays fast and cheap.
| Check | Cost | When |
|---|---|---|
| Structure: frontmatter, names, referenced files | Free | Every commit |
| Script unit tests | Free | Every commit |
| Trigger runs against a prompt set | Model tokens | Nightly, or when the description changes |
| Full output review in a container | Tokens and time | Before a release |
The every-commit job
Our packing script is this job: it refuses to build if a skill's name does not match its folder, its description is missing or too long, a referenced file is missing, or any file contains an em dash. It runs in the normal build, so a broken skill cannot ship.
The nightly job
Run the prompt set in a clean container, as in a Docker harness, record both trigger numbers, and compare with the last run. A drop is worth a look even if you changed nothing; see regression testing after a Claude update.
Questions
What can be checked in CI without calling a model?
Frontmatter, the name matching the folder, description length, referenced files existing, banned characters, and whether scripts pass their own tests.
Should trigger tests run on every commit?
Usually not. They call the model and cost money. Run them nightly, on a schedule, or when SKILL.md or its description changes.
Where do I keep the API key for CI runs?
In your CI provider's encrypted secrets, passed as an environment variable at run time. Never in the repository.