This skill measures whether a Python test suite is a good check. It asks whether the suite would
notice broken code, whether it is fast and stable enough to run on every change, and which tests
cost time without adding detection. A pinned runner does the measuring; the skill reports and
does not change tests. Upstream source:
timini/agentic-verification.
@agentic-verify instead to check a single change.Arguments: $ARGUMENTS. The first word is the mode: quick (default) or deep. Anything
else goes to the runner. To check a single change rather than the whole suite, use theagentic-verify skill instead.
An agent can only be trusted as far as the checks it runs. This skill measures whether a test
suite is a good check: whether it would notice broken code, whether it is fast and stable enough
to run on every change, and which tests cost time without adding detection.
AV='uvx --from git+https://github.com/timini/agentic-verification@07608b3b9e4e359bc0b93b3eca94bb78d67e369e#subdirectory=runner agentic-verification'
$AV quick|deep <path> [--budget-minutes N]
$AV prune-check <path> --tests-file <file> --from-run <run-dir>
$AV replay --run <run-dir> --mutant <id>
The runner works in a disposable copy and writes only to .test-suite-audit/ in the project;
suggest adding that to .gitignore. It needs Python 3.12+ and pytest-cov in the project's
environment, and pytest-randomly for order checks. Use a Bash timeout of the budget plus 2
minutes. The last stdout line is JSON with the report paths.
never upgrade a label (an inferred finding stays inferred).
user asks for a specific change after reading the report.
are not redundant together. Only a removal set that prune-check did not refute may be
presented as removable, and even then as "not refuted", not "safe".
equivalent yourself; say it needs human review.
the study evidence instead of just declining.
test-suite-audit.toml; ifabsent, the runner discovers source packages, test directories and the interpreter
(.venv/bin/python, then python3). If discovery looks wrong (for example a monorepo),
write a test-suite-audit.toml only after asking. Keys: python, source, tests,pytest_args, env, workers, protected (node ID prefixes or marker names of tests that
must never be proposed for removal). If pytest-cov is missing, say coverage will be "not
assessed" and offer the install command; do not install anything without asking.
overlap, no mutants. Deep is 30 minutes by default: three file-order and three random-order
runs, coverage, every deletion mutant, then a seeded sample of classical mutants until the
budget ends. In our benchmark, 12 to 14 minutes completed deep audits of projects up to about
3,300 executable lines; dateutil (about 3,600) and click (about 5,000) needed more.
report.json (full evidence) and report.md. Seereading the report for statuses, labels and the detection rate.
study such sets covered about 80% of each suite and every one lost kills on held-out mutants.
Run prune-check on them with--from-run <run-dir>, so the check uses mutants the set was not chosen from. Give it at
least 10 minutes. Report its verdict: refuted (with the kills or lines lost) or not refuted
(with the number of killed mutants checked and the upper bound it gives).
If it is refuted, keep the tests it names as sole killers, and say the rest is still
unproven. Never present a refuted set as removable.
evidence and next validation step:
weak tests, uncovered code, and whether property-based tests exist.
slowest and setup-heavy tests, intermittent and order-dependent tests.
Overlap pairs are listed only as things to look at.
Give the reproduction command for at least one mutant finding. List what was not assessed.
Link report.md.
original and fails on a confirmed mutant (the agentic-verify skill does this for changed code);
removing a not-refuted set in its own commit; or a deeper run with a larger budget.
If the run is partial (budget or interruption), say so first and explain which steps are missing.
AV='uvx --from git+https://github.com/timini/agentic-verification@07608b3b9e4e359bc0b93b3eca94bb78d67e369e#subdirectory=runner agentic-verification'
$AV quick . --budget-minutes 5
$AV deep . --budget-minutes 30
$AV prune-check . --tests-file removal.txt --from-run .test-suite-audit/<run-dir>
pytest-cov; order checks need pytest-randomly.not refuted removal set is not proof that removal is safe.uvx from GitHub at tag v0.3.0 (commit 07608b3b9e4e359bc0b93b3eca94bb78d67e369e). Review it before first use..test-suite-audit/.@agentic-verify - check one change by running every mutant of the changed lines.A temporary copy is not a security sandbox: pytest, fixtures, plugins, conftest and dependency hooks execute code and may access network, credentials and external services. Inspect the pinned runner and target test setup first; use a low-privilege isolated environment with no production credentials and controlled network access. Scope secrets and environment inheritance before running mutation/replay commands. A pin identifies code, not a completed safety review.