Test the server that users will run. A schema review or a passing unit test is
not runtime evidence.
This skill complements security review. It focuses on protocol behavior,
published-contract drift, transport correctness, and reproducible release
evidence.
initialize, notifications/initialized, discovery, and invocation inthe same session. A new process is a new STDIO session.
as a contract that must match it.
"looks correct."
sandbox, or stop and name the missing safe test environment.
run.
Identify:
Prefer repository-native commands. Inspect package.json, pyproject.toml,Makefile, CI workflows, and contributor instructions before inventing a test
harness.
Build the candidate and start the documented entry point with test-safe
configuration. Capture:
For STDIO, stdout is protocol-only. Logs, banners, and stack traces belong on
stderr. For HTTP transports, record the status, relevant MCP headers, and
session identifier handling without printing credentials.
If the server cannot start from its documented instructions, report that as a
release failure and preserve the startup error verbatim.
Run this sequence through a real MCP client or the repository's integration
harness:
initialize with a protocol version the server claims to support.notifications/initialized.ping.tools/listresources/listresources/templates/listprompts/listcapability class.
Do not send post-initialization requests through separate one-shot processes.
That accidentally tests several incomplete sessions instead of one valid
session.
Build four inventories from current evidence:
| Surface | Evidence |
|---|---|
| Source | Registered tool, resource, template, and prompt definitions |
| Runtime | Results from the live discovery methods |
| Generated metadata | Catalogs, manifests, or generated indexes |
| Documentation | README, reference pages, and install output |
Compare by stable identifier. Report:
Regenerate derived files with the repository's own build command, then fail if
the working tree still contains unexplained generated changes.
For every discovered item, verify the runtime definition against its source:
inputSchema defines types, required fields, enums, and bounds where needed.outputSchema when one is published.traces.
prompts/get returns usable messages for valid arguments.At minimum, probe:
Verify that each response has the correct request ID, a useful error message,
and no successful side effect. For STDIO, also confirm every stdout line is a
complete protocol message and a healthy session leaves stderr clean unless the
server explicitly documents diagnostic output.
When the project publishes an install command:
An install string that was only inspected is unverified.
Use this format:
# MCP Release QA
Candidate: [commit]
Transport: [STDIO | Streamable HTTP | SSE]
Verdict: PASS | PASS WITH CAVEATS | FAIL
## Commands and results
- `[exact command]` — [exit status and result]
## Session transcript
- initialize: [result]
- discovery: [result]
- representative calls: [result]
- negative paths: [result]
## Parity
| Identifier | Source | Runtime | Metadata | Docs | Result |
|---|---|---|---|---|---|
## Findings
| Severity | Evidence | Impact | Narrowest fix |
|---|---|---|---|
## Missing evidence
- [check that could not run and why]
Use FAIL for a server that cannot start, complete a valid session, keep the
transport parseable, or safely reject invalid input. Use PASS WITH CAVEATS
only for bounded documentation or metadata drift that does not misrepresent a
dangerous capability. Otherwise use PASS.