MCP server vetting for agents

Your agent trusts every
tool you hand it.

Adding an MCP server to Claude Code is one line of JSON. After that line, a server you have never audited can describe its own tools to your agent, and your agent will believe the description. There is no grade, no gate, and no record of what you let in.

Graded by being used, not by being read. Do not trust the letter. Replay the tape.

3 scored layers 6 probes 3 runs each, temperature 0 full transcripts kept

How a grade is built

Three layers, and one of them is honest about not running

Most scanners read a server's manifest and score its conformance. That catches a missing field. It does not catch the thing that actually costs you: an agent picking the wrong tool because two descriptions say nearly the same thing.

30weight

Static conformance

Wraps the mcpscore CLI: protocol version, TLS, schema validity, annotations, pagination behaviour. Its raw output is a score over a moving denominator, because rules get skipped per server. We normalise to a percentage before anything is compared. A server scoring 78 out of 91 is worse than one scoring 64 out of 73, and only the normalised number shows it.

50weight

Behavioural probes

A real agent, a pinned model, temperature 0, three runs per probe. The probes drive the server the way your agent would and score what happens. This is the half of the grade that reading cannot produce.

20weight

Guidance delta

The same cold task, re-run with the drafted recipe in hand. It measures how much of the gap to a perfect run the recipe closes, not the raw point difference, so a server that was already strong is not punished for having little room left to improve.

A layer that did not run is not a zero. If the guidance delta was never measured, its weight is removed and the other two renormalise to 37.5 and 62.5. Scoring an unmeasured layer as zero would drag every honest partial audit into a failing band, which is a way of lying with arithmetic.

A grade is relative to the model that produced it. The model id is stored on the audit, printed in the report, and rendered on the badge. Grades from different models are not comparable and the report says so.

A · 85 and above B · 70 to 84 C · 50 to 69 F · below 50, or any hard fail

What gets measured

Six probes, each answering one question

Two of them do not always run. Ambiguity only fires when tool descriptions genuinely overlap, and Chain is skipped under three tools. A probe that did not apply is recorded with its reason and left out of the average, rather than counted as a failure the server did not earn.

Probe 1

Handshake and inventory

Connect, list tools, lint the schemas. A server with no tools exits here without spending anything on probes.

Is there anything here at all?

Probe 2

Cold open

A fresh agent, the server's own descriptions, and a task built from what you said you needed it for. Scored on first-try tool choice, first-try schema validity, completion, and how much wandering it took.

Can an agent that has never seen this succeed on the first try?

Probe 3

Ambiguity gauntlet

Fires only when two tools overlap. The task is worded in the wrong tool's vocabulary, so only a description that draws a real line will save it.

Do these two descriptions actually distinguish themselves?

Probe 4

Bad input recovery

Omit a required parameter on purpose, then measure the error message. What is graded here is the message, not the agent. "Invalid input" and "missing required 'repoName' (string, e.g. facebook/react)" are the same failure and completely different products.

Can an agent fix itself from this error in two turns?

Probe 5

Chain test

Two steps, where the second must consume a real value the first returned. Servers that look fine tool by tool fall over here, when an id comes back under a different name than the next tool asks for.

Do these outputs compose, or only look like they should?

Probe 6

Injection sniff

Scan-only. Reads every description and schema string an agent would see before choosing a tool, looking for content shaped like an instruction rather than documentation. It never calls a tool: we do not execute a server to find out whether it is hostile.

Is this documentation, or is it giving my agent orders?

Two findings cap a grade at F outright: injection-shaped content in the advertised strings, and a transport that is not TLS. A hard fail overrides every other score, and a hard-failed server with a 90 percent static result is still an F.

Because an F is a public accusation about somebody else's software, the sniffer is deliberately conservative and every hit records the pattern that matched plus the offending text, verbatim, so the claim can be checked rather than taken on trust.

The part that runs on your machine

A gate that fails closed

The grading service is interesting. The hook is the part that protects you. It runs before every MCP tool call, reads a local file, and blocks anything it cannot vouch for.

claude code · PreToolUse · matcher mcp__.*

> summarise what the acme-helper server can tell us about this repo

· calling mcp__acme_helper__search
BLOCKED by doorman

  doorman: 'acme_helper' is UNKNOWN. Blocking until it has been graded.

  An ungraded MCP server is not a trusted one. Nothing about
  'mcp__acme_helper__search' has been verified: not its tool
  descriptions, not its error handling, not whether its descriptions
  contain instructions aimed at you.

  To grade it:   /vet <server-url>

> /vet https://acme-helper.example.com/mcp "search our internal docs"

  queued · claimed by runner · mcpscore 64/73 = 87.67%
  probe cold_open 55 · ambiguity 33 · bad_input 45 · chain skipped
  VERDICT: C (61.4/100)

  Why: agent reached for 'search_all' when 'search_docs' was the fit
       error for missing 'query' does not name the field
       'search_all' and 'search_docs' overlap 41%

  Recipe drafted: 3 rules. Proposed registry diff below.

No network: The gate contains no network command at all. A gate that asks a server for permission is offline the moment the network is, and offline would mean allow.
No dependencies: Bash builtins and coreutils only. No jq, no node, no python. A gate that fails to start is a gate that fails open.
Fails closed: Unparseable input, missing registry, unknown server, unreadable file: all block. The default answer is no.
Exit 2, never exit 1: Only exit 2 blocks a tool call. Exit 1 is treated as a script error and the call proceeds. Every refusal path exits 2, and a test asserts the file contains no exit 1.
One tool, one job: The doorman subagent holds Read and exactly one MCP tool. The agent that decides what to trust does not also carry capabilities an untrusted server could talk it into using.
Humans own the list: Nothing writes to the allowlist automatically. The poller reports what changed and refuses to allowlist a hard fail even when the service says allow.

Live viewport

Queue an audit, watch the ledger

Grading does not happen in the browser, and it does not happen in the Worker either. The static layer shells out to a Python CLI, which a Cloudflare Worker cannot do, so the request goes into a queue and a probe runner claims it. The ledger below polls every two seconds. There is no websocket here on purpose: streaming would need Durable Objects, and this runs on the free tier.

Request an audit

The URL must be https. A plaintext transport is a hard fail, so it is refused before any work is queued.

Ledger

--:--:--idlewaiting for the scorecard

Not connected to a scorecard yet.

What you get back

Four artifacts, one hash over all of them

Every audit produces the same bundle, and the bundle is hashed with a canonical serialisation so two runs over the same evidence agree on any machine. Change a transcript and the hash changes. That is what makes the anchor worth anything.

grade.json: The machine record: score, band, per-layer breakdown with effective weights, per-probe scores, hard fail, pinned model, mcpscore version.
report.md: One page, and the cap is enforced rather than hoped for. It drops optional sections until it fits and says so when it had to. Carries an ungraded provenance section that describes what actually ran, so a static-only audit never claims probe runs it did not do.
recipe.md: Usage rules drafted from observed failures, each traced to the probe that produced it. If nothing failed, no rules are written. Nothing here is invented.
transcripts.jsonl: Every turn of every run of every probe. Never truncated, never edited, never sampled. This is the tape.

Your agent trusts everytool you hand it.