MCP server vetting for agents

Your agent trusts every tool you hand it.

Adding an MCP server to Claude Code is one line of JSON. After that line, a server you have never audited describes its own tools to your agent, and your agent believes the description.

Graded by being used, not by being read. Do not trust the letter. Replay the tape.

DeepWiki live

A 85.71out of 100

Static 85.7%

Behavioral not measured

Guidance not measured

claude-sonnet-5 2cc2178dbe66

Every scanner reads the manifest. None of them use the server.

Reading catches a missing field or a stale protocol version. It does not catch the thing that actually costs you: an agent picking the wrong tool because two descriptions say nearly the same thing, or burning three turns on an error that never names the field it is complaining about.

How a grade is built

Three layers, and one of them is honest about not running

Weighted 30, 50 and 20. The layer that only reads the manifest is deliberately the smallest of the three.

30Static

What the manifest claims

Wraps the mcpscore CLI: protocol version, TLS, schema validity, annotations, pagination. Its raw output is a score over a moving denominator, because rules get skipped per server, so it is normalised before anything is compared. A server scoring 78 out of 91 is worse than one scoring 64 out of 73, and only the normalised number shows it.

50Behavioral

What happens when an agent uses it

A real agent, a pinned model, temperature 0, three runs per probe. The probes drive the server the way your agent would and score what actually happens. This is the half of the grade that reading cannot produce, and it is the reason the rest of this exists.

20Guidance

How much a recipe rescues it

The same cold task, re-run with the drafted recipe in hand. Scored as recovered headroom rather than raw point difference, so a server that was already strong is not punished for having little room left to improve.

A layer that did not run is not a zero. If the guidance delta was never measured, its weight is removed and the other two renormalise to 37.5 and 62.5. Scoring an unmeasured layer as zero would drag every honest partial audit into a failing band, which is a way of lying with arithmetic.

A grade is relative to the model that produced it. The model id is stored on the audit, printed in the report, and rendered on the badge. Grades from different models are not comparable, and the report says so rather than leaving you to find out.

A · 85 and above B · 70 to 84 C · 50 to 69 F · below 50, or any hard fail

What gets measured

Six probes, each answering one question

Two of them do not always run. A probe that did not apply is recorded with its reason and left out of the average, rather than counted as a failure the server never earned.

Handshake and inventory

Connect, list tools, lint the schemas. A server with no tools exits here without spending anything on probes.

Is there anything here at all?

Cold open

A fresh agent, the server's own descriptions, and a task built from what you said you needed it for. Scored on first-try tool choice, first-try schema validity, completion, and how much wandering it took.

Can an agent that has never seen this succeed on the first try?

03conditional

Ambiguity gauntlet

Fires only when two tool descriptions genuinely overlap. The task is worded in the wrong tool's vocabulary, so only a description that draws a real line will save it.

Do these two descriptions actually distinguish themselves?

Bad input recovery

Omit a required parameter on purpose, then measure the error. What is graded here is the message, not the agent. "Invalid input" and "missing required repoName (string, e.g. facebook/react)" are the same failure and completely different products.

Can an agent fix itself from this error in two turns?

05conditional

Chain test

Two steps, where the second must consume a real value the first returned. Servers that look fine tool by tool fall over here, when an id comes back under a different name than the next tool asks for. Skipped under three tools.

Do these outputs compose, or only look like they should?

Injection sniff

Scan-only. Reads every description and schema string an agent sees before choosing a tool, looking for content shaped like an instruction rather than documentation. It never calls a tool: we do not execute a server to find out whether it is hostile.

Is this documentation, or is it giving my agent orders?

Two findings cap a grade at F outright: injection-shaped content in the advertised strings, and a transport that is not TLS. A hard fail overrides everything else, and a hard-failed server with a 90 percent static result is still an F.

Because an F is a public accusation about somebody else's software, the sniffer is deliberately conservative, and every hit records both the pattern that matched and the offending text, verbatim, so the claim can be checked rather than taken on trust.

The part that runs on your machine

A gate that fails closed

The grading service is the interesting half. The hook is the half that protects you. It runs before every MCP tool call, reads a local file, and blocks anything it cannot vouch for.

claude code · PreToolUse · matcher mcp__.*

> summarise what the acme-helper server can tell us about this repo

· calling mcp__acme_helper__search
BLOCKED by doorman

  doorman: 'acme_helper' is UNKNOWN. Blocking until it has been graded.

  An ungraded MCP server is not a trusted one. Nothing about
  'mcp__acme_helper__search' has been verified: not its tool
  descriptions, not its error handling, not whether its descriptions
  contain instructions aimed at you.

  To grade it:   /vet <server-url>

> /vet https://acme-helper.example.com/mcp "search our internal docs"

  queued · claimed by runner · mcpscore 64/73 = 87.67%
  probe cold_open 55 · ambiguity 33 · bad_input 45 · chain skipped
  VERDICT: C (61.4/100)

  Why: agent reached for 'search_all' when 'search_docs' was the fit
       error for missing 'query' does not name the field
       'search_all' and 'search_docs' overlap 41%

  Recipe drafted: 3 rules. Proposed registry diff below.

No network: The gate contains no network command at all. A gate that asks a server for permission is offline the moment the network is, and offline would have to mean allow.
No dependencies: Bash builtins and coreutils only. No jq, no node, no python. A gate that fails to start is a gate that fails open.
Fails closed: Unparseable input, missing registry, unknown server, unreadable file: all block. The default answer is no.
Exit 2, never exit 1: Only exit 2 blocks a tool call. Exit 1 is treated as a script error and the call proceeds. Every refusal path exits 2, and a test asserts the file contains no exit 1.
One tool, one job: The doorman subagent holds Read and exactly one MCP tool. The agent that decides what to trust does not also carry capabilities an untrusted server could talk it into using.
Humans own the list: Nothing writes to the allowlist automatically. The poller reports what changed and refuses to allowlist a hard fail even when the service says allow.

Live viewport

Queue an audit, watch the ledger

Grading does not happen in the browser, and it does not happen in the Worker either. The static layer shells out to a Python CLI, which a Cloudflare Worker cannot do, so the request goes into a queue and a probe runner claims it. The ledger polls every two seconds. There is no websocket here on purpose: streaming would need Durable Objects, and this runs on the free tier.

Request an audit

The URL must be https. A plaintext transport is a hard fail, so it is refused before any work is queued.

Ledger

Connecting to the scorecard.

Polling every two seconds.

What you get back

Four artifacts, one hash over all of them

Every audit produces the same bundle, hashed with a canonical serialisation so two runs over the same evidence agree on any machine. Change a transcript and the hash changes. That is the only thing that makes an anchor worth anything.

grade.json: The machine record: score, band, per-layer breakdown with effective weights, per-probe scores, hard fail, pinned model, mcpscore version.
report.md: One page, and the cap is enforced rather than hoped for. It drops optional sections until it fits and says so when it had to. Its provenance section describes what actually ran, so a static-only audit never claims probe runs it did not do.
recipe.md: Usage rules drafted from observed failures, each traced to the probe that produced it. If nothing failed, no rules are written. Nothing here is invented.
transcripts.jsonl: Every turn of every run of every probe. Never truncated, never edited, never sampled. This is the tape.