MCP Scorecard
Deterministic, CI-first quality scorecard for MCP servers.
MCP Scorecard is an open-source infrastructure tool for reviewing MCP servers before they enter
real workflows. It launches a server locally over stdio, discovers its tools, applies a
deterministic ruleset, and produces reviewable scores and findings across:
conformancesecurityergonomicsmetadata
The output is built for CI: stable terminal summaries, a machine-readable JSON scorecard report,
and SARIF for code-scanning systems.
This project is intentionally not an AI wrapper. It does not depend on LLM scoring, hidden
judgment, or hosted analysis. The goal is a repeatable, auditable baseline that engineering teams
can gate on in pull requests and release pipelines.
Quick value:
- run locally against a real MCP server
- fail CI below a deterministic threshold
- export JSON and SARIF for automation
- review risky MCP surfaces deterministically
What This Is
MCP Scorecard is a deterministic quality scorecard for MCP servers.
It is designed for cases where teams need to answer questions like:
- Is this server surface reviewable before we adopt it?
- Does it expose capabilities that deserve extra scrutiny in CI?
- Are the tool names, descriptions, and schemas clear enough for human review?
- Can we produce a stable machine-readable report for automation and policy?
Today the tool focuses on local stdio MCP servers and a deterministic scoring model that is easy
to explain, test, and version.
Why This Exists
MCP servers are infrastructure. They define callable tool surfaces that agents, runtimes, and
automation can invoke. That means they should be reviewed with the same seriousness as other
integration boundaries.
In practice, teams often evaluate MCP servers ad hoc:
- descriptions are vague
- schemas are weak or underconstrained
- high-risk capabilities are discovered late
- CI has no consistent baseline
MCP Scorecard turns that first-line review into a deterministic contract:
- run it locally
- run it in CI
- inspect category scores and findings
- export JSON and SARIF
- keep the result reviewable over time
What It Checks
The current score model uses four explicit buckets.
Conformance here means deterministic interface-level conformance and schema reviewability checks,
not full protocol certification.
Conformance
Checks whether the server surface is structurally well-formed and reviewable as an MCP interface.
Examples:
- duplicate tool names
- missing schema type
- arbitrary top-level properties
- critical input fields not marked required
Security
Checks for exposed capabilities that materially increase blast radius or deserve explicit review.
Examples:
- command execution
- filesystem mutation
- network and HTTP request primitives
- download-and-execute patterns
Ergonomics
Checks whether the server surface is understandable enough for humans and automation to review
without guessing.
Examples:
- overly generic tool names
- vague descriptions
- weak input schemas
- filesystem mutation tools without visible scope hints
Metadata
Checks whether basic descriptive metadata is present and whether destructive behavior is made easy
to spot.
Examples:
- missing tool descriptions
- descriptions that explicitly advertise broad destructive access
What It Does Not Promis
... View full README on GitHub
