plzebo

Cost-parity bench for agent loops. Point it at two or more ways of getting the same task done (a cheaper model mix, a different setup, a tool on vs off) and it measures which one delivers the outcome for less, in dollars and in minutes, and whether the gap is real or noise. Runs locally, on your own keys or your own subscription. An agent can call it on itself over MCP.

The headline metric is cost per successful outcome: total metered spend divided by successes, compared across arms with a 95% confidence interval. Time per successful outcome is reported beside it. When the data cannot support a claim, plzebo says so instead of making it.

How it measures

• Arms are whole loops. An arm is a shell command that runs an agent to completion on a task, plus model rewrite rules applied at a local metering gateway. Loop code can be byte-identical across arms while the gateway maps every claude-* call to each arm's model.
• Every model call transits the gateway. Arms receive a placebo key (sk-placebo-...) and a redirected base URL; the gateway swaps in the real key upstream and records provider-reported token usage per call. A call that bypasses the gateway fails at the provider instead of billing invisibly, and a trial with zero metered calls is flagged, never compared.
• Trials are paired and interleaved. K pairs run back to back with a seeded coin flip deciding arm order per pair, so provider drift lands inside pairs. Prompt caching is disabled and every trial carries a unique nonce, so no arm warms a cache for another.
• Grading never trusts the agent's tree. The diff is filtered against a protected-path list (tests, runner config), applied to a pristine checkout, and the test command runs there. Editing the tests fails the trial.
• Statistics refuse to overclaim. Success differences use an exact McNemar test with Holm correction across arms. Parity requires an equivalence test, not just a failed significance test. Cost and time deltas come from a paired bootstrap. Fewer than five successes on either side blocks cost rulings. Timeouts are graded failures with spend included; crashes and provider throttling (429/529) are excluded with spend disclosed. Every refusal is printed as an explicit do-not-claim line.
• Receipts are the artifact. Every run writes a JSON receipt: the spec and its pre-registration hash, every trial's raw rows (tokens, per-call status, changed paths, output tails), the stats inputs, the pricing table version, and a generated estimand statement. plzebo verify receipt.json recomputes every derived number from the raw rows and reports any divergence.

Requirements

Node 22.6 or later (the CLI runs its TypeScript sources through Node type stripping; the launcher supplies the flag on versions that need it). Git, plus whatever your arms and oracle need (for the examples here: python3).

Quick start

npx plzebo run spec.yaml --estimate   # prices the run from prior receipts; spends nothing
npx plzebo run spec.yaml              # runs the experiment
npx plzebo verify <receipt.json>      # recompute a receipt's claims from its raw rows

A spec:

name: sonnet-vs-opus
fixture:
  repo: ./my-fixture-repo        # a git repo; every trial gets a fresh clone
  ref: main
task: "Make the failing tests in this repo pass. Do not edit the tests."
arms:
  sonnet:
    cmd: claude --print "$PLZEBO_TASK" --dangerously-skip-permissions --strict-mcp-config --mcp-config '{"mcpServers":{}}' --model claude-sonnet-5
    auth: passthrough            # runs on your logged-in Claude subscription
    rewrite:
      claude-*: claude-sonnet-5  # pin every call this arm makes, including auxiliary ones
  opus:
    cmd: claude --print "$PLZEBO_TASK" --dangerously-skip-permissions --strict-mcp-config --mcp-config '{"mcpServers":{}}' --model claude-opus-4-7
    auth: passthrough
    rewrite:
      claude-*: claude-opus-4-7
baseline: opus
oracle:
  cmd: [python3, -m, unittest, discover, 

... [View full README on GitHub](https://github.com/wkoverfield/plzebo#readme)