AgentMark

Git-native AI agents.
_{Prompts and datasets in your repo. Evals in CI. Traces in your OTEL backend.}

Homepage · Docs · Cloud

AgentMark is an open-source platform for building reliable AI agents. Define prompts in Markdown, run them with the SDK you already use, evaluate against datasets locally or in CI, and trace every call with OpenTelemetry.

• Prompt management. Prompts are .prompt.mdx files with type-safe inputs, tools, structured outputs, conditionals, loops, and reusable components. They live in your repo, get reviewed in PRs, and roll back with git revert.
• Datasets. JSONL files in your repo. Each row is a line, so git diffs show exactly which test cases changed.
• Evaluations. Run prompts over datasets with built-in or custom evaluators. Use the CLI or call the SDK from your own pipelines. Block merges on regressions, the way tests do.
• Tracing. Every LLM call emits an OpenTelemetry span. Inspect traces in the local dev UI, or forward them to AgentMark Cloud (or any OTEL backend) for search, dashboards, and alerts in production.
• Type safety. Auto-generated TypeScript types from your prompts. JSON Schema validation in your editor.

Quick start

Requires: Node.js 18 or newer.

# Scaffold a new project (interactive: picks your language)
npm create agentmark@latest my-agents
cd my-agents

# Start the dev server (API + trace UI + hot reload)
agentmark dev

# Run a single prompt
agentmark run-prompt agentmark/my-prompt.prompt.mdx

# Run an experiment against a dataset
agentmark run-experiment agentmark/my-prompt.prompt.mdx

About five minutes from npm create to a traced prompt running locally (assuming you have an LLM API key set up).

What a prompt looks like

---
name: customer-support-agent
text_config:
  model_name: anthropic/claude-sonnet-4-20250514
  max_calls: 2
  tools:
    - search_knowledgebase
test_settings:
  props:
    customer_question: "How long does shipping take?"
input_schema:
  type: object
  properties:
    customer_question:
      type: string
  required: [customer_question]
---

<System>
You are a helpful customer service agent. Use the search_knowledgebase tool
when customers ask about shipping, warranty, or returns.
</System>

<User>{props.customer_question}</User>

The frontmatter declares which tools the prompt may call; the implementations live in your code, resolved where you call the model. See Tools and agents.

Run it:

agentmark run-prompt customer-support.prompt.mdx

The prompt is version-controlled, type-checked, and traced. The same file works with any SDK — the Vercel AI SDK, the raw OpenAI or Anthropic client, Pydantic AI, or your own bespoke client. AgentMark renders the prompt to a neutral { messages, ...config } shape; your SDK makes the call.

Why git-native

Most AI tooling treats the dashboard as the primary workspace. Prompts are rows in a database. Edits happen in a b

... View full README on GitHub