Model Context Protocol (MCP) Agent Frameworks Demo & Benchmarking Platform

This repository demonstrates LLM Agents using tools from Model Context Protocol (MCP) servers with several frameworks:

• Google Agent Development Kit (ADK)
• LangGraph Agents
• OpenAI Agents
• Pydantic-AI Agents

Repository Structure

• Agent with a single MCP server - Learning examples and basic patterns
• Agent with multiple MCP servers - Advanced usage with MCP server coordination
• Evaluation suite - Comprehensive benchmarking tools
  • Evaluation Dashboard: Interactive Streamlit UI for model comparison
  • Multi-Model Benchmarking: Parallel/sequential evaluation across multiple LLMs
  • Rich Metrics: Usage analysis, cost comparison, and performance leaderboards

The repo also includes Python MCP Servers:

• example_server.py based on MCP Python SDK Quickstart - Modified to include a datetime tool and run as a server invoked by Agents
• mermaid_validator.py - Mermaid diagram validation server using mermaid-cli

Tracing is done through Pydantic Logfire.

Quickstart

1. cp .env.example .env

• Add GEMINI_API_KEY and/or OPENAI_API_KEY
  • Individual scripts can be adjusted to use models from any provider supported by the specific framework
• [Optional] Add LOGFIRE_TOKEN to visualise evaluations in Logfire web ui

2. Test an Agent framework script e.g.:

• uv run agents_mcp_usage/basic_mcp/basic_mcp_use/pydantic_mcp.py

  • Requires GEMINI_API_KEY by default

• uv run agents_mcp_usage/basic_mcp/basic_mcp_use/oai-agent_mcp.py

  • Requires OPENAI_API_KEY by default

3. Run Dockerised single-model benchmark (isolated dependencies)

• make benchmark MODEL="openai:gpt-5.2 (none)" RUNS=5 PARALLEL=1

Check console, Logfire, or the ADK web UI for output

Project Overview

This project aims to teach:

1. How to use MCP with multiple LLM Agent frameworks
   • Agent using a single MCP server (basic_mcp)
   • Agent using multiple MCP servers (multi_mcp)
2. How to see traces LLM Agents with Logfire
3. How to evaluate LLMs with PydanticAI evals

Repository Structure

• agents_mcp_usage/basic_mcp/ - Single MCP server integration examples

  • basic_mcp_use/ - Contains basic examples of single MCP usage:
    • adk_mcp.py - Example of using MCP with Google's Agent Development Kit (ADK 1.3.0)
    • langgraph_mcp.py - Example of using MCP with LangGraph
    • oai-agent_mcp.py - Example of using MCP with OpenAI Agents
    • pydantic_mcp.py - Example of using MCP with Pydantic-AI
    • pydantic_mcp_factory.py - Example demonstrating the model factory pattern for multi-provider support

• agents_mcp_usage/multi_mcp/ - Advanced multi-MCP server integration examples

  • multi_mcp_use/ - Contains examples of using multiple MCP servers simultaneously:
    • pydantic_mcp.py - Example of using multiple MCP servers with Pydantic-AI Agent

• agents_mcp_usage/evaluations/ - Evaluation modules for benchmarking

  • mermaid_evals/ - Comprehensive evaluation suite for mermaid diagram fixing tasks
    • evals_pydantic_mcp.py - Core evaluation module for single-model testing
    • run_multi_evals.py - Multi-model benchmarking with parallel execution
    • merbench_ui.py - Interactive dashboard for result visualization

• Demo Python MCP Servers

  • mcp_servers/example_server.py - Simple MCP server that runs locally, i

... View full README on GitHub