Spark SQL MCP Server

An MCP server that enables AI assistants to query Spark SQL clusters via the Thrift/HiveServer2 protocol.

Works with any HiveServer2-compatible system: Apache Spark, AWS EMR, Hive, Impala, Presto.

Features

• Query Spark SQL — Execute read-only SQL queries against your Spark cluster
• Schema Discovery — List databases, tables, and describe table structures
• Multiple Auth Methods — NONE, LDAP, NOSASL, CUSTOM, and Kerberos authentication
• EMR Compatible — Works with AWS EMR clusters out of the box
• Read-Only Enforcement — Only SELECT, SHOW, DESCRIBE, EXPLAIN, and WITH statements are allowed
• Safety Defaults — Automatic LIMIT clause on unbounded queries, sanitized error messages

Installation

pip install spark-sql-mcp-server

Or run directly with uvx:

uvx spark-sql-mcp-server

Quick Start

1. Set Environment Variables

export SPARK_HOST="your-emr-master-node.amazonaws.com"
export SPARK_PORT="10000"        # default
export SPARK_DATABASE="default"  # default
export SPARK_AUTH="NONE"         # NONE | LDAP | KERBEROS | CUSTOM | NOSASL

2. Add to Claude Code

Global (all projects) — add to ~/.claude.json under your project's mcpServers:

{
  "mcpServers": {
    "spark-sql": {
      "command": "uvx",
      "args": ["spark-sql-mcp-server"],
      "env": {
        "SPARK_HOST": "your-emr-master-node.amazonaws.com",
        "SPARK_PORT": "10000",
        "SPARK_AUTH": "NONE"
      }
    }
  }
}

Project-level — add to .claude/mcp.json in your repo:

{
  "mcpServers": {
    "spark-sql": {
      "command": "uvx",
      "args": ["spark-sql-mcp-server"],
      "env": {
        "SPARK_HOST": "your-emr-master-node.amazonaws.com",
        "SPARK_PORT": "10000",
        "SPARK_AUTH": "NONE"
      }
    }
  }
}

3. Add to Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "spark-sql": {
      "command": "uvx",
      "args": ["spark-sql-mcp-server"],
      "env": {
        "SPARK_HOST": "your-emr-master-node.amazonaws.com",
        "SPARK_PORT": "10000"
      }
    }
  }
}

4. Query

Ask Claude things like:

• "What databases are available in our Spark cluster?"
• "Show me the schema of the sales.transactions table"
• "Query the top 10 customers by revenue from the analytics database"

Available Tools

Authentication

No Auth (default)

export SPARK_AUTH="NONE"

LDAP

export SPARK_AUTH="LDAP"
export SPARK_USERNAME="your-username"
export SPARK_PASSWORD="your-password"

Kerberos

export SPARK_AUTH="KERBEROS"
export SPARK_KERBEROS_SERVICE_NAME="hive"  # default
# Ensure you have a valid Kerberos ticket (kinit)

AWS EMR Setup

1. Security Group — Allow inbound traffic on port 10000 from your IP
2. SSH Tunnel (recommended):

   ssh -i your-key.pem -L 10000:localhost:10000 hadoop@your-emr-master
   

3. Set SPARK_HOST=localhost

Development

git clone https://github.com/aidancorrell/spark-sql-mcp-server.git
cd spark-sql-mcp-server
pip install -e ".[dev]"
pytest
ruff check .

Local Testing with Docker

A Docker Compose setup provides a local Spark Thrift Server with sample data for integration testing.

# Start the Spark Thrift Server
cd docker && docker compose up -d

# Wait for it to be ready (takes ~30s on first start)
docker logs -f spark-thrift-server  # look for "Sample data loaded."

# Run integration tests
pytest -m integration -

... [View full README on GitHub](https://github.com/aidancorrell/spark-sql-mcp-server#readme)