Interactive Data Systems Lab

Runnable Python and Rust data-system internals with an MCP-native runtime copilot layer.

This repository combines two things:

• a local-first engineering lab for PostgreSQL-like and Databricks-like internals,
• a Runtime Copilot MCP surface for diagnostics, explainability, regression checks, and operational memory.

Core capabilities

• Storage and planner internals: heap tables, B-tree indexing, selectivity, and plan choice.
• Persistence and replay: WAL/checkpoint style flows and deterministic state transitions.
• Workflow and write-path modeling: event-first architecture, idempotency, retry semantics.
• Explainable runtime operations: traced runs, failure summaries, regression verdicts, baseline compare.
• MCP access: machine-usable operational interface instead of ad hoc shell scripts.

Who this is for

• Data engineers learning warehouse and query-engine internals.
• Platform and infrastructure engineers teaching storage and execution fundamentals.
• Teams building onboarding labs, workshops, and demo environments.

Quick start

Requirements:

• Python 3.10+ (tested with Python 3.14)

Setup:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install "psycopg[binary]"

Run end-to-end flow:

cargo run --bin e2e_flow

Run core demos:

.venv/bin/python mini_pg_like.py
.venv/bin/python mini_databricks_clone.py
cargo run --bin mini_pg_like
cargo run --bin mini_databricks_clone

What this repository contains

• mini_pg_like.py: PostgreSQL-like toy engine with heap table, B-tree index, and planner output.
• mini_databricks_clone.py: Databricks-like toy platform with versioning, partitions, DAGs, and events.
• src/bin/mini_pg_like.rs: Rust PostgreSQL-like demo.
• src/bin/mini_databricks_clone.rs: Rust Databricks-like demo.
• src/lib.rs, src/common.rs, src/pg.rs: shared Rust core modules.
• mcp_engine_server.py: MCP runtime adapter for diagnostics and regression workflows.

Why this exists

Most internals content stops at diagrams. This project stays runnable and inspectable:

• compare Python and Rust implementations of the same system ideas,
• trace write-path behavior with concrete events and state transitions,
• run explainable regression checks through MCP,
• turn runtime operations into a discoverable control surface for AI clients.

MCP Adapter Layer

This repo also includes a minimal MCP server that wraps the lab operations:

• mcp_engine_server.py
• Cursor config: .cursor/mcp.json

Current MCP tool list for this release (47 tools total):

Engine state and runtime:

• init_engine
• insert_row
• upsert_row
• create_index
• explain_customer
• reindex_project
• run_e2e_flow

Explainability and demos:

• explain_run
• demo_explain_run
• demo_explain_run_failure
• demo_explain_semantic_failure
• demo_explain_idempotency_conflict
• demo_explain_concurrency_failure_storm
• explain_regression_suite

Trace and retrieval:

• record_tool_trace
• similar_incidents
• refresh_trace_path
• refresh_docs_path
• memory_upsert
• memory_search

SLO and ROI:

• health_check
• benchmark_calls
• scenario_load_test
• capture_roi_baseline
• report_drift_bug
• decision_gate

Schema evaluation (verdict + report via MCP, no external Postgres):

• schema_load_tool
• schema_explain_tool
• schema_evaluate_tool
• schema_evaluate_full_tool

Project contract and regression:

• project_manifest
• project_capabilities
• project_tool_catalog
• project_get_defaults
• project_run_regression
• project_capture_baseline
• project_compare_baseline

Generic project state:

• project_list_entities
• project_get_entity
• project_upsert_entity
• project_delete_entity
• project_append_event
• project_ingest_trace
• project_explain_run
• project_export_state

Generic heuristics:

• project_list_heuristics
• project_run_heuristic

For machine-readable discovery, prefer

... View full README on GitHub