DataSQRL

DataSQRL is a data automation framework for building reliable data pipelines, data APIs (REST, MCP, GraphQL), and data products in SQL using open-source technologies.

DataSQRL provides three key elements for AI-assisted data platform automation:

1. World Model: DataSQRL builds a source-to-sink computational graph of the data processing including schemas, connectors, and mappings, which provides a comprehensive world model to ground generative AI.
2. Simulation: DataSQRL includes a runtime and testing framework to ensure data integrity and act as a simulator in iterative refinement loops with real-world feedback.
3. Verification: Since the entire data pipeline is defined in SQL, it is easy to understand and verify. DataSQRL produces detailed execution plans and lineage graphs to assist automated and manual analysis.

DataSQRL generates the deployment artifacts to execute the entire pipeline on open-source technologies like PostgreSQL, Apache Kafka, Apache Flink, and Apache Iceberg on your existing infrastructure with Docker, Kubernetes, or cloud-managed services.

DataSQRL models data pipelines with the following requirements:

• 🛡️ Data Consistency Guarantees: Exactly-once processing, data consistency across all outputs, schema alignment, and data lineage tracking.
• 🔒 Production-grade Reliability: Robust, highly available, scalable, secure, access-controlled, and observable data pipelines.
• 🚀 Developer Workflow Integration: Local development, quick iteration with feedback, CI/CD support, and comprehensive testing framework.

To learn more about DataSQRL, check out the documentation.

To see how DataSQRL provides feedback and guides AI coding agents to build data products autonomously, view this demo video.

Getting Started

To create a new data project with DataSQRL, use the init command in an empty folder.

 docker run --rm -v $PWD:/build datasqrl/cmd init api messenger

(Use ${PWD} in Powershell on Windows).

This creates a new data API project called messenger with some sample data sources and a simple data processing script called messenger.sqrl.

Run the project with

docker run -it --rm -p 8888:8888 -p 8081:8081 -v $PWD:/build datasqrl/cmd run messenger-prod-package.json

This launches the entire data pipeline for ingesting, processing, storing, and serving messages. You can access the API in your browser http://localhost:8888/v1/graphiql/ and add messages with the following mutation:

mutation {
    Messages(event: {message: "Hello World"}) {
    message_time
  }
}

Query messages with:

{
    Messages {
    message
    message_time
  }
}

Alternatively, you can query messages through REST or MCP. Once you are done, terminate the pipeline with CTRL-C.

For additional data processing, edit the messenger.sqrl script - for example to aggregate messages:

TotalMessages := SELECT COUNT(*) as num_messages, MAX(m

... [View full README on GitHub](https://github.com/DataSQRL/sqrl#readme)