Overview
Merlin is an AI-powered assistant designed specifically for Data Engineers. It provides a chat-based UI integrated with MCP (Model Context Protocol) and a Skills system on the DeerFlow backend, enabling engineers to query databases, generate pipelines, and debug ETL jobs through natural language.
Why Build This?
Existing AI coding assistants are general-purpose. Data Engineers have specific needs:
- Querying across multiple databases and warehouses
- Understanding complex schema relationships
- Generating and debugging SQL, Spark, and Airflow code
- Accessing documentation spread across many tools
Architecture
The system uses a multi-model orchestration approach:
- DeepSeek — for code generation and SQL tasks (cost-efficient)
- Kimi — for long-context document analysis
- Claude — for complex reasoning and architecture decisions
An intelligent router selects the best model based on the task type, reducing token costs by over 10x compared to using a single premium model for everything.
MCP Integration
The Model Context Protocol allows Merlin to:
- Connect to multiple backends — databases, file systems, APIs
- Access cross-server resources — query a Postgres database while reading from S3
- Execute Skills — pre-built workflows for common DE tasks (schema migration, data profiling, etc.)
Design Decisions
- Skills over prompts — Instead of relying on prompt engineering, common workflows are encoded as deterministic "Skills" that the AI orchestrates
- Token budget management — Each request has a token budget; the router picks models that fit within it
- Streaming responses — Long-running queries show intermediate results to keep the UX responsive
Lessons Learned
- Model selection matters more than prompting — Choosing the right model for each sub-task had a bigger impact than tweaking prompts
- MCP is still early — Had to work around protocol limitations and implement custom retry logic
- Data Engineers want control — The UI shows exactly which queries are being run, not just the final answer