Overview
One of the biggest pain points in data engineering is testing ETL pipelines without impacting production data. In this project, I built an automated infrastructure provisioning system using Terraform that spins up isolated sandbox environments on both AWS and GCP for running integration tests.
The Problem
Traditional ETL testing approaches suffer from several issues:
- Environment drift — dev and staging environments slowly diverge from production
- Shared state — multiple engineers running tests against the same database leads to flaky results
- Slow feedback — waiting for a shared test environment to become available
Architecture & Design
The solution uses Terraform modules to provision ephemeral infrastructure:
module "sandbox" {
source = "./modules/sandbox"
cloud_provider = var.provider # "aws" or "gcp"
vpc_cidr = "10.0.0.0/16"
instance_type = "t3.medium"
}
Each test run gets its own:
- VPC with isolated networking
- Compute instance (VM) for running ETL jobs
- Gateway for controlled external access
- Storage bucket for test data
Key Design Decisions
- Multi-cloud support — The same Terraform modules work on both AWS and GCP, using provider-agnostic abstractions where possible
- Zero production impact — Sandboxes are completely isolated; no shared databases or queues
- Reproducibility — Every sandbox uses pinned OS versions and library versions, eliminating "works on my machine" issues
Lessons Learned
- State management is critical — Use remote Terraform state with locking to prevent concurrent sandbox conflicts
- Cost control — Always set TTL on sandbox resources; forgot once and ran up a bill on idle VMs
- Integration tests need real data shapes — Synthetic test data should mirror production schemas, not just simple fixtures