A Two-Year Roadmap: From Data Engineering to AI Agents and Harness Engineering

Why I am writing this down

I have spent three years building warehouses, pipelines and dashboards for banking, telecom and retail clients. Over the last year more and more of my side projects have been agents: an MCP server for IBM Cognos, a two-tier goal agent on Cloud Run, a RAG dataset for Vietnamese business law.

I do not want to leave data engineering. I want to be the data engineer who can also build the part that surrounds a model: its tools, its context, its permissions, its evaluation, its cost controls. People have started calling that part the harness, and the short version is agent = model + harness.

This post is the plan for the next two years. I researched it on 12 November 2026. Exam prices, formats and eligibility rules come mostly from third-party write-ups that do not always agree with each other, so treat every number here as approximate and check the vendor page before paying for anything.

The plan at a glance

Each phase ends with one certification and one project.

Phase	When	Certification	Fee (USD)	Valid for
0	Nov–Dec 2026	None. Groundwork	0	n/a
1	Dec 2026 – Mar 2027	Google Cloud Professional Agentic Architect	about 200	1 year
2	Apr – Jul 2027	Databricks Certified Data Engineer Professional	200	2 years
3	Aug – Nov 2027	Databricks Certified Generative AI Engineer Associate	about 200	2 years
4	Dec 2027 – Apr 2028	AWS Certified Generative AI Developer – Professional	about 300	3 years
5	May – Nov 2028	Claude Certified Architect, or NVIDIA NCP-AAI as a fallback	125 to 200	1 to 2 years

Exam fees add up to roughly 1,000 to 1,200 USD over two years, before retakes and renewals.

Why these certifications

The agent side

Google Cloud Professional Agentic Architect launched on 11 November 2026. It has two parts: a multiple-choice exam on system design, cost and security, and hands-on labs. It covers the Agent Development Kit, agent identity and model safety tooling. I already have a Google Cloud data background and agents running on Cloud Run, so this is the shortest path to a current, agent-focused credential.

Claude Certified Architect is the closest match to harness work: agentic architecture, Claude Code, prompt engineering and MCP. The catch is access. At the time of writing the exams are open to people at organizations in the Claude Partner Network, so I have put it last and will move it earlier if I become eligible.

NVIDIA NCP-AAI is open to anyone and covers agent architecture, retrieval and memory, orchestration, deployment and security. It is my fallback for phase 5.

The data side

Databricks Data Engineer Professional fills the largest gap in my profile. I have Spark, Kimball modelling and slowly changing dimensions in production, but no public proof of lakehouse work. The data engineering job posts I looked at in Vietnam mention Databricks and Snowflake more than anything else.

Databricks Generative AI Engineer Associate connects the two sides: retrieval, vector search, model serving, evaluation and governance on top of governed data.

AWS Generative AI Developer – Professional builds on my AWS data engineering certification and adds Bedrock agents plus a full domain on AI safety and governance.

What I left out

Snowflake SnowPro: a good choice if your target jobs run on Snowflake. I picked one lakehouse platform instead of two.
Microsoft AI-103 and DP-700: only worth it for me if a project lands on Azure.
Agent courses without a proctored exam: useful for learning, weaker as evidence.

Phase by phase

Phase 0: groundwork (Nov–Dec 2026)

No exam. I will audit my existing certifications and their expiry dates, tidy the public repositories, and build the first project.

Project P0, a minimal eval harness. I will pull the evaluation cases out of my goal agent into a reusable package: a test-case set, automatic scoring, and a regression report whenever a prompt or a model changes. Every later project reuses it.

Phase 1: Google Cloud Professional Agentic Architect (Dec 2026 – Mar 2027)

Project P1, a data-ops agent on Google Cloud. The agent receives a failed-pipeline alert, reads the logs, classifies the cause and proposes a fix. It may only execute actions from an allowlist. It runs on Cloud Run, is provisioned with Terraform, is scored by the P0 harness and reports its cost per run.

Phase 2: Databricks Data Engineer Professional (Apr – Jul 2027)

Project P2, a Medallion lakehouse. Streaming air-quality sensor data from my university lab work and a public retail dataset go through Bronze, Silver and Gold layers on Delta, with slowly changing dimensions, data-quality checks that block bad batches, and access control.

Phase 3: Databricks Generative AI Engineer Associate (Aug – Nov 2027)

Project P3, retrieval and agents on the lakehouse. I will move my Vietnamese business-law corpus (2,315 articles, 841 chunks) to vector search and add an agent that answers with article-level citations. A second agent answers questions over the Gold tables from P2 with permission-aware text-to-SQL. I will measure retrieval accuracy, citation correctness and cost per question.

Phase 4: AWS Generative AI Developer – Professional (Dec 2027 – Apr 2028)

Project P4, an agent on my AWS data platform. I will extend my existing Terraform platform (Kinesis, Lambda, S3, Glue, Athena) with an agent that queries the catalog through tools scoped by IAM, with guardrails for sensitive data and logging for every tool call. The same P0 test set runs against both the Google Cloud and the AWS agent, so the two can be compared.

Phase 5: Claude Certified Architect or NVIDIA NCP-AAI, plus a capstone (May – Nov 2028)

Project P5, a harness for data agents. This combines everything: tools served over MCP behind a permission registry, context management, model routing by task difficulty, a sandbox for execution, deterministic gates (tests and data checks) that must pass before the agent's work is accepted, regression evals, and a dashboard for cost and success rate. I will publish the results with numbers.

The harness skills no exam covers

I did not find a certification that tests harness engineering directly, so these have to be shown through the projects.

Skill	Learned in	Evidence
Evals: test sets, automatic scoring, regression	Phase 0, reused everywhere	Eval harness repository, numbers in P1 to P5
Tool design, MCP and permissions	Phases 0 and 1	Cognos MCP server, P1
Context engineering: what to load, compress and remember	Phases 1 and 3	P1, P3
Safety: action allowlists, sandboxing, guardrails	Phases 1 and 4	P1, P4
Observability and cost: traces, cost per run, model routing	Phases 3 and 4	P3, P4
Deterministic gates before accepting agent output	Phases 2 and 5	Data-quality gates in P2, P5

Risks I am planning around

Agent certifications expire fast. The Google and Claude credentials last one year. By the time they lapse, the projects should carry the proof.
Exams keep changing. Several of these exam guides were rewritten in 2026. I will download the current guide before starting each phase.
The plan may be wrong. I will review it after every phase against real job posts and the technology I am actually working with, and swap a phase if the market points elsewhere.
Pace. Four months per phase alongside a full-time job is tight. If I fall behind, phases 1 to 3 stay and phases 4 and 5 slide.

What happens next

Phase 0 starts this month. I plan to post an update here at the end of each phase with what I built, what the exam was like, and what I would change.