Designing an autonomous multi-agent data-engineering swarm

The dream, and the trap

The promise of agentic AI is seductively simple: describe the task, get a working result. The reality, for anything non-trivial, is that a single agent asked to build an entire data pipeline will hand you something that looks done and isn't. It invents columns that don't exist in the source. It quietly skips the deduplication step, because that part is hard to verify. And because the same model wrote the spec, the code, and the verdict on whether the code is correct, there is no independent place to look when something is wrong.

I wanted a system that could take a vague data-engineering request and return a machine-checkable pass or fail — not a paragraph assuring me it worked. This is the story of how breaking one agent into three, each unable to do the others' job, got me there.

First, do it the wrong way

My first version was the obvious one: one capable agent, a long prompt, full tool access. "Here's the source data; build a tested pipeline and tell me if it's correct." It produced impressive output and was wrong in ways I couldn't easily see. Three failure modes kept repeating:

1. Conflated roles. The agent that wrote the schema was also the agent that "validated" the schema. A model rarely catches its own mistakes — it has every incentive to declare its own work correct.
2. Unverifiable claims. "The pipeline handles duplicates correctly" is a sentence a model can write whether or not it is true. There was no machine-readable signal behind the claim.
3. Context pollution. Spec, code, errors, and justification all lived in one context window. When it went wrong, the failure was buried in a wall of text.

The key idea: separate capability from authority

The fix came from a shift in how I thought about agents. An LLM is broadly capable — given tools, it can read, write, and reason about almost anything. But capability is not the same as authority. A senior engineer is capable of writing code, doing QA, and designing architecture, but on a real team we still don't let the same person merge their own pull request. We separate roles not because people aren't capable, but because separation makes the result checkable.

So instead of one agent with all the tools, I built three agents, each with a deliberately restricted toolset. The constraint isn't "please focus on X" — it's "you physically cannot do Y."

The three roles

The swarm is a plan → build → verify loop. Each stage is a different agent with the wrong tools for the other two.

• The Planner (a data architect). Reads the request and produces two documents: a plan, and a data contract. The contract is the spec the rest of the system works against — it names the grain (one row per what?), every column with its type, the freshness SLA, the partition key, the idempotency strategy, and the acceptance criteria. Its toolset is Read, Grep, Glob, and Write to docs/ — but no Edit and no Bash. It cannot write or run code even if it tries. Its output is a specification, by construction.
• The Builder (a pipeline engineer). Implements the pipeline against the contract — bronze (raw), silver (cleaned), gold (business-facing) layers, typed schemas, incremental models, partitioning. It has Read, Edit, Write, Bash, Glob, Grep — full implementation power. But its brief is "implement to spec; do not invent scope." It builds exactly what the contract says exists, no more.
• The Verifier (a data-quality engineer). Checks the result against the contract on two axes: do the code tests pass, and is the data actually correct? It checks schema drift, nulls, uniqueness, referential integrity, row-count sanity, freshness, and idempotency (a re-run yields identical output). Its toolset is Read, Grep, Glob, Bash — and crucially, no Write and no Edit. It cannot fix a bug it finds. It can only report one.

What the contract actually looks like

The contract is the backbone of the whole system, so it's worth showing its shape. A real one is longer, but the skeleton is this:

# docs/data-contract.md
grain:          one row per order_line
source:         raw.orders + raw.order_items
columns:
  order_line_id : uuid, not null, unique
  order_id      : uuid, not null   # FK to orders
  amount        : decimal(10,2), not null, >= 0
  created_at    : timestamptz, not null
partition_key:  created_at
freshness_sla:  max(now() - max(created_at)) < 24 hours
idempotency:    re-running on the same input yields byte-identical output
acceptance:
  - no duplicate order_line_id
  - every order_id exists in orders
  - no null amount

Every line there is something the Verifier can turn into a query with a yes/no answer. That is the point: correctness expressed as checkable facts, not prose.

Why restricted tools beat better prompting

This is the part I want to emphasise, because it is the opposite of how most people try to improve agents. A prompt that says "don't edit code" is a request the model can ignore, especially under complexity. An agent that does not have the Edit tool cannot edit code — that is enforced by the system, not by the model's compliance. You have moved the constraint from something the model has to remember to something the system guarantees.

Anthropic's write-up on multi-agent research systems makes the same point from the other direction: the orchestrator-worker pattern works because each subagent gets its own context window and a focused objective, and handoffs happen through files rather than long in-context monologues. My swarm is a stricter version of that idea — not just separate contexts, but separate and deliberately incompatible toolsets.

The contract is the only hand-off

Agents don't pass messages to each other. The only thing that moves between stages is a file: the data contract. The Planner writes it; the Builder reads it before writing a line of code; the Verifier reads it to know what "correct" means. This is what turns correctness from a vibe into a checkable property. "The model said it worked" becomes a set of questions with yes/no answers: does the actual schema match the contract? Are the keys unique where the contract requires uniqueness? Is freshness inside the SLA? Does a re-run produce identical output?

If you know design by contract or typed functional programming, this should feel familiar — it is the same instinct as "make illegal states unrepresentable," applied at the boundary between two agents instead of inside one program.

The loop returns a verdict, not a story

plan (architect) ──▶ build (engineer) ──▶ verify (qa)
      ▲                                       │
      └────── fix-back (max 2 rounds) ────────┘  fail

If the Verifier's verdict is fail, the specific failures go back to the Builder — not "try again," but "these three acceptance criteria failed; here they are." After two failed rounds the system stops and reports failure rather than looping forever. The final output is a structured object:

{
  "passed": false,
  "qa_failures": [
    "duplicate order_line_id: 3 rows",
    "2 order_id values missing in orders"
  ]
}

A human — or a CI step — can act on that without reading anything the model wrote. That is the goal: a result you trust because it was checked by an independent agent against an explicit contract, not because a model asserted it.

Trade-offs (for the sceptics)

This isn't free, and I don't want to oversell it. Multi-agent systems burn more tokens — cost roughly scales with the number of agents, because each carries its own context. They are slower: stages run sequentially, and the fix-back loop adds round-trips. They are more complex to build and debug than a single prompt. So this pattern earns its keep on tasks where correctness matters and "looks plausible but is wrong" is expensive — building a pipeline someone will make decisions on, generating code that runs in production, anything where a silent failure costs more than the extra tokens. For a one-shot "summarise this," a single agent is the right call. Don't reach for three agents when one will do.

What I'd do differently

The contract can be wrong. A Planner that misunderstands the source produces a confidently incorrect spec, and the Builder will faithfully implement the wrong thing — the Verifier catches implementation bugs, not spec bugs. Catching spec bugs means a human (or a fourth, adversarial agent) reviewing the contract itself before any code is written. I would also add hard cost guardrails: token caps and a strict fix-back ceiling per run, so a stuck loop fails fast instead of burning budget.

What a run actually looks like

Abstract architecture is cheap; let me ground it. Suppose the request is: "build a daily pipeline that turns raw order events into one row per order line, fresh within 24 hours." Here is what each agent does.

The Planner reads the source schema and writes the contract: grain is one row per order line; columns are order_line_id (uuid, unique), order_id (foreign key), amount (decimal, non-negative), created_at; partition on created_at; freshness under 24 hours; idempotent on input; and acceptance criteria — no duplicate ids, every order_id resolves, no null amount. It cannot touch code, so it spends its budget on getting the spec right.

The Builder implements three models — bronze (raw copy), silver (cleaned, typed, deduped), gold (one row per order line, joined) — plus the tests for those acceptance criteria. It works strictly inside the contract: if the contract doesn't mention a column, it doesn't invent one.

The Verifier runs the tests and queries the data directly — counts duplicate order_line_ids, checks every order_id exists in the orders table, asserts freshness, and re-runs the pipeline to confirm identical output. It returns a structured verdict: pass, or fail with the specific broken criteria. It writes nothing; it only reports.

Notice what is missing from this loop. There is no point at which a model simply declares the pipeline correct. Correctness is established by an independent agent querying the data against an explicit spec. That is the entire reason the architecture exists.

The takeaway

Reliability in agentic systems comes from architecture, not prompting. Splitting roles, restricting tools, and replacing agent-to-agent chat with a file-based contract are all ways of making the system's state inspectable and its failures specific. The goal was never a smarter agent. It was a system where, when something is wrong, you can tell.

Further reading:

• Anthropic — Multi-agent research systems (the orchestrator-worker pattern, context isolation, file-based handoffs): anthropic.com/engineering
• Medallion architecture — the bronze / silver / gold layering the Builder targets: databricks.com/glossary/medallion-architecture