Reliability for LLM apps: routing, retries, and deterministic control

The output is the product

I built an agentic journaling app — Voice Debrief — that turns a daily debrief into structured, queryable rows. The interesting work isn't getting the model to produce text. It's making sure the model's output is reliable enough to store. The moment you persist what an LLM generates, every failure mode becomes data corruption: a hallucinated value doesn't disappear when you refresh the page, it sits in your database quietly poisoning every downstream query.

So the design problem is sharp: how do you make malformed or hallucinated output unable to reach the database? Here are the techniques that carried most of the weight — and, more importantly, the single principle underneath all of them.

Start with the right mental model

Before any technique, adopt this framing: an LLM is an untrusted subprocess that produces text. Treat its output the way you treat user input, form submissions, or third-party API responses. You would never pipe raw user input straight into your database; you validate it at the boundary first. Model output deserves the same distrust — arguably more, because a model's output is plausible by design, which makes its errors harder to spot than a user's typos.

Everything below follows from that one shift.

1. Route by stakes, not by capability

The app uses two models: a fast, cheap one and a strong, expensive one. The naive approach is "use the strong model for everything important." The better question is: what's the cost of being wrong here?

• The interview driver and the two-sentence overview are low-stakes — if they're a little off, nothing breaks. A fast model handles them, and its fluency is more than good enough.
• Structured extraction is high-stakes — these rows become the data layer. Precision matters far more than speed or cost. The strong model does this.

Routing by cost-of-being-wrong lets you spend the expensive model where mistakes are expensive and a cheap model everywhere else. It is a reliability decision dressed up as a cost decision. The heuristic I use: if the output feeds a decision or gets stored verbatim, route to the strong model; if it is ephemeral or cosmetic, the fast model is fine.

2. Schema-validated extraction with a retry loop

The strong model is asked to return JSON matching a schema. The moment it comes back, a validator (Zod) checks it. If it fails — wrong shape, missing field, bad type — the error is fed straight back to the model and it tries again, up to a fixed number of attempts.

for attempt in range(max_attempts):
    raw = model.complete(prompt, response_format="json")
    parsed = try_parse_json(raw)         # must be valid JSON
    result = schema.safe_parse(parsed)  # must match the schema
    if result.ok:
        return result.value
    prompt = prompt + "\n\nThat failed: " + result.error
raise ExtractionError   # fail loud, store nothing

The contract is absolute: nothing reaches the store unless it passes the schema. A malformed payload isn't "stored with a warning" — it is rejected, retried, and if it keeps failing, the whole extraction fails loudly. There is no code path that writes unvalidated data, so the store physically cannot be poisoned by bad model output.

If you have followed the platforms lately, you know there are now stronger versions of this. Plain "JSON mode" only guarantees the output is valid JSON — it does not guarantee it matches your schema. Structured outputs (OpenAI) and tool/function calling (OpenAI, Anthropic) go further: they constrain decoding so the model's tokens are forced to conform to a JSON Schema as they are generated. That is a harder guarantee than validating after the fact — it makes many malformed outputs ungeneratable. But it does not replace the boundary validator. Constrained decoding catches structural errors; your schema at the boundary catches semantic ones (a well-formed number that is still wrong). Use both.

3. Deterministic control via an explicit checklist

The interview has to know when it has covered enough. The tempting design is to ask the model "are you done?" — which hands a reliability-critical decision to the least reliable component in the system. A model that wants to be helpful will often say yes too early; a model that is uncertain will ramble. Either way, you have given control of your state machine to a probability distribution.

Instead, the model is never asked that. It is asked only one thing: which of these three fields did the user's latest message actually address? The model sets coverage flags; ordinary code merges them into a running checklist. Completion is a deterministic computation — "all three flags are true" — not a feeling the model has. The model can't decide to wrap up early, because the decision isn't the model's to make.

The general pattern: use the LLM as a sensor (it reads the user's message and reports what it saw), not as a controller (it decides what happens next). Sensors can be noisy without breaking the system, because the control logic that consumes them is deterministic. This is just the classic state-machine rule — keep control flow out of the unreliable component — rediscovered for the LLM era.

4. Make the write atomic and idempotent

One more, because it saves you on bad days. When extraction succeeds, write the results in a single database transaction, so a partial failure can't leave half a session stored. And make the write idempotent on the session id, so a retry (from a network blip, or the user re-submitting) updates the same rows instead of creating duplicates. The model does not need to know about any of this; the storage layer enforces it regardless of what the model produced.

The principle underneath all of them

Each technique does the same thing in a different place: it makes wrongness either impossible (the schema rejects it; the unvalidated write path does not exist) or loudly detected (the checklist exposes it; the transaction rolls back). You do not make an LLM reliable by hoping. You accept that it will sometimes produce garbage, and you engineer the surrounding system so that garbage can't pass silently. The model stays generative; the guardrails do the verifying.

If you remember one thing, remember the boundary. Put the check at the place untrusted data crosses into trusted territory — once, thoroughly — and let everything inside that boundary assume the data is good.

Trade-offs and limits

• Latency. Retries add round-trips. A failing extraction can take several attempts before it gives up, and the user waits. Set max_attempts low, and fall back gracefully (show the raw input, let the user edit) rather than blocking.
• Valid but wrong. A schema cannot catch a confident hallucination that happens to be well-formed. For high-stakes fields, pair the schema with a human-in-the-loop edit step — which is exactly why my app lets the user correct extracted rows and writes those corrections back with a flag.
• Free text. Not every field is structured. Overview paragraphs and summaries can't be schema-validated the way a typed row can; for those, length limits and a separate review path are the best you can do. Don't pretend a schema covers what it can't.

Test the model like you test code

Schemas catch malformed output. But how do you know the model's correct output is still correct after you change a prompt, swap a model, or bump a temperature? The answer is the same as for any other code: you write tests.

Keep a golden set — a few dozen real inputs paired with the output you expect — and run the extraction over them on every change. If a prompt tweak makes three of them flip, your test suite tells you before a user does. This is just regression testing, applied to a component people forget is code. The model is a function; the prompt is its source; treat changes to the prompt with the same discipline as changes to a function body.

The same logic guards the schema itself. A schema that rejects everything is technically "safe" and totally useless — it fails every extraction. Your golden set catches that too: if a schema change drops your success rate from 95% to 40%, that is a regression, not a reliability win. Reliability means reliably succeeding, not just reliably refusing bad output.

Takeaway

Treat the model as an untrusted subprocess. Validate its output at the boundary, route by the cost of being wrong, keep control flow deterministic, and make writes atomic and idempotent. Do those four things and your database stops being at the mercy of a probability distribution — which, if you are building anything you want people to depend on, is the whole game.

Further reading:

• OpenAI — Structured Outputs (constrained decoding to a JSON Schema, vs plain JSON mode): platform.openai.com/docs/guides/structured-outputs
• OpenAI — Function calling (tool/function-calling as another constrained-output mechanism): platform.openai.com/docs/guides/function-calling
• Zod — schema validation at the TypeScript boundary: zod.dev