How to Get Reliable JSON from LLMs: Structured Outputs & Schemas

Why LLMs produce broken JSON

An LLM generates text one token at a time by sampling from probabilities. Unless something constrains it, nothing forces the output to be well-formed JSON. The model can wrap the response in markdown code fences, add a friendly sentence before or after the object, emit trailing commas, use single quotes, or simply run out of tokens mid-object and return truncated JSON.

These failures are not bugs in the model so much as a mismatch between free-form generation and a format that has strict rules. The fix is to remove the model's freedom to break the rules, and to verify whatever it does produce.

The reliability ladder

Think of reliability as three rungs, each tighter than the last:

• Prompt engineering (roughly 80 to 95 percent) — asking the model to 'respond only with JSON' and giving an example. Cheap, works often, fails unpredictably under load.
• Function calling and tool use (roughly 95 to 99 percent) — the provider exposes a tool with a JSON Schema and the model fills in arguments. More reliable because the schema shapes the request, but not a hard guarantee.
• Native structured output with constrained decoding (effectively 100 percent valid) — the runtime restricts token generation so the model literally cannot emit a token that would violate the schema. Invalid JSON becomes impossible by construction.

Native structured outputs in 2026

By early 2026 the ecosystem has converged: OpenAI, Anthropic, and Google Gemini all support native structured output, and plain JSON mode has become a footnote. It helps to distinguish two guarantees. JSON mode guarantees the output is syntactically valid JSON but does not enforce your schema. Structured outputs go further and use constrained decoding to guarantee the response matches your JSON Schema — correct field names, correct types, and all required properties present.

The practical advice is simple: if your provider offers schema-enforced structured outputs, use them. They eliminate the entire class of syntax and shape errors at the generation step, which is far more robust than parsing and hoping.

Provider differences worth knowing

Implementations are not identical, and the details matter. Notably, Anthropic's SDK transforms schemas before generation by removing fine-grained constraints such as minimum, maximum, minLength, and pattern, relocating them into field descriptions as plain-language guidance, and then validating the generated output against those constraints afterward with automatic retries.

The lesson: constrained decoding reliably guarantees structure (the right fields and types), but value-level constraints like ranges and regex patterns may be enforced by post-generation validation rather than by the decoder. You still need your own validation layer for those.

Validate everything with JSON Schema

Even with structured outputs, never let unvalidated model output flow into your database or downstream services. Define a JSON Schema as the contract, and validate every response against it. In Python, libraries like Pydantic turn a schema into a typed model and raise on mismatch; in TypeScript, Zod does the same. This validation step is your safety net for the value-level rules the decoder may not enforce, and for the rare case where a provider returns something unexpected.

A schema also doubles as documentation and as the thing you hand to the model: the same JSON Schema that constrains generation is the one you validate against, keeping a single source of truth.

from pydantic import BaseModel

class Result(BaseModel):
    name: str
    score: int

Result.model_validate_json(llm_output)  # raises on mismatch

Repairing broken JSON when you cannot constrain it

Sometimes you cannot use structured outputs — an older model, a streamed response, or output you received as plain text. For those cases, a repair step recovers a surprising amount: stripping markdown code fences, removing trailing commas, quoting unquoted keys, converting single quotes, and closing brackets on truncated objects. Rule-based repair handles the most common LLM failure modes without another expensive model call.

Treat repair as a fallback, not a primary strategy. The order of preference is: constrain generation first, validate always, and repair only what slips through.

A production-ready pipeline

1. define a JSON Schema for the output
2. request structured output (schema-constrained)
3. validate the response against the schema
4. on failure → run a repair pass
5. still invalid → retry, feeding back the error

• Define a JSON Schema that describes the exact output you need, including required fields and types.
• Request structured output using that schema so the provider constrains generation.
• Validate the response against the schema (Pydantic, Zod, or a schema validator) to enforce value-level rules.
• If validation fails, attempt an automatic repair pass for common syntactic damage.
• If repair still does not produce schema-valid JSON, retry the call, optionally feeding back the validation error so the model can correct itself.
• Only then pass the result downstream — every record entering your system is schema-valid by construction.