JSON Drift
Back to blog
AI· Updated 9 min read

Why LLMs Produce Invalid JSON and How to Fix It

Understand why ChatGPT and other LLMs break JSON syntax — markdown fences, trailing commas, truncation, schema drift — and learn prompting, validation, repair, and schema strategies for production AI apps.

By Saifur Rahman Shochib

Large language models are trained to predict text, not to satisfy a grammar. When you ask one for JSON, it usually obliges — but it also wraps the payload in markdown, adds a trailing comma, leaves a comment, or runs out of tokens mid-object. In a production AI system, model output must be treated as untrusted until it has passed through validation and, ideally, a schema. This guide explains why these failures happen and how to build a pipeline that turns unreliable model text into JSON your code can trust.

Why models break JSON

An LLM generates one token at a time from a probability distribution shaped by its training data. That training data is mostly prose and source code, where trailing commas, single quotes, comments, and markdown code fences are all common and correct. The model has no parser checking each token, so it reproduces those habits. It also has a hard token budget: when a long object exceeds the limit, generation simply stops, leaving the JSON truncated. None of this is a bug in the model — it is a mismatch between free-form text generation and a strict grammar.

Common LLM JSON failure modes

Most broken model output falls into a handful of recognizable patterns. A good repair-and-validation step is built specifically to catch these.

The failure modes a repair step must handletext
```json            <- wrapped in a markdown code fence
Sure, here it is:  <- leading prose
{"ok": true,}      <- trailing comma
{'ok': true}       <- single quotes
{"ok": tr          <- truncated mid-value
  • Markdown code fences (```json ... ```) wrapping the payload.
  • Leading prose such as 'Here is the JSON you requested:' before the object.
  • Trailing commas and inline comments carried over from code in training data.
  • Single-quoted strings or unquoted keys.
  • Truncated output when the response hits the token limit.
  • Schema drift — extra fields, missing required fields, or values of the wrong type.
  • Hallucinated enum values that look plausible but are not in your allowed set.

Step one: prompt for cleaner output

Prevention is cheaper than repair. Tell the model explicitly to return only JSON with no markdown, no comments, and no surrounding text, and give it a concrete example of the exact shape you expect. Provide the target schema or a sample object in the prompt so the model anchors on real field names and types. Keep responses small — ask for one object per call rather than a giant batch — to reduce the chance of hitting the token limit and truncating.

A prompt that reduces JSON errorstext
Return ONLY valid JSON matching this shape, with no markdown
fences, no comments, and no explanation:
{ "sentiment": "positive|neutral|negative", "score": 0.0 }

Step two: use structured-output modes

Modern model providers offer constrained generation that guarantees well-formed JSON. Anthropic's Claude models support tool use, where you define a tool with a JSON Schema for its input and the model is constrained to produce arguments that fit that schema. OpenAI offers a JSON mode and structured outputs. When such a mode is available, prefer it: constraining generation at the source eliminates whole categories of syntax error before they reach your code. It does not, however, free you from validation — the data can still be semantically wrong even when it parses.

Step three: repair and validate (defense in depth)

Because no single safeguard is perfect, layer them. First, extract the JSON from any surrounding text by stripping markdown fences and leading prose. Next, run a repair pass that fixes trailing commas, single quotes, and unquoted keys. Then parse with a real JSON parser and reject anything that fails. Finally, validate the parsed object against a JSON Schema to confirm required fields, types, and enum values. Only data that survives all four layers should reach your business logic — never call JSON.parse on raw model text and act on the result directly.

A defense-in-depth pipelinetext
model output
  -> strip fences / leading prose
  -> repair (commas, quotes, keys)
  -> JSON.parse  (reject on failure)
  -> schema validate (types, required, enums)
  -> trusted object

Handling truncation and retries

Truncation needs its own handling because a cut-off object is not repairable in general — the missing data is simply gone. Detect it (the parser will report an unexpected end of input), then retry the call, ideally with a higher token limit or a smaller requested payload. For agents that loop, treat a validation failure as a signal to re-prompt the model with the error message so it can correct itself. Cap the number of retries to avoid runaway cost, and log failures so you can spot prompts that fail repeatedly.

Tools for debugging AI JSON

When a model's output fails in production, you need to see exactly where. A JSON Validator gives you the precise error location, a JSON Repair tool applies the common mechanical fixes, and a JSON Schema Validator confirms the structure matches your contract. JsonDrift is building AI-native repair and schema-enforcement tools on top of its private, client-side toolkit, so you can debug model output quickly without pasting potentially sensitive payloads into a third-party server.

Frequently asked questions

Should I fix LLM JSON with better prompts or with post-processing?

Both. Better prompts and structured-output modes reduce how often errors occur, while validation, repair, and schema checks guarantee safety regardless of how the model behaves on any given call. Treat prompting as prevention and post-processing as the guarantee — never rely on prompting alone in production.

Can I repair a truncated JSON response?

Not reliably. If the model stopped mid-object because it hit the token limit, the remaining data does not exist and cannot be reconstructed. Detect the truncation, then retry with a larger token budget or a smaller requested payload rather than trying to patch the fragment.

Does using a provider's JSON mode mean I can skip validation?

No. Structured-output and JSON modes guarantee well-formed syntax, but they do not guarantee the data is correct. The model can still return wrong values, miss optional context, or pick an out-of-range enum. Always validate against a schema even when generation is constrained.

What is schema drift and how do I catch it?

Schema drift is when output parses as valid JSON but does not match your expected structure — extra fields, missing required keys, or values of the wrong type. Catch it by validating every payload against a JSON Schema before acting on it, and fail loudly when a payload does not conform.

Why does the model wrap JSON in ```json fences even when I ask it not to?

Markdown fences are extremely common around code in training data, so the habit is strong. Reduce it by being explicit and giving an example with no fences, but always strip fences in code as a safety net rather than trusting the instruction to hold every time.

Related tools

Try these tools

Keep reading

Related guides