Structured outputs vs. plain prompts: when constrained decoding is worth it

Getting a language model to return machine-readable data used to be a matter of careful prompting and defensive parsing. Today, many APIs and inference servers offer structured outputs: you provide a JSON schema, and the system guarantees the response matches it. It sounds like the problem is solved.

It is partly solved. Constrained decoding eliminates syntax errors and schema violations, which removes a whole class of production failures. It does not make values correct, it can change model behaviour in subtle ways, and it is not available everywhere. This guide explains how structured outputs work, compares them with plain prompting, and gives a test plan to decide which to use for your task.

Three levels of structure

Tool or function calling is a closely related mechanism: the model fills in arguments for a declared function signature, often using the same constrained decoding under the hood.

How constrained decoding works

At each step, the model produces probabilities for every possible next token. With constrained decoding, the system tracks where the output currently is within the schema, for example "inside a string value for the key status, which must be one of three options," and masks out every token that would make the output invalid. The model then picks from the remaining tokens.

Open-source libraries and inference engines implement this with grammars or finite-state machines compiled from the schema; llama.cpp, for instance, supports grammar files, and several serving frameworks support JSON schemas directly. Hosted APIs implement their own versions, often with a list of supported schema features.

The result is a hard guarantee: if the response completes, it parses and matches the schema.

What structured outputs do not fix

• Wrong values. The schema says amount is a number; it cannot know whether 129.00 is the right number.
• Invented values. If a field is required, the model must fill it, even when the source does not contain the information. Make fields nullable where information may be missing.
• Truncation. If the response hits the maximum output length, you can still get an incomplete object. Check the stop reason.
• Refusals and errors. Some APIs return refusals in a separate field; handle that path explicitly.

Validation of values and grounding checks in code remain essential. The techniques in structured output without JSON mode, especially schema validation and checking values against the source, apply equally with constrained decoding.

Can constraints hurt quality?

There is evidence that strict format requirements can reduce performance on some tasks. Tam et al. (2024), "Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models," reported that requiring structured formats such as JSON degraded results on several reasoning tasks compared with free-form answers, while classification-style tasks were less affected or improved.

Plausible reasons include:

• No room to reason. When the model must start with {"answer":, it commits to an answer before thinking.
• Token masking. Forcing tokens the model considered unlikely can push it off its preferred path, especially with unusual key names or orderings.
• Key order matters. Placing a reasoning field before the answer field lets the model think first; placing it after is pointless.

The practical takeaway is not to avoid structured outputs but to design schemas that leave room for thinking when the task needs it, and to measure.

Designing schemas that work well

• Put reasoning fields first when the task needs reasoning, then the answer fields.
• Use descriptive key names close to natural language (delivery_date, not dd).
• Use enums for fixed choices, with clear values.
• Make fields nullable when the source may not contain them.
• Keep nesting shallow and avoid huge optional schemas.
• Add descriptions to schema fields if the API passes them to the model.

A test plan: plain prompting vs. structured outputs

1. Build a set of 100 to 200 real inputs with correct expected outputs. See build an eval set in an afternoon.
2. Run three variants with the same instructions: plain prompting with a schema example, JSON mode, and schema-constrained structured outputs.
3. Use the same temperature for all variants, ideally your production setting; see LLM temperature explained.
4. Measure separately:

• parse success rate (did it produce valid JSON?);
• schema compliance (right keys and types?);
• field-level accuracy (are the values correct?);
• null handling (does it use null when the source lacks a value?);
• latency and cost.

1. Read the differences. Look at items where variants disagree; that is where the trade-offs live.

Operational details that bite

A few practical issues come up repeatedly when teams move to structured outputs:

• Schema feature support varies. Providers often support a subset of JSON Schema. Features such as certain string patterns, recursive definitions or complex combinations may be rejected or silently ignored. Test the exact schema you plan to ship.
• First-request latency. Some systems compile the schema into a grammar on first use, which adds a delay. Subsequent requests with the same schema are usually faster, so reuse schemas rather than generating new ones per request.
• Schema changes are API changes. Adding a required field or renaming a key changes model behaviour and downstream code at once. Version schemas, and run your eval set before deploying a change.
• Streaming partial JSON. If you stream structured responses, use a parser that tolerates incomplete objects, or wait for completion before acting on the data.

Recommendations by situation

• Extraction and classification with a supported API: use structured outputs, with nullable fields and value validation.
• Reasoning-heavy tasks: add a reasoning field first, or reason in free text and extract the structure in a second, cheap call.
• Small or open models without schema support: use plain prompting with delimiters, validation and repair.
• Multi-vendor systems: keep a schema definition in code, and generate both the prompt example and the provider-specific schema from it.

Wherever the output format is defined, keep it in a predictable place in the system prompt; see anatomy of a production system prompt.

Frequently asked questions

What are structured outputs in LLM APIs?

Structured outputs are a feature where you supply a JSON schema and the system constrains the model's generation so the response is valid JSON matching that schema, typically by masking tokens that would break the schema.

What is the difference between JSON mode and structured outputs?

JSON mode guarantees syntactically valid JSON but not a particular shape. Structured outputs guarantee the response matches your schema: the required keys, types and allowed values.

Do structured outputs reduce model quality?

On some reasoning tasks, strict formats have been shown to reduce accuracy, partly because the model must commit to an answer early. Placing a reasoning field before the answer, or separating reasoning from formatting, usually reduces the effect.

Do I still need validation if I use structured outputs?

Yes. Structured outputs guarantee format, not correctness. Validate value ranges and cross-field rules, and check extracted values against the source.