Structured output without JSON mode: 7 patterns that hold up

MMany model APIs now offer a JSON mode or schema-constrained decoding that guarantees syntactically valid output. When it is available and fits your use case, use it. But plenty of real situations do not have it: an open-weights model behind a minimal server, an older endpoint, a provider whose schema support excludes the features you need, or a pipeline that must work identically across several vendors.

Even with constrained decoding, valid JSON is not the same as *correct* JSON. The keys can be right while the values are wrong, empty or invented. This guide covers seven patterns for getting structured output that is both parseable and trustworthy, with or without JSON mode.

Why structured output fails

When a model is asked for JSON in plain text, failures fall into a few categories:

• Wrapper text. "Sure! Here is the JSON:" before the object, or an explanation after it.
• Markdown fences around the payload.
• Syntax errors: trailing commas, single quotes, unescaped newlines in strings, truncated output when the response hits the token limit.
• Schema drift: renamed keys, extra fields, strings where numbers were expected.
• Semantic errors: valid shape, wrong content, especially invented values for fields the input does not contain.

Patterns one to four mostly address syntax and shape. Patterns five to seven address correctness, which is the harder problem. For a comparison of constrained decoding and plain prompting, see structured outputs vs. plain prompts.

Pattern 1: show the schema, not a description of it

Models follow a concrete template more reliably than a prose description. Put a literal example of the target shape in the prompt, with types and allowed values written inline.

The null instruction matters. Without an explicit way to say "not present," models fill gaps with plausible inventions.

Pattern 2: fence the output with sentinels

Ask the model to put the payload between unique markers, then extract only what is between them. This removes wrapper text and markdown fences in one step and is robust across models.

import json, re

def extract_payload(text: str) -> dict:
    m = re.search(r"<<<JSON\s*(.*?)\s*JSON>>>", text, re.S)
    if not m:
        raise ValueError("no payload markers")
    return json.loads(m.group(1))

Instruct: "Write the JSON between <<<JSON and JSON>>> and nothing else." Even if the model adds commentary, your parser ignores it.

Pattern 3: keep the output shape small and flat

Deeply nested objects with many optional fields fail more often than flat ones. If you need a complex structure, consider:

• splitting extraction into two calls (first identify entities, then extract details for each);
• using arrays of small, uniform objects instead of nested dictionaries;
• moving derived fields into code rather than asking the model to compute them.

The more your schema looks like something the model has seen thousands of times, the more reliable it is.

Pattern 4: put free-text reasoning before the payload, not inside it

If the task needs reasoning, let the model think in a separate section first, then emit the structure. Mixing long reasoning into JSON string fields invites escaping errors, and forcing the model to answer in JSON immediately can lower quality on harder tasks. Research such as Tam et al. (2024), "Let Me Speak Freely?", reported that strict format restrictions can hurt reasoning performance on some tasks.

<analysis>
short reasoning in plain text
</analysis>
<<<JSON
{ ...payload... }
JSON>>>

Pattern 5: validate against a real schema in code

Parsing is not validation. Define the schema once, in code, and validate every response against it: types, required keys, enums, value ranges and cross-field rules (for example, qty must be positive).

from pydantic import BaseModel, Field
from typing import Literal

class Item(BaseModel):
    sku: str
    qty: int = Field(gt=0)

class Order(BaseModel):
    order_id: str | None
    items: list[Item]
    delivery: Literal["standard", "express"] | None
    complaint: bool

order = Order.model_validate(extract_payload(response_text))

Generate the prompt's schema example from the same model definition so the two never drift apart.

Pattern 6: repair, then retry, with the error message

When validation fails, send the model its own output plus the exact validation error and ask for a corrected version. A targeted repair succeeds far more often than a blind retry, and it is cheaper because the correction is short.

1. Try to parse and validate.
2. On failure, send: the original task, the previous output, and the error ("items[1].qty: must be greater than 0").
3. Validate again. After two failed repairs, stop and route the item to a fallback or a human.

Log every repair. A rising repair rate is an early warning that a prompt or model change has degraded quality.

Pattern 7: check values against the source

The hardest errors are well-formed but wrong. For extraction tasks, add cheap grounding checks in code:

• every extracted ID or quoted string should appear verbatim in the input;
• numbers should match a number present in the source;
• enums inferred from text ("express") should be justified by a keyword in the input.

When a value fails the check, set it to null or flag it for review rather than trusting it. This is the same principle as prompting for citations that check out: make the model's claims verifiable, then verify them.

Handling truncation and long outputs

A frequent and confusing failure is output that stops mid-object because the response hit the maximum output token limit. The JSON is not malformed because the model made a mistake; it simply ran out of room. Three habits prevent it:

• Set the output limit with headroom. Estimate the largest realistic payload and allow at least 50% more.
• Detect truncation explicitly. Most APIs report a stop reason such as "length" or "max_tokens." Treat that as a distinct error and retry with a higher limit or a smaller batch, rather than sending it to the repair step.
• Paginate large extractions. If a document contains hundreds of items, ask for them in pages ("items 1–50," then "items 51–100") or process the document in sections and merge the results in code.

Streaming makes truncation easier to spot early, and it lets you start validating items as they arrive when the output is a list.

Choosing a combination

If you add few-shot examples, pick them carefully; examples with filled-in values for every field teach the model never to use null. Our guide to few-shot examples that don't backfire covers this, and anatomy of a production system prompt shows where output-format instructions belong.

Frequently asked questions

How do I get an LLM to return valid JSON without JSON mode?

Show a literal schema example in the prompt, ask for the payload between unique delimiter markers, parse only what is between the markers, validate against a schema in code, and send validation errors back to the model for a targeted repair.

Does JSON mode guarantee correct data?

No. JSON mode and schema-constrained decoding guarantee valid syntax and, with schemas, the right shape. They do not guarantee that values are accurate or present in the source, so you still need validation and grounding checks.

Why does my model add text before the JSON?

Chat-tuned models are trained to be conversational. Delimiters solve this reliably because your parser ignores everything outside the markers, regardless of what the model adds.

Should I ask the model to explain its reasoning inside the JSON?

Usually not. Long free text inside JSON strings causes escaping errors. Put reasoning in a separate section before the payload, or skip it if the task is simple.