The Structured Output Problem
LLMs generate free-form text by default. When your application needs the output in a specific format — JSON for an API response, a list with defined fields, a table with precise columns — getting reliable structured output requires deliberate engineering. The naïve approach of asking the model to “respond in JSON” works most of the time but fails unpredictably: occasionally the model adds a preamble before the JSON, wraps it in markdown code fences, produces malformed JSON with trailing commas, or decides mid-response to add an explanatory note after the closing brace. These failures are infrequent but when they occur in a production system that parses the output programmatically, they cause errors that are hard to reproduce and debug.
Three approaches reliably solve this: JSON mode (where supported), tool forcing, and output parsers with retry logic. Understanding when each applies and how to implement each correctly is one of the most practically important skills in LLM application development.
JSON Mode
OpenAI’s API supports a JSON mode that constrains the model to only produce valid JSON output. Enable it by setting response_format: {"type": "json_object"} in your API call. When JSON mode is active, the model will always return parseable JSON — it will never add preamble text or explanation before or after the JSON object. The catch: JSON mode does not constrain the schema of the JSON, only that it is valid JSON. You will always get parseable output, but the fields, nesting, and values within that JSON are still determined by your prompt. Describe your desired schema clearly in the system prompt and include an example of the expected output format.
Claude’s API does not have a direct JSON mode equivalent, but tool forcing achieves the same guarantee with schema control. Gemini’s API provides both a JSON mode and a more powerful structured output mode that enforces a specific schema using a provided JSON Schema definition — the strongest built-in structured output mechanism available from any major provider.
Tool Forcing: The Most Reliable Approach
Tool forcing — setting tool_choice to require a specific function call — is the most reliable way to guarantee structured output conforming to a specific schema. The model is required to populate the function’s parameter schema with values, and the API validates that the output conforms to the schema before returning it. You receive a structured object, not text — no parsing required, schema violations are rejected at the API level.
For extraction tasks, define a function whose parameter schema matches your desired output structure exactly. For a contact extraction task, define a function with fields for name, email, phone, and company. Force the model to call this function, and the output is always a valid object with those fields. This works identically across OpenAI (via tool_choice: {"type": "function", "function": {"name": "..."}}) and Anthropic (via tool_choice: {"type": "tool", "name": "..."}). The schema validation guarantee is absolute — you will never receive output that doesn’t conform to your schema.
Pydantic and the Instructor Library
The instructor library wraps the major LLM APIs to make structured output extraction using Pydantic models clean and automatic. Define your desired output as a Pydantic model, pass it to instructor-wrapped client, and receive a validated Pydantic object:
import anthropic, instructor
from pydantic import BaseModel
client = instructor.from_anthropic(anthropic.Anthropic())
class ContactInfo(BaseModel):
name: str
email: str | None
company: str | None
role: str | None
contact = client.messages.create(
model="claude-sonnet-4-6", max_tokens=512,
response_model=ContactInfo,
messages=[{"role":"user","content":
"Extract contact: Hi, I'm Sarah Chen, Head of Engineering at Acme Corp. Email me at sarah@acme.com"}]
)
print(contact.name) # "Sarah Chen"
print(contact.email) # "sarah@acme.com"Instructor handles tool definition, response parsing, Pydantic validation, and automatic retry on validation failure — if the model produces output that fails Pydantic validation, instructor retries with the validation error added to the context, typically resolving in one retry. For any application doing systematic structured data extraction, instructor is the recommended approach.
Figure 1 — Structured Output Methods Compared
Output Parsers with Retry Logic
For models or scenarios where tool forcing is not available, robust output parsers with retry logic handle the failure cases that plain prompting cannot guarantee away. The pattern: attempt to parse the model’s output, catch parsing failures, and retry with the error included in the prompt context so the model can self-correct. LangChain’s OutputFixingParser implements this automatically — it wraps any parser and adds a retry call with the parsing error if the first attempt fails. For most applications, one retry is sufficient to recover from the occasional format failure that even well-prompted models produce.
Nested and Complex Schema Extraction
Extracting complex nested structures — a contract with multiple parties, each with their own obligations and dates — works well with tool forcing but requires careful schema design. Deeply nested schemas with many optional fields are harder for models to populate correctly than flat or shallowly nested ones. When possible, break complex extractions into multiple targeted calls rather than one massive schema: extract parties first, then obligations, then dates. Each extraction is simpler, the model makes fewer errors, and the results are easier to validate. The added API cost of multiple calls is usually justified by the improved extraction accuracy on complex documents.
Validating Extracted Data Beyond Schema
Schema validation (is the JSON valid? do the required fields exist?) is necessary but not sufficient for production extraction pipelines. Semantic validation — does the extracted content actually make sense? — requires additional checks. An email field that contains “Not provided” instead of an email address passes schema validation but is semantically wrong. A date field populated with “2023-13-45” passes string validation but is not a valid date. Build validation layers that check semantic correctness beyond schema conformance: email format validation, date parsing validation, cross-field consistency checks (the end date should not precede the start date), and domain-specific business logic checks. These validation failures, returned to the model with a specific error message, enable targeted self-correction rather than generic retry that may reproduce the same error.
Streaming and Structured Outputs
JSON mode and tool forcing are incompatible with streaming in most implementations — the schema validation happens at the end of generation, so you cannot stream partial output and validate it simultaneously. For applications that need both streaming and structured output, the practical approach is to stream the generation for user feedback (showing the model “thinking”), then parse the complete output once generation is done. Alternatively, design your structured extraction as a non-streaming background call while streaming a separate narrative response to the user — the structured data is processed silently while the user sees the conversational output, and the extracted data is used to update application state once available.
Testing Structured Output Pipelines
Structured output pipelines require specific testing strategies beyond standard LLM evaluation. Test your schema coverage: for each field in your schema, test inputs that should and should not trigger population of that field. Test your validation logic: generate intentionally malformed outputs and verify that validation catches them. Test your retry logic: simulate parsing failures and verify that the retry mechanism produces correct output. Test edge cases in the input: empty documents, documents in unexpected formats, documents where the target information is absent. And test at realistic volume to find the failure rate — even well-designed structured output pipelines have non-zero failure rates, and understanding your actual failure rate under production conditions is essential for deciding how much human review is needed before structured outputs are used in downstream systems.
When to Use Each Approach
JSON mode is appropriate for simple use cases where the schema is flexible and any valid JSON is acceptable — quick prototyping, simple key-value extraction, cases where you define the schema loosely in the prompt. Tool forcing is the production standard for most structured extraction tasks — it provides schema guarantees, works across providers, and integrates naturally with the function-calling patterns already used in agentic applications. Instructor with Pydantic is the recommended approach for Python applications doing systematic extraction where type safety, validation, and automatic retry are valuable — it adds minimal overhead and significantly improves reliability. Avoid prompt-only structured output in production: the failure rate is unpredictable, parsing errors cause application failures at the worst moments, and the alternatives are both reliable and easy to implement. Structured output is a solved problem in 2026 — the tools exist to make it reliable, and there is no good reason to rely on fragile string parsing of LLM text responses in a production application.
Structured Outputs in Agentic Pipelines
Structured outputs are foundational to reliable agentic systems. When an agent must make a decision — which tool to call next, how to route a query, whether a task is complete — expressing that decision as a structured object rather than natural language makes the downstream logic deterministic rather than dependent on string matching. An agent that returns {"action": "search", "query": "Q3 revenue figures", "confidence": 0.87} is far more reliable to build downstream logic around than one that returns “I think I should search for the Q3 revenue figures next.” Build structured decision objects into your agent architecture from the start — the debugging, testing, and reliability improvements are substantial compared to parsing natural language agent outputs.
The investment in structured output infrastructure — tool definitions, Pydantic schemas, validation logic — pays dividends across every part of an agentic system that depends on the model’s outputs being machine-readable, not just human-readable.