Structured Outputs from LLMs: JSON Mode vs Tools

An LLM outputs text. Your application needs structured data it can parse without guessing: JSON with predictable keys and types. The gap between those two facts is where most AI integration bugs live, and it’s why tool calls and structured outputs exist as a category of their own rather than “just ask nicely for JSON.”

The naive approach: ask the model to return JSON, parse the response, catch the errors. This works in development when you’re writing the prompts yourself and checking every output. It fails in production when the model wraps the JSON in a markdown code block, adds an explanation before it, or returns a slightly different structure than you specified.

The 2026 approach is different. Every major provider now exposes a structured output mechanism, tool calls with a forced schema being the most reliable of them, that constrains what the model can produce. Getting this right removes an entire category of runtime errors.

JSON Mode vs Structured Outputs

JSON mode and structured outputs are related but different: JSON mode only guarantees the response parses as JSON, with whatever keys and shape the model chooses. Structured outputs go further, forcing the response to match a schema you define, so the fields, types, and nesting are fixed and predictable every time.

JSON mode fixes parse errors. Structured outputs fix the undefined-property crashes that follow a successful parse, because the shape of the response is guaranteed rather than merely well-formed.

Concretely: with JSON mode, the model is still free to use any key names, any nesting structure, any field types. With structured outputs (function/tool calling with strict schema enforcement), if your schema says { name: string, age: number }, the model will always produce exactly that.

Structured outputs require more setup but are what you want for production systems.

Using Tool Calls for Data Extraction

The most reliable way to get structured data from a model today is through tool/function calling. You define a tool with an input schema, the model “calls” the tool, and you receive the arguments as a parsed object.

Structured outputs pair naturally with the hallucination mitigations covered in our guide to reducing LLM hallucination in production: a forced tool call is itself a form of constrained generation, since the model cannot wander into free text where a factual claim might be invented.

In TypeScript with the Anthropic SDK, using Anthropic’s tool use API:

import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic();

const extractContactInfo = async (text: string) => {
  const response = await client.messages.create({
    model: 'claude-opus-4-6',
    max_tokens: 1024,
    tools: [
      {
        name: 'extract_contact',
        description: 'Extract contact information from text',
        input_schema: {
          type: 'object',
          properties: {
            name: { type: 'string', description: 'Full name' },
            email: { type: 'string', description: 'Email address' },
            phone: { type: 'string', description: 'Phone number with country code' },
            company: { type: 'string', description: 'Company or organization name' },
          },
          required: ['name'],
        },
      },
    ],
    tool_choice: { type: 'tool', name: 'extract_contact' },
    messages: [
      {
        role: 'user',
        content: `Extract the contact information from this text:\n\n${text}`,
      },
    ],
  });

  const toolUse = response.content.find(block => block.type === 'tool_use');
  if (!toolUse || toolUse.type !== 'tool_use') {
    throw new Error('Model did not call the expected tool');
  }

  return toolUse.input as {
    name: string;
    email?: string;
    phone?: string;
    company?: string;
  };
};

const result = await extractContactInfo(
  'Please contact Sarah Chen at sarah@acme.com or +1-555-0123 for more information about Acme Corp.'
);

console.log(result);
// { name: 'Sarah Chen', email: 'sarah@acme.com', phone: '+1-555-0123', company: 'Acme Corp' }

tool_choice: { type: 'tool', name: 'extract_contact' } forces the model to call that specific tool. Without this, the model may decide to respond with text instead.

With OpenAI’s structured outputs API:

import OpenAI from 'openai';
import { zodResponseFormat } from 'openai/helpers/zod';
import { z } from 'zod';

const client = new OpenAI();

const ContactSchema = z.object({
  name: z.string(),
  email: z.string().email().optional(),
  phone: z.string().optional(),
  company: z.string().optional(),
});

const response = await client.beta.chat.completions.parse({
  model: 'gpt-4o-2024-11-20',
  messages: [
    {
      role: 'user',
      content: `Extract contact info: ${text}`,
    },
  ],
  response_format: zodResponseFormat(ContactSchema, 'contact'),
});

const contact = response.choices[0].message.parsed;
// TypeScript knows the type is z.infer<typeof ContactSchema>

OpenAI’s structured outputs guarantee the output matches your Zod schema. The parsed field is already typed.

The practical difference between the two SDK styles above is where the guarantee lives. Anthropic’s tool-calling approach guarantees the model invokes the named tool with arguments matching the tool’s schema. OpenAI’s structured outputs API bakes the schema constraint directly into generation, so the response is guaranteed to validate against the Zod schema before your code ever sees it. Both remove the same category of bug: a model that answers in prose, or in JSON shaped differently than you asked for.

Schema Validation on the Output

Even with structured output APIs, runtime validation is worth doing. The model’s output passes the provider’s schema validation, but your application’s logic may have additional constraints that schemas can’t express cleanly.

import { z } from 'zod';

const LineItemSchema = z.object({
  description: z.string().min(1),
  quantity: z.number().int().positive(),
  unit_price: z.number().positive(),
  total: z.number().positive(),
});

const InvoiceSchema = z.object({
  invoice_number: z.string(),
  vendor: z.string(),
  date: z.string().regex(/^\d{4}-\d{2}-\d{2}$/),
  line_items: z.array(LineItemSchema).min(1),
  subtotal: z.number().positive(),
  tax: z.number().nonnegative(),
  total: z.number().positive(),
});

// Validate and get typed result
const parseInvoice = (rawOutput: unknown) => {
  const result = InvoiceSchema.safeParse(rawOutput);
  if (!result.success) {
    console.error('Invoice validation failed:', result.error.flatten());
    return null;
  }
  return result.data;
};

The safeParse approach avoids thrown exceptions and gives you a clean error path when something unexpected comes through. The same pattern, define a schema once and reuse it for both the tool definition and the runtime check, is covered in more depth in our Zod schema validation guide if you’re building this on the TypeScript side.

Provider Differences That Change How You Write the Schema

“Structured outputs” is not one API. Each provider enforces the schema differently, and the differences change what you can safely put in a schema definition:

OpenAI’s strict mode requires every property to be listed in required and forbids additionalProperties. There is no way to say “this field is optional” the way JSON Schema normally allows; instead you make a field nullable (["string", "null"]) and treat a returned null as the equivalent of absent. Deeply nested anyOf unions are also more restricted than plain JSON Schema allows, so a schema that validates fine with a generic JSON Schema validator can still be rejected by the strict-mode compiler.

Anthropic’s tool use takes a standard JSON Schema for input_schema and is more permissive about optional fields: anything not listed in required is genuinely optional, and the model can omit it rather than being forced to emit a null placeholder. This makes Anthropic’s approach a better fit for schemas with many optional fields, at the cost of the model occasionally omitting a field OpenAI’s stricter mode would have forced it to address.

Smaller and self-hosted models vary widely. Some support grammar-constrained decoding (forcing the token-level output to match a JSON grammar) through libraries like Outlines or the vLLM guided_json parameter; others only support prompt-based JSON mode with no hard guarantee at all. Before assuming a schema will be enforced, check whether the specific model and serving stack you’re using does constrained decoding or just prompts for compliance.

Porting a schema between providers without checking these differences is a common source of “it worked with OpenAI but breaks on Claude” bug reports. The schema shape that compiles under one provider’s strict mode isn’t automatically valid under another’s.

Python with Pydantic

from pydantic import BaseModel, Field
from typing import Optional
import anthropic

class SentimentResult(BaseModel):
    sentiment: str = Field(description="positive, negative, or neutral")
    confidence: float = Field(ge=0.0, le=1.0)
    key_phrases: list[str] = Field(description="phrases that drove the sentiment")
    summary: str

client = anthropic.Anthropic()

def analyze_sentiment(text: str) -> SentimentResult:
    response = client.messages.create(
        model="claude-opus-4-6",
        max_tokens=512,
        tools=[
            {
                "name": "report_sentiment",
                "description": "Report sentiment analysis results",
                "input_schema": SentimentResult.model_json_schema(),
            }
        ],
        tool_choice={"type": "tool", "name": "report_sentiment"},
        messages=[{"role": "user", "content": f"Analyze the sentiment: {text}"}],
    )

    tool_use = next(
        block for block in response.content if block.type == "tool_use"
    )

    return SentimentResult.model_validate(tool_use.input)

result = analyze_sentiment("The product works but customer support took 3 days to respond.")
print(result.sentiment)      # negative
print(result.confidence)     # 0.72
print(result.key_phrases)    # ['customer support', '3 days to respond']

model_json_schema() generates a JSON Schema from the Pydantic model, which the tool definition uses. model_validate() validates the model’s output against the Pydantic model with full type checking; our Pydantic v2 validation guide covers the validator patterns worth knowing if you’re using this class for more than tool schemas.

Provider-Specific Constraints

Not every schema feature is supported the same way across providers, and this is where structured-output code that worked in a demo breaks in production.

Anthropic’s forced tool_choice is close to a hard guarantee that the named tool gets called, documented in the tool use guide, but the JSON Schema in input_schema still needs to be something the model can actually fill in: deeply nested optional unions produce worse results than flatter schemas, even though the API accepts them syntactically.

OpenAI’s strict structured outputs mode rejects several JSON Schema keywords outright, among them minLength, maxLength, and pattern on strings, and it requires every property to appear in required (optional fields have to be modeled as nullable types instead of simply omitted). A schema that works fine with ordinary tool calling can fail validation in strict mode until those constraints are flattened out.

Gemini’s function calling accepts a subset of the OpenAPI schema format rather than full JSON Schema, so a schema generated by a tool like Zod needs a translation step, not a direct pass-through, before it’s valid for a Gemini tool definition. The lesson in all three cases is the same: test the schema against the specific provider you’re shipping to, since something that validates fine offline against the JSON Schema spec can still be rejected by a provider’s narrower subset.

If schema validation keeps failing for reasons that trace back to the model inventing values it shouldn’t, that’s usually a hallucination problem wearing a validation-error costume; mitigating hallucination in production covers the layers that catch it before it reaches your schema check.

Handling Model Refusals and Partial Outputs

Structured output modes reduce but don’t eliminate the need to handle bad outputs. Models can still refuse to fill in fields when information isn’t present, return null for required fields, or hit token limits mid-structure.

const extractWithRetry = async (text: string, maxAttempts = 2) => {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      const result = await extractContactInfo(text);
      
      // Business logic validation the schema can't catch
      if (!result.name || result.name.trim().length === 0) {
        if (attempt < maxAttempts - 1) continue;
        return null;
      }
      
      return result;
    } catch (err) {
      if (attempt === maxAttempts - 1) return null;
    }
  }
  return null;
};

Keep retry counts low. One retry is usually appropriate. More than two suggests your prompt or schema needs work, not more retries. If retries or fallback structures are becoming a real pattern in your codebase, it’s worth reading them against the broader hallucination mitigation playbook, since a model that won’t fill in a structured field reliably is often exhibiting the same underlying uncertainty a hallucination check would catch.

The Schema Is Reusable Infrastructure, Not Just a Prompt Detail

A schema you write for structured outputs isn’t throwaway prompt scaffolding. The same JSON Schema that constrains a tool call is what Model Context Protocol servers use to describe tools to any MCP-compatible client, and it’s the same schema shape OpenAPI and most API gateways already expect. Defining it once, in Pydantic or Zod, and generating the JSON Schema from that definition means the extraction tool, the API contract, and the documentation all stay in sync instead of drifting apart as the feature evolves.

This matters more as a codebase adds more of these calls. A team that hand-writes a fresh JSON Schema for every extraction task ends up maintaining the same field definitions in three or four places: the schema, the validation layer, the TypeScript or Python types, and the docs. A team that generates the schema from one typed definition maintains it in one place, and the other three follow automatically.

When to Use Which Approach

Use tool/function calling when you need guaranteed structure, the schema is specific, and you’re building a production feature that will run at volume.

Use JSON mode when you need valid JSON but the structure is flexible, or when you’re prompting an older model that doesn’t support structured outputs well.

Use prompt-only JSON extraction when the schema is very simple (1-2 fields), you’re prototyping, or the model is a small local model where tool calling support is limited.

Use streaming with structured outputs for long-form structured responses where you need to show progress. Most providers support streaming even with schema constraints, the JSON builds incrementally, and you parse when the stream completes. For the retry and validation patterns that apply once you’re past the basic tool-call setup, see our broader guide to reliable JSON from LLMs.

The investment in structured outputs pays off quickly. The parser errors, the “model responded with markdown instead of JSON” bugs, the undefined-access crashes on missing fields, these stop happening. What remains is prompt work and schema design, which are problems worth solving once rather than debugging indefinitely in production. If the extracted data is feeding a RAG pipeline downstream, the same schema discipline applies to whatever you retrieve and inject back into context, not just what the model returns.