Structured Extraction: Documents In, Validated JSON Out

Invoices, applications, contracts, intake forms — a huge share of real-world AI value is one shape: unstructured document in, structured record out. Modern models are genuinely good at this. What separates a demo from a production system is everything wrapped around the model call: a schema that encodes your rules, a validation loop that catches bad output before your database does, confidence that means something, and a router that knows when to hand a document to a human.

01The schema is the contract

Design the target schema the way you'd design an API: explicit types, enums for anything with a known value set, and — critically — room for honest uncertainty. A schema that forces a value for every field forces hallucination on every missing field. Make nullable fields genuinely nullable, and require the model to flag what it couldn't find rather than improvise it.

from datetime import date
from typing import Literal
from pydantic import BaseModel, Field

class LineItem(BaseModel):
    description: str
    quantity: float = Field(gt=0)
    unit_price_cents: int = Field(ge=0)

class InvoiceExtraction(BaseModel):
    vendor_name: str
    invoice_number: str
    invoice_date: date
    due_date: date | None = None          # genuinely optional in the wild
    currency: Literal["USD", "EUR", "GBP"]
    total_cents: int = Field(ge=0)
    line_items: list[LineItem]
    # honest-uncertainty channel — the model reports rather than invents
    missing_fields: list[str] = []
    ambiguities: list[str] = []           # "two dates present; used the later"

    def consistency_errors(self) -> list[str]:
        errs = []
        items_total = sum(
            round(li.quantity * li.unit_price_cents) for li in self.line_items
        )
        if self.line_items and abs(items_total - self.total_cents) > 1:
            errs.append(
                f"line items sum {items_total} != stated total {self.total_cents}"
            )
        if self.due_date and self.due_date < self.invoice_date:
            errs.append("due_date precedes invoice_date")
        return errs

02The validation loop: let the error message do the prompting

The single highest-leverage pattern in extraction: when validation fails, don't just retry — feed the validator's error message back to the model and ask it to correct its own output. Pydantic's errors are precise ("total_cents: input should be a valid integer"), and models are markedly better at fixing a named mistake than at avoiding all mistakes in one shot. Two rounds of this loop resolve the large majority of validation failures; what survives the loop is genuinely hard and belongs with a human.

Note the layering: schema validation (types, enums, ranges) is mechanical; consistency checks (line items sum to total, dates ordered) encode business rules the type system can't see. Run both, feed both back.

from pydantic import ValidationError

MAX_REPAIRS = 2

def extract_invoice(document_text: str, call_llm) -> tuple[str, object]:
    """Returns (route, result): route is 'auto' | 'review' | 'reject'."""
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},   # schema + rules + examples
        {"role": "user", "content": document_text},
    ]
    last_error = None
    for attempt in range(1 + MAX_REPAIRS):
        raw = call_llm(messages, json_schema=InvoiceExtraction)  # provider seam
        try:
            parsed = InvoiceExtraction.model_validate_json(raw)
        except ValidationError as e:
            last_error = f"Schema errors:\n{e}"
        else:
            issues = parsed.consistency_errors()
            if not issues:
                return route_by_confidence(parsed), parsed
            last_error = "Consistency errors:\n- " + "\n- ".join(issues)
        # feed the exact failure back and let the model repair its output
        messages.append({"role": "assistant", "content": raw})
        messages.append({"role": "user", "content":
            last_error + "\nReturn corrected JSON only. If a value is not in "
            "the document, use null and add the field to missing_fields — "
            "never invent it."})
    return "reject", last_error   # exhausted repairs → human queue with context

def route_by_confidence(parsed: InvoiceExtraction) -> str:
    if parsed.missing_fields or parsed.ambiguities:
        return "review"           # honest uncertainty → human eyes
    return "auto"

03Confidence you can act on

Raw model self-confidence ("rate your certainty 1–10") is poorly calibrated and shouldn't gate anything alone. Production confidence is an ensemble of signals you can measure: did validation pass on the first attempt or need repairs? Did the model populate missing_fields or ambiguities? Do cheap independent checks agree (regex-extract the invoice number and compare)? For high-stakes fields, does a second extraction pass — same document, different prompt phrasing — produce the same value?

Each signal is weak alone; together they partition documents into a clean-pass majority that flows straight through and a flagged minority that earns human review. Tune the thresholds against your golden set until the auto lane's field-level error rate is below the error rate of your manual process — that's the honest bar, since humans mistype too.

04The routing tiers

Every reviewer correction in the review lane is gold: log the (document, model output, human correction) triple. That log is simultaneously your error analysis, your regression eval set, and your evidence for which prompt changes actually help.

• ▸Auto lane. First-pass validation, no uncertainty flags, checks agree. Writes to the system of record with an extraction-audit row (document hash, model, output, signals) for later reconstruction.
• ▸Review lane. Validation needed repairs or the model flagged uncertainty. The UI shows the document and extraction side by side with flagged fields highlighted — the reviewer confirms in seconds because attention is pre-aimed.
• ▸Reject lane. Failed the repair loop, or the document isn't the expected type at all. These route to the human process with the error context attached — and get logged as candidates for prompt or schema fixes.

05Evaluating extraction like an engineer