Traditional software fails loudly: exceptions, 500s, alerts. LLM workflows fail politely — the response arrives on time, well-formatted, and wrong. Nothing in your existing monitoring stack notices, because nothing is broken in the way that stack understands. The consequence: observability for LLM systems has to be designed in, and it has to capture semantics (what was asked, what was answered, was it good) rather than just mechanics (status codes, latency).
The architecture has four layers: traces (capture everything), cost accounting (know what each workflow spends), offline evals (measure quality on a fixed set before users do), and drift detection (notice when production quietly changes). None requires a vendor platform to start — a Postgres table and discipline cover the first six months.
01Traces: capture enough to replay
The unit of LLM observability is the workflow run, composed of steps (model calls, retrievals, tool executions). For every model call, capture: the full input (or a pointer to it), the full output, model and version, token counts, latency, and the workflow run it belongs to. The test of sufficiency is replayability — given the trace, could you re-run the exact call and diff the result? If not, you're logging vibes.
Prompt templates change over time, so store a hash of the rendered prompt alongside a template id + version. When quality shifts, the first question is always "what changed?" — and prompt-hash frequencies answer it in one query.
CREATE TABLE runs (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
workflow text NOT NULL, -- 'invoice_extract', 'kb_answer', ...
trigger_ref text, -- ticket id, document id, user id
status text NOT NULL DEFAULT 'running'
CHECK (status IN ('running','ok','error')),
started_at timestamptz NOT NULL DEFAULT now(),
ended_at timestamptz
);
CREATE TABLE steps (
id bigserial PRIMARY KEY,
run_id uuid NOT NULL REFERENCES runs(id),
kind text NOT NULL, -- 'llm', 'retrieval', 'tool'
name text NOT NULL, -- 'draft_reply', 'hybrid_search'
model text, -- provider/model@version for llm steps
prompt_id text, -- template id + semver
prompt_hash text, -- sha256 of rendered input
input jsonb NOT NULL,
output jsonb,
tokens_in int,
tokens_out int,
cost_microusd bigint, -- integers; float money is a bug
latency_ms int,
error text,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX steps_run_idx ON steps (run_id);
CREATE INDEX steps_cost_idx ON steps (name, created_at);import contextvars, functools, hashlib, json, time
current_run = contextvars.ContextVar("current_run", default=None)
def traced_step(kind: str, name: str):
"""Wrap any pipeline step; captures i/o, latency, errors to the ledger."""
def deco(fn):
@functools.wraps(fn)
def wrapper(*args, **kwargs):
run_id = current_run.get()
t0 = time.monotonic()
error, output = None, None
try:
output = fn(*args, **kwargs)
return output
except Exception as e:
error = repr(e)
raise
finally:
record_step( # single INSERT into steps
run_id=run_id, kind=kind, name=name,
input=safe_json(args, kwargs),
output=safe_json(output),
prompt_hash=hashlib.sha256(
json.dumps(safe_json(args, kwargs),
sort_keys=True).encode()
).hexdigest()[:16],
latency_ms=int((time.monotonic() - t0) * 1000),
error=error,
)
return wrapper
return deco
# usage:
# @traced_step("llm", "draft_reply")
# def draft_reply(ticket: dict) -> str: ...02Cost accounting: per workflow, not per invoice
The provider bill tells you what you spent; it can't tell you what you spent it on. With per-step token and cost capture, the question "which workflow, which step, which prompt version is the money going to?" is a GROUP BY — and the answers are routinely surprising: one noisy retrieval-summarize step quietly costing more than the flagship drafting step, a retry loop nobody remembers doubling a workflow's cost. Store cost in integer micro-dollars at write time, computed from the model's current rates; rates change, and back-computing historical cost from token counts is how finance meetings go wrong.
03Offline evals: your quality regression suite
An eval is to an LLM workflow what a test suite is to code: a fixed set of real cases, a scoring rule, and a number that must not go down. The discipline that matters most is scorer selection — use the cheapest scorer that captures the failure you fear, and layer them:
- ▸Mechanical scorers first. Schema validity, required fields present, citation indices in range, length bounds, banned-phrase checks. Free, deterministic, and they catch a surprising share of regressions.
- ▸Reference scorers second. Exact or fuzzy match against golden answers for extraction and classification; recall@k for retrieval. Deterministic where your task allows it.
- ▸LLM-as-judge last, with hygiene. For fluency, tone, and answer quality there's no mechanical scorer — a judge model scoring against a written rubric works, but treat it as a noisy instrument: pin the judge model version, randomize A/B presentation order to dodge position bias, spot-check judge verdicts against human ratings quarterly, and never let the judge grade its own model family without checking for self-preference.
import json, statistics
def run_eval(golden_path: str, workflow_fn, scorers: list) -> dict:
"""golden: [{\"input\": ..., \"expected\": ...}]; scorer -> float in [0,1]"""
cases = json.load(open(golden_path))
rows = []
for case in cases:
output = workflow_fn(case["input"])
scores = {s.__name__: s(output, case) for s in scorers}
rows.append({"case": case.get("id"), "scores": scores,
"output": output})
summary = {
s.__name__: round(statistics.mean(r["scores"][s.__name__]
for r in rows), 3)
for s in scorers
}
worst = sorted(rows, key=lambda r: sum(r["scores"].values()))[:5]
return {"summary": summary, "n": len(rows), "worst_cases": worst}
# scorers stay tiny and composable:
def schema_valid(output, case) -> float:
try:
TicketReply.model_validate_json(output); return 1.0
except Exception:
return 0.0
def cites_real_sources(output, case) -> float:
reply = TicketReply.model_validate_json(output)
valid = set(case["retrieved_ids"])
return 1.0 if all(c in valid for c in reply.citations) else 0.004Drift: the failure mode with no error message
Production drifts three ways: inputs drift (the documents users submit change shape), behavior drifts (a provider updates the model behind your API), and standards drift (reviewers get pickier or sloppier). None throws an exception. All show up in trends you can query — if you're capturing traces and review outcomes.
The most reliable early-warning signals, in practice: the human override rate on reviewed outputs (rising = quality falling), the validation-repair rate in extraction loops, the escalation-lane share, and week-over-week token-length distributions of inputs (a cheap proxy for "the inputs changed"). Put them on one dashboard query and look at it weekly — drift caught in week two is a prompt fix; drift caught in month three is a credibility incident.
- ▢Every model call in production writes a trace row — no exceptions for "quick internal tools."
- ▢Evals run in CI on every prompt/model/retrieval change, and the summary lands in the PR.
- ▢The golden set grows from production: every human override becomes a candidate case.
- ▢Someone owns the weekly drift query the way someone owns the on-call rotation.
SELECT date_trunc('week', r.started_at) AS week,
r.workflow,
count(*) AS reviewed,
round(avg(CASE WHEN rv.outcome = 'overridden'
THEN 1.0 ELSE 0.0 END), 3) AS override_rate,
round(avg(s.latency_ms)) AS avg_latency_ms,
round(sum(s.cost_microusd) / 1e6, 2) AS cost_usd
FROM runs r
JOIN steps s ON s.run_id = r.id AND s.kind = 'llm'
JOIN reviews rv ON rv.run_id = r.id -- human decisions on outputs
WHERE r.started_at > now() - interval '8 weeks'
GROUP BY 1, 2
ORDER BY 2, 1;
-- Alert rule of thumb: override_rate doubling vs its 4-week median
-- is a page-someone signal, whatever the absolute number is.OPERATOR NOTE — Vendors will sell you all of this as a platform, and later you may want one. But the discipline is the product: a team that won't maintain a golden set won't be saved by a dashboard.
Put this architecture to work.
A 30-minute strategy call with an operator — we'll map your first deployment path, not send a deck.