Retrieval Architecture for Internal Knowledge Assistants

Every internal knowledge assistant is a retrieval system wearing a chat interface. The model matters far less than what you put in its context window — a frontier model with the wrong three paragraphs loses to a mid-tier model with the right ones, every time. This schematic covers the retrieval architecture that holds up in production: how to chunk, how to search, how to fuse and rerank, and how to prove the answers came from somewhere.

The reference stack here is Postgres with the pgvector extension plus built-in full-text search. Not because dedicated vector databases are bad — because most mid-market deployments already run Postgres, one fewer moving part is a feature, and this stack comfortably handles corpora into the millions of chunks before you need anything exotic.

01Chunking is a structure problem, not a size problem

The default advice — "split every 500 tokens with overlap" — treats documents as undifferentiated text streams. Real internal documents have structure: headings, sections, tables, lists. A chunk that respects structure carries its own context; a chunk that slices mid-thought forces the retriever to find three fragments instead of one answer.

The rules that matter in practice: never split inside a heading's first paragraph, prepend the heading path to every chunk (a chunk that begins "Security Policy > Data Retention: backups are kept…" is self-describing in a way "backups are kept…" never is), keep tables atomic, and size chunks by structure first with a token ceiling as the backstop.

import re
from dataclasses import dataclass

MAX_TOKENS = 512          # backstop, not target
OVERLAP_SENTENCES = 2     # only used when forced to split a long section

@dataclass
class Chunk:
    text: str            # heading breadcrumb + body
    heading_path: str    # "Security Policy > Data Retention"
    source_id: str       # document id in your store
    position: int        # order within document, for citation display

def split_markdown(doc_text: str, source_id: str) -> list[Chunk]:
    chunks: list[Chunk] = []
    heading_stack: list[str] = []
    body: list[str] = []
    pos = 0

    def flush():
        nonlocal pos
        text = "\n".join(body).strip()
        if not text:
            return
        crumb = " > ".join(heading_stack) or "Document"
        for piece in split_by_tokens(text, MAX_TOKENS, OVERLAP_SENTENCES):
            chunks.append(Chunk(
                text=crumb + ": " + piece,
                heading_path=crumb,
                source_id=source_id,
                position=pos,
            ))
            pos += 1
        body.clear()

    for line in doc_text.splitlines():
        m = re.match(r"^(#{1,4})\s+(.*)", line)
        if m:
            flush()
            level = len(m.group(1))
            del heading_stack[level - 1:]
            heading_stack.append(m.group(2).strip())
        else:
            body.append(line)
    flush()
    return chunks

02Hybrid search: vectors miss what keywords catch

Embedding search excels at paraphrase — "how long do we keep customer data" finds the retention policy even when no words overlap. It is famously bad at exact tokens: error codes, SKU numbers, people's names, internal project codenames. Full-text search has exactly the opposite profile. Production retrieval runs both and fuses the results.

The fusion method that works without tuning is reciprocal rank fusion (RRF): each result contributes 1/(k + rank) from each list it appears in, with k≈60. RRF needs no score normalization — it only uses ranks — which is exactly why it's robust across two systems whose raw scores aren't comparable.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id           bigserial PRIMARY KEY,
  source_id    text NOT NULL,          -- document identifier
  heading_path text NOT NULL,
  position     int  NOT NULL,
  body         text NOT NULL,
  embedding    vector(1024) NOT NULL,  -- match your embedding model's dim
  tsv          tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED,
  updated_at   timestamptz NOT NULL DEFAULT now()
);

-- HNSW for vector search: good recall, no training step, fine for most corpora
CREATE INDEX chunks_embedding_idx ON chunks
  USING hnsw (embedding vector_cosine_ops);

CREATE INDEX chunks_tsv_idx ON chunks USING gin (tsv);

WITH vec AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS rank
  FROM chunks
  ORDER BY embedding <=> $1        -- <=> is cosine distance in pgvector
  LIMIT 40
),
kw AS (
  SELECT id, row_number() OVER (
           ORDER BY ts_rank_cd(tsv, websearch_to_tsquery('english', $2)) DESC
         ) AS rank
  FROM chunks
  WHERE tsv @@ websearch_to_tsquery('english', $2)
  LIMIT 40
)
SELECT c.id, c.source_id, c.heading_path, c.body,
       COALESCE(1.0 / (60 + vec.rank), 0) +
       COALESCE(1.0 / (60 + kw.rank),  0) AS rrf_score
FROM chunks c
LEFT JOIN vec ON vec.id = c.id
LEFT JOIN kw  ON kw.id  = c.id
WHERE vec.id IS NOT NULL OR kw.id IS NOT NULL
ORDER BY rrf_score DESC
LIMIT 12;

03Reranking: the cheap stage that buys the most quality

Hybrid search optimizes recall — getting the right chunk somewhere into the top 40. The generator needs precision — the right chunks in the top handful, because context windows are finite and models over-attend to whatever you include. A reranker bridges the gap: it scores each candidate against the query with a cross-encoder (or an LLM scoring pass) and reorders.

The operational win: retrieval fetches 30–40 candidates cheaply, the reranker reads all of them carefully, and only the top 5–8 reach the prompt. In evals, adding a reranking stage routinely improves answer accuracy more than upgrading the generation model — at a fraction of the cost delta.

04Citations that survive an audit

An internal assistant without citations is a rumor generator. The mechanism: every chunk carries its source_id and heading_path through the whole pipeline, the prompt instructs the model to reference sources by index, and the UI renders those indices as links to the source document — opened to the right section.

Two production rules. First, citations are metadata, not model output: render links from your retrieval metadata keyed by the model's cited index — never let the model compose URLs, which it will happily hallucinate. Second, log the full retrieval set (not just what the model cited) with the answer; when someone challenges an answer next quarter, you can reconstruct exactly what the system knew.

05Evaluating retrieval before users do

Retrieval quality is measurable without any model in the loop. Build a golden set — 30 to 50 real questions paired with the chunk(s) that answer them — and track recall@k: how often the right chunk appears in the top k results. Run it on every change to chunking, embeddings, or fusion weights.

import json

def recall_at_k(golden_path: str, search_fn, k: int = 8) -> dict:
    """golden file: [{\"query\": str, \"relevant_chunk_ids\": [int]}]"""
    cases = json.load(open(golden_path))
    hits, results = 0, []
    for case in cases:
        got = [row.id for row in search_fn(case["query"], limit=k)]
        hit = any(cid in got for cid in case["relevant_chunk_ids"])
        hits += hit
        results.append({"query": case["query"], "hit": hit, "top_ids": got})
    score = hits / len(cases)
    misses = [r["query"] for r in results if not r["hit"]]
    return {"recall_at_k": round(score, 3), "n": len(cases), "misses": misses}

# Wire search_fn to your hybrid query. Run in CI on every retrieval change:
# a chunking \"improvement\" that drops recall from 0.92 to 0.71 should fail
# the build, not a customer.