The Data Foundations Manual

There are two failure modes in data readiness. The famous one is chaos: the workflow's inputs live in a veteran's inbox and a spreadsheet named FINAL_v7_ACTUAL. The quieter one is over-preparation: eighteen months of platform work before anything intelligent ships, cleaning fields nothing consumes, guessing at requirements no workflow has stated.

This manual charts the path between them: per-workflow data foundations, built to the standard of "sufficient for the build in front of you," in weeks. It pairs with DSP-11 (the argument); this is the protocol.

01What "reachable" actually means

The readiness bar for one workflow is concrete. Its inputs are reachable if a credentialed system can get them without a human ferrying files:

• ▸API or export. The system of record offers an API, a scheduled export, or at minimum a report someone can automate pulling. "IT says it's possible" is not reachable; a working credential is.
• ▸Digital at the source. Inputs arrive as email, PDF, form, or record — not paper, not phone calls transcribed from memory. If they're physical, digitization is honestly step one of the build.
• ▸Identifiable. Records carry stable identifiers so the workflow can connect an input to an account, a ticket, a person. Fuzzy matching is possible but it's a build item, not an assumption.
• ▸Permitted. Someone with authority has said yes to this use of this data. Access debates masquerade as data problems constantly — flush them out early, they have different owners and different timelines.

02The read-only mirror pattern

The highest-leverage data move for most mid-market builds is also the least invasive: a read-only mirror. Sync the relevant slice of the system of record — nightly or hourly is almost always enough — into a Postgres schema the build owns. The legacy system stays untouched and authoritative; the automation gets a queryable, indexable, joinable copy it can't corrupt.

The mirror decouples everything: the build iterates without touching production credentials on every request, retrieval and reporting get real SQL instead of rate-limited API calls, and when the legacy system finally gets replaced, the automation barely notices — it reads the same schema fed by a different sync. Write-backs, where needed, go through the vendor API deliberately, with the approval gates of SCH-02 in front of anything irreversible.

03Minimum viable structure, by workflow type

• ▸Drafting workflows (proposals, replies, reports): need examples more than schemas — a few dozen past outputs, reachable, plus the source inputs they were built from. The structure investment is a folder convention and an export, not a warehouse.
• ▸Extraction workflows (documents → records): need the target schema designed (FM-08's contract thinking, SCH-03's honesty channels) and a destination table. The inputs can stay messy — that's the point of extraction.
• ▸Retrieval workflows (knowledge assistants): need the source documents enumerated, an owner per source, and a staleness policy — which docs are authoritative, which are archive. Chunking and indexing are build items (SCH-01); knowing what's true is yours.
• ▸Reporting workflows (recurring summaries): need the metrics defined with owners who agree on the definitions. The hard part of automated reporting is never the automation — it's two systems disagreeing about what "active customer" means, and that fight is organizational.

04Foundations that compound

Each workflow build leaves data assets behind — if you let it. The mirror schema grows a table at a time, every one load-bearing. The extraction pipelines mint clean records that the next workflow reads. The golden sets and miss logs become eval infrastructure. Two or three builds in, you notice the thing the data-lake pitch promised: a genuinely useful internal data layer — except every piece of yours exists because a shipping workflow demanded it, and none of it was built on speculation.

That's the honest version of a data strategy for the mid-market: bottom-up, workflow-forced, paid for as it goes.