The AI Vendor Diligence Manual

AI procurement has a specific hazard profile right now: the gap between demo and production has never been wider, the category is crowded with thin wrappers, and the pressure to "pick something" is coming from the board. Standard software diligence — feature matrices, pricing comparisons — was built for a world where the product either had the feature or didn't. AI tools all have the feature. The question is whether it survives your data, your volume, and your compliance reality.

This manual is the diligence protocol for that question. It pairs with DSP-10 (the twelve questions); this is the process wrapped around them.

01Triage before diligence

Full diligence is expensive — reserve it. Before investing anyone's week, run the three-gate triage:

• ▸Gate one: the layer test. Which layer of the stack is this (per DSP-05)? If it's workflow logic — the layer that encodes your judgment — buying is usually the wrong altitude and no vendor will fix that. Diligence only what belongs at a buyable layer.
• ▸Gate two: the data-practices one-pager. Before any demo: is customer data used for training, where does it live, can it stay in-tenant, what's the retention on churn? Vendors who can't answer in writing in a week aren't ready for regulated or serious buyers.
• ▸Gate three: production references exist. Not pilots. Named customers, in production, a year or more. No references, no diligence — you're not evaluating a product, you're being recruited as one.

02The proof-of-work test

The centerpiece of AI diligence is a structured trial on your data, designed by you — not the vendor's guided pilot. The design: pick 30–50 real cases from the actual workflow, including the ugly ones (the scanned PDFs, the run-on email threads, the edge-case vendors). Define the scoring rule before the trial starts — field accuracy, draft usability, whatever the workflow actually needs. Run the cases through the tool, score blind if you can, and compare against your manual baseline.

Two rules keep it honest. First, you drive: the vendor can advise on configuration, but your team operates the tool during the trial — operability is part of what you're buying. Second, no cherry-picking in either direction: the case set is fixed before anyone sees results, and the ugly cases stay in.

03Reading vendor behavior as signal

• ▸Watch the failure conversation. Ask what the tool is bad at. A real team answers instantly with specifics and mitigations — they've been living with the failure modes. Evasion here predicts evasion during incidents.
• ▸Watch the trial resistance. Vendors who resist a fixed-case-set trial are telling you what it would show. The strong ones ask to add cases.
• ▸Watch the pricing math at 10x. Run your realistic production volume through their pricing model in front of them. Watch whether the number surprises them.
• ▸Watch the roadmap ratio. Count how many of your requirements are answered with shipped features versus roadmap slides. Above one-third roadmap, you're funding their build with your risk.

04The reference call script

References are pre-selected to be happy, so the yield is in the specifics. Fifteen minutes, four questions: What broke in the first ninety days, and how did the vendor respond? What does your team still do manually around the tool? What did the invoice do between year one and year two? If you were buying again today, what would you negotiate differently?

The first question is the whole call. Every production deployment has a first-ninety-days story; a reference who claims otherwise hasn't deployed it, and a vendor whose references all say "it just worked" has coached them into uselessness.

05Contract terms that matter more than price