AI vendor evaluation is hard right now for a structural reason: the demo and the product have never been further apart. A demo runs on curated inputs, a warm cache, and a presenter who knows which questions not to ask. Your production reality has none of those. The questions below are designed to surface the gap before the contract does.
01Data questions
- ▢Is our data used to train your models or anyone else's — and can you disable that in writing, in the contract?
- ▢Where does our data physically live, who can access it, and what's the retention schedule after we leave?
- ▢Can the system run inside our tenant or compliance perimeter if we need it to — and what's the real cost difference?
02Reliability questions
- ▢What happens when the model is wrong? Walk me through the failure path a user experiences — not the roadmap slide about it.
- ▢What accuracy do you measure, on what data, and how? "95% accurate" means nothing without the denominator.
- ▢What's your fallback when your upstream model provider has an outage or deprecates a version?
03Integration questions
- ▢Show me the API docs — not the integrations logo wall. Can we read AND write everything the UI can?
- ▢What does exporting all of our data look like on the day we churn? Formats, completeness, cost.
- ▢How do you handle our systems' rate limits and schema changes? (The honest answer includes the word "retry" and a queue.)
04Business questions
- ▢What does pricing look like at 10x our usage? Token-based pricing that's cute at pilot scale can be a hostage situation at production scale.
- ▢Which of your features are actually your product, and which are thin wrappers over a model API we could call ourselves?
- ▢Who are three customers running this in production for over a year — not pilots, production?
05Red flags that end the meeting early
- ▸The training dodge. Any hedging on whether your data trains their models — pending policies, tiered opt-outs buried in enterprise plans, or a link instead of a contract clause. If data governance is improvised, everything downstream is too.
- ▸The accuracy number with no denominator. A precision claim with no dataset, no task definition, and no failure profile is marketing wearing a lab coat. Ask what it was measured on; watch what happens.
- ▸The pilot-only reference list. Every reference is six weeks in, enthusiastic, and pre-revenue on the tool. Production customers exist or they don't — and a vendor who can't produce one is asking you to be it, at full price.
- ▸The roadmap answer, repeatedly. One roadmap answer is honest; five is a product that doesn't exist yet. Count them in the meeting and decide with the count, not the enthusiasm.
- ▸The demo that can't leave the rails. You ask to try your own example live and the room gets creative about why not. Whatever the reason sounds like, the answer was no — and it's the most informative no in the whole evaluation.
06Reading the answers
Good vendors answer these directly, sometimes with an unflattering honesty that should raise your confidence, not lower it. A vendor who says "our accuracy drops on scanned documents older than 2015 and here's our review-queue design for exactly that" understands production. A vendor who answers every question with "the AI handles that" is selling you a demo.
The pattern to fear isn't a missing feature — it's a missing failure story. Any team that has actually run AI in production has scars and mechanisms. If the pitch contains no scars, the scars are scheduled for your rollout.
OPERATOR NOTE — Ask question twelve first. Reference customers in production for a year answer half the other questions for free.
Put this thinking to work.
A 30-minute strategy call with an operator — we'll map your first deployment path, not send a deck.