You just watched a demo. The interface was clean, the AI answered every question correctly, the presenter walked through your exact use case with what looked like effortless precision. You came in skeptical and left almost convinced.
That feeling — that slight vertigo between "this looks real" and "I have no idea if it actually works" — is not a failure of judgment. It's a completely rational response to a controlled environment. Demos are designed to succeed. The data is prepped. The edge cases are avoided. The presenter has given this pitch forty times and knows exactly where to pause for effect.
The problem isn't that vendors run good demos. The problem is that most of us have no framework for what to ask next.
This piece gives you one. Seven questions you can ask any AI vendor or consultant — whether they're selling you software or their own services — that separate a durable solution from a polished slide. None of them require a technical background. All of them require a vendor with something real to offer.
Why Standard Due Diligence Doesn't Quite Work Here
When you evaluate a new piece of accounting software or a staffing agency, you have intuitions built from experience. You know what questions to ask, what red flags look like, what a reasonable pricing structure is.
AI vendors have figured out that most buyers don't have those intuitions yet, and some of them — not all, but some — are taking advantage of that gap. The language is just unfamiliar enough to make you hesitant to push back. Terms like "machine learning pipeline" and "LLM-powered" and "intelligent automation" carry an authority that's hard to interrogate if you're not sure what they mean.
The good news: the questions that reveal whether something real is underneath the language are mostly ordinary business questions. You just have to know which ones to ask, and what a telling answer looks like.
Seven Questions That Separate Real from Polished
1. "Show me it failing."
This is the most important question on the list, and the one vendors least expect.
Every AI system has failure modes. It will misread ambiguous inputs, generate plausible-sounding answers that are factually wrong, or handle edge cases poorly. A vendor who claims their product doesn't have these problems either hasn't tested it honestly or is lying to you.
What you're looking for isn't a flawless answer — it's the character of the failure. Ask them to demonstrate a case where the system gets it wrong. Watch what happens: does the AI produce a confident, polished wrong answer, or does it flag uncertainty and tell the user to verify? The former is a genuine risk to your business. The latter is a sign the system is designed to be used responsibly.
This question also tells you a lot about the vendor. A good one will answer without hesitation, probably with a prepared example, and will explain what guardrails exist. A bad one will pivot to another feature or tell you failures are rare.
AI systems that produce confident wrong answers without flagging uncertainty are a documented, pervasive problem — if you want to understand the mechanics of why, the piece on how AI confidently gets things wrong is worth your time before you go into any vendor meeting.
2. "What does this look like on my data?"
Demo data is clean. Your data isn't.
Most small and midsize businesses are running on a mix of spreadsheets that evolved over years, records exported from three different systems that don't agree with each other, customer names entered four different ways, and fields that someone decided to repurpose halfway through 2019. That's not a knock on you — it's just what real business data looks like.
Ask the vendor to connect to a sample of your actual data during the evaluation. Not a tidy export you prepared — a messy one. If they need time to prep it first, ask how long that prep typically takes and who does it. That preparation is a one-time cost you'll pay at implementation, and often a recurring cost every time your data structure changes.
The answer to "who cleans the data" matters a lot. If the answer is "your team," that's a real ongoing labor cost. If the answer is "we handle it," ask how and what it costs. If the answer is evasive, that's a tell.
3. "Who else my size can I call?"
Not "can you send references" — that gives them time to curate a list of their happiest customers. Ask who else at your approximate size and in your approximate industry is using this, and whether you can speak with them directly.
When you get a name, call them. And don't just ask whether they're happy with the product. Ask:
- What took longer than you expected?
- What do you still do by hand that you thought the tool would handle?
- If you were starting over, what would you do differently?
Happy customers will still answer these questions honestly if you ask them directly. Unhappy customers are telling you something more valuable. The vendor who encourages you to call references and hears you asking these specific questions is the vendor who isn't afraid of the answers.
4. "What happens to my data if I leave?"
This question gets less attention than it deserves, and it's especially important with AI tools that learn from or store your business data over time.
You want to know: Can you export everything? In what format? Does the vendor retain your data after the contract ends, and for how long? Do they have the right to use your data to train their models?
Some of these are pure privacy questions. Some are practical business continuity questions — if you switch vendors in two years, do you have to start from scratch? And some are legal questions worth having your lawyer look at before you sign. The piece on what happens to the data you put into AI tools covers the underlying risks here in more detail.
A vendor with nothing to hide on this will have a clear, written answer. A vendor who is vague, who points you to a forty-page terms-of-service document, or who tells you "that's not something most of our customers ask about" — that's a tell.
5. "What's the total cost in year two?"
AI tool pricing is genuinely hard to evaluate from a sales deck, because it's often structured to look small at the point of sale and grow significantly afterward.
The questions to ask:
- Are there per-seat costs that scale with headcount?
- Is usage metered — and what does "usage" mean in this context?
- What does integration with our current systems cost, and who does it?
- What has the renewal increase been for existing customers?
- Are there tiers above our current plan that we'd likely hit?
Most vendors will give you honest answers if you ask directly. What you're guarding against is signing a contract based on the entry-level number and discovering a year later that you're paying three times that.
This is one of the most common ways AI tool costs catch businesses off guard — and it's worth going in with eyes open. The full landscape of hidden costs is something we've covered in depth in the AI costs nobody warns you about, and it's worth reading before you sign anything.
6. "What does your product not do?"
This is the character test.
Every product has limits. Honest vendors know them and will name them, because they'd rather scope you correctly than lose you to churn in six months. If a vendor can't give you a clear, specific answer to this question — or if every limitation they mention comes with a "but we're working on that" — that's a meaningful signal.
The best answer sounds something like: "We're strong at X, we're not the right tool for Y, and if you need Z you'll want to pair us with another system." That specificity is a sign they know their product well and respect your time enough to say so.
An answer that sounds like "we do everything you need" should make you significantly more skeptical, not less.
7. "How will we know if this is working?"
Obvious, and almost never asked. Before anyone builds or installs anything, agree on the one number that defines success — not "improved efficiency" but "average handle time drops from eight minutes to five," or "first-draft writing time falls by 60%." A vendor who helps you name that metric up front, and ties it to a business outcome you actually care about, is signing up to be judged on results. One who offers only activity metrics — prompts processed, workflows automated — is steering you toward numbers that look impressive without proving anything changed.
The Tells Worth Knowing Without Asking
Sometimes you don't need to ask any questions. The pitch tells you.
Guaranteed outcomes. No reputable AI vendor guarantees specific performance results, because outcomes depend heavily on your data, your processes, and how you implement the tool. "Guaranteed to reduce your support tickets by 40%" is either a lie or a contractual commitment with asterisks that will become your problem.
Percentages with no baseline. "95% accuracy" sounds impressive until you ask: accuracy at what, measured how, on whose data? A percentage without a denominator is a number chosen for marketing, not measurement.
"AI-powered" as the feature, not the outcome. If the main selling point is that the product uses AI, rather than what it does for your business as a result, ask yourself what you're actually buying. The technology is the mechanism, not the benefit. A vendor who can't clearly articulate the business outcome is selling the buzzword.
Manufactured urgency. "This pricing expires Friday" or "we only have two implementation slots left this quarter" — pressure tactics are pressure tactics regardless of what's being sold. Real tools don't evaporate.
A demo you're never allowed to drive. If you can't touch the product during evaluation — if every interaction goes through the vendor's hands — ask why. There's usually a reason, and it's usually not flattering to the product.
What Good Actually Looks Like
A vendor worth working with will do a few things without being asked.
They'll name the limits of their product before you find them yourself. They'll scope a small pilot — a defined test on a real part of your business before a full commitment — rather than pushing you straight to an annual contract. They'll be specific about what "implementation" involves and who does the work. They'll give you real customer references and encourage you to ask the hard questions. And for anything bigger than a quick tool purchase, a credible firm starts with a diagnostic phase — a paid scoping exercise that tells you what's worth doing before you commit to building it; what that kind of audit should actually involve, and cost, is worth knowing before you sign.
And if you ask any of the seven questions above, they'll answer directly. Not defensively. Not with a pivot. Just directly.
A Quick Checklist Before You Sign
If you can check every box, you've probably found someone worth working with:
- They asked more about your business than they talked about their technology.
- They named the limits of their product or approach without being prompted.
- They have a clear plan for how this connects to your existing data and systems.
- Success is defined in numbers you can measure before the work starts.
- You understand what year-two costs and ongoing maintenance look like.
- There's an exit plan that doesn't require them to be involved forever.
- Nothing was guaranteed that couldn't reasonably be guaranteed.
A Note on Consulting Firms (Including Ours)
Everything above applies equally to consultants who help you select and implement AI tools, not just to software vendors. You should ask a consulting firm the same questions: show me a project that didn't go as planned, tell me what you don't do well, give me a reference I can call who'll be honest with me.
We hold ourselves to the same standard. If you want to see whether a strategy engagement with us would be worth your time, the right move is a conversation where you get to ask us the hard questions — not a sales deck. The thinking we do on your behalf should be demonstrable before you pay for it.
If this is the kind of due diligence you want someone in your corner for, let's talk. No commitment, no pressure — just a straight conversation about whether we're the right fit.