How To Evaluate An AI Vendor Without A Technical Team
Six questions that separate a working system from a demonstration, none of which require you to understand the model.
Most businesses buying artificial intelligence software have no one in-house able to evaluate the technology. That is a genuine disadvantage but a smaller one than it appears, because the questions that predict whether a deployment succeeds are commercial rather than technical.
What happens when it is wrong
Every such system is wrong sometimes. The question is what the product does about it: whether errors are surfaced for review, whether a human can intervene before an action is taken, and whether there is an audit trail afterwards.
A vendor who responds to this question with accuracy statistics has not answered it.
Where the data goes
Establish whether your data leaves your environment, whether it is retained, whether it is used to improve the vendor's product for other customers, and whether that can be switched off. Get the answer in the contract rather than from a salesperson.
If you handle regulated data, this determines whether you can use the product at all.
What happens on the demonstration data
Demonstrations run on inputs chosen to work. Ask to run a pilot on a sample of your own real material, including the awkward cases — the badly scanned document, the unusual format, the edge case your staff complain about. Performance on your mess is the only relevant number.
Who is accountable for the output
If the system produces something that causes harm, the contract allocates responsibility. Many vendor agreements disclaim liability for outputs entirely. That may be acceptable; it should be a decision rather than a discovery.
What it costs at scale
Usage-based pricing that is trivial in a pilot can be significant at full volume. Model the cost at realistic production volumes before committing, and ask what happens if usage doubles.
What replacing it looks like
Ask how you would export your data and move to another supplier. A vendor who cannot answer clearly is describing a system you will find expensive to leave, and that fact should be priced into the decision now.
