Vendor demonstrations are built to impress. They run on the vendor's data, on cases the vendor chose, with the failures cropped out. Nobody is lying. The demonstration just answers a different question from the one you have.
Your question is how this behaves on your records on a bad day, and you can answer it in about three weeks.
Build the test before you talk to anyone
Take a few hundred real cases out of your own records and agree the correct answer for each one. Keep that set aside and put every vendor through exactly the same cases. Somebody in your business who knows the work can produce it in roughly two days, which is far less than most people assume when they object to the idea.
Build it before the vendor conversations start. Build it afterwards and you will unconsciously shape it around what you have already been shown.
Include the cases you dread
Ask that person which records they hate receiving. The document scanned at an angle. The category that appears twice a year. The record where two fields contradict each other and somebody has to decide which one is lying.
Three thousand easy cases tell you less than two hundred difficult ones. Most of these test sets are too easy, everybody scores well, and the exercise proves nothing while feeling rigorous.
Agree the pass mark in writing
Before anybody runs anything, decide what score would make this worth buying. Write it down and circulate it.
Leave it until afterwards and the number will drift to fit whichever vendor the loudest person in the room already preferred. I have watched that happen and it is very hard to argue against once it starts.
Score the things that are not accuracy
Every time the system answers, it costs you money, so work that cost out at your volume rather than at the volume in their example. Where the data physically goes. What the provider keeps and for how long. What the price does when your usage doubles. What it costs to leave.
That last one sets your negotiating position at renewal and almost nobody scores it.
Keep the test afterwards
When the comparison is finished you should still own the set of test cases and the way you scored them. Models change every few months and prices move faster. Being able to run the whole comparison again yourself in an afternoon is worth more than the original decision was.
If a supplier is unwilling to be measured this way, you already have your result.
Working through this on a live programme?
A 45-minute call with the engineer who would run the work. We will tell you whether AI is the answer, including when it is not.