How accurate does the system have to be

I ask early on what level of accuracy would make a system worth having. The usual answer is one hundred per cent. Occasionally somebody says ninety-nine and looks like they are being generous.

It is a reasonable instinct and it is the wrong target, because it is not a standard anything in your business currently meets.

Start by measuring the people

Ask what the error rate on this process is today. Almost nobody knows. It has been running for years, everyone agrees it works, and there has never been a reason to count.

So count. Pull a few hundred cases from last quarter, have somebody senior check them properly, and write down how many were wrong and how badly wrong. Two days of work. The number is usually higher than the room expects, and once it exists the conversation changes, because you now have something to be better than instead of an ideal to fall short of.

You also learn what sort of mistakes your people make, which is rarely the same sort a machine makes. That is more useful than the total. Your staff miss things when they are tired at the end of the month. Software is wrong in the same way every time, which is easier to find and easier to fix.

Not all mistakes cost the same

There are two ways to be wrong and they are almost never equally expensive. The system approves something it should have refused. The system refuses something it should have approved.

Put a rough figure on each. If a wrongly approved invoice costs you the value of the invoice, and a wrongly refused one costs you a phone call from an irritated supplier, those are different numbers and the system should be built to lean one way. If a missed fault costs you a day of lost production and a false alarm costs an engineer an hour, it should lean the other way.

This is the most useful hour anybody spends before a build, and it is nearly always skipped in favour of arguing about a single overall percentage that hides both.

Handling less is often worth more than being right more

The question is not only how accurate. It is also how much the system attempts. One that handles six cases in ten to a high standard and hands the rest to a person, having flagged which ones it was unsure about, is normally worth more than one that handles all ten at eighty-five per cent.

The first you can trust and build a process around. With the second, everything has to be checked anyway, so you have added a step rather than removed one. Ask any supplier what their number becomes if the system is allowed to decline the cases it is unsure of. If they have never thought about that, it tells you something about how much production use they have seen.

Agree the number before, with the person who carries the risk

Write down what score would make this worth using, and have the person who answers for the process sign it. Not the project sponsor. The one who takes the complaint.

Left until afterwards, the threshold drifts to fit whatever the result turned out to be. I have watched a target quietly become an aspiration in the same meeting where the score was presented, and it is very hard to argue against once it starts.

Some processes should be left alone

Where a mistake cannot be undone, where it touches somebody's safety or their health, or where it is a filing that carries a legal consequence, the answer is either do not automate it or keep a person in the decision with the time and the information to disagree with the machine.

Before software I was an air safety officer at India's aviation regulator, investigating accidents. Nobody there asked whether a system was usually right. The questions were what happens the one time it is not, whether anybody would find out, and how long that would take. Those questions apply perfectly well to an invoice run.

So a short test. If it is wrong on a Tuesday, can you undo it, and would you know? If either answer is no, keep a person in the loop and design the system to make that person faster rather than to replace them.

None of this argues for lower standards. It argues for a number that means something, arrived at by measuring what you achieve now and pricing what each kind of mistake costs you. If you want help putting that number together for one specific process, it is a couple of days of work, and it will also tell you whether to go ahead at all.