Most weeks somebody describes the same system to me. Twenty years of documents on a shared drive, or in a records system, or both plus a few inboxes. They want their people to ask a question in ordinary language and get an answer with the page it came from attached.
We have built several. It is also the request where the gap between the demonstration and the working system is widest, because a demonstration runs on ten clean documents somebody chose in advance.
Can the documents be found at all
Before anything can read your documents, something has to reach them. In practice they are spread across a file server, a document system bought in 2014 and half abandoned, email attachments, and one long-serving person's laptop.
The harder half is duplicates. There are four versions of the policy and one of them is signed. There is a drawing marked issued for construction with three earlier ones beside it. If the system answers from the wrong one it will do it confidently and cite it properly, which is worse than no answer. Deciding which copy is the truth is not an engineering job, and it is usually the longest task in the project.
Can the document actually be read
A document that is text the whole way through is easy. A scan is a picture of a page, and the words have to be recognised before anything can be done with them. That recognition is much better than it was five years ago, and it is still where these projects lose weeks.
The awkward ones are predictable. A page photographed at an angle on somebody's phone. A stamp in the corner of a drawing carrying the revision number. A handwritten note in the margin that changes the clause above it. An Arabic document, or one in both languages. Take your fifty worst examples rather than your best and test on those before anybody signs anything.
We built document search for an engineering consultancy where the drawings and the calculations were the point. Structural beam design went from four hours to about five minutes, and it works because reading the source material properly was treated as the main task.
Does it respect who is allowed to see what
This is the one that gets organisations into real trouble, and the one demonstrations skip. The rule is easy to state. The system must only search what the person asking is already allowed to open.
It goes wrong in a specific way. The system is given one powerful login that can read everything, because that is far easier to build, and permission is checked only at the front door: are you an employee, yes, come in. Then somebody junior asks about salary bands and gets an answer drawn out of a board paper. Nothing was hacked. It was built that way.
Run this before go-live and again every quarter. Take an account with the lowest access you have, ask it ten questions only the executive team should be able to answer, then repeat across departments, because the sensitive material is rarely all at the top. If the answers come back, you have built a leak with a search box on the front.
Even a refusal can leak. If the system replies that it cannot show you the document called Redundancy plan March, it has told you the document exists.
Does it show you where the answer came from
Every answer should arrive with the document, the version and the page, and the reader should be able to click through to the paragraph where it sits. Two reasons, and the second matters more than people expect.
The first is trust. Nobody in a professional job will act on an answer they cannot check, and if they have to go and find the source themselves you have saved them nothing. The second is that showing the source makes a wrong answer survivable. These systems are wrong sometimes. That is tolerable when the reader can see in two seconds that it quoted the superseded revision, and not tolerable when the answer arrives as a confident paragraph with nothing behind it.
If a vendor demonstrates a system that answers without sources, treat that as a different product rather than a missing feature.
How to test it in an afternoon
- Fifty real questions your staff ask, with the right answers known, written by somebody who does the job.
- The worst documents you hold, including scans, drawings, handwriting and Arabic.
- A question whose answer sits in a superseded version, to see whether it picks the current one.
- The same questions again from an account with the lowest access in the building.
That set takes a day or two of somebody's time and tells you more than any demonstration. Keep it, because you will want to run it again when the documents grow or the supplier changes something.
If you are not sure whether your documents are in a state anybody could use, send a description of what you hold and where it lives. That is usually a short call rather than a project.
Working through this on a live programme?
A 45-minute call with the engineer who would run the work. We will tell you whether AI is the answer, including when it is not.