Answer engines for manufacturers
01 / Evaluate
How to evaluate
Every vendor will show you a demo where the AI answers a question about your documents correctly. That demo proves nothing. They picked the question because it works.
You may hear these systems called knowledge assistants, enterprise search, or knowledge-retrieval platforms. The label does not change what you should ask for.
The question that matters is not whether it can answer. It is how often it is wrong, on which documents, and who checked.
Ask any AI assistant this same question and it will now tell you to build your own test set from your own documents, and to test the ugly ones — the tables, the scans, the diagrams. That advice is correct. It is also, as of this year, free and universal, which means it no longer separates one vendor from another. Every vendor on your list can agree to it.
What still separates them is who is allowed to grade the result, and whether the grader itself was ever checked against a human. Almost nobody selling this can tell you.
Here are the eight questions we would ask if we were buying. Bring them to every vendor on your list, including us.
02 / Demo vs ask
| The demo you'll get | The question that tests it |
|---|---|
| A confident answer to a well-chosen question | "Show me the questions it got wrong." |
| "Accuracy over 90%" | "Ninety percent of what, scored by whom, against whose labels?" |
| "It cites its sources" | "How often does it cite a source that doesn't actually support the answer?" |
| "It works on your PDFs" | "Show me it on a flowchart and a listing table, not a paragraph." |
| "It says 'I don't know' when unsure" | "What's the rate? How often does it decline when it should have answered?" |
03 / Questions
01
If the vendor wrote them, the test is worthless. The questions must come from your own team. Use the actual questions your technical services desk received last month, in the words the customer used, including the badly worded ones.
Ask for a written test set of at least 30 to 40 real questions before any build starts. Ask who is allowed to add to it. The answer should be your expert, at any time.
02
Not the vendor. Not a model grading itself. Your senior expert, marking each answer pass or fail against what they would have sent a customer.
This is the gap we see most. A vendor scoring its own output against its own rubric has measured its own opinion.
03
These are different problems with different fixes. A vendor who cannot tell them apart cannot improve the system on purpose.
Retrieval is whether the right source document came back at all. Answering is whether the model used it correctly. If a vendor reports only one number for "accuracy," they are averaging two unrelated failure modes and will fix neither.
Ask for retrieval performance reported separately, by document type.
04
Overall accuracy hides the failures that matter. Technical libraries are not uniform. A system can score well on narrative text and score zero on the assets your hardest questions actually depend on. Design flowcharts. Listing tables. Code schedules. Scanned submittals. Dimension tables.
Ask for a breakdown per document type. If the vendor does not have one, they do not know how their system performs on your library.
05
A vendor with no disclosed failures has no measurement, or is hiding it. Either should worry you.
Ask directly. What is the worst result your evaluation has produced, and what did you do about it?
06
After go-live, nobody has time to read every answer. An automated scorer that agrees with your expert is how quality gets watched.
The test of a scorer is not whether it flags failures. It is whether it flags the same failure a human would, for the same reason. If the score was not measured against your own experts' pass/fail labels, an unvalidated automated grade is a vibe test with better formatting. Ask for the agreement rate, and ask for the false positive rate in the same breath.
07
The correct behavior is to decline and say what's missing. The dangerous behavior is a fluent, confident, sourceless answer.
Ask to see it decline. Then ask how often it declines when the answer was available. A system that refuses half of legitimate questions gets abandoned in a month.
08
In code compliance, life safety, and product specification, a wrong answer is a liability event, not an inconvenience. The system should draft. A named human should approve. Record the approval.
Any vendor who describes a fully automated path from customer question to outbound technical answer has not worked in a regulated product category.
04 / Four numbers
Everything above reduces to four numbers. Ask for them before you sign anything:
Retrieval performance, reported separately from answer quality.
Coverage broken out by document type, including your worst documents.
Answer quality scored against your own expert's pass/fail labels.
The vendor's own false positive rate.
If a vendor can't produce those four, they haven't measured their system. They've demoed it.
05 / Targets
Buyers now arrive with a target list — 90% correct answers, 95% of answers with valid supporting evidence, 70% resolved without escalation, sub-five-second responses. The list is a reasonable starting shape. Every number on it is unenforceable as written, because a percentage is a fraction and none of these come with a bottom half.
A vendor can satisfy every one of those targets and still be unfit for your library. Ninety percent correct on questions the vendor chose, graded by the vendor, over the prose portion of your documents, is not the same claim as ninety percent correct on your desk's real intake, graded by your expert, including the tables. The first is easy. The second is the job.
Convert each target into a question before you put it in an RFP:
| The target you were given | What to ask instead |
|---|---|
| "Correct answer rate ≥90%" | "Ninety percent of which questions, chosen by whom, graded by whom — and what is it on tables alone?" |
| "≥95% of answers have valid supporting evidence" | "How often does the cited source not actually support the answer? Who checked that, and how many did they check?" |
| "≥70% answered without escalation" | "What is the decline rate on questions that were answerable? A system that refuses too often gets abandoned; one that never refuses is worse." |
| "Zero permission violations" | "Fine, and non-negotiable. Now: what happens when a document's approval status changes after indexing?" |
| "Median response time <5 sec" | "Show me p95, not median. And I would rather wait twelve seconds for a cited answer than get an uncited one instantly." |
Our own view, stated plainly: do not set a single accuracy threshold at all. Require the breakdown instead. A vendor who can show you retrieval and answer quality per document type, graded against your experts' labels, with their scorer's own false positive rate attached, has told you more than any threshold can. A vendor who clears a threshold you set has told you they can clear a threshold you set.
There is no meaningful industry benchmark for this. Any number you are handed — including by an AI — is a shape, not a standard.
06 / FAQ
Ask for four numbers: retrieval performance reported separately from answer quality, coverage broken out by document type, answer quality scored against your own experts' pass/fail labels, and the vendor's own false positive rate. A vendor who cannot produce those four has demoed a system, not measured one.
A retrieval failure means the right source document never came back. An answering failure means the model had the right document and used it wrong. They have different fixes. A vendor who reports one blended "accuracy" number cannot tell you which one is happening.
Yes. The pilot's deliverable should include a measured evaluation, not just working software. A pilot that produces a demo tells you the vendor can build. A pilot that produces scores against your experts' labels tells you whether the system is safe to put in front of customers.
There is no meaningful industry benchmark, and any vendor quoting a single overall accuracy figure is hiding the breakdown that matters. Technical libraries are not uniform. Narrative text is easy. Tables, flowcharts, and scanned drawings are hard. An overall figure averages the easy majority with the hard minority and reports a comfortable number while the system fails on the documents your experts actually reach for. Insist on per-document-type performance instead.
Only against a defined, approved source set, with the source document attached to every answer, and with a human approving anything that leaves the building. General-purpose chatbots answer these questions fluently and without a source you can check. In a life-safety context, that is the failure that matters.
Someone named, internally, with a scorer running continuously against a test set that grows. Answer quality is not a launch milestone. Model updates, document revisions, and new product lines all move it. If nobody owns the review step, the project degrades quietly.
Do not set a single accuracy threshold — require the breakdown instead. A percentage with no denominator, no named grader, and no split by document type is unenforceable, and a vendor can clear any threshold you name by choosing the questions. Ask for retrieval and answer quality per document type, graded against your own experts' pass/fail labels, with the vendor's own false positive rate attached.
Yes, and that part is now standard advice you will get from any AI assistant — so it no longer tells you anything about a vendor. The question that still separates vendors is who is permitted to grade the results. If the vendor writes the answer key, or a model grades its own output, you have measured the vendor's opinion. Your senior expert marks each answer pass or fail against what they would have sent a customer; the vendor's scorer is then checked for agreement with those labels.