Accuracy · Benchmarks
PlantPulse AI re-measures accuracy with the same evaluation set at every release. All figures below are based on our in-house evaluation set and heavy-industry demo data, not figures measured on customer field data. For each item, we note what was measured. Items that vary from run to run are given as a median and a range.
Even for the same question, results vary with how well-organized the data is. During onboarding validation, we re-run the same kind of evaluation using your own equipment names, tags, and documents to confirm.
Chat AI
An automated gate measures this at every release. If it falls short of the bar, the release does not ship. The figures below are based on the public pipeline for Re-release 6 (2026-10-08) of the 2026.10 cycle.
| Item | Result | What was measured |
|---|---|---|
| Tool selection | 22 / 25 (88%) — bar 18 | Machine-scored whether the AI called the expected system lookup tool on 25 field questions |
| Grounding fidelity | 6 / 7 | Whether the numbers returned by the tool were carried into the answer unchanged — the correct answer is produced by calling the same tool directly |
| Ontology disambiguation | 16-question set, median 14 / 16 over 5 runs (13 ~ 15) | Whether answers correctly keep distinct axes apart, such as equipment, process, and order |
| Daily briefing | 8 / 8 | Whether the morning briefing avoids inventing content not present in the data |
| What-if calculation citation | "If a part is N days late" 6 / 6 — 0 fabricated numbers, "situations where it should not be called" 2 / 2 | Whether only numbers produced by the calculation engine were carried over, and whether the calculation was correctly withheld for questions that should not trigger it |
System quality
Based on the Re-release 6 public pipeline.
| Item | Result | What was measured |
|---|---|---|
| Service responsiveness | All 6 MCP servers and 19 components responded | Health checks on each server and component at the release gate |
| REST lookups | 174 GET calls — 0 server errors (5xx) | Result of calling every public lookup API once each |
| Platform tool contracts | 63 types — 0 regressions | Whether the input/output contracts of platform tools changed from the previous version |
Video · audio search
Evaluation set v2 — 83 questions run 3 times, querying the search engine directly (including long training videos, distractor videos, and questions not covered by the material).
| Item | Result | What was measured |
|---|---|---|
| Correct video ranked #1 | 90% | Rate at which the correct video appears first in search results |
| Correct video in top 5 | 97% | Rate at which the correct video appears in the top 5 results |
| Scene timing error | median 7.6 seconds (within 15 seconds 68% of the time) | Difference between the playback start time and the correct scene's time |
| "I don't know" | 95 ~ 100% — 0 video chips shown in that case | Whether, when asked about content not in the material, the system answers "I don't know" instead of fabricating an answer |
The path through the chat window is still improving on the same metrics (26 questions, correct video in top 5 74 ~ 84%).
Transcription
| Item | Result | What was measured |
|---|---|---|
| Long training video | 12 of 12 utterances transcribed from a 22-minute video | Whether every spoken segment is transcribed without omission |
| Short training video | 2-minute video, 5 segments — text and timing 5 / 5 matched | Whether the transcribed text and timing match the reference captions |
Drawings — fit formulas
| Item | Result | What was measured |
|---|---|---|
| Tolerance · deviation calculation | 0 mismatches against published reference values across IT5 ~ IT8 tolerances and g · f deviations, over the full 3 ~ 500 mm range | Checked the upper/lower limits produced by ISO formulas against published reference values |
| Digits not calculated | 6 digits differing from the standard by 1 µm | For digits where the formula would not match the standard value, no number is produced — it is left as "not calculated" |
Upper/lower limits produced by formula carry the disclaimer "formula estimate — verify against your own standards documents" at the end of the answer → Asking via drawings
Drawing reading · revision comparison
Measured against in-house complex-part test drawings (hydraulic manifold HMB-2040, gearbox housing GBH-7300) and reference answer sheets.
| Item | Result | What was measured |
|---|---|---|
| Reading | All reference-sheet items matched — title block, dimensions, geometric tolerances, surface roughness, parts list, notes, revision history (for both PDF and DXF) | Compared values read from the drawing against the reference sheet, item by item |
| Revision comparison | HMB-2040 C → D, GBH-7300 B → C — all changed dimensions, parts list entries, notes, and geometric tolerances detected, 0 false positives | Whether actual changes between two revisions are caught in full, with no changes invented |
Related documents
- AI findings and accuracy — how field-recorded dismissal reasons, accept/reject decisions, and outcomes feed back into accuracy
- Asking via video · audio
- Asking via drawings