AI Discoveries and Accuracy — What the Engine Finds and What People Leave Behind as Judgment
The time series engine doesn't wait to be asked — it finds problems on its own and records them as "discoveries." Copilot receives these: urgent ones are alerted immediately, the rest are bundled and shown in the morning, and anything requiring human judgment is raised as a proposal card. The judgment a person leaves on a proposal is used to measure the AI's accuracy, and the AI uses it to improve its own judgment criteria.
엔진이 발견을 적는다 (설비 이상 · 오더 목표 미달 · 데이터 끊김 …)
→ 코파일럿이 1분마다 받아 온다
→ 급함 · 높음은 즉시 알림 / 나머지는 아침 브리핑 카드 / 판단이 필요하면 AI 제안 카드
→ 사람이 제안에서 무시(사유) · 수락/거절 · 효과를 남긴다
→ 그 표시로 정확도를 재고, 설비별 판정 기준을 스스로 조정
No button in this document controls equipment or changes orders or schedules. The only thing the AI changes on its own is the anomaly judgment criteria (per-equipment sensor thresholds).
The «Reports › AI Accuracy» tab, the AI discoveries · equipment to watch · alarm shelf · post-repair confirmation on the monitoring screen (/watch), the AI verdict (correct · false alarm · unsure) and report missed detection on the anomaly diagnosis operations console, and Pending Validation (/review) do not exist in the new screens. The judgment a person leaves is now consolidated into a single Proposal, and the numbers are viewed in the admin console and automated reports — see below.
Where do AI discoveries go
There is no screen that shows only AI discoveries collected separately. Discoveries go to the following places depending on their type and urgency.
| Discovery | Where it goes |
|---|---|
| Equipment anomaly | Proposals › To-Do › Sudden Issues' Anomaly to Review. If severity is high, it's escalated to an incident |
| Urgent or severity High | Immediate alert — in-app notification · push, and if configured, email · Slack. One message per source equipment |
| Everything else | Morning Briefing card — bundled per source equipment every day before dawn (default 05:00) as a single card under Proposals › Plan |
| Order target shortfall · delay recovery plan · failure risk | Separately from the above, an AI Suggestion card under Proposals › Plan → AI Suggestion Card |
- The same discovery is alerted only once at the same level. If severity rises or urgency turns on, it alerts again.
- You can also ask in Ask, e.g. "Summarize the AI discoveries currently open," to get a list — the discovery list tool (
list_findings) is on from the start (2026.10 re-release (2026-10-20)) → Time Series Insight Tools - Shadow verdicts that haven't yet passed validation are not sent to alerts, proposals, or Ask — see "Shadow" below.
Types of discoveries
| Type | What |
|---|---|
| Equipment anomaly | Sensor · OEE · RAM · energy deviates from normal |
| Order target shortfall | An in-progress order is projected to miss its target quantity by the planned end |
| Data loss · Data quality issue | Values aren't coming in, or incoming values look abnormal (a data issue, not an equipment issue) |
| Chain stoppage | An upstream stop caused a downstream stop |
| Failure risk | May fail in the near future |
| Limit reached (projected) · Power peak (projected) | At the current trend, a limit or peak will be reached |
| OEE degradation · Defect rate anomaly signal · Changeover delay | Production metric discoveries |
| Parts delay propagation · Delay recovery plan | Parts delay spreading into the schedule, and its recovery plan |
| AI config change | A record that the AI changed its own criteria |
Equipment anomalies are judged by both the existing anomaly detection and AI discoveries, using the same rules. To avoid alerting the same anomaly twice,
by default (anomaly.source=both) anomaly detection does the alerting, and the equipment anomaly from AI discoveries is used only for comparison.
If an operator switches it to findings, AI discoveries do the alerting, and in that case low/medium equipment anomalies go to the morning briefing instead of an immediate alert.
Equipment to check first today
The engine ranks a Equipment to Watch list per site — scored (0–100) and ordered by open discoveries · health score · 24-hour trend · equipment importance, along with three reasons and the rank change from yesterday. There is no ranking screen; ask "Which equipment should I check first today?" in Ask, and it answers with the same ranking (this tool is on by default).
- Data quality issues such as data loss are not included in the score.
- Equipment importance (critical · important · normal · low) is set in the admin console's Equipment Importance Rating.
When alerts flood in — chattering · flooding
If the same equipment · same discovery type reaches a third immediate alert within 60 minutes, that message is tagged with "[Chattering 3x/60min]" and no further alerts follow (chattering). If a site exceeds 10 immediate alerts in 10 minutes, a single "Alert Flood" message is sent instead of individual alerts, and the rest are bundled into a summary (flooding). Either way, the row and the discovery remain. High/urgent alerts for critical equipment are not buried by chattering or flooding. The numbers are defaults and can be changed in settings.
The old alarm shelf, where you could temporarily suppress alerts with a deadline and reason, and the screen that showed suppressed alerts collectively, do not exist in the new screens. Shelved items already placed come down automatically once the deadline passes.
The chattering/flooding threshold numbers are alert.chatter.count · alert.chatter.window.min · alert.flood.threshold · alert.flood.window.min in System Settings.
Judgment left by people
Judgment is left in the proposal detail view → Proposals
| Judgment | Where | Values | What it's used for |
|---|---|---|---|
| Dismiss reason | Dismiss on an anomaly to review | During setup/changeover · equipment stopped/idle · under maintenance · sensor issue · other | Marks that anomaly as a false alarm. Anomalies bundled in the same card are marked together. Material for per-equipment precision and threshold tuning |
| Accept · Reject | AI Suggestion Judgment on the AI suggestion card | Accept / Reject (already done · not needed · suggestion was wrong · other) | AI suggestion's score. "Already done" triggers automatic post-repair confirmation |
| Effect | On an accepted suggestion | Effective · Not effective · Still unknown (+ note) | Used by the AI to choose its next suggestions |
| Diagnosis verdict · actual cause | Record result on an incident | Diagnosis right · diagnosis wrong / mechanical · electrical · process/operation · other (all optional) | Admin Console Answer Quality › Diagnosis Accuracy |
| Action plan rejection reason | Reject on an incident awaiting approval | Diagnosis is wrong · actions do not fit · already done · cannot do it now (people, schedule) · not an anomaly (false alarm) · other | «Diagnosis is wrong» is recorded as diagnosis verdict «wrong», and «not an anomaly (false alarm)» records that incident as a false alarm → Proposals — Action Plan Review |
- Judgments are record-only — pressing them doesn't turn off alerts or change equipment, orders, or schedules.
- If you're not sure, don't dismiss. A guessed reason distorts accuracy. If the effect is unknown, leave it as Still unknown.
- Marks can be pulled via the public API
GET /api/v1/ai-feedback→ API Integration Guide
There are two marks the server applies automatically, without a person pressing anything.
| Automatic mark | What |
|---|---|
| Stop/failure rule | Cross-checks hourly whether the equipment was actually stopped/failed at that time and tags correct · false alarm · missed (accuracy.rule.enabled). If a stop/failure began and there was no anomaly verdict within ±5 minutes, it's missed. Counted separately from human-entered marks |
| Post-repair automatic confirmation | After a set time following action, answers effect confirmed · no effect · inconclusive → Anomaly Detection · Incident Response |
There is no field in the new screens for a person to mark an anomaly as "correct" or to report a missed detection by the AI. What a person leaves is the five items in the table above (dismiss reason · accept/reject · effect · diagnosis verdict · action plan rejection reason), and correct · missed are answered automatically by the "stop/failure rule" above.
Where to see accuracy
The «Reports › AI Accuracy» screen is gone. Accuracy-related numbers are viewed in four places.
| What | Where | Who |
|---|---|---|
| Per-equipment threshold tuning progress · reason · recent judgments | Admin Console Accuracy Tuning (/admin/tuning) | Admins + holders of "threshold tuning promote/revert" permission |
| Answer quality periodic test scores for AI (last 90 days) | Admin Console Answer Quality (/admin/quality) | Admins |
| Diagnosis accuracy of incident AI diagnoses — correct rate of human verdicts (30 · 90 · 180 days), by incident type · model · equipment · week · actual cause | Lower section of Admin Console Answer Quality (/admin/quality) | Admins |
| How accurate AI anomaly judgments were over the last 30 days — correct · false alarm · missed · precision totals, and equipment with low precision | Automation template "AI Accuracy Check" — report (default weekly, Mon 09:00) | The person who created that automation and the recipient role (default: plant manager) |
-
The score for actions taken (validation passed · recurrence) is on the header line of the Proposals › Completed tab.
-
The fields on the Accuracy Tuning · Answer Quality screens are in Copilot Admin Console (diagnosis accuracy is in Answer Quality), and "AI Accuracy Check" is in Use Cases — Automations.
-
Benchmark figures and the measurement method are in Accuracy · Benchmarks.
-
The "AI Accuracy Check" report includes only totals and the ten equipment with the lowest precision (within your equipment scope; if the sample is insufficient, "insufficient sample"). There is no new screen for viewing the mark rate, false-alarm reasons, or equipment still in training as a table across all equipment.
How accuracy is counted
This is how the server counts per equipment — the calculation stays the same even if the screens change.
| Value | Meaning |
|---|---|
| Precision | Correct ÷ (Correct + False alarm). Measured separately for human-entered marks and for marks from the stop/failure rule |
| Missed | Counted per source — e.g. cases where a stop/failure began but there was no anomaly verdict within ±5 minutes |
| Mark rate | Marked rows ÷ Judged rows — if low, don't trust the precision |
| False alarm reason | Setup · idle · maintenance · sensor · other |
If there are fewer than 10 marks, precision is not calculated and it's treated as "insufficient sample." A percentage from only a few cases is close to chance. To get real numbers, judgments need to be left in the field.
The diagnosis accuracy of incidents (Answer Quality screen) has a different rule — with fewer than 20 verdicts it attaches "Small sample", but the rate is still shown.
In training · Cannot be judged
For new equipment or equipment that was stopped for a long time, no score is produced until a baseline is established per tag.
| Status | Meaning |
|---|---|
| Cannot be judged | All tags are in training — no anomaly judgment is made |
| In training | Only some tags are in training — judgment is made using the rest |
| Unable to score | The engine was unable to score this equipment (unrelated to training — e.g., inference failure) |
🔴 No anomalies shown in Proposals doesn't mean normal. Equipment marked as cannot-be-judged won't surface anomalies as "anomaly to review" even if there is a problem. Remaining time is known only to the extent the engine reports; if unknown, it's shown as "unknown" — it is never guessed.
Autonomous per-equipment threshold tuning
The AI uses human marks and the stop/failure rule to autonomously tune per-equipment sensor thresholds. The only thing that changes is the judgment criteria — nothing is done to the equipment or process.
매일 밤 지난 3일로 후보 평가 → 더 나은 값이 있으면 그림자 3일
→ 통과하면 승격 → 7일 감시 → 나빠지면 자동 되돌림 (연속 2번이면 그 설비 일시정지)
- Default mode is auto (promotes without human involvement). In propose mode, values that pass shadow validation wait for an admin to promote them. In off mode, it stops (existing values remain in use).
- If marks are insufficient, nothing changes ("insufficient sample").
- In the admin console's Accuracy Tuning, view per-equipment status ("shadow day N · ready to promote · applying · observing day N · stopped") along with threshold · reason · recent judgments, and press Promote · Revert · Resume.
Shadow — record-only before notifying the field
AI verdicts that haven't yet passed validation (equipment sensor anomaly alerts · short-term failure risk) run as shadow — judged, but only recorded,
and not sent to alerts, proposals, or Ask. In the admin console's Field Alert Preview (/admin/field-modes), you can view discoveries recorded as shadow (last 48 hours),
and if false alarms aren't excessive, enter a reason and switch it on for the field → Copilot Admin Console
The old Pending Validation screen, where people marked shadow verdicts correct/incorrect, does not exist in the new screens. The permission "review shadow verdicts" remains in the permission bundle but has no place to be used in the new screens → Access Control
What admins configure
| What | Where (admin console) |
|---|---|
| Equipment importance (critical · important · normal · low) — note required when changing | Equipment Importance Rating |
| Shadow ↔ Field — turning on a judgment feature for the field (reason required) | Field Alert Preview |
| Threshold tuning promote · revert · resume | Accuracy Tuning |
| Global values and switches | System Settings — table below |
| Setting | Default | Meaning |
|---|---|---|
finding.enabled | true | Receiving AI discoveries and morning briefing |
finding.repair.enabled | true | Post-repair automatic confirmation |
anomaly.source | both | Who issues equipment anomaly alerts — pipeline · both · findings |
attention.enabled | true | Equipment to watch ranking |
accuracy.rule.enabled | true | Stop/failure rule cross-check |
anomaly.tuning.mode | auto | Autonomous per-equipment threshold tuning — auto · propose · off |
Related documents
- Proposals — Where to leave dismiss · accept/reject · effect
- Anomaly Detection · Incident Response — Detection cycle · alert channels · recurrence validation · post-repair automatic confirmation
- Copilot Admin Console — Accuracy Tuning · Answer Quality · Field Alert Preview · Equipment Importance Rating
- Use Cases — Automations — "AI Accuracy Check"
- Time Series Insight — The engine's tools