Troubleshooting

The AI stack is a single docker compose. So troubleshooting always starts from one place — find which container is misbehaving, and look at that container's logs.

cd /opt/kopens/plantpulse-ai-docker/bin
./doctor.sh      # 아무것도 바꾸지 않고 진찰만 합니다. 장애 중에 먼저 돌리십시오
./status.sh      # 무엇이 떠 있나
./health.sh      # 디스크 · 자원 · 서비스별 최근 로그까지

The stack won't come up

First, check that both sibling repositories exist

ls /opt/kopens
# plantpulse-ai-docker
# plantpulse-ai-inference-server        ← 없으면 compose 가 뜨지 않습니다
# plantpulse-timeseries-insight-server  ← 없으면 compose 가 뜨지 않습니다

compose bind-mounts files from these two as relative paths (for example, gateway config litellm-config.yaml). If they're missing, the installer refuses to start and prints the name of the missing repository → Installation Overview.

Check whether the config file exists

sudo test -f /etc/kopens/plantpulse-ai.env && echo 있음 || echo 없음

If it doesn't, rerun bin/install.sh or copy compose/env.gb10.example to that path.

🔴 Do not edit compose/.env. Nobody reads it — the source of truth is /etc/kopens/plantpulse-ai.env.

Check whether the pre-boot validation passed

bin/up.sh · restart.sh · update.sh validate the env file before bringing up compose, and if something's off, they stop without creating a single container. As of 2026.10, two causes are the most common.

cd /opt/kopens/plantpulse-ai-docker
sudo bash bin/lint-env.sh     # 줄마다 제자리인지 — 바꾸지 않습니다

Output	Cause	Action
`lines that do not belong in …` with per-line `release` · `profile` · `redundant` · `derived` · `unknown`	A 2026.09-or-earlier env was brought up as-is — image references · GPU allocation · values identical to defaults · `PP_HOST` · leftover old platform account lines	Delete with the single `sed` line it prints. Before deleting, fill in the two platform credential values → Setting Environment Variables
`renamed keys still use their old names`	The old name `CH_*` is still present	Don't delete it — run the `sed` it prints (just renames it) → see "CH_ prefix" below
`PP_HW_PROFILE is not set`	No hardware profile line	Add `PP_HW_PROFILE=gb10` for GB10, or `g7e` for AWS g7e
`still has template placeholders`	`CHANGE_ME_*` is still present	Fill in the printed key. It won't start with the value from the public template
`[FATAL] [platform-credentials] …`	Pre-boot DB provisioning couldn't get platform credentials	Same cause as "Container keeps stopping with FATAL" below
`PI_INBOUND_API_KEY is empty and PI_INBOUND_AUTH_DISABLED is not true`	Inbound key is empty	See "Backend service refuses to start" below
`PI_MEDIA_ENABLED=<value> — true 또는 false 만 받는다`	The video/audio switch was set to a different value	Fix with `true` or `false` → Video/Audio Collection
`ASR_URL … media` appears in a list of lines	The host env's `ASR_URL` differs from the value given by the video/audio switch	Delete that line. Turning on transcription is a single `PI_MEDIA_ENABLED=true` line

Lines printed as [warn] (lines to tidy — the stack starts as is) don't block startup — they're the old name MEDIA_INGEST_ENABLED (aliased for this release, alias), a ASR_URL that equals the switch's value (media-same), or a leftover old line equal to the new name's value (oldname). Clean them up with the single sed line printed alongside. The aliases are removed in 2026.11, so migrate before then.

To check whether values required by compose itself (PP_DATALAKE_ADMIN_URL · PP_DATALAKE_ADMIN_API_KEY · PP_HW_PROFILE, etc.) are empty, do this.

docker compose --env-file /etc/kopens/plantpulse-ai.env -f compose/docker-compose.yml config >/dev/null

A single container keeps restarting

docker compose ps                              # 어느 것이 Restarting 인가
docker compose logs --tail=200 <컨테이너>       # 왜 죽는가

Common cause	Check
Can't grab GPU	`nvidia-smi` · `docker compose logs plantpulse-llm`
Can't connect to platform DB	See "Platform DB" below
Out of memory	`docker stats` · `dmesg -T \| grep -i oom`
Port conflict	`ss -ltnp \| grep -E '80\|443\|4000\|5959\|5960\|7115\|8970'` — the installer may have already warned about a «port held by someone else» → Service Port Map
Couldn't get platform credentials	See "Container keeps stopping with FATAL" below
Backend service stopped with `inbound` · `INBOUND_API_KEY is empty`	See "Backend service refuses to start" below
Only the copilot refuses to start	See "Only the copilot won't come up" below

Container keeps stopping with FATAL — platform credentials

Starting with 2026.10, Copilot · Time Series Insight API · LightRAG · RAG API · Digital Thread · LiteLLM receive their account and password from the platform at startup. If they can't get them, they won't fall back to old values — they stop with FATAL — so if several of these six keep restarting at once, check here first.

docker logs --tail=50 plantpulse-copilot-web 2>&1 | grep -iE 'credentials|fatal'
sudo grep -E '^PP_DATALAKE_ADMIN_(URL|API_KEY)=' /etc/kopens/plantpulse-ai.env
ls -l /etc/kopens/ca/root.crt /etc/kopens/ca/CA.crt

Cause	Action
Only one of `PP_DATALAKE_ADMIN_URL` · `PP_DATALAKE_ADMIN_API_KEY` is present	Add both. If only one is present, the container rejects
`/etc/kopens/ca/root.crt` is missing	Set up the platform trust anchor (also as `CA.crt`). There's no way to disable certificate validation
Key is wrong, or the AI host IP isn't on the platform allowlist (4xx)	Copy `PP_DATALAKE_ADMIN_API_KEY` from the platform host `/etc/kopens/plantpulse-platform.env` verbatim, and add the AI host IP to the allowlist. 4xx stops immediately without retry
Can't reach the platform `:4950` · 5xx	Retries briefly, then stops. First check whether `curl -sk https://<플랫폼 호스트>:4950` is reachable

Host scripts (pre-boot DB provisioning · the pre-checks in install.sh START=1) also receive the same values. If it fails there, a single [FATAL] [platform-credentials] … line states the reason and stops before the container even starts — the action is the same as the table above.

On success, the log prints a line starting with credentials: showing only the key names and count. After fixing the values, recreate with bin/restart.sh → Installation Overview

Backend service refuses to start — inbound key

RAG · Time Series Insight API · Document Understanding · Vision · Digital Thread won't start if the inbound key is empty (before the 2026.10 re-release, they'd come up healthy and open without authentication).

sudo grep -c '^PI_INBOUND_API_KEY=.' /etc/kopens/plantpulse-ai.env     # 1 이면 값이 있습니다
docker logs --tail=50 plantpulse-rag-api 2>&1 | grep -iE 'inbound|fatal'

Symptom	Cause	Action
`bin/up.sh` creates no containers and prints `[fail] PI_INBOUND_API_KEY is empty …`	The line is missing from env, or the value is empty — pre-boot validation stops	Add it via `sudo bin/passwd.sh --generate PI_INBOUND_API_KEY` (bring the stack up again after editing the file)
`[fail] PI_INBOUND_AUTH_DISABLED=<value> — write true or false`	The no-auth switch was set to a different value	Fix with `true` · `false`, or delete the line
A line like `FATAL: RAG_INBOUND_API_KEY is empty — refusing to start …` in the logs	An empty key got into the container — it was started bypassing compose, or the image was run separately	Set the key as above and recreate with `bin/restart.sh`
The stack comes up, but RAG/time-series calls from the admin console or chat get `401`	Only some services were recreated after changing the key	Recreate everything with `bin/restart.sh` — see "Admin console shows 502 · upstream_auth" below

The only way to run without authentication is the dev-only switch PI_INBOUND_AUTH_DISABLED=true, and if a key is present, that switch is ignored ([note] … is ignored — PI_INBOUND_API_KEY is set and the key wins). For production installs, setting the key is the answer → Setting Environment Variables

Only the copilot won't come up — stored credential encryption key

If the rest of the stack is green but only plantpulse-copilot-web refuses to start, check PI_COPILOT_SECRET_KEY.

docker logs --tail=50 plantpulse-copilot-web 2>&1 | grep -iE 'SECRET_KEY|secrets'

Log	Cause	Action
Prints to add the `PI_COPILOT_SECRET_KEY=…` line	The key is missing or malformed — hosts upgraded from 2026.09 or earlier lack the line (the installer doesn't add it to an existing env)	Add it with the `openssl rand -base64 32` value and recreate → Setting Environment Variables
`Stored secrets cannot be decrypted`	The key was changed, but the old value wasn't provided	Place the old value in `PI_COPILOT_SECRET_KEY_PREVIOUS` → Rotation Procedure. If the old value is lost, clear the sealed row with the SQL printed in the log and re-enter that credential from the screen

If Digital Thread's PI_DT_SECRET_KEY is empty, the stack still comes up but only the connector registry responds with config_error (503). Pre-boot validation (bin/lint-env.sh) reports if both keys are missing as «stored credential encryption key is missing.»

Changed settings, but copilot ignores them — CH_ prefix

In 2026.10, the copilot environment variable prefix changed from CH_ → CO_, and there's no fallback. Old env lines like CH_API_V1_KEY · CH_LLM_MAX_COMPLETION_TOKENS · CH_OTEL_GENAI_CONTENT are no longer read by the copilot. Simply deleting the lines reverts the value to default — for example, an external integration calling via CH_API_V1_KEY will no longer be able to get in with that key.

Pre-boot validation flags these lines as renamed and halts startup, printing a single rename-only sed line per line (preserving the value, leaving a backup). Run it and recreate with bin/restart.sh plantpulse-copilot-web.

cd /opt/kopens/plantpulse-ai-docker
sudo bash bin/lint-env.sh 2>&1 | grep -A2 'renamed'

If both the old and new names exist with different values, the new name wins — verify which value is correct, then delete the old line → Setting Environment Variables

No logs after bringing it up, or writes fail — volume ownership

Starting with 2026.10, every container except for one plantpulse-dcgm runs non-root (our images use uid 10001, gid 0). Write volumes that were created as root in an old install (app logs · time series fine-tuning weights · LightRAG · Qdrant · Neo4j · Phoenix storage, vLLM cache, proxy certificates) can no longer be written to directly. These failures are usually silent (e.g. showing up only on console instead of in a log file), Qdrant dies immediately, and the proxy fails to read the certificate key, prints a prescription, and stops.

bin/up.sh · restart.sh · update.sh fix ownership once with bin/volumes-nonroot.sh before bringing things up. If you called docker compose up directly by hand, you skipped this step.

cd /opt/kopens/plantpulse-ai-docker/bin
./volumes-nonroot.sh     # 이미 맞으면 아무것도 바꾸지 않습니다
./restart.sh

If ownership can't be fixed, it prints 볼륨 소유권을 <uid:gid> 로 맞추지 못했다 and halts startup → Operating Services

Can't connect to the platform DB

🔴 The AI stack doesn't run PostgreSQL · Aster TSDB · Redis · RustFS. It shares the platform's installation. If any of these four go down, AI stops too.

# 플랫폼 호스트는 이 URL 의 호스트입니다 (PP_HOST 는 따로 적지 않습니다)
sudo grep -E '^PP_DATALAKE_ADMIN_URL=' /etc/kopens/plantpulse-ai.env

# 실제로 닿는가
nc -zv <플랫폼 호스트> 5432    # PostgreSQL
nc -zv <플랫폼 호스트> 9042    # Cassandra

If unreachable, it's a platform-side problem → Service Port Map.

Platform accounts aren't in env (2026.10)

Accounts, passwords, and ports for PG · Aster TSDB · Redis · RustFS are received from the platform at startup, by both containers and host scripts. If authentication is rejected, look not at env but at the platform's values and "Container keeps stopping with FATAL" above. If you changed a password on the platform, you must restart the AI containers for the new value to take effect.

Admin console shows 502 · upstream_auth

If one menu in the console screen fails with … rejected the console's inbound API key (401) - check <변수>, the service behind the console has rejected the console's inbound key — this is not console session expiry. Older versions interpreted this 401 as session expiry and simply redirected to the login screen (re-logging in didn't help).

The cause is that the backend and console are running with different PI_INBOUND_API_KEY. Usually only one side was recreated, or only some containers were restarted after changing the key, so recreate the entire stack.

cd /opt/kopens/plantpulse-ai-docker && bin/restart.sh

The admin console address as of 2026.10 is http://<AI 서버>:5959 · https://<AI 서버>:5960 (formerly 9080).

LLM screen shows 502 · llm_upstream

If the LLM read screens in the console (API keys · logs · model registry list · upstream status, etc.) fail with 502 and reason llm_upstream, the gateway (LiteLLM) or its DB returned a 5xx — the console backend itself isn't broken. The upstream_status in the response is the status code the gateway returned. Older versions passed this through as 500, which looked like a console failure.

docker compose logs --tail=100 plantpulse-litellm    # 게이트웨이가 왜 5xx 를 냈나 (DB 접속 등)

If the gateway is completely unreachable, the same 502 shows reason llm_unreachable. For 4xx (input error), the gateway's reason is shown as-is on screen.

No answer comes back

Narrow it down in order — gateway → model → screen.

docker compose logs --tail=100 plantpulse-litellm    # 게이트웨이가 요청을 받았나
docker compose logs --tail=100 plantpulse-llm        # 모델이 답했나
docker compose logs --tail=100 plantpulse-copilot-web

Symptom	Common cause
No request appears in gateway logs	Wrong gateway address/key set on screen
Gateway returns 401	`PI_LLM_API_KEY` mismatch
Only external integrations fail with model name error	Calling an old name (`aura-35b` · `gpt-4o` · `text-embedding-3-large`, etc.) — now it's `aura-lite` · `aura-embedding-4b` → API Integration Guide
Model takes a long time then disconnects	Out of GPU memory — `nvidia-smi`
"Could not generate an answer ... the model returned an empty response"	Retried, but the model still returned an empty answer (via API this is `502`). If repeated, check thinking mode and generation cap → Setting Environment Variables
"⚠ The same content repeated, so answer generation was stopped"	The model repeated itself and the copilot cut it off. Narrow the question and ask again

"Connection lost. The server is still generating an answer..."

Only the connection was dropped mid-question (network switch, proxy disconnect, etc.). The new screen's message reads "Connection lost. The server is still finishing the answer, and it will be loaded here as soon as it's done." Since the server finishes generating and saves the answer, the screen displays this message and re-checks that conversation's answer every 5 seconds, for up to 5 minutes, loading it once it arrives. Just wait — the same applies on mobile. If not found within 5 minutes, it switches to an error message with a Retry button remaining — in that case, first reopen the conversation to check whether the answer was saved.

Can't find documents (RAG)

docker compose logs --tail=100 plantpulse-rag-api
docker compose logs --tail=100 plantpulse-lightrag
curl -s http://127.0.0.1:6333/healthz     # Qdrant(벡터 색인)

Symptom	Where to look
Chat only says "Document search is unavailable right now (the MCP gateway is not responding) …" and does not answer	Document search in chat goes only through the MCP Gateway. Check whether the gateway is up — `docker compose logs --tail=100 plantpulse-mcp-gateway-api`, and `[MCP_GATEWAY] unreachable` · `[DOC_SEARCH_GUARD]` in the copilot log. At the same time, answers to other questions start with "The tool gateway is unreachable …"
Chat only says "Document search is turned off or not allowed for this account …" and does not answer	The gateway responds but `rag_search` does not reach the model. In AI Admin Console › MCP Gateway, check that the RAG Gateway under Servers and `rag_search` under Tools are on (both are on from the start, so someone turned them off). If it is on, check that person's chat tool permission (`tool:mcp`) and the gateway's «Permissions» tab → RAG — Document search in chat
An uploaded document doesn't show up in search	Whether indexing finished — `plantpulse-rag-api` log · Sync History's skip reason → RAG
Document upload fails	`plantpulse-docling-serve` · `plantpulse-document-understanding-api`
The answer cites the wrong document	Index is stale — re-upload and reindex
The AI only says "couldn't find it in the documents"	May be expected behavior — if search comes back empty, it doesn't fabricate an answer and returns `retrieved: false` instead. Check with `rag_list_documents` first whether that document is indexed
RAG `/health` is `degraded`	None of `ingest` · `neo4j` · `qdrant` in the response is `ok`

A large PDF fails partway through indexing

LightRAG extracts entities per document chunk using the LLM. If a single chunk doesn't finish in time, the whole document fails, and the remaining chunks aren't even attempted.

docker compose logs plantpulse-lightrag | grep -E 'Worker execution timeout|workers initialized'

Log	Meaning
`extract LLM func: Worker execution timeout after 480s`	The install is running with the old default (240 seconds per call)
`(Timeouts: Func: 3600s, Worker: 7200s, …)`	This is the current default

The time budget is LIGHTRAG_LLM_TIMEOUT in /etc/kopens/plantpulse-ai.env (default 3600 seconds), and the actual limit is double that. If indexing gets cut off during busy periods when chat and queries flood the model and slow it down, check this value. After changing it, you need to recreate (bin/restart.sh), not just restart.

Don't just increase the timeout

The per-chunk output cap (LIGHTRAG_LLM_MAX_TOKENS, default 32768) is tied to this and is what makes a longer budget safe. If you remove the cap and only raise the budget, a single runaway request can hold a GPU slot for an hour.

Document/photo tools fail with "refused to fetch"

Document Understanding · Vision tools refuse a URL that resolves to an internal address (loopback · private network · link-local). If the reason shows resolves to a non-public address, this is the case.

To intentionally allow an internal file server, list that hostname in FETCH_ALLOWED_HOSTS and recreate both services. It must match exactly what's used in the URL (if a port is attached, only that port is allowed).

echo 'FETCH_ALLOWED_HOSTS=files.example.local' | sudo tee -a /etc/kopens/plantpulse-ai.env >/dev/null
cd /opt/kopens/plantpulse-ai-docker
bin/restart.sh plantpulse-document-understanding-api plantpulse-vision-api

→ Setting Environment Variables · MCP Tool Map

Drawings get indexed as regular documents, or the drawing screen doesn't appear

Drawing collection is off by default (DRAWING_INGEST_ENABLED=false). While off, rag-api doesn't even have a drawing API, and DXF · PDF drawings aren't read as drawings.

docker exec plantpulse-rag-api printenv DRAWING_INGEST_ENABLED

If false, set DRAWING_INGEST_ENABLED=true in the host env, recreate with bin/restart.sh plantpulse-rag-api, then re-upload the drawings — files uploaded before turning this on won't be reread as drawings. PDFs must be given the type «drawing» at upload time to be read as drawings → Setting Environment Variables

Console message	Meaning
`drawing is platform-managed — update it in the platform, not here` (`409`)	Tried to overwrite via the console a drawing that came in as a platform asset document (2026.10 re-release 3). Fix this on the platform
`drawing is platform-managed — delete it in the platform ledger, or exclude it from RAG (POST …/exclude)` (`409`)	Tried to delete a platform ledger drawing via the console. Deletion happens in the platform ledger; to only exclude it from search, «exclude from corpus»
`manually uploaded drawings cannot be excluded — delete the drawing or re-index it instead (platform-ledger drawings can be excluded)` (`409`)	A drawing uploaded directly via the console can't be «excluded from search.» Delete it or reindex it. Platform ledger drawings can be excluded
`extension not allowed` (`400`)	It's a DWG. Export to DXF or PDF and upload

Files added to chat cannot be read

Files added with Add file in the Ask input box are received by the copilot, kept in object storage, and handed to the Document Understanding service (plantpulse-document-understanding-api) to read — documents with docling, drawings with drawing reading, video · audio with transcription (/api/v1/transcribe → plantpulse-asr). How to use it is in AI Chat — Adding files.

docker logs --tail=200 plantpulse-copilot-web 2>&1 | grep -E '\[CHAT_ATTACH|\[attach\]|\[CHAT_MEDIA'   # upload · reading · transcription · failure reasons
docker compose logs --tail=100 plantpulse-document-understanding-api
docker compose logs --tail=100 plantpulse-docling-serve
docker compose logs --tail=100 plantpulse-asr                       # video · audio — runs only when PI_MEDIA_ENABLED=true

Rejected at upload — the chip shows the sentence below as is, and that file is not sent.

What the chip says	Cause · what to do
"This file type cannot be added. Add an office file (xlsx, xls, csv, docx, pptx), a PDF, a drawing (DXF, PDF), a video (mp4, mov, mkv, webm) or an audio file (m4a, mp3, wav)."	Not an accepted format. The format is judged by content, so just changing the extension does not get it in
"The name and the content of this video or audio file do not match. …"	The extension says video · audio, but the file header is not that format. If only the extension was changed in a converter, export it again in its original format
"The video or audio file is too large (max 1 GB)."	Video · audio can be up to 1GB. Split it or lower the resolution, then add it
"The video or audio is too long (max 60 minutes). Cut it and add the parts."	It is longer than 60 minutes. The first part alone is not taken either — cut it and add the parts
"Too many files are waiting for transcription right now. Please add it again in a moment."	Across the whole installation, 4 video · audio files are waiting for transcription or being uploaded. Add it again once the earlier ones are processed
"DWG files cannot be read. Export the drawing to DXF or PDF and add that."	Export to DXF or PDF from CAD and add that
"Add pictures as images (drop them on the input and they attach as images)."	A photo file came in through the attachment path. Add it to the input box again and it attaches as a photo
"The file is too large (max 50 MB)."	One file can be up to 50MB. Split it and add the parts
"You can add up to 20 files to one conversation."	This conversation already has 20. Add it in a new conversation
"The file is empty."	A 0-byte file
"Could not read this file: it is damaged or was not fully received."	The file is broken or was cut off while uploading. Open the original to check, then add it again
"The file structure is too large to accept (unpacked size or entry count limit)."	The Office file contains more than 20,000 entries or unpacks to more than 500MB
"Could not store the file. Please add it again in a moment."	It could not be written to object storage. Check the copilot logs and the object storage status

Reading failed after upload — the chip shows "Could not read this file: reason". The reason is an English sentence left by the server. This file can still be sent, and the AI knows it as «a file that could not be read».

Reason	Cause · what to do
`document reader unreachable` · `drawing reader unreachable`	The Document Understanding service could not be reached. Check that `plantpulse-document-understanding-api` is up
`document reader is not configured`	The copilot has no Document Understanding service address. This does not happen with the default compose — check whether the settings were changed by hand
`reading took longer than 10 min`	Reading took more than 10 minutes. It is a large PDF, or it overlapped with indexing work. Remove the file and add it again when things are quiet, or cut out only the pages you need
`too many files are being read — remove it and add it again`	Too many files are being read at once. Remove the chip and add it again shortly
`reading was interrupted — remove the file and add it again`	The copilot restarted during reading (if there is no progress for more than 15 minutes it is folded this way). Remove it and add it again
`document reader answered 503` and other numbers	The Document Understanding service is busy or the key does not match. Check the Document Understanding logs
Any other sentence	The reason exactly as given by the Document Understanding service (e.g. an old Office file that cannot be opened). Re-save the original as PDF or a newer format (xlsx · docx · pptx) and add it

Video · audio transcription failed — the chip shows "Could not transcribe: reason" or "Not accepted: reason". The reason is the English sentence given by the Document Understanding service, as is. Temporary failures such as connection · congestion are retried three times, after 1 · 5 · 30 seconds, before being folded as a failure; configuration · input problems are folded as a failure at once. A failed file can still be sent and its original still plays, but the AI knows it as «a file that could not be transcribed».

Reason	Cause · what to do
`media transcription is disabled: ASR_URL is not set`	Video · audio is off on this installation. An administrator adds `PI_MEDIA_ENABLED=true` to the host env, pulls the transcription model and brings the stack up again → Video · Audio Ingestion
`media transcription is misconfigured: …`	The transcription settings do not match (serving name, etc.). Check Document Understanding's `/health` and logs
`transcription service unreachable` · `transcription service answered 503` and others	The Document Understanding service or the transcription serving (`plantpulse-asr`) could not be reached or was busy. Check that both are up, then remove the file and add it again
`transcription service is not configured`	The copilot has no Document Understanding service address. This does not happen with the default compose
"Not accepted: …" (length exceeded)	The length could not be read from the header, and the accepted file went past 60 minutes during transcription. Cut it and add the parts

Other symptom	Cause
Send cannot be pressed and "Reading files — you can send once they are read" stays	Reading has not finished. It is released when reading completes or fails. Removing the file being read also releases it
"The attachment is still being read. Send once it is ready."	At the moment of sending, a file was still being read on the server. Send again after the chip shows «Read»
Clicking a chip shows "This file was not found. It was deleted or you cannot view it"	The file was deleted, or it belongs to someone else's conversation. Attachments open only for the conversation owner (administrators too)
Clicking a chip shows "Could not load the file. Please open it again in a moment"	The original or the read text could not be fetched from object storage
The file shows «Read», but the AI says it does not know the file's content	If the person lacks the chat tool permission `tool:builtin`, the AI does not get the tool that reads attachments → Permissions
The AI does not know the latter part of a long document	Only the first 100 pages are read (each sheet · slide counts as one page). Cut out the part you need and add it
Dimensions · BOM of a PDF drawing are read wrongly	It was read as a document. Press the chip's Drawing button to have it re-read as a drawing, then send
You added a video and asked, but the AI says it is "still transcribing"	Transcription has not finished. For 5 minutes · 200MB or less the question waits up to 20 seconds; longer files are transcribed in the background. Ask again once the chip shows "Transcript ready"
The chip stays at "Waiting for transcription" for a long time	Transcription runs one file at a time across the whole installation, in the order added (short files first). It is waiting for the earlier files to finish. If it stays that way, check the `[CHAT_MEDIA` logs
"No speech or subtitles found"	There is neither speech nor embedded subtitles. On-screen text · burned-in subtitles are not read
The video does not play	It is a codec the browser cannot open (e.g. H.265). Transcription and answers still work
HTML 413 appears while uploading a large video	The proxy's chat attachment path is open up to 1040MB (it does not follow the host's `MEDIA_MAX_FILE_MB`). Check whether the proxy image was changed by hand or another proxy sits in front

"Voice Question" ends with "Voice input is currently unavailable"

Chat's Voice Question sends the recording to Document Understanding's transcription endpoint, and Document Understanding calls the transcription serving (plantpulse-asr). This message ("...the transcription service is off or busy") means somewhere in that path is closed. The status code returned by Document Understanding is left in the copilot logs.

docker logs --tail=100 plantpulse-copilot-web 2>&1 | grep '\[voice\]'      # 예: [voice] doc-und answered 503: …
docker exec plantpulse-document-understanding-api printenv ASR_URL          # 비었으면 영상 · 음성이 꺼진 설치
cd /opt/kopens/plantpulse-ai-docker && bin/status.sh                        # plantpulse-asr · plantpulse-asr-audio

Cause	What you'll see	Action
Video/audio is off (default)	Document Understanding's `ASR_URL` is empty, and the transcription endpoint is `503` (`media transcription is disabled: ASR_URL is not set`)	Add `PI_MEDIA_ENABLED=true` to host env, get the transcription model, then `bin/restart.sh` → Video/Audio Collection
Turned it on, but Document Understanding is still running with old values	`ASR_URL` is still empty	Recreate (not restart) with `bin/restart.sh`, not `docker restart`, after switching
`plantpulse-asr` is `unhealthy` · keeps restarting	Document Understanding `/health`'s `components.asr.status` is `down`, overall `status` is `degraded`	`docker logs plantpulse-asr` — if the model (`/opt/kopens/models/Aura-ASR-Turbo`) is missing, fetch it with `bin/fetch-models.sh`. First startup may take long due to compilation
`plantpulse-asr-audio` exited with a non-zero code	Not "Exited (0)" in `bin/status.sh`. Anything depending on this, such as `plantpulse-asr`, also won't start	Check `docker logs plantpulse-asr-audio`, confirm the image was pulled, then `bin/restart.sh`
Transcription serving is busy	Overlapping with video processing	Try again shortly

plantpulse-asr-audio remaining as "Exited (0)" is normal — it's a one-shot service that copies audio libraries to a volume and finishes, and bin/status.sh counts this as normal too.

Other message	Cause
"Voice input only works on HTTPS addresses"	Browsers don't grant mic access on plain-text addresses. Connect via `https://`
"Microphone permission was denied"	Address bar lock icon → Allow microphone
"Recording is too long (max 60 seconds)" · "Recording is too large (max 4MB)"	A single question is capped at 60 seconds · 4MB
"Could not understand the speech"	Transcription returned empty. Get closer to the mic and try again

MCP tools don't show up

If PP_API_URL · PP_API_TOKEN are empty, the stack won't start, but if the token is wrong, the stack still comes up normally and only the platform MCP tools in chat show 0. So it doesn't look like a failure.

sudo grep -E '^PP_API_(URL|TOKEN)' /etc/kopens/plantpulse-ai.env

After filling in the values, recreate instead of restart is needed — docker restart doesn't re-read env.

cd /opt/kopens/plantpulse-ai-docker && bin/restart.sh

Time series analysis isn't working

docker compose logs --tail=100 plantpulse-timeseries-insight-api    # :8970
docker compose logs --tail=100 plantpulse-timeseries-insight-server # :8980 (호스트에는 안 열립니다)

Both of these read values from the platform's Aster TSDB. Check "Platform DB" above first.

Forecasting returns 500, or everything comes back NAIVE

A config error doesn't kill the server — since the same process also serves anomaly detection, unrelated features aren't halted. First check the inference server's /health.

# 이 이미지에는 curl 이 없습니다 — python 으로 찌릅니다
docker compose exec plantpulse-timeseries-insight-server \
  python -c "import urllib.request; print(urllib.request.urlopen('http://localhost:8980/health').read().decode())"

If models.ttm.loaded: false shows status: "degraded", the forecast model failed to load, and the reason is in error. If the models item is missing entirely, it's an older version that the healthcheck won't show — identify it from the logs below instead.

docker compose logs plantpulse-timeseries-insight-server | grep -E 'MODEL_LOAD|WARMUP|INFER_FORECAST|fallback naive'

Log	Meaning	Action
`[MODEL_WARMUP_FAIL] TTM ch=… 설정이 체크포인트와 맞지 않는다`	Checkpoint generation and the loader class mismatch. The forecast returns 500	Either leave `TS_TTM_MODEL_CLASS` blank (letting the checkpoint decide) or match it to the generation → Explanation
`[MODEL_LOAD_FAIL] … 무작위 초기화 모델로 대체하지 «않는다»`	Same cause. The point where loading was rejected	Same as above
`[MODEL_WARMUP_SKIP] TTM ch=…`	Same cause in a version prior to 2026-09-21. In that version, forecasts come back 200 with `model_used="NAIVE"`	Same as above
`TTM 을 돌릴 데이터가 모자라다 -> fallback naive. reason=…`	Insufficient data. Returns 200 with `NAIVE` — the current version only absorbs this case with simple extrapolation	Read `reason`
`TTM failed -> fallback naive. reason='<태그>_prediction_q0.1'`	All forecasts come back as `NAIVE`. The R3 weights are loaded, but it's an image missing the quantile column name fix (e.g. `2026.09-20260924-1004`)	Re-pull an image with the fix (`./update.sh`)
`TTM failed -> fallback naive. reason=…` (other)	Fallback from an older version — that version also swallowed failures other than data shortage as 200 `NAIVE`	Read `reason`

Where it answered with NAIVE, the same reason is also carried in the response's fallback_reason — check there for places not captured in logs.

"All NAIVE" and "500" are two versions of the same disease

A config error that disguised itself as simple extrapolation was fixed on 2026-09-21. Older versions could silently run with /health normal · forecast 200 · every answer quietly being simple extrapolation. Either way, look in the same places — the logs above and TS_TTM_MODEL_CLASS.

Don't rely on the version number — this month's image is re-baked multiple times as 2026.09-<date>-<시각>. The log lines above are the most reliable way to tell.

Changed a setting, but not a single number moved

That means the setting didn't take effect. Some parameters in the model library silently swallow unknown names, so a wrongly passed value gets ignored without error. In fact, anomaly detection's aggregation/smoothing parameters (aggr_win_size · smoothing_window_size) were found to have been completely ignored after a library argument rename (2026-07-24 audit), with no error or warning the entire time.

After changing a setting, check "did the numbers actually change"

"No error occurred" is not confirmation. Compare the score distribution or detection count before and after the change — if not a single number moved, assume the setting didn't take effect.

The fastest test is trying an extreme value — if you change a value drastically and the result is identical, that setting isn't getting through.

Disk is filling up

docker system df
df -h /data1

Container logs are rotated by docker, so there's nothing to manually delete there. Usually it's images and volumes.

docker image prune -f      # 태그 없는 옛 이미지

Don't run docker volume prune carelessly

Neo4j · Qdrant · LightRAG storage live on volumes. If run while the containers are down, the knowledge graph and vector index disappear.

Emergency recovery

cd /opt/kopens/plantpulse-ai-docker/bin
./restart.sh                       # 스택 전체. 0 이면 «쓸 수 있다» 는 뜻입니다
docker compose restart plantpulse-lightrag   # 하나만

To pull fresh images and bring things up:

./update.sh

Where the data lives

LightRAG source documents · indexes · knowledge graph live on named docker volumes (pi-lightrag-inputs · pi-lightrag-storage · pi-qdrant-data · pi-neo4j-data, and as of 2026.10 the factory graph mirror's pi-digital-thread-neo4j-data and time series fine-tuning weights pi-ttm-tuned-models). Since these aren't host directories, rm -rf won't delete them, but conversely, a single docker volume prune wipes them all out together. Always back up before deleting — bin/backup.sh captures the compose volumes (excluding logs/cache).

The stack won't come up​

First, check that both sibling repositories exist​

Check whether the config file exists​

Check whether the pre-boot validation passed​

A single container keeps restarting​

Container keeps stopping with FATAL — platform credentials​

Backend service refuses to start — inbound key​

Only the copilot won't come up — stored credential encryption key​

Changed settings, but copilot ignores them — CH_ prefix​

No logs after bringing it up, or writes fail — volume ownership​

Can't connect to the platform DB​

Admin console shows 502 · upstream_auth​

LLM screen shows 502 · llm_upstream​

No answer comes back​

"Connection lost. The server is still generating an answer..."​

Can't find documents (RAG)​

A large PDF fails partway through indexing​

Document/photo tools fail with "refused to fetch"​

Drawings get indexed as regular documents, or the drawing screen doesn't appear​

Files added to chat cannot be read​

"Voice Question" ends with "Voice input is currently unavailable"​

MCP tools don't show up​

Time series analysis isn't working​

Forecasting returns 500, or everything comes back NAIVE​

Changed a setting, but not a single number moved​

Disk is filling up​

Emergency recovery​

See also​