Skip to main content

Logging and Monitoring

Checking Logs​

Logs are managed by Docker. You do not need to manually delete log files on the host — rotation is configured in compose.

cd /opt/kopens/plantpulse-ai-docker/bin

./logs.sh                    # 스택 전체를 따라 본다
./logs.sh plantpulse-copilot-web   # 서비스 하나 — compose 서비스 이름 그대로 (여러 개 나열 가능)
./logs.sh --tail 50          # 기본보다 «적게» 보고 싶을 때
No need to append -f

logs.sh is already followed via -f. The value of --tail appended at the end overrides the default.

You can also invoke compose directly.

cd /opt/kopens/plantpulse-ai-docker
docker compose logs -f --tail=200 plantpulse-rag-api
docker compose logs -f plantpulse-litellm

Which containers to check​

SymptomContainers to check
UI does not loadplantpulse-proxy · plantpulse-copilot-web
No responseplantpulse-litellm · plantpulse-llm
Cannot find documentsplantpulse-rag-api · plantpulse-lightrag
Cannot ingest documentsplantpulse-docling-serve · plantpulse-document-understanding-api
Time series analysis failsplantpulse-timeseries-insight-api · -server
Admin console is abnormalplantpulse-admin-api · plantpulse-admin-web
Connector · graph mirror · equipment timeline are abnormalplantpulse-digital-thread-api · plantpulse-digital-thread-neo4j
Video is not indexed · voice queries do not work (in installations with video · audio enabled)plantpulse-asr · plantpulse-asr-audio · plantpulse-document-understanding-api · plantpulse-rag-api

The full list of containers and their roles is in Installation Overview.

Observing AI Tool Calls — The Tool Row in Queries and Phoenix​

There are two places to see what the AI did while generating a response.

LocationWhatHow to open
Tool row above the query answer (「Used N tools · N seconds」)Tools called for that answer, values queried, failed toolsDirectly in Query (/chat)
Arize Phoenix (plantpulse-phoenix)LLM call traces — including prompt · response bodyNot open on the host — open with PP_DEBUG_PORTS=1 only when diagnosing → Service Port Map
Phoenix is enabled by default — only the host port is closed

Phoenix is part of the default stack and continuously collects traces. From 2026.09, port 6006 is simply not exposed on the host — because it would expose query and response bodies without authentication. The Logs screen for LLM in the Admin Console shows only model · token · latency; the body is available only in Phoenix.

Expanding the tool row attached to each query answer lets you see which tools were called with which values, and if any failed. Rounds executed by Automation are recorded in the automation execution history as 「Evidence — N tool calls」.

Service Status Monitoring​

cd /opt/kopens/plantpulse-ai-docker/bin

./status.sh    # compose ps + 외부 노출 지점(copliot-web :80) 응답 확인
./health.sh    # 위에 더해 호스트 디스크 · 컨테이너 자원 · 서비스별 최근 로그
./doctor.sh    # 무엇이 잘못됐는지 진찰 — 아무것도 바꾸지 않습니다

For a quick check, use status.sh; during an outage, view broadly with health.sh then narrow down with doctor.sh.

Health Check Endpoints​

ServiceHealth Check URLNotes
AI Chat Webhttp://127.0.0.1/actuator/healthAccessed through the proxy
RAGhttp://127.0.0.1:7115/healthrag-api. Works without a key (exempt path)
Time Series Insightshttp://127.0.0.1:8970/healthWorks without a key (exempt path)
LiteLLMhttp://127.0.0.1:4000/health/liveliness/health may require the master key
Admin Consolehttp://127.0.0.1:5959/healthContainer-native health is at /ping. Port is PI_ADMIN_WEB_PORT(default 5959) — use that value if changed. Service status aggregation is in the console UI
Digital ThreadNot reachable from the host — Digital Thread API card on the Admin Console home7140 is not exposed. The card reads /health, and if the graph mirror projection is failed, it degrades to degraded(yellow). The status of connector periodic synchronization (connector_sync) is also shown here, but it is informational only and does not trigger a yellow indicator
Document UnderstandingNot reachable from the host — docker exec plantpulse-document-understanding-api python3 -c "import urllib.request;print(urllib.request.urlopen('http://localhost:7120/health').read().decode())"In installations with video · audio enabled, components.asr.status must be up(disabled installations are disabled — normal). If transcription serving dies, it is degraded
7114 is a dead address

Previously LightRAG was directly exposed via 7114, but now rag-api(7115) is the only gateway and LightRAG itself is only inside the container. Do not probe 7114 and conclude it is dead — it was not there in the first place.

Things not exposed on the host

Qdrant · vLLM · Embedding · Reranker · docling · Time series inference · admin-api do not expose host ports(2026-08-21). Phoenix from 2026.09, and the Digital Thread API(7140) and its dedicated Neo4j from 2026.10 are the same. Because they only call each other inside the container, probing with 127.0.0.1 will not work. If diagnosis is needed, revive them via the debug overlay → Service Port Map

Unified MCP is part of the Platform

Since the Platform provides server-web, the health check also looks at the Platform side.

Metrics — /metrics​

Since the 2026.10 re-release, five Python backends(RAG · Time Series Insights API · Document Understanding · Vision · Digital Thread) Prometheus text format metrics at GET /metrics. There is no new port; it is a path attached to each service's original port.

ServiceMetrics addressFrom host
RAGhttp://127.0.0.1:7115/metricsReachable
Time Series Insights APIhttp://127.0.0.1:8970/metricsReachable
Document Understanding · Vision · Digital Threadplantpulse-document-understanding-api:7120 · plantpulse-vision-api:7130 · plantpulse-digital-thread-api:7140 of /metricsNot reachable — scrape within the same compose network
curl -s -H "X-API-Key: <PI_INBOUND_API_KEY>" http://127.0.0.1:7115/metrics | grep '^http_requests_total' | head
/metrics requires a key

Unlike the health paths(/ping · /health), it is behind the inbound key. When scraping with Prometheus, provide PI_INBOUND_API_KEY via the X-API-Key header or authorization:(Bearer) configuration. Calling without a key results in 401.

MetricMeaning
http_requests_totalNumber of processed requests — by method · path template · status class(2xx etc.)
http_request_duration_secondsRequest latency histogram — same labels
http_requests_in_flightNumber of currently in-progress requests
app_build_infoService name · version · release(release label, e.g. 2026.10). Value is always 1

The path label is a path template, not the actual path, and requests that do not match any route are collapsed into <unmatched> — no time series is created for each path containing an ID. Basic metrics such as process · GC are also output.

Tracing a single request in logs — X-Request-ID​

The same five services assign an id to each request, return it in the response header X-Request-ID, and populate it in every log line during that request via request_id. If the caller sends a value in X-Request-ID(1–128 characters, alphanumeric · . _ -), that value is used as-is. Use this single value to match errors displayed in the UI to logs.

docker compose logs plantpulse-rag-api 2>&1 | grep '<request_id 값>'

When the Disk is Full​

You do not need to manually delete log files — container logs are rotated by Docker. What usually consumes disk space is images and volumes.

docker system df                 # 무엇이 얼마나 쓰고 있나
docker image prune -f            # 태그 없는 옛 이미지 정리
Do not casually run docker volume prune

Two Neo4j instances · Qdrant · LightRAG storage · time series models trained per installation(pi-ttm-tuned-models) and TLS certificates are in the volumes. If run while containers are down, the knowledge graph and vector index, as well as models trained in the field, will be lost.

Since health.sh shows both host disk and container resources, check it before cleaning up.

See Also​