Troubleshooting Guide
Service Startup Failures
Symptom: A specific service is in the [DOWN] state
./status.sh
[DOWN] CASSANDRA (9042)
[DOWN] RAG (7114)
Causes and resolutions:
| Cause | Resolution |
|---|---|
| Leftover process from a previous run | Check the PID file, then terminate the process |
| Port conflict | Identify the occupying process with ss -tlnp | grep <port> |
| Dependent service not started | The DB must start first (verify the startup order) |
| Disk full | Check the disk with df -h, then run ./clean.sh |
# PID 파일로 프로세스 확인
cat /home/kopens/plantpulse-ai/<module>/*.pid
kill -9 <PID>
rm /home/kopens/plantpulse-ai/<module>/*.pid
# 모듈 재시작
cd /home/kopens/plantpulse-ai/<module>/bin
./stop.sh
./start.sh
Symptom: The entire system fails to start
Check in this order:
# 1. 환경 변수 파일 확인
source /home/kopens/plantpulse-ai/template/env.sh
echo $PP_HOME # /home/kopens 이어야 함
echo $JAVA_HOME # JDK 경로 확인
# 2. Java 확인
$JAVA_HOME/bin/java -version
# 3. Python 확인
python3 --version
# 4. 설치 로그 확인
tail -100 /home/kopens/plantpulse-ai/logs/setup.log
Database Problems
Cassandra starts slowly / times out
Cassandra takes 20–60 seconds to start. start.sh waits automatically, but when starting manually you must allow enough time.
# Cassandra 상태 확인
nodetool status
# Cassandra 로그 확인
tail -50 /home/kopens/plantpulse-ai/db/cassandra/logs/*.log
# 키스페이스 재생성 (초기화 필요 시)
cd /home/kopens/plantpulse-ai/db/cassandra/support
./keyspace-create.sh
PostgreSQL connection failure
# PostgreSQL 상태 확인
pg_isready -h 127.0.0.1 -p 5432
# PostgreSQL 로그 확인
tail -50 /home/kopens/plantpulse-ai/db/postgres/logs/*.log
# 연결 테스트
psql -h 127.0.0.1 -U ch -d ch -c "SELECT 1"
Main causes:
max_connectionsexceeded → increase it in the PostgreSQL settings- Insufficient disk space → check
df -h - Stale PID file → delete the
.pidfile indb/postgres/and restart
Neo4j startup failure
# Neo4j 로그 확인
tail -50 /home/kopens/plantpulse-ai/db/neo4j/logs/neo4j.log
# 힙 메모리 부족 시 neo4j.conf 수정
vi /home/kopens/plantpulse-ai/db/neo4j/conf/neo4j.conf
# server.memory.heap.initial_size=512m
# server.memory.heap.max_size=1g
Qdrant startup failure
# Qdrant 로그 확인
tail -50 /home/kopens/plantpulse-ai/db/qdrant/logs/*.log
# 초기화 상태 확인
cat /home/kopens/plantpulse-ai/db/.qdrant-initialized
# 헬스 체크
curl http://127.0.0.1:6333/healthz
AI Chat Web Problems
Symptom: Cannot reach the web page
# 웹 서비스 상태 확인
nc -z 127.0.0.1 80 && echo "OK" || echo "DOWN"
# 웹 서비스 로그 확인
tail -50 /home/kopens/plantpulse-ai/web/logs/system.log
# WAR 파일 존재 확인
ls -la /home/kopens/plantpulse-ai/web/app/*.war
Symptom: No chat response (SSE streaming failure)
Chat is handled by the AI Chat Web built-in agent (AgentOrchestrator), which calls LiteLLM (gpt-4o).
Check in this order:
# 1. LiteLLM 프록시가 실행 중인지 확인
nc -z 127.0.0.1 4000 && echo "OK" || echo "DOWN"
# 2. LiteLLM LLM 응답 확인
curl http://127.0.0.1:4000/health
# 3. 통합 MCP(플랫폼 server-web) 접근 및 api_key 확인
# web/config/application.properties의 mcp.api.url / mcp.api.token 확인
# 4. AI Chat Web 로그 확인
tail -50 /home/kopens/plantpulse-ai/web/logs/system.log
Symptom: Charts/diagrams do not render
- Check the browser console (F12) for JavaScript errors
- ECharts/Mermaid library failed to load → check CDN access in the Network tab
- If external CDNs are blocked in an on-premise environment → configure the system to use local libraries
Integrated MCP Problems
MCP and ontology tools are provided by the PlantPulse platform server-web integrated MCP (/api/v5/mcp). The former standalone services (mcp-server:50000, ontology:8888) have been archived.
Symptom: MCP tool call fails
# 통합 MCP 도구 목록 조회 (api_key 필요)
curl -X POST <mcp.api.url>/api/v5/mcp \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <mcp.api.token>" \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
# AI Chat Web 로그에서 MCP 호출 오류 확인
tail -50 /home/kopens/plantpulse-ai/web/logs/system.log | grep -i mcp
Main causes:
| Error | Cause | Resolution |
|---|---|---|
| Connection refused | Platform server-web unreachable | Check mcp.api.url and the network |
| 401 Unauthorized | api_key authentication failure | Check mcp.api.token, reissue the token |
| Tool not listed | tools.enabled disabled | Check mcp.tools.enabled=true |
| Timeout | Large data query | Use pagination |
TimeSeries-Insight Problems
Symptom: Anomaly detection / prediction fails
# TimeSeries 헬스 체크
curl http://127.0.0.1:8970/health
# 로그 확인
tail -50 /home/kopens/plantpulse-ai/timeseries/logs/system.log
Main causes:
| Error | Cause | Resolution |
|---|---|---|
| No data found | No sensor data | Increase lookback_minutes |
| Model loading failed | AI model failed to load | Check GPU memory, switch to CPU mode |
| Cassandra timeout | Time series DB response delay | Check Cassandra status, check TS_CASS_HOST |
| CUDA out of memory | Insufficient GPU memory | Reduce the batch size or use CPU mode |
When GPU memory is insufficient
# GPU 상태 확인
nvidia-smi
# GPU 메모리 정리 (프로세스 확인 후)
nvidia-smi --query-compute-apps=pid --format=csv,noheader | xargs kill -9
RAG (LightRAG) Problems
The RAG engine is the LightRAG 1.5.4 server. Document management is performed in the WebUI (/webui).
Symptom: No document search results
# LightRAG 헬스 체크
curl http://127.0.0.1:7114/health
# Qdrant(벡터 저장소) 확인
curl http://127.0.0.1:6333/collections
# LightRAG 로그 확인
docker logs --tail 50 lightrag
Main causes:
| Error | Cause | Resolution |
|---|---|---|
| Empty results | No documents registered | Upload documents in the LightRAG WebUI (/webui) |
| Embedding timeout | Embedding server response delay | Check LiteLLM/embedding status |
| Qdrant connection error | Vector DB unreachable | Check the Qdrant port (6333) |
| Reranker error | Reranker model failure | Temporarily work around it by disabling the reranker setting |
Symptom: Document indexing fails
# LightRAG 인덱싱/파싱 로그 확인
docker logs --tail 100 lightrag | grep -i "parse\|docling\|error"
Ontology (Knowledge Graph) Problems
info
The ontology is no longer a standalone service; it has been absorbed into the platform integrated MCP (/api/v5/mcp). Diagnose graph tool errors and synchronization issues using the Integrated MCP Problems procedure, and check the platform server-web side for the DB↔Neo4j synchronization logic.
# Neo4j 연결 확인 (LightRAG/온톨로지 그래프 저장소)
curl http://127.0.0.1:7474
# 그래프 통계 조회는 통합 MCP 도구 ontology_get_stats 호출
Main causes:
- Neo4j not started → start Neo4j first
- Integrated MCP unreachable → check
mcp.api.url/mcp.api.token
LiteLLM (LLM Proxy) Problems
Symptom: No AI response
# LiteLLM 헬스 체크
curl http://127.0.0.1:4000/health
# 모델 목록 확인
curl -H "Authorization: Bearer 설치-시-변경" http://127.0.0.1:4000/v1/models
# 로그 확인
tail -50 /home/kopens/plantpulse-ai/lib/litellm/logs/*.log
Main causes:
| Error | Cause | Resolution |
|---|---|---|
| Model not found | Model configuration error | Check template/proxy.yaml |
| Connection to vLLM failed | Inference server unreachable | Check the network to the inference server (192.168.0.240) |
| Rate limit exceeded | Too many requests | Wait and retry, or adjust num_retries |
| API key invalid | Authentication key mismatch | Check PI_INFERENCE_SERVER_API_KEY |
Insufficient Disk Space
# 디스크 사용량 확인
df -h
# 큰 파일 찾기
du -sh /home/kopens/plantpulse-ai/*/logs/ | sort -rh
# 로그 및 임시 파일 정리
cd /home/kopens/plantpulse-ai/bin
./clean.sh
# Cassandra 데이터가 큰 경우
du -sh /data1/pp-data/
Insufficient Memory
# 메모리 사용량 확인
free -h
# 프로세스별 메모리 확인
ps aux --sort=-%mem | head -20
Network Problems
Verifying communication between internal services
# 전체 포트 스캔
for port in 4000 5432 6333 6379 7114 7687 8970 9001 9042 80; do
nc -z 127.0.0.1 $port 2>/dev/null && echo "[OK] $port" || echo "[FAIL] $port"
done
Verifying the connection to the PlantPulse IIoT server
# IIoT 서버 접근 확인
curl -k https://100.68.69.41:7443/health
# SSL 인증서 확인
openssl s_client -connect 100.68.69.41:7443 -brief
Emergency Recovery
Restarting the entire system
cd /home/kopens/plantpulse-ai/bin
./stop.sh
sleep 10
./start.sh
Restarting only a specific module
# 예: LightRAG(RAG)만 재시작
docker compose restart lightrag
Resetting while preserving data
# 1. 전체 종료
./stop.sh
# 2. 로그/캐시만 정리 (데이터는 보존)
./clean.sh
# 3. 재설치 (의존성 재설치)
./setup.sh
# 4. 재시작
./start.sh
warning
The /data1/pp-data/ directory holds core data. Deleting this directory removes all LightRAG documents/indexes and more. Always back it up before deleting.