Diagnosis and Inspection
A collection of diagnostic scripts for use when failures or abnormal behavior are suspected.
The procedures below are for native boxes (bin/ps.sh, bin/log-viewer.sh). For container mode:
bash /opt/kopens/install/bin/status.sh # 한 줄 상태
bash /opt/kopens/install/bin/health.sh # 종합 헬스 (exit 0/1)
bash /opt/kopens/install/bin/logs.sh -f tomcat # 로그 follow
sudo bash /opt/kopens/install/bin/doctor.sh # 진단 일괄 tarball (support escalation)
Details: Container Mode Operations.
1. Step-by-Step Diagnosis Procedure
# 1. 누가 죽었나?
$PE_HOME/bin/ps.sh
# 2. 어디가 시끄러운가?
$PE_HOME/bin/log-viewer.sh
# (실시간 — Ctrl+C 로 빠져나옴)
# 3. Cassandra 의심
$PE_HOME/bin/node-info.sh
# 4. Disk 의심
df -h /data1
du -sh $PE_HOME/*/log/* /data1/* 2>/dev/null | sort -h | tail -20
# 5. 다 안 맞으면 — 안전하게 단계적 재시작
$PE_HOME/bin/restart-app.sh # 1차: 앱(Tomcat)만 — 30~90초
# 그래도 이상 시
$PE_HOME/bin/restart.sh # 2차: 전체 스택 (stop + start 를 한 프로세스로)
# 그래도 이상 시
sudo $PE_HOME/bin/reboot.sh # 3차: OS 재부팅 (최후의 수단)
2. Caution — Do Not Call log-viewer.sh in Non-Interactive Shells
log-viewer.sh is an infinite tail -f. If called in a non-interactive environment like CI/cron, the SSH session will never end. Use it only in an interactive shell.
3. log-viewer.sh — Unified Log Tail
$PE_HOME/bin/log-viewer.sh
View 7 logs in real-time on one screen:
- Tomcat (
server/logs/catalina.out) - timeseries-engine
- Cassandra (
db/logs/system.log) - HiveMQ (
mqtt/log/hivemq.log) - Node-RED (
node/log/node-red.log) - Redis cache
- Others
Exit with Ctrl+C. This is the tool to run first when a fault occurs.
4. ps.sh
$PE_HOME/bin/ps.sh
List of live Java processes filtered by the plantpulse keyword. In a normal state, all of the following must be visible:
| Process | Meaning | PID Environment Variable |
|---|---|---|
apache.cassandra.service.CassandraDaemon | Cassandra | cassandra.pid |
hivemq.jar | MQTT | (None) |
plantpulse.timeseries.engine.Main | Time series engine | (None) |
org.apache.catalina.startup.Bootstrap | Tomcat | CATALINA_PID |
node-red (Node.js) | Node-RED | (None) |
If any are missing, that component is dead — revive it with start.sh or the bin/start.sh of the corresponding sub-component.
5. node-info.sh — Cassandra Status
$PE_HOME/bin/node-info.sh
Representative output (nodetool info):
ID : 8a4d...
Gossip active : true
Native Transport active: true
Load : 1.21 GiB
Generation No : 1778176430
Uptime (seconds) : 1234
Heap Memory (MB) : 824.10 / 2048.00
| Inspection Point | Meaning |
|---|---|
Native Transport active : true | 9042 client port is listening |
Heap > 80% | OOM imminent — clean up data / increase heap |
Load exceeds 80% of disk | sstable cleanup (node-cleanup.sh) or disk expansion |
6. node-cql.sh — Interactive cqlsh
$PE_HOME/bin/node-cql.sh
cqlsh -u cassandra -p ... runs automatically and connects to the keyspace pe. Use for manual data inspection — direct INSERT/UPDATE during operation is not recommended (cache not reflected, replica inconsistency, etc.).
Frequently used query examples:
USE pe;
SELECT count(*) FROM app_tag;
SELECT opc_id, opc_type FROM app_opc;
DESCRIBE TABLE app_tag;
7. network-speed-test.sh
$PE_HOME/bin/network-speed-test.sh
Receives speedtest.py from product.kopens.io and runs it to measure external network speed. If the line is slow, upgrades also take longer — this is for pre-upgrade diagnosis.
8. Common Pitfalls — Summary
| Symptom | Cause / Solution |
|---|---|
restart.sh after OPC shows 0/N connection | Rare startup race. Call restart.sh once more |
SSH session suddenly drops after clean.sh | /tmp/* kills even the SSH socket — reconnect from another session |
Download failure during upgrade.sh | Check external network / product.kopens.io. network-speed-test.sh |
start.sh stuck for too long ([4] DB START) | Cassandra commitlog recovery — usually waits 30s+. If it still doesn't finish, check db/logs/system.log |
| Tomcat keeps dying | Possible OOM — check server/logs/catalina.out's OutOfMemoryError / heapdump directory |
| Node-RED disappears after Deploy | Corrupted userDir. Designed to be recovered from the master copy at $PE_HOME/node/conf — node/bin/start.sh syncs automatically |
| Cannot access gateway from outside | Check firewall (firewall-cmd --list-all or iptables -L) / SELinux policy / router NAT |
9. Learn More
- When disk is full: Cleanup and Purge
- Step-by-step failure diagnosis scenarios: Operational Scenarios
- Monitoring (REST metrics): Monitoring (Operator/REST)