Diagnostics / Inspection
A collection of diagnostic scripts to use when a fault or abnormal behavior is suspected.
The procedures below are for native boxes (bin/ps.sh, bin/log-viewer.sh). For container mode:
bash /opt/kopens/install/bin/status.sh # 한 줄 상태
bash /opt/kopens/install/bin/health.sh # 종합 헬스 (exit 0/1)
bash /opt/kopens/install/bin/logs.sh -f tomcat # 로그 follow
sudo bash /opt/kopens/install/bin/doctor.sh # 진단 일괄 tarball (support escalation)
Details: Container Mode Operations Guide + /opt/kopens/install/RUNBOOK.md.
1. Step-by-step diagnostic procedure
# 1. 누가 죽었나?
$PE_HOME/bin/ps.sh
# 2. 어디가 시끄러운가?
$PE_HOME/bin/log-viewer.sh
# (실시간 — Ctrl+C 로 빠져나옴)
# 3. Cassandra 의심
$PE_HOME/bin/node-info.sh
# 4. Disk 의심
df -h /data1
du -sh $PE_HOME/*/log/* /data1/* 2>/dev/null | sort -h | tail -20
# 5. 다 안 맞으면 — 안전하게 단계적 재시작
$PE_HOME/bin/restart.sh # 1차: Tomcat 만
# 그래도 이상 시
$PE_HOME/bin/stop.sh
$PE_HOME/bin/start.sh # 2차: 전체
# 그래도 이상 시
sudo $PE_HOME/bin/reboot.sh # 3차: OS 재부팅 (최후의 수단)
2. Caution — do not call log-viewer.sh from a non-interactive shell
log-viewer.sh is an infinite tail -f. If called in a non-interactive environment such as CI or cron, the ssh session will never end. Use it only from an interactive shell.
3. log-viewer.sh — consolidated log tail
$PE_HOME/bin/log-viewer.sh
Seven logs on a single screen in real time:
- Tomcat (
server/logs/catalina.out) - timeseries-engine
- Cassandra (
db/logs/system.log) - HiveMQ (
mqtt/log/hivemq.log) - Node-RED (
node/log/node-red.log) - Redis cache
- Others
Exit with Ctrl+C. This is the first tool to run when a fault occurs.
4. ps.sh
$PE_HOME/bin/ps.sh
Lists the live Java processes matching the plantpulse keyword. In a healthy state, all of the following should appear:
| Process | Meaning | PID environment variable |
|---|---|---|
apache.cassandra.service.CassandraDaemon | Cassandra | cassandra.pid |
hivemq.jar | MQTT | (none) |
plantpulse.timeseries.engine.Main | Time series engine | (none) |
org.apache.catalina.startup.Bootstrap | Tomcat | CATALINA_PID |
node-red (Node.js) | Node-RED | (none) |
If any is missing, that component is down — bring it back up with start.sh or with the bin/start.sh of the corresponding sub-component.
5. node-info.sh — Cassandra status
$PE_HOME/bin/node-info.sh
Representative output (nodetool info):
ID : 8a4d...
Gossip active : true
Native Transport active: true
Load : 1.21 GiB
Generation No : 1778176430
Uptime (seconds) : 1234
Heap Memory (MB) : 824.10 / 2048.00
| Check point | Meaning |
|---|---|
Native Transport active : true | Client port 9042 is listening |
Heap > 80% | OOM imminent — clean up data or increase heap |
Load exceeds 80% of disk | Clean up sstables (node-cleanup.sh) or expand disk |
6. node-cql.sh — interactive cqlsh
$PE_HOME/bin/node-cql.sh
cqlsh -u cassandra -p ... runs automatically and connects to the pe keyspace. Intended for manual data inspection — direct INSERT/UPDATE during operation is not recommended (cache not refreshed, replicas out of sync, etc.).
Common query examples:
USE pe;
SELECT count(*) FROM app_tag;
SELECT opc_id, opc_type FROM app_opc;
DESCRIBE TABLE app_tag;
7. network-speed-test.sh
$PE_HOME/bin/network-speed-test.sh
Fetches speedtest.py from product.kopens.io and runs it to measure external network speed. If the line is slow, an upgrade will also take longer — use this as a pre-upgrade diagnostic.
8. Common pitfalls — summary
| Symptom | Cause / resolution |
|---|---|
OPC shows 0/N connections after restart.sh | A rare startup race. Call restart.sh once more |
ssh session drops abruptly after clean.sh | /tmp/* also removes the ssh socket — reconnect with another session |
Download fails during upgrade.sh | Check the external network / product.kopens.io. network-speed-test.sh |
start.sh hangs for too long ([4] DB START) | Cassandra commitlog recovery — typically 30s+ of waiting. If it still does not finish, check db/logs/system.log |
| Only Tomcat keeps dying | Possible OOM — check OutOfMemoryError / heapdump directory in server/logs/catalina.out |
| Node-RED disappears after Deploy | userDir corruption. Designed to recover from the master copy in $PE_HOME/node/conf — node/bin/start.sh syncs automatically |
| Gateway unreachable from outside | Check the firewall (firewall-cmd --list-all or iptables -L), SELinux policy, and router NAT |
9. Learn more
- When disk space runs low: Cleanup / Purge
- Step-by-step fault diagnosis scenarios: Operations Scenarios
- Monitoring (REST metrics): Monitoring (Operator/REST)