Skip to main content

Operational Scenarios (Runbook)

A runbook that lays out the complete operating procedures for common situations so you can follow them as-is. Just copy and run one section from top to bottom.

Runbook for boxes in container mode

If systemctl is-active plantpulse-edge.service is active, the box is in container mode — refer to /opt/kopens/install/RUNBOOK.md inside the box (or the Container Mode Operations Guide) instead of this page. RUNBOOK bundles a 5-scenario failure matrix (A–E), the OTA workflow, the password table, and a command cheat sheet into a single document.

This page covers native box scenarios.

1. Deploying a New Version — The Safest Procedure​

# 0. 작업 공지 (관제 / Slack)
echo "[$(date +%F\ %T)] 게이트웨이 업그레이드 시작 — 약 15분 소요"

# 1. 사전 백업 (자동 backup 도 있지만 수동으로 한 번 더)
$PE_HOME/bin/backup.sh

# 2. 외부 NAS / S3 로 즉시 미러
LATEST=$(ls -t /data1/pp-backup | head -1)
rsync -av /data1/pp-backup/$LATEST nas:/backup/edge/

# 3. 업그레이드 (자동으로 backup → install → start)
$PE_HOME/bin/upgrade.sh

# 4. 검증
$PE_HOME/bin/ps.sh
curl -s http://127.0.0.1/api/v1/edge | jq                       # 버전 확인
curl -s http://127.0.0.1/ui/opcua/tree | jq '.data.tree | length'  # 사이트/태그 정상
$PE_HOME/bin/log-viewer.sh                                       # 30초 정도 봐서 SEVERE 없는지
# (Ctrl+C)

# 5. 외부 SCADA / Sparkplug 수신측 정상 도착 여부 확인
echo "[$(date +%F\ %T)] 업그레이드 완료"

Details: Upgrade (upgrade.sh), Backup (backup.sh)

2. Quick Restart When Things Feel Stuck​

$PE_HOME/bin/restart.sh
# 6초 후
curl -s http://127.0.0.1/api/v1/edge | jq

Code and configuration changes are applied at the same time. Cassandra/HiveMQ/Node-RED stay up.

Details: Restart (restart.sh)

3. Fault Diagnosis — Step by Step​

# 1. 누가 죽었나?
$PE_HOME/bin/ps.sh

# 2. 어디가 시끄러운가?
$PE_HOME/bin/log-viewer.sh
# (실시간 — Ctrl+C 로 빠져나옴)

# 3. Cassandra 의심
$PE_HOME/bin/node-info.sh

# 4. Disk 의심
df -h /data1
du -sh $PE_HOME/*/log/* /data1/* 2>/dev/null | sort -h | tail -20

# 5. 다 안 맞으면 — 안전하게 단계적 재시작
$PE_HOME/bin/restart.sh           # 1차: Tomcat 만
# 그래도 이상 시
$PE_HOME/bin/stop.sh
$PE_HOME/bin/start.sh             # 2차: 전체
# 그래도 이상 시
sudo $PE_HOME/bin/reboot.sh       # 3차: OS 재부팅 (최후의 수단)

Details: Diagnosis / Inspection

4. Responding to Low Disk Space​

# 1. 무엇이 차지 중?
df -h
du -sh $PE_HOME/*/log/* /data1/* 2>/dev/null | sort -h | tail -20

# 2. 로그 / 임시 파일 정리
$PE_HOME/bin/clean.sh

# 3. Cassandra 디스크 회수
$PE_HOME/bin/node-cleanup.sh

# 4. 오래된 백업 정리 (30 일 이상)
find /data1/pp-backup -maxdepth 1 -type d -mtime +30 -exec rm -rf {} +

# 5. 결과 확인
df -h

Details: Cleanup

5. Automating Scheduled Backups (cron example)​

# /etc/cron.d/plantpulse-backup
# 매일 새벽 1시에 백업, 30일 이상은 자동 삭제
0 1 * * * root /opt/kopens/plantpulse-edge/bin/backup.sh >>/var/log/pp-backup.log 2>&1
30 1 * * * root find /data1/pp-backup -maxdepth 1 -type d -mtime +30 -exec rm -rf {} +

After applying the cron entry, check the next day at 01:00 that a new /data1/pp-backup/<date>/ has been created.

Details: Backup (backup.sh)

6. OS Security Patches (Quarterly)​

# 1. 사전 백업
$PE_HOME/bin/backup.sh

# 2. 게이트웨이 정지 (커널 업데이트 가능성 대비)
$PE_HOME/bin/stop.sh

# 3. OS 업데이트
$PE_HOME/bin/firmware.sh

# 4. 커널 업데이트 시 재부팅 필요
needs-restarting -r
# 결과가 "Reboot is required" 이면
$PE_HOME/bin/reboot.sh
# 부팅 후 ssh 다시 들어가서
$PE_HOME/bin/start.sh

# 5. 검증
$PE_HOME/bin/ps.sh

Details: Upgrade (upgrade.sh / firmware.sh)

7. After a Power Outage / OS Reboot​

Services may start automatically via systemd registration, but verify manually:

$PE_HOME/bin/ps.sh
# 출력이 비어있거나 일부만 살아있다면
$PE_HOME/bin/start.sh
# 또는 백그라운드로
$PE_HOME/bin/start-daemon.sh

Details: Start (start.sh)

8. Restoring from Backup (Disaster Recovery)​

Restoring overwrites data — do it with someone experienced

The following is the big picture of a full restore. Do not copy and run it as-is in a production environment; carry it out together with your operations team.

Detailed procedure: Backup (backup.sh) — Restoring from Backup

9. Learn More​

  • For details on each script, see the left-hand menu: Start / Stop / Restart / Backup / Upgrade / Diagnosis / Cleanup
  • All scripts at a glance: At a Glance