System Startup and Shutdown

Overview

PlantPulse operates as a Docker Compose stack. Startup / shutdown / restart are all managed through a single set of common verbs in /opt/kopens/plantpulse-platform-docker/bin/, and the inter-container dependency order is applied automatically by compose's depends_on, so operators don't need to worry about ordering by hand.

Exit code 0 means something different here

For up.sh and restart.sh, 0 doesn't mean "the command succeeded" — it means "it's now usable." During the 2026-08-14 credential rotation, an incident occurred where the next step proceeded on seeing just "started," and the process stalled on a broker that hadn't come up yet. Since then, both verbs were changed to wait until ready.

status.sh is different — 0 = normal / 2 = abnormal.

Recommended Operation Commands (Summary)

cd /opt/kopens/plantpulse-platform-docker/bin

./up.sh                # 기동 (멱등) — 준비될 때까지 대기. 0 = 준비 완료
./down.sh              # 안전 종료 (의존 순서의 역순, 상태 보존)
./restart.sh           # graceful drain → 정지 → 기동 → 준비 대기
./status.sh            # 서비스 / health / 볼륨 요약. 0 = 정상 / 2 = 비정상
./ops-check.sh         # 헬스 + 최근 critical log
./logs.sh [서비스]     # 로그 보기 — 1회 출력, -f 로 따라가기 (인자 없으면 전체)
./stack-verify-boot.sh # 스택 전체 준비 판정
./doctor.sh            # 진단 tarball (시스템 변경 없음)

All verbs support --help. The old names (stack-run.sh · stack-stop.sh · stack-bash.sh · stack-update.sh · stack-remove.sh) still work as-is.

Environment Variable	Default	Meaning
`PP_READY_TIMEOUT`	`900`	Upper bound for readiness wait (seconds)
`PP_READY_INTERVAL`	`15`	Check interval (seconds)
`PP_WAIT=0`	—	Don't wait. In this case `0` does not mean readiness has completed

Startup Order (Automatic)

Between Containers

compose's depends_on maintains ordering with service_healthy conditions — it waits for "usable," not "process has started." This order isn't arbitrary; it's carried over directly from the application orchestrator's boot sequence.

The last four (warehouse · opcua · aasx · ha) have no ordering relative to each other.

Inside the Datalake Container

Infrastructure components boot in order again inside a single plantpulse-datalake container.

Order	Area	Key Components	Representative Ports
1	Storage	Cassandra, PostgreSQL, Valkey, MinIO	9042 / 5432 / 6379 / 9000
2	Analytics	Spark, Hive, Kyuubi, Gravitino	7077 / 9083 / 10000 / 19001
3	Time Series	TSE Engine, UI	7800 / 3000
4	Messaging	Kafka, MQTT	9092 / 1883
5	Workflow	Temporal, Kestra	7233 / 8233 / 8380
6	Processing	CEP, Data Gateway, SQL, Monitor	7400 / 5500 / 4000 / 4950

Time Required

Process startup itself takes 3–5 minutes (JVM warm-up), but a clean install takes 15–18 minutes for all components to stabilize, due to Cassandra schema migration and stabilization.

Measured (2026-08-31, 32 vCPU / 128GiB): datalake 217 seconds, web server 316 seconds. A restart with existing data is much faster.

Shutdown Order

cd /opt/kopens/plantpulse-platform-docker/bin
./down.sh

You don't need to enforce the order by hand. compose stops in reverse dependency order, so the data-writing side (apps) stops first and the datalake stops last. Each container has a generous stop grace period (stop_grace_period) — 180 seconds for the datalake — giving Cassandra time to flush its Memtables.

Volumes (pp-data · pp-temp · pp-backup · pp-security · pp-proxy-certs) are always preserved regardless of stopping or removal.

Don't use docker kill or a forced host shutdown

Unflushed Memtable data in Cassandra can be lost. Please always use ./down.sh. If a container is already unresponsive, follow the emergency procedures.

Restart

Full Restart

cd /opt/kopens/plantpulse-platform-docker/bin
./restart.sh

Request a graceful drain from the datalake (if running)
Stop the entire stack in reverse dependency order
Boot back up in depends_on order
Wait until ready — exit code 0 means it's usable

Use this for bin/env.sh changes, locale/timezone changes, resolving transient issues, and periodic restarts.

Restarting a Single App

Since each of the six apps runs in its own container, restarting just that container leaves the other apps and infrastructure running. This is one of the reasons the containers were split.

cd /opt/kopens/plantpulse-platform-docker
docker compose -f compose/docker-compose.yml restart plantpulse-server-web

docker compose restart does not honor dependency order

If you need to bring multiple apps back up together, it's safer to restart the whole stack with ./restart.sh.

Restarting Only Infrastructure Components

./shell.sh                                                              # 데이터레이크 진입
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd restart storage
exit

The available scripts are pd restart storage · pd restart analytics · pd restart messaging · pd restart timeseries · pd restart workflow · pd restart cep · pd restart data-gateway · pd restart sql · restart-monitor.sh. For details, see the Startup Guide.

Status Check

One-Line Summary

./status.sh            # 0 = 정상 / 2 = 비정상

status.sh treats five things as abnormal.

Condition	Description
Can't read the service list	compose parsing failed or docker inaccessible
No container for a service declared by compose	Not started
A persistent container is not `running`	One-shots (`plantpulse-certs`) are excluded
A persistent container's health is `unhealthy`	`starting` / `none` are treated as pending
Health API is not OK	probe from inside the datalake

Volumes and conf are reported only, and not factored into the exit code — because in a 2-node split installation (PP_TIER=APP), it's normal for some to be missing.

A one-shot's Exited (0) is success

plantpulse-certs bakes the certificate and finishes on its own. compose has explicitly disabled the healthcheck for this container (healthcheck: disable), so the health column is blank — because attaching a health probe to a one-shot would make "a container operating normally" look "unhealthy forever," forcing everyone to remember this single exception. status.sh and ops-check.sh judge only this container by exit code — please view it the same way when checking manually.

Checking Containers

docker ps                       # 여덟 개가 (healthy), certs 는 Exited (0)
docker stats --no-stream        # 컨테이너별 CPU / 메모리

Healthcheck API

# 호스트 / 외부에서 — 4950 이 publish 되어 있습니다
curl -kfsS https://<server-ip>:4950/api/health | jq

# 컨테이너 안에서 — 어떤 구성에서도 동작합니다
docker exec plantpulse-datalake curl -kfsS https://127.0.0.1:4950/api/health | jq

The value of "status" can only be OK · WARN · FAIL, and OK/WARN are the normal range.

Port 4949 serves the same API too

The console and health API are served on both ports — 4950 (HTTPS) and 4949 (plain HTTP). Same console, same API, just a different scheme. 4949 no longer redirects to 4950.

4949 is plaintext — login passwords and session cookies flow unencrypted. Use 4950 on untrusted networks. 4949 exists as an option for boxes where self-signed certificate warnings actually block operators.

Checking Logs

./logs.sh                              # 여덟 컨테이너를 시간순으로 한 화면에
./logs.sh plantpulse-server-web -n 200 # 특정 컨테이너
./logs.sh cassandra                    # 데이터레이크 안 컴포넌트 로그 파일
./logs.sh --list                       # 볼 수 있는 대상 전체
./logs.sh -f plantpulse-proxy          # 계속 따라가기 (Ctrl-C 로 종료)

By default, this prints the last N lines (200 by default) and exits. Add -f to follow.

Auto-Start Configuration

No separate configuration is needed. Persistent containers are declared with restart: always in compose, so when the Docker daemon comes up, they come up with it even after a host reboot. One-shots (plantpulse-certs) are restart: "no", so they don't run again — running again would bake a new certificate.

The only thing to check is the Docker daemon's own auto-start.

systemctl is-enabled docker     # enabled 여야 합니다
sudo systemctl enable docker    # 아니라면

The proxy may briefly show red right after a reboot

A host reboot is the one path where depends_on ordering doesn't apply — since everything comes up all at once, the proxy's healthcheck may briefly fail until the web server comes up (measured up to 316 seconds). Since the retry budget is set to 360 seconds and docker doesn't restart a container just because it's unhealthy, this is a transient state that resolves on its own.

Emergency Procedures

Web Console Not Responding

cd /opt/kopens/plantpulse-platform-docker/bin

# 1. 어느 컨테이너가 문제인가
./status.sh

# 2. 프록시와 웹 서버 로그
./logs.sh plantpulse-proxy -n 100
./logs.sh plantpulse-server-web -n 200

# 3. 웹 서버만 재시작 (인프라는 유지)
cd /opt/kopens/plantpulse-platform-docker
docker compose -f compose/docker-compose.yml restart plantpulse-server-web

# 4. 회복되지 않으면 진단 번들
bin/doctor.sh

If the proxy is green but the screen shows 502

The proxy's healthcheck only checks itself + upstream reachability (backend health is judged by the web server's own healthcheck). If the proxy is green but the screen doesn't render, check plantpulse-server-web.

OOM (Out Of Memory) Occurs

# 1. 어느 컨테이너가 OOM 인가
docker inspect plantpulse-server-web --format '{{.State.OOMKilled}} {{.State.ExitCode}} {{.RestartCount}}'

# 2. 호스트 커널 로그
dmesg | grep -i "out of memory\|oom"

# 3. 컨테이너별 사용량
docker stats --no-stream

Each container has its own separate limit — the datalake uses DOCKER_DATALAKE_MEMORY (default 80G), apps use DOCKER_SERVER_MEMORY · DOCKER_BATCH_MEMORY, etc. (Environment Variables). After adjusting values, apply with ./restart.sh.

One app's OOM does not bring down the whole system

Giving each app its own mem_limit is the first purpose of splitting into separate containers. Even if one app hits its limit, infrastructure and other apps stay alive.

DB Connection Failure

All databases live inside plantpulse-datalake.

cd /opt/kopens/plantpulse-platform-docker/bin
./shell.sh                                                            # 데이터레이크 진입

/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node psql       # PostgreSQL
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node cql        # Cassandra
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node status     # Cassandra 링 상태

To bring back only the infrastructure, use pd restart storage inside the container, and if that doesn't work, use ./restart.sh from the host.

If Data Corruption Is Suspected

cd /opt/kopens/plantpulse-platform-docker/bin

./down.sh                                  # 즉시 정지 (추가 손상 방지)
ls -lh /data1/pp-backup/docker-volume/     # 최신 백업 확인
./doctor.sh                                # 진단 번들 (데이터를 임의로 수정하지 마세요)

Please send the generated tarball to webmaster@kopens.com.

Operation Checklist

Daily Inspection

Inspection Item	Command / Check Method	Normal Criteria
Overall service status	`./status.sh`	exit code 0
Operational health	`./ops-check.sh`	no critical logs
Health API	`curl -kfsS https://<server-ip>:4950/api/health`	`status` is `OK` or `WARN`
Containers	`docker ps`	eight `(healthy)`, certs is `Exited (0)`
Disk usage	`df -h /data1`	usage below 80%
Container resources	`docker stats --no-stream`	headroom against limits
Cassandra status	`pd node status` inside the container	all nodes UN (Up/Normal)

Weekly Inspection

Inspection Item	Command / Check Method	Normal Criteria
Cassandra Compaction	`pd node compactionstats` inside the container	no excessive pending tasks
Cassandra tables	`pd node table-stats` inside the container	no abnormal growth
Kafka topics / lag	`pd node topic` inside the container	delayed message count within normal range
Backup check	`ls -lh /data1/pp-backup/docker-volume/`	valid backup exists
Docker usage	`docker system df`	no accumulation of unused images
Log volume	`du -sh /opt/kopens/plantpulse-platform-docker/logs`	no abnormal growth
Security updates	check OS package updates	no known vulnerabilities

Binary (Native) Environment

The current release does not support binary (native) installation

The content below is kept for existing systems built via binary installation. Since the current shipped release is solely the Docker Compose stack above, do not use the procedures below (systemd services, Windows services, individual process startup) in a container environment.

In a binary environment, the operation scripts are located at /opt/kopens/plantpulse-platform/plantpulse-startup/ — start-daemon.sh · stop.sh · restart.sh · status.sh · kill.sh · log-viewer.sh. For details, see Binary Installation.

Linux Startup (systemd)

If systemd services are registered, you can start with the following commands.

# 1. 스토리지 서비스 시작
sudo systemctl start plantpulse-postgresql
sudo systemctl start plantpulse-redis
sudo systemctl start plantpulse-cassandra

# Cassandra가 완전히 시작될 때까지 대기 (약 30~60초)
until cqlsh 127.0.0.1 -e "DESCRIBE KEYSPACES" > /dev/null 2>&1; do
    echo "Cassandra 시작 대기 중..."
    sleep 5
done
echo "Cassandra 시작 완료"

# 2. 메시징 서비스 시작
sudo systemctl start plantpulse-kafka
sudo systemctl start plantpulse-mqtt

# 3. 엔진 서비스 시작
sudo systemctl start plantpulse-timeseries
sudo systemctl start plantpulse-cep

# 4. 웹서버 시작
sudo systemctl start plantpulse-server

# 5. 에이전트 시작
sudo systemctl start plantpulse-agent

Linux Manual Startup

If not using systemd, you can run each module's startup script directly.

# 스토리지
/opt/kopens/plantpulse-platform/plantpulse-storage/db/postgres/bin/pg_ctl start -D /opt/kopens/plantpulse-platform/plantpulse-storage/db/postgres/data
/opt/kopens/plantpulse-platform/plantpulse-storage/db/valkey/bin/valkey-server /opt/kopens/plantpulse-platform/plantpulse-storage/db/valkey/conf/valkey.conf &
/opt/kopens/plantpulse-platform/plantpulse-storage/db/cassandra/bin/cassandra

# 메시징
/opt/kopens/plantpulse-platform/plantpulse-messaging/kafka/bin/kafka-server-start.sh -daemon /opt/kopens/plantpulse-platform/plantpulse-messaging/kafka/config/server.properties
/opt/kopens/plantpulse-platform/plantpulse-messaging/mqtt/bin/startup.sh

# 엔진
/opt/kopens/plantpulse-platform/plantpulse-timeseries/bin/startup.sh
/opt/kopens/plantpulse-platform/plantpulse-cep/bin/startup.sh

# 웹서버
/opt/kopens/plantpulse-platform/plantpulse-server/bin/startup.sh

# 에이전트
/opt/kopens/plantpulse-platform/plantpulse-plugin/opc-ua/bin/startup.sh

Startup Script (with Dependency Checks)

Below is an example script that starts sequentially while checking dependencies.

#!/bin/bash
# plantpulse-start-all.sh - 전체 플랫폼 시작 스크립트

KOPENS_HOME="/opt/kopens"
LOG_FILE="/var/log/plantpulse/startup.log"

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}

wait_for_port() {
    local host=$1 port=$2 timeout=${3:-60}
    local elapsed=0
    while ! nc -z "$host" "$port" 2>/dev/null; do
        if [ $elapsed -ge $timeout ]; then
            log "ERROR: $host:$port 연결 타임아웃 (${timeout}초)"
            return 1
        fi
        sleep 2
        elapsed=$((elapsed + 2))
    done
    log "OK: $host:$port 연결 확인"
    return 0
}

# 1. PostgreSQL
log "PostgreSQL 시작 중..."
sudo systemctl start plantpulse-postgresql
wait_for_port 127.0.0.1 5432 30 || exit 1

# 2. Redis
log "Redis 시작 중..."
sudo systemctl start plantpulse-redis
wait_for_port 127.0.0.1 6379 15 || exit 1

# 3. Cassandra
log "Cassandra 시작 중..."
sudo systemctl start plantpulse-cassandra
wait_for_port 127.0.0.1 9042 120 || exit 1

# 4. Kafka & MQTT
log "Kafka 시작 중..."
sudo systemctl start plantpulse-kafka
wait_for_port 127.0.0.1 9092 30 || exit 1

log "MQTT 시작 중..."
sudo systemctl start plantpulse-mqtt
wait_for_port 127.0.0.1 1883 15 || exit 1

# 5. TSE
log "시계열 엔진 시작 중..."
sudo systemctl start plantpulse-timeseries
wait_for_port 127.0.0.1 7800 30 || exit 1

# 6. CEP
log "CEP 엔진 시작 중..."
sudo systemctl start plantpulse-cep
wait_for_port 127.0.0.1 7400 30 || exit 1

# 7. 웹서버
log "PlantPulse 웹서버 시작 중..."
sudo systemctl start plantpulse-server
wait_for_port 127.0.0.1 80 60 || exit 1

# 8. OPC Agent
log "OPC Agent 시작 중..."
sudo systemctl start plantpulse-agent
wait_for_port 127.0.0.1 60000 30 || exit 1

log "전체 플랫폼 시작 완료"

Windows Startup

Windows Service

If registered as a Windows service, you can start it from Service Manager (services.msc) or the command prompt.

# 서비스 시작 (관리자 권한 PowerShell)
Start-Service PlantPulse-PostgreSQL
Start-Service PlantPulse-Redis
Start-Service PlantPulse-Cassandra
Start-Service PlantPulse-Kafka
Start-Service PlantPulse-MQTT
Start-Service PlantPulse-TSE
Start-Service PlantPulse-CEP
Start-Service PlantPulse-Server
Start-Service PlantPulse-Agent

Starting via Batch File

@echo off
REM plantpulse-start-all.bat - 전체 시작 배치 파일

echo [%date% %time%] PostgreSQL 시작 중...
net start PlantPulse-PostgreSQL
timeout /t 10 /nobreak > nul

echo [%date% %time%] Redis 시작 중...
net start PlantPulse-Redis
timeout /t 5 /nobreak > nul

echo [%date% %time%] Cassandra 시작 중...
net start PlantPulse-Cassandra
timeout /t 60 /nobreak > nul

echo [%date% %time%] Kafka 시작 중...
net start PlantPulse-Kafka
timeout /t 10 /nobreak > nul

echo [%date% %time%] MQTT 시작 중...
net start PlantPulse-MQTT
timeout /t 5 /nobreak > nul

echo [%date% %time%] 시계열 엔진 시작 중...
net start PlantPulse-TSE
timeout /t 10 /nobreak > nul

echo [%date% %time%] CEP 시작 중...
net start PlantPulse-CEP
timeout /t 10 /nobreak > nul

echo [%date% %time%] 웹서버 시작 중...
net start PlantPulse-Server
timeout /t 30 /nobreak > nul

echo [%date% %time%] OPC Agent 시작 중...
net start PlantPulse-Agent
timeout /t 10 /nobreak > nul

echo [%date% %time%] 전체 시작 완료
pause

Shutdown Order

Shutdown must be performed in the reverse order of startup. Stop the data-collecting agents first, and stop the database last.

Order	Service	Description
1	OPC Agent	stop data collection
2	PlantPulse Web Server (Tomcat)	stop web service
3	CEP	stop event processing
4	TSE	stop time series engine
5	MQ (Kafka / MQTT)	stop message broker
6	Cassandra	stop time series DB
7	Redis (Valkey)	stop cache
8	PostgreSQL	stop meta DB

Caution: Stopping Cassandra before other services can cause loss of unflushed Memtable data. Be sure to stop the agents and web server first to halt data writes, then stop Cassandra. Before stopping Cassandra, it's recommended to force-flush Memtables using the nodetool drain command.

Linux Shutdown Commands

# 1. 에이전트 종료
sudo systemctl stop plantpulse-agent

# 2. 웹서버 종료
sudo systemctl stop plantpulse-server

# 3. 엔진 종료
sudo systemctl stop plantpulse-cep
sudo systemctl stop plantpulse-timeseries

# 4. 메시징 종료
sudo systemctl stop plantpulse-mqtt
sudo systemctl stop plantpulse-kafka

# 5. Cassandra 안전 종료 (Memtable 플러시 후 종료)
/opt/kopens/plantpulse-platform/plantpulse-storage/db/cassandra/bin/nodetool drain
sudo systemctl stop plantpulse-cassandra

# 6. Redis 종료
sudo systemctl stop plantpulse-redis

# 7. PostgreSQL 종료
sudo systemctl stop plantpulse-postgresql

Windows Shutdown Commands

# 역순으로 서비스 중지 (관리자 권한 PowerShell)
Stop-Service PlantPulse-Agent
Stop-Service PlantPulse-Server
Stop-Service PlantPulse-CEP
Stop-Service PlantPulse-TSE
Stop-Service PlantPulse-MQTT
Stop-Service PlantPulse-Kafka
Stop-Service PlantPulse-Cassandra
Stop-Service PlantPulse-Redis
Stop-Service PlantPulse-PostgreSQL

@echo off
REM plantpulse-stop-all.bat - 전체 종료 배치 파일

echo [%date% %time%] OPC Agent 종료 중...
net stop PlantPulse-Agent
timeout /t 5 /nobreak > nul

echo [%date% %time%] 웹서버 종료 중...
net stop PlantPulse-Server
timeout /t 10 /nobreak > nul

echo [%date% %time%] CEP 종료 중...
net stop PlantPulse-CEP
timeout /t 5 /nobreak > nul

echo [%date% %time%] 시계열 엔진 종료 중...
net stop PlantPulse-TSE
timeout /t 5 /nobreak > nul

echo [%date% %time%] MQTT 종료 중...
net stop PlantPulse-MQTT
timeout /t 5 /nobreak > nul

echo [%date% %time%] Kafka 종료 중...
net stop PlantPulse-Kafka
timeout /t 10 /nobreak > nul

echo [%date% %time%] Cassandra 종료 중...
net stop PlantPulse-Cassandra
timeout /t 30 /nobreak > nul

echo [%date% %time%] Redis 종료 중...
net stop PlantPulse-Redis
timeout /t 5 /nobreak > nul

echo [%date% %time%] PostgreSQL 종료 중...
net stop PlantPulse-PostgreSQL
timeout /t 10 /nobreak > nul

echo [%date% %time%] 전체 종료 완료
pause

Restart

Normal Restart

To restart the entire platform, perform shutdown followed by startup in order.

# 전체 종료 (역순)
sudo systemctl stop plantpulse-agent
sudo systemctl stop plantpulse-server
sudo systemctl stop plantpulse-cep
sudo systemctl stop plantpulse-timeseries
sudo systemctl stop plantpulse-mqtt
sudo systemctl stop plantpulse-kafka
/opt/kopens/plantpulse-platform/plantpulse-storage/db/cassandra/bin/nodetool drain
sudo systemctl stop plantpulse-cassandra
sudo systemctl stop plantpulse-redis
sudo systemctl stop plantpulse-postgresql

# 전체 시작 (정순)
sudo systemctl start plantpulse-postgresql
sudo systemctl start plantpulse-redis
sudo systemctl start plantpulse-cassandra
sleep 60  # Cassandra 시작 대기
sudo systemctl start plantpulse-kafka
sudo systemctl start plantpulse-mqtt
sudo systemctl start plantpulse-timeseries
sudo systemctl start plantpulse-cep
sudo systemctl start plantpulse-server
sudo systemctl start plantpulse-agent

Rolling Restart (Zero-Downtime Restart)

In a cluster environment, you can perform a Rolling Restart that restarts nodes one at a time without service interruption.

#!/bin/bash
# rolling-restart.sh - 클러스터 Rolling Restart
# 사용법: ./rolling-restart.sh node1 node2 node3

NODES=("$@")

for NODE in "${NODES[@]}"; do
    echo "=== $NODE 재시작 시작 ==="

    # 1. 로드밸런서에서 노드 제거
    echo "$NODE 로드밸런서에서 제거 중..."
    # curl -X POST http://loadbalancer/api/remove-node -d "node=$NODE"

    # 2. 연결 드레인 대기 (기존 요청 처리 완료 대기)
    echo "기존 연결 드레인 대기 (30초)..."
    sleep 30

    # 3. 서비스 재시작
    echo "$NODE 서비스 재시작 중..."
    ssh "$NODE" "sudo systemctl restart plantpulse-server"

    # 4. 헬스체크 통과 대기
    echo "$NODE 헬스체크 대기 중..."
    until ssh "$NODE" "curl -sf http://localhost/api/v5/ping > /dev/null 2>&1"; do
        sleep 5
    done

    # 5. 로드밸런서에 노드 재등록
    echo "$NODE 로드밸런서에 재등록 중..."
    # curl -X POST http://loadbalancer/api/add-node -d "node=$NODE"

    echo "=== $NODE 재시작 완료 ==="
    echo "다음 노드 진행 전 안정화 대기 (60초)..."
    sleep 60
done

echo "Rolling Restart 완료"

Note: Rolling Restart applies to the web server (Tomcat). For a Rolling Restart of the Cassandra cluster, perform nodetool drain and then restart each node sequentially.

Status Check

Checking Processes

# 전체 PlantPulse 관련 프로세스 확인
ps aux | grep plantpulse

# 특정 서비스 프로세스 확인
ps aux | grep plantpulse-server
ps aux | grep cassandra
ps aux | grep kafka

Checking Ports

# 핵심 포트 한 번에 확인
for port in 5432 6379 9042 9092 1883 7800 7400 80 60000; do
    if nc -z 127.0.0.1 $port 2>/dev/null; then
        echo "OK: 포트 $port 열림"
    else
        echo "FAIL: 포트 $port 닫힘"
    fi
done

Healthcheck API

The PlantPulse web server provides a healthcheck via the /api/v5/ping endpoint.

# 기본 헬스체크
curl -sf http://localhost/api/v5/ping
# 응답: {"status":"OK","timestamp":1709884800000}

# HTTP 상태 코드만 확인
curl -sf -o /dev/null -w "%{http_code}" http://localhost/api/v5/ping
# 200이면 정상

Checking Logs

# PlantPulse 웹서버 로그
tail -f /opt/kopens/plantpulse-platform/plantpulse-server/logs/system.log

# Cassandra 로그
tail -f /opt/kopens/plantpulse-platform/plantpulse-storage/db/cassandra/logs/system.log

# Kafka 로그
tail -f /opt/kopens/plantpulse-platform/plantpulse-messaging/kafka/logs/server.log

# 에이전트 로그
tail -f /opt/kopens/plantpulse-platform/plantpulse-plugin/opc-ua/logs/agent.log

# 시작 시 에러 로그만 필터링
grep -i "error\|exception\|fail" /opt/kopens/plantpulse-platform/plantpulse-server/logs/system.log | tail -20

Checking JVM Status

# Java 프로세스 목록 확인
jps -lv

# PlantPulse 서버 힙 메모리 확인
jstat -gc $(pgrep -f plantpulse-server) 1000 5

# GC 로그 확인
tail -f /opt/kopens/plantpulse-platform/plantpulse-server/logs/gc.log

# 스레드 덤프 (문제 진단 시)
jstack $(pgrep -f plantpulse-server) > /tmp/thread-dump-$(date +%Y%m%d%H%M%S).txt

systemd Service Status

# 전체 PlantPulse 서비스 상태 확인
systemctl list-units 'plantpulse-*' --all

# 특정 서비스 상세 상태
sudo systemctl status plantpulse-server
sudo systemctl status plantpulse-cassandra

Auto-Start Configuration

Linux (systemd enable)

Configure services to start automatically on server boot.

# 자동 시작 활성화
sudo systemctl enable plantpulse-postgresql
sudo systemctl enable plantpulse-redis
sudo systemctl enable plantpulse-cassandra
sudo systemctl enable plantpulse-kafka
sudo systemctl enable plantpulse-mqtt
sudo systemctl enable plantpulse-timeseries
sudo systemctl enable plantpulse-cep
sudo systemctl enable plantpulse-server
sudo systemctl enable plantpulse-agent

# 자동 시작 상태 확인
systemctl list-unit-files 'plantpulse-*' | grep enabled

Note: By using the After= directive in the systemd unit file to set startup order dependencies between services, they will start in the correct order even at boot. Example:

[Unit]
Description=PlantPulse Server
After=plantpulse-postgresql.service plantpulse-redis.service plantpulse-cassandra.service
Requires=plantpulse-postgresql.service plantpulse-redis.service

[Service]
Type=forking
User=kopens
ExecStart=/opt/kopens/plantpulse-platform/plantpulse-server/bin/startup.sh
ExecStop=/opt/kopens/plantpulse-platform/plantpulse-server/bin/shutdown.sh
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Windows Service Auto-Start

# 서비스 자동 시작 설정
Set-Service -Name "PlantPulse-PostgreSQL" -StartupType Automatic
Set-Service -Name "PlantPulse-Redis" -StartupType Automatic
Set-Service -Name "PlantPulse-Cassandra" -StartupType Automatic
Set-Service -Name "PlantPulse-Kafka" -StartupType Automatic
Set-Service -Name "PlantPulse-Server" -StartupType Automatic
Set-Service -Name "PlantPulse-Agent" -StartupType Automatic

# 자동 시작 상태 확인
Get-Service PlantPulse-* | Select-Object Name, StartType, Status

Emergency Procedures

Server Not Responding

If the web server is not responding, take action in the following order.

# 1. 헬스체크 확인
curl -sf --connect-timeout 5 http://localhost/api/v5/ping
echo "HTTP 응답 코드: $?"

# 2. 프로세스 상태 확인
ps aux | grep plantpulse-server

# 3. 포트 점유 확인
ss -tlnp | grep ':80'

# 4. 스레드 덤프 (행(hang) 의심 시)
jstack $(pgrep -f plantpulse-server) > /tmp/thread-dump-$(date +%Y%m%d%H%M%S).txt

# 5. GC 상태 확인
jstat -gcutil $(pgrep -f plantpulse-server) 1000 3

# 6. 웹서버만 재시작 (다른 서비스는 유지)
sudo systemctl restart plantpulse-server

# 7. 재시작 후 헬스체크 확인
sleep 30
curl -sf http://localhost/api/v5/ping

OOM (Out Of Memory) Occurs

# 1. OOM 발생 확인
dmesg | grep -i "out of memory\|oom"

# 2. 힙 덤프 확인 (자동 생성된 경우)
ls -la /opt/kopens/plantpulse-platform/plantpulse-server/logs/heapdump*

# 3. 메모리 사용량 확인
free -h
ps aux --sort=-%mem | head -10

# 4. JVM 힙 크기 조정 (catalina.sh 또는 setenv.sh)
# JAVA_OPTS="-Xms4g -Xmx8g -XX:+HeapDumpOnOutOfMemoryError"
# 설정 변경 후 재시작
sudo systemctl restart plantpulse-server

DB Connection Failure

# 1. PostgreSQL 연결 확인
psql -h 127.0.0.1 -U plantpulse -d plantpulse -c "SELECT 1"

# 2. Cassandra 연결 확인
cqlsh 127.0.0.1 -e "DESCRIBE KEYSPACES"

# 3. Redis 연결 확인
redis-cli -h 127.0.0.1 ping

# 4. 연결 수 확인 (PostgreSQL)
psql -h 127.0.0.1 -U plantpulse -d plantpulse -c "SELECT count(*) FROM pg_stat_activity"

# 5. DB 서비스 재시작 (필요 시)
# 주의: DB 재시작 전 반드시 웹서버와 에이전트를 먼저 종료해 주세요
sudo systemctl stop plantpulse-agent
sudo systemctl stop plantpulse-server
sudo systemctl restart plantpulse-postgresql
sudo systemctl start plantpulse-server
sudo systemctl start plantpulse-agent

Operation Checklist

Daily Inspection

Inspection Item	Command / Check Method	Normal Criteria
Overall service status	`systemctl list-units 'plantpulse-*'`	all services active (running)
Core port check	port check script (see above)	all ports open
Healthcheck API	`curl http://localhost/api/v5/ping`	HTTP 200, status OK
Disk usage	`df -h`	usage below 80%
Memory usage	`free -h`	usage below 85%
Error logs	`grep ERROR plantpulse.log \| tail -20`	no recurring errors
Cassandra status	`nodetool status`	all nodes UN (Up/Normal)

Weekly Inspection

Inspection Item	Command / Check Method	Normal Criteria
Cassandra Compaction	`nodetool compactionstats`	no excessive pending tasks
PostgreSQL statistics	query `pg_stat_activity`	no excessive idle connections
Kafka Consumer Lag	`kafka-consumer-groups.sh --describe`	delayed message count within normal range
JVM GC statistics	`jstat -gcutil`	low frequency of Full GC
Backup check	check recent backup files	valid backup exists
Log file volume	`du -sh */logs/`	no abnormal growth
Security updates	check OS package updates	no known vulnerabilities

Overview​

Recommended Operation Commands (Summary)​

Startup Order (Automatic)​

Between Containers​

Inside the Datalake Container​

Shutdown Order​

Restart​

Full Restart​

Restarting a Single App​

Restarting Only Infrastructure Components​

Status Check​

One-Line Summary​

Checking Containers​

Healthcheck API​

Checking Logs​

Auto-Start Configuration​

Emergency Procedures​

Web Console Not Responding​

OOM (Out Of Memory) Occurs​

DB Connection Failure​

If Data Corruption Is Suspected​

Operation Checklist​

Daily Inspection​

Weekly Inspection​

Binary (Native) Environment​

Linux Startup (systemd)​

Linux Manual Startup​

Startup Script (with Dependency Checks)​

Windows Startup​

Windows Service​

Starting via Batch File​

Shutdown Order​

Linux Shutdown Commands​

Windows Shutdown Commands​

Restart​

Normal Restart​

Rolling Restart (Zero-Downtime Restart)​

Status Check​

Checking Processes​

Checking Ports​

Healthcheck API​

Checking Logs​

Checking JVM Status​

systemd Service Status​

Auto-Start Configuration​

Linux (systemd enable)​

Windows Service Auto-Start​

Emergency Procedures​

Server Not Responding​

OOM (Out Of Memory) Occurs​

DB Connection Failure​

Operation Checklist​

Daily Inspection​

Weekly Inspection​

Overview

Recommended Operation Commands (Summary)

Startup Order (Automatic)

Between Containers

Inside the Datalake Container

Shutdown Order

Restart

Full Restart

Restarting a Single App

Restarting Only Infrastructure Components

Status Check

One-Line Summary

Checking Containers

Healthcheck API

Checking Logs

Auto-Start Configuration

Emergency Procedures

Web Console Not Responding

OOM (Out Of Memory) Occurs

DB Connection Failure

If Data Corruption Is Suspected

Operation Checklist

Daily Inspection

Weekly Inspection

Binary (Native) Environment

Linux Startup (systemd)

Linux Manual Startup

Startup Script (with Dependency Checks)

Windows Startup

Windows Service

Starting via Batch File

Shutdown Order

Linux Shutdown Commands

Windows Shutdown Commands

Restart

Normal Restart

Rolling Restart (Zero-Downtime Restart)

Status Check

Checking Processes

Checking Ports

Healthcheck API

Checking Logs

Checking JVM Status

systemd Service Status

Auto-Start Configuration

Linux (systemd enable)

Windows Service Auto-Start

Emergency Procedures

Server Not Responding

OOM (Out Of Memory) Occurs

DB Connection Failure

Operation Checklist

Daily Inspection

Weekly Inspection