System Startup and Shutdown
Overview
PlantPulse operates as a Docker Compose stack. Startup / shutdown / restart are managed in one place by the common verbs in /opt/kopens/plantpulse-platform-docker/bin/, and the dependency order between containers is applied automatically by depends_on in compose, so operators don't need to worry about the sequence.
The 0 of up.sh and restart.sh does not mean "the command succeeded" but rather "it is now usable." During the credential rotation on 2026-08-14, an incident occurred where someone saw "started" and moved to the next step, only to hang on a broker that wasn't up yet, so these two verbs were changed to wait until ready.
status.sh is different — 0 = OK / 2 = unhealthy.
Recommended Operational Commands (Summary)
cd /opt/kopens/plantpulse-platform-docker/bin
./up.sh # 기동 (멱등) — 준비될 때까지 대기. 0 = 준비 완료
./down.sh # 안전 종료 (의존 순서의 역순, 상태 보존)
./restart.sh # graceful drain → 정지 → 기동 → 준비 대기
./status.sh # 서비스 / health / 볼륨 요약. 0 = 정상 / 2 = 비정상
./ops-check.sh # 헬스 + 최근 critical log
./logs.sh [서비스] # 로그 보기 — 1회 출력, -f 로 따라가기 (인자 없으면 전체)
./stack-verify-boot.sh # 스택 전체 준비 판정
./doctor.sh # 진단 tarball (시스템 변경 없음)
All verbs support --help. The legacy names (stack-run.sh · stack-stop.sh · stack-bash.sh · stack-update.sh · stack-remove.sh) still work.
| Environment Variable | Default | Meaning |
|---|---|---|
PP_READY_TIMEOUT | 900 | Readiness wait upper limit (seconds) |
PP_READY_INTERVAL | 15 | Check interval (seconds) |
PP_WAIT=0 | — | No waiting. In this case, 0 does not mean ready |
Startup Sequence (Automatic)
Between Containers
The depends_on of compose maintains order under the service_healthy condition — waiting for "usable," not "process is up." This order is not arbitrary; it mirrors the boot sequence of the application orchestrator.
The last four (warehouse · opcua · aasx · ha) have no order between them.
Inside the Datalake Container
Infrastructure components start sequentially again within the single plantpulse-datalake container.
| Order | Area | Key Components | Representative Ports |
|---|---|---|---|
| 1 | Storage | Aster TSDB, PostgreSQL, Valkey, RustFS | 9042 / 5432 / 6379 / 9000 |
| 2 | Analysis | Spark, Hive, Kyuubi, Gravitino | 7077 / 9083 / 10000 / 19001 |
| 3 | Time Series | TSE Engine, UI | 7800 / 3000 |
| 4 | Messaging | Kafka, MQTT | 9092 / 1883 |
| 5 | Workflow | Temporal, Kestra | 7233 / 8233 / 8380 |
| 6 | Processing | CEP, Data Gateway, SQL, Monitor | 7400 / 5500 / 4000 / 4950 |
Process startup itself takes 3–5 minutes (JVM warm-up), but for a clean installation, due to Aster TSDB schema migration and stabilization, it takes 15–18 minutes for all components to become stable.
Measured (2026-08-31, 32 vCPU / 128GiB): datalake 217 seconds, web server 316 seconds. Restarts with existing data are much faster.
Shutdown Sequence
cd /opt/kopens/plantpulse-platform-docker/bin
./down.sh
You don't need to maintain the order manually. Compose stops containers in the reverse order of dependencies, so the data-writing side (apps) stops first, and the datalake stops last. Each container has a generous stop grace period (stop_grace_period) — 180 seconds for the datalake — giving Aster TSDB time to flush its Memtable.
Volumes (pp-data · pp-temp · pp-backup · pp-security · pp-proxy-certs) are always preserved, regardless of stop or removal.
docker kill or force-kill the hostUnflushed Memtable data in Aster TSDB could be lost. Be sure to use ./down.sh. If a container is already unresponsive, follow the Emergency Procedure.
Restart
Full Restart
cd /opt/kopens/plantpulse-platform-docker/bin
./restart.sh
- (If running) send a graceful drain request to the datalake
- Stop the entire stack in the reverse order of dependencies
- Bring it back up in
depends_onorder - Wait until ready — if the exit code is
0, it's usable
Use for bin/env.sh changes, locale/timezone changes, resolving temporary issues, and periodic restarts.
Restarting a Single App
The six apps each run in their own container, so bringing just that container back up leaves the other apps and infrastructure alive. This is one of the reasons for splitting the containers.
cd /opt/kopens/plantpulse-platform-docker
docker compose -f compose/docker-compose.yml restart plantpulse-server-web
docker compose restart does not respect dependency orderIf you need to bring multiple apps back together, it's safer to restart the entire stack with ./restart.sh.
Restarting Infrastructure Components Only
./shell.sh # 데이터레이크 진입
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd restart storage
exit
The names that pd restart <서비스> accepts are only the eight listed in services/order.txt — storage · analytics · messaging · timeseries · cep · workflow · data-gateway · admin-api. Any other name ends with unknown service. For details, refer to the Startup Guide.
Status Check
One-Line Summary
./status.sh # 0 = 정상 / 2 = 비정상
The status.sh considers five things unhealthy.
| Check | Meaning |
|---|---|
| Cannot read service list | Compose parsing failure or Docker inaccessible |
| No container for a service declared by compose | Not running |
Container not running | All nine run continuously — no exceptions |
Health of an always-on container is unhealthy | starting / none are evaluated as pending |
| Health API not OK | Probe inside the datalake |
Volumes and conf are only reported and not included in the exit code — because it's normal for some to be absent depending on the installation configuration.
plantpulse-certs bakes the certificate once and stays up to monitor expiry. You can judge all nine by the same criteria (healthy).
Healthchecks are baked into the image. It's intentional that compose has no probe block, and the pp.healthcheck: image label makes that explicit. If you write different probe paths into compose, they'll break at the customer site the moment the repository moves the file.
Container Check
docker ps # 아홉 개가 모두 (healthy)
docker stats --no-stream # 컨테이너별 CPU / 메모리
Healthcheck API
# 호스트 / 외부에서 — 4950 이 publish 되어 있습니다
curl -kfsS https://<server-ip>:4950/api/health | jq
# 컨테이너 안에서 — 어떤 구성에서도 동작합니다
docker exec plantpulse-datalake curl -kfsS https://127.0.0.1:4950/api/health | jq
The "status" value can be one of three — OK · WARN · FAIL, with OK/WARN being the normal range.
The console and health API are served on both ports — 4950 (HTTPS) and 4949 (plaintext HTTP). Same console, same API, only the scheme differs. 4949 no longer redirects to 4950.
4949 is plaintext — login passwords and session cookies flow through unencrypted. On untrusted networks, use 4950. 4949 is an option for sites where a self-signed certificate warning actually stops operators in their tracks.
Log Check
./logs.sh # 여덟 컨테이너를 시간순으로 한 화면에
./logs.sh plantpulse-server-web -n 200 # 특정 컨테이너
./logs.sh cassandra # 데이터레이크 안 컴포넌트 로그 파일
./logs.sh --list # 볼 수 있는 대상 전체
./logs.sh -f plantpulse-proxy # 계속 따라가기 (Ctrl-C 로 종료)
The default is print the last N lines (default 200) and exit. To follow, append -f.
Auto-Start Configuration
No separate configuration is needed. Containers have a restart policy declared in compose, so even if the host reboots, they come up when the Docker daemon starts. plantpulse-certs is restart: unless-stopped — if someone manually stopped a container, it stays down even after the daemon restarts. Certificates are not re-baked on restart.
The only thing to check is that the Docker daemon's auto-start is enabled.
systemctl is-enabled docker # enabled 여야 합니다
sudo systemctl enable docker # 아니라면
A host reboot is the only path where the depends_on order doesn't apply — everything comes up at once, so the proxy's healthcheck may fail briefly until the web server is up (measured up to 316 seconds). The retry budget is set at 360 seconds, and Docker won't restart a container for being unhealthy, so this is a temporary condition that resolves on its own.
Emergency Procedure
Web Console Not Responding
cd /opt/kopens/plantpulse-platform-docker/bin
# 1. 어느 컨테이너가 문제인가
./status.sh
# 2. 프록시와 웹 서버 로그
./logs.sh plantpulse-proxy -n 100
./logs.sh plantpulse-server-web -n 200
# 3. 웹 서버만 재시작 (인프라는 유지)
cd /opt/kopens/plantpulse-platform-docker
docker compose -f compose/docker-compose.yml restart plantpulse-server-web
# 4. 회복되지 않으면 진단 번들
bin/doctor.sh
The proxy's healthcheck only checks itself + upstream reachability (the backend's health is judged by the web server's own healthcheck). If the proxy is green but the screen isn't rendering, look at the plantpulse-server-web side.
OOM (Out Of Memory) Occurred
# 1. 어느 컨테이너가 OOM 인가
docker inspect plantpulse-server-web --format '{{.State.OOMKilled}} {{.State.ExitCode}} {{.RestartCount}}'
# 2. 호스트 커널 로그
dmesg | grep -i "out of memory\|oom"
# 3. 컨테이너별 사용량
docker stats --no-stream
Each container has its own limit — the datalake is DOCKER_DATALAKE_MEMORY (default 96G), and apps are DOCKER_SERVER_MEMORY · DOCKER_BATCH_MEMORY etc. (Environment Variables). After adjusting the value, apply it with ./restart.sh.
Having a separate mem_limit per app is the first purpose of container separation. Even if one app hits its limit, infrastructure and other apps stay alive.
DB Connection Failure
All databases are inside plantpulse-datalake.
cd /opt/kopens/plantpulse-platform-docker/bin
./shell.sh # 데이터레이크 진입
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node psql # PostgreSQL
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node cql # Cassandra
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node status # Cassandra 링 상태
To bring back up only the infrastructure, use pd restart storage inside the container; if that doesn't work, use ./restart.sh on the host.
Suspected Data Corruption
cd /opt/kopens/plantpulse-platform-docker/bin
./down.sh # 즉시 정지 (추가 손상 방지)
ls -lh /data1/pp-backup/docker-volume/ # 최신 백업 확인
./doctor.sh # 진단 번들 (데이터를 임의로 수정하지 마세요)
Please send the generated tarball to webmaster@kopens.com.
Operational Checklist
Daily Inspection
| Inspection Item | Command / Check Method | Normal Criteria |
|---|---|---|
| Overall service status | ./status.sh | Exit code 0 |
| Operational health | ./ops-check.sh | No critical logs |
| Health API | curl -kfsS https://<server-ip>:4950/api/health | status is OK or WARN |
| Containers | docker ps | All nine are (healthy) |
| Disk usage | df -h /data1 | Usage below 80% |
| Container resources | docker stats --no-stream | Sufficient headroom against limits |
| Aster TSDB status | pd node status inside the container | All nodes UN (Up/Normal) |
Weekly Inspection
| Inspection Item | Command / Check Method | Normal Criteria |
|---|---|---|
| Aster TSDB Compaction | pd node compactionstats inside the container | No excessive pending tasks |
| Aster TSDB tables | pd node table-stats inside the container | No abnormal growth |
| Kafka topics / lag | pd node topic inside the container | Delayed message count in normal range |
| Backup check | ls -lh /data1/pp-backup/docker-volume/ | Normal backups exist |
| Docker usage | docker system df | No accumulation of unused images |
| Log volume | du -sh /opt/kopens/plantpulse-platform-docker/logs | No abnormal growth |
| Security updates | Check OS package updates | No known vulnerabilities |