Skip to main content

Cluster Installation

This describes how to configure PlantPulse as a master + worker node cluster for large-scale workloads. You can start with a single node and scale out progressively as throughput hits its limits.

Two multi-node configurations — different purposes​

ConfigurationPurposeDocument
① MASTER / WORKER clusterHorizontally scale infrastructure (Cassandra·Kafka·Spark) — distribute the same layer across multiple nodesThis document
② DATALAKE / APP 2-node splitSplit the storage/processing layer and console/batch layer onto separate boxes (vertical separation) — reduce per-node memory usage and isolate load2-Node Split Installation

Use ① when throughput is insufficient and you need to scale out infrastructure; use ② when a single box lacks memory or you need to isolate batch load from the console. The two are not mutually exclusive — PP_MODE (clustering) and PP_TIER (tier gates) are orthogonal.

When should you scale to a cluster?​

Environment sizeRecommended configurationNotes
~5,000 tagsSingle masterCluster not needed
5,000 ~ 50,000 tagsMaster + 1-2 workersDistribute analytics / messaging
50,000+ tagsMaster + 3+ workersFull-scale distributed operation

Cluster topology​

Hardware requirements​

Recommended specs for each worker node:

CategoryMinimumStandardLarge-scale
CPU16 vCPU32 vCPU48 vCPU
Memory64 GB128 GB200+ GB
Data disk200 GB NVMe1 TB NVMe4 TB NVMe
Network1 Gbps10 Gbps10 Gbps

Inter-node network: 10 Gbps or higher is recommended between cluster nodes. Cassandra repair, Kafka replication, and Spark shuffle traffic are heavy.

Prerequisites​

The following must be completed on the master node and all worker nodes.

  • Meet system requirements
  • Hostname / static IP / DNS configuration
  • Time synchronization (chronyd) — time difference between nodes must be 100ms or less
  • OS tuning (limits / sysctl / swap off)
  • Private network connectivity between nodes (RFC 1918 or Tailscale)
  • Allow cluster-internal ports (Cassandra 7000/7001/9042, Kafka 9092/9093, Spark 7077/8081, etc.)

/etc/hosts configuration (same on all nodes)​

# /etc/hosts — 마스터 / 워커 모든 노드에 동일하게
192.168.0.41   plantpulse-master   plantpulse-master.local
192.168.0.101  plantpulse-worker-1 plantpulse-worker-1.local
192.168.0.102  plantpulse-worker-2 plantpulse-worker-2.local
192.168.0.103  plantpulse-worker-3 plantpulse-worker-3.local

In a Docker/one-line installation environment, workers are brought up as a compose overlay. With the base stack already running on the master node, bin/worker-add.sh adds workers one at a time. Secrets and certificates are copied between nodes as sidecar files following the procedure below, so you never need to transcribe values by hand.

The single source of truth for the worker list — compose/workers.roster​

The single source of truth for which workers exist is the one file compose/workers.roster. Each line follows the <id> <ip> format, and it is normal for this to be empty on a single-node install.

# compose/workers.roster
1 10.99.0.101
ItemMeaning
idPP_WORKER_ID. Suffix for the container name (plantpulse-worker-<id>) and volume name (pw-<id>-*)
ipThe static address on pp-net(10.99.0.0/24). .1 is the gateway and .100 is the data lake, so use from .101 onward

compose/docker-compose.worker.yml is generated from this file (bin/gen-worker-compose.sh), and bin/env.sh also derives the worker IPs and node list from the same file. This ensures compose, operational scripts, and secret rotation guards never see different sets of workers.

Do not hand-edit generated files

docker-compose.worker.yml is a generated artifact. Editing it by hand will cause CI's gen-worker-compose.sh --check to fail. Adding or removing workers is done not by editing lines in the roster, but via worker-add.sh / worker-decommission.sh + worker-remove.sh below — workers hold Cassandra token ranges and PostgreSQL replication slots, so they cannot be created or removed simply by editing the list.

VariableDefaultDescription
DOCKER_PW_NAMEplantpulse-workerWorker container name prefix (plantpulse-worker-1 …)
DOCKER_PW_MEMORYSame value as DOCKER_DATALAKE_MEMORY (80G, 90% of RAM if the host is small)Worker container memory limit
DOCKER_PW_IP_<n> · PP_WORKER_NODESDerived from rosterDo not write by hand
Do not arbitrarily lower worker memory

Since workers use the same image as the data lake master (unified as of 2026-08-31), they inherit the JVM sizing baked into that image as-is — a single Cassandra instance requires -Xms16G/-Xmx16G. Sizing proportional to host size is incorrect. The required amount is determined by service composition, not the host.

Observed on 2026-08-31: a worker with the old 12g default was OOMKilled before Cassandra could reach the ring, and while the container died with OOMKilled=true, it remained in health: starting without ever joining.

Adding a worker​

cd /opt/kopens/plantpulse-platform-docker

bin/worker-add.sh                  # 빈 id·빈 주소 자동 선택
bin/worker-add.sh 3                # id 지정
bin/worker-add.sh 3 10.99.0.103    # id·주소 지정

worker-add.sh performs the following in order:

  1. Register in the roster (rejected as not an «addition» if the id already exists)
  2. Regenerate the overlay compose
  3. Create volume → pull image → up -d
  4. Verify ring join — validate directly with the master's nodetool
"The container is up" is not proof of joining

A worker also carries a Cassandra token range, Spark worker registration, PostgreSQL replication slot, and Redis replication link. Because the container's own readiness check only looks at "can it reach the master," a worker that failed to join reports as healthy, exactly like one that succeeded.

Observed on 2026-08-31: a worker whose Cassandra was OOMKilled remained in health: starting while the ring stayed at 1 node. This is why worker-add.sh queries the ring itself.

Worker operations​

cd /opt/kopens/plantpulse-platform-docker

# 진입 (워커는 기본 스택의 서비스가 아니라 shell.sh 로는 잡히지 않습니다)
docker exec -ti plantpulse-worker-3 /bin/bash

# 정지 — compose 동사 그대로
docker compose -f compose/docker-compose.yml -f compose/docker-compose.worker.yml \
  --profile worker-3 stop plantpulse-worker-3

# 이미지 갱신 (이미 도는 워커. 추가 시점의 pull 은 worker-add.sh 안에 들어 있습니다)
docker compose -f compose/docker-compose.yml -f compose/docker-compose.worker.yml \
  --profile worker-3 pull plantpulse-worker-3
docker compose -f compose/docker-compose.yml -f compose/docker-compose.worker.yml \
  --profile worker-3 up -d plantpulse-worker-3

Removing a worker — this order is mandatory​

bin/worker-decommission.sh 3       # 데이터 이관 + 링 이탈 확인 + 복제 슬롯 정리
bin/worker-remove.sh               # 그 다음에 컨테이너 제거
docker rm on a live worker is not removal — it's abandonment

Cassandra holds that node's token range and host id as DN. With RF=3, QUORUM still holds, so no alert fires at all. And the master continues to hold that worker's PostgreSQL physical replication slot, pinning the WAL until the disk fills up.

worker-remove.sh refuses if the container is still running, and instructs you to call worker-decommission.sh first.

Deleted worker scripts (2026-08-31)

The four scripts worker-run.sh · worker-stop.sh · worker-update.sh · worker-bash.sh have been removed. They were built on the assumption of a manually managed list of worker services, and when the design moved to roster generation, they weren't just renamed — they were eliminated. Their replacements are worker-add.sh (includes pull) · compose verbs · compose pull + up -d · docker exec respectively.

Copying the secrets sidecar (do not copy by hand)​

Service secrets generated on the master live in the /etc/kopens/plantpulse-platform.env sidecar file. Copying this file to a worker node lets the worker join with the same credentials — do not transcribe passwords one by one (same pattern as the join bundle in the 2-node split document).

# 마스터 노드에서 각 워커로
scp /etc/kopens/plantpulse-platform.env root@<worker-ip>:/etc/kopens/

Copy the shared CA → auto-seed​

Inter-node TLS trust is established via a shared cluster CA. Copying the master's /etc/kopens/ca/ to a worker causes it to be auto-seeded into the pp-security volume at container startup, so you don't need to place certificates manually.

# 마스터 노드에서 각 워커로 (컨테이너 경로 기준 자동 seed)
scp -r /etc/kopens/ca root@<worker-ip>:/etc/kopens/

platform.node.env is a per-node identity file, so do not copy it. Only the two items above (plantpulse-platform.env, ca/) should be copied.

Cluster verification (Docker environment)​

# Cassandra 링 — 모든 노드가 UN (Up Normal) 이어야 합니다
docker exec plantpulse-datalake \
  /opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node status

# Spark — 워커가 ALIVE 인지
#   브라우저에서 http://<마스터IP>:4440/ 의 Workers 탭

Native (binary) cluster procedure​

The current release does not support binary (native) installation

The following procedure is retained for existing systems built via binary installation. Since the current shipped release only offers the Docker Compose path above, please do not use this for new builds.

1. Install the master node​

The master node follows the same procedure as a single-node installation.

1.1 Install the master​

# 마스터 노드에서
mkdir -p /opt/kopens
cd /opt/kopens
tar -xzvf plantpulse-platform-2026.05.tgz

cd /opt/kopens/plantpulse-platform/tools
./setup.sh
# 노드 모드 입력: MASTER
# HOST IP: 192.168.0.41
# SERVICE IP: 192.168.0.41 (외부 노출 IP)

1.2 Adjust env.local.sh (cluster options)​

env.sh is the SSOT (single source of default truth) that gets overwritten on deployment, so do not edit it directly. Write machine-specific values as overrides in env.local.sh in the same directory (env.sh sources it first at the top, so it always takes priority). For details, see Binary Installation §4.

vi /opt/kopens/plantpulse-platform/plantpulse-startup/env.local.sh
# 마스터 노드
export PP_MODE=MASTER
export PP_HOST_IP=192.168.0.41
export PP_SERVICE_IP=192.168.0.41
export PP_MASTER_IP=192.168.0.41
export PP_PUBLIC_IP=192.168.0.41

# 클러스터 자원
export PP_CLUSTER_CORES=64       # 마스터 + 워커 코어 합 (Spark 사용)
export PP_CLUSTER_MEMORY_BY_CORE=2G

# TLS SAN 에 모든 노드 IP / 도메인 포함
export PP_TLS_SAN_IPS="192.168.0.41,192.168.0.101,192.168.0.102,192.168.0.103,127.0.0.1"
export PP_TLS_SAN_DNS="plantpulse-master,plantpulse-worker-1,plantpulse-worker-2,plantpulse-worker-3,localhost"
export PP_TLS_NODE_NAMES="master worker-1 worker-2 worker-3"

1.3 Start the master​

cd /opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin
./configure.sh
./prepare-ssl.sh
./start-daemon.sh
./status.sh

2. Add worker nodes​

2.1 Install workers​

Install on each worker node using the same procedure.

# 워커 노드 (예: 192.168.0.101) 에서
mkdir -p /opt/kopens
cd /opt/kopens
tar -xzvf plantpulse-platform-2026.05.tgz

cd /opt/kopens/plantpulse-platform/tools
./setup.sh
# 노드 모드 입력: WORKER
# HOST IP: 192.168.0.101
# SERVICE IP: 192.168.0.101
# MASTER IP: 192.168.0.41

2.2 Adjust worker env.local.sh (node-specific values only)​

Write only node-specific machine values, such as per-node IPs, to env.local.sh (do not edit env.sh directly ✗).

vi /opt/kopens/plantpulse-platform/plantpulse-startup/env.local.sh
# 워커 노드
export PP_MODE=WORKER
export PP_HOST_IP=192.168.0.101            # 이 워커의 IP
export PP_SERVICE_IP=192.168.0.101
export PP_MASTER_IP=192.168.0.41           # 마스터의 IP
export PP_PUBLIC_IP=192.168.0.101

Do not transcribe passwords/keystore passwords here by hand — synchronize them between nodes using the secrets sidecar below.

2.3 Copy the secrets sidecar​

Service secrets live in the master's /etc/kopens/plantpulse-platform.env sidecar file. Copying this file to a worker lets it join with the same credentials (same pattern as the join bundle in the 2-node split document). Do not transcribe passwords one by one.

# 마스터에서 워커로 (env.sh 보다 먼저 source 되는 사이드카)
scp /etc/kopens/plantpulse-platform.env root@192.168.0.101:/etc/kopens/

2.4 Copy the shared CA → auto-seed​

Inter-node TLS trust is established via a shared cluster CA. Copying the master's /etc/kopens/ca/ to a worker causes it to be auto-seeded based on the container path at startup, so you don't need to place certificates manually.

# 마스터에서 워커로 (공유 CA — 크로스노드 TLS 신뢰)
scp -r /etc/kopens/ca root@192.168.0.101:/etc/kopens/

platform.node.env is a per-node identity file, so do not copy it.

2.5 Start the worker​

cd /opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin
./configure.sh
./start-daemon.sh
./status.sh

The worker node's status.sh output is normal if the following are RUNNING:

  • Cassandra (:9042)
  • Kafka (:9092)
  • Spark Worker (:8081)

Note: Worker nodes do not run application modules such as server / cep / batch. Only infrastructure / distributed processing components are clustered.

3. Cluster verification​

3.1 Cassandra cluster​

# 마스터에서
cd /opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin
./pd node status

Expected output (example):

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load        Tokens  Owns  Host ID                               Rack
UN  192.168.0.41   45.2 GiB    256     ?     <uuid>                                rack1
UN  192.168.0.101  44.8 GiB    256     ?     <uuid>                                rack1
UN  192.168.0.102  45.5 GiB    256     ?     <uuid>                                rack1

If UN (Up Normal) is shown for all nodes, it is normal.

3.2 Kafka cluster​

cd /opt/kopens/plantpulse-platform/plantpulse-messaging/kafka/bin
./kafka-broker-api-versions.sh --bootstrap-server 192.168.0.41:9092 | head

If 3 brokers are shown, it is normal.

3.3 Spark cluster​

Access the Spark UI on the master from a browser:

http://192.168.0.41:4440/

In the Workers tab, verify that all workers are in ALIVE status.

4. Cluster operations​

Adding a worker (dynamic scaling)​

Procedure for adding a worker node to an already running cluster:

Removing a worker​

# 1. 워커를 안전하게 비우기 (Cassandra)
ssh root@192.168.0.103 \
  /opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node drain

# 2. 마스터에서 노드 제거
cd /opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin
./pd node remove
# 호스트 ID 입력

# 3. 워커 정지 후 제거
ssh root@192.168.0.103 \
  /opt/kopens/plantpulse-platform/plantpulse-startup/stop.sh

Node replacement​

When replacing a failed node with a new one, refer to the Admin: Node Replacement procedure. The key point is to keep the same IP / hostname and pull data using bootstrap mode after booting the new node.

5. Cluster backup​

The plantpulse-backup on the master node is responsible for the entire cluster.

cd /opt/kopens/plantpulse-platform/plantpulse-backup/bin

# Cassandra: 모든 노드에서 snapshot 수집
./backup.sh --cassandra

# PostgreSQL: 마스터에서만
./backup.sh --postgres

See Backup and Recovery.

Common issues​

SymptomCauseAction
Worker won't joinSeed node not configuredVerify master IP is included in seeds of cassandra.yaml
Cassandra token imbalanceImproper joinpd node cleanup + reassign tokens
Kafka under-replicatedBroker downkafka-topics.sh --describe + check replication factor
Spark Worker not registeringFirewall blocking 7077Allow port 7077, 8081 bidirectionally on the private network
Certificate mismatchKeystore not synchronizedRe-run prepare-ssl.sh on the master + redeploy to workers
Time mismatchNTP not configuredCheck chronyc tracking, keep under 100ms

Removing a cluster​

# 각 워커에서
ssh root@<WORKER_IP> /opt/kopens/plantpulse-platform/plantpulse-startup/stop.sh
ssh root@<WORKER_IP> rm -rf /opt/kopens/plantpulse-platform

# 마스터에서
cd /opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin
./stop.sh
rm -rf /opt/kopens/plantpulse-platform

# 데이터 디스크는 별도 정책으로 관리 (필요 시 백업 후 삭제)