Skip to main content

Cluster Installation

This guide describes how to configure PlantPulse as a master + worker node cluster for large-scale workloads. You can start with a single node and scale up incrementally when throughput reaches its limits.

Multi-node is only about «horizontal scaling»​

This document covers a master / worker cluster that distributes the infrastructure (Aster TSDB · Kafka · Spark) across multiple nodes. Use it when throughput reaches its limits.

The tiered installation (separating the data lake and applications onto different boxes) was removed on 2026-09-02. Every box now gets a full installation.

When should you scale to a cluster?​

Environment SizeRecommended ConfigurationNotes
~5,000 tagsSingle masterCluster not required
5,000 ~ 50,000 tagsMaster + 1-2 workersDistribute analytics / messaging
50,000+ tagsMaster + 3+ workersFull distributed operation

Cluster Topology​

Hardware Requirements​

Recommended specifications for each worker node (the master node follows System Requirements — data lake master 96 GB or more):

ItemMinimumStandardLarge-Scale
CPU16 vCPU32 vCPU48 vCPU
Memory67 GB or more (the 60 GB worker container must fit within 90% of host RAM — 64 GB is below spec)128 GB200+ GB
Data Disk200 GB NVMe1 TB NVMe4 TB NVMe
Network1 Gbps10 Gbps10 Gbps

Inter-node network: 10 Gbps or higher is recommended between cluster nodes. Aster TSDB repair, Kafka replication, and Spark shuffle traffic are significant.

Prerequisites​

The following must be completed on the master node and all worker nodes.

  • Meet the system requirements
  • Configure hostname / static IP / DNS
  • Time synchronization (chronyd) — time difference between nodes must be less than 100ms
  • OS tuning (limits / sysctl / swap off)
  • Private network communication between nodes (RFC 1918 or Tailscale)
  • Allow internal cluster ports (Aster TSDB 7000/7001/9042, Kafka 9092/9093, Spark 7077/8081, etc.)

/etc/hosts Configuration (Identical on All Nodes)​

# /etc/hosts — 마스터 / 워커 모든 노드에 동일하게
192.168.0.41   plantpulse-master   plantpulse-master.local
192.168.0.101  plantpulse-worker-1 plantpulse-worker-1.local
192.168.0.102  plantpulse-worker-2 plantpulse-worker-2.local
192.168.0.103  plantpulse-worker-3 plantpulse-worker-3.local

In a Docker/one-line installation environment, workers are brought up as a compose overlay. With the base stack running on the master node, bin/worker-add.sh adds workers one at a time. Secrets and certificates are copied between nodes via sidecar files using the procedure below, so you don't have to manually transcribe values.

The Source of Truth for the Worker List — compose/workers.roster​

The source of truth for which workers exist is a single file, compose/workers.roster. It follows the <id> <ip> format on each line; if it's a single-node installation, an empty file is normal.

# compose/workers.roster
1 10.99.0.101
ItemMeaning
idPP_WORKER_ID. Suffix for container names (plantpulse-worker-<id>) and volume names (pw-<id>-*)
ipFixed address above pp-net(10.99.0.0/24). .1 is the gateway, .100 is the data lake, so use .101 onwards

compose/docker-compose.worker.yml is generated from this file (bin/gen-worker-compose.sh), and bin/env.sh also derives worker IPs and the node list from the same file. This prevents compose, operations scripts, and secret rotation guards from seeing different sets of workers.

Do not hand-edit generated files

docker-compose.worker.yml is a generated artifact. Hand-editing it will cause the CI's gen-worker-compose.sh --check to fail. To add or remove workers, you don't insert or remove lines in the roster — you use the worker-add.sh / worker-decommission.sh + worker-remove.sh procedures below, because workers carry Aster TSDB token ranges and PostgreSQL replication slots, so they can't be created or removed just by editing a list.

VariableDefaultDescription
DOCKER_PW_NAMEplantpulse-workerPrefix for worker container names (plantpulse-worker-1 …)
DOCKER_PW_MEMORY60G; if the host is small, 90% of RAM (sized separately from the master's DOCKER_DATALAKE_MEMORY)Memory limit for worker containers
DOCKER_PW_IP_<n> · PP_WORKER_NODESDerived from rosterNot manually specified
Do not arbitrarily lower worker memory

Workers use the same image as the data lake master (integrated on 2026-08-31), so they also inherit the JVM size baked into that image — a single Aster TSDB instance requires -Xms16G/-Xmx16G. Sizing it proportionally to host size is wrong. The required amount is determined by the service configuration, not the host.

Measured on 2026-08-31: A worker with the old 12G default was OOMKilled before Aster TSDB could join the ring, and the container died with OOMKilled=true — yet the ring remained at 1 node, and the join never happened, while health: starting was still running.

Adding Workers​

cd /opt/kopens/plantpulse-platform-docker

bin/worker-add.sh                  # 빈 id·빈 주소 자동 선택
bin/worker-add.sh 3                # id 지정
bin/worker-add.sh 3 10.99.0.103    # id·주소 지정

worker-add.sh performs the following steps in order.

  1. Register in the roster (rejects existing IDs since they aren't «additions»)
  2. Recreate the overlay compose
  3. Create volumes → pull images → up -d
  4. Verify ring join — directly check via the master's nodetool
«Container is up» is not proof of a join

Workers carry Aster TSDB token ranges, Spark worker registration, PostgreSQL replication slots, and Redis replication links. The container's own readiness check only looks at whether it can reach the master, so a worker that failed to join reports as healthy just like one that succeeded.

Measured on 2026-08-31: A worker whose Aster TSDB was OOMKilled kept health: starting running while the ring remained at 1 node. So worker-add.sh queries the ring itself.

Worker Operations​

cd /opt/kopens/plantpulse-platform-docker

# 진입 (워커는 기본 스택의 서비스가 아니라 shell.sh 로는 잡히지 않습니다)
docker exec -ti plantpulse-worker-3 /bin/bash

# 정지 — compose 동사 그대로
docker compose -f compose/docker-compose.yml -f compose/docker-compose.worker.yml \
  --profile worker-3 stop plantpulse-worker-3

# 이미지 갱신 (이미 도는 워커. 추가 시점의 pull 은 worker-add.sh 안에 들어 있습니다)
docker compose -f compose/docker-compose.yml -f compose/docker-compose.worker.yml \
  --profile worker-3 pull plantpulse-worker-3
docker compose -f compose/docker-compose.yml -f compose/docker-compose.worker.yml \
  --profile worker-3 up -d plantpulse-worker-3

Removing Workers — Must Follow This Order​

bin/worker-decommission.sh 3       # 데이터 이관 + 링 이탈 확인 + 복제 슬롯 정리
bin/worker-remove.sh               # 그 다음에 컨테이너 제거
Using docker rm on a live worker is «abandonment», not removal

Aster TSDB holds onto that node's token range and host ID as DN. With RF=3, the QUORUM is still satisfied, so no alarm is raised. And the master continues to hold that worker's PostgreSQL physical replication slot, pinning WAL until the disk fills up.

worker-remove.sh rejects the operation if the container is running and instructs you to call worker-decommission.sh first.

Previously removed worker scripts (2026-08-31)

The four scripts worker-run.sh · worker-stop.sh · worker-update.sh · worker-bash.sh have been removed. They were built on the assumption of a manually managed worker service list, so when we switched to the roster-generation approach, we didn't just rename them—we deleted them. The replacements are worker-add.sh (includes pull) · compose verbs · compose pull + up -d · docker exec respectively.

Copying Secret Sidecars (No Manual Copying)​

Service secrets created on the master reside in the /etc/kopens/plantpulse-platform.env sidecar file. Copy this file to the worker node, and the worker joins with the same credentials—don't manually transcribe passwords one by one (it's the same pattern as the join bundle from the two-node separation document).

# 마스터 노드에서 각 워커로
scp /etc/kopens/plantpulse-platform.env root@<worker-ip>:/etc/kopens/

Copying the Shared CA → Automatic Seed​

Inter-node TLS trust is established by a shared cluster CA. Copying the master's /etc/kopens/ca/ to the worker results in automatic seeding into the pp-security volume at container startup, so you don't need to manually place certificates.

# 마스터 노드에서 각 워커로 (컨테이너 경로 기준 자동 seed)
scp -r /etc/kopens/ca root@<worker-ip>:/etc/kopens/

platform.node.env is a per-node identity file, so do not copy it. Only the two items above (plantpulse-platform.env, ca/) are copied.

Cluster Verification (Docker Environment)​

# Cassandra 링 — 모든 노드가 UN (Up Normal) 이어야 합니다
docker exec plantpulse-datalake \
  /opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node status

# Spark — 워커가 ALIVE 인지
#   브라우저에서 http://<마스터IP>:4440/ 의 Workers 탭

Frequently Encountered Issues​

SymptomCauseVerification
Worker doesn't join the ringPP_MASTER_IP is unreachable from this boxFix /etc/kopens/platform.node.env and bin/restart-datalake.sh
Kafka clients on the worker only connect to the masterAdvertised address is loopbackChange PP_KAFKA_ADVERTISED_HOST to that box's actual address
Sidecar values missing on the workerSecret files manually copiedUse the «Copying Secret Sidecars» procedure above — manual copying breaks permissions and line endings
Workers showing as DN in nodetool statusFirewall blocks the 7000 seriesPort configuration

Cluster Removal​

# 각 워커에서 — 컨테이너를 제거하고 데이터 볼륨은 남긴다
ssh root@<WORKER_IP> 'cd /opt/kopens/plantpulse-platform-docker && bin/remove.sh'

# 마스터에서
cd /opt/kopens/plantpulse-platform-docker && bin/remove.sh
remove.sh and down.sh are different
VerbContainerVolume
bin/down.shStop only — can be brought back with up.sh as-isLeft intact
bin/remove.shRemovedLeft intact (default)

Use bin/tools/remove-all-volumes.sh separately to delete volumes as well. Add RM_IMAGE=1 · RM_NETWORK=1 to also remove images and networks.

There is no command named bin/stop.sh — the common verbs were consolidated to up · down · remove · restart · status · logs · update during the compose transition (startup guide).

Data disks are managed under a separate policy (back them up before deletion if necessary).