High Availability (HA) Configuration
This configuration runs the platform in pairs (A·B), ensuring that even if one side fails completely, not a single piece of operational data is lost. It is intended for power plants, large process facilities, and 24/365 continuous-operation equipment.
HA handles availability, consistency, and zero data loss. The safety state of the equipment is independently guaranteed by the PLC / Safety PLC (SIS) under any software failure (IEC 61508 / 61511 SIL). The platform is not the entity that ensures safety.
Two Promises
1 · No Data Loss
If one side has been down, upon recovery it retroactively retrieves only the missed interval that accumulated on the peer. It does not calculate or guess by time—because the Kafka commit offset cursor knows exactly how far it has received.
2 · Control Goes to Only One Place
Control commands are the exact opposite of data. They are not sent to both sides; they go only to a healthy single leader. If the rightful authority is ambiguous, they are not sent (fail-closed).
A refused connection occurs before transmission, so it can be safely handed over to the other side. However, a timeout leaves you in a state of ‘did it go or not’, so it is not retried—because a retry would result in dual actuation.
Configuration — Edge-Platform Pairs
Two edge devices each read the same PLC and send data to different platforms. Thus, both platforms receive the same data live.
One PLC is independently collected by two edge devices, with Edge #1 sending only to A, and Edge #2 only to B. This is different from attaching two edge devices to a single platform — that would be redundant storage.
If they are connected to the same switch, that switch becomes a single point of failure and defeats the purpose of redundancy → Edge HA
Does Nothing Under Normal Conditions
plantpulse-ha The daemon runs continuously on each platform, but in a healthy state it does not assume the traffic. Under normal conditions, the edge pair described above handles the supply to both sides. The daemon only begins its work after one side has failed and been restored, and then it does two things.
| Task | What It Does |
|---|---|
| Time Series Backfill | Subscribes to the peer's pp-batch topic. Once the local side is restored, it re-publishes only the interval missed during the downtime to the local pp-batch. The existing batch-web ingests it as-is, so no changes are needed at the destination |
| PG Meta Synchronization | Configuration that is not carried on Kafka—such as tag metadata, asset structure, and alarm definitions—is pulled from the primary cluster's PG by the standby cluster every 60 seconds |
PG metadata sync also carries the Frontline tablet app's data — operator accounts and scope (including PIN hashes), kiosk settings · per-equipment stop reason lists · paired tablets. The AI key is carried only with metasync.include.tokens=true, and sessions · audit logs · the sign-in cache stay per node (Frontline HA).
event and not pp-batchpp-batch is the form after it has already passed through CEP. Therefore, replaying it does not re-trigger CEP at the destination. The old model (continuous fan-out of the event topic) was discarded due to CEP re-triggering and the constant duplicate delivery load.
How Backfill Works
INIT → GUARD → WAIT_LOCAL_READY → CATCH_UP → TAIL
↑ ↓
PAUSED ←─────┘ (헬스 상실 / 회복)
| State | What It Does |
|---|---|
GUARD | If the peer's and local clusterId are the same, startup is refused—prevents the accidental case of treating itself as its own peer |
WAIT_LOCAL_READY | Waits until the local server-web health becomes stable |
CATCH_UP | Re-publishes the interval before the boundary timestamp T_b to the local side |
TAIL | Segments after T_b are ones the edge may have already delivered to the local side, so they are discarded |
PAUSED | If health is lost, it stops and does not commit |
INCIDENT | If there is an unreadable interval beyond the retention period, it records it and resumes from earliest |
The transition criteria are based on «record timestamp vs. fixed boundary T_b», not on consumer lag. Judging by lag on a low-traffic partition would result in misjudgment and flapping.
TAIL also performs a 60-second delayed commit (commit.lag.seconds)—to ensure that a short window just before a health loss is naturally re-covered by the next cycle of CATCH_UP.
Loops are Structurally Prevented
A pp-ha-origin header is attached to re-published records. Records with that header are never re-published again—it only goes original (h=0) → re-publish (h=1), and stops at h=1.
Duplicates are Absorbed
The same record may be delivered twice near the boundary, but Aster TSDB reload is idempotent based on tag_id + timestamp, so neither tag point loss nor duplication remains.
Guarantees and Limitations
If it is down longer than the peer's pp-batch retention period, that much cannot be automatically recovered. That interval is recorded in INCIDENT, and to restore it, a manual re-replay via backfill.force.from is required. Set the retention period longer than the expected maximum downtime.
| Limitation | Details |
|---|---|
| Statistical counters are non-idempotent | Tag point reload is fully idempotent, but count = count + ? type statistical counters may be over-counted during duplicate replays |
| Asymmetric failures cannot be corrected | If only one edge device fails and supply to one side is cut, the local platform itself is healthy, so the health gate cannot detect it → manual re-replay |
| PG synchronization is one-way | Primary → Standby. Always make console configuration changes on the primary cluster only.—if changed on the standby, they will be overwritten on the next cycle. |
Sites with Only One Edge Device — Dual-Transmission Client
At sites where deploying two edge devices is not possible, the collector directly transmits to both clusters. plantpulse-api-ha provides the same surface as the existing APIClient, so only the creation part needs to be replaced.
- Writes are fanned out to all destinations and return immediately (bound queue + worker + backoff per destination).
- Reads go leader-first + failover + sticky.
- The topology is discovered by polling the ha daemon's
/api/clusterevery 10 seconds and cached (last-known-good).
Operations
# 기동 · 정지 (각 플랫폼에서)
sh/start.sh [설정파일] # 기본 config/plantpulse-ha.properties, 이미 떠 있으면 WARN + exit 0
sh/stop.sh # SIGTERM → 15초 대기 → SIGKILL
sh/log-viewer.sh
The deployment layout is <서비스 루트>/plantpulse-ha/{sh,lib,logs,config,etc}.
When a Failure Occurs
| Situation | What Happens |
|---|---|
| Cluster B down | A continues to ingest normally. When B is restored, B's daemon reads the interval accumulated only on A via CATCH_UP and re-publishes to B → convergence |
| Only the daemon down | The commit offset is on the peer broker, not on the daemon. Upon restart, it resumes from the last committed offset (it must pass the health gate again) |
| Configuration change on the primary cluster | The standby cluster reflects the diff on the next cycle (60 seconds) |