Backup and Recovery

Data protection is the most critical operational responsibility. PlantPulse integrates backup of PostgreSQL / Cassandra / files / container volumes through the plantpulse-backup module.

Backup commands run «inside» the data lake container

plantpulse-backup is built into the plantpulse-datalake image. Because all databases run in that container.

cd /opt/kopens/plantpulse-platform-docker/bin
./shell.sh                                              # 데이터레이크 진입
cd /opt/kopens/plantpulse-platform/plantpulse-backup/bin

Docker volume backup only runs from the host (see §3 below).

Protected Assets at a Glance

Backup Tool Matrix

Tool	Target	Method	Compression	Deduplication
pgBackRest	PostgreSQL	Full / Differential / Incremental	gzip / zstd	✓
Medusa	Cassandra	Full / Differential (snapshot)	gzip	-
Restic	Files / Directories	Incremental	zstd	✓
mc mirror	MinIO	Incremental sync	-	-
backup-volume.sh	Docker Volumes	tar.gz snapshot	gzip	-

RPO / RTO Target Objectives

Data	RPO (Recovery Point)	RTO (Recovery Time)	Recommended Backup Frequency
PostgreSQL Metadata	1 hour	30 minutes	1× Full daily + hourly WAL
Cassandra Time Series	24 hours	2 hours	1× Diff daily + 1× Full weekly
MinIO Objects	24 hours	4 hours	External S3 mirror + 1× daily
Configuration / Certificates	On change	15 minutes	Immediate on change + 1× monthly

Production data center: The above targets are based on single-site simple operation. If Multi-AZ / Geo-redundant is required, please design external site replication (rsync, mc mirror, Kafka MirrorMaker) together.

Backup Storage Structure

/data1/pp-backup/
├── pgbackrest/postgres/         # PostgreSQL 백업 (pgBackRest)
│   ├── archive/                  # WAL archive
│   └── backup/                   # Full / Diff / Incr
├── medusa/                       # Cassandra 백업 (Medusa)
│   └── <hostname>/<backup-name>/
├── restic/                       # 파일 백업 (Restic)
│   ├── data/
│   ├── index/
│   └── snapshots/
└── docker-volume/                # Docker 볼륨 (tar.gz)
    ├── pp-data-YYYYMMDD.tar.gz
    ├── pp-security-YYYYMMDD.tar.gz
    └── ...

1. plantpulse-backup Integrated Command

The plantpulse-backup module wraps all backup tools.

Initial Setup (First Run Only)

cd /opt/kopens/plantpulse-platform/plantpulse-backup/bin
./backup.sh --install

Automatically performs:

pgBackRest configuration (/etc/pgbackrest/pgbackrest.conf)
Medusa configuration (/etc/medusa/medusa.ini)
Directory / permission creation
pgBackRest stanza creation

PostgreSQL Backup

cd /opt/kopens/plantpulse-platform/plantpulse-backup/bin

./backup.sh --postgres            # Differential (기본, 빠름)
./backup.sh --postgres full       # Full (전체)
./backup.sh --postgres incr       # Incremental (가장 빠름)

Cassandra Backup

./backup.sh --cassandra           # Differential
./backup.sh --cassandra full      # Full

File Backup (Restic)

/opt/kopens/plantpulse-platform/plantpulse-backup/etc/run_restic.sh

/data1/pp-data incremental backup
Retention policy: last 7 days + 1 per month
Automatic integrity check

Backup Cleanup

./backup.sh --purge

pgBackRest: repo1-retention-full (default 2 sets)
Medusa: CASS_PURGE_KEEP_DAYS (default 10 days)
Restic: automatic per policy

2. Automated Backup — Already Enabled

Nothing to install

Automated backup is already baked into the image with five systemd timers and enabled by default. You do not need to run --cron-install.

Timer	Function
`plantpulse-backup-postgres-diff`	PostgreSQL Differential
`plantpulse-backup-postgres-full`	PostgreSQL Full
`plantpulse-backup-cassandra-diff`	Cassandra Differential
`plantpulse-backup-cassandra-full`	Cassandra Full
`plantpulse-backup-purge`	Retention policy cleanup

Default Schedule

Job	`OnCalendar`	Time
PostgreSQL Diff	`--* 01:30:00`	Daily 01:30
PostgreSQL Full	`Sun --* 01:00:00`	Weekly Sunday 01:00
Cassandra Diff	`--* 01:20:00`	Daily 01:20
Cassandra Full	`Sun --* 00:10:00`	Weekly Sunday 00:10
Purge	`Sun --* 03:00:00`	Weekly Sunday 03:00

Enable and Disable

Write a switch to the host's secret sidecar and restart.

# 호스트에서
sudo vi /etc/kopens/plantpulse-platform.env
#   export PP_BACKUP_SCHEDULE_ENABLED=false

cd /opt/kopens/plantpulse-platform-docker/bin
./restart.sh

「Disabled」is not failure, it is the third state

The timer always fires; the wrapper script reads the switch to decide. When disabled, backup is skipped while recording to journal and history.jsonl as result="skipped" — neither ok nor fail. This allows you to distinguish «disabled» from «ran» later.

Do not edit the unit file directly to disable it. The next image build will quietly resurrect the original.

The switch must be in the sidecar

If you write export PP_BACKUP_SCHEDULE_ENABLED=false only to bin/env.sh, it never reaches the container. The compose line that passes this name to the container is all there is; the value comes from the sidecar (or shell export). Incorrect placement results in «appears disabled but keeps running».

3. Docker Volume Backup (Container Environments)

One-liner installation / Docker installation additionally supports volume-level backup.

Run from the host.

cd /opt/kopens/plantpulse-platform-docker/bin

# 기본 세트를 한 번에 (pp-data · pp-security)
./backup.sh

# 볼륨 하나만
./tools/backup-volume.sh pp-data 7 /data1/pp-backup/docker-volume
./tools/backup-volume.sh pp-security 30 /data1/pp-backup/docker-volume

Two are missing from the default set—not by accident: pp-backup is the destination of backup (putting itself inside itself doubles it every time), and pp-temp is useless for recovery. If needed, you can specify them as arguments to back up.

Configuration templates are not volumes

Configuration templates are a host directory /etc/kopens/conf bound-mounted, so they are not Docker volume backup targets. Back them up together as host files. The secret sidecar /etc/kopens/plantpulse-platform.env is the same.

Argument	Description	Default
1	Volume name	(required)
2	Retention days	7
3	Save path	`/data1/pp-backup/docker-volume`

4. Environment Variables (`backup.sh`)

Variable	Default	Description
`BACKUP_ROOT`	`/data1/pp-backup`	Backup root
`PG_VER`	`18`	PostgreSQL version
`PG_DATA`	`/data1/pp-data/postgres/data`	PG data directory
`CASS_BASE`	`/opt/kopens/plantpulse-platform/plantpulse-storage/db/cassandra`	Cassandra installation path
`CQL_USER`	`${PP_CASSANDRA_USER}`	CQL user
`CQL_PASS`	`${PP_CASSANDRA_PASSWORD}`	CQL password
`PG_RETENTION_FULL`	`2`	PG Full retention
`CASS_PURGE_KEEP_DAYS`	`10`	Medusa retention days

5. Off-Site Replication

Local backup alone is vulnerable to data center failure. Recommended pattern:

# 매일 새벽 4시 — 외부 S3 로 복제
0 4 * * * rclone sync /data1/pp-backup/ s3:kopens-pp-backup/$(hostname)/ \
  --bwlimit 50M --log-file /var/log/pp-backup-sync.log

# MinIO 객체는 mc 로 별도 미러
0 5 * * * mc mirror --overwrite local/ s3-offsite/

6. Recovery

6.1 PostgreSQL Recovery

# 1. plantpulse 중지 (PG 의존 모듈 + PG)
cd /opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin
./stop.sh

# 2. 백업 정보 확인
sudo -u postgres pgbackrest --stanza=pg18 info

# 3. 최신 복구
sudo -u postgres pgbackrest --stanza=pg18 restore

# 3-2. 시점 복구 (PITR)
sudo -u postgres pgbackrest --stanza=pg18 --type=time \
  --target="2026-05-30 12:00:00" restore

# 4. 재시작
./start-daemon.sh
./status.sh

6.2 Cassandra Recovery

# 백업 목록
medusa list-backups

# 단일 노드 복구
medusa restore-node --backup-name full_20260530_0010

# 클러스터 전체 복구 (마스터에서)
medusa restore-cluster --backup-name full_20260530_0010

Warning: restore-cluster overwrites data on all nodes. Always validate in a stage environment before running.

6.3 File Recovery (Restic)

# 스냅샷 목록
restic -r /data1/pp-backup/restic snapshots

# 최신 스냅샷 → 임시 위치 복구
restic -r /data1/pp-backup/restic restore latest --target /data1/pp-data-restored

# 특정 파일만 복구
restic -r /data1/pp-backup/restic restore latest --target / --include /data1/pp-data/cassandra/data/pp/tm_tag_point

6.4 Partial Time Series Recovery (plantpulse-recovery)

To reprocess only a specific asset's data for a specific time period:

cd /opt/kopens/plantpulse-platform/plantpulse-recovery/bin
./recovery.sh ASSET_01032 20260501 20260530

6.5 Partial Time Series Export (plantpulse-exporter)

Export to CSV or bulk process with dsbulk:

cd /opt/kopens/plantpulse-platform/plantpulse-exporter/bin

# 에셋 단위 CSV
./export.sh ASSET_01032 20260101 20260530 /tmp

# 테이블 벌크 언로드 → gz
./bulk-unload.sh pp tm_tag_point
# /data1/pp-backup/cassandra/bulk/tm_tag_point.gz

# 다시 로드
./bulk-load.sh pp tm_tag_point

7. Disaster Recovery (DR) Scenarios

Scenario A: Server Hardware Failure (Cold DR)

Expected RTO: 4–8 hours (varies by data volume)

Scenario B: Data Corruption (PITR)

# 1. 손상 시점 직전으로 시점 복구
./stop.sh
sudo -u postgres pgbackrest --stanza=pg18 --type=time \
  --target="2026-05-30 14:30:00" restore

# 2. Cassandra 도 동일 시점 가까운 백업으로
medusa restore-cluster --backup-name diff_20260530_0120

# 3. 재시작
./start-daemon.sh
./status.sh

Scenario C: External Site Failover (Hot DR)

Maintaining a standby cluster at an external site. Requires advance design:

PostgreSQL: streaming replication or pgBackRest async replication
Cassandra: multi-DC cluster + automatic hinted handoff
Kafka: MirrorMaker 2 bidirectional replication
MinIO: site replication or mc mirror --watch

8. Post-Recovery Validation Checklist

cd /opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin

# 1. 전체 모듈 상태
./status.sh

# 2. 헬스 체크
./ops-check.sh

# 3. Cassandra 클러스터 정상
./pd node status

# 4. PostgreSQL 연결
./pd node psql -c "SELECT count(*) FROM site;"

# 5. Kafka 토픽 / Consumer Group
./pd node topic

Browser:

Web console access (http://[HOST]/)
Admin login (admin / admin123!)
Dashboard loads
Alarm list displays
OPC connection status CONNECTED
Recent data shows in time series trend
CEP rule execution confirmed

9. Backup Operations Best Practices

3-2-1 Rule: 3 copies of data, 2 media types, 1 off-site
Monthly Recovery Drill: attempt actual recovery in stage environment
Backup Integrity Validation: restic check, pgbackrest verify, medusa verify
Backup Monitoring: alerts on backup failure (Slack, email)
Encryption: TLS for off-site transfer, LUKS/SSE-S3 for storage
Permission Separation: backup disk on separate mount + read-only mount option
Documentation: record recovery procedures in operations team shared wiki

Technical Support

Backup / recovery inquiries: webmaster@kopens.com

Protected Assets at a Glance​

Backup Tool Matrix​

RPO / RTO Target Objectives​

Backup Storage Structure​

1. plantpulse-backup Integrated Command​

Initial Setup (First Run Only)​

PostgreSQL Backup​

Cassandra Backup​

File Backup (Restic)​

Backup Cleanup​

2. Automated Backup — Already Enabled​

Default Schedule​

Enable and Disable​

3. Docker Volume Backup (Container Environments)​

4. Environment Variables (backup.sh)​

5. Off-Site Replication​

6. Recovery​

6.1 PostgreSQL Recovery​

6.2 Cassandra Recovery​

6.3 File Recovery (Restic)​

6.4 Partial Time Series Recovery (plantpulse-recovery)​

6.5 Partial Time Series Export (plantpulse-exporter)​

7. Disaster Recovery (DR) Scenarios​

Scenario A: Server Hardware Failure (Cold DR)​

Scenario B: Data Corruption (PITR)​

Scenario C: External Site Failover (Hot DR)​

8. Post-Recovery Validation Checklist​

9. Backup Operations Best Practices​

Related Documentation​

Technical Support​