Backup and Recovery

Data protection is the most critical operational responsibility. PlantPulse performs integrated backups of PostgreSQL / Aster TSDB / files / container volumes through the plantpulse-backup module.

Backup commands are executed «inside» the datalake container

plantpulse-backup is contained within the plantpulse-datalake image because all databases run inside that container.

cd /opt/kopens/plantpulse-platform-docker/bin
./shell.sh                                              # 데이터레이크 진입
cd /opt/kopens/plantpulse-platform/plantpulse-backup/bin

Only Docker volume backups are performed on the host (see §3 below).

Protection Targets Overview

Backup Tool Matrix

Tool	Target	Method	Compression	Deduplication
pgBackRest	PostgreSQL	Full / Differential / Incremental	gzip / zstd	✓
Medusa	Aster TSDB	Full / Differential (Snapshot)	gzip	-
Restic	File / Directory	Incremental	zstd	✓
rc mirror	RustFS	Incremental Sync	-	-
backup-volume.sh	Docker Volume	tar.gz Snapshot	gzip	-

Recommended RPO / RTO Targets

Data	RPO (Recovery Point)	RTO (Recovery Time)	Recommended Backup Frequency
PostgreSQL Meta	1 hour	30 min	Daily Full + Hourly WAL
Aster TSDB Time Series	24 hours	2 hours	Daily Diff + Weekly Full
RustFS Object	24 hours	4 hours	Off-site S3 Mirror + Daily
Config / Certificates	On change	15 min	Immediate on change + Monthly

Operational Grade Data Center: The targets above are based on simple single-site operations. If Multi-AZ / Geo-redundant is required, please design external site replication (rsync, rc mirror, Kafka MirrorMaker) concurrently.

Backup Storage Structure

/data1/pp-backup/
├── pgbackrest/postgres/         # PostgreSQL 백업 (pgBackRest)
│   ├── archive/                  # WAL archive
│   └── backup/                   # Full / Diff / Incr
├── medusa/                       # Cassandra 백업 (Medusa)
│   └── <hostname>/<backup-name>/
├── restic/                       # 파일 백업 (Restic)
│   ├── data/
│   ├── index/
│   └── snapshots/
└── docker-volume/                # Docker 볼륨 (tar.gz)
    ├── pp-data-YYYYMMDD.tar.gz
    ├── pp-security-YYYYMMDD.tar.gz
    └── ...

1. plantpulse-backup Integrated Command

The plantpulse-backup module wraps all backup tools.

Initial Installation (First time only)

cd /opt/kopens/plantpulse-platform/plantpulse-backup/bin
./backup.sh --install

Automatic actions:

pgBackRest configuration (/etc/pgbackrest/pgbackrest.conf)
Medusa configuration (/etc/medusa/medusa.ini)
Directory / Permission creation
pgBackRest stanza creation

PostgreSQL Backup

cd /opt/kopens/plantpulse-platform/plantpulse-backup/bin

./backup.sh --postgres            # Differential (기본, 빠름)
./backup.sh --postgres full       # Full (전체)
./backup.sh --postgres incr       # Incremental (가장 빠름)

Aster TSDB Backup

./backup.sh --cassandra           # Differential
./backup.sh --cassandra full      # Full

File Backup (Restic)

/opt/kopens/plantpulse-platform/plantpulse-backup/etc/run_restic.sh

/data1/pp-data incremental backup
Retention policy: Last 7 days + 1 per month
Automatic integrity check

Backup Cleanup

./backup.sh --purge

pgBackRest: repo1-retention-full (2 sets by default)
Medusa: CASS_PURGE_KEEP_DAYS (10 days by default)
Restic: Automatic based on policy

2. Automatic Backup — Enabled by Default

Nothing to install

Automatic backup is enabled by default. You do not need to run separate commands to install schedules.

Job	Action
`postgres-diff`	PostgreSQL Differential
`postgres-full`	PostgreSQL Full
`cassandra-diff`	Aster TSDB Differential
`cassandra-full`	Aster TSDB Full
`purge`	Retention Policy Cleanup

Not a systemd timer — Changed on 2026-09-08

The datalake image contains neither systemd nor crontab (PID 1 is the entrypoint itself). Schedules are stored as one row per job in the backup_schedule table of the datalake DB, which is read and triggered by the Quartz scheduler inside plantpulse-datalake-admin-api (only in PP_MODE=MASTER).

pd backup schedule                     # 읽기
pd backup schedule set --job <잡> --cron "<초 분 시 일 월 요일>"
pd backup schedule reset [--job <잡>]

Do not search using systemctl list-timers — it will not appear.

:::

Default Schedule

Task	`OnCalendar`	Time
PostgreSQL Diff	`--* 01:30:00`	Daily 01:30
PostgreSQL Full	`Sun --* 01:00:00`	Every Sunday 01:00
Aster TSDB Diff	`--* 01:20:00`	Daily 01:20
Aster TSDB Full	`Sun --* 00:10:00`	Every Sunday 00:10
Purge	`Sun --* 03:00:00`	Every Sunday 03:00

Enabling and Disabling

Write the switch in the host's secret sidecar and restart.

# 호스트에서
sudo vi /etc/kopens/plantpulse-platform.env
#   export PP_BACKUP_SCHEDULE_ENABLED=false

cd /opt/kopens/plantpulse-platform-docker/bin
./restart.sh

«Off» is the third column, not a failure

The timer itself always fires, and the wrapper script decides based on the switch. If off, it skips the backup and records it as result="skipped" in the journal and history.jsonl — not as ok or fail. This allows distinguishing between «off» and «executed» later.

Do not disable it by modifying the unit file directly. It will be silently restored when the next image build reloads the original.

Switches must be written to the sidecar

Writing export PP_BACKUP_SCHEDULE_ENABLED=false only to bin/env.sh will not reach the container. The only part of the switch that the compose file passes to the container is the name; the value comes from the sidecar (or shell export). If written incorrectly, the system will enter a state where it «appears off but continues to run».

3. Docker Volume Backup (Container Environment)

For one-line/Docker installations, additional volume-level backup is possible.

Execute on the host.

cd /opt/kopens/plantpulse-platform-docker/bin

# 기본 세트를 한 번에 (pp-data · pp-security)
./backup.sh

# 볼륨 하나만
./tools/backup-volume.sh pp-data 7 /data1/pp-backup/docker-volume
./tools/backup-volume.sh pp-security 30 /data1/pp-backup/docker-volume

Two items are missing from the default set for specific reasons: pp-backup is the destination of the backup (including itself would double the size every time), and pp-temp is useless for recovery. They can be backed up by specifying them as arguments if needed.

Configuration templates are not volumes

Configuration templates are bind-mounts of the host directory /etc/kopens/conf, so they are not targets for Docker volume backup. Please back them up as host files. The same applies to the secret sidecar /etc/kopens/plantpulse-platform.env.

Argument	Description	Default
1	Volume Name	(Required)
2	Retention Days	7
3	Storage Path	`/data1/pp-backup/docker-volume`

4. Environment Variables (`backup.sh`)

Variable	Default	Description
`BACKUP_ROOT`	`/data1/pp-backup`	Backup Root
`PG_VER`	`18`	PostgreSQL Version
`PG_DATA`	`/data1/pp-data/postgres/data`	PG Data Directory
`CASS_BASE`	`/opt/kopens/plantpulse-platform/plantpulse-storage/db/cassandra`	Aster TSDB Installation Path
`CQL_USER`	`${PP_CASSANDRA_USER}`	CQL User
`CQL_PASS`	`${PP_CASSANDRA_PASSWORD}`	CQL Password
`PG_RETENTION_FULL`	`2`	PG Full Retention
`CASS_PURGE_KEEP_DAYS`	`10`	Medusa Retention Days

5. Off-site Replication

Local backups alone are vulnerable to data center failures. Recommended pattern:

# 매일 새벽 4시 — 외부 S3 로 복제
0 4 * * * rclone sync /data1/pp-backup/ s3:kopens-pp-backup/$(hostname)/ \
  --bwlimit 50M --log-file /var/log/pp-backup-sync.log

# RustFS 객체는 S3 클라이언트로 별도 미러
0 5 * * * rc mirror --overwrite local/ s3-offsite/

6. Recovery

6.1 PostgreSQL Recovery

# 1. plantpulse 중지 (PG 의존 모듈 + PG)
cd /opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin
./stop.sh

# 2. 백업 정보 확인
sudo -u postgres pgbackrest --stanza=pg18 info

# 3. 최신 복구
sudo -u postgres pgbackrest --stanza=pg18 restore

# 3-2. 시점 복구 (PITR)
sudo -u postgres pgbackrest --stanza=pg18 --type=time \
  --target="2026-05-30 12:00:00" restore

# 4. 재시작
./start-daemon.sh
./status.sh

6.2 Aster TSDB Recovery

# 백업 목록
medusa list-backups

# 단일 노드 복구
medusa restore-node --backup-name full_20260530_0010

# 클러스터 전체 복구 (마스터에서)
medusa restore-cluster --backup-name full_20260530_0010

Caution: restore-cluster overwrites data on all nodes. Always verify in a stage environment before executing.

6.3 File Recovery (Restic)

# 스냅샷 목록
restic -r /data1/pp-backup/restic snapshots

# 최신 스냅샷 → 임시 위치 복구
restic -r /data1/pp-backup/restic restore latest --target /data1/pp-data-restored

# 특정 파일만 복구
restic -r /data1/pp-backup/restic restore latest --target / --include /data1/pp-data/cassandra/data/pp/tm_tag_point

6.4 Partial Time Series Recovery (plantpulse-recovery)

To reprocess data for a specific period of a specific asset:

cd /opt/kopens/plantpulse-platform/plantpulse-recovery/bin
./recovery.sh EQUIPMENT_000001 20260501 20260530

6.5 Partial Time Series Extraction (plantpulse-exporter)

Extract to CSV or bulk process with dsbulk:

cd /opt/kopens/plantpulse-platform/plantpulse-exporter/bin

# 에셋 단위 CSV
./export.sh EQUIPMENT_000001 20260101 20260530 /tmp

# 테이블 벌크 언로드 → gz
./bulk-unload.sh pp tm_tag_point
# /data1/pp-backup/cassandra/bulk/tm_tag_point.gz

# 다시 로드
./bulk-load.sh pp tm_tag_point

7. Disaster Recovery (DR) Scenarios

Scenario A: Server Hardware Failure (Cold DR)

Estimated RTO: 4~8 hours (varies by data volume)

Scenario B: Data Corruption (PITR)

# 1. 손상 시점 직전으로 시점 복구
./stop.sh
sudo -u postgres pgbackrest --stanza=pg18 --type=time \
  --target="2026-05-30 14:30:00" restore

# 2. Cassandra 도 동일 시점 가까운 백업으로
medusa restore-cluster --backup-name diff_20260530_0120

# 3. 재시작
./start-daemon.sh
./status.sh

Scenario C: Off-site Failover (Hot DR)

When maintaining a standby cluster at an external site. Pre-design required:

PostgreSQL: Streaming replication or pgBackRest asynchronous replication
Aster TSDB: Multi-DC cluster + automatic hinted handoff
Kafka: MirrorMaker 2 bidirectional replication
RustFS: Site replication or rc mirror --watch

8. Post-Recovery Verification Checklist

cd /opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin

# 1. 전체 모듈 상태
./status.sh

# 2. 헬스 체크
./ops-check.sh

# 3. Cassandra 클러스터 정상
./pd node status

# 4. PostgreSQL 연결
./pd node psql -c "SELECT count(*) FROM site;"

# 5. Kafka 토픽 / Consumer Group
./pd node topic

Browser:

Web console access (http://[HOST]/)
Administrator login (admin / admin123!)
Dashboard load
Alarm list display
OPC connection status CONNECTED
Recent data displayed in time series trends
CEP rule execution confirmed

9. Backup Operation Best Practices

3-2-1 Rule: 3 copies of data, 2 media types, 1 off-site
Monthly Recovery Drill: Attempt actual recovery in stage environment
Backup Integrity Verification: restic check, pgbackrest verify, medusa verify
Backup Monitoring: Notifications for backup failures (Slack, email)
Encryption: TLS for off-site transfer, LUKS/SSE-S3 for storage
Permission Separation: Separate mount for backup disk + read-only mount option
Documentation: Record recovery procedures in the operations team's shared wiki

Technical Support

For backup/recovery inquiries: webmaster@kopens.com

Protection Targets Overview​

Backup Tool Matrix​

Recommended RPO / RTO Targets​

Backup Storage Structure​

1. plantpulse-backup Integrated Command​

Initial Installation (First time only)​

PostgreSQL Backup​

Aster TSDB Backup​

File Backup (Restic)​

Backup Cleanup​

2. Automatic Backup — Enabled by Default​

Default Schedule​

Enabling and Disabling​

3. Docker Volume Backup (Container Environment)​

4. Environment Variables (backup.sh)​

5. Off-site Replication​

6. Recovery​

6.1 PostgreSQL Recovery​

6.2 Aster TSDB Recovery​

6.3 File Recovery (Restic)​

6.4 Partial Time Series Recovery (plantpulse-recovery)​

6.5 Partial Time Series Extraction (plantpulse-exporter)​

7. Disaster Recovery (DR) Scenarios​

Scenario A: Server Hardware Failure (Cold DR)​

Scenario B: Data Corruption (PITR)​

Scenario C: Off-site Failover (Hot DR)​

8. Post-Recovery Verification Checklist​

9. Backup Operation Best Practices​

Related Documents​

Technical Support​