Skip to main content

Backup and Restore

There are two types of Data Lake backups. Which one you choose depends on what you have lost.

Logical DumpPhysical Backup
Commandpd backup (no arguments)pd backup run …, plus the daily scheduler job
WhatPostgreSQL user databases as SQL files (pg_dump)PostgreSQL via pgBackRest, Aster TSDB via Medusa — the data files themselves
Where/data1/pp-data/postgres/dump/<db>-YYYYmmdd_HHMMSS.sql/data1/pp-backup/ (pgbackrest/ · medusa/) + journal history.jsonl
Restorepd recover <파일> — restores a single pp databasepd backup restore … — the entire data directory
Full / diff · point-in-time recoveryNone (always full)Yes. PostgreSQL supports point-in-time recovery (PITR) via --target

The backups taken by the scheduler at night and those taken manually by an operator via pd backup run travel through the same toolkit (plantpulse-backup) and end up in the same history.jsonl. The backup screen in the admin console invokes the same commands.

Not a systemd timer

The authoritative source for the backup schedule is the backup_schedule table in the datalake database, and the Quartz scheduler inside plantpulse-datalake-admin-api reads and executes it (it runs on the master only).

Unit files are shipped with the image, but none of them are installed — because PID 1 in the container is bash, not systemd, so they could never run anyway. Trying to check via systemctl list-timers only returns «System has not been booted with systemd».

docker exec plantpulse-datalake pd backup schedule --json | head -c 400
# "manager":"quartz","systemctl_present":false,"unit_dir":null,
# "scheduler":{"name":"plantpulse-datalake-admin-api","kind":"quartz","runs_on":"MASTER"}

The place to change it is pd backup schedule set --job <잡> --cron "<Quartz cron>" or the backup screen in the admin console. cron follows the Quartz format (seconds minutes hours day-of-month month day-of-week), so it has six fields.

Automatic backup — already enabled​

JobWhenWhat
postgres-diffDaily 01:30PostgreSQL differential
postgres-fullSunday 01:00PostgreSQL full
cassandra-diffDaily 01:20Aster TSDB differential
cassandra-fullSunday 00:10Aster TSDB full
purgeSunday 03:00Clean up according to the retention policy
docker exec plantpulse-datalake pd backup schedule     # 일정 · 마지막 결과 · 다음 실행
docker exec plantpulse-datalake pd backup list         # 세트 목록
docker exec plantpulse-datalake pd backup status       # 지금 도는 것이 있나

If the LAST column in pd backup schedule shows never, it's never run (normal); if it shows fail (…), that's the error from the last run. For an Aster TSDB set with - for SIZE, the size info is just missing — it's not zero. Do not restore from a set with error and incomplete for STATUS.

Changing the schedule​

docker exec plantpulse-datalake pd backup schedule set --job postgres-diff --calendar "*-*-* 02:45:00"
docker exec plantpulse-datalake pd backup schedule set --job purge --enabled false
docker exec plantpulse-datalake pd backup schedule reset --job postgres-diff     # 그 잡만 기본으로
docker exec plantpulse-datalake pd backup schedule reset                         # 전부 기본으로

The calendar uses systemd syntax — *-*-* 02:45:00 (daily at 02:45), Sun *-*-* 01:00:00 (Sunday at 01:00), Mon..Fri *-*-* 23:30:00 (weekdays). An incorrect expression is caught at validation and nothing is written. The choice remains in /data1/pp-data/backup/schedule.json, so even if the container is recreated, it is applied again at every start.

The global switch wins. If you put PP_BACKUP_SCHEDULE_ENABLED=false in the sidecar, the scheduler only records «skipped» and does nothing, no matter what you've enabled per-job.

Before a major operation — take one more snapshot​

Right before a major operation like an upgrade, restore, or configuration change, take one more snapshot of "this very moment."

docker exec plantpulse-datalake pd backup run --engine postgres --type diff
docker exec plantpulse-datalake pd backup run --engine cassandra --type full
docker exec plantpulse-datalake pd backup                  # 그리고 SQL 사본 하나

Only one runs at a time — if one is in progress, it rejects with exit code 3. Even if the tool exits with 0, if the repository hasn't grown, it is judged as fail.

Restore — from a physical backup​

This is the last resort. All writes since that set are lost. This verb enforces a procedure — if you run it without --yes, it refuses and tells you to run --dry-run first.

# 1. 세트 고르기 — STATUS 가 ok 인 것만
docker exec plantpulse-datalake pd backup list

# 2. 드라이런 — 전제조건 일곱과 «정확히 무슨 명령이 무엇을 덮어쓸지». 아무것도 안 건드린다
docker exec plantpulse-datalake pd backup restore --engine postgres --set 20260903-013040F_20260904-135107D --dry-run
#    preconditions (all pass - ready) 가 나올 때까지 실행하지 않는다

# 3. pp 를 쓰는 서비스를 역순으로 내린다 — storage 는 둔다 (PostgreSQL 은 이 동사가 스스로 멈추고 띄운다)
docker exec plantpulse-datalake pd stop admin-api
docker exec plantpulse-datalake pd stop data-gateway
docker exec plantpulse-datalake pd stop workflow
docker exec plantpulse-datalake pd stop cep
docker exec plantpulse-datalake pd stop timeseries

# 4. 실행
docker exec plantpulse-datalake pd backup restore --engine postgres --set 20260903-013040F_20260904-135107D --yes

# 5. 끝 줄 "done in NNs - restart the dependent services" 를 보고
docker exec plantpulse-datalake pd restart

# 6. 확인
docker exec plantpulse-datalake pd status            # 0 STOPPED
docker exec plantpulse-datalake pd backup status     # 락이 free
  • Point-in-time recovery (PITR) — for PostgreSQL only, you can give a time like --target "2026-09-04 12:00:00". After restoring that set, the WAL is replayed up to that time. If you pass --target to Aster TSDB, it exits with code 2.
  • For Aster TSDB restore, Medusa stops and restarts Aster TSDB itself. If you choose a diff set, it also installs the full backup that it references.
  • It doesn't run on workers. Restore is the master's job; workers follow it.

Restore — from a logical dump​

This restores one pp database to a point from the logical dump. It's irreversible and has no dry run — as soon as you pass the file name, DROP DATABASE "pp" is executed.

ls -l /data1/pp-data/postgres/dump/                            # 날짜와 크기를 먼저 눈으로
docker exec -it plantpulse-datalake pd recover                 # 목록을 찍고 파일 이름을 묻는다 (Ctrl+C 취소)
docker exec plantpulse-datalake pd recover pp-20260904_131500.sql

The procedure is the same as steps 3–6 of a physical restore (stop the service first, start it again after it's done). Even if you load a dump from another database (temporal · hive · kestra), it goes into pp — those databases are the ones that need physical backups.

Do not​

  • Don't run --yes without --dry-run. A restore you haven't inspected the plan for is a restore where you don't know which point in time you're going back to.
  • Don't restore over a running service. Services that were connected hold on to the broken session and spew errors.
  • Don't manually delete the lock file (/data1/pp-backup/pd-backup.lock). A lock from a dead PID is read as free by pd.
  • Don't copy the dump to /data1/pp-temp. pd clean will delete it.