DataLake Overview
DataLake is the infrastructure layer of the PlantPulse platform. Everything that stores and streams data — databases, message brokers, analytics engines, time-series engines, workflow engines — lives here, and applications such as the web console, batch jobs, and warehouses sit on top of it.
- Where are the configuration files? — what to change so it survives a restart
- How to change settings — values · node addresses · templates · memory
- Changing passwords and API keys — one command for
passwd.sh - Admin console — view status · logs · backups in the browser
- Data queries — choose between PostgreSQL · CQL · Spark SQL
- Time-series database — a general-purpose time-series DB for sensor values. Time-series functions · compression · full-text search
- Time-series engine — a metrics-only engine (TSE). TQL · Prometheus · PlantPulse Dashboard
- CEP — judges streaming events in real time. EQL · backends · event types
- Data gateway — the single door where external systems, BI, and AI get in
- FAQ
One Container, Eight Services
The DataLake runs as a single plantpulse-datalake container on the host. Inside it, eight services start in a fixed order. The authoritative sequence is a single file, plantpulse-datalake-cli/services/order.txt, inside the container.
storage → analytics → messaging → timeseries → cep → workflow → data-gateway → admin-api
| Order | Service | Components | Ports | What it does |
|---|---|---|---|---|
| 1 | storage | Valkey · PostgreSQL · Aster TSDB · RustFS | 6379 · 5432 · 9042 · 9000 | Cache · metadata · time series · object storage |
| 2 | analytics | Spark · Hive · Gravitino · Kyuubi · Flink(+ SQL Gateway) | 7077/4440 · 9083 · 19001 · 10000 · 8181 · 8083 | Distributed analytics and SQL gateway |
| 3 | messaging | Kafka · HiveMQ(MQTT) | 9092 · 1883 | Field data collection and inter-module event delivery |
| 4 | timeseries | Time-series engine · PlantPulse Dashboard | 7800 · 3000 | Metric storage · query · dashboards (sensor values live on 9042 of storage) |
| 5 | cep | CEP server | 7400 | Complex event processing — EQL registration · validation · console, and the engine |
| 6 | workflow | Temporal · Kestra | 7233/8233 · 8380 | Long-running jobs and data pipelines |
| 7 | data-gateway | Data gateway | 5500 | The front door for external systems to query data (including SQL console) |
| 8 | admin-api | Admin console backend | 4949 · 4950 | Operational console and health API for this node |
storage and analytics are the gatekeepers. If these two do not come up, the services behind them do not even attempt to start. The rest, if they fail to start, log a warning and move on.
Where You Do What
There are three places you work with the DataLake. Once you know what each place is for, the rest of this section becomes reference material.
| Place | Command · Screen | What you do |
|---|---|---|
Host (/opt/kopens/plantpulse-platform-docker/bin/) | up.sh down.sh restart.sh restart-datalake.sh status.sh logs.sh shell.sh passwd.sh | Start and stop the container, change passwords, edit configuration files. The operator's first entry point |
Inside the container (docker exec plantpulse-datalake pd …) | pd status pd doctor pd logs pd config diff pd restart <서비스> pd backup | Restart a single service, verify that settings rendered correctly, take a backup |
Browser (https://<server-ip>:4950/) | Admin console | View status · metrics · logs · diagnostics · backups · configuration diff on screen. You can also run the pd verbs allowed by the command screen |
Every configuration file inside the container (/opt/kopens/plantpulse-platform/<module>/conf/…) is a generated artifact. Anything you edit there is gone on the next startup. The source of truth for values is the set of files under /etc/kopens/ on the host; for details, see Where are the configuration files?.
First Five Minutes — How to Confirm It Is Healthy
The commands below are all read-only. They change nothing, so you can run them any time.
# 1. 호스트에서 — 컨테이너와 헬스 요약. 종료 코드 0 = 정상, 2 = 비정상
cd /opt/kopens/plantpulse-platform-docker/bin
./status.sh
# 2. 컨테이너 안 — 서비스별 포트 표. 마지막 줄 "0 STOPPED" 면 정상
docker exec plantpulse-datalake pd status
# 3. 컨테이너 안 — 환경이 조용히 어긋난 곳은 없나. 마지막 줄 "FAIL 0" 이면 정상
docker exec plantpulse-datalake pd doctor
# 4. 브라우저 — 관리 콘솔
# https://<server-ip>:4950/?lang=ko
On pd status, seeing a single STOPPED for the first few dozen seconds after startup is normal. Kyuubi inside analytics starts last (measured 54 seconds). Check again after one minute.
What This Section Contains
| Page | When to read it |
|---|---|
| Path map | When you ask "where is that file" — host · container · volumes · logs |
| Configuration | Configuration file locations, change procedures, variable reference |
| Credentials | Service passwords · API key rotation, web login accounts |
| Admin console | Console access · login · screen layout |
pd CLI | The full set of operational commands inside the container |
| Operations | Start · stop · restart, logs · health, backup · restore |
| CEP | Write and run real-time rules with EQL — language · event types · operations · limits |
| Data gateway | Four languages behind one REST — API · controls and limits |
| Components | Details for storage · messaging · analytics · time series · CEP · workflow · gateway |
| FAQ | Frequently encountered symptoms and answers |
Related Documents
- System startup and shutdown — starting and stopping the entire stack
- Port and service management — complete list of public ports and firewall
- Backup and recovery — backup tools and disaster recovery scenarios
- Changing initial passwords — commissioning checklist