plantpulse-analytics (Analytics Layer)
Role
A big data stack that handles high-volume analytics, SQL queries, and data catalogs. It distributes analytics across Cassandra, PostgreSQL, and Iceberg (MinIO) data. pd In the startup sequence, it is the second gatekeeper.
| Item | Value |
|---|---|
| Module name | plantpulse-analytics |
| Installation path | /opt/kopens/plantpulse-platform/plantpulse-analytics/ |
| Size | Approximately 4.4GB (largest in the platform) |
pd Services | analytics — MASTER: spark-master → hive → gravitino → kyuubi / WORKER: spark-worker → kyuubi |
| Accounts | PP_SPARK_USER / PP_SPARK_PASSWORD (client), PP_HIVE_USER / PP_HIVE_PASSWORD (metastore), PP_GRAVITINO_USER / PP_GRAVITINO_PASSWORD (catalog backend) |
Architecture
Directory Structure
plantpulse-analytics/
├── spark/ # bin/ (spark-submit · beeline · spark-sql) · conf/ (생성물) · logs/ · work/
├── hive/ # bin/ · conf/hive-site.xml (생성물) · logs/
├── kyuubi/ # bin/ · conf/kyuubi-defaults.conf (생성물) · logs/ · work/
├── gravitino/ # bin/ · conf/gravitino-iceberg-rest-server.conf (생성물) · logs/
└── hadoop/ # 라이브러리 — 단일 노드에서는 HDFS 를 띄우지 않습니다
Components
Spark — Distributed Processing Engine
| Item | Value |
|---|---|
| Ports | 7077 (master RPC) · 4440 (master UI) · 8081 (worker UI) |
| Mode | Standalone. Worker nodes register their spark-worker with the master |
| Configuration | spark-defaults.conf.template · spark-env.sh.template · metrics.properties.template |
| Workloads | SQL from Kyuubi sessions, Iceberg archive jobs from the warehouse |
The master UI is at http://<server-ip>:4440/ (not 8080). The Iceberg catalog template is rendered to point to Gravitino's REST endpoint (PP_GRAVITINO_ICEBERG_REST_PORT, 19001).
docker exec -it plantpulse-datalake /opt/kopens/plantpulse-platform/plantpulse-analytics/spark/bin/spark-sql --master spark://127.0.0.1:7077
The Cassandra → Iceberg archiver (plantpulse-warehouse) has been handled by the plantpulse-warehouse container in the platform image since 2026-09-06. This container does not include that module. Monitor archive job status in pd retention lines cold_tier · archive_job → warehouse module
Hive Metastore
| Item | Value |
|---|---|
| Port | 9083 (thrift) |
| Backend | PostgreSQL DB hive-metastore-230 (PP_HIVE_DB), account hive |
| Configuration | hive-site.xml.template (rendered in both hive and spark) |
PP_HIVE_PASSWORD has a value that works both ways — the metastore presents it when connecting to PostgreSQL, and clients validate it as hive-auth.properties when connecting to Kyuubi. That is why passwd.sh handles both account changes and re-rendering.
Kyuubi — SQL Gateway
The gateway through which SQL clients (DBeaver, Tableau, data gateway) connect to Spark via JDBC / Thrift. It provides session isolation and queueing.
| Item | Value |
|---|---|
| Ports | 10000 (thrift) · 10099 (web UI) |
| Configuration | kyuubi-defaults.conf.template |
| Authentication | hive-auth.properties — account PP_SPARK_USER / PP_SPARK_PASSWORD is created by this file |
docker exec -it plantpulse-datalake bash -c \
'/opt/kopens/plantpulse-platform/plantpulse-analytics/spark/bin/beeline -u "jdbc:hive2://127.0.0.1:10000/default" -n "$PP_SPARK_USER" -p "$PP_SPARK_PASSWORD"'
Because Kyuubi is last in the startup sequence, seeing PP_ANALYTICS[KYUUBI] STOPPED in pd status for tens of seconds right after startup is normal.
Gravitino — Iceberg REST Catalog
A catalog server for Iceberg tables. It runs only on the MASTER; Spark on worker nodes points to the master's instance.
| Item | Value |
|---|---|
| Port | 19001 (REST API) — there is no web UI. / returns 404 |
| Backend | PostgreSQL account gravitino (PP_GRAVITINO_PASSWORD) |
| Configuration | gravitino-iceberg-rest-server.conf.template |
Integrated Operations
docker exec plantpulse-datalake pd status analytics
docker exec plantpulse-datalake pd restart analytics # 실행 중인 SQL · Spark 잡이 전부 중단됩니다
docker exec plantpulse-datalake pd logs --lines 100 analytics
If an archive job is running (archive_job in pd retention is NA + «running»), restart after it finishes.
Common Queries
-- Cassandra 직접 조회 (Spark SQL)
SELECT date_format(time, 'yyyy-MM-dd HH:00') AS hour, avg(value) AS avg_value
FROM cassandra.pp.tm_tag_point
WHERE tag_id = 'TAG_001' AND time >= date_sub(current_date(), 1)
GROUP BY 1 ORDER BY 1;
-- Iceberg (콜드 티어)
USE iceberg;
SHOW TABLES;
SELECT count(*), min(time), max(time) FROM tw_tag_point WHERE dt BETWEEN '2026-01-01' AND '2026-05-30';
Logs
docker exec plantpulse-datalake pd logs --list | grep -i "spark\|hive\|kyuubi\|gravitino"
Troubleshooting
| Symptom | Cause | Action |
|---|---|---|
| Spark job OutOfMemory | executor memory exhausted | spark.executor.memory in spark-defaults.conf.template |
| Kyuubi never starts | core count larger than actual | Check if host DOCKER_PP_CLUSTER_CORES is passed to the container — a 16 vCPU box believing it has 30 cores causes this |
| Kyuubi session timeout | idle sessions accumulate | kyuubi.session.idle.timeout |
| Metastore connection failure | PostgreSQL down · hive password mismatch | pd status storage, passwd.sh PP_HIVE_PASSWORD |
| Iceberg small file buildup | optimize not run | warehouse container optimization job → warehouse module |
| Spark UI unreachable | firewall | 4440 |
Tuning Points
| Item | Template | Recommended |
|---|---|---|
| Spark executor memory | spark-defaults.conf.template | 25–50% of host memory |
| Spark cores | PP_CLUSTER_CORES (host DOCKER_PP_CLUSTER_CORES) | host cores − 2 (automatic) |
| Kyuubi engine share level | kyuubi-defaults.conf.template | kyuubi.session.engine.share.level=USER |
For detailed tuning, see Performance Tuning.
Related Documentation
- Data Lake Overview
- warehouse module — Iceberg archival
- plantpulse-data-gateway — SQL console