plantpulse-analytics (Analytics Layer)
Role
This is the big data stack responsible for large-scale analytics, SQL queries, and the data catalog. It performs distributed analysis of Aster TSDB, PostgreSQL, and Iceberg (RustFS) data. It is the second gatekeeper in the pd startup sequence.
| Item | Value |
|---|---|
| Module name | plantpulse-analytics |
| Installation path | /opt/kopens/plantpulse-platform/plantpulse-analytics/ |
| Size | Approx. 4.4GB (largest of the platform) |
pd services | analytics — MASTER: spark-master → hive → gravitino → kyuubi / WORKER: spark-worker → kyuubi |
| Accounts | PP_SPARK_USER / PP_SPARK_PASSWORD (client), PP_HIVE_USER / PP_HIVE_PASSWORD (metastore), PP_GRAVITINO_USER / PP_GRAVITINO_PASSWORD (catalog backend) |
Configuration
Directory Structure
plantpulse-analytics/
├── spark/ # bin/ (spark-submit · beeline · spark-sql) · conf/ (생성물) · logs/ · work/
├── hive/ # bin/ · conf/hive-site.xml (생성물) · logs/
├── kyuubi/ # bin/ · conf/kyuubi-defaults.conf (생성물) · logs/ · work/
├── gravitino/ # bin/ · conf/gravitino-iceberg-rest-server.conf (생성물) · logs/
└── hadoop/ # 라이브러리 — 단일 노드에서는 HDFS 를 띄우지 않습니다
Components
Spark — Distributed Processing Engine
| Item | Value |
|---|---|
| Ports | 7077 (Master RPC) · 4440 (Master UI) · 8081 (Worker UI) |
| Mode | Standalone. When a worker node connects, its spark-worker registers with the master |
| Configuration | spark-defaults.conf.template · spark-env.sh.template · metrics.properties.template |
| Workloads | SQL from Kyuubi sessions, Iceberg archive jobs in the warehouse |
The Master UI is at http://<server-ip>:4440/ (not 8080). The template renders the Iceberg catalog to point at Gravitino's REST endpoint (PP_GRAVITINO_ICEBERG_REST_PORT, 19001).
docker exec -it plantpulse-datalake /opt/kopens/plantpulse-platform/plantpulse-analytics/spark/bin/spark-sql --master spark://127.0.0.1:7077
The Aster TSDB → Iceberg archiver (plantpulse-warehouse-web) has been running in the plantpulse-warehouse-web container of the platform image since 2026-09-06. This module is not in that container. Check archive job status in pd retention, lines cold_tier and archive_job → warehouse module
Hive Metastore
| Item | Value |
|---|---|
| Ports | 9083 (thrift) |
| Backend | PostgreSQL DB hive-metastore-230 (PP_HIVE_DB), account hive |
| Configuration | hive-site.xml.template (rendered in both hive and spark) |
PP_HIVE_PASSWORD is a single value that works in both directions — it's presented when the metastore connects to PostgreSQL, and validated as hive-auth.properties when clients connect through Kyuubi. Therefore, passwd.sh handles both account changes and re-rendering.
Kyuubi — SQL Gateway
This is the entry point for SQL clients (DBeaver, Tableau, Data Gateway) to access Spark via JDBC/Thrift. It provides session isolation and queuing.
| Item | Value |
|---|---|
| Ports | 10000 (thrift) · 10099 (web UI) |
| Configuration | kyuubi-defaults.conf.template |
| Authentication | hive-auth.properties — this file creates the PP_SPARK_USER / PP_SPARK_PASSWORD accounts |
docker exec -it plantpulse-datalake bash -c \
'/opt/kopens/plantpulse-platform/plantpulse-analytics/spark/bin/beeline -u "jdbc:hive2://127.0.0.1:10000/default" -n "$PP_SPARK_USER" -p "$PP_SPARK_PASSWORD"'
Since Kyuubi starts last in the startup sequence, it's normal to see PP_ANALYTICS[KYUUBI] STOPPED in pd status for a few tens of seconds right after startup.
Gravitino — Iceberg REST Catalog
This is the catalog server for Iceberg tables. It runs only on MASTER; Spark on workers points to the master instance.
| Item | Value |
|---|---|
| Ports | 19001 (REST API) — there is no web UI. / responds with 404 |
| Backend | PostgreSQL account gravitino (PP_GRAVITINO_PASSWORD) |
| Configuration | gravitino-iceberg-rest-server.conf.template |
Integrated Operations
docker exec plantpulse-datalake pd status analytics
docker exec plantpulse-datalake pd restart analytics # 실행 중인 SQL · Spark 잡이 전부 중단됩니다
docker exec plantpulse-datalake pd logs --lines 100 analytics
If an archive job is running (when archive_job in pd retention is NA + «running»), restart after it finishes.
Frequently Used Queries
-- Cassandra 직접 조회 (Spark SQL)
SELECT date_format(time, 'yyyy-MM-dd HH:00') AS hour, avg(value) AS avg_value
FROM cassandra.pp.tm_tag_point
WHERE tag_id = 'TAG_001' AND time >= date_sub(current_date(), 1)
GROUP BY 1 ORDER BY 1;
-- Iceberg (콜드 티어)
USE iceberg;
SHOW TABLES;
SELECT count(*), min(time), max(time) FROM tw_tag_point WHERE dt BETWEEN '2026-01-01' AND '2026-05-30';
Logs
docker exec plantpulse-datalake pd logs --list | grep -i "spark\|hive\|kyuubi\|gravitino"
Common Problems
| Symptom | Cause | Action |
|---|---|---|
| Spark job OutOfMemory | Insufficient executor memory | spark.executor.memory in spark-defaults.conf.template |
| Kyuubi never starts | Core count set higher than actual | Check whether the host's DOCKER_PP_CLUSTER_CORES is passed to the container — this happens when a 16-vCPU box is told it has 30 cores |
| Kyuubi session timeout | Idle session accumulation | kyuubi.session.idle.timeout |
| Metastore connection failure | PostgreSQL down · hive password mismatch | pd status storage, passwd.sh PP_HIVE_PASSWORD |
| Iceberg small-file buildup | optimize not run | Run optimization job in the warehouse container → warehouse module |
| Can't access Spark UI | Firewall | 4440 |
Tuning Points
| Item | Template | Recommended |
|---|---|---|
| Spark executor memory | spark-defaults.conf.template | 25–50% of host memory |
| Spark cores | PP_CLUSTER_CORES (host DOCKER_PP_CLUSTER_CORES) | Host cores − 2 (automatic) |
| Kyuubi engine sharing level | kyuubi-defaults.conf.template | kyuubi.session.engine.share.level=USER |
For more details on tuning, see Performance Tuning.
Related Documents
- Datalake Overview
- warehouse module — Iceberg archive
- plantpulse-data-gateway — SQL console