Performance Tuning

Overview

PlantPulse performance improves only when bottlenecks in each layer, collection → processing → storage, are removed one by one. The key is not to blindly increase every value, but to monitor the metrics and tune only the bottleneck.

3 Tuning Principles

1. Change one thing at a time: If you change multiple variables at once, you cannot determine which change had the effect.
2. Base decisions on operational data: Reproduce actual traffic in a stage environment before making changes.
3. Record everything: Document changed values and before/after metrics in your GitOps / operations wiki.

Recommended Starting Points by Environment

Environment	CPU	Memory	Disk	Recommended PIPELINE_THREADS	Recommended PIPELINE_RATELIMIT
Development	16 vCPU	64 GB	SSD	16	10,000 MPS
Standard	32 vCPU	128 GB	NVMe	36	40,000 MPS
Large-scale	48+ vCPU	200+ GB	NVMe	64	100,000 MPS

Key Tuning Variables Based on env.sh

The variables most frequently adjusted in operational environments are managed in the host's bin/env.sh and compose/docker-compose.yml → Environment Variable Reference

Variable	Default	Impact
PP_CLUSTER_CORES	30	Spark / analytics core allocation
PP_CLUSTER_MEMORY_BY_CORE	2G	Spark executor memory/cores
PP_LANG / PP_TZ	ko / Asia/Seoul	JVM locale / timezone
JAVA_TOOL_OPTIONS	Auto-generated	Automatically applied to all child JVMs

For detailed variables, refer to the env.sh variable index in the Properties Reference.

Measurement Tools

Tools for comparing before and after tuning:

cd /opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin

./status.sh                            # 모듈별 CPU / 메모리(PSS) 요약
./ops-check.sh                         # 헬스 + critical log
./memory-check.sh                      # 모듈별 메모리 상세
./memory-rank.sh                       # 메모리 순위
./pd node tpstats                      # Cassandra 스레드풀
./pd node compactionstats              # Cassandra 컴팩션
./pd node table-stats                  # 테이블별 통계
./pd node proxyhistograms              # Cassandra 지연 히스토그램

# Kafka consumer lag
./pd node topic

# JVM Heap / GC (컨테이너 진입 후)
./shell.sh
jcmd <pid> GC.heap_info
jcmd <pid> GC.heap_dump /tmp/heap.hprof

For external monitoring (Grafana / Prometheus), refer to the System Monitoring page.

Note: Optimal performance tuning values can vary depending on server specifications, data volume, and usage patterns. We recommend starting with the recommended settings below and adjusting gradually while monitoring the results.

Engine Pipeline Tuning

Configuration file: plantpulse-server/config/plantpulse-engine.properties

Note: The engine pipeline is the critical path through which sensor-collected data is processed. These settings allow you to adjust the data processing speed and capacity.

Core Pipeline Parameters

Parameter	Default	Description	Tuning Guide
engine.pipeline.threads	36	Number of pipeline worker threads	Recommended 70–80% of host core count (48 cores in validation environment → 36)
engine.pipeline.ratelimit	40000	Maximum messages per second (MPS)	Adjust according to server specifications
engine.pipeline.queue.size	1200000	Maximum pipeline queue size	Adjust according to available memory
engine.pipeline.queue.o3.delay	50	Out-of-Order delay processing (ms)	Adjust according to network latency
engine.pipeline.task.mode	SINGLE	Pipeline task mode	SINGLE or MULTI
engine.async.parallelism	256	Number of asynchronous parallel tasks	Recommended 8–16 times the number of CPU cores

Streaming Settings

Parameter	Default	Description
engine.stream.processor	DIRECT	Stream processing mode
engine.streaming.messaging.warning.ms	5000	Message delay warning threshold (ms)
engine.streaming.messaging.timeout.ms	10000	Message timeout (ms)
engine.streaming.messaging.timeout.store.type	FILE_QUEUE	Timeout message storage method
engine.streaming.messaging.timeout.recovery.type	DB	Timeout recovery method

Worker Threads

Parameter	Default	Description
engine.job.thread.asset	8	Asset worker thread count
engine.job.thread.point	8	Point worker thread count

Recommended Settings by Specification

The table below shows recommended settings based on server specifications. Please refer to the column matching your operational environment's specifications.

Item	Small-scale (8 cores/64GB)	Medium-scale (16 cores/128GB)	Large-scale (32 cores/256GB)
pipeline.threads	16	36	64
pipeline.ratelimit	10,000	40,000	100,000
pipeline.queue.size	300,000	1,200,000	3,000,000
async.parallelism	64	256	512

---

Aster TSDB Tuning

Configuration file: plantpulse-storage/db/cassandra/conf/cassandra.yaml

Note: Aster TSDB is the core database for storing time series data. As the data volume grows, Aster TSDB tuning has a significant impact on overall performance, so we recommend adjusting it for your operational environment.

Memory Settings

Parameter	Default	Description
memtable_heap_space	2GiB	Memtable heap memory
memtable_offheap_space	2GiB	Memtable off-heap memory
memtable_allocation_type	offheap_objects	Memory allocation type
memtable_flush_writers	8	Memtable flush thread count

Concurrency Settings

Parameter	Default	Description	Tuning Guide
concurrent_reads	256	Concurrent read threads	Recommended disk count x 16
concurrent_writes	256	Concurrent write threads	Recommended CPU cores x 8
concurrent_counter_writes	256	Concurrent counter writes	Set the same as concurrent_writes
concurrent_compactors	8	Concurrent compaction threads	Recommended 1/4 of CPU core count

Commit Log

Parameter	Default	Description
commitlog_sync	periodic	Synchronous mode
commitlog_sync_period	10000ms	Synchronous period
commitlog_segment_size	320MiB	Segment size
commitlog_total_space	32GiB	Total commit log space

Timeouts

Parameter	Default	Description
read_request_timeout	10000ms	Read timeout
write_request_timeout	60000ms	Write timeout
counter_write_request_timeout	10000ms	Counter write timeout
cas_contention_timeout	1000ms	CAS contention timeout
truncate_request_timeout	600000ms	Truncate timeout

Compaction Strategy

PlantPulse uses the UCS (Unified Compaction Strategy) designed for time series data characteristics.

Note: Compaction is the process by which Aster TSDB consolidates and merges data files (SSTables) stored on disk. Using an appropriate compaction strategy can save disk space and improve read performance.

storage.properties Settings:

Parameter	Default	Description
storage.table.compaction.strategy	UCS	Compaction strategy (UCS or TWCS)
storage.table.compaction.strategy.ucs.scailing_parameter	T8	Scaling parameter
storage.table.compaction.strategy.ucs.min_sstable_size	128MiB	Minimum SSTable size
storage.table.compaction.strategy.ucs.target_sstable_size	512MiB	Target SSTable size
storage.table.compaction.strategy.ucs.base_shard_count	8	Number of base shards
storage.table.compaction.strategy.ucs.max_sstables_to_compact	6	Maximum SSTables for compaction

Compression

Uses ZStandard dictionary compression:

Parameter	Default	Description
storage.compression.zstd.type	ZstdDictionaryCompressor	Compression method
storage.compression.zstd.level	3	Compression level (1–22; higher levels increase compression ratio but slow down speed)

TTL (Data Retention Period)

Set the TTL for each table in storage.properties. Once the TTL expires, the corresponding data is automatically deleted, so set it according to your data retention requirements.

Parameter	Default	Description
storage.tag.point.ttl	62 (days)	Tag point source data
storage.tag.point.map.ttl	1 (days)	Tag point map
storage.tag.point.sampling.ttl	93 (days)	Sampling data
storage.tag.point.snapshot.ttl	93 (days)	Snapshot data
storage.tag.point.aggregation.ttl	93 (days)	Aggregation data
storage.tag.point.archive.ttl	365 (days)	Archive data
storage.tag.blob.ttl	93 (days)	BLOB data
storage.asset.data.ttl	10 (days)	Asset data
storage.asset.data.sampling.ttl	31 (days)	Asset sampling data

Aster TSDB Performance Inspection Commands

You can inspect the performance status of Aster TSDB using the following commands:

# 컴팩션 상태 확인
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node compactionstats

# 테이블별 통계 확인
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node table-stats

# 테이블 히스토그램 (읽기/쓰기 지연)
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node table-histograms

# SSTable 크기 확인
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node sstable-size

# 스레드풀 통계
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node tpstats

---

Spark Tuning

Configuration file: plantpulse-analytics/spark/conf/spark-defaults.conf

Note: Apache Spark is a distributed processing engine for large-scale data analysis. Please refer to the settings below to improve the speed and stability of analytical jobs.

Resource Allocation

Parameter	Default	Description
spark.cores.max	16	Maximum Spark cores
spark.driver.cores	2	Driver core count
spark.driver.memory	4g	Driver memory
spark.executor.cores	1	Executor core count
spark.executor.memory	2G	Executor memory
spark.executor.memoryOverhead	2g	Executor overhead memory

Dynamic Allocation

Parameter	Default	Description
spark.dynamicAllocation.enabled	true	Dynamic executor allocation
spark.dynamicAllocation.minExecutors	2	Minimum executor count
spark.dynamicAllocation.maxExecutors	16	Maximum executor count
spark.dynamicAllocation.executorIdleTimeout	30	Idle executor release time (seconds)

Shuffle and Parallelism

Parameter	Default	Description
spark.default.parallelism	64	Default parallelism
spark.sql.shuffle.partitions	128	SQL shuffle partition count
spark.sql.adaptive.enabled	true	Adaptive Query Execution
spark.sql.adaptive.coalescePartitions.enabled	true	Automatic partition coalescing

JVM Settings

Both Spark driver and executor use ZGC:

-XX:+UseZGC
-XX:+AlwaysPreTouch
-XX:+UseTLAB
-XX:+ResizeTLAB
-XX:+DisableExplicitGC

S3 (RustFS) Integration Settings

Parameter	Default	Description
spark.hadoop.fs.s3a.connection.maximum	400	Maximum S3 connections
spark.hadoop.fs.s3a.threads.max	128	Maximum S3 threads
spark.hadoop.fs.s3a.max.total.tasks	1024	Maximum async operations
spark.hadoop.fs.s3a.fast.upload	true	Fast upload enablement
spark.hadoop.fs.s3a.fast.upload.buffer	disk	Upload buffer type
spark.hadoop.fs.s3a.block.size	128MB	S3 block size

Iceberg Settings

Parameter	Default	Description
spark.sql.iceberg.write.target-file-size-bytes	256MB	Target file size
spark.sql.iceberg.column-stats.enabled	true	Column statistics enablement
spark.sql.iceberg.vectorization.enabled	true	Vectorized read enablement
spark.sql.iceberg.manifest.merge.enabled	true	Manifest merging
spark.sql.parquet.compression.codec	zstd	Parquet compression codec

---

Kafka Tuning

Configuration file: plantpulse-messaging/kafka/config/server.properties

Note: Kafka is a message broker that delivers sensor data in real time. As data volume grows, you can adjust Kafka settings to improve processing performance.

Core Settings

Parameter	Default	Description
num.network.threads	3	Network processing threads
num.io.threads	8	I/O processing threads
socket.send.buffer.bytes	102400	Socket send buffer
socket.receive.buffer.bytes	102400	Socket receive buffer
socket.request.max.bytes	104857600	Maximum request size (100MB)
num.partitions	1	Default partition count
log.retention.hours	168	Log retention period (7 days)
log.segment.bytes	1073741824	Log segment size (1GB)

Tuning for High-Volume Processing

If you process large volumes of IoT data, please refer to the settings below and adjust accordingly:

# 파티션 수 증가 (메시지 병렬 처리)
num.partitions=4

# I/O 스레드 증가
num.io.threads=16

# 네트워크 스레드 증가
num.network.threads=8

# 소켓 버퍼 증가
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576

---

Messaging Settings

Configuration file: plantpulse-server/config/plantpulse-mq.properties

Kafka DDS Topics

Parameter	Default	Purpose
mq.dds.kafka.topic.tag.point	pp-tag-point	Tag point values
mq.dds.kafka.topic.tag.alarm	pp-tag-alarm	Tag alarms
mq.dds.kafka.topic.asset.data	pp-asset-data	Asset data
mq.dds.kafka.topic.asset.alarm	pp-asset-alarm	Asset alarms
mq.dds.kafka.topic.asset.event	pp-asset-event	Asset events
mq.dds.kafka.topic.asset.aggregation	pp-asset-aggregation	Asset aggregation
mq.dds.kafka.topic.domain.changed.event	pp-domain-changed-event	Metadata changes

MQTT DDS Topics

Parameter	Default	Purpose
mq.dds.mqtt.topic.tag.point	tag/point/{tag_id}	Point values per tag
mq.dds.mqtt.topic.tag.alarm	tag/alarm/{tag_id}	Alarms per tag
mq.dds.mqtt.topic.asset.data	asset/data/{asset_id}	Data per asset
mq.dds.mqtt.topic.asset.command	asset/command/{asset_id}	Asset control commands

---

Scheduler Settings

Configuration file: quartz.properties (web app internal classpath resource — not a candidate for externalization)

Parameter	Default	Description
org.quartz.threadPool.threadCount	8	Scheduler thread count
org.quartz.threadPool.threadPriority	5	Thread priority

---

Mail / Notification Settings

Configuration location: PostgreSQL pp.mm_config table. On 2026-09-17, plantpulse-mail.properties was removed, and the path that read the old copy remaining in the external conf was also removed on 2026-09-18 — values enter only via the console or pp.mm_config. Key names and defaults remain unchanged.

Parameter	Default	Description
mail.smtp.host	192.168.0.41	SMTP server address
mail.smtp.port	25	SMTP port
mail.smtp.auth	false	Authentication usage
mail.smtp.starttls.enable	false	TLS usage

---

Application Settings

The application.properties file was removed in 2026.06. Console behavior settings such as theme and homepage are managed in the console's System > Settings Management (PostgreSQL mm_config table), and alarm.duplicate.check.minutes(duplicate alarm check interval, minutes) has been migrated to plantpulse-server/config/plantpulse-engine.properties.

---

Performance Bottleneck Diagnosis

When performance issues occur, please refer to the diagnostic methods and solutions below for each symptom.

1. Pipeline Queue Backlog

Symptoms: The pipeline wait queue persistently increases

# PlantPulse Dashboard 에서 "파이프라인 대기 큐" 패널을 확인해 주세요
# 또는 JMX: pipeline.queue.size 값을 확인해 주세요

Solutions:

• Increase the engine.pipeline.threads value
• Adjust the engine.pipeline.ratelimit value upward
• Check Aster TSDB write performance

2. Aster TSDB Write Latency

Symptoms: Write timeouts or WriteTimeoutException occur

# 컴팩션 상태 확인
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node compactionstats

# Memtable 플러시
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node flush

Solutions:

• Increase the concurrent_writes value
• Increase the compaction_throughput value
• Check disk I/O performance (SSD usage is recommended)

3. Insufficient Memory

Symptoms: Frequent JVM GC or OutOfMemoryError occurrences

Solutions:

• In bin/start.sh of the service module, increase the -Xmx value
• Reduce the engine.pipeline.queue.size value to conserve memory usage
• Disable unnecessary caches

4. Network Latency

Symptoms: Increased data reception delays (verifiable in the PlantPulse Dashboard "Network Latency" panel)

Solutions:

• Check network status between the OPC server and PlantPulse
• Adjust the engine.streaming.messaging.timeout.ms value
• Check the MQTT QoS level

---

Technical Support

For performance tuning inquiries: webmaster@kopens.com