Performance Tuning

Overview

PlantPulse performance improves by eliminating bottlenecks sequentially through each layer: collection → processing → storage. The key is not to blindly increase all values, but to monitor metrics and adjust only the bottleneck in focus.

Three tuning principles

1. One variable at a time: If you change multiple parameters simultaneously, you cannot tell which change had an effect.
2. Based on operational data: Make changes in a stage environment after reproducing real traffic.
3. Keep records: Document changed values and before/after metrics in GitOps or your operations wiki.

Recommended starting points by deployment size

Environment	CPU	Memory	Disk	Recommended PIPELINE_THREADS	Recommended PIPELINE_RATELIMIT
Development	16 vCPU	64 GB	SSD	16	10,000 MPS
Standard	32 vCPU	128 GB	NVMe	36	40,000 MPS
Large scale	48+ vCPU	200+ GB	NVMe	64	100,000 MPS

Core tuning variables in env.sh

The most frequently adjusted variables in production are managed in the host's bin/env.sh and compose/docker-compose.yml → Environment Variable Reference

Variable	Default	Impact
PP_CLUSTER_CORES	30	Spark / analytics core allocation
PP_CLUSTER_MEMORY_BY_CORE	2G	Spark executor memory/core
PP_LANG / PP_TZ	ko / Asia/Seoul	JVM locale / timezone
JAVA_TOOL_OPTIONS	Auto-generated	Automatically applied to all child JVMs

For detailed variables, see the env.sh variable index in Properties Reference.

Measurement tools

Tools to compare before and after tuning:

cd /opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin

./status.sh                            # 모듈별 CPU / 메모리(PSS) 요약
./ops-check.sh                         # 헬스 + critical log
./memory-check.sh                      # 모듈별 메모리 상세
./memory-rank.sh                       # 메모리 순위
./pd node tpstats                      # Cassandra 스레드풀
./pd node compactionstats              # Cassandra 컴팩션
./pd node table-stats                  # 테이블별 통계
./pd node proxyhistograms              # Cassandra 지연 히스토그램

# Kafka consumer lag
./pd node topic

# JVM Heap / GC (컨테이너 진입 후)
./shell.sh
jcmd <pid> GC.heap_info
jcmd <pid> GC.heap_dump /tmp/heap.hprof

For external monitoring (Grafana / Prometheus), see the System Monitoring page.

Note: Optimal tuning values depend on server specifications, data volume, and usage patterns. We recommend starting with the recommended settings below and adjusting gradually while monitoring results.

Engine Pipeline Tuning

Configuration file: plantpulse-server/config/plantpulse-engine.properties

Note: The engine pipeline is the critical path where data collected from sensors is processed. These settings control the processing speed and capacity of data.

Pipeline core parameters

Parameter	Default	Description	Tuning guide
engine.pipeline.threads	36	Number of pipeline worker threads	70–80% of host cores recommended (example: 48 cores → 36)
engine.pipeline.ratelimit	40000	Maximum messages per second (MPS)	Adjust to match server capacity
engine.pipeline.queue.size	1200000	Maximum pipeline queue size	Adjust to available memory
engine.pipeline.queue.o3.delay	50	Out-of-Order delay handling (ms)	Adjust based on network latency
engine.pipeline.task.mode	SINGLE	Pipeline task mode	SINGLE or MULTI
engine.async.parallelism	256	Async parallelism	8–16× CPU cores recommended

Streaming configuration

Parameter	Default	Description
engine.stream.processor	DIRECT	Stream processing mode
engine.streaming.messaging.warning.ms	5000	Message delay alert threshold (ms)
engine.streaming.messaging.timeout.ms	10000	Message timeout (ms)
engine.streaming.messaging.timeout.store.type	FILE_QUEUE	Timeout message storage method
engine.streaming.messaging.timeout.recovery.type	DB	Timeout recovery method

Work threads

Parameter	Default	Description
engine.job.thread.asset	8	Asset work thread count
engine.job.thread.point	8	Point work thread count

Recommended settings by server capacity

The table below shows recommended configuration values based on server specifications. Refer to the column matching your production environment.

Item	Small (8 cores/64GB)	Medium (16 cores/128GB)	Large (32 cores/256GB)
pipeline.threads	16	36	64
pipeline.ratelimit	10,000	40,000	100,000
pipeline.queue.size	300,000	1,200,000	3,000,000
async.parallelism	64	256	512

---

Cassandra Tuning

Configuration file: plantpulse-storage/db/cassandra/conf/cassandra.yaml

Note: Cassandra is the core database for storing time series data. As data volume grows, Cassandra tuning has an increasing impact on overall performance, so it is recommended to adjust these settings for your production environment.

Memory settings

Parameter	Default	Description
memtable_heap_space	2GiB	Memtable heap memory
memtable_offheap_space	2GiB	Memtable Off-Heap memory
memtable_allocation_type	offheap_objects	Memory allocation type
memtable_flush_writers	8	Memtable flush thread count

Concurrency settings

Parameter	Default	Description	Tuning guide
concurrent_reads	256	Concurrent read threads	Disk count × 16 recommended
concurrent_writes	256	Concurrent write threads	CPU cores × 8 recommended
concurrent_counter_writes	256	Concurrent counter writes	Set equal to concurrent_writes
concurrent_compactors	8	Concurrent compaction threads	CPU core count ÷ 4 recommended

Commit log

Parameter	Default	Description
commitlog_sync	periodic	Sync mode
commitlog_sync_period	10000ms	Sync period
commitlog_segment_size	320MiB	Segment size
commitlog_total_space	32GiB	Total commit log space

Timeouts

Parameter	Default	Description
read_request_timeout	10000ms	Read timeout
write_request_timeout	60000ms	Write timeout
counter_write_request_timeout	10000ms	Counter write timeout
cas_contention_timeout	1000ms	CAS contention timeout
truncate_request_timeout	600000ms	Truncate timeout

Compaction strategy

PlantPulse uses UCS (Unified Compaction Strategy) optimized for time series data characteristics.

Note: Compaction is the process by which Cassandra organizes and merges data files (SSTables) stored on disk. Using an appropriate compaction strategy saves disk space and improves read performance.

storage.properties configuration:

Parameter	Default	Description
storage.table.compaction.strategy	UCS	Compaction strategy (UCS or TWCS)
storage.table.compaction.strategy.ucs.scailing_parameter	T8	Scaling parameter
storage.table.compaction.strategy.ucs.min_sstable_size	128MiB	Minimum SSTable size
storage.table.compaction.strategy.ucs.target_sstable_size	512MiB	Target SSTable size
storage.table.compaction.strategy.ucs.base_shard_count	8	Base shard count
storage.table.compaction.strategy.ucs.max_sstables_to_compact	6	Maximum compaction SSTables

Compression

ZStandard dictionary compression is used:

Parameter	Default	Description
storage.compression.zstd.type	ZstdDictionaryCompressor	Compression codec
storage.compression.zstd.level	3	Compression level (1–22; higher = better compression, slower speed)

TTL (Data retention period)

TTL for each table is configured in storage.properties. Data is automatically deleted when TTL expires, so set it according to your data retention requirements.

Parameter	Default	Description
storage.tag.point.ttl	62 (days)	Tag point raw data
storage.tag.point.map.ttl	1 (days)	Tag point map
storage.tag.point.sampling.ttl	93 (days)	Sampling data
storage.tag.point.snapshot.ttl	93 (days)	Snapshot data
storage.tag.point.aggregation.ttl	93 (days)	Aggregate data
storage.tag.point.archive.ttl	365 (days)	Archive data
storage.tag.blob.ttl	93 (days)	BLOB data
storage.asset.data.ttl	10 (days)	Asset data
storage.asset.data.sampling.ttl	31 (days)	Asset sampling data

Cassandra performance inspection commands

You can check Cassandra performance status using the following commands:

# 컴팩션 상태 확인
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node compactionstats

# 테이블별 통계 확인
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node table-stats

# 테이블 히스토그램 (읽기/쓰기 지연)
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node table-histograms

# SSTable 크기 확인
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node sstable-size

# 스레드풀 통계
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node tpstats

---

Spark Tuning

Configuration file: plantpulse-analytics/spark/conf/spark-defaults.conf

Note: Apache Spark is a distributed processing engine for large-scale data analytics. Refer to the settings below to improve the speed and stability of analysis jobs.

Resource allocation

Parameter	Default	Description
spark.cores.max	16	Maximum Spark cores
spark.driver.cores	2	Driver cores
spark.driver.memory	4g	Driver memory
spark.executor.cores	1	Executor cores
spark.executor.memory	2G	Executor memory
spark.executor.memoryOverhead	2g	Executor overhead memory

Dynamic allocation

Parameter	Default	Description
spark.dynamicAllocation.enabled	true	Dynamic executor allocation
spark.dynamicAllocation.minExecutors	2	Minimum executor count
spark.dynamicAllocation.maxExecutors	16	Maximum executor count
spark.dynamicAllocation.executorIdleTimeout	30	Idle executor timeout (seconds)

Shuffle and parallelism

Parameter	Default	Description
spark.default.parallelism	64	Default parallelism
spark.sql.shuffle.partitions	128	SQL shuffle partitions
spark.sql.adaptive.enabled	true	Adaptive Query Execution
spark.sql.adaptive.coalescePartitions.enabled	true	Partition coalescing

JVM configuration

Both Spark driver and executor use ZGC:

-XX:+UseZGC
-XX:+AlwaysPreTouch
-XX:+UseTLAB
-XX:+ResizeTLAB
-XX:+DisableExplicitGC

S3 (MinIO) integration settings

Parameter	Default	Description
spark.hadoop.fs.s3a.connection.maximum	400	Maximum S3 connections
spark.hadoop.fs.s3a.threads.max	128	Maximum S3 threads
spark.hadoop.fs.s3a.max.total.tasks	1024	Maximum async operations
spark.hadoop.fs.s3a.fast.upload	true	Fast upload enabled
spark.hadoop.fs.s3a.fast.upload.buffer	disk	Upload buffer type
spark.hadoop.fs.s3a.block.size	128MB	S3 block size

Iceberg configuration

Parameter	Default	Description
spark.sql.iceberg.write.target-file-size-bytes	256MB	Target file size
spark.sql.iceberg.column-stats.enabled	true	Column statistics enabled
spark.sql.iceberg.vectorization.enabled	true	Vectorized read enabled
spark.sql.iceberg.manifest.merge.enabled	true	Manifest merge
spark.sql.parquet.compression.codec	zstd	Parquet compression codec

---

Kafka Tuning

Configuration file: plantpulse-messaging/kafka/config/server.properties

Note: Kafka is the message broker that delivers sensor data in real time. As data volume increases, you can improve processing performance by adjusting Kafka settings.

Core settings

Parameter	Default	Description
num.network.threads	3	Network processing threads
num.io.threads	8	I/O processing threads
socket.send.buffer.bytes	102400	Socket send buffer
socket.receive.buffer.bytes	102400	Socket receive buffer
socket.request.max.bytes	104857600	Maximum request size (100MB)
num.partitions	1	Default partition count
log.retention.hours	168	Log retention period (7 days)
log.segment.bytes	1073741824	Log segment size (1GB)

High-volume processing tuning

For handling large amounts of IoT data, refer to the settings below:

# 파티션 수 증가 (메시지 병렬 처리)
num.partitions=4

# I/O 스레드 증가
num.io.threads=16

# 네트워크 스레드 증가
num.network.threads=8

# 소켓 버퍼 증가
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576

---

Messaging Configuration

Configuration file: plantpulse-server/config/plantpulse-mq.properties

Kafka DDS topics

Parameter	Default	Purpose
mq.dds.kafka.topic.tag.point	pp-tag-point	Tag point values
mq.dds.kafka.topic.tag.alarm	pp-tag-alarm	Tag alarms
mq.dds.kafka.topic.asset.data	pp-asset-data	Asset data
mq.dds.kafka.topic.asset.alarm	pp-asset-alarm	Asset alarms
mq.dds.kafka.topic.asset.event	pp-asset-event	Asset events
mq.dds.kafka.topic.asset.aggregation	pp-asset-aggregation	Asset aggregates
mq.dds.kafka.topic.domain.changed.event	pp-domain-changed-event	Metadata changes

MQTT DDS topics

Parameter	Default	Purpose
mq.dds.mqtt.topic.tag.point	tag/point/{tag_id}	Per-tag point values
mq.dds.mqtt.topic.tag.alarm	tag/alarm/{tag_id}	Per-tag alarms
mq.dds.mqtt.topic.asset.data	asset/data/{asset_id}	Per-asset data
mq.dds.mqtt.topic.asset.command	asset/command/{asset_id}	Asset control commands

---

Scheduler Configuration

Configuration file: quartz.properties (web app internal classpath resource — not externalized)

Parameter	Default	Description
org.quartz.threadPool.threadCount	8	Scheduler thread count
org.quartz.threadPool.threadPriority	5	Thread priority

---

Mail / Notification Configuration

Configuration file: plantpulse-server/config/plantpulse-mail.properties

Parameter	Default	Description
mail.smtp.host	192.168.0.41	SMTP server address
mail.smtp.port	25	SMTP port
mail.smtp.auth	false	Authentication required
mail.smtp.starttls.enable	false	TLS enabled

---

Application Configuration

The application.properties file was removed in 2026.06. Console behavior settings such as theme and home page are now managed in the console's System > Settings (PostgreSQL mm_config table), and alarm.duplicate.check.minutes (duplicate alarm check interval in minutes) has been moved to plantpulse-server/config/plantpulse-engine.properties.

---

Performance bottleneck diagnosis

When performance issues occur, refer to the diagnosis methods and solutions for each symptom below.

1. Pipeline queue backlog

Symptom: The pipeline wait queue grows continuously

# Grafana 대시보드에서 "파이프라인 대기 큐" 패널을 확인해 주세요
# 또는 JMX: pipeline.queue.size 값을 확인해 주세요

Solution:

• Increase the engine.pipeline.threads value
• Adjust the engine.pipeline.ratelimit value upward
• Check Cassandra write performance

2. Cassandra write delay

Symptom: Write timeouts or WriteTimeoutException occurs

# 컴팩션 상태 확인
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node compactionstats

# Memtable 플러시
/opt/kopens/plantpulse-platform/plantpulse-datalake-cli/bin/pd node flush

Solution:

• Increase the concurrent_writes value
• Increase the compaction_throughput value
• Check disk I/O performance (SSD use is recommended)

3. Memory shortage

Symptom: JVM GC occurs frequently or OutOfMemoryError occurs

Solution:

• Increase the bin/start.sh value in the service module's -Xmx
• Reduce the engine.pipeline.queue.size value to save memory usage
• Disable unnecessary caches

4. Network latency

Symptom: Data reception latency increases (check in the Grafana "network latency" panel)

Solution:

• Check the network status between the OPC server and PlantPulse
• Adjust the engine.streaming.messaging.timeout.ms value
• Check the MQTT QoS level

---

Technical support

For performance tuning inquiries: webmaster@kopens.com