Skip to main content

Field Case Book

This is not a "do it this way" guide — it's a record of "this happened, and this is how we solved it." It contains only what was actually encountered and resolved in the field between February and September 2026.

How to Read This Document

General procedures and basic usage are not here. For that, Troubleshooting and Technical Troubleshooting are the authoritative references.

What's here follows three lines: Symptom → Cause → Action. Find something similar to the symptom you're facing and use it as a candidate cause. The remaining numbers are only things that are knowledge in themselves — protocol standards or library defaults — and IPs, hostnames, and customer names have been removed.

1. Field Installation · Network​

Can't ping the PLC, no data exchange​

  • Symptom Unable to communicate with the PLC after field installation. No response to ping.
  • Cause The PLC IP and the IP to be assigned to the edge PC were not distinguished.
  • Action Set the edge PC IP to the same subnet · different host · same mask as the PLC. Before heading out, fill in both IPs in a table beforehand and confirm they are different values.
  • Verification If communication on both sides drops at the same time, suspect an IP conflict first. Narrow it down in the order ping → port → read.

Not knowing the Mitsubishi port leaves engineers waiting idle​

  • Symptom Work stalls because the MC protocol port is unknown.
  • Cause Mitsubishi has no officially fixed port. It's a value specified in "Open Settings" in GX Works.
  • Action Ask the engineer to specify just four things — ① the Open Settings screen ② Protocol · Open System · own station port ③ whether MC protocol is configured ④ connection-permitted IP restrictions. Don't open with a vague "how do I set up communication?"
  • Reference Conventional values are built-in Ethernet 1025 · module 2001, and 5001/5002 are reserved. 5007 is a library default, not a Mitsubishi default.

ping and port work, but values can't be read​

  • Symptom Connection succeeds, but memory values don't come in.
  • Cause Issue with the PLC DB block memory address value.
  • Action Ask the PLC engineer to verify the DB block address value. Field data shows that if the address is correct and there's no issue with the connection options, there is no scan problem on the edge side. From that point on, action lies with the PLC side.

Communication becomes unstable when the panel door is closed​

  • Symptom Network becomes very unstable when the panel door is closed, but is fine when open.
  • Cause A configuration where the edge PC, LTE router, and even the antenna were all placed inside a thick panel.
  • Action Install the antenna outside the panel. The electrical room staff designates the drilling location. When installing externally, silicone sealing of the joint is mandatory — it prevents mounting failure.

Missing a checklist item during the site survey causes issues on-site​

  • Symptom You thought you were prepared, but variables keep coming up on-site.
  • Action Fix 6 site survey items — panel structure/enclosure type / PLC model · condition / network communication method / AP zone ping status / edge PC installation location / antenna location. The enclosure is a configuration of edge mini PC + LTE router + 2-outlet power strip, and the power adapter location must be considered.

Could not finalize the number of edge PCs to install​

  • Symptom Site survey was completed but equipment quantity couldn't be determined.
  • Cause The number of collection tags was not secured. The tag count and collection interval are needed to calculate load per equipment and determine the number of units, but no list existed, and the installation location was also limited, making additional installation difficult. A 1:1 mapping was assumed, but the customer required 1:N.
  • Action Before the site survey, obtain ① the PLC tag list/count per site, ② the installation info form (PLC/SCADA · IP/PORT · protocol), and ③ the network configuration plan for new segments.

PLC has no Ethernet port, or routing through a SCADA PC is required​

  • Symptom The assumption of direct Ethernet TCP connection does not hold. Requests like "can't you just take it from the PC connected to the PLC?", or a structure where multiple PLCs only write to a SCADA PC's DB.
  • Action During the site survey, always check whether the PLC has an Ethernet port / whether it routes through SCADA / the SCADA DB type and write method, and decide the collection method in advance.

Attaching a test tool broke collection​

  • Symptom After connecting a verification tool, existing collection was rejected or disrupted.
  • Cause Opened a second TCP connection to a single-connection-slot device like MELSEC.
  • Action Three things are forbidden.
    1. No second connection to a PLC currently being collected from — stop the relevant OPC first.
    2. No writes — a single tool argument can trigger a write.
    3. No 1-second probe timeout — measured device accept delay is about 2.6 seconds. Set it to 3–5 seconds.

2. Boot · Startup​

All endpoints unresponsive after boot + ping retry logs flooding​

  • Symptom All requests time out (code=000).
  • Cause dev.mode=EDGE, but the boot was blocked because the external platform's web port was down.
  • Action Change to dev.mode=STANDALONE and reboot, then revert once the external side recovers. Also check server.host.

Web screen hangs every 3–4 minutes​

  • Symptom The web screen periodically dies and comes back.
  • Cause Platform not connected. When it fails to connect, the web context falls into a reload loop every 3–4 minutes. It stops once connected. It is not a problem with the web module itself.
  • Action Restoring the platform connection is the root fix.

Infinite NPE storm after changing the auth key​

  • Symptom dm is null NPEs pile up endlessly in the logs, and all DB-related functions die.
  • Cause With an incorrect key, API client initialization hard-fails with 403, which leaves DatalakeManager uninitialized, collapsing the DB layer. With a correct key, it recovers via the fallback.
  • Action Revert to the original key and reboot. Confirm via the DataSource initialized — ready line in the boot log.

3. Platform Upstream (Transmission) Disconnection​

Edge web and collection are normal, but no data reaches the platform​

This is the most frequently encountered form.

  • Symptom Field collection and the web are fine, but only the upstream has stopped. TIME-WAIT on the web port spikes to over 100.

  • Cause After a brief line outage, the MQTT reconnect CONNECT is missing credentials, so the platform rejects it and it falls into an infinite reconnect loop. Since credentials are only inserted on the initial boot path, once disconnected the app cannot recover on its own.

  • Action Restarting the web app restores the session. The watchdog below prevents recurrence.

  • Verification The fingerprint is "the platform web port is open, but there is no MQTT session."

    nc -z -w2 <플랫폼IP> <웹포트> && echo ROUTE_OK
    ss -tan | grep -c "ESTAB.*<플랫폼IP>:<MQTT포트>"   # 1 정상 / 0 끊김

A few seconds of line outage causes days of upstream stoppage​

  • Symptom A 7-second outage caused a 3-hour stoppage; a single outage caused an 11-day stoppage.
  • Cause Same as above. Nobody noticed because the web, PLC collection, and the line were all normal.
  • Action Deploy automatic recovery (watchdog). Since outage duration and stoppage duration are unrelated, prioritize devices without a watchdog.

Kept digging into the edge even though the problem was platform storage​

  • Symptom Upstream transmission works, but no data remains on the platform.
  • Cause It was the platform storage path, not the edge. Capturing the topic before the broker confirmed edge publishing was normal; the actual blocking point was that platform storage initialization had permanently failed to initialize due to a DB query timeout.
  • Action Reboot the platform server. Final recovery actually came from resolving several causes simultaneously — a missing broker auth file + tag description null + stopped platform storage buffer.
  • Verification Successful ingestion is judged not by SSTable creation but by the actual existence of the target tag. There was a history of misjudging a pass based only on creation.

watchdog Auto-Recovery Configuration​

  • Symptom Upstream disconnections only got resolved when a human discovered them.
  • Action Deploy three types of watchdog as independent services each — line (log only) · equipment network (log only) · platform session (restarts the web app). Since the others must stay alive even if one dies, they are not merged into one.
  • Judgment Fingerprint confirmed on 4 consecutive checks at 30-second intervals (about 2 minutes) → restart → no reactivation for 15 minutes.
  • Result Zero false positives across 6 deployed units. An actual recovery: outage at 05:27 → triggered at 05:39 → recovered at 05:40.

Installed watchdog, but auto-recovery didn't work​

  • Symptom Stopped for 11 days, but the watchdog did nothing.
  • Cause It was in observation-only (DRY_RUN=1) mode. would restart only logged 1,013 records. The judgment was accurate; this was the only point of failure.
  • Action Deploy in observation mode → verify actual restart on 1 unit → switch to execution mode while in a normal state. Switching while disconnected triggers an immediate restart.

A manual restart command killed its own shell​

  • Symptom After running the restart command, the shell died instantly and the web app was down for 3 minutes 34 seconds.
  • Cause The command line of pkill -9 -ef tomcat contained "tomcat", matching itself. The subsequent startup command that was chained after it never ran.
  • Action Separate the kill and startup into distinct executions. For status checks, use brackets as in pgrep -f 'catalina[.]startup' to avoid self-matching.
  • Caution restart.sh has a log tail at the end, so calling it from automation will hang.

watchdog reports the PLC service as DOWN even though the PLC is normal​

  • Symptom Session is alive, but 1,422 false positives occurred.
  • Cause ① The judgment pattern missed IPv6-mapped notation ([::ffff:<PLC_IP>]:<PLC포트>). ② nc, used as a fallback, was rejected by a single-session device.
  • Action ① Fix the pattern to accept both bracketed and non-bracketed forms. ② Change the judgment order to prefer ss ESTAB passively, falling back to nc only when unavailable.

Restarting watchdog caused a 2-minute collection gap​

  • Symptom When watchdog settings were changed, the web app also shut down together.
  • Cause The web app launched by watchdog is incorporated into the watchdog service's cgroup and dies along with it.
  • Action It self-heals via re-detection/restart after 90 seconds, but a 2-minute stoppage still occurs. Change watchdog settings only during periods with no impact on operations.

Diagnostic channel repeatedly logs 401​

  • Symptom The diagnostic port is chronically in SYN-SENT, and diagnostic auth 401 appears in logs every 2–3 seconds.
  • Cause This is a platform-side matter; the 401 occurs due to a version mismatch where the diagnostic transmitter doesn't carry a key, and this happens identically even under normal operating combinations.
  • Action Since restarting the edge doesn't fix it, exclude it from the watchdog trigger. Judge whether integration is actually normal by whether real tag data arrives within 60 seconds.

4. PLC Collection Stability​

Collection repeatedly stops and restarts on every brief communication fluctuation​

  • Symptom Collection stops and reconnects repeatedly on minor transient communication changes.

  • Cause A single ping failure immediately stopped the collector.

  • Action Put the grace period on stopping and recovery. The NOT_COLLECT decision itself has no grace period.

    WhatValue
    Ping interval · ICMP timeout5 seconds · 800 ms
    How many consecutive ping failures before stopping the collector (reset on success)3 times
    Window in which a recent successful read skips the ping15 seconds or collection interval ×3, whichever is longer
    When it drops to NOT_COLLECTNo grace period — in any cycle with even one error or no values received at all, immediately in that cycle
    How long NOT_COLLECT must persist before recovery (reconnect) is requestedMore than 60 seconds
    Cannot Be Changed via Configuration

    All of the values above, including ping 3 times and recovery 60 seconds, are hardcoded constants. app.properties has no corresponding key — changing them requires a product change. Some older materials describe them as config keys, but that is not true.

    Also, per-tag retries were removed. When the connection drops, all tags fail simultaneously, so retries only delay the scan. Instead, a per-tag read timeout was added so that a delay on one tag doesn't block the entire scan.

    Connection failures are subject to a per-connection reconnect backoff (2026-10-01). It waits from 1 second, doubling up to 60 seconds, before reconnecting. A successful connect or a changed IP or port retries immediately without waiting.

  • Result A brief fluctuation may show up briefly as NOT_COLLECT on screen. It still does not stop the collector or trigger a reconnect — the collector is not stopped until ping fails 3 times in a row, and recovery is requested only when NOT_COLLECT persists for more than 60 seconds.

Monitoring-layer ICMP false positives stop otherwise healthy collection​

  • Symptom Collection is working, but monitoring stops the collector.
  • Cause The monitoring layer judged based on ping results only.
  • Action Change the monitoring criterion to "traffic itself is proof of liveness." If the most recent read succeeded and is fresh, skip the probe.

Socket leak caused the PLC to reject everything for 50 minutes​

  • Symptom All connections — collection, probing, and recovery — were rejected.
  • Cause When the driver was discarded, it was abandoned without close, leaking the socket, and because it was a single-slot device, the leaked socket permanently occupied the slot. connected=false is just a logical flag, separate from the actual TCP socket.
  • Action Ensure that code that removes/replaces a driver always accompanies it with close.

Recovery never opens up due to probe timeout​

  • Symptom Recovery doesn't happen even though the slot is free.
  • Cause The device's accept delay is about 2.6 seconds, but the probe timeout is 1 second, so it was judged as a permanent failure.
  • Action Raise the timeout to 4 seconds. When attaching a new device, measure the actual connection time in logs and verify there's enough margin.
Don't Trust a Single Passing Verification

The three cases just above had the same symptom but three stacked causes. Each time one was fixed, it was judged "resolved," but another cause remained.

Incidents mostly appear within 1 minute of boot. Focus observation on the first 3 minutes, and only grant a pass when the boot decision window + 15 minutes of no stoppage + 0 log entries are all satisfied.

Could not determine whether communication instability was due to the line​

  • Symptom Unable to pinpoint the root segment of intermittent collection failures.
  • Observation In an actual case, connection delay spiked to 6x the average (6,125ms), the link then dropped over several days, and without any action it auto-recovered and collection resumed. A spike in delay is the precursor.
  • Action Set up continuous quality monitoring. Summarize 1-second pings in 10-minute buckets (loss rate · RTT min/avg/max), and when loss rate exceeds 5%, simultaneously snapshot interface error counters and routing. Judgment levels are Normal / Caution 5% / Critical 10%, and logs are retained for 15 days.

Could not reproduce the packet loss scenario​

  • Symptom Unable to reproduce the field symptom in-house, blocking verification.

  • Action Use tc/netem to inject loss only on communication to a specific PLC IP. SSH or other communications are unaffected.

    tc qdisc del dev <NIC> root 2>/dev/null            # 항상 먼저 제거
    tc qdisc add dev <NIC> root handle 1: htb default 30
    tc class add dev <NIC> parent 1: classid 1:1 htb rate 100mbit
    tc class add dev <NIC> parent 1:1 classid 1:10 htb rate 100mbit
    tc qdisc add dev <NIC> parent 1:10 handle 10: netem loss 5%
    tc filter add dev <NIC> protocol ip parent 1:0 prio 1 u32 match ip dst <PLC_IP> flowid 1:10
    tc qdisc del dev <NIC> root                        # 종료

Reading watchdog Logs​

Event blocks are only logged when the state changes. Use grep "EVENT" to see only events.

CategorySignalMeaning
LTECABLE_DISCONNECTEDPhysical link
LTEROUTER_DOWNRouter ping failure
LTELTE_NO_RETURN_TRAFFICTX present but RX is 0 — upstream disconnection
LTELTE_SIGNAL_LOSTExternal ping failure
PLCLINK_DOWN · ARP_L2_ISSUELink · L2
PLCPHYSICAL_LAYER_ISSUEIncreasing CRC — cable/port physical issue
PLCPLC_NO_RESPONSE · PLC_SERVICE_DOWNNo response · port closed

⚠ Some NICs do not support rx_crc_errors. CRC:+0 is a fallback value, so don't use it for physical layer judgment.

5. Values Missing or Abnormal​

HMI screen value differs from the edge-collected value​

  • Cause The HMI value may already be a value with logic applied.
  • Action Narrow it down in 3 stages — raw PLC memory → HMI panel (logic applied) → edge (data type · collection interval · formula). First compare against raw PLC data, then check the edge's type/interval/formula for the final comparison.

Counter occasionally shows strange intermediate values​

  • Cause During the PLC scan cycle, if the counter update timing overlaps with the read moment, an intermediate value with broken consistency between the upper/lower words gets read. This is a timing characteristic, not a fault.

Tag address was entered but keeps returning empty values​

  • Cause In JSON-style addresses, PATH mode notation ($.) was omitted, causing a fall-through to KEY mode. KEY mode does not follow the path and instead looks for the exact top-level key name. This applies to Kafka · MQTT · WebSocket, but not HTTP.

  • Action Check three things.

    • Array indices are dot plus number (.0.); bracket notation is not supported.
    • Field names must be exact, including case (agvid fails / agvId succeeds).
    • Wildcards and full array expansion are not supported.
    TOPIC:$.body.list.0.battery     ✅
    TOPIC:body.list[0].battery      ❌

A specific tag keeps returning empty values​

  • Cause The field name in the publisher's payload was misspelled (status → stutaus). Registering it with the correct spelling results in empty values with no error.
  • Action Before registering, visually confirm the original field names from the actual payload.
  • Also note Empty strings are not ingested at all (creating data gaps during periods with no alarms). JSON null is ingested as string "null", so registering it as a numeric type will fail to parse.

Collected count is far lower than the published count​

  • Cause This is normal by design. The driver overwrites messages with the latest value, and reading is a last-value polling sample, not queue consumption. If the broker LAG is 0, everything is being received.
  • Action If full collection is required, a queue-based redesign is needed. This is a contract change matter.

Old data floods in right after registration, or conversely doesn't come in at all​

  • Cause This is the direction of Kafka's auto-offset (default latest). latest only catches messages published after the consumer group is created, while earliest bulk-consumes existing messages already queued at the moment of registration. Any value other than these two silently falls back to latest.
  • Caution This applies only to Kafka. MQTT has no offset concept — the equivalent spot is clean-session.
  • Action Verify using real-time publishing.

Data extraction returns 0 records with no error log​

  • Cause A jar conflict — a repackaged JSON library conflicted with the standard library, missing a required constructor.
  • Action Replace it with the standard jar, move the conflicting jar to backup, clear the cache, and reboot.
  • Verification Normal operation is a silent success. It's normal if the polling thread is alive, LAG is 0, and the VALUE_NULL log has stopped. No success log is printed.

Type exceptions occur​

  • Symptom NumberFormatException or Invalid tag data type.
  • Cause All types were registered as Integer in bulk.
  • Action Split them by value characteristics — Double for current/temperature/voltage/weight/speed/coordinates, String for alarm/status/zone/direction/color, Integer for battery/error codes. The edge tag type and platform type must match. Even if a value arrives as a string, it will parse correctly as long as the declared type is correct.

Metadata changes don't take effect​

  • Cause The cache was not reloaded.
  • Action Reload all three locations — the platform server, the OPC-UA plugin, and the edge. Missing even one breaks consistency. If re-registering OPC causes the tag ID to be reissued, remove the leftover tag on the platform and stop/start OPC to resync.

A REST-based registration causes the driver to not even connect to the broker​

  • Symptom Looks like a "PLC connection timeout", but TCP is fine and not even the consumer group is created.
  • Cause The OPC creation request body was missing required fields, causing an internal exception and failing the address map scan. Registering via the UI avoids this because the UI fills them in, so it only breaks on the REST path.
  • Action Fill in all eight required fields — opc_id · opc_name · opc_type · site_id · opc_agent_ip · opc_agent_port · timecycle · async. Also check address notation and offset direction.
  • What changed Previously, PUT would blank out unspecified fields, making "delete then recreate" the safe approach. This was fixed via partial merge on 2026-07-20 — now only the fields sent are changed. However, this also means a field can no longer be cleared with null.

Ping works, but collection is 0​

  • Cause Ping succeeds and ARP is fine, but the TCP port was simply closed. The read error counter was 4.8 million while successes were 0.
  • Action This is a matter for PLC-side communication port/module action and cannot be resolved by watchdog. Hand it off to the equipment owner.

Querying the latest value returns old data​

  • Cause SELECT * ... LIMIT 5 is partition order, so old data appears first. Real-time ingestion was normal.
  • Action Verify using WHERE tag_id=... ORDER BY timestamp DESC.

6. Resource Exhaustion · Logs​

Every 30 days, all kinds of errors erupt simultaneously​

  • Symptom Widespread error explosion after 30 days of uptime.

  • Cause A monitoring timer attempts a docker ping every 10 seconds, but if docker is absent, 2 file descriptors are not reclaimed on the failure path. The exception isn't logged, so it accumulates silently. The math checks out — 30.3 days divided by 10 seconds is 261,700 iterations, times 2 is 523,400, and the measured value was 523,690.

  • Action Install the same docker version as the reference working device. A plain reboot will only cause it to recur 30 days later.

  • Verification

    P=$(pgrep -f catalina.startup.Bootstrap | head -1)
    ls /proc/$P/fd | wc -l      # 정상 ~600, 누수 시 50만+
    wc -l < /proc/net/unix      # 정상 ~200, 누수 시 50만+
    ls /var/run/docker.sock     # 없으면 누수 진행 확정
Don't Misdiagnose Secondary Symptoms as Separate Bugs

/proc/stat failures, Mount point not found, JMX NPE, "too many open files", queue WAL exceptions (data loss windows during collection), and log explosions all derive from file descriptor exhaustion. Before the exhaustion, logs had 0 ERROR entries.

Threads/FDs keep growing and OOM occurs​

  • Cause Infinite reconnection to a dead target. With the target's ping working but the protocol port dead, dozens of OPCs reconnect every 1 second, creating a new client on each attempt. The measured net growth was 2.25 per second, accounting for 76% of all active threads. On another device, driver threads grew to 40,900 before OOM occurred, and collection stopped for 3 days with no shutdown log, so it went unnoticed.
  • Action Always stop unused OPCs. The same leak recurs if a PLC goes down, so pay extra attention on long-running equipment.

Tens of thousands of identical log entries pile up​

  • Cause The polling interval is hardcoded, so there is no setting to disable it.
  • Action Don't patch the class — suppress it by adding a WARN logger for the relevant package in the logging config.

7. Deployment · Initialization​

Needed to leave no trace of installation on the customer's equipment​

  • Action Instead of uploading files, feed the script via stdin.

    ssh <호스트> 'bash -s' < install.sh

    No installation files are created on the server at all.

  • Verification Confirm no leftovers exist in temp directories, no backup files are created, and that any new files within 24 hours are only the intended outputs. Backup originals are preserved in the local report.

A one-line install command returns 404​

  • Cause The deployment path was restructured, and the top-level domain had changed.
  • Also encountered
    • Symlinks return 404 — set up pointers using hard links instead.
    • Running it via curl | bash causes read inside the script to hit EOF, aborting the whole run. Fix by inheriting the terminal.
    • A single failed step was causing firewall/tuning/SSL/service registration steps afterward to all be skipped, so that step was made non-mandatory.

Resetting a box to leave only the OS​

  • Action Order Remove containers before removing the engine. Treat the presence of a resetup source as a safety check at step 0 — abort if it's missing.
Never Touch the Firewall

Resetting the firewall during a remote session will disconnect SSH, leaving no way to recover without console access. The resetup script only opens the necessary ports; it never closes any.

  • Intentionally left untouched Hostname, locale, timezone, and auxiliary packages. The resetup process configures these again.

Changed server.host, but data direction doesn't change​

  • Cause That device used a pull method. The platform directly reads the edge's tag value API. Changing the edge's push target doesn't change the direction.
  • Action To receive on a different platform, configure pull settings there, pointing to the edge's address and key.

Integration fails due to mismatched edge/platform versions​

  • Symptom API server is not responding loops infinitely. All ports are open, so the network is not at fault.
  • Cause The new platform only exposes the new API, while the old edge has the old path hardcoded. The handshake must occur before publishing can start.
  • Action Upgrade the edge to the same release as the platform. Configuration cannot fix this — the handshake is compiled in.
  • Token pitfall Saving and deploying are different (you must click "Deploy" on the screen). An allowed IP must be specified, and a 200 on a public ping is not proof of authentication.