Monitoring and Troubleshooting

the node writes structured logs to stdout and serves a health endpoint and prometheus metrics on port 9090 together they answer the three questions that matter in production is it connected, is it publishing, and is anything being dropped logs logs are json, one object per line, on stdout; the container runtime owns retention set the level with le logging level at info the node logs startup and session state, connectivity, birth set summaries, host application state changes, and shutdown per device births, litmus edge events, subscription activity, and buffering progress are debug every line carries a component field, which is the fastest way to filter component covers lifecycle startup and shutdown config configuration loading nats the litmus edge nats broker connection mqtt broker connection and reconnects sparkplug birth sets, publishing, commands forward store and forward observability the metrics server other fields appear where they apply node , group , device , tag , topic , seq , bdseq , server , and count {"component" "sparkplug","dbirth published" 4,"dbirth skipped" 0,"devices" 4,"duration ms" 120,"group" "plant1","level" "info","msg" "birth set published","node" "line1","status" "success","time" "2026 07 24t10 30 00 123456789z"} health endpoint curl i http //\<host> 9090/healthz 200 with body ok means the node has a connected mqtt session — and, when host application gating is enabled, that the host application is online anything else returns 503 with body unavailable this is readiness rather than liveness a healthy process waiting for its host application reports 503 by design if your orchestrator distinguishes the two, wire this endpoint to readiness metrics prometheus metrics are served at /metrics on the same port, all prefixed le sparkplug both endpoints are unauthenticated, so bind them to an internal interface or front them with a proxy turn them off entirely with obs enabled=false , or move them with obs addr publishing metric type meaning nbirth total counter nbirth messages published climbing steadily means a rebirth loop ndata total , ddata total counter data messages published ddata metrics total counter individual metrics inside those data messages ddata batch size histogram metrics per data message mqtt publish duration seconds histogram publish duration, labelled by message type connection and session metric type meaning mqtt reconnect total counter reconnect attempts pha state gauge host application state 1 unknown, 0 offline, 1 online nats decode errors total counter messages from litmus edge the node could not decode api request duration seconds histogram litmus edge api calls, labelled by endpoint and status operation duration seconds histogram internal operations, labelled by operation commands metric type meaning ncmd total counter node commands, labelled type rebirth , next server , unknown dcmd total counter device commands received rebirth suppressed total counter rebirth requests dropped because one was already running, labelled reason store and forward metric type meaning sf backlog size gauge messages waiting on disk sf dropped total counter buffered messages discarded instead of replayed, labelled reason sources metric type meaning devicehub entity count gauge latest counts, labelled entity devices , devices published , registers , assigned tags devicehub devices filtered total counter devices dropped by a publish filter, labelled reason source disabled , exclude , not in include digital twins instances filtered total counter twin instances dropped by a publish filter, labelled reason digital twin quarantine total counter twin instances skipped for want of a usable contract, labelled reason digital twin attribute fallback total counter attributes registered as string for want of a type signal digital twin attribute inference total counter attributes typed from a non devicehub source digital twin attribute quarantine total counter attributes dropped while planning the contract digital twin attribute value mismatch total counter runtime values that did not match the birth datatype digital twin attribute value skipped total counter values skipped after such a mismatch scrape configuration scrape configs \ job name le sparkplug static configs \ targets \['le sparkplug 9090'] scrape interval 15s worth alerting on expression what it means rate(le sparkplug mqtt reconnect total\[5m]) > 0 1 broker instability, or host application state flapping le sparkplug pha state == 0 for 5 minutes the host application is down, and data is being buffered or dropped le sparkplug sf backlog size > 10000 the drain is not keeping up with the outage increase(le sparkplug sf dropped total\[15m]) > 0 buffered data was discarded, which happens for a removed or filtered device rate(le sparkplug nats decode errors total\[5m]) > 0 unreadable messages from litmus edge increase(le sparkplug nbirth total\[15m]) > 5 repeated rebirths, so host applications keep rebuilding their tag lists increase(le sparkplug digital twin quarantine total\[15m]) > 0 a twin stopped publishing because its contract no longer compiles increase(le sparkplug digital twin attribute value mismatch total\[15m]) > 0 runtime values disagree with a declared datatype troubleshooting start with the node's logs and, where a symptom involves the broker, a capture of the sparkplug namespace setting le logging level=debug and restarting adds per device and per event detail mosquitto sub h \<broker host> t 'spbv1 0/#' f '%t' no node appears in the host application read the logs in order each component logs connected when its half of the connection succeeds log state cause fix no connected from nats the node cannot reach the litmus edge nats broker inside litmus edge, check the docker gateway, or set edge docker gateway ip outside, check the api account key and that the nats proxy is enabled see deployment # no connected from mqtt the broker is unreachable, or credentials or certificates are rejected check mqtt url , credentials, and the tls settings in deployment # waiting for primary host application and nothing after the host application has not reported online confirm primary host id matches the host exactly, and clear any retained state message left from testing see configuration # birth set published with devices as 0 the node connected but litmus edge returned no devices confirm devices exist and are enabled in devicehub, and that filters are not excluding them a device or tag is missing a whole device check the publish filters le sparkplug devicehub devices filtered total names the reason; compare devicehub entity count{entity="devices"} with {entity="devices published"} one tag on a device either its value type is not one the node maps, or its name has no alphanumeric characters both are logged; the second logs invalid tag name with the tag see sparkplug data # a removed tag still visible in the host application the node publishes ddeath before a shrunk tag set for exactly this reason if the stale tag persists, request a rebirth, or confirm the host processed the ddeath a digital twin publishes no data is there a dbirth for the twin at all? if not, the instance is disabled, has no topic, or its contract compiled to nothing — check le sparkplug digital twin quarantine total does the dbirth contain attributes/… metrics, or only device properties/digital twin/… and device control/rebirth ? only the latter means no dynamic attribute produced a metric are values arriving but not appearing? check le sparkplug digital twin attribute value mismatch total , whose labels name the metric, the expected datatype, and what arrived every digital twin attribute is a string no type signal reached the node, so it registered the honest fallback rather than guessing confirm it with the typesource property on the birth metrics, which reads untyped , and count the affected attributes with le sparkplug digital twin attribute fallback total sparkplug data # explains the fix in the model editor data stops after a reconnect check whether the node is publishing at all le sparkplug ddata total should still be climbing climbing, but the host application shows nothing — the host is rejecting the data, and its own logs will say why flat, with le sparkplug pha state at 0 — the host application went offline and the node stopped publishing by design flat, with reconnects climbing — the session is cycling check the broker and, with several brokers configured, whether the node is walking the list the host application reports a sequence gap almost always two nodes sharing one sparkplug identity, whose sequence counters interleave confirm only one container runs with your group id and node id — including any leftover from an upgrade — then restart the surviving one so the host sees a clean session the node reads as stale after every restart the data directory is not persistent, so the session counter restarts and the host reads each new session as out of order mount /le sparkplug/data on a volume and keep it across upgrades see deployment # duplicate data after a restart expected after an unclean stop with store and forward enabled a buffered message is deleted only after the broker acknowledges it, so a message published just before the process died is replayed once replayed metrics are marked historical, which is how a host application distinguishes them buffered data never arrives le sparkplug sf backlog size stays flat the node still cannot publish check the broker connection and, if gating is enabled, host application state le sparkplug sf dropped total{reason="unbirthed device"} is climbing the data belongs to a device with no birth certificate in this session, because it was removed in litmus edge or excluded by a filter that data is discarded by design the backlog fell without data arriving messages may have passed their le ttl duration certificate errors on the broker connection the node logs certificate problems and keeps running check the mqtt component against the message table in deployment # , which names each decode and key pair failure and its cause the metrics endpoint is unreachable obs enabled may be false , the port may not be published from the container, or obs addr may have moved it verify from inside the container first, then check the port mapping what to include in a support request the node version, and whether it runs inside or outside litmus edge litmus edge version logs at debug covering a restart a broker capture of spbv1 0/# covering the same window the host application's own logs, when it is involved your group id and node id, and the number of devices and twins expected