Monitoring Unify with Prometheus metrics

unify exposes its metrics in prometheus format on the metrics endpoint (default http //\<host> 9090/metrics ) every metric is a per broker instance except where noted elsewhere here in a cluster deployment, each node exposes its own values and dashboards normally sum them across nodes mqtt retained messages is the one exception as described below event driven counters update in real time gauges that require scanning connected clients or shared states ( mqtt client inflight utilization , mqtt clients send quota saturated , mqtt wildcard subscriptions , mqtt topics tracked , mqtt retained messages , and the broker level connection/subscription gauges) refresh on an internal sampler every 10 seconds by default set metrics sample interval to change the interval the sampler refreshes connector metrics every 15 seconds the following sections example queries in promql, the query language used by prometheus and grafana note the following conventions counters ( total ) only ever increase query them with rate( \[5m]) to get a per second rate or increase( \[1h]) to get a total over a window the raw counter value is rarely useful on its own gauges are point in time values query them directly and wrap them in sum( ) to aggregate across cluster nodes message throughput these are the primary load indicators the rate of mqtt messages received total is the inbound publish load and mqtt messages sent total is delivery load ( received rate × average fan out ) the byte counters distinguish "many small messages" from "few large messages", which is relevant for capacity planning a growing gap between received rate and sent rate can indicate messages with no subscribers, acl denials, or drops (see message drops and rejections docid\ r3 d2ogyse1fbxb5oyovh ) metric description type labels mqtt messages received total total publish messages received from clients, broken down by qos level counter qos (0/1/2) mqtt messages sent total total publish messages delivered to subscribers, by qos one inbound message fanned out to n subscribers counts n times here counter qos mqtt message bytes received total total payload bytes of publish messages received from clients, by qos counter qos mqtt message bytes sent total total wire bytes of publish packets sent to subscribers, by qos counter qos mqtt published total legacy counter of all published messages, any qos this counter remains for backward compatibility; prefer mqtt messages received total counter — mqtt packets sent total legacy counter of all mqtt packets sent (all packet types, not just publish) counter — mqtt packets read total legacy counter of all mqtt packets read (all packet types) counter — example queries \# inbound publish rate (messages/sec) across the cluster sum(rate(mqtt messages received total\[5m])) \# delivery rate broken down by qos level sum by (qos) (rate(mqtt messages sent total\[5m])) \# average fan out how many subscribers each published message reaches sum(rate(mqtt messages sent total\[5m])) / sum(rate(mqtt messages received total\[5m])) \# outbound bandwidth (bytes/sec) — capacity planning sum(rate(mqtt message bytes sent total\[5m])) \# average inbound payload size (bytes per message) sum(rate(mqtt message bytes received total\[5m])) / sum(rate(mqtt messages received total\[5m])) delivery latency the broker's internal delivery time indicates how long a message resides in the broker before it goes out to a subscriber it does not include the network time between broker and client watch the p95/p99 a rising tail under constant load indicates broker or downstream pressure (such as slow subscribers or nats backpressure) for clusters, when the publisher and subscriber connect to different nodes, the measurement spans the inter node nats hop and includes any clock skew between nodes metric description type labels mqtt publish e2e latency seconds broker side latency from receiving a publish to handing it to a subscriber's connection, by qos buckets range from 0 5 ms to 5 s histogram qos example queries \# p99 delivery latency (99% of messages arrive faster than this) histogram quantile(0 99, sum by (le) (rate(mqtt publish e2e latency seconds bucket\[5m]))) \# p95 latency per qos level — expect qos 1/2 to be slower than qos 0 histogram quantile(0 95, sum by (le, qos) (rate(mqtt publish e2e latency seconds bucket\[5m]))) \# mean latency (useful as a trend line, hides the tail) sum(rate(mqtt publish e2e latency seconds sum\[5m])) / sum(rate(mqtt publish e2e latency seconds count\[5m])) \# fraction of messages delivered within 10 ms (slo style check) sum(rate(mqtt publish e2e latency seconds bucket{le="0 01"}\[5m])) / sum(rate(mqtt publish e2e latency seconds count\[5m])) message drops and rejections in a healthy system, reasons stay flat at zero any non zero rate is message loss or rejection and identifies the cause directly the reason label points at either a misbehaving client ( packet too large , acl denied , or quota exceeded ) or a consumption problem ( queue full , inflight expired , or packet id exhausted ) metric description type labels mqtt messages dropped total total messages the broker dropped or rejected, by reason the broker pre initializes all reason series to 0, so a flat zero line means "nothing dropped" (not "no data") counter reason the following table shows what each drop reason indicates reason meaning indicates queue full a subscriber's outbound queue was full that subscriber is consuming slower than messages arrive for it (slow consumer) inflight expired an in flight qos 1/2 message expired or the broker abandoned it before acknowledgment subscribers are not acknowledging in time possibly due to disconnected, stuck, or overloaded clients packet id exhausted the broker aborted a qos delivery because the client's send window had no free packet ids the client's entire qos window sits in unacknowledged messages acl denied an access control rule rejected a publish a client attempted to publish to a topic for which it lacks permission (also see mqtt acl denied total ) packet too large a packet exceeded the configured maximum packet size a client is sending payloads above the configured limit quota exceeded receive maximum or quota enforcement rejected a packet a client is publishing faster than its negotiated qos receive window allows example queries \# drop rate by reason — the primary "is anything being lost?" panel sum by (reason) (rate(mqtt messages dropped total\[5m])) \# total messages dropped in the last hour, by reason sum by (reason) (increase(mqtt messages dropped total\[1h])) \# drop percentage relative to delivery volume 100 sum(rate(mqtt messages dropped total\[5m])) / sum(rate(mqtt messages sent total\[5m])) \# alert condition any drops at all in the last 15 minutes sum(increase(mqtt messages dropped total\[15m])) > 0 qos reliability and backpressure these metrics reveal qos delivery health before messages start dropping resends mean subscribers failed to acknowledge within the retry interval, which is the earliest sign of a struggling consumer or lossy network a steadily climbing mqtt inflight messages means acknowledgments are not keeping up with deliveries the utilization histogram and saturated clients gauge identify how close clients are to their send window limit clients that sit near 1 0 stop receiving new qos messages entirely until they acknowledge, which surfaces as packet id exhausted drops if it persists metric description type labels mqtt qos resends total total qos 1/2 publish retransmissions to subscribers counter qos (1/2) mqtt inflight messages qos 1/2 messages currently in flight the broker has sent these and is waiting for acknowledgment gauge — mqtt client inflight utilization distribution of per client outbound qos window utilization (consumed ÷ negotiated) across all connected clients 1 0 means a client's send window is fully exhausted histogram — mqtt clients send quota saturated number of connected clients whose outbound qos window is at least 25% consumed gauge — example queries \# resend ratio retransmissions as a fraction of qos 1/2 deliveries sum(rate(mqtt qos resends total\[5m])) / sum(rate(mqtt messages sent total{qos= "1|2"}\[5m])) \# in flight backlog trend — sustained growth means acks are falling behind sum(mqtt inflight messages) \# number of clients close to their send window limit right now sum(mqtt clients send quota saturated) \# fraction of client samples with more than 90% of the qos window consumed 1 ( sum(rate(mqtt client inflight utilization bucket{le="0 9"}\[5m])) / sum(rate(mqtt client inflight utilization count\[5m])) ) connections and sessions mqtt clients connected is the basic fleet size or availability signal the disconnect reason breakdown separates normal churn ( client initiated ) from problems a spike in keepalive timeout or network error points at network instability or dying devices recurring takeover means two devices share a client id (a common fleet misconfiguration) a protocol error indicates a broken client implementation the connects total labels show reconnect behavior a high rate with session present="true" is a reconnect storm of existing clients rather than fleet growth metric descripiton type labels mqtt clients connected currently connected mqtt clients gauge — mqtt clients disconnected disconnected persistent sessions the broker is retaining (clients that connected with a session expiry and may return) gauge — mqtt connects total client connections established, by clean session flag and whether the client resumed an existing session counter clean , session present mqtt disconnects total client disconnections, by reason (see below) counter reason mqtt sessions expired total persistent sessions the broker removed after they reached their expiry counter — disconnect reasons reason meaning client initiated normal, client requested disconnect keepalive timeout the client stopped sending within its keepalive interval (silent death, network partition) takeover another client connected with the same client id and took over the session protocol error malformed packet or protocol violation server initiated the broker closed the connection (e g shutdown) network error transport level error (connection reset, tls failure, etc ) other any cause not covered above example queries \# connected fleet size across the cluster (and per node with by (instance)) sum(mqtt clients connected) \# abnormal disconnect rate by reason — excludes normal client requested disconnects sum by (reason) (rate(mqtt disconnects total{reason!="client initiated"}\[5m])) \# connect rate — a sudden spike is a reconnect storm or a fleet rollout sum(rate(mqtt connects total\[1m])) \# share of connections that resumed an existing session sum(rate(mqtt connects total{session present="true"}\[5m])) / sum(rate(mqtt connects total\[5m])) \# session takeovers in the last hour — duplicate client ids in the fleet sum(increase(mqtt disconnects total{reason="takeover"}\[1h])) authentication and access control a rising outcome="failure" rate indicates clients with bad credentials including misconfigured devices, expired tokens, or a brute force attempts the cached label shows how much authentication load hits the database versus the cache a low cache ratio under high connect rates puts pressure on postgresql mqtt acl denied total shows clients trying to publish or subscribe outside their permitted topics this is usually a misconfigured device or an incorrectly scoped acl rule investigate either case metric description type labels mqtt auth total authentication attempts, by outcome and whether the broker validated the credential from the in memory auth cache rather than the database counter outcome (success/failure), cached mqtt acl denied total denied acl permission checks by action counter action (publish/subscribe) example queries \# authentication failure rate (attempts/sec) sum(rate(mqtt auth total{outcome="failure"}\[5m])) \# failure percentage of all auth attempts — spikes suggest bad credentials or an attack 100 sum(rate(mqtt auth total{outcome="failure"}\[5m])) / sum(rate(mqtt auth total\[5m])) \# auth cache hit ratio — a low value under load means the database is doing the work sum(rate(mqtt auth total{outcome="success",cached="true"}\[5m])) / sum(rate(mqtt auth total{outcome="success"}\[5m])) \# acl denials by action (publish vs subscribe) over the last hour sum by (action) (increase(mqtt acl denied total\[1h])) subscriptions and topics subscription counts track consumer side scale wildcard subscriptions get their own metric because each one matches many topics and multiplies fan out cost a rising wildcard count with rising delivery latency is a common correlation mqtt topics tracked reflects the size of the topic namespace unbounded growth usually means clients are embedding unique values (timestamps or uuids) in topic names metric description type labels mqtt subscriptions currently active subscriptions on the broker gauge — mqtt subscribe total subscriptions the broker granted, split by whether the topic filter contains a wildcard ( + / # ) counter wildcard (true/false) mqtt unsubscribe total unsubscriptions, with the same split counter wildcard mqtt wildcard subscriptions active wildcard subscriptions across connected clients gauge — mqtt topics tracked topics the broker currently tracks in its local topic tree gauge — example queries \# active subscriptions across the cluster sum(mqtt subscriptions) \# wildcard share of active subscriptions — high values multiply fan out cost sum(mqtt wildcard subscriptions) / sum(mqtt subscriptions) \# topic namespace growth over the last 24 hours — sustained growth suggests \# unique values (timestamps, uuids) leaking into topic names sum(delta(mqtt topics tracked\[24h])) \# subscribe churn rate, split by wildcard vs exact filters sum by (wildcard) (rate(mqtt subscribe total\[5m])) retained messages retained messages persist indefinitely, the broker delivers them to every new matching subscriber, and the gauge tracks storage growth in the shared store comparing the add rate against a flat gauge distinguishes "clients refreshing retained values on the same topics" (healthy) from "retained messages accumulating on ever new topics" (growth to watch) metric description type labels mqtt retained messages active retained messages in the cluster shared store this value is cluster global every node reports the same number, so aggregate it with max , not sum , in cluster queries gauge — mqtt retained added total retained messages that clients set on this broker instance counter — example queries \# retained messages in the cluster — max, not sum (every node reports the same \# cluster global value; summing would multiply it by the node count) max(mqtt retained messages) \# rate at which clients set retained messages (per node, summed) sum(rate(mqtt retained added total\[5m])) \# net growth of the retained store over the last hour — near zero while the add \# rate is non zero means existing topics are being refreshed, not accumulating max(mqtt retained messages) max(mqtt retained messages offset 1h) connector (bridge) metrics connector connected is the connector health signal 0 for an enabled connector means the remote endpoint is unreachable a repeatedly resetting uptime seconds or a climbing reconnect attempts total means the link is flapping queue depth is the backpressure indicator a persistently growing queue means the remote endpoint cannot absorb the local publish rate and queue full drops follow if it saturates the forwarded counters, split by direction, verify data is actually flowing each way across the bridge these cover the mqtt to mqtt and kafka connectors (bridges) every metric carries a connector id label enabling you to track each configured connector independently metric description type labels connector connected 1 if the connector is currently connected to its remote endpoint, 0 otherwise gauge connector id connector uptime seconds seconds since the connector last (re)connected gauge connector id connector messages forwarded total messages forwarded across the bridge direction="in" is remote → local broker (ingress); direction="out" is local broker → remote (egress) counter connector id , direction connector messages dropped total messages the connector dropped loop (message would echo back to its origin), oversized (exceeds the remote's size limit), queue full (connector's outbound buffer full) counter connector id , reason connector routing errors total transient errors while routing a message (decode, marshal, or publish failures) counter connector id connector reconnect attempts total reconnect attempts to the remote endpoint counter connector id connector queue depth current depth of the connector's outbound queue gauge connector id connector last error timestamp seconds unix timestamp of the connector's most recent error; 0 if no error has occurred gauge connector id example queries \# which connectors are currently down connector connected == 0 \# flapping detection reconnect attempts in the last 15 minutes, per connector sum by (connector id) (increase(connector reconnect attempts total\[15m])) \# throughput per connector and direction (in = remote→local, out = local→remote) sum by (connector id, direction) (rate(connector messages forwarded total\[5m])) \# backpressure connectors whose outbound queue is growing over the last 10 minutes deriv(connector queue depth\[10m]) > 0 \# connectors that logged an error in the last 5 minutes (time() connector last error timestamp seconds < 300) and (connector last error timestamp seconds > 0) process and runtime metrics these metrics separate application level symptoms from resource level causes rising delivery latency alongside rising gc pause time or cpu saturation indicates the broker instance needs more resources latency growth with flat resource usage points at a downstream cause (slow consumers, nats, or database) the metrics endpoint also serves the standard prometheus collectors go go runtime health memory in use ( go memstats ) goroutine count ( go goroutines ) garbage collection pause durations ( go gc duration seconds ) process os level process stats cpu time ( process cpu seconds total ) resident memory ( process resident memory bytes ) open file descriptors ( process open fds each mqtt connection consumes one monitor against process max fds at high connection counts) go build info the go module and version of the running binary example queries \# cpu usage per broker instance (1 0 = one full core) rate(process cpu seconds total\[5m]) \# resident memory per instance process resident memory bytes \# file descriptor headroom — each mqtt connection consumes one descriptor process open fds / process max fds \# goroutine count — a steady climb with flat connection count suggests a leak go goroutines