MQTT explained: topics, QoS levels and retained messages

MQTT is the protocol most connected devices use to hand their readings to whatever needs them, and it sits at the messaging layer of the IoT protocol stack. It is small enough to run on a sensor node and simple enough to explain in one article, and yet the same three or four concepts are misunderstood in almost every first project: what a topic is, what the quality-of-service levels actually promise, and why a retained message exists. This is those concepts, in order.

How MQTT works

MQTT has three roles. A broker is a server that every other participant connects to. A publisher is a client that sends messages to the broker, each addressed to a topic. A subscriber is a client that tells the broker which topics it wants and then receives every message published to them. A client can be both at once.

Publishers and subscribers never talk to each other. A sensor publishes a temperature to a topic and has no idea whether zero or a hundred subscribers are listening. A dashboard subscribes to that topic and has no idea which device produced the value. The broker is the only thing that knows both sides, and that decoupling is the whole reason the pattern is used: producers and consumers can be added, removed and replaced independently.

A client connects to the broker over a single TCP connection, usually secured with TLS (the alternative for devices too small for that, CoAP, runs over UDP instead), and authenticates with a username and password or a certificate. The connection stays open. Messages in both directions flow over it, and a keep-alive mechanism lets both sides notice when the other has gone quiet.

Topics and wildcards

A topic is a string, and the protocol treats it as a path with slashes. The broker does not care what the segments mean; you decide. The convention almost everyone converges on is a hierarchy from general to specific: a site or organization, then an area or building, then a device, then the measurement. Something like a warehouse, a zone, a sensor identifier and the word for the quantity it reports.

Two wildcards let a subscriber match many topics at once. The single-level wildcard, a plus sign, stands in for exactly one segment: a subscription with the plus sign in the device position receives that measurement from every device in the zone. The multi-level wildcard, a hash sign, stands in for everything from that point down and must be the last character: a subscription ending in the hash receives every message under that branch. Publishers cannot use wildcards; every published message goes to one exact topic.

Some rules of thumb for designing a topic tree. Keep it consistent, so the same kind of value sits at the same depth everywhere. Put the thing that varies most (device identifiers) deeper than the things that vary least (site, area). Never put values in topic names; values go in payloads. Avoid leading slashes and spaces. Keep topics short, because every byte is sent with every message. And write the scheme down, because the topic tree is the closest thing an MQTT system has to a schema.

The payload is bytes. MQTT does not define a format, so a project must choose one: plain numbers for simplicity, JSON for readability, or a compact binary encoding for constrained devices. Whichever it is, include units and timestamps in the convention, because a subscriber receiving a bare number has no way to know either.

QoS levels 0, 1 and 2

Every message is published with a quality-of-service level that says how hard the sender and the broker will work to deliver it. The levels are easy to misread, so here is what each actually promises.

QoS 0, at most once, sends the message and forgets it. No acknowledgment, no retry. If the connection is lost at the wrong moment the message is gone. It is the cheapest option and the right one for readings that will be replaced by the next reading in a few seconds: a temperature every minute, a signal strength, a heartbeat.

QoS 1, at least once, requires an acknowledgment. The sender keeps the message until the receiver acknowledges it, and resends if it does not. The message will arrive; it may arrive more than once, because a lost acknowledgment causes a resend of a message that was already delivered. Subscribers must therefore cope with duplicates, which is easy when messages carry a timestamp or a sequence number and awkward when they are commands such as “increment the counter”. QoS 1 suits events and alarms: things that must not be lost and that can be safely applied twice.

QoS 2, exactly once, uses a four-step handshake so that the message is delivered once and only once. It costs the most round trips and the most state on both sides. It suits commands that must not be lost and must not be repeated, and billing-type events, and it is rarely needed for sensor data.

Two things surprise people. The level applies per hop: a publisher’s QoS is between the publisher and the broker, and each subscriber’s QoS is between the broker and that subscriber, and the broker delivers at the lower of the two. Publishing at QoS 2 to a subscriber that subscribed at QoS 0 gets QoS 0 delivery. And QoS is not persistence: a QoS 1 or 2 message to a subscriber that is offline is queued only if the subscriber has a persistent session, described below.

Retained messages

A retained message is one the broker keeps as the last known value for its topic. When a new subscriber subscribes to that topic, the broker immediately sends it the retained message, before any new ones arrive. Without retention, a dashboard that starts up has to wait for every sensor to report before it can show anything; with it, the dashboard fills in at once with the last known reading from each.

Retention is per topic, and only the most recent retained message is kept. Publishing a new retained message replaces the old one. Publishing an empty retained message clears it. A message is retained only if the publisher sets the flag; ordinary messages pass through and are forgotten.

Use retained messages for state: the current temperature, whether a door is open, the last known position, a device’s configuration. Do not use them for events, because a new subscriber would receive an old alarm as if it had just happened. And clear the retained message when a device is decommissioned, or its last reading will sit in the broker for years.

Last will, keep-alive and sessions

Three features deal with clients that disappear.

Keep-alive is the interval a client promises to send something within; if it has nothing to publish it sends a tiny ping. If the broker hears nothing for longer than that interval plus a margin, it concludes the client is gone and closes the connection. Choose an interval long enough that a sleepy device is not disconnected between reports, and short enough that a dead device is noticed in a useful time.

The last will is a message the client registers when it connects, with a topic and payload of its choosing. If the client disconnects without sending a proper disconnect, the broker publishes the last will on its behalf. Combined with a retained “online” message published on connect, this gives every subscriber a reliable view of which devices are alive: connect, publish “online” retained; die, the broker publishes “offline” retained. Every dashboard that shows a device’s status is built on this.

Sessions decide what the broker remembers about a client between connections. With a clean session, everything is forgotten on disconnect: subscriptions must be re-established and nothing is queued. With a persistent session, identified by a client identifier, the broker keeps the subscriptions and queues QoS 1 and 2 messages that arrive while the client is away, delivering them on reconnect. Persistent sessions are how a device on an unreliable link avoids missing commands. They also accumulate queued messages while a client is away, so a broker with many long-absent clients needs limits.

MQTT version 5, now widely supported, adds features on top of all this: message expiry, so a queued reading does not arrive hours late; user properties for metadata; reason codes that say why something failed; and shared subscriptions that spread a topic’s load across several subscribers. Version 3.1.1 remains common on devices, and the concepts above are the same in both; both specifications are published by OASIS.

Designing a topic tree

Bringing it together, a small but sound design for a site looks like this. Topics run from the site to the area to the device to the measurement. Sensor readings are published at QoS 0, retained, with a JSON payload carrying the value, a unit and a timestamp. Events and alarms are published at QoS 1, not retained, with a sequence number so duplicates can be dropped. Each device publishes a retained status message on connect and registers a last will that sets the same topic to offline. Commands to devices go on a separate branch, at QoS 1 or 2 depending on whether repeating one would matter, and devices that sleep use persistent sessions so commands wait for them.

Write that scheme in a document that every team member and every device firmware follows; in a plant the OPC UA and MQTT article shows where that document sits. An MQTT system with a documented topic tree and payload format stays understandable for years; one without becomes a pile of strings that only its first author could read.