MQTT 5 fleet design is the discipline of turning a flexible publish-subscribe protocol into a stable product contract. A broker can route arbitrary topics and delivery levels, but a fleet needs rules for identity, ownership, schema, authorization, offline behavior, expiry, retained state, congestion, and evolution. Without those rules, topic names become hidden APIs and broker queues become accidental product storage.
Design from the decision or action each message supports. Telemetry that can be sampled, an alarm that must arrive, a command that must not execute late, and a desired-state snapshot need different topics and delivery semantics. MQTT 5 supplies useful mechanisms such as reason codes, session expiry, message expiry, receive maximum, topic aliases, subscription options, and shared subscriptions. Architecture must assign each mechanism a bounded fleet policy.
Define the fleet message contract before topic strings
List message classes: observations, state, events, alarms, commands, acknowledgements, configuration, software lifecycle, diagnostics, and presence. For each, name publisher authority, subscriber purpose, payload schema, expected frequency and size, ordering key, loss tolerance, duplication handling, maximum useful age, sensitivity, and retention. This exposes contradictions early, such as a critical command on a lossy channel or high-rate telemetry placed in a persistent offline session.
Keep MQTT transport metadata separate from domain payload. Use payload fields for event time, schema version, measurement unit, sequence or sample identity, and domain correlation. MQTT properties can carry content type, response topic, correlation data, message expiry, and bounded user properties where broker support and policy permit. Do not rely on a property that a selected managed broker omits; platform implementations differ from the complete OASIS standard.
| Message class | Typical QoS | Expiry or retention | Required consumer behavior |
|---|---|---|---|
| High-rate observation | QoS 0 when occasional loss is acceptable | Short message expiry; usually not retained | Aggregate, detect gaps, avoid assuming every sample |
| Alarm or lifecycle event | QoS 1 | Expiry tied to response usefulness | Deduplicate by stable event ID and record outcome |
| Desired state | QoS 1 | Retained or shadowed current value with version | Reject stale version and report applied state |
| Device command | QoS 1 | Short explicit expiry | Authorize, deduplicate command ID, check precondition, acknowledge result |
| Diagnostic stream | QoS selected by session need | Bounded campaign and storage | Enforce rate, access, and automatic campaign end |
Build a topic taxonomy around authority and stable identity
Use a small, documented hierarchy such as organization or tenant, environment, product, stable device ID, channel, and message class. Put only routing dimensions in the topic. Avoid mutable labels such as customer name, room, owner, or firmware version when they force resubscription and policy rewrites after change. Keep personal or sensitive values out of topic names because brokers and logs often expose them more broadly than encrypted payload fields.
Separate device-published and service-published namespaces so authorization can express who controls each direction. A device might publish observed state and command results while subscribing to desired state and commands. Reserve platform and system prefixes. Define wildcard policy explicitly: a device should generally receive only its own namespace, while fleet services may subscribe across a product or tenant under controlled identities. Lint topic names and policies in CI against canonical examples.
Choose QoS from loss and duplication consequences
QoS 0 is best-effort delivery and can be appropriate for frequent replaceable samples. QoS 1 adds acknowledgement and redelivery, providing at-least-once behavior that requires idempotent consumers. QoS 2 adds a handshake for exactly-once protocol delivery but does not make a downstream business effect exactly once, and many managed IoT brokers do not support it. AWS IoT Core, for example, documents support for QoS 0 and 1, not QoS 2.
Assign QoS by topic class in a registry and cap what devices may request. Higher QoS consumes in-flight state, bandwidth, broker memory, and acknowledgement capacity. Test loss and reconnect under real link conditions. A QoS 1 command still needs a command ID, target version or precondition, expiry, authorization, and result topic. A duplicate PUBLISH after reconnect must not unlock a door, debit an account, or apply firmware twice.
Bound session persistence and offline queues
In MQTT 5, Clean Start determines whether a connection begins with a new session, while Session Expiry Interval controls how long session state persists after disconnect. Choose policy by client role. A device that reconnects briefly may benefit from preserved subscriptions and bounded QoS 1 delivery. An ephemeral analytics consumer should usually start clean. A decommissioned or replaced device must not leave a persistent session consuming broker storage.
Session persistence is not an instruction to queue everything offline. Set message expiry on time-sensitive publishes so an old command disappears before reconnection. Cap queued count and bytes per client or shared group, and decide overflow behavior by class. Prefer current desired state over a long sequence of obsolete updates. Record session present on reconnect and resubscribe when absent. Monitor expired sessions and offline queue age across device generations.
| Client role | Clean Start | Session expiry | Offline policy |
|---|---|---|---|
| Intermittent field device | Resume when compatible | Longer than normal outage, shorter than abandonment window | Queue bounded QoS 1 events; expire commands |
| Always-on gateway | Resume after brief process restart | Short operational recovery window | Backpressure locally and prioritize alarms |
| Stateless worker | Usually new session | Zero or short | Use shared subscription and external durable work state |
| Interactive support tool | New session | Zero | No offline messages; query current state explicitly |
| Device being retired | New connection blocked after revocation | Delete session | Purge queued commands and retained authority state |
Use retained messages only for authoritative last-known values
A retained PUBLISH stores one last message for a topic and sends it to a qualifying new subscriber. Use it for small current-state values such as desired configuration pointer, presence state with timestamp, or latest supported metadata when the publisher is authoritative. Do not treat retained topics as event history. New subscribers receive the latest retained value, not the sequence that led there, and broker-specific wildcard behavior may differ.
Include schema, source, event time, version, and expiry where supported. Define deletion by publishing the protocol's zero-length retained payload and protect that permission. Avoid a retained command that can execute each time a client subscribes; model desired state with version and idempotent convergence instead. Distinguish retained messages from persistent-session queues and from a device-shadow service, which may add structured versioning, indexing, and conflict behavior.
Scale consumers with shared subscriptions and stable processing keys
MQTT 5 shared subscriptions allow one matching publication to be delivered to one session in a named subscriber group, enabling parallel processing. The standard uses the $share/{ShareName}/{filter} form. Use groups for stateless or externally stateful fleet consumers, not for every device. Keep the shared group name stable across deployments and give each consumer idempotent handling because QoS and failures can still produce retries.
Partition or serialize by device or aggregate when order matters. Shared delivery does not guarantee that different messages for one device land on the same worker or complete in arrival order. Carry device sequence and event time, reject stale state updates, and use an external queue or partitioned processor if strict per-key sequencing is required. Test broker behavior for retained messages, offline group members, group queue limits, and fairness before relying on it.
Apply backpressure at device, broker, and consumer
Backpressure begins at the source. Set a telemetry budget by operating mode, aggregate or sample at the edge, batch where latency allows, and prioritize alarms over diagnostics. Devices need bounded local buffers with drop policy, age limits, and a way to report loss. When disconnected, do not replay hours of stale high-rate samples at unlimited speed. Pace catch-up traffic and preserve live safety or control messages.
MQTT 5 Receive Maximum limits unacknowledged QoS 1 and 2 PUBLISH packets in one direction and helps bound in-flight work, but it does not replace application flow control. Monitor publish rejection and reason codes, acknowledgement latency, inflight count, offline queue depth, shared-group queue depth, consumer lag, dropped and expired messages, connection churn, and broker quotas. AWS publishes specific IoT Core quotas; verify the selected service's current limits rather than encoding portable assumptions.
Authorize topic actions and evolve contracts deliberately
Authenticate every client with a device or workload identity and authorize connect, publish, subscribe, retained access, and administrative operations separately. Bind device identity to allowed topic variables server-side; do not trust a client-supplied device ID in the topic. Separate production from test namespaces and credentials. Rate-limit identities, rotate credentials, revoke retired devices, and audit wildcard grants and shared consumer permissions.
Version payload schemas and compatibility rules, not every topic by default. Additive optional fields can evolve in place when consumers tolerate them; breaking semantic changes need a new schema version and often a parallel topic class or negotiated capability. Test old devices against new consumers and new devices against supported old paths. Publish deprecation dates and measure active subscribers before removal. Device lifetimes usually exceed backend release cycles.
Validate the design under failure and fleet scale
Test duplicate QoS 1 messages, reconnect with and without session present, expired commands, retained-state replacement and deletion, authorization denial, consumer crash, broker failover, link flapping, clock error, out-of-order delivery, full device buffers, and slow acknowledgements. Use several firmware generations and realistic payload sizes. Verify the business effect and message-loss accounting, not only that clients reconnect.
Load tests should include synchronized fleet reconnect and queued-message drain, not only steady publish rate. Stage device cohorts and constrain reconnect jitter so a service recovery does not cause a connection storm. Capacity models include connections, subscriptions, topic policies, publish and delivery rates, inflight messages, stored sessions, retained count, and logs. Set warning thresholds below provider quotas and rehearse degradation that drops replaceable data before critical traffic.
Key takeaways
- Define message classes, authority, schema, useful age, and loss or duplication tolerance before creating a topic tree.
- Use stable identity and directional namespaces; keep mutable and sensitive labels out of topic names.
- Treat QoS 1 as at-least-once and make commands and consumers idempotent regardless of broker acknowledgement.
- Bound persistent sessions, offline queues, retained state, and replay rate so disconnected fleets cannot create unlimited backlog.
- Measure backpressure across device, broker, and consumer, and validate provider-specific MQTT support and quotas.
Frequently asked questions
Should all important messages use QoS 2? No. Broker support is not universal, protocol exactly-once does not cover downstream effects, and the handshake costs more. QoS 1 with stable message identity and idempotent processing is the common durable choice for important fleet events.
Are retained messages a device shadow? Not necessarily. A retained message stores one value per topic. A shadow service may provide document structure, desired versus reported state, versions, conflict handling, APIs, and indexing. Choose the contract needed rather than assuming equivalence.
How long should a session persist? Longer than expected intermittent outages but shorter than the point at which queued work becomes useless or a device is considered abandoned. Tie the value to product behavior, message expiry, broker capacity, and revocation operations.
Conclusion
MQTT 5 provides expressive delivery building blocks, not a finished fleet architecture. A durable topic and payload contract, deliberate QoS, bounded sessions, safe retained state, scalable consumers, and end-to-end backpressure turn those primitives into predictable product behavior. The resulting fleet can survive intermittent links and bursts without making the broker an unbounded hidden database.