MQTT: the lightweight IoT messaging protocol explained

MQTT is a publish/subscribe messaging protocol that allows for communication between any program or device that implements the protocol. It is lightweight, open, simple, and designed so as to be easy to implement. These characteristics make it ideal for use in Internet of Things (IoT) and Machine to Machine (M2M) contexts, typically constrained environments where a small code footprint is required and/or network bandwidth is at a premium.

We recently used MQTT as the main driver for communication in a project where we developed a kind of home automation platform. MQTT facilitated communication between a web UI, a number of microservices and PLCs and we have been delighted with how well the system has worked.

The goal of this post is to provide a quick introduction to MQTT’s core architecture and concepts. We will consider the topic of security outside the scope of this post (though some features are mentioned in passing), but I will say it is very much at the forefront in the design of MQTT and provides a number of means to make your implementation secure.

A brief history of MQTT

MQTT started off as a proprietary protocol in 1999. It was used by IBM internally until they released MQTT 3.1 as a royalty-free version in 2010. MQTT has been an OASIS standard since 2013 and the specification is managed by the OASIS MQTT Technical Committee.

The protocol’s swiftness, simplicity, efficiency and scalability – both in and of itself but also compared to other protocols – have not gone unnoticed and over the years it has been widely adopted and is generally used in environments where real-time access to data from devices and sensors is critical. Use cases would include for example smart homes, transportation and manufacturing, but notably even Facebook has used the protocol in their Messenger app.

But where does the name come from?

The “MQ” in “MQTT” originally referred to the IBM MQ product line, where it stands for “Message Queue”. IBM referred to the protocol by the name “MQ Telemetry Transport” in the version 3.1 specification we mentioned above. Looking up MQTT on the internet today, you will find that the four letters are often expanded to “Message Queuing Telemetry Transport”. While this name is relevant from a historical standpoint, the Technical Committee agrees that at least as of 2013 “MQTT” does not stand for anything.

MQTT Architecture basics

Client

The MQTT client can be any program or device that implements the MQTT protocol. Clients connect to a server that handles all message passing, where each client is identified by a unique client ID. A client can publish messages that other clients may be interested in and likewise subscribe to request messages that it is interested in receiving.

Clients are decoupled by design and do not communicate directly with each other. All communication is brokered by a server component which sits in between clients and handles the routing of messages. This decoupling is the foundation of MQTT’s efficient one-to-many capability.

Server

An MQTT server (also commonly “broker”) is responsible for managing which clients are subscribed to which topics, receiving messages published on a particular topic and forwarding that message to any client subscribed for updates. When the connection between a client and server is lost, the server is also responsible for caching the message and delivering it to the client when the connection is re-established.

The server handles security implementation. For example clients can be required to authenticate or to connect using TLS. The server can also restrict access to topics using general rules, or even making rules for specific client IDs.

A demonstration of the MQTT architecture

Topics

All communication in MQTT is grouped into topics. Clients can publish messages to topics and subscribe to receive messages from others. A topic can be any string and is intended to group subjects of common interest, for example, sensor updates would be published to a topic. Topics are hierarchical, which means you will typically see topics structured as level1/level2/level3 where the slash acts as a separator for the levels. Topic subscriptions support wildcards, a powerful and convenient feature.

A subscription using a single-level wildcard (+) will result in a subscription that matches any topic that contains an arbitrary string in place of the wildcard. Subscribing to a topic such as level1/+/level3 means you will be subscribed to level1/foo/level3, level1/bar/level3 etc.

The multi-level wildcard (#) covers multiple topic levels and must be the last character in the topic. Subscribing to the topic level1/# would result in a subscription to level1/foo , level1/bar/baz etc. Essentially the subscription matches any topic that begins with the pattern preceding the wildcard character. As you may have guessed, subscribing to just # creates a subscription to all topics.

Reliability

Many clients connect to servers over unreliable networks, which necessitates the ability to recover gracefully from network outages and other such failures. This is what MQTT’s quality of service (QoS) addresses. QoS functions as an agreement between the message sender and receiver that defines the level of delivery guarantee for a specific message. The protocol defines three levels of quality of service.

QoS 0: at most once delivery – messages can be lost and neither client nor server take any additional steps to confirm delivery.
QoS 1: at least once delivery – messages are confirmed and re-sent if necessary. As messages may be delivered more than once, the receiving client should be able to handle duplication.
QoS 2: exactly once delivery – messages are confirmed and re-sent until they are received by the subscriber exactly once. This level is suitable for scenarios where neither message duplication nor loss is acceptable.

Note that the QoS defined by a publisher and subscriber may differ. The publishing client might publish its message using QoS 2 while the subscribing client defines QoS 1 for its subscription. This poses no issues, as the server will simply use the QoS defined by the recipient to deliver the message.

A higher QoS level comes with the cost of higher overhead. This along with the tolerance for data loss and duplication are important factors to take into account when choosing the appropriate level for your use case.

Implementations

A large number of both proprietary and open source MQTT server implementations are available. You will also find that readily available MQTT client libraries exist for many popular programming languages (Python, Java, JavaScript, C#). Development-oriented readers may check out for example mqtt.js and aiomqtt, both of which make getting a client up and running a breeze.

MQTT does not define a payload specification. This affords the implementing party immense freedom, with the valuable benefit of flexibility to transfer payloads between older and newer systems. On the other hand it can prove a challenge in terms of ensuring compatibility between clients, as communication is essentially based on silent agreements.

Conclusion

This has been a short introduction to MQTT and has hopefully provided insight into why it has become the de facto messaging protocol for IoT and M2M environments. We’ve learned about the the concepts of clients, topics and servers in MQTT in an effort to give you a head start in understanding the protocol’s messaging architecture. A brief look at the levels of quality of service gave us an understanding of how the protocol approaches the issue of reliability in communication.

We’ve seen that MQTT is commonly used in IoT and M2M contexts, but it need not be pigeonholed into those few contexts only. The versatility of potential use cases is one of MQTT’s many appealing features. The protocol is ideal for both small hobby projects and larger-scale applications. Due to the protocol being payload agnostic, the structuring of your payloads will play a large role in whether MQTT is a suitable tool for your needs.