Definition and meaning of MQTT (MQ Telemetry Transport)
MQTT (Message Queuing Telemetry Transport) is an efficient and lightweight messaging protocol designed for communication from devices with limited resources or bandwidth such as the Internet of Things (IoT). It is based on a transport protocol that enables sorted and bidirectional connections, usually such as TCP/IP, as well as in QUIC (more widespread in Google Chrome browser), under the OASIS standard and ISO recommendations (ISO/IEC 20922).
Origin and purpose of the MQTT protocol
Conceived by Dr. Andy Stanford-Clark and Arlen Nipper in 1999, MQTT has evolved to become a fundamental standard for data transmission between connected devices, such as light, motion, temperature sensors, mobile phones, and computers. Its standardization by OASIS in 2013 cemented its position as a key tool in the IoT arena. Simplicity, scalability, and the ability to operate on low power networks have contributed to MQTT’s continued success in facilitating efficient communication in diverse IoT environments.
History and Evolution of MQTT
Development of MQTT up to the current version, as OASIS and ISO standard
Originally, MQTT was invented in 1999 for the oil and gas industry, due to the need for a protocol with the minimum bandwidth and that also ensures a minimum battery loss, in order to be able to monitor the pipelines via satellite. At the time, the protocol was called Message Queue Server Telemetry Transport by IBM’s MQ Series product, which was the first to accept it. Then, in 2010, IBM released MQTT 3.1 as a free and open protocol, so that anyone could implement it. In 2013, OASIS, the Organization for the Advancement of Structured Information Standards (OASIS), was sent for standardization.
In 2019, OASIS released version 5 of MQTT and even makes the acronym the official name of the protocol for IoT and is recognized as an ISO standard (ISO/IEC PRF 20922) of the publication/subscription-based messaging protocol.
Operation of MQTT
Publish/Subscribe Communication Model
The MQTT protocol works according to the principles of the publishing or subscription model. In traditional network communication, clients and servers communicate directly with each other, where clients request resources or data from the server, and then the server processes and sends a response. However, MQTT uses a publishing or subscription pattern to decouple the message sender (publisher) from the message receiver (subscriber), and a third component, called a message agent, controls communication between publishers and subscribers. The agent’s job is to filter all incoming messages from publishers and distribute them correctly to subscribers. Agent decouples publishers and subscribers as follows:
- Spatial decoupling: The publisher and subscriber do not know each other’s network location and do not exchange information, such as IP addresses or port numbers.
- Time Decoupling: Publisher and subscriber do not run or have network connectivity at the same time.
- Sync decoupling: Publishers and subscribers can send or receive messages without interrupting each other. That is, the subscriber does not have to wait for the publisher to send a message.
MQTT Components: Client and Agents
- MQTT Client: Any device running an MQTT library. If the client sends messages, it is the editor; if it receives messages, it is the receiver.
- Agent MQTT: It is the back-end system that coordinates messages between clients. It is responsible for receiving and filtering messages, identifying customers subscribed to each message and sending them the messages. It is also responsible for the authorization and authentication of MQTT clients, passing messages to other systems for lost message and client session analysis and control.
Connection and communication process between clients and agents
MQTT Connection: Clients initiate the connection in MQRR by sending a CONNECT message to an agent. The agent confirms the established connection with a CONNACK message. Both the client and the agent require TCP or IP to communicate – they do not connect to each other without the agent.
The event-driven architecture of MQTT stands out for its ability to handle small and clearly defined messages. In addition, it introduces the concept of Quality of Service (QoS) with levels 0, 1 and 2 (which we will explain later in the benefits section), allowing message delivery to be adapted to the specific needs of each application. With these features, MQTT stands as a versatile and effective protocol for communication in IoT environments, offering a strong and scalable infrastructure.
Topics in MQTT are hierarchical strings used to categorize messages. Subscribers can filter messages by subscribing to specific topics, thereby achieving efficient data distribution. These topics are organized hierarchically by using the “/” character as a delimiter, facilitating an orderly and understandable structure. An important feature is message retention, ensuring that new subscribers receive the latest information. In addition, MQTT incorporates the concept of “last will and testament,” providing a strategy for handling unexpected disconnections by setting a default message to be sent in such situations. These elements enrich the flexibility and reliability of MQTT, consolidating it as a versatile and robust protocol in the context of the Internet of Things (IoT).
Another important aspect is that MQTT is distinguished by four main actions that make up its core function:
- Publish: It allows customers to message a specific topic, sharing relevant information.
- Subscribe: It enables customers to receive messages on a given topic, allowing two-way communication.
- Ping: It keeps the connection between clients and brokers, ensuring the validity and efficiency of the exchange of information.
- Disconnect: It allows customers to terminate their connection in an orderly manner.
Each of these actions plays a key role in the MQTT protocol, making agile and reliable communication easier in the dynamic environment of the Internet of Things (IoT).
Benefits and Applications of MQTT
Advantages of using MQTT in IoT and Industrial IoT (IioT) environments
- Lightweight, with a fixed header of 2 to 5 bytes, so its “weight” in the network is light. It not only optimizes bandwidth when having many devices connected and sending messages simultaneously, but it also allows it to be used in areas where the internet connection is unstable or very limited.
- Highly compatible, being a standard supported by multiple IoT devices using this protocol. To implement it, a minimum amount of code is required, which is a clear advantage for devices from different manufacturers or with limited memory.
- Reliable, with three levels of quality of service (QoS):
- Level 0 (throw and forget). At this level, each message is sent once to each subscriber, without waiting for confirmation, which minimizes bandwidth. The message is not stored. In case customers are disconnected at that time, they will not receive the message, so it is recommended for cases where message loss is not a problem.
- Level 1 (delivered at least once). As the name implies, at this level, the customer must confirm the message received. The advantage is that it ensures that messages are all delivered at least once. The disadvantage is that more bandwidth is used and the client can receive duplicate messages if it takes time to check their receipt.
- Level 2 (deliver only once). This is the level that requires the most bandwidth, although the advantage is that it is more reliable because it waits for the client to confirm the message received and, before sending a message again, asks wether the message has been received. This level ensures that customers received the message only once. In addition to not consuming bandwidth, messages received without being duplicated are guaranteed.
- It provides security as it supports authentication methods such as OAuth or TLS 1.3, allowing it to be used on networks that are not entirely safe.
- Information for real-time analysis, as it provides real-time data, which is very useful in preventive maintenance and monitoring in environments either in a smart home, such as logistics, distribution and manufacturing.
- Open, supported by major cloud vendors such as AWS, Google Cloud, IBM Cloud, Oracle, and Microsoft Azure.
Practical applications of MQTT in multiple industries, such as automotive, energy and telecommunications
Through MQTT, IoT allows obtaining and processing data to manage data that leads to decisions and actions aimed at productivity and optimization of operations in different economy sectors. For instance:
- Transport
Support for mobile applications, giving recommendations of the nearest available vehicles. - Manufacturing
In interaction with robotics in manufacturing cycles or preventive maintenance. - Automotive
Together with artificial intelligence data, it can autonomously recognize and solve deficiencies in human information. - Energy
Remote control and automation of power distribution. - Telecommunications
To provide exceptional mobile and network connectivity services that support smart mobile applications in the home or real-time monitoring systems. - Health
Monitoring of the state of well-being by means of a smart watch that emits data, allowing critical patient information to be identified.
Disadvantages and Challenges of MQTT
Security, Interoperability and Authentication Limitations and Challenges
One of the disadvantages of MQTT is associated with potential security issues, as the protocol does not have any security mechanisms such as encryption or authentication, which puts messages at risk of being intercepted, altered or forged by hackers.
Another disadvantage stems from the fact that the protocol can be a single failure point, which can expose the IoT system to attacks or interruptions.
The above makes us recommend that additional security measures be implemented, such as SSL/TLS encryption, user authentication, passwords or access control lists. It is of utmost importance to protect data and devices, which are the preferred targets of cybercriminals.
Comparison to other transfer protocols such as CoAP and AMQP
There are other protocols for connecting devices such as CoaP (Constrained Application Protocol, internet application layer protocol for devices with restricted resources) and AMQP (Advanced Message Queuing Protocol, is the open standard application layer protocol for message-oriented middleware). Compared to these, MQTT has some disadvantages:
MQTT vs CoAP:
- Transmission Cycles: MQTT has a slower transmission cycle than CoAP.
- Support for RESTful (interface to exchange information securely over the Internet): MQTT is not RESTful (a RESTful API is an interface that two computer systems use to exchange information securely over the Internet), while CoAP is RESTful.
- Resource Discovery: MQTT works on flexible theme subscriptions. CoAP has a stable resource discovery mechanism.
- Encryption: MQTT is not encrypted, although you may use TLS/SSL to implement security and encryption. CoAP works with DTLS (Data Transport Layer Security).
MQTT vs AMQP:
- Message Patterns: The AMQP protocol supports a more sophisticated routing mechanism. Messages first go to an exchange which then routes them to the correct queue, using some predefined rules.
- Message Routing: AMQP supports multiple exchange types with unique routing strategies, this allows this protocol to support different communication patterns, while MQTT is a simple message routing mechanism.
- Versatility: AMQP offers several features in message persistence and transactionality. This makes it very versatile for different use cases that require more features.
Differences between MQTT and REST in terms of architecture and communication model
To understand the difference between MQTT and rest, let’s take a look at the following:
The previous figure shows MQTT architecture, which consists of a centralized intermediary, where all communications between devices (endpoints) go through the intermediary, and the broker can be installed on any public server. Remember that MQTT is based on a publishing/subscribing architecture, where at each end devices can publish topics and subscribe to any topic. In the MQTT protocol, a username and password are needed to establish the connection.
Now rest (Representational State Transfer) is an interface to connect several systems based on the HTTP protocol and used to obtain and generate data and operations, returning data in very specific formats (for example XML and JSON).
This figure shows the REST protocols. See that REST is built on HTTP/TCP layers. The REST protocol uses a bus-based architecture, where no broker component is needed and devices (endpoints) can communicate directly. In this case, request and response messages between end devices are used to exchange the information. With this, unlike MQTT, in rest:
- Messages are GET, PUT, POST, and DELETE.
- The architecture is REQUEST/RESPONSE.
- Broker is not required, as communication is direct.
- The security protocol is HTTPS.
- Interoperability is semantic.
- Fault tolerance is of Server in SPoF (single point of failure)- If one part fails in the System, the operation of the rest of the system stops.
Exploring how MQTT version 5 introduces features similar to REST
The reason we explained the above is that MQTT version 5.0 now has rest-like features, making it more robust:
- Reasons for disconnection: You may now provide a reason code for each acknowledgement packet, giving us a better understanding of why a disconnection or failure took place.
- With MQTT 5.0, a specific time period can be defined during which the session must remain active after disconnection. This provides greater flexibility in managing session duration and maintains resources on the server.
- MQTT 5.0 introduces theme aliases to reduce overhead in message headers. In previous versions, the theme name had to be included in each message, resulting in larger packets.
- Custom metadata, which allows users to include custom metadata in MQTT packet headers. This can be particularly useful for applications that need to send additional information with their MQTT messages, such as message timestamp, device location, or other application.
- Subscription options, to specify how you wish to receive messages for each subscribed topic. For example, clients can now specify whether they wish to receive retained messages for a particular subscription, or whether they wish to receive messages if they have the same level of QoS (Quality of Service) as the subscription.
- The request/response function allows a client to specify a theme that the server can use to send a direct response. This makes communication more efficient and direct.
- Shared subscription; that is, when a message is published on a shared topic, the server distributes the message to one of the clients of the shared subscription, thus balancing the message load.
Also, in this release, customers can connect devices using MQTT5 or leverage a combination of devices connected to MQTT versions 3 and 5, interacting with each other to support heterogeneous deployments.
Security in MQTT
Implementation of security protocols such as SSL/TLS to protect communication in MQTT
Security at MQTT is addressed through multiple layers to ensure the integrity and confidentiality of communication on the Internet of Things (IoT). The first layer focuses on network security, implementing measures to protect the underlying infrastructure. Even though MQTT offers the option to use usernames and passwords, it is crucial to note that this information is transmitted in unencrypted text, which presents an aspect to be considered in terms of privacy. In addition, there is the possibility to strengthen security through the use of SSL/TLS, although this option comes with additional overhead. Together, these strategies seek to balance accessibility and protection, adapting to the specific requirements of each MQTT deployment in IoT environments.
MQTT v5.0 Updates
The evolution of MQTT to version 5.0 set a milestone in its development. Officially published in 2019, this standard introduced significant innovations that expanded its versatility. Highlights include reason codes, which provide greater clarity in event interpretation. Shared subscriptions provide an efficient way to distribute messages to multiple subscribers, improving scalability. Message expiration provides more precise control over information persistence, adapting to multiple application scenarios. In addition, the addition of topic aliases simplifies management and optimizes data sharing. These updates reinforce MQTT’s status as a dynamic, cutting-edge protocol capable of meeting the changing demands of the Internet of Things (IoT) efficiently.
Authentication, authorization and encryption aspects in MQTT
As mentioned earlier, using SSL/TLS helps us provide authentication, encryption and integrity when using the MQTT protocol. i.e.;
- Authentication: The person sending the message is who they say they are.
- Encryption: No one on the road can read the message.
- Integrity: Messages cannot be modified.
This is done through a digital signature (certified) and public and private keys to encrypt and decrypt the message. To add this in MQTT, the steps to follow are:
- Creating a public and private password to authorize certification (CA).
- Create a certificate for the CA and sign with the previously generated private password.
- Generate a public and private password for the MQTT broker.
- Create a certificate signature requirement for the passwords from the previous step.
- Use the certificate from step 2 to sign the requirement from the previous step.
- Copy all certificates into a MQTT broker directory.
- Copy the CA certificate to the client.
- Edit client settings to use TLS and CA certificate.
MQTT and WSS (MQTT on WebSockets)
Description of MQTT on WebSockets and its implementation for receiving data in web browsers
Each browser can be an MQTT device with MQTT over Websockets, which are protocols that make it possible to open an interactive communication session between the user’s browser and a server. This protocol runs over Transport Layer Security (TLS) or Secure Sockets Layer (SSL), providing a safe means of exchanging data. Using WebSockets, directly in a browser, communication efficiency can be achieved by sending messages to a server and receiving event-driven responses without having to query the server for a response. This is because the client and server connect through the WebSocket URL (there are multiple MQTT control packages in a single WebSocket data frame).
Comparison to MQTT standard communication
WebSocket was originally designed for web applications. A WebSocket feature is that it keeps a steady connection to the server, enabling low latency communication compared to other traditional HTTP-based architectures. WebSocket also revolutionizes communication by enabling real-time data transmission, rather than creating multiple short-lived connections for each interaction. With the accelerated growth of IoT device usage, latency and scalability have become more critical to real-time data sharing.
Conclusions and considerations
With infrastructures extending beyond data centers on devices and the Internet of Things, MQTT offers the benefits of being lightweight, reliable, compatible, open and secure. Now with version 5.0 available, an update is recommended. To do this migration process, the following is recommended:
1. Update MQTT brokers, taking great care not to impact all MQTT customers, preferably in a non-production environment, before implementing it.
2. Update customer libraries, first in a non-production environment too. Make sure your application code is up to date to handle the new features in MQTT 5.0.
3. Address new safety considerations. For example, with the new user ownership feature, customers can now send custom data to the broker. While this is a powerful feature, it can be exploited if not used correctly. Therefore, it is important to evaluate all new features from a safety perspective.
4. Monitor after migration. Monitoring should not only be limited to technical aspects, such as message delivery or customer connections, but should also monitor the use of the new MQTT 5.0 features in your applications.
And the recommendation we always make is to combine it with a technology expert who preferably understands the particular needs where the industry your organization belongs to is developed.
Can one tool have global visibility?