MQTT is the open protocol. This is used for asynchronous message queuing. This has been developed and matured over several years. MQTT is a machine to machine protocol. It's been widely used with embedded devices. Microsoft is having its own MQTT tool with huge support. Here, we are going to overview the MQTT protocol & its details.

MQTT Protocol:

MQTT is a very simple publish / subscribe protocol. It allows you to send messages on a topic (channels) passed through a centralized message broker.

The MQTT module of API will take care of the publish/ subscribe mechanism along with additional features like authentication, retaining messages and sending duplicate messages to unreachable clients.

There are three parts of MQTT architecture –

• MQTT Broker – All messages passed from the client to the server should be sent via the broker.
• ‍MQTT Server – The API acts as an MQTT server. The MQTT server will be responsible for publishing the data to the clients.
• ‍MQTT Client – Any third party client who wishes to subscribe to data published by API, is considered as an MQTT Client.
  

The MQTT Client and the MQTT Server need to connect to the Broker in order to publish or subscribe messages.

Suppose our API is sending sensor data to get more ideas on MQTT.
API gathers the sensor data through the Monitoring module, and the MQTT module publishes the data to provide different channels. On the successful connection of external client to the MQTT module of the API, the client would receive sensor data on the subscribed channel.

Below diagram shows the flow of data from the API Module to the External clients.

MQTT Broker – EMQTT:

EMQTT (Erlang MQTT Broker) is a massively scalable and clusterable MQTT V3.1/V3.1.1 broker, written in Erlang/OTP.

Main responsibilities of a Broker are-

• Receive all messages
• Filter messages
• Decide which are interested clients
• Publish messages to all the subscribed clients

All messages published are passed through the broker. The broker generates the Client ID and Message ID, maintains the message queue, and publishes the message.

There are several brokers that can be used. Default EMQTT broker developed in ErLang.

MQTT Topics:

A topic is a string(UTF-8). Using this string, Broker filters messages for all connected clients. One topic may consist of one or more topic levels. Forward slash(topic level separator) is used for separating each topic level.

When API starts, the Monitoring API will monitor the sensor data and publish it in a combination of topics. The third party client can subscribe to any of those topics, based on the requirement.

The topics are framed in such a way that it provides options for the user to subscribe at level 1, level 2, level 3, level 4, or individual sensor level data.

While subscribing to each level of sensor data, the client needs to specify the hierarchy of the IDs. For e.g. to subscribe to level 4 sensor data, the client needs to specify level 1 id/ level 2 id/ level 3 id/ level 4 id.

The user can subscribe to any type of sensor by specifying the sensor role as the last part of the topic.

If the user doesn’t specify the role, the client will be subscribed to all types of sensors on that particular level.

The user can also specify the sensor id that they wish to subscribe to. In that case, they need to specify the whole hierarchy of the sensor, starting from project id and ending with sensor id.

Following is the list of topics exposed by API on startup.

Features supported by MQTT:

1. Authentication:

EMQTT provides authentication of every user who intends to publish or subscribe to particular data. The user id and password is stored in the API database, into a separate collection called ‘mqtt

While connecting to EMQTT broker, we provide the username name and password, and the MQTT Broker will validate the credentials based on the values present in the database.

2. Access Control:

EMQTT determines which user is allowed to access which topics. This information is stored in MongoDB under the table ‘mqtt_acl’

By default, all users are allowed to access all topics by specifying ‘#’ as the allowed topic to publish and subscribe for all users.

3. QoS:

The Quality of Service (QoS) level is the Quality transfer of messages which ensures the delivery of messages between sending body & receiving body. There are 3 QoS levels in MQTT:

• At most once(0) –The message is delivered at most once, or it is not delivered at all.
• At least once(1) – The message is always delivered at least once.
• ‍Exactly once(2) – The message is always delivered exactly once.
  

4. Last Will Message:

MQTT uses the Last Will & Testament(LWT) mechanism to notify ungraceful disconnection of a client to other clients. In this mechanism, when a client is connecting to a broker, each client specifies its last will message which is a normal MQTT message with QoS, topic, retained flag & payload. This message is stored by the Broker until it it detects that the client has disconnected ungracefully.

5. Retain Message:

MQTT also has a feature of Message Retention. It is done by setting TRUE to retain the flag. It then retained the last message & QoS for the topic. When a client subscribes to a topic, the broker matches the topic with a retained message. Clients will receive messages immediately if the topic and the retained message are matched. Brokers only store one retained message for each topic.

6. Duplicate Message:

If a publisher doesn’t receive the acknowledgement of the published packet, it will resend the packet with DUP flag set to true. A duplicate message contains the same Message ID as the original message.

7. Session:

In general, when a client connects with a broker for the first time, the client needs to create subscriptions for all topics for which they are willing to receive data/messages from the broker. Suppose a session is not maintained, or there is no persistent session, or the client lost a connection with the broker, then users have to resubscribe to all the topics after reconnecting to the broker. For the clients with limited resources, it would be very tedious to subscribe to all topics again. So brokers use a persistent session mechanism, in which it saves all information relevant to the client. ‘clientId’ provided by client is used as ‘session identifier’, when the client establishes a connection with the broker.

Features not-supported by MQTT:

1. Not RESTful:

MQTT does not allow a client to expose RESTful API endpoints. The only way to communicate is through the publish /subscribe mechanism.

2. Obtaining Subscription List:

The MQTT Broker doesn’t have the Client IDs and the subscribed topics by the clients. Hence, the API needs to publish all data to all possible combinations of topics. This would lead to a problem of network congestion in case of large data.

MQTT Wildcards:

MQTT clients can subscribe to one or more topics. At a time, one can subscribe to a single topic only. So we can use the following two wildcards to create a topic which can subscribe to many topics to receive data/message.

1. Plus sign(+):

This is a single level wildcard. This is used to match specific topic level. We can use this wildcard when we want to subscribe at topic level.

Example: Suppose we want to subscribe for all Floor level ‘AL’(Ambient light) sensors, we can use Plus (+) sign level wild card instead of a specific zone level. We can use following topic:

<project_id>/<building_id>/<floor_id>/+/AL</floor_id></building_id></project_id>

2. Hash Sign(#):

This is a multi level wildcard. This wildcard can be used only at the end of a topic. All data/messages get subscribed which match to left-hand side of the ‘#’ wildcard.

Example: In case we want to receive all the messages related to all sensors for floor1 , we can use Hash sing(#) multi level wildcard after floor name & the slash( / ). We can use following topic-

<level 1_id="">/<level 2_id="">/<level 3_id="">/#</level></level></level>

MQTT Test tools:

Following are some popular open source testing tools for MQTT.

1. MQTT Lens
2. MQTT SPY
3. MQTT FX
   

Difference between MQTT & AMQP:

MQTT is designed for lightweight devices like Embedded systems, where bandwidth is costly and the minimum overhead is required. MQTT uses byte stream to exchange data and control everything. Byte stream has optimized 2 byte fixed header, which is prefered for IoT.

AMQP is designed with more advanced features and uses more system resources. It provides more advanced features related to messaging, topic-based publish & subscribe messaging, reliable queuing, transactions, flexible routing and security.

Difference between MQTT & HTTP:

MQTT is data-centric, whereas HTTP is document-centric. HTTP is a request-response protocol for client-server, on the other hand, MQTT uses publish-subscribe mechanism. Publish/subscribe model provides clients with the independent existence from one another and enhances the reliability of the whole system. Even if any of the client is out of network, the system keeps itself up and running

As compared to HTTP, MQTT is lightweight (very short message header and the smallest packet message size of 2 bytes), and allows to compose lengthy headers and messages.

MQTT Protocol ensures high delivery guarantees compared to HTTP.

There are 3 levels of Quality of Services:

- at most once: it guarantees that message will be delivered with the best effort.

- at least once: It guarantees that message will be delivered at a minimum of one time. But the message can also be delivered again..

- exactly once: It guarantees that message will be delivered one and only one time.

Last will & testament and Retained messages are the options provided by MQTT to users. With Last Will & Testament, in case of unexpected disconnection of a client, all subscribed clients will get a message from the broker. Newly subscribed clients will get immediate status updates via Retained message.

HTTP Protocol has none of these abilities.

Conclusion:

MQTT is one of its kind message queuing protocols, best suited for embedded hardware devices. On the software level, it supports all major operating systems and platforms. It has proven its certainty as an ISO standard in IoT platforms because of its more pragmatic security and message reliability.