Maurice Kalinowski

Optimizing Device Communication with Qt MQTT

Published Wednesday May 16th, 2018
5 Comments on Optimizing Device Communication with Qt MQTT
Posted in Automation, Dev Loop, Embedded | Tags: , ,

Qt for Automation has been launched in conjunction with Qt 5.10 as an AddOn to Qt for Application Development or Qt for Device Creation. One module in that offering is our client-side solution for MQTT called Qt MQTT.

Qt MQTT focusses on the client side, helping developers to create sensor devices and or gateways managing data to be sent via MQTT.  MQTT describes itself to be lightweight, open and simple. The principle idea is to reduce the protocol overhead as much as possible. The most used version of this protocol is MQTT 3.1.1, usually referred to as MQTT 4. The MQTT 5 standard has been agreed on recently, and we are working on adding support for it in Qt MQTT. But that will be part of another announcement later this year.

To verify the functionality of a module and also to provide guidelines for developers on its usage, The Qt Company created a demo which is called SensorTag. We will use this demo as a basis for this blog series.

A brief introduction can be viewed in this video:

The source code of this demo is located in our Boot 2 Qt demo repository here ( http://code.qt.io/cgit/qt-apps/boot2qt-demos.git/tree/tradeshow/iot-sensortag ).

Basically, this demo describes a scenario in which multiple sensors report data to a gateway (in this case Raspberry Pi3) via Bluetooth, which then again sends data to an MQTT broker in the cloud. Multiple gateways mesh up to generate a sensor network.

automation

Each sensor propagates updates to what they measure, more specifically

  • Ambient Temperature
  • Object Temperature
  • Acceleration
  • Orientation / angular velocity
  • Magnetism
  • Altitude
  • Light
  • Humidity

As the focus of this series is about data transmission, the Sensor / Gateway combination will be summed up as “device”.

The demo serves two purposes; It has a pleasant user interface, which appeals to many, and it showcases how easy it is to integrate MQTT (and Qt MQTT).

MQTT is a publish/subscribe protocol, which implies that data is sent related to a specific topic and other recipients register themselves to a broker (server) to receive notifications on each message published.

For the devices above, available messages are:

  • [Topic: Sensors/active, Data: ID]: Every 5 seconds a sensor publishes an “Online” message, notifying subscribers that the device is still active and sending data. On initial connect it also includes a Will message “Offline”. A will message is sent whenever a device disconnects to notify all subscribers with a “Last Will”. Hence, as soon as the network is disconnected, the broker broadcasts the last will to all subscribed parties.
  • [Topic: Sensor/<sensorID>/<datatype>/, Data:<value>]: This is send from each sensor whenever a value for a specific datatype like in above list changes. The user interface from the video subscribes to these messages and updates the graphics accordingly.

Side-note: One additional use-case could be to check the temperature of all devices. In that case, “wildcard” subscriptions can be very useful. Subscribing to “Sensor/+/temperature will lead to receiving temperature updates from all sensors.

Currently, there are around 10 sensors registered at the same time at peak times, mostly when the demo is presented at tradeshows. As mentioned above, demo code exists to showcase an integration, not necessarily the most performant or energy-saving solution. When going towards production, a couple of questions need to be asked:

  • What would happen in a real-world scenario? Would the demo scale up to thousands of devices reporting data?
  • Does the demo fit to requirements on hardware and/or battery usage?
  • Is it within the boundaries of communication limitations of Low Power WAN solutions?

In case you are not familiar with LPWAN solutions like LoRA or Sigfox, I highly recommend Massimo’s talk (https://www.youtube.com/watch?v=s9h7osaSWeU ) at the Qt World Summit 2017.

 

To simplify the source to look at and to reduce it to a non-UI minimal example, there is a minimal representation available here . In that the SensorInformation class has a couple of properties, which get updated frequently:

class SensorInformation : public QObject

{
    Q_OBJECT
    Q_PROPERTY(double ambientTemperature READ ambientTemperature WRITE setAmbientTemperature NOTIFY ambientTemperatureChanged)
    Q_PROPERTY(double objectTemperature READ objectTemperature WRITE setObjectTemperature NOTIFY objectTemperatureChanged)
    Q_PROPERTY(double accelerometerX READ accelerometerX WRITE setAccelerometerX NOTIFY accelerometerXChanged)
    Q_PROPERTY(double accelerometerY READ accelerometerY WRITE setAccelerometerY NOTIFY accelerometerYChanged)
    Q_PROPERTY(double accelerometerZ READ accelerometerZ WRITE setAccelerometerZ NOTIFY accelerometerZChanged)
    Q_PROPERTY(double altitude READ altitude WRITE setAltitude NOTIFY altitudeChanged)
    Q_PROPERTY(double light READ light WRITE setLight NOTIFY lightChanged)
    Q_PROPERTY(double humidity READ humidity WRITE setHumidity NOTIFY humidityChanged)
[..]

 

Whenever a property is updated the client publishes a message:

    m_client->publish(QString::fromLatin1("qtdemosensors/%1/ambientTemperature").arg(m_id),
                      QByteArray::number(ambientTemperature));

 

The device update rate depends on the type of sensors, items like temperature and light intensity are updated less frequently than, for instance, acceleration.

To get an idea of how much data is sent, example Part1B hooks into the client’s transport capabilities. MQTT in its standard has only a limited number of requirements for the transport. Whatever transmits the data must send it ordered, lossless and bi-directional. Theoretically, anything starting from QIODevice is capable of this. QMqttClient allows to specify a custom transport via QMqttClient::setTransport(). Here, we use the following:

 

class LoggingTransport : public QTcpSocket
{
public:
    LoggingTransport(QObject *parent = nullptr);

protected:
     qint64 writeData(const char *data, qint64 len) override;
private:
    void printStatistics();
    QTimer *m_timer;
    QMutex *m_mutex;
    int m_dataSize{0};
};

Inside the constructor a timer is created to invoke printStatistics() at a regular interval. writeData simply stores the length of data to be send and then passes on to QTcpSocket::writeData().

To add the LoggingTransport to a client, all one needs to do is:

    m_transport = new LoggingTransport(this);
    m_client = new QMqttClient(this);
    m_client->setTransport(m_transport, QMqttClient::AbstractSocket);

The output shows that after 11 seconds one sensor sends about 100KB of data. This is just one sensor. Considering the “real-world” example this is clearly unacceptable for distribution. Hence, the effort is now to reduce the number of bytes to be published without losing information.

The demo itself uses one connection at all times, meaning it does not need to dis- or reconnect. The connect statement in MQTT 3.1.1 is around 10 bytes plus the ID of the client. In case a device would reconnect each time to send data, this can add up to a significant amount. But in this case, we “optimized” it already.

Consequently, we move on to the publishing step. There are two options, reduce the size of each message and reduce the number of messages.

For the former, the design of a publish message for ambient temperature is the following:

Bytes Content
1 Publish Statement & Configuration
1 Remaining Length of the message
2 Length of the topic
71 Topic: qtdemosensors/{8f8fde60-933d-44cf-b3a7-8dac62425a63}/ambientTemperature
2 ID of the message
1..8 Value (String for double)

It is obvious that the topic itself is to be taken responsible for the size of the message, especially as the payload storing the value is just a fraction of the size of a message.

Concepts which can be applied here are:

  • Shorten the “root” topic (qtdemosensors -> qtds)
  • Shrink the size of the ID (UUID -> 8 digit number)
  • Replace the clear text of sensortype with an enum (ambientTemperature -> 1)

All those approaches come with a price. Using enums instead of clear text property reduces the readability of a message, the subscriber always has to know what type of data corresponds to an ID. Reducing the size of the ID potentially limits the maximum number of connected devices, etc. But applying those items leads to more than three times longer period until the 100KB barrier is reached (See example Part 1C).

At this stage, it becomes impossible to reduce the message overhead without losing information. In the showcase, I mentioned that the simulated sensors are to be used in the field, potentially sending data via LPWAN. Typically, sensors in this area should not send data on such a high frequency. If that is the case, there are two additional options.

First, the messages are getting combined to one single message containing all sensor properties. The table above showed that the amount of data for the value part has just been a fraction of the message overhead.

One approach is to use JSON to propagate properties, this can be seen in example Part 1d. QJsonObject and QJsonDocument are very handy to use, and QJsonDocument::toJson() exports the content to a QByteArray, which fits perfectly to an MQTT message.

 

void SensorInformation::publish()
{
    QJsonObject jobject;
    jobject["AmbientTemperature"] = QString::number(m_ambientTemperature);
    jobject["ObjectTemperature"] = QString::number(m_objectTemperature);
    jobject["AccelerometerX"] = QString::number(m_accelerometerX);
    jobject["AccelerometerY"] = QString::number(m_accelerometerY);
    jobject["AccelerometerZ"] = QString::number(m_accelerometerZ);
    jobject["Altitude"] = QString::number(m_altitude);
    jobject["Light"] = QString::number(m_light);
    jobject["Humidity"] = QString::number(m_humidity);
    QJsonDocument doc( jobject );
    m_client->publish(QString::fromLatin1("qtds/%1").arg(m_id), doc.toJson());
}

 

The size of an MQTT publish message is now at around 272 bytes including all information. As mentioned before, it comes at the cost of losing information, but also at significantly reducing the bandwidth required.

To summarize this first part, we have looked into various solutions to reduce the amount of data being sent from a device to an MQTT broker with minimal effort. Some approaches trade off readability or extensibility, others come with a loss of information before data transmission. It really depends on the use-case and the scenario in which an IoT solution is put into production. But all of these can easily be implemented with Qt. Until now, we have only covered Qt’s feature set on JSON, MQTT, networking, and connectivity via Bluetooth.

In the meantime, you can also find more information on our website and write your comments below.

Do you like this? Share it
Share on LinkedInGoogle+Share on FacebookTweet about this on Twitter

Posted in Automation, Dev Loop, Embedded | Tags: , ,

5 comments

MIkeC says:

You could save a few more bytes by using doc.toJson(QJsonDocument::Compact) in the last line.

Customwritingservice says:

Maurice Kalinowski, thank you ever so for you post.Much thanks again.

Bram Van der Streeck says:

By converting all values to a QString, will you not also lose 2 bytes per value because they get surrounded by ” in the json?

Maurice Kalinowski Maurice Kalinowski says:

Thanks for the feedback, you are both perfectly right.

I quickly went through your suggestions and here’s the output:

Blog Post output(299 bytes):
“{\n \”AccelerometerX\”: \”-5.65587\”,\n \”AccelerometerY\”: \”2.1666\”,\n \”AccelerometerZ\”: \”-0.206962\”,\n \”Altitude\”: \”19.9548\”,\n \”AmbientTemperature\”: \”9.95402\”,\n \”Humidity\”: \”3.2807\”,\n \”Light\”: \”533.492\”,\n \”ObjectTemperature\”: \”12.0596\”\n}\n”
QJsonDocument::Compact Output (239 bytes)
“{\”AccelerometerX\”:\”-5.65587\”,\”AccelerometerY\”:\”2.1666\”,\”AccelerometerZ\”:\”-0.206962\”,\”Altitude\”:\”19.9548\”,\”AmbientTemperature\”:\”9.95402\”,\”Humidity\”:\”3.2807\”,\”Light\”:\”533.492\”,\”ObjectTemperature\”:\”12.0596\”}”
Using QJsonValue(double) instead of QString (353 bytes)
“{\n \”AccelerometerX\”: -5.655866499480448,\n \”AccelerometerY\”: 2.1665967228545893,\n \”AccelerometerZ\”: -0.20696171921429762,\n \”Altitude\”: 19.954768386910622,\n \”AmbientTemperature\”: 9.95401833819818,\n \”Humidity\”: 3.2806976138032145,\n \”Light\”: 533.4915353384683,\n \”ObjectTemperature\”: 12.059570236620361\n}\n”
Using QJsonValue(double) in conjunction with QJsonDocument::Compact (293 bytes)
“{\”AccelerometerX\”:-5.655866499480448,\”AccelerometerY\”:2.1665967228545893,\”AccelerometerZ\”:-0.20696171921429762,\”Altitude\”:19.954768386910622,\”AmbientTemperature\”:9.95401833819818,\”Humidity\”:3.2806976138032145,\”Light\”:533.4915353384683,\”ObjectTemperature\”:12.059570236620361}”

Bram Van der Streeck says:

Nice! Great article by the way! I forgot about the fact that QString::number limits the default precision to 6 digits after the comma , so now this leads to more data :(. I suppose rounding could be added depending on whether the focus is purely on data size or precision or performance.

Leave a Reply

Your email address will not be published.

Get started today with Qt Download now