LabPlot's MQTT in the finish line

Hello everyone. GSoC is coming to its end, so I think that I should give a report about what's been done since the last post, and also make a brief evaluation, summary of the project itself. 

As I've written in my last post, the main focus was on improving the quality of the code, cleaning, optimizing and properly documenting it. And also making it more comestible for other developers. 

The next step was searching for bugs and then fixing them. In order to do this properly, I implemented a unit test for the main MQTT related features. This proved to be useful since it helped discover several hidden bugs and errors which were all corrected. The main features, that tests were developed for, are: checking if a topic contains another one, checking if two topics are "common topics" (meaning they only differ at only one level, and are the same size), managing messages, subscribing&unsubscribing.

As I said in the previous post, a problem was that LabPlot couldn't plot QDateTime, so using an index column was necessary. However, fortunately Alexander Semke, as he promised, dealt with the matter, so plotting of data from a single topic is now possible without needing any plus/additional data. I'm truly thankful to Alexander for this.

The main improvements were related to algorithms needed for subscribing and unsubscribing. This process is now more optimal, and I really hope that is bug-free, since a lot of time was spent on testing every possible scenario for this feature. Not only the algorithms were improved regarding this feature, but the user interface as well. Now we use two tree widgets, which really does look better than the previous list widget (used for listing subscriptions). Using the tree widget for listing subscriptions made room for further improvements. Now not only the subscription is listed, but if the subscription contains wildcards, then every topic contained by the subscription and present in the topic tree will be added as children of the subscription, in order to make the user's life easier. Searching for root subscriptions in the tree widget is possible just like in the case of the topic tree widget.

 

New UI for subscribing&unsubscribing

Another improvement is dealing properly with multiple MQTTClient objects, which wasn't quite alright at the time of the last post. Now it works fine, and the user can easily control each of the MQTTClients using the LiveDataDock. Another bug/absurdity was fixed. Namely, the user could add more MQTTClients with the same hostname, which is quite illogical (since the user can control every topic of a broker with a single MQTTClient). Another minor visual improvement is that an icon for MQTTClient and MQTTSubscripiton was added.

 

Dealing with multiple MQTTClients

As I presented the major improvements, I think it's high time I showed you a possible and practical use of the features developed and the benefits of LabPlot having MQTT support. MQTT, as I mentioned in earlier posts, is mainly used to communicate data collected by sensors. So if one had the possibility and adequate sensors, then one can save&plot data collected by those sensors. However, there are less sophisticated uses as well. As we all know our phones have quite a few sensors, which could be put to use. And there is an application for this, which can be used by everyone who owns a smartphone: Sensor Node Free. In the app the user can choose from multiple sensors, the data of which can be sent to a preferred MQTT broker using a preferred topic name. As you can see in the next picture.

Choosing the sensors, setting the broker and the topic name in the app

Of course, any app, that has these features, could be used (for example a fitness app), but my mentor suggested this one.  The data of these sensors will be ploted in the demo video. Almost every sensor sends data divided to x, y, z axis. These 3 will be shown in the same plot, their data set as Y value, and the QDateTime allocated to the values as X value. the curves based on data from x axis will be red, the y axis green, and the z axis purple. The plotting was done for a while before starting to record the video. So here is the demo video:

Demo video

And finally here comes the evaluation/summary. I truly think that every feature presented in my proposal is implemented and working. So I think the main aim is met: LabPlot now has full support for MQTT. There were difficulties along the way, but with work, and help from my mentor, everything was dealt with. As I said everything works, but some unforeseen bugs or errors might appear in the future. Some steps for the future may be to improve the overall performance of the new features.

Working on this project for the last 3 months has been a very great experience, and I can honestly say that I improved my coding skills and way of thinking in a way I haven't even dreamt for. I'm more than grateful to my mentor, Kristóf, who always helped me and had my back if I encountered any hardship. I'd also like to express my gratitude towards Alexander Semke, my mentor's former mentor and an invaluable member of the LabPlot team, who also helped me a great deal. I am determined to further improve the features, even after GSoC ended. I would be more than happy to stay in this amazing team and help them whenever they need me. It's my goal for next summer to join GSoC  again and work on LabPlot with Kristof, since I really liked being part of this team. I truly think that people should contribute more to the open source community, and the end of GSoC shouldn't mean the end of the involvement as well.

This is it, guys. Thank you for reading the post, and thank you for your interest in my project. If there's any development regarding my project or a new task (in the future) I'll let you know here. Bye :)

Support for MQTT has evolved in LabPlot

Hey guys. It's been a while since my last post, however we haven't lazed since then. We solved the problems presented in the previous post, and also implemented the "next steps". To get in the picture you may want to read the previous post.

So let's just go through the new things step by step. I'll try to explain the respective feature, and also give examples using videos or screenshots.

The biggest problem was that these topics are completely INDIVIDUAL, so they may send totally different amount of data (this amount of data may differ from message to message as well, given one topic) and this hasn't made possible putting the data of these messages in the same container (spreadsheet). The idea used for the solution came from my mentor Kristóf and his former mentor Alexander Semke. 3 new classes were implemented, in order to represent better the tree like structure of the topics: 

• MQTTClient which contains the information about the QMqttClient which connects to the broker and manages it, manages the connection, contains the reading and update options, presented in the previous post, is responsible for subscribing and unsubscribing
• MQTTSubscription which represents the subscriptions made by the client. This is the core of the tree like structure, because if the subscribed topic contains wildcards, then every topic from which a message was received by the subscription will be added as its child
• MQTTTopic which is responsible for saving the data in columns according to the reading and update options of the MQTTClient. This is basically a spreadsheet containing at least 2 columns:
• value: the data received from the message
• timestamp: the time when the message was received in order to be able to plot the values of a single topic, without using any other topic's values
• one optional index column

This not has only solved the problem of sets of messages with not equal amount of data, but also we no longer have to enforce synchrony for the topics, since they are handled individually.

The "tree" structure of the new classes in the project explorer

The next change was brought by the previous one. Since we represent the subscribed topic in a tree like structure in the project explorer, I couldn't stand the fact that in the ImportFileWidget we could only select the topics to be subscribed via a QComboBox. Therefore I improved the ui of ImportFileWidget. Now we use a QTreeWidget to list the available topics, in order to represent them in their natural tree structure. Searching for "roots" is also possible for better navigation, the QTreeWidget scrolls to the given root item. Another upgrade of the ui is that its content is now scrollable, this change was needed because the amount of ui items needed for the MQTT setup has become quite large.

 

Upgraded ImportFileWidget design and functionality

The next feature is setting the will message of the client. Each client can specify its last will message (a normal MQTT message with topic, retained flag, QoS and payload) when connecting to a broker. The broker will store the message until it detects that the client has disconnected ungracefully. If the client disconnect abruptly, the broker sends the message to all subscribed clients on the topic, which was specified in the last will message. The stored LWT message will be discarded if a client disconnects gracefully by sending a DISCONNECT message.

The user can choose the will topic from the topics of the MQTTSubscriptions. The user can set the QoS, retain flag and payload as well. For the payload we implemented 3 variants:

1. Own message: the user can set any message using a QLineEdit
2. Last message received (from the will topic)
3. Statistics: several options which can be selected by list of checkboxes (for instance Minimum, Maximum, Median, Variance, etc.)

The user can also set when and how to set/update the will message. There are 2 options: by click and after a set time interval.

 

Setting Will Message

Another implemented feature is subscribing and unsubscribing to topics during runtime in LiveDataDock. If we subscribe to a topic that "contains" one that we are already subscribed to, this can only happen if the new contains # or + wildcard(s) and it includes the "old" topic. If this happens we don't throw away the data which was received until that moment by the to be deleted MQTTSubscription, but we "reparent" the MQTTTopics of the "old" MQTTSubscribtion to the new subscription.

 

Subscribe & Unsubscribe

The last big feature which was implemented is saving and loading the objects of the newly implemented classes. Saving and loading is a really important part of LabPlot so it was necessary to enable this for MQTT as well.

 

Save & Load

The preview of data in ImportFileWidget was also improved, now it works with messages containing different amount of data as well.
There was added another option, the user can choose from interpreting the retained messages or not.   

The next steps. There may be a lot. One short come still is that  LabPlot can't plot QDateTime for now, but as I've talked with Alexander, he said that this matter will be dealt with before the end of the project. One important thing that is still left is to modify PlotDataDialog, because now, at first, we can only plot from columns of a single MQTTTopic, then we can only later set other MQTTTopic's column as X or Y value. After this is dealt with, since, slowly, nearly every feature presented in the project proposal is implemented, we'll focus on improving, optimizing cleaning the code, making it easily readable, "comestible" for other developers. Writing documentation for the new classes and functions. Searching for bugs and correcting them, in order to LabPlot work smoothly. Writing tests for certain functions.

This is it for now guys. I will continue to work on the project with Kristóf and Alexander, who really help me a lot. I'm very thankful to them. When anything new will be finished and running I'll let you know. 

See you soon!

Bye!

LabPlot getting support for MQTT

Hello everyone! I'm participating in Google Summer of Code, I'm working on KDE's LabPlot. At first, let me introduce you my project:

"Currently, LabPlot has some support for the plotting of live data, which can be read from file/named pipe, TCP socket, UDP socket, local socket and serial port. The goal of this project is to provide support for the plotting of data received via MQTT protocol from a certain broker. LabPlot currently doesn’t provide any support for MQTT, even though it would be quite important for LabPlot to provide this kind of support, since it is a scientific data plotting software and MQTT protocol is widely used to transmit certain sensor data.

This project would increase LabPlot’s usability regarding analyses of scientific data, monitoring data collected by sensors, that is transmitted by MQTT protocol, or even make LabPlot usable in the field of Internet of Things or Smart Home Appliances."

If you are more interested in the project, you can check out my proposal, to find out in detail what the project really is about.

It is also important to know that this project is based on my mentor's work, who implemented the live data source functionality. To find out more check out: http://krajszgsoc.blogspot.com/2017

I can happily tell you that I managed to adapt quite a few of the already existing features of live data sources to MQTT protocol.

But let's start with the beginning. The very first task was to update the Import File Widget. Using the widget the user can set:

• the MQTT client's host and port to the host and port of the broker, which the user wants to connect to,
• the username and password (if authentication is required by the broker) 
• the client ID. 

Setting the MQTT client options

After successfully connecting to the broker. The MQTT client subscribes to the "#" wildcard. This means that the client gets every message published on the broker, so we can add every active topic to a combo box. The user can choose from these topics the ones the MQTT client will subscribe to. Since lots of topics are added to the combo box, by starting to type the topic's name we can narrow down the list the user has to choose from. Subscribing and unsubscribing to a topic is also implemented.

The next logical step is to offer a preview of the data, that is received via MQTT messages, so the user is able to see what kind of data is transmitted in every topic that the user subscribed to. This is useful, because the user can decide whether the expected data was received or not. If not the user can unsubscribe from the topic, as shown in the previous video.

The next step was adapting already existing options and functionality of live data sources to MQTT protocol. In the already existing implementation every data was read from one QIODevice, data for every column was present in every line read from the QIODevice. However this method can't be used for MQTT, since data for every column is received from individual topics in multiple messages. These messages may or may not be sent synchronously. After discussing the matter with my mentor we decided that we will enforce synchrony. What does this mean? It means that we only process a set of data, if the client received a message for every subscribed topic. The next problem is that these topics are completely INDIVIDUAL, so they may send totally different amount of data (this amount of data may differ from message to message as well given one topic). After talking this problem through with my mentor, we decided on implementing the features assuming that every set of messages (the currently processed messages, one from each topic) contains the same amount of data, the amount may differ from one set to another (for starters at least).

Here's a list of the options I adapted to MQTT:

• Reading type
• Continuously fixed
• From end
• Till the end

•  Update type
• Periodically
• On new data

• Keep last N values
• Update now
• Pause/Continue
• Update interval
• Sample rate

I will demonstrate these features with some videos, using brownians motion data, provided by another Qt application I wrote:

This Video Presents how Read Type and Sample rate work

This video presents how the update types and update interval work

This video presents the keep last N values option

This video presents the Pause/Continue and Update now options

The next steps most probably will be investigating for further errors the already adapted options, setting will messages for the client using some statistical data, run time subscribe/unsubscribe, probably thinking through and finding a solution for sets of messages with not equal amount of data, and of course any new idea that may come up regarding the project.

This is it for now. I will continue to work on the project with my mentor Kristóf, who really helps me a lot, and if I'm not sure about something he guides me. When anything new will be finished and running I'll let you know.

See you soon!