LabPlot's Welcome screen and Dataset feature in the finish line

Hello Everyone! This year's GSoC is coming to its end. Therefore I think that I should let you know what's been done since my last blog post. I would also like to evaluate the progress I managed to make and the goals set up at the beginning of this project. 

As I told you in my last post, my main goal, in this last period, was to clean up, properly document, refactor, optimise the code and make it easier to read, so it would be fit to be brought to the master branch and to be used by the community.

My next proposition was to search for bugs and fix them, in order to make the implemented features more or less flawless. I can happily state, that I succeeded in this.

As proposed, I implemented a unit test for the main dataset related features. This proved to be useful since it helped discover some hidden problems which were all corrected. The main features, that tests were developed for, are: 

• processing the metadata files in order to list the available collections, categories, subcategories and datasets
• downloading and processing of datasets based on the information stored in the metadata files
•  adding new datasets to the already existent ones (by creating new collections, categories and subcategories)

 I managed to create some "real" example projects so the users can explore the possibilities provided by LabPlot. These include the already existing ones (3 by count) and I added some dataset related ones. I also proceeded with the collecting of datasets. I managed to double the previous amount. Now we have 2000 datasets categorized, which is already a considerable amount.

As I said, the main priority was to improve and correct the code and not to develop major new features. However, some minor new ideas were still implemented.  The first step was to improve the downloading and naming of the datasets. I also developed some caching mechanism for the downloaded files. When downloading a file, if an earlier copy is available it needs to be checked whether or not it was created earlier than 1 day. If it was, then the file needs to be deleted in order to proceed with the downloading, otherwise the downloading isn't necessary.

Another, more observable, feature is that the users can now easily display the full name and the description of the selected dataset, so they may be able to retrieve additional information before choosing a dataset to work with. Previously the users could also see the short name of the dataset which wasn't really practical since it didn't really offer much information about the dataset itself. I also added a counter to every collection, category and subcategory so now the users can see how many datasets belong to the previously listed units of categorization. I thought it might be useful. The following demo video will present these new features in use: 

 

Show Details functionality of ImportDatasetWidget

I also created a backup functionality for ImportDatasetWidget. When the users press the "Refresh" button the metadata files "shipped with LabPlot" are copied to the "dataset folder", and the files which were previously there are kept as a backup. This was needed because by pressing refresh any newly added dataset disappears. This can be very unpleasant if done unintentionally, I speak out of experience. So by pressing the "Restore" button the effect of the "Refresh" button can be undone, meaning that the current metadata files are deleted, the backup files are restored and the widget is updated accordingly. The following video illustrates this functionality:

 

Restore backup functionality of ImportDatasetWidget

As I've already presented you the improvements and new features, I think it's high time I present to you an overall image of what has been accomplished during this summer's GSoC. I will present the new features in two groups: dataset related and welcome screen related. I will provide a demo video for each of them.

I'd like to start with the welcome screen since that will be the first thing people will see when they start using LabPlot. I think that the welcome screen turned out to be quite good looking, clean and simple (in the best possible meaning). It is quite responsive and by allowing the users to resize the different sections, it is customisable to a degree. I think it provides a great deal of functionality that can be useful for the great majority of the users, and it also helps in the process of getting to know LabPlot. In the next demo video I will present every aspect of LabPlot's welcome screen:

 

LabPlot's welcome screen

The other main part of the project was to implement functionality to import online datasets. This could be useful for those who like working with data or in the field of statistics. These datasets can be visualized by creating different plots: curves and histograms.  This way LabPlot may be used for educational purposes as well. LabPlot comes with a great number of datasets which may be expanded individually by adding own datasets. I really hope that the users will find this new functionality useful and practical. Here is a video that demonstrates adding a new dataset and creating some plots based on it:

 

LabPlot's Dataset Functionality

And finally here comes the evaluation/summary. I truly think that every feature presented in my proposal is implemented and working. So I think the main aim is met: LabPlot now has full support for downloading and processing online datasets and has a gorgeous welcome screen. There were difficulties along the way, but with work, and help from my mentors, everything was dealt with. As I said everything works, but some unforeseen bugs or errors might appear in the future. Some steps for the future may be to improve the overall performance of the new features and improving the overall look of the welcome screen.

Working on this project for the last 3 months has been a great experience, and I can honestly say that I improved my coding skills and way of thinking in a way I haven't even dreamt for. I also improved my ability to work alone and solve problems on my own. I'm more than grateful to my mentor, Kristóf, who always helped me and had my back if I encountered any hardship. I'd also like to express my gratitude towards Alexander Semke, my mentor's former mentor and an invaluable member of the LabPlot team, who also helped me a great deal. I am determined to further improve the features, even after GSoC ended. I would be more than happy to stay in this amazing team and help them whenever they need me. It's my goal to work, in the future, on LabPlot with them, since I really liked being part of this team. I truly think that people should contribute more to the open-source community, and the end of GSoC shouldn't mean the end of the involvement as well.

This is it, guys. Thank you for reading the post, and thank you for your interest in my project. If there's any development regarding my project or a new task (in the future) I'll let you know here. Bye :)

LabPlot has got some beautifying and lots of datasets

Hello everyone! The second part of this year's GSoC is almost over, so I was due to let you know the progress made in the last 3 weeks. I can assure you we haven't lazed since then. I think I managed to make quite good progress, so everything is going as planned, or I could say that even better. If you haven't read about this year's project or you just want to go through what has already been accomplished you can check out my previous post.

So let's just go through the new things step by step. I'll try to explain the respective feature, and also give examples using videos or screenshots. 

The first step was to improve the welcome screen and make it easily usable, dynamic, clean and intuitive for users. This step was very important since the welcome screen is what the users will first get in contact with when they start using LabPlot. 

We had a great idea, which was to make a screenshot of the main window whenever the user saves a project. This screenshot will be saved with the project itself and will be used as a thumbnail for Recent and Example projects in the welcome screen. The code section, that deals with making and saving the screenshot, is already committed on the master branch. You can see these thumbnails put to use in the following picture:

Thumbnails put to use

As you might recall, when I wrote the last post, the only section of the welcome screen which wasn't functional was the examples section. Implementing this feature was the next step. You could already catch a glimpse of it in the previous picture. Me and my mentors really like QtCreator's Example section, so that's where I got my inspiration. The example projects are shown in a GridView, and they look quite nice, thanks to the thumbnails. Every example project has a name and one or more tags assigned to itself. Just like in QtCreator's approach, the example projects are searchable based on their names and also on one or more tags. There is a search bar providing this functionality. Unfortunately, we didn't manage to create example projects just yet, so I used some temporary projects for implementing and testing, and also these will be shown in the following demo video:

 

The functionality of the Example Projects section

The next step was to make the section and also their content more dynamic. In my last post, these sections/widgets (whichever name you prefer) were static, having a fixed size and their content wasn't adapting really well to the resizing of the main window. As I said that was only a prototype. During the last weeks, I managed to make it really dynamic, as a modern welcome screen should be. The result is pretty nice, at least according to my mentors :). So how does this work? When the user drags the mouse over the frame of a section a line appears, with which the user can easily resize a section just by dragging it. If the user doesn't like the layout he/she created, it can always be reset to the original layout. You can see how it works in the next video:

  

Resizing the welcome screen and its content

Another new feature is also connected to the welcome screen. I made it possible for LabPlot to save the layout of the welcome screen whenever it's closed (either because opening/creating a project or because of closing the application). When the welcome screen is displayed next time, it's layout isn't the standard one (to which the user can reset the current layout in the settings) but the saved one. It seemed a good idea and it might be useful since no one would modify the layout, if he/she liked it. This feature is presented in the next video:

  

Save welcome screen's layout

Lastly but not least, the welcome screen got another new feature. Now the user can maximize a section, so he/she can interact with the given section much more easily. When the section doesn't need to be maximized, then the user can minimize it and the former layout is restored. This function is particularly useful since there are many sections and they might not be big enough without the others getting particularly small. This was the issue that resulted in figuring out and implementing this idea. The maximizing and minimizing can be done with the icon in the upper-left corner of every section. In addition to this, I also added a go forward and go backwards icon/button to the "Release section" since its implemented using a WebView and users can navigate away from the starting page:

 

Maximizing&Minimizng the welcome screen's widgets

Some other changes were made to the categorizing of datasets too. We thought it would be better to organise the datasets into collections (for example collection of R Datasets etc.) then into categories and subcategories. This made possible having a single file for a dataset collection, rather than having a metadata file for every dataset (as it previously was implemented). This, of course, caused some changes in the ImportDatasetWidget and DatasetMetadataManagerWidget, but their functionality stayed the same:

 

Changes on ImportDatasetWidget and DatasetMetadataManagerWidget

As you might remember from the post, our main problem was that uploading with KNS3 is disabled for an indefinite amount of time due to errors caused by the library. This made us question whether we should use it at all or not. Given the mentioned problem and the fact that we found the library's functionality quite limited for our purposes, we decided not to use it. Instead, we'll provide a considerable collection of datasets, which should suffice. I already managed to collect and categorize not less than 1000 datasets. And I'm planning to collect some more.

Finally, I'd like to say some words about the next steps. We still have to make some "real" example projects so the users can explore the possibilities provided by LabPlot. I'll have to proceed with the collecting of datasets, in order to provide the users of LabPlot with a considerable dataset collection. As the finish line is getting closer, there won't be new "big features", maybe some minor new ideas, if some will come up, but instead I'd like to focus on cleaning, documenting, refactoring and optimising the code so it will be fit to be brought to the master branch.  I'd also like to search for hidden bugs and errors, to make the new and already implemented features more or less flawless. Some tests should also be written for the dataset management part of the code.

This is it for now. I will continue to work on the project alongside Kristóf and Alexander. I truly enjoy working with them, mostly the "brainstorming part", I think we form a quite good team. I'm thankful to them, for their guidance. When anything new will be finished and running I'll let you know. 

See you soon!

Bye!

LabPlot getting prettier and also support for online datasets

Introduction

Hello everyone! I'm participating in Google Summer of Code for the second time. I'm working on KDE's LabPlot, just like last year. I'm very happy that I can work again with my former and current mentor Kristóf Fábián, and with Alexander Semke, an invaluable member of the LabPlot team, who is like a second mentor to me. At first, let me introduce you my current project: 

"There are many internet pages providing data sets for educational and academic purposes concerning various fields of science, and not only (astrophysics, statistics, medicine, etc.). Some tools used in the scientific field provide some "wrappers" for such online sources and allow the user to easily investigate these data sets and work with them in all kinds of applications, whilst the technical details and methodology like the fetching of data from the server and parsing are done completely transparent for the user. The user doesn’t even know what happens in the “background”.
The goal of this project is to add similar functionality to LabPlot. This would make LabPlot more fit for educational purposes, students and teachers could use LabPlot for visualizing and analyzing data connected to the currently studied field. And also could bring LabPlot into the life of the average student."

If the synopsis caught your attention and you are interested in the project itself, you can check out my proposal, to find out in detail what the project really is about.

Bonding period

Let's start with the bonding period. I used this time to investigate/analyze already existing solutions for uploading/downloading with KNS3 also its API documentation, checking out various welcome screens of other applications to get some inspiration, checking out some simpler caching implementations.

I communicated with my mentor and others from the LabPlot team to properly  design the project and the course of implementation. I also tried to get involved in the KDE community.

First month

At the end of the first month I can state that fortunately I was able to make a quite good progress. Everything is successfully implemented from what was proposed for the first month, and I could also proceed to other tasks. Now let's see what's been done.

Dealing with datasets

The very first step was to implement a new widget, called ImportDatasetWidget which could provide the functionality to:

• list the available categories and subcategories of datasets
• list the available datasets for a certain subcategory
• refresh the list of datasets and delete the downloaded metadata files

ImportDatasetWidget

The user can select from the categories and subcategories of the available datasets, as visible in the picture above. In order to visualize these I used a QTreeWidget. When the user clicks on a subcategory then every dataset, belonging to it, is listed in a QListView. The user is also provided with the possibility to search a particular category/subcategory, since we estimate that there will be a considerable amount of datasets by the end of the project. This is also the case with the dataset list.

We had to create metadata files in order to record additional information about datasets, and also to divide them into categories and subcategories. We use a metadata data file which contains every category and subcategory and a list of datasets for every subcategory. Additionally there is a metadata file for every dataset containing various data about the dataset itself.

In the "Datasets" section we highlight every dataset the metadata of which is locally available (in the labplot directory located in the user's home directory). When the user clicks on the "Clear cache" button every file is deleted from the above mentioned directory. The "Refresh" button provides the possibility to refresh the locally available metadata file, which contains the categories and subcategories.

In order to make possible the import of datasets into LabPlot, and saving them into Spreadsheets I had to implement a helper class: DatasetHandler. This class processes a dataset's metadata file, configures the Spreadsheet into which the data will be loaded, downloads the dataset, processes it (based on the preferences present in the metadata file) then loads its content into the spreadsheet.

 

ImportDatasetWidget basic functionality

There also is a "Add new Dataset" button. This makes it possible for the user to add own datasets to LabPlot's list. When the button is clicked a new dialog is shown to the user: DatasetMetadataManagerDialog

Adding new dataset to the collection

This dialog provides an interface for the user, so the user can easily set the options necessary for the dataset's metadata file. The user doesn't have to create the metadata file himself/herself, the dialog does this instead based on the data provided by the user. The dialog also adds the new dataset's name to the categories' metadata file. Therefore the user can easily add new datasets, and later load them easily, using the ImportDatasetWidget. 

 Adding new dataset

While implementing these functionalities we were faced with some problems, which still need to be solved. We wanted to make possible the download and upload to store.kde.org, using the KNS3 library.  So users could really add new datasets to the basic list, and could download only those metadata files which are needed. This was our intention, however KNS3 gave us a hard time. It provides its functionality only by two dialogs, which we wouldn't prefer to use. We want to incorporate the dialogs' functionality into LabPlot somehow, but we didn't figure this out yet. but, we are thinking about it. Another problem with KNS3 is that according to KDE's mailing list, uploading with KNS3 is disabled for an indefinite amount of time due to errors caused by the library. Therefore the question arises: Do we need downloading if uploading is not possible? 

Initial Welcome Screen

Due to successfully implementing quite fast the dataset part and also to the difficulties caused by KNS3, I started to design and to create a prototype for our Welcome Screen. The first step was to decide what technology we want to adapt in order to implement this welcome screen. We considered creating a widget based GUI, or using QML for this purpose. We chose QML, because despite  it being more complex and cumbersome to work with, it offers a greater freedom to implement ideas. So this is the main reason this technology was chosen. Then we had to think through, what functionalities we want to be provided by the welcome screen. We came up with:

• Recently opened projects
• Help section: Documentation, FAQ, etc.
• Exploring datasets
• Example projects
• Latest release information 
• News section

The current state of the welcome screen

Every part, except the Examples section, is fully functional. In the recent projects section the user can choose from the lastly opened projects and load them only by a click. The help section navigates the user to the Documentation, FAQ, Features and Support part of LabPlot's web page. In the release section the user can read about the last release. The news section is connected to the RSS feed of LabPlot's webpage, so the user can see the new posts of the web page. A central piece is the "Start exploring data" section where the user can browse the available datasets, display information about them, and open them with only one click. The functionality of the welcome screen is presented in the following video:

Finally I'd like to say some words about the next steps. The first will be implementing the "Examples" section of the welcome screen. In order to do so we'll have to design a metadata file for the example  projects and nonetheless we'll need to create some example projects to have something to work with. When functionally the welcome screen works, we can proceed to refining the design itself, making it more pleasant and adapting it to the user's theme's color. We'll still have to figure out what to do with the KNS3 library, should we use it or not, and if yes then how, in what manner. Nonetheless, we have to collect more datasets in order to provide the users of LabPlot with a considerable dataset collection.

This is it for now. I will continue to work on the project alongside Kristóf and Alexander. I think we form a quite good team. I'm thankful to them, for their guidance. When anything new will be finished and running I'll let you know. 

See you soon!

Bye!

LabPlot's MQTT in the finish line

Hello everyone. GSoC is coming to its end, so I think that I should give a report about what's been done since the last post, and also make a brief evaluation, summary of the project itself. 

As I've written in my last post, the main focus was on improving the quality of the code, cleaning, optimizing and properly documenting it. And also making it more comestible for other developers. 

The next step was searching for bugs and then fixing them. In order to do this properly, I implemented a unit test for the main MQTT related features. This proved to be useful since it helped discover several hidden bugs and errors which were all corrected. The main features, that tests were developed for, are: checking if a topic contains another one, checking if two topics are "common topics" (meaning they only differ at only one level, and are the same size), managing messages, subscribing&unsubscribing.

As I said in the previous post, a problem was that LabPlot couldn't plot QDateTime, so using an index column was necessary. However, fortunately Alexander Semke, as he promised, dealt with the matter, so plotting of data from a single topic is now possible without needing any plus/additional data. I'm truly thankful to Alexander for this.

The main improvements were related to algorithms needed for subscribing and unsubscribing. This process is now more optimal, and I really hope that is bug-free, since a lot of time was spent on testing every possible scenario for this feature. Not only the algorithms were improved regarding this feature, but the user interface as well. Now we use two tree widgets, which really does look better than the previous list widget (used for listing subscriptions). Using the tree widget for listing subscriptions made room for further improvements. Now not only the subscription is listed, but if the subscription contains wildcards, then every topic contained by the subscription and present in the topic tree will be added as children of the subscription, in order to make the user's life easier. Searching for root subscriptions in the tree widget is possible just like in the case of the topic tree widget.

 

New UI for subscribing&unsubscribing

Another improvement is dealing properly with multiple MQTTClient objects, which wasn't quite alright at the time of the last post. Now it works fine, and the user can easily control each of the MQTTClients using the LiveDataDock. Another bug/absurdity was fixed. Namely, the user could add more MQTTClients with the same hostname, which is quite illogical (since the user can control every topic of a broker with a single MQTTClient). Another minor visual improvement is that an icon for MQTTClient and MQTTSubscripiton was added.

 

Dealing with multiple MQTTClients

As I presented the major improvements, I think it's high time I showed you a possible and practical use of the features developed and the benefits of LabPlot having MQTT support. MQTT, as I mentioned in earlier posts, is mainly used to communicate data collected by sensors. So if one had the possibility and adequate sensors, then one can save&plot data collected by those sensors. However, there are less sophisticated uses as well. As we all know our phones have quite a few sensors, which could be put to use. And there is an application for this, which can be used by everyone who owns a smartphone: Sensor Node Free. In the app the user can choose from multiple sensors, the data of which can be sent to a preferred MQTT broker using a preferred topic name. As you can see in the next picture.

Choosing the sensors, setting the broker and the topic name in the app

Of course, any app, that has these features, could be used (for example a fitness app), but my mentor suggested this one.  The data of these sensors will be ploted in the demo video. Almost every sensor sends data divided to x, y, z axis. These 3 will be shown in the same plot, their data set as Y value, and the QDateTime allocated to the values as X value. the curves based on data from x axis will be red, the y axis green, and the z axis purple. The plotting was done for a while before starting to record the video. So here is the demo video:

Demo video

And finally here comes the evaluation/summary. I truly think that every feature presented in my proposal is implemented and working. So I think the main aim is met: LabPlot now has full support for MQTT. There were difficulties along the way, but with work, and help from my mentor, everything was dealt with. As I said everything works, but some unforeseen bugs or errors might appear in the future. Some steps for the future may be to improve the overall performance of the new features.

Working on this project for the last 3 months has been a very great experience, and I can honestly say that I improved my coding skills and way of thinking in a way I haven't even dreamt for. I'm more than grateful to my mentor, Kristóf, who always helped me and had my back if I encountered any hardship. I'd also like to express my gratitude towards Alexander Semke, my mentor's former mentor and an invaluable member of the LabPlot team, who also helped me a great deal. I am determined to further improve the features, even after GSoC ended. I would be more than happy to stay in this amazing team and help them whenever they need me. It's my goal for next summer to join GSoC  again and work on LabPlot with Kristof, since I really liked being part of this team. I truly think that people should contribute more to the open source community, and the end of GSoC shouldn't mean the end of the involvement as well.

This is it, guys. Thank you for reading the post, and thank you for your interest in my project. If there's any development regarding my project or a new task (in the future) I'll let you know here. Bye :)

Support for MQTT has evolved in LabPlot

Hey guys. It's been a while since my last post, however we haven't lazed since then. We solved the problems presented in the previous post, and also implemented the "next steps". To get in the picture you may want to read the previous post.

So let's just go through the new things step by step. I'll try to explain the respective feature, and also give examples using videos or screenshots.

The biggest problem was that these topics are completely INDIVIDUAL, so they may send totally different amount of data (this amount of data may differ from message to message as well, given one topic) and this hasn't made possible putting the data of these messages in the same container (spreadsheet). The idea used for the solution came from my mentor Kristóf and his former mentor Alexander Semke. 3 new classes were implemented, in order to represent better the tree like structure of the topics: 

• MQTTClient which contains the information about the QMqttClient which connects to the broker and manages it, manages the connection, contains the reading and update options, presented in the previous post, is responsible for subscribing and unsubscribing
• MQTTSubscription which represents the subscriptions made by the client. This is the core of the tree like structure, because if the subscribed topic contains wildcards, then every topic from which a message was received by the subscription will be added as its child
• MQTTTopic which is responsible for saving the data in columns according to the reading and update options of the MQTTClient. This is basically a spreadsheet containing at least 2 columns:
• value: the data received from the message
• timestamp: the time when the message was received in order to be able to plot the values of a single topic, without using any other topic's values
• one optional index column

This not has only solved the problem of sets of messages with not equal amount of data, but also we no longer have to enforce synchrony for the topics, since they are handled individually.

The "tree" structure of the new classes in the project explorer

The next change was brought by the previous one. Since we represent the subscribed topic in a tree like structure in the project explorer, I couldn't stand the fact that in the ImportFileWidget we could only select the topics to be subscribed via a QComboBox. Therefore I improved the ui of ImportFileWidget. Now we use a QTreeWidget to list the available topics, in order to represent them in their natural tree structure. Searching for "roots" is also possible for better navigation, the QTreeWidget scrolls to the given root item. Another upgrade of the ui is that its content is now scrollable, this change was needed because the amount of ui items needed for the MQTT setup has become quite large.

 

Upgraded ImportFileWidget design and functionality

The next feature is setting the will message of the client. Each client can specify its last will message (a normal MQTT message with topic, retained flag, QoS and payload) when connecting to a broker. The broker will store the message until it detects that the client has disconnected ungracefully. If the client disconnect abruptly, the broker sends the message to all subscribed clients on the topic, which was specified in the last will message. The stored LWT message will be discarded if a client disconnects gracefully by sending a DISCONNECT message.

The user can choose the will topic from the topics of the MQTTSubscriptions. The user can set the QoS, retain flag and payload as well. For the payload we implemented 3 variants:

1. Own message: the user can set any message using a QLineEdit
2. Last message received (from the will topic)
3. Statistics: several options which can be selected by list of checkboxes (for instance Minimum, Maximum, Median, Variance, etc.)

The user can also set when and how to set/update the will message. There are 2 options: by click and after a set time interval.

 

Setting Will Message

Another implemented feature is subscribing and unsubscribing to topics during runtime in LiveDataDock. If we subscribe to a topic that "contains" one that we are already subscribed to, this can only happen if the new contains # or + wildcard(s) and it includes the "old" topic. If this happens we don't throw away the data which was received until that moment by the to be deleted MQTTSubscription, but we "reparent" the MQTTTopics of the "old" MQTTSubscribtion to the new subscription.

 

Subscribe & Unsubscribe

The last big feature which was implemented is saving and loading the objects of the newly implemented classes. Saving and loading is a really important part of LabPlot so it was necessary to enable this for MQTT as well.

 

Save & Load

The preview of data in ImportFileWidget was also improved, now it works with messages containing different amount of data as well.
There was added another option, the user can choose from interpreting the retained messages or not.   

The next steps. There may be a lot. One short come still is that  LabPlot can't plot QDateTime for now, but as I've talked with Alexander, he said that this matter will be dealt with before the end of the project. One important thing that is still left is to modify PlotDataDialog, because now, at first, we can only plot from columns of a single MQTTTopic, then we can only later set other MQTTTopic's column as X or Y value. After this is dealt with, since, slowly, nearly every feature presented in the project proposal is implemented, we'll focus on improving, optimizing cleaning the code, making it easily readable, "comestible" for other developers. Writing documentation for the new classes and functions. Searching for bugs and correcting them, in order to LabPlot work smoothly. Writing tests for certain functions.

This is it for now guys. I will continue to work on the project with Kristóf and Alexander, who really help me a lot. I'm very thankful to them. When anything new will be finished and running I'll let you know. 

See you soon!

Bye!