Artem Sidyakin

Qt Quick WebGL release in Qt 5.12

Published Friday November 23rd, 2018
12 Comments on Qt Quick WebGL release in Qt 5.12
Posted in Automation, Dev Loop, QPA, Qt Quick | Tags: , , , ,

One of the Qt 5.12 new features is Qt Quick WebGL platform plugin (also known as WebGL streaming). It was actually available as a technology preview from Qt 5.10 already, but starting with Qt 5.12 it is a released feature.

TLDR

$ ./your-qt-application -platform webgl:port=8998

Intro

If you missed previous blog-posts, here they are:

There is also a good article by Jeff Tranter from ICS.

This post is intended to be a kind of an “unboxing” experience from the perspective of an average Qt “user” – I never tried Qt Quick WebGL streaming myself, neither did I participate in its development.

What is WebGL streaming

If you read past blog-posts, you can just skip this section. I would however recommend to at least read the documentation (the link will point to 5.12 docs after release).

WebGL streaming is a QPA plugin that sends (“streams”) OpenGL calls of your Qt Quick application over the network and in turn those are translated into WebGL calls and thus can be rendered at HTML5 Canvas. What it means in practice is that you can have an application running on a remote host and render its GUI in a local web-browser.

Here’s how it looks schematically:

Here’s also a video from KDE Akademy with a more detailed explanation by Jesus Fernandez.

But since I’m a simple Qt “user”, I don’t really care about any of that (it’s all hidden from me anyway), and to me everything looks like this:

So I can have a Qt-based application running on some device and work with it from Safari on my iPad. Sounds alright.

Naturally, instead of a device (Raspberry Pi in this case) there could be a desktop computer “hosting” the application, but I think WebGL streaming will be used mostly on embedded platforms (see use cases section).

Let’s now highlight a couple of points we’ve just learnt about the feature:

  • The application itself does not run inside a web-browser. Web-browser only renders its GUI;
  • So it is neither video-streaming, nor mirroring. It is about “decoupling” application’s GUI and showing it in a web-browser;
  • Since it’s for OpenGL (ES) things only, WebGL streaming does not work with Widgets or any other non-OpenGL stuff.

In fact, if you try to launch some “non-compatible” Qt application using WebGL QPA, most likely you’ll get the following error:

qt.qpa.webgl: WebGL QPA platform plugin: Raster surfaces are not supported

How to use it

You only need to install it:

…or, if you are not into installers, build Qt from sources as usual – no special configuration options needed. Actually, with earlier versions -opengl es2 option was required, but there is no need in that as Qt Quick Scene Graph can use ES subset even if there is a later version of OpenGL available.

Having installed Qt itself, build any Qt Quick application of yours and launch it with the following command line arguments:

$ ./your-qt-application -platform webgl

Yes, you don’t need to make any modifications in your source code, it just works. Open the following address in your web-browser: 127.0.0.1:8080, where 127.0.0.1 is the IP address of the host running your application.

If you want to use a different port, you can specify it like that:

$ ./your-qt-application -platform webgl:port=8998

Needless to say, Qt WebGL is cross-platform, and it works equally fine on Linux, Mac OS and Windows. Although, there are some differences in launching applications:

Linux:

./your-qt-application -platform webgl:port=8998

Mac OS:

QT_QPA_PLATFORM=webgl:port=8998 ./your-qt-application.app/Contents/MacOS/your-qt-application

…because Qt version I have at the moment apparently ignores -platform option (which sounds like a bug that needs to be reported).

Windows:

your-qt-application.exe -platform webgl:port=8998

And of course you can do it in a cross-platform (duh) way via qputenv() in your main.cpp:

// ...
qputenv("QT_QPA_PLATFORM", "webgl:port=8998");

QGuiApplication app(argc, argv);
// ...

Speaking about web-browsers support, I tried several ones (except for the one with trident and his younger brother), and it worked in all of them, so it looks like WebGL is well supported in modern browsers nowadays. Although, I did experience a couple of occasional page reloads, and also Chrome on Android tablet even crashed once, so apparently not that well, but that’s really outside the Qt’s scope.

With regards to performance, the most “busy” time is during the initialization phase, when web-browser is receiving buffers, textures, glyphs, atlases and so on. After the first draw call, the bandwidth usage is pretty low. And by the way, since OpenGL ES calls are sent as binary data, it should be more “light-weight” than VNC. I am actually thinking about writing another blog-post to compare WebGL streaming and VNC in terms of network utilization.

Some demos

There is already a fair amount of demo videos in previous blog-posts, and here’s also another nice compilation, so I decided to create a couple of my own.

Device Information

This one is a rather simple demo application. It gathers some information about the platform it is running on. For example, here’s what it shows when I run it on my Mac:


If video doesn’t play in your browser, you can download it here

Let’s now run it on Raspberry Pi using WebGL streaming plugin, connect to it from a web-browser on the same Mac and ascertain that it no longer reports Mac OS as operating system and that platform is webgl now. There is one more thing to see here – pay attention to changing values of the “screen” resolution:

As you can see, when I resize the browser window, application (beside nicely adapting its layout) reports changed screen resolution values. This is because it takes canvas dimensions for the screen resolution. Let’s check that in web-browser inspector:

By the way, we can take a look at user input events here as well:

So application does indeed run on Raspberry, and what I have in my browser is just its “streamed” GUI.

Camera

This demo is a bit more practical one – it’s a camera controlled by robotic-ish arm which is mounted on a Raspberry Pi device:

The idea is to control the camera (its pan and tilt) with a Qt-based application running on the device, but to do that remotely from a web-browser on some tablet. And of course we would like to see what cameras is looking at (its viewfinder). And it also would be nice to make photos with the camera.

Here’s a list of required hardware for such a setup:

And that’s the manual I used to assemble it.

Now let’s take a little detour (spoiler: I will be promoting Qt’s commercial features). You might have noticed that for Device Information demo I used Boot to Qt image, saving myself quite some time and efforts with regards to building Qt-based application for Raspberry Pi and deploying it there.

But Boot to Qt is a commercial-only feature, and without it you’ll have to go though some more steps setting up system environment. Here’s what it takes with a regular Raspbian Stretch Lite image as an example:

  • Since Qt Multimedia module is used, you need to make sure that GStreamer is installed in the system and you have correct plugins available;
  • Get the latest Qt build (5.12) for Raspberry Pi. Either set a cross-compilation toolchain on your desktop or build it right on the device. Building Qt from sources directly on Raspberry Pi is actually a viable option (especially if you failed with cross-compilation toolchain), although compilation itself takes around 10 hours and requires some dances with increasing available swap size (1 GB of RAM is really not enough);
  • Build V4L driver to make camera discoverable by GStreamer and thus Qt;
  • Come up with a convenient way of building/deploying your applications on device.

Even though this list of steps is not too long, in practice it can take you up to several days before you get a working setup, whether with Boot to Qt image you get everything working out-of-the-box, and you can run your applications on the connected device right from the Qt Creator. But enough with the promotional part, let’s get back to demo.

The GUI layout looks like the following:

Most of the space on the first tab is taken by the camera’s viewfinder, which is implemented with VideoOutput and Camera itself:

Camera { id: camera }

VideoOutput {
    anchors.fill: parent
    fillMode: VideoOutput.PreserveAspectCrop
    source: camera
}

There are two sliders, horizontal and vertical – for controlling pan and tilt of the camera:

Slider {
    id: sliderTilt
    orientation: Qt.Vertical
    from: root.maxValue
    value: 0
    stepSize: 1
    to: -root.maxValue

    onPressedChanged: {
        if (!pressed)
        {
            backend.movePanTilt(basePath, sliderPan.value, sliderTilt.value)
        }
    }
}

Pan-Tilt HAT servos are interfaced via I2C, and due to the lack of time I went with fast and dirty solution – by using Pimoroni’s Python library. At some point I would like to do it properly with C/C++, although it’s really not the point of the demo.

There is also a button for taking photos using CameraCapture:

Button {
    scale: hovered ? (pressed ? 0.9 : 1.1) : 1

    background: Rectangle {
        color: "transparent"
    }

    Image {
        anchors.fill: parent
        source: "/img/camera.png"
    }

    onClicked: {
        camera.imageCapture.captureToLocation(basePath + "shots/" + getCurrentDateTime() + ".jpg");
    }
}

Second tab contains a list of taken photos:

…which is implemented by FolderListModel:

ListView {
    FolderListModel {
        folder: "file:" + basePath + "shots/"
        nameFilters: ["*.jpg"]
    }
    
    model: folderModel
    
    delegate: ItemDelegate {
        text: model.fileName
    }
}

Full application source code is available here.

Now let’s see it in action. There are 3 spirits placed on my table, surrounding the camera, and I want to take photos of each. I built and ran the application on device with -platform webgl, and connected to it over Wi-Fi from Safari on my iPad:

As you can see, the plan worked out just fine: seeing camera’s viewfinder, I can remotely control its position and take photos of the objects I’m interested in.

Use cases

Most obvious use case for WebGL streaming is the ability to have a decent GUI for some low-end device with limited computing power, without GPU, and quite often without any display at all. For instance, that is a common scenario for industrial automation domain, where you can have lots of headless devices installed all over the factory: they can be distributed over quite a significant area or even mounted in places with hazardous environment – being able to control/configure those remotely comes to be rather handy.

Reading discussion at Hacker News, I stumbled upon a “reverse” idea: what if it’s the other way around, what if “device” is actually a very powerful server, and you work with it from your regular desktop. That way you can can perform some heavy calculations on the server while having GUI in your web-browser (which actually begs for an HTML-based frontend but more on this in the next section).

Another possible use-case is an anti-piracy measure. Let’s say you want to protect your software from being “cracked” or “pirated”. Obviously, if there is nothing running on the client, then there is nothing to crack as your users only have GUI rendered in their browsers, and the application itself is running on your server. Sounds interesting, but there are several drawbacks here:

  • While WebGL streaming performs well in local network, using it over the internet will result in significant latency;
  • Connection is not encrypted, so it is not secure;
  • Currently only one connection at a time is supported (so only one user).

Overall, supporting only one connection at a time fairly reduces the number of possible use cases, and unfortunately it is unlikely that current implementation of the feature will improve in that regard, so it is more of a task for Qt 6. By the way, there is an idea to complement streaming with an ability of mirroring as in some cases having the latter is more important.

Speaking about mirroring, I would like to mention our recent webinar that we had together with Toradex. There you can see an interesting combination of WebGL streaming and Remote Objects, which allows you to implement mirroring functionality as of now already.

Another noticeable aspect of WebGL streaming is so-called “zero install” concept – you don’t have to install/deploy anything on clients (desktops/tablets/smartphones/etc) as the only thing needed is just a web-browser. However, Qt for WebAssembly seems to be a bit more suitable for that purpose.

WebGL streaming vs actual web

Some of you might ask, what is the point of relying on WebGL streaming in the first place? Since it’s all about web-browser, one can just take a regular web-server and create a web-application – result will be almost the same: backend is hosted on the remote device and HTML-based GUI is rendered in the web-browser.

That is a very good and fair question. I actually have some experience in web-development, so I asked this question myself. Let’s try to answer it, hopefully without starting yet another holy war.

Indeed, in some cases it is enough just to have a simple REST API, especially if you only need to get some plain text data values. So it is likely that Qt-based application with WebGL streaming would be an overkill for such purpose.

However, in more sophisticated scenarios (for example, when you need to control some hardware) Qt-based application with WebGL-streamed GUI might fit better, because that way you’ll get a powerful backend (C++/Qt), and I would also mention that creating a complex, appealing and performant frontend is (considerably) easier with Qt Quick rather than with HTML/CSS/JS, but this statement does look like a beginning of yet another holy war, so I’ll keep that as my personal opinion.

And the last thing worth to mention here – if you already have a Qt-based application, then WebGL streaming is an obvious option, because it will cost you nothing to have a remote GUI for it.

Licensing/pricing

WebGL streaming plugin is available under commercial and Open Source licenses (but GPLv3 only). And for commercial customers it is included in both Application Development and Device Creation products with no additional charge.

Conclusion

So you’re now able to use web-browser as a remote GUI client for your Qt Quick applications with no efforts – it only takes one command line parameter.

In terms of further development, I reckon the next thing to be expected is connection security/encryption, both for WebSocket and WebServer. WebSocket part should be pretty straightforward as QWebSocket already supports secure connection (wss://). And WebServer part, if you remember, from the very beginning was a temporary solution, and research on proper implementation (including support for HTTPS) is still ongoing.

Meanwhile, if you have any other feature-requests or maybe bugs to report, please use our tracker for that: http://bugreports.qt.io/ (choose QPA: WebGL component). Your feedback will help our product management team to shape the feature’s roadmap.

Do you like this? Share it
Share on LinkedInGoogle+Share on FacebookTweet about this on Twitter

Posted in Automation, Dev Loop, QPA, Qt Quick | Tags: , , , ,

12 comments

Jakub says:

Great news, WebGL looks very promising and has a really huge potential!

It would be perfect if WebGL could work with simultaneously with rendering on host, so it would act as a remote control.

Also WebGL for Qt allows only one client to connect with application, so I really don’t see a point of using it except simple enthusiast projects with RasPi or quick ‘client-server’ app mockups.

Dmitry says:

I like WebAssembly. I can to create back-end on Golang/C++ and used C++ on front-end. For example https://github.com/RPG-18/qt-wasm-chat-example/releases.
This example working on Chrome, Firefox and Safari but don’t work on iOS Safari

Marcel Charlebois says:

I have a question to The Qt Company: did you ask your customers about their preferences? I doubt that many people are using or need this plugin. Maybe you should consider deprecation of this module?

As far as I know, the whole thing started with a request from Bosch (you might have seen their project on this video), so we did not come up with it ourselves. And we continue to receive requests for this functionality from other customers – not every day, but demand is definitely there.

d3fault says:

I think we should keep it, but my rationale is simply that it’s cool af. WebAssembly is cool af too.

Businesses tend to use existing technology to fill in needs, rather than creating technology outright. Qt is all about creating new technology, so querying customers for their opinions is going to take you in the opposite direction.

Tere says:

that’s actually a really nice feature that can provide a fast solution to remote control, either embedded or not. I tried tech preview version on some of my animated uis and the perfomance was not that good, lower fps was noticeable on animated items, should we anticipated performance improvements upon release or maybe it was my local issuie, can you comment performance wise please, or anyone share their experience. anyway, thanks for making this all possible, web technologies are really of importance these days, keep up the good work and good luck!

d3fault says:

typo: doesn’t not

re: qt.qpa.webgl: WebGL QPA platform plugin: Raster surfaces are not supported
isn’t it trivial to embed raster painting into non-raster painting? will be a performance hit, but this WebGL streaming is already not trying to win the performance race.

“I would also mention that creating a complex, appealing and performant frontend is (considerably) easier with Qt Quick rather than with HTML/CSS/JS”
You forgot to mention that you can [relatively seamlessly(meh, marshaling QMLC++)] use Qt/C++ for business logic! That’s the best benefit of all, because C++ is the best programming language and is more productive than JavaScript and Python.

Thanks, fixed the typo.

As I remember, there was some other problem with raster painting other than performance hit, but I can’t tell for sure. I’ll ask the main developer of the feature, hopefully he can elaborate on this.

Yes, you’re right, C++/Qt backend is at your service. It just seemed too obvious at the moment, but now I think it’s better to state this explicitly. I’ll add it to the text.

d3fault says:

I thought about it some more and answered my own question: If you want to use raster painting (such as QWidgets) in the browser, you’re better off sticking with the VNC QPA and using a Web-based (JS) VNC Client… because embedding non-raster painting into raster painting will give you the same performance. WebGL is a huge performance boost over VNC… but you have to use OpenGL painting (QtQuick/etc) to take advantage of it.

Jason says:

Per https://codereview.qt-project.org/#/c/215908/

You can specify the port as interface:port so that your exposure is limited to a specific interface (i.e. localhost)

Jason says:

I had problems with using QTextDocuments and any sizable document (100s of lines) It took too long after a click to move the cursor, etc.

I still think that something like QMLWeb ( https://qmlweb.github.io ) is the way to go and should be part of Qt.

However the WebGL is very good for embedded systems that need a remote accessible GUI… just don’t expect it to be performant. You might need to take into account WebGL bottlenecks when designing your UI.

Sai says:

No use at all for webgl in QT . CPU = 100% ….

Commenting closed.

Get started today with Qt Download now