Jesús Fernández

Qt WebGL Streaming merged

Published Friday July 7th, 2017
54 Comments on Qt WebGL Streaming merged
Posted in Dev Loop, OpenGL, QPA, Qt, Qt Quick, Qt Quick 2, Qt Quick Controls | Tags: , , ,

Some time ago I published a couple of blog posts talking about Qt WebGL Streaming plugin. The time has come, and the plugin is finally merged into the Qt repository. In the meantime, I worked on stabilization, performance and reducing the calls sent over the network. It also changed a bit in the way the connections are handled.

New client approach

In the previous implementations, the client was accepting more than one concurrent connections. After the latest changes, the plugin is going to behave like a standard QPA plugin. Now, only one user per process is allowed. If another user tries to connect to the web server, it will see a fancy loading screen until the previous client disconnects.
The previous approach caused some problems with how the desktop applications and GUI frameworks are designed. Everyone can agree that desktop applications are not intended to work with concurrent physical users, even if the window instances were different for all users.

No more boilerplate code

Previously the application had to be modified to support this platform plugin. This code was needed to make the application work with the plugin:

class EventFilter : public QObject
{
public:
    virtual bool eventFilter(QObject *watched, QEvent *event) override
    {
        Q_UNUSED(watched);
        if (event->type() == QEvent::User + 100) {
            createWindow(true);
            return true;
        } else if (event->type() == QEvent::User + 101) {
            window->close();
            window->deleteLater();
            return true;
        }

        return false;
    }
};

And install the event filter into the QGuiApplication.

No more modifications in applications are needed anymore.

How to try

So, if you want to give it a try before Qt 5.10 is released (~November 2017) do the following:

Prerequisites

Since WebGL was modelled using OpenGLES2 as reference, first thing you will need is to have an OpenGLES2 version of Qt built. To do that, you need to pass the parameter -opengl es2 to the configure before building.
Example:

./configure -opensource -confirm-license -opengl es2

Depending on your system it is possible you will need some aditional headers and libraries to be able to use es2.

Testing the plugin

After building everything, you can try to run a Qt Quick example.

To try the photoviewer example we need to build it and run with the -platform webgl parameters:

./photoviewer -platform webgl

If you want to try the Qt Quick Controls 2 Text Editor:

./texteditor -platform webgl

Supported options

Currently, the plugin only supports an option to configure the port used by the embedded HTTP Server. If you want to listen in the default HTTP port you should write -platform webgl:port=80.

The future

The plugin will be part of Qt 5.10 release as a technology preview (TP), as it needs to be improved. Currently, the plugin contains an HTTP Server and a Web Sockets server to handle the browser connections. I’m planning to remove the servers from the plugin and start using a Lightweight QtHttpServer we are working on right now. Once it’s ready, you will be able to create an application server to launch different process inheriting the web socket to communicate with the browsers. This will allow for supporting more than one concurrent user instead of sharing applications among users.

Note: The plugin only works using the Threaded Render Loop. If you use Windows or a platform using a different render loop ensure you set QSG_RENDER_LOOP environment variable to threaded

Do you like this? Share it
Share on LinkedInGoogle+Share on FacebookTweet about this on Twitter

Posted in Dev Loop, OpenGL, QPA, Qt, Qt Quick, Qt Quick 2, Qt Quick Controls | Tags: , , ,

54 comments

Philip S says:

This is pretty cool, but what’s the use case for it?

Jesús Fernández Jesús Fernández says:

Interact with remote applications. For example from your phone.

Jean-Michaël Celerier says:

Very nice! It would also be fantastic to get the webassembly patches merged : http://qtandeverything.blogspot.fr/2017/06/qt-for-web-assembly.html

rd says:

Well, I’m guessing the whole point of this is that you can have Qt running on the server, instead of compiling it into JavaScript for the client – with just the rendering happening in the browser.

Oussama Khmis says:

Congrats !! Great work, Have you managed to use the QWebChannel (for full web functionalities support)?

Jesús Fernández Jesús Fernández says:

It’s in my backlog 🙂 First, we need to redesign the server part. It will give us more flexibility.

Tom says:

you guys do an awesome job!

Jos says:

Is it possible to share a docker image with this demo as a low level entry for testing out this demo?

Jesús Fernández Jesús Fernández says:

I don’t know how to do that 🙁 I will look into it. If I have something, I will definitely let you know!

jason says:

“Everyone can agree that desktop applications are not intended to work with concurrent physical users, even if the window instances were different for all users.”

I disagree. Your basic MVC design pattern should alleviate this. I commented on the expresso blog post (proceeding this blog entry) and everything being written today should be using a microservices architecture.

Your Qt app, I think as most of your embedded customers want to do – is to use Qt for the local GUI, and Qt for a remote web GUI. The architecture you are creating in this post is one that will not adequately synchronize the two. Your local UI will always be connected. That may be a nativeGL or WebGL UI. The only logical conclusion is to create a REST microservice and native and WebGL clients that are synced.

So your statement is misaligned. It’s not desktop apps we want to create with this, it’s embedded UIs and Web UIs with the same stack.

Jesús Fernández Jesús Fernández says:

Thank you for your comment.
No, there is no possibility to give different UIs to different users in the same process, there are some platform limitations… Maybe Qt 6 can bring some changes to that. The solution I’m going to provide soon is spawning different processes connected via web socket to different browsers. This solution will be more scalable (I tried it).

apollo13 says:

Are widget based applications also support (given that it is a platform plugin I’d assume so). How is the performance?

Jesús Fernández Jesús Fernández says:

Widget applications are a problem here because they are written relying on rasterization. You can write a custom application using GLES2 calls directly, but not widgets, unfortunately.

If we implement support for widgets right now, the performance would be really poor.

Tonu Jaansoo says:

I wrote WebGL backend for Qt a year ago .. I hit bottlenecks. Too much GL commands to flow over the web. Qt isn’t optimized painstakingly for this… or maybe now it is… cool to demo on localhost though.

Jesús Fernández Jesús Fernández says:

WebGL streaming is not a perfect solution, it’s not meant to be used on the Internet to have interactive applications. I spent time reducing the query calls to the remote WebGL context. If a frame needs to be redrawn only draw calls are sent over the network, the query calls are cached on the server side, so the response time is immediate.
I already used the QML Presentation System in a presentation over the Internet recently and the times were more than acceptable.

Tonu Jaansoo says:

Thanks for the reply. I got similar performance to typical VNC setup when not using too many animations in QML. I used Huffman coding to squeeze data about 2x (many zeros in calls). I don’t remember exactly, but I tried to buffer all verts and pixels on client side.. using VAO’s, PBO’s …

Anyway. Maybe it’s useful to try this kind of approach. Although I am looking for someone to port Qt to Emscripten for me 🙂

Jesús Fernández Jesús Fernández says:

I have a pending patch with permessage-deflate support in our Web Socket implementation. When it’s done, it will save a lot of bandwidth for all calls, not just for submitting buffer data. Also, the new scenegraph is helping a lot with performance. If you consider contributing, it would be much appreciated 🙂

Tonu Jaansoo says:

Unfortunately, I don’t find the most recent video I made … this was made soon after I got something rendering in a browser – https://drive.google.com/open?id=0B7nGK-1U4QjnM0xDR3ZNa0FfQzg

Jesús Fernández Jesús Fernández says:

Thanks for sharing!

Stefan says:

What is the idea behind QHttpServer? Is this developed in the public?

Jesús Fernández Jesús Fernández says:

For the IoT world, we would need some sort of HTTP server. We are starting to research a bit on this area.
I’m not sure about the license yet. So stay tuned 🙂

Michael C says:

Is the repository available to anyone? I assumed so since you have a ‘How To Try’ section, but I’m having trouble downloading it. These are my steps:

git clone –branch dev https://github.com/qt/qt5.git
cd qt5
./init-repository –module-subset=default

Everything seems to go OK with init-repository until it gets to qtwebglplugin:

Cloning into ‘qtwebglplugin’…
remote: Repository not found.
fatal: repository ‘https://github.com/qt/qtwebglplugin.git/’ not found
git clone –no-checkout https://github.com/qt/qtwebglplugin.git qtwebglplugin exited with status 32768 at ./init-repository line 198.

Jesús Fernández Jesús Fernández says:

github is not the proper place to clone it.
You should use:
git clone https://codereview.qt-project.org/qt/qtwebglplugin

hmng says:

This would be really cool!

Stephan says:

Can I use WebGL in an application that uses Qt with Desktop OpenGL? I read what you write in the prerequisites that ES2 is needed instead of Desktop.

Jesús Fernández Jesús Fernández says:

If you can build Qt with ES2 support. ES2 works on Linux and Windows (via ANGLE).

Stephan says:

Some parts of the application need desktop OpenGL, so switching to ES2 is not an option.
Or are you suggesting to compile Qt with both desktop and ES2 support? Is that even possible?

Jesús Fernández Jesús Fernández says:

Then the answer is no. You can use custom OpenGL command. But just ES2 commands.

Tobias Koenig says:

Hej Jesús,

have you actually tested running an QtQuick application with webgl plugin on Windows/ANGLE?
If I do not set SG_RENDER_LOOP to ‘basic’ explicitly, the app crashes right away inside the scenegraph code.
When setting SG_RENDER_LOOP to ‘basic’, the application starts up, but crashes sometimes later (haven’t investigated further yet). So have you ever tried it on Windows/ANGLE and this is just a bug in my local setup, or could there be a general problem with this setup?

Jesús Fernández Jesús Fernández says:

Hi Tobias,

I tried some time ago on Windows. Currently, I haven’t a Windows machine to test the plugin. Can you try using ‘threaded’ render loop?
I will try to debug it on Windows ASAP.

Sławomir Lach says:

Many years ago I’ve wrote integrated web server into my business logic to generate GUI/CUI/CLI library. This library could works both as client and server, so this a way to allow running many applications by one client. We need also proxy, which run our first application (app written to use my library in client mode). This proxy is not real proxy, because is both server for tao-network-client and our remote application, but the way it’s working is the same – only difference is way it starts. You can read description at:
https://nintyfan.wordpress.com/2015/04/30/server-buildin-into-libgreattao-and-tao-network-client/

Source code for libgreattao and tao-network-client are on sourceforge.net.

Jesús Fernández Jesús Fernández says:

Thank you!

Mike Lothian says:

Are there any plans to have both OpenGL and OpenGL ES supported at the same time, or rather have it as a runtime rather than compile time switch?

Jesús Fernández Jesús Fernández says:

It makes no sense to support Desktop OpenGL. This plugin is designed to work with ES2.

Tom Hirst says:

Could this be used in conjunction with the Qt Wayland Compositor framework to stream a Wayland desktop to a WebGL client?

Jesús Fernández Jesús Fernández says:

I have not enough info to answer this question.

Konstantin Podsvirov says:

I am try it on Windows 7 64bit with MSVC2017 and it is crash because QWebGLIntegration::createPlatformBackingStore(QWindow*) method always return nullptr.

Jesús Fernández Jesús Fernández says:

Are you trying to run a widgets application?

Konstantin Podsvirov says:

Yes.

Jesús Fernández Jesús Fernández says:

Widgets are not supported because of performance issues. We only support Qt Quick and QOpenGLWindows calling raw GLES2 commands.

Konstantin Podsvirov says:

Grief. I just wanted to try this with QtIFW installers 🙁

Jesús Fernández Jesús Fernández says:

I’m sorry. The way widgets work does not fit here 🙁

Michael says:

I tried it on Windows 7 64bit with MSVC2015 and it crashes in void QOpenGLFunctions::initializeOpenGLFunctions()
I compared the stack with -platform webgl and without and I noticed a call of syncSceneGraph() but it looks like there is no context yet.

with -platform webgl
Qt5Guid.dll!qt_gl_functions(QOpenGLContext * context) <- crash because returned context is nullptr
Qt5Guid.dll!QOpenGLFunctions::initializeOpenGLFunctions()
Qt5Quickd.dll!QSGBatchRenderer::Renderer::Renderer(QSGDefaultRenderContext * ctx)
Qt5Quickd.dll!QSGDefaultRenderContext::createRenderer()
Qt5Quickd.dll!QQuickWindowPrivate::syncSceneGraph() <- sync without currentContext?
Qt5Quickd.dll!QSGWindowsRenderLoop::renderWindow(QQuickWindow * window)
Qt5Quickd.dll!QSGWindowsRenderLoop::render()
Qt5Quickd.dll!QSGWindowsRenderLoop::exposureChanged(QQuickWindow * window)

Qt5Guid.dll!QPlatformWindow::setVisible(bool visible)

without:
Qt5Guid.dll!qt_gl_functions(QOpenGLContext * context)
Qt5Guid.dll!QOpenGLFunctions::QOpenGLFunctions(QOpenGLContext * context)
Qt5Guid.dll!QOpenGLExtraFunctions::QOpenGLExtraFunctions(QOpenGLContext * context)
Qt5Guid.dll!QOpenGLExtensions::QOpenGLExtensions(QOpenGLContext * context)
Qt5Guid.dll!QOpenGLContext::functions()
Qt5Guid.dll!QOpenGLContext::makeCurrent(QSurface * surface)
Qt5Quickd.dll!QSGWindowsRenderLoop::show(QQuickWindow * window)
Qt5Quickd.dll!QQuickWindow::showEvent(QShowEvent * __formal)

Qt5Guid.dll!QWindow::setVisible(bool visible)

Jesús Fernández Jesús Fernández says:

I’m currently investigating it the crash.

Jesús Fernández Jesús Fernández says:

Can you try to set the “QSG_RENDER_LOOP” environment variable to “threaded” before running the application?

Michael says:

Now it works great. Thanks

Casper says:

HI Jesús,
Just started with Qt. I made a little calculation tool with PyQt5. Is it also possible to run this in a webbrowser using WebGL Streaming?

Jesús Fernández Jesús Fernández says:

We don’t support PyQt5 🙁

Casper says:

Ah too bad! I think the best way to solve this would be to build the tool in QT Quick and use Qt WebGl do you agree? Will take some time but should be worth it.

Konstantin Podsvirov says:

Hi Jesús,
the Michael’s problem (context) solved for me with QSG_RENDER_LOOP=threaded, but I use MSVC2017.

Jesús Fernández Jesús Fernández says:

Thank you Konstantin 🙂

jani yang says:

“I’m going to provide soon is spawning different processes connected via web socket to different browsers” Why not re-implementing custom qabstracteventdispatcher to support epoll backend, so that already hand made code with qtcpsocket class could gain advantage this epoll and possibly iocp on windows?

Jesús Fernández Jesús Fernández says:

What benefits do you think this could bring?

Commenting closed.

Get started today with Qt Download now