Jesús Fernández

Qt Quick WebGL Streaming

Published Wednesday February 22nd, 2017
61 Comments on Qt Quick WebGL Streaming
Posted in Dev Loop, Qt Quick 2, WebGL

WebGL Streaming is optimized for Qt Quick and allows you to run remote Qt Quick applications in a browser.

I’m working on a platform plugin to run remote applications in your browser, using OpenGL command streaming.

When the remote application runs using this new platform plugin, the application will create a lightweight web server. When a user connects to the application, a web socket connection is opened between the server and the client, using QWebSocketServer.
The application running on a remote computer will serialize all GL calls to binary data and sends it using the web socket connection.

The local browser will send the event (mouse, touch or keyboard) to the remote application, so user interaction is allowed. Even multi-touch support! (up to 6 fingers):

And some Qt Quick demos running in the browser:

The “calqlatr” example:

“clocks” example:

“emitters” (particles) example:

“samegame” example:

Desktop browsers are also supported:
webgldemo

It allows multiple connections to the same application.

screenshot_20170222_183836

New windows are shown in a different canvas in the browser.screenshot_20170222_184419

To improve the performance, I’m also working to support web sockets compression. To be able to use the permessage-deflate extension. It’s actually working but needs more testing.

This feature is going to be introduced in Qt 5.10 as appears in the 2017 roadmap.

Do you like this? Share it
Share on LinkedInGoogle+Share on FacebookTweet about this on Twitter

Posted in Dev Loop, Qt Quick 2, WebGL

61 comments

Jason says:

YAS! I’ve been wanting this for years!

But [how] are audio events supported?

Jason says:

[How] will concurrency be supported?

Like does each connection get it’s own QGuiApplication, or is there only one? I’d write to to handle many connections, then let the application worry about the concurrency.

Jesús Fernández Jesús Fernández says:

You create a single QGuiApplication and different windows.

There is a signal to notify when new clients connect to the HTTP server. When the signal is emitted, you create a different QWindow.

The windows are independent. And a window can create more windows (like a popup or a message box). Those child windows will create a new canvas in the browser.

Jason says:

“Independent” isn’t really that independent. They are share an address space. A crash in one takes all of them down.

I realize this is an experiment, but I think having process isolation in the end product is a must-have.

Jesús Fernández Jesús Fernández says:

The windows are independent. The input methods are independent. Of course, the different windows are running in the same application. A crash will “kill” all remote sessions.

There are different approaches, using a single process is only one of them.

Jesús Fernández Jesús Fernández says:

Audio events are not supported. But a custom implementation could add it. Host machine will play the sound.

Alexander says:

Is the rendering performed on the remote or local machine? If the rendering happens on the local computer (in the browser), this will be amazing for embedded devices and a good web alternative to http://qmlweb.github.io/

Jesús Fernández Jesús Fernández says:

The OpenGL calls are sent over the network and transformed into WebGL calls in the browser. It’s not video-streaming. The browser is executing WebGL in a canvas. Textures, buffers, glyphs, etc. are sent as well.

You need a browser with WebGL support. Tested on IE11, Firefox, Chrome, Edge, Chromium.
The browser of the videos is a Google Chrome running on Android.

Alexander says:

Awesome! Then Qt is taking away a major argument for HTML5 based UIs.

Vlad says:

great alternative to android auto/car play

Jesús Fernández Jesús Fernández says:

Thank you 🙂

Frank says:

Is this already on the dev branch (or gerrit)? Would it be possible to play around with it (provided that I built from source) or is it too early?

Jesús Fernández Jesús Fernández says:

WIP. Almost finished, but it needs more internal testing and probably some changes.

Frank says:

Does the system which runs the application/webserver need a GPU?

Jesús Fernández Jesús Fernández says:

This platform plugin will act as a virtual GPU to your Qt application. So there is no need to have a local GPU.

luffy says:

This is nice.
Are there any plans for the foreseeable future to support QML in the Browser?
Something like: http://qmlweb.github.io/

dbskcnc says:

I’m just wondering the bandwidth usage?

Jesús Fernández Jesús Fernández says:

The peak is during initialization. Because the browser needs to receive the buffers, textures, glyphs, atlases, etc.
After the first draw call, the bandwidth usage is low or null, thanks to the amazing work in QtQuick optimization. (more info: http://blog.qt.io/blog/2013/09/02/new-scene-graph-renderer/)

Jesús Fernández Jesús Fernández says:

The videos shown in the post correspond to my phone connected via Wi-Fi to my desktop computer.

I also tested it over the Internet (+VPN), and the results were promising.

dev-sst says:

This looks fantastic, it can be a game changer when it comes to devices without a display.

The topic raised a lot of questions, I placed them in a list to increase readability.

There probably are some limitations with local files/services:
1. a. Can the same window be shown to multiple clients, and would it be possible to identify the source of input events in such a case?
1. b. IDK about the state of the art browser security settings, but will it be possible to alter/store files on the client file system (or provide them as “download”)?
1. c. Can the application trigger a local notification mechanism?
1. d. Is it possible for the user to switch the webgl/canvas part of the browser into full screen mode?
1. e. Will Uploading files via drag & drop (from the file browser into the web browser) be supported ?

Some performance and scalability questions:
2. a. How much input latency is noticeable (on top of the connection latency), e.g. is there any perceptible delay with a slider/flickable or other “live” changing components?
2. b. Since textures need to be transfered to the client, is it possible to cache these? Can the client download a resource cache from a content server and then use the resources when connecting to the application server?
2. c. How many clients can concurrently use the application (on a dedicated server, virtual server, or even a RPI2)?

A few questions about compatibility:
3. a. Does drawing of QQuickPaintedItem work (e.g. a QWidget painted as a texture in qml)?
3. b. What about raw OpenGL (e.g. QNanoPainter https://github.com/QUItCoding/qnanopainter)?

Jesús Fernández Jesús Fernández says:

1.a: Different connections are receiving different instances of the same QWindow. The user interaction is sent to your Window in the application. Your clicks won’t be dispatched to other users connected.
1.b: No, you cannot modify the client filesystem. You can alter the host filesystem.
1.c: What do you mean?
1.d: Full-screen browsing is supported. The top level window is resized if you change the size of the browser window.
1.e: Not supported at the beginning. No idea if it can be supported. The same applies to the clipboard, for security reasons (imposed by the browser) a script cannot read/modify your clipboard. If you copy a text in a TextField a new tab in your browser will be open allowing you to copy the remote clipboard content, it requires explicit user interaction.

2.a: ~(4-6)times the latency between the host and the client. Depends on the application. In local the input latency is low. The problem is, some GL commands need an answer from OpenGL so the browser should answer the request.
2.b: Further improvements can be done. Maybe a checksum of the textures can be used to store data in client-side and avoid uploading the same data everytime.
2.c: Tested with 3 concurrent connections.

3.a: It works, but don’t expect good results.
3.b: Raw OpenGLES2 calls works. You can create a QQuickFramebufferObject and start using raw OpenGL calls.

jason says:

1.b Could it be possible to also design in usage LocalStorage API, ( https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API ) so that local objects can be persisted on the client?

Jesús Fernández Jesús Fernández says:

It could be implemented.

dev-sst says:

Thanks for answering the questions.

Regarding 1. c. the idea is to show a notification like it is done with libnotify, e.g. Chromium (and Firefox) has support for push notifications (https://blog.chromium.org/2015/04/reaching-and-re-engaging-users-on.html – see also https://www.w3.org/TR/push-api/).

Jesús Fernández Jesús Fernández says:

Technically it’s possible.
Custom functionality can be accessed using QGuiApplication::platformFunction. So adding a “showPopup” function, it’s possible.

Laszlo Agocs Laszlo Agocs says:

Re. 3.b: All OpenGL code has to go through QOpenGLFunctions to get properly redirected. Therefore external engines may not be directly compatible with the current streaming approach.

Jesús Fernández Jesús Fernández says:

True! 😀

IPeculiar says:

Regarding the local storage and some other functionalities that’s not supported (at least yet), can we use qwebchannel.js (like with qwebengine) to get those things done using html/js ? sorry for bad explanation, hope you understand what I’m trying to say.
Thanks

Jesús Fernández Jesús Fernández says:

I didn’t realize about QWebChannel. Definitively I will try!

Thank you for the suggestion. 🙂

IPeculiar says:

Glad to help you, I Hope to see it in action soon.
Can you share with us a public website that’s running an app so we can test thing like bandwidth usage, how many users can connect to the server at the same time…

Jesús Fernández Jesús Fernández says:

I cannot have a public server for testing services. I will try to push it to a public repository ASAP. But I don’t know where should this go yet.

There is no limit on the users. But after 3 or 4 concurrent users the performance decreases considerably. But there are different approaches we can follow to improve it.
As I said before the use case of this is not to use it with a massive number of concurrent clients or publish apps over the Internet.

jason says:

While this is not due for Qt 5.10, is there a branch/repo where I can play with it myself?

Jesús Fernández Jesús Fernández says:

Not yet.

Andre H. Beckedorf says:

Yes. This looks fantastic!
I’ve been developing a somewhat similar technology as pet project for the past 5 years, streaming QtQuick 1/2 and QGraphicsScene based scenes to the browser, supporting WebSockets, Server-Sent Events and Long Polling as transport channels.
It is currently used in three commercial applications and is open sourced in project Viridity:
https://github.com/evilJazz/viridity/tree/master
(https://github.com/evilJazz/viridity/tree/v3 for the latest development version)

Jesús Fernández Jesús Fernández says:

I will check it! 🙂
After a first glance, it looks a different approach.

Good work!

bnogal says:

What a nice work.

A good structure allows infinite possibilities.

I dont know if it could be use in a teaching environment, when host (teacher) render and has event inputs, and students can follow the application rendering. Video streaming requires high bandwidth, this could help a lot in this area.

Jesús Fernández Jesús Fernández says:

Screen mirroring is not the use case, unfortunately. But for example, I have a mock-up where your phone acts as a remote “touch-device” sending touch events to an application. It can be used to control a slideshow remotely or draw.

bnogal says:

Well, once the commands are in a stream, it should be easy to stream it to multiple clients and the host, and pick inputs and opengl answers just from the host.

i will give a try when it is available.

Jesús Fernández Jesús Fernández says:

No, it’s no easy. Because you need to replay the buffer, texture, glyph, etc. creation and upload every time a new client connects. Also, the application should use an OpenGL ES2 version, so you have to use a GLES2 compatible device.
About the answers, WebGL glGet* commands can respond with different values than the host device.

joni says:

So, what do you think by having iocp/epoll/kqueue support in QWebSocketServer 🙂 ?

Jesús Fernández Jesús Fernández says:

What do you mean?

joni says:

My though was that Qt should not forget provide it 🙂 Something like not to relying select behavior in QAbstractEventDispatcher in headless environment

Jesús Fernández Jesús Fernández says:

But, to use this, you should use a dispatcher. It’s a QGuiApplication.

QWebSocketSeerver works with QCoreApplication as well.

Vlad says:

Is it possible to create lightweight Qt host app which will render all the received events instead of using browser with webgl?

Jesús Fernández Jesús Fernández says:

Do you consider WebEngine Qt apps lightweight? 😛

Vlad says:

nope 🙂

Jesús Fernández Jesús Fernández says:

Yes, it’s possible, but a different platform plugin is needed then.

Bruno says:

This is the coolest think, specially when pyside 2 becomes available. This will change our ideas of web development! Great work!

Jesús Fernández Jesús Fernández says:

Thank you.

But remember, the target of the plugin is not streaming applications over the Internet. You should use it in LAN/WLAN environment, not only because of connection speed but also because of the security.

Tham says:

>You should use it in LAN/WLAN environment, not only because of connection speed but also because of the security.

Do you mean this project do not suit for building public, world wide “website” like Qt blog, github, google mail and other?

it would be cool if Qt company could rebuild the websites by this project, could be a nice show case. Like wt do–“eat your own dog food”

Jesús Fernández Jesús Fernández says:

Unfortunately, it’s not possible. 🙁

Anonymous says:

Would there be any future plans to work a similar system using wasm in the future?

Alex K. says:

I have tried to do something similar:
https://github.com/bbvch/OpenGLStreamingWithQt

Juergen_M says:

Congratulations. I also waited for this feature for years.
We did some researches as well about this topic.
https://github.com/bbvch/OpenGLStreamingWithQt
Is it possible to have a concurrent local and remote view?
If yes, have you also considered about frame synchronization between a fast local server and slow remote client?

Jesús Fernández Jesús Fernández says:

No, it’s not possible. At least at the moment.

Gilbert Lee says:

Could you tell me the difference between WebToolkit( https://www.webtoolkit.eu/wt ) and Qt Quick WebGL?

Jesús Fernández Jesús Fernández says:

Tell me a similarity!!!

One is a C++ web API, to write typical HTTP AJAX applications. This one is Qt Quick streaming to an HTML canvas using WebGL to render the received GL calls.

Adam Simmons says:

Is there some protection mechanism built in so that only intended users can remotely access devices running this plugin?

Jesús Fernández Jesús Fernández says:

You can modify the code and add some IP address verification in the embedded HTTP Server or the WebSockets server.

Commenting closed.

Get started today with Qt Download now