Andy Nichols (nezticle)

Introducing Qt Quick 3D: A high-level 3D API for Qt Quick

Published Wednesday August 14th, 2019
16 Comments on Introducing Qt Quick 3D: A high-level 3D API for Qt Quick
Posted in Dev Loop, Graphics, OpenGL, Qt Quick 3D

As Lars mentioned in his Technical Vision for Qt 6 blog post, we have been researching how we could have a deeper integration between 3D and Qt Quick. As a result we have created a new project, called Qt Quick 3D, which provides a high-level API for creating 3D content for user interfaces from Qt Quick. Rather than using an external engine which can lead to animation synchronization issues and several layers of abstraction, we are providing extensions to the Qt Quick Scenegraph for 3D content, and a renderer for those extended scene graph nodes.

Does that mean we wrote yet another 3D Solution for Qt?  Not exactly, because the core spatial renderer is derived from the Qt 3D Studio renderer. This renderer was ported to use Qt for its platform abstraction and refactored to meet Qt project coding style.

Complex 3D scene visualized with Qt Quick 3D

“San Miguel” test scene running in Qt Quick 3D

What are our Goals?  Why another 3D Solution?

Unified Graphics Story

The single most important goal is that we want to unify our graphics story. Currently we are offering two comprehensive solutions for creating fluid user interfaces, each having its own corresponding tooling.  One of these solutions is Qt Quick, for 2D, the other is Qt 3D Studio, for 3D.  If you limit yourself to using either one or the other, things usually work out quite fine.  However, what we found is that users typically ended up needing to mix and match the two, which leads to many pitfalls both in run-time performance and in developer/designer experience.

Therefore, and for simplicity’s sake, we aim have one runtime (Qt Quick), one common scene graph (Qt Quick Scenegraph), and one design tool (Qt Design Studio).  This should present no compromises in features, performance or the developer/designer experience. This way we do not need to further split our development focus between more products, and we can deliver more features and fixes faster.

Intuitive and Easy to Use API

The next goal is for Qt Quick 3D is to provide an API for defining 3D content, an API that is approachable and usable by developers without the need to understand the finer details of the modern graphics pipeline.  After all, the majority of users do not need to create specialized 3D graphics renderers for each of their applications, but rather just want to show some 3D content, often alongside 2D.  So we have been developing Qt Quick 3D with this perspective in mind.

That being said, we will be exposing more and more of the rendering API over time which will make more advanced use cases, needed by power-users, possible.

At the time of writing of this post we are only providing a QML API, but the goal in the future is to provide a public C++ API as well.

Unified Tooling for Qt Quick

Qt Quick 3D is intended to be the successor to Qt 3D Studio.  For the time being Qt 3D Studio will still continue to be developed, but long-term will be replaced by Qt Quick and Qt Design Studio.

Here we intend to take the best parts of Qt 3D Studio and roll them into Qt Quick and Qt Design Studio.  So rather than needing a separate tool for Qt Quick or 3D, it will be possible to just do both from within Qt Design Studio.  We are working on the details of this now and hope to have a preview of this available soon.

For existing users of Qt 3D Studio, we have been working on a porting tool to convert projects to Qt Quick 3D. More on that later.

First Class Asset Conditioning Pipeline

When dealing with 3D scenes, asset conditioning becomes more important because now there are more types of assets being used, and they tend to be much bigger overall.  So as part of the Qt Quick 3D development effort we have been looking at how we can make it as easy as possible to import your content and bake it into efficient runtime formats for Qt Quick.

For example, at design time you will want to specify the assets you are using based on what your asset creation tools generate (like FBX files from Maya for 3D models, or PSD files from Photoshop for textures), but at runtime you would not want the engine to use those formats.  Instead, you will want to convert the assets into some efficient runtime format, and have them updated each time the source assets change.  We want this to be an automated process as much as possible, and so want to build this into the build system and tooling of Qt.

Cross-platform Performance and Compatibility

Another of our goals is to support multiple native graphics APIs, using the new Rendering Hardware Interface being added to Qt. Currently, Qt Quick 3D only supports rendering using OpenGL, like many other components in Qt. However, in Qt 6 we will be using the QtRHI as our graphics abstraction and there we will be able to support rendering via Vulkan, Metal and Direct3D as well, in addition to OpenGL.

What is Qt Quick 3D? (and what it is not)

Qt Quick 3D is not a replacement for Qt 3D, but rather an extension of Qt Quick’s functionality to render 3D content using a high-level API.

Here is what a very simple project with some helpful comments looks like:

import QtQuick 2.12
import QtQuick.Window 2.12
import QtQuick3D 1.0
 
Window {
  id: window
  visible: true
  width: 1280
  height: 720
     
  // Viewport for 3D content
  View3D {
    id: view
         
    anchors.fill: parent
    // Scene to view
    Node {
      id: scene
          
      Light {
             
        id: directionalLight
               
      }

      Camera {
        id: camera
        // It's important that your camera is not inside 
        // your model so move it back along the z axis
        // The Camera is implicitly facing up the z axis,
        // so we should be looking towards (0, 0, 0)
        z: -600
      }

      Model {
        id: cubeModel
        // #Cube is one of the "built-in" primitive meshes
        // Other Options are:
        // #Cone, #Sphere, #Cylinder, #Rectangle
        source: "#Cube"
                 
        // When using a Model, it is not enough to have a 
        // mesh source (ie "#Cube")
        // You also need to define what material to shade
        // the mesh with. A Model can be built up of 
        // multiple sub-meshes, so each mesh needs its own
        // material. Materials are defined in an array, 
        // and order reflects which mesh to shade
                 
        // All of the default primitive meshes contain one
        // sub-mesh, so you only need 1 material.
                 
        materials: [
                     
          DefaultMaterial {
                         
            // We are using the DefaultMaterial which 
            // dynamically generates a shader based on what
            // properties are set. This means you don't 
            // need to write any shader code yourself.  
            // In this case we just want the cube to have
            // a red diffuse color.
            id: cubeMaterial
            diffuseColor: "red"
          }
        ]
      }
    }
  }
}

The idea is that defining 3D content should be as easy as 2D.  There are a few extra things you need, like the concepts of Lights, Cameras, and Materials, but all of these are high-level scene concepts, rather than implementation details of the graphics pipeline.

This simple API comes at the cost of less power, of course.  While it may be possible to customize materials and the content of the scene, it is not possible to completely customize how the scene is rendered, unlike in Qt 3D via the its customizable framegraph.  Instead, for now there is a fixed forward renderer, and you can define with properties in the scene how things are rendered.  This is like other existing engines, which typically have a few possible rendering pipelines to choose from, and those then render the logical scene.

Camera Orbiting Car

A Camera orbiting around a Car Model in a Skybox with Axis and Gridlines (note: stutter is from the 12 FPS GIF )

What Can You Do with Qt Quick 3D?

Well, it can do many things, but these are built up using the following scene primitives:

Node

Node is the base component for any node in the 3D scene.  It represents a transformation in 3D space, and but is non-visual.  It works similarly to how the Item type works in Qt Quick.

Camera

Camera represents how a scene is projected to a 2D surface. A camera has a position in 3D space (as it is a Node subclass) and a projection.  To render a scene, you need to have at least one Camera.

Light

The Light component defines a source of lighting in the scene, at least for materials that consider lighting.  Right now, there are 3 types of lights: Directional (default), Point and Area.

Model

The Model component is the one visual component in the scene.  It represents a combination of geometry (from a mesh) and one or more materials.

The source property of the Mesh component expects a .mesh file, which is the runtime format used by Qt Quick 3D.  To get mesh files, you need to convert 3D models using the asset import tool.  There are also a few built-in primitives. These can be used by setting the following values to the source property: #Cube, #Cylinder, #Sphere, #Cone, or #Rectangle.

We will also be adding a programmatic way to define your own geometry at runtime, but that is not yet available in the preview.

Before a Model can be rendered, it must also have a Material. This defines how the mesh is shaded.

DefaultMaterial and Custom Materials

The DefaultMaterial component is an easy to use, built-in material.  All you need to do is to create this material, set the properties you want to define, and under the hood all necessary shader code will be automatically generated for you.  All the other properties you set on the scene are taken into consideration as well. There is no need to write any graphics shader code (such as, vertex or fragment shaders) yourself.

It is also possible to define so-called CustomMaterials, where you do provide your own shader code.  We also provide a library of pre-defined CustomMaterials you can try out by just adding the following to your QML imports:

import QtQuick3D.MaterialLibrary 1.0

Texture

The Texture component represents a texture in the 3D scene, as well as how it is mapped to a mesh.  The source for a texture can either be an image file, or a QML Component.

A Sample of the Features Available

3D Views inside of Qt Quick

To view 3D content inside of Qt Quick, it is necessary to flatten it to a 2D surface.  To do this, you use the View3D component.  View3D is the only QQuickItem-based component in the whole API.  You can either define the scene as a child of the View3D or reference an existing scene by setting the scene property to the root Node of the scene you want to renderer.

If you have more than one camera, you can also set which camera you want to use to render the scene.  By default, it will just use the first active camera defined in the scene.

Also it is worth noting that View3D items do not necessarily need to be rendered to off-screen textures before being rendered.  It is possible to set one of the 4 following render modes to define when the 3D content is rendered:

  1. Texture: View3D is a Qt Quick texture provider and renders content to an texture via an FBO
  2. Underlay: View3D is rendered before Qt Quick’s 2D content is rendered, directly to the window (3D is always under 2D)
  3. Overlay: View3D is rendered after Qt Quick’s 2D content is rendered, directly to the window (3D is always over 2D)
  4. RenderNode: View3D is rendered in-line with the Qt Quick 2D content.  This can however lead to some quirks due to how Qt Quick 2D uses the depth buffer in Qt 5.

 

2D Views inside of 3D

It could be that you also want to render Qt Quick content inside of a 3D scene.  To do so, anywhere where an Texture is taken as a property value (for example, in the diffuseMap property of default material), you can use a Texture with its sourceItem property set, instead of just specifying a file in the source property. This way the referenced Qt Quick item will be automatically rendered and used as a texture.

animated_cubes

The diffuse color textures being mapped to the cubes are animated Qt Quick 2D items.

3D QML Components

Due to Qt Quick 3D being built on QML, it is possible to create reusable components for 3D as well.  For example, if you create a Car model consisting of several Models, just save it to Car.qml. You can then instantiate multiple instance of Car by just reusing it, like any other QML type. This is very important because this way 2D and 3D scenes can be created using the same component model, instead of having to deal with different approaches for the 2D and 3D scenes.

Multiple Views of the Same Scene

Because scene definitions can exist anywhere in a Qt Quick project, its possible to reference them from multiple View3Ds.  If you had multiple cameras in a scene, you could even render from each one to a different View3D.

teapots

4 views of the same Teapot scene. Also changing between 3 Cameras in the Perspective view.

Shadows

Any Light component can specify that it is casting shadows.  When this is enabled, shadows are automatically rendered in the scene.  Depending on what you are doing though, rendering shadows can be quite expensive, so you can fine-tune which Model components cast and receive shadows by setting additional properties on the Model.

Image Based Lighting

In addition to the standard Light components, its possible to light your scene by defining a HDRI map. This Texture can be set either for the whole View3D in its SceneEnvironment property, or on individual Materials.

Animations

Animations in Qt Quick 3D use the same animation system as Qt Quick.  You can bind any property to an animator and it will be animated and updated as expected. Using the QtQuickTimeline module it is also possible to use keyframe-based animations.

Like the component model, this is another important step in reducing the gap between 2D and 3D scenes, as no separate, potentially conflicting animation systems are used here.

Currently there is no support for rigged animations, but that is planned in the future.

How Can You Try it Out?

The intention is to release Qt Quick 3D as a technical preview along with the release of Qt 5.14.  In the meantime it should be possible to use it already now, against Qt 5.12 and higher.

To get the code, you just need to build the QtQuick3D module which is located here:

https://git.qt.io/annichol/qtquick3d

What About Tooling?

The goal is that it should be possible via Qt Design Studio to do everything you need to set up a 3D scene. That means being able to visually lay out the scene, import 3D assets like meshes, materials, and textures, and convert those assets into efficient runtime formats used by the engine.

designstudio3d

A demonstration of early Qt Design Studio integration for Qt Quick 3D

Importing 3D Scenes to QML Components

Qt Quick 3D can also be used by writing QML code manually. Therefore, we also have some stand-alone utilities for converting assets.  Once such tool is the balsam asset conditioning tool.  Right now it is possible to feed this utility an asset from a 3D asset creation tool like Blender, Maya, or 3DS Max, and it will generate a QML component representing the scene, as well as any textures, meshes, and materials it uses.  Currently this tool supports generating scenes from the following formats:

  • FBX
  • Collada (dae)
  • OBJ
  • Blender (blend)
  • GLTF2

To convert the file myTestScene.fbx you would run:

./balsam -o ~/exportDirectory myTestScene.fbx

Thiswould generate a file called MyTestScene.qml together with any assets needed. Then you can just use it like any other Component in your scene:

import QtQuick 2.12
import QtQuick.Window 2.12
import QtQuick3D 1.0

Window {
 width: 1920
 height: 1080
 visible: true
 color: "black"

 Node {
   id: sceneRoot
   Light { 
   }
   Camera {
   z: -100
   }
   MyTestScene {
   }
 }
 
 View3D {
   anchors.fill: parent
   scene: sceneRoot
 }
}

We are working to improve the assets generated by this tool, so expect improvements in the coming months.

Converting Qt 3D Studio Projects

In addition to being able to generate 3D QML components from 3D asset creation tools, we have also created a plugin for our asset import tool to convert existing Qt 3D Studio projects.  If you have used Qt 3D Studio before, you will know it generates projects in XML format to define the scene.  If you give the balsam tool a UIP or UIA project generated by Qt 3D Studio, it will also generate a Qt Quick 3D project based on that.  Note however that since the runtime used by Qt 3D Studio is different from Qt Quick 3D, not everything will be converted. It should nonetheless give a good approximation or starting point for converting an existing project.  We hope to continue improving support for this path to smooth the transition for existing Qt 3D Studio users.

qt3dstudio_sample

Qt 3D Studio example application ported using Qt Quick 3D’s import tool. (it’s not perfect yet)

What About Qt 3D?

The first question I expect to get is why not just use Qt 3D?  This is the same question we have been exploring the last couple of years.

One natural assumption is that we could just build all of Qt Quick on top of Qt 3D if we want to mix 2D and 3D.  We intended to and started to do this with the 2.3 release of Qt 3D Studio.  Qt 3D’s powerful API provided a good abstraction for implementing a rendering engine to re-create the behavior expected by Qt Quick and Qt 3D Studio. However, Qt 3D’s architecture makes it difficult to get the performance we needed on an entry level embedded hardware. Qt 3D also comes with a certain overhead from its own limited runtime as well as from being yet another level of abstraction between Qt Quick and the graphics hardware.  In its current form, Qt 3D is not ideal to build on if we want to reach a fully unified graphics story while ensuring continued good support for a wide variety of platforms and devices ranging from low to high end.

At the same time, we already had a rendering engine in Qt 3D Studio that did exactly what we needed, and was a good basis for building additional functionally.  This comes with the downside that we no longer have the powerful APIs that come with Qt 3D, but in practice once you start building a runtime on top of Qt 3D, you already end up making decisions about how things should work, leading to a limited ability to customize the framegraph anyway. In the end the most practical decision was to use the existing Qt 3D Studio rendering engine as our base, and build off of that.

What is the Plan Moving Forward?

This release is just a preview of what is to come.  The plan is to provide Qt Quick 3D as a fully supported module along with the Qt 5.15 LTS.  In the meantime we are working on further developing Qt Quick 3D for release as a Tech Preview with Qt 5.14.

For the Qt 5 series we are limited in how deeply we can combine 2D and 3D because of binary compatibility promises.  With the release of Qt 6 we are planning an even deeper integration of Qt Quick 3D into Qt Quick to provide an even smoother experience.

The goal here is that we want to be able to be as efficient as possible when mixing 2D and 3D content, without introducing any additional overhead to users who do not use any 3D content at all.  We will not be doing anything drastic like forcing all Qt Quick apps to go through the new renderer, only ones who are mixing 2D and 3D.

In Qt 6 we will also be using the Qt Rendering Hardware Interface to render Qt Quick (including 3D) scenes which should eliminate many of the current issues we have today with deployment of OpenGL applications (by using DirectX on Windows, Metal on macOS, etc.).

We also want to make it possible for end users to use the C++ Rendering API we have created more generically, without Qt Quick.  The code is there now as private API, but we are waiting until the Qt 6 time-frame (and the RHI porting) before we make the compatibility promises that come with public APIs.

Feedback is Very Welcome!

This is a tech preview, so much of what you see now is subject to change.  For example, the API is a bit rough around the edges now, so we would like to know what we are missing, what doesn’t make sense, what works, and what doesn’t. The best way to provide this feedback is through the Qt Bug Tracker.  Just remember to use the Qt Quick: 3D component when filing your bugs/suggestions.

Do you like this? Share it
Share on LinkedInGoogle+Share on FacebookTweet about this on Twitter

Posted in Dev Loop, Graphics, OpenGL, Qt Quick 3D

16 comments

Jakub Narolewski says:

Amount of different ‘3D’ solutions under Qt’s umbrella is staggering : P
But hey, n + 1 is always better than n, right? ;]

The RHI approach sounds awesome! Can’t wait to try it out in Qt 6.
This new module will be available under LGPL?

Eli says:

This is exactly what we are needing for our product! Nice work!

Off topic:
Can you guys ditch jira for a nice fully featured gitlab instance? It would be so much more convenient to contribute with gitlab!

Andy Nichols (nezticle) Andy Nichols (nezticle) says:

Regarding your suggestion of Jira and Gitlab: I don’t disagree 😉 bring it up with Qt Project.

zack says:

I don’t see how JIRA is a problem and what gitlab can solve here.

Andy Nichols (nezticle) Andy Nichols (nezticle) says:

This is very off-topic but I suspect what @Eli is referring to here is that there is a disconnect between all of our developer infrastructure. Why not have one instance of GitLab or Github where it’s easy to contribute, review, and track open issues all in once place, rather than have lots of convoluted infrastructure and processes. I’m only comfortable with the current setup because I’ve been around since it was added, but I can’t imagine how hard it must be for someone new to the community to jump in and contribute in a casual way.

TTGil says:

It sounds like Qt3D’s architecture is fundamentally flawed in a way that is beyond saving. Is the eventual goal to reach feature parity with it and deprecate? Having a customizable framegraph is something that seems a shame to lose, especially since modern graphics require things like deferred rendering, multiple passes, and so on. Seems wrong to have Qt 6 launch with a 3D component that renders content on the level of a 1999 game engine.

Andy Nichols (nezticle) Andy Nichols (nezticle) says:

I don’t think that reaching feature parity with Qt3D even makes sense, because they are abstractions on a different level.
Regarding the frame graph API, this is not something the average user needs and in most cases is an implementation detail. Also it is unfair to characterize a forward renderer as an archaic technology considering this technique is used today on Mobile and Embedded hardware in addition to AR and VR applications. This technique is the priority to start with because it has the broadest reach, but we like other high level rendering engines we have the option to offer additional renders for more advanced techniques. There is also the question of whether such a renderer even should be a priority considering the plethora of options already available for Desktop and Console level hardware needed to leverage it.

TTGil says:

That’s fair, but I think the expectation nowadays, even on phones and tablets, is PBR rendering on a par with: https://www.pbrt.org/scenes_images/sanmiguel_cam25.jpg

Anyway, thanks for continuing to invest in 3D and making things fit together. The vision of a mixed 2D/3D qml world makes a lot of sense.

Andy Nichols (nezticle) Andy Nichols (nezticle) says:

I agree, and you can certainly see the difference between the San Miguel screenshot I posted vs the one from the book. We are currently working on a Principled BSDF version of the DefaultMaterial component that should provide an easy way to take advantage of the Metal/Roughness workflow (and enable more correct materials for GLTF2 imports). Expect an update on that soon 🙂

Alexandre GRANVAUD says:

Will it handle transparency?

Sean harmer says:

As Andy mentioned, they are for different use cases. We are still actively working on Qt 3D and have a bunch of work in progress to reduce CPU and memory overhead and to reduce latency and optimise the common case of 2D Qt Quick over a Qt 3D underlay.

Sylv says:

It seems difficult to maintain 2 different graphical APIs in Qt. All strategic decisions will be made as a priority for QtQuick 3D. Similarly, the documentation will probably be more complete for QtQuick3D than for Qt3D, and therefore the number of users will be more important for QtQuick3D. As a result, the evolutions will be mainly on QtQuick3D.
In addition, the roadmap of Qt6 does not mention Qt3D at all.
IMHO, it’s sad but it looks like a small death of Qt3D.

Will says:

I know you’ve mentioned Qt3D, but uhh…. What about Qt3D? I’d love to hear a bit more about it. You mention the “Qt Rendering Hardware Interface.” What all will that cover. Is it going to include generalizations of things like QOpenGLTextureBlitter, QOpenGLShaderProgram, QOpenGLTexture and QOpenGLWidget ? Or lower in the stack? higher? Will Qt3D be ported to using that QRHI as a living project? Or is Qt3D basically abandoned?

Mainly, what does the C++ story look like, outside of QML? Qt3D was always mainly documented and supported as QML rather than C++. This QtQuick3D is clearly QML. But literally every person I have ever met/worked with / talked to about doing high performance 3D graphics is mainly working in C++, and probably on some fairly “old fashioned” Widgets app. So what is the 3D story for the developers who actually do 3D stuff?

Laszlo Agocs Laszlo Agocs says:

As the post says, a C++ API may appear for Quick3D in the future. That would not be any different conceptually, though. You would build up the exact same scene with Models and Materials, just from C++, instead of describing the same object tree in QML (or having the QML code generated by the designer tool, which is and will be a very important aspect in the Quick3D story)

Below this we (will) have the RHI. This indeed replaces many of the QOpenGL* classes, but not everything has (or needs) an equivalent. In the end (Qt 6) moving away from direct OpenGL usage will have consequences for many bits and pieces of Qt, not just in Qt Quick (3D). For example, the details of what this all means for widgets in the future (QOpenGLWidget, QQuickWidget, OpenGL paint engine) is still under research.

https://alpqr.github.io/qtrhi/qtrhi-index.html gives a good overview of the RHI API for those who are interested in details. (this is a not fully up-to-date documentation snapshot, broken here and there, but readable). Remember though that this is all private API for now, so not meant to be directly used by applications. It remains to be seen if it makes sense to open this up at some point.

Under this there is the platform adaptation and windowing system glue, where applicable. (QOpenGLContext and its platform-specific implementations, QOpenGLFunctions, QVulkanInstance, surface and/or layer management behind QWindow in the platform plugins, etc.) This is not going to change much in Qt 6 either.

And then there are the native graphics APIs. The possibility of directly using OpenGL, Vulkan, Metal, etc. is going to be there, like today with OpenGL. This may of course come at the expense of portability (due to being tied to a single graphics API). So people doing their own high performance renderers building on the latest greatest graphics techniques will still be able to integrate that into a QWidget or QML application. For instance, we have spent (and will still need to spend) quite a lot of work on keeping the various approaches to custom rendering integration (underlay, overlay, QSGRenderNode, etc.) operational on the new RHI-based rendering path of Qt Quick as well.

From this perspective the story overall is not that different from Qt 5, except for the introduction of a first class way to add 3D content in Qt Quick applications, combined with a strong focus on the associated design tools. For many users, including those who are targeting embedded devices with varying power and performance characteristics in the many industries Qt is used, this should provide a satisfactory solution, with the added bonus of offering tools for designers as well (covering both 2D and 3D).

This may not cover 100% of the Qt users, of course. Developers who prefer writing their own 3D engines in C++, typically targeting high end desktops, and then combine this with a 2D UI provided by widgets or Quick, can continue to do so in Qt 6 and beyond as well.

Mousavi says:

Glad to hear it 👍🏻
I was happy to meet Qt Quick 3D 😊

Linucs says:

I just wish all the best for Qt3D API, I don’t care what is used under the hood though.

Leave a Reply

Your email address will not be published.

Get started today with Qt Download now