Andy Nichols (nezticle)

Making Movies with QML

Published Tuesday February 21st, 2017
10 Comments on Making Movies with QML
Posted in Dev Loop, Graphics, Qt Quick

One of the interesting things about working with Qt is seeing all the unexpected ways our users use the APIs we create.  Last year I got a bug report requesting an API to set a custom frame rate for QML animations when using QQuickRenderControl.  The reason was that the user was using QQuickRenderControl as an engine to render video output from Qt Quick, and if your target was say 24 frames per second, the animations were not smooth because of how the default animation driver behaves.  So inspired by this use case I decided to take a stab at creating such an example myself.

screen-shot-2017-02-21-at-12-46-27

This may not be the most professional looking User Interface, but what it does is still pretty neat.  The objective is to feed it an animated QML scene and it should output an image file for each frame of the animation.  These images can then be converted into a video or animated image using an external tool.  The challenge is that Qt Quick is a UI tool, not a movie generator.

The naive approach to this would be to create a QQuickWindow, set the window size to the output target size, and then grab the frame by calling QQuickWindow::grabWindow() each time the frameSwapped() signal is emitted.  There are a couple of issues with this approach though.  First is that the video would need to render in realtime.  If you wanted to render an animation that was 5 minutes long, it would take 5 minutes because it would just be like recording your application for 5 minutes.  The second issue is that under the best case scenario you would be rendering video at the refresh rate of your monitor. This would even require a reasonably powerful machine, because the QQuickWindow::grabWindow() call involves a glReadPixels call which is quite expensive.  It is also problematic if you need to render at a different frame rate than your monitor refresh (which is what the user that inspired me was complaining about).  So here is how I addressed both of these issues.

QQuickRenderControl

QQuickRenderControl is a magical class that lets you do all kinds of crazy things with Qt Quick content.  For our purposes we will use it to render Qt Quick content to an offscreen surface as fast as we can.  Rather than creating an on-screen QQuickWindow, we can create a dummy QQuickWindow and via render control we can render content to an QOpenGLFramebufferObject instead.

    // Setup Format
    QSurfaceFormat format;
    format.setDepthBufferSize(16);
    format.setStencilBufferSize(8);

    // Setup OpenGL Context
    m_context = new QOpenGLContext;
    m_context->setFormat(format);
    m_context->create();

    // Setup dummy Surface (to create FBO with)
    m_offscreenSurface = new QOffscreenSurface;
    m_offscreenSurface->setFormat(m_context->format());
    m_offscreenSurface->create();

    // Setup Render Control and dummy window 
    m_renderControl = new QQuickRenderControl(this);
    m_quickWindow = new QQuickWindow(m_renderControl);

    // Setup QML Engine
    m_qmlEngine = new QQmlEngine;
    if (!m_qmlEngine->incubationController())
        m_qmlEngine->setIncubationController(m_quickWindow->incubationController());

    // Finish it all off
    m_context->makeCurrent(m_offscreenSurface);
    m_renderControl->initialize(m_context);

The above gets QQuickRenderControl setup, then when the size is know and you can actually create the QOpenGLFramebuffer object and tell the dummy QQuickWindow thats where it will be rendering.

void MovieRenderer::createFbo()
{
    m_fbo = new QOpenGLFramebufferObject(m_size * m_dpr, QOpenGLFramebufferObject::CombinedDepthStencil);
    m_quickWindow->setRenderTarget(m_fbo);
}

And once that is done it’s just a matter of loading up the QML content and rendering it.  Unlike with QQuickWindow, QQuickRenderControl allows you to control when the steps of the rendering process occurs.  In our case we want to render as fast as possible so this is what our rendering setup looks like:

void MovieRenderer::renderNext()
{

    // Polish, synchronize and render the next frame (into our fbo).
    m_renderControl->polishItems();
    m_renderControl->sync();
    m_renderControl->render();
    m_context->functions()->glFlush();

    m_currentFrame++;
 
    // Grab the contents of the FBO here ...

    if (m_currentFrame < m_frames) { 
        // Schedule the next update 
        QEvent *updateRequest = new QEvent(QEvent::UpdateRequest); 
        QCoreApplication::postEvent(this, updateRequest);
    } else { 
        //Finished cleanup();
    } 
} 
bool MovieRenderer::event(QEvent *event) { 
    if (event->type() == QEvent::UpdateRequest) {
        renderNext();
        return true;
    }
    return QObject::event(event);
}

The above sets up an event driven loop that will render as fast as possible while still handling events between frames, which is needed for progressing animations with Qt Quick.

Custom QAnimationDriver

The second issue we need to address is that the animation behavior is wrong.  To remedy this we need a custom QAnimationDriver that enables us to advance animations at our own frame rate.  The default behavior is to try and advance the animation’s in steps as close as possible to the refresh rate of the monitor your application is running on.  Since we never present the content we render to the screen that behavior doesn’t make sense for us.  Instead we can install our own QAnimationDriver which can be manually advanced each frame we generate based on a pre-determined frame rate.  Here is the whole implementation of my custom Animation driver:

class AnimationDriver : public QAnimationDriver
{
public:
    AnimationDriver(int msPerStep)
        : m_step(msPerStep)
        , m_elapsed(0)
    {}

    void advance() override
    {
        m_elapsed += m_step;
        advanceAnimation();
    }
    qint64 elapsed() const override
    {
        return m_elapsed;
    }
private:
    int m_step;
    qint64 m_elapsed;
};

Now to use this you just need to install the new QAnimationDriver.  When you call QAnimationDriver::install() it will replace the current one, so Qt Quick will then behave like we need it to.  When we start the movie renderer we also install the custom AnimationDriver:

    m_frames = m_duration / 1000 * m_fps;
    m_animationDriver = new AnimationDriver(1000 / m_fps);
    m_animationDriver->install();

    // Start the renderer
    renderNext();

And finally since we control the render loop, we need to manually advance the animation driver.  So before the end of the renderNext() method make sure to call:

m_animationDriver->advance();

And that is it.  Now we can render as fast as possible, and our animation engine will step perfectly for the frame rate we are generate frames for.  It is important to remember that you must process events after calling advance() on your animations though, because these are handled through the Qt Event and Signal and Slots system.  If you don’t do this, then you will generate the same frame many times.

Results

Once you run the MovieRenderer you end up with a folder full of images representing each frame.  To prepare video files from the generated output I used ffmpeg:

ffmpeg -r 24 -f image2 -s 1280x720 -i output_%d.jpg -vcodec libx264 -crf 25 -pix_fmt yuv420p hello_world_24.mp4

In the above command it will generate a 720p video at 24 fps from a series of files called output_*.jpg.  It would also be possible to create an example that either called this tool for you via QProcess, or even included an encoder library to generate the video directly.  I went for the simplest approach using only what Qt had built-in for this example.  Here are a few example movies I generated:

This first video is rendered at 60 FPS and the second is at 24 FPS.  Notice how they animate at the same speed but one is smoother than the other.  This is the intended behavior in action.

Well thats all I have to show, the rest is up to you.  I’ve published the code for the QML Movie Renderer here so go check it out now!  I hope this example inspires you as well to make other cool projects, and I look forward to seeing what new unexpected ways you’ll be using Qt in the future.

Do you like this? Share it
Share on LinkedInGoogle+Share on FacebookTweet about this on Twitter

Posted in Dev Loop, Graphics, Qt Quick

10 comments

Ilya Aleshkov says:

Amazing! Thank you for sharing!

aportale says:

Thank you Andy! I was fantasizing about creating movies from QML, but the missing fps control kept me from even trying it. Your custom QAnimationDriver is really the missing puzzle piece. Cool stuff 🙂

Jinming Chen says:

Amazing! In your first piece of code, it seems QQuickWindow cannot accept a QQuickRenderControl for its constructor, doesn’t it? Maybe a typo?

Andy Nichols (nezticle) Andy Nichols (nezticle) says:

QQuickWindow does have a constructor that accepts a QQuickRenderControl. It’s documented as part of the QQuickRenderControl documentation, but is marked as “internal” for the QQuickWindow documentation:
http://code.qt.io/cgit/qt/qtdeclarative.git/tree/src/quick/items/qquickwindow.cpp#n1254

Nikita Skovoroda says:

Looks cool!

The naive approach would be to use apitrace though =). It supports recording OpenGL calls (with resources) into a trace and supports extracting png images for each frame. It doesn’t give you a convenient control over the surface format, though, and it also doesn’t gurantee a constant framerate, so it won’t be suitable for making a recording — but it still would be better than the «naive approach» mentioned here.
It has other usecases, though — like an ability to edit the trace itself, extract partial frames, and very compact trace size.

Jason says:

I did this for 4.X, where it easy much easier. I do recommend saving the frames in another thread. The performance improvements are substantial when dealing with PNG files, and is noticable in other for formats. Remember QPixmap is implicitly shared, so tough can convert to QImage and save in another thread. This allows each core of your machine to handle a frame – be it the DCT of jpg or the zlib compression of PNG.

Jason says:

Oops, looks like to put are doing this already. Carry on.

Jason says:

It would be cool if we didn’t have to set a duration. I’d rather have the animation be able to trigger the completion by itself.
Even if we have to fire a signal at the end. But very cool.

RajaRaviVarma says:

Cool, I always wanted to do this, but didn’t give it a try since there is no direct way.

Are there any plans to make it a part of Qt Quick core. A public api which eases this process
will be a lot helpful.

Sorry if that sounds so naive. I don’t know the internals of Qt or OpenGL.

lqsa says:

This works on Android and iPhone?

Commenting closed.

Get started today with Qt Download now