Pimp my video: shader effects and multimedia

Introduction

A topic which has been receiving quite a bit of attention recently is the use of shader programs to apply effects within Qt Quick applications. Once you have grasped the basics of the shader programming language, embedding the shaders in your application is made really easy by Qt Quick, with the result that stunning visual effects can be achieved from a surprisingly small amount of code.

What hasn't been mentioned previously in this blog is that applying shader effects to multimedia content (video playback, or a camera viewfinder) is just as easy as transforming any other QML element. This post shows some of the results you can achieve by combining shader programming with QtMultimedia.

The best way to demonstrate some of the effects you can achieve is ... with a demo, so that's where we'll start.

The videos above show a Qt Quick 2 demo app, running on a desktop Linux machine. Aside from some ancillary C++ code (listening to scene graph signals in order to calculate the frame rate, and reading shader programs from the file system), the app is entirely written in QML.

Implementation details

As previously described, Qt Quick 2 comes with built-in support for shader effects via the ShaderEffect QML element. Using this to apply an effect to video or viewfinder content really isn't any more difficult than applying the same effect to any other QML element. To illustrate this, the code snippet below shows a time-varying wobble effect being applied to a video clip which is played from a file.

import QtQuick 2.0

import QtMultimedia 5.0
Rectangle {

    width: 600

    height: 400
    MediaPlayer {

        id: mediaPlayer

        autoplay: true

        source: "test.ogg"

    }
    VideoOutput {

        id: videoOutput

        anchors.fill: parent

        source: mediaPlayer

    }
    ShaderEffect {

        anchors.fill: parent
        // Properties which will be passed into the shader as uniforms

        property real amplitude: 0.02

        property real frequency: 20

        property real time: 0
        NumberAnimation on time {

            loops: Animation.Infinite

            from: 0

            to: Math.PI * 2

            duration: 600

        }
        property variant source: ShaderEffectSource {

            sourceItem: videoOutput

            hideSource: true

        }
        fragmentShader: "

            uniform highp float amplitude;

            uniform highp float frequency;

            uniform highp float time;

            uniform sampler2D source;

            uniform lowp float qt_Opacity;

            varying highp vec2 qt_TexCoord0;

            void main() {

                highp vec2 p = sin(time + frequency * qt_TexCoord0);

                highp vec2 tc = qt_TexCoord0 + amplitude * vec2(p.y, -p.x);

                gl_FragColor = qt_Opacity * texture2D(source, tc);

            }

        "

    }

}

Applying the effect to a viewfinder stream rather than to video playback is simply a matter of replacing the Video element with a Camera element.

In the qmlvideofx demo, each effect is implemented as a QML element which inherits from ShaderEffect; these elements are dynamically loaded and applied to the video content when the user selects the corresponding effect from the menu. Similarly, the different inputs (image, video and camera) are represented by QML elements which are dynamically loaded when required.

The set of parameters supported by each effect (e.g. 'granularity' for Pixelate; 'radius and diffraction' for Magnify) is exposed as a ListModel which is used to construct sliders via which the parameter values can be adjusted.

The code (Qt 5.x)

The source for the qmlvideofx demo can be found in the qtmultimedia repository here:

https://qt.gitorious.org/qt/qtmultimedia/trees/master/examples/video/qmlvideofx

Until Qt 5.0 is released, you'll have to build it from source in order to build and run the demo. Instructions for how to do that can be found on the wiki here. The required subset of Qt modules can be cloned as follows:

$QTDIR/init-repository --module-subset=qtbase,qtdeclarative,qtjsbackend,qtmultimedia,qtxmlpatterns

The code (Qt 4.x)

While Qt Quick 1 does not have built-in support for shader effects, it is provided by the Qt.labs.shaders plugin, which is shipped with Qt from version 4.7.4 onwards. A Qt 4 version of the demo is available here:

https://qt.gitorious.org/qt-mobility/qt-mobility/trees/master/demos/video/qmlvideofx

In addition to running on desktop platforms (tested on Linux and Windows), the Qt 4 version of the demo will also run on mobile devices - the video below shows it running on a Symbian device (Nokia C-701).

Keen-eyed viewers will notice that the app has a slightly different UI to the desktop version - thanks to the flexibility of Qt Quick, that is achieved by substituting a single QML file which describes the layout.

It should be noted that the app will not run on currently available versions of the Symbian platform. The reason is that it requires the output of the video decoder to be available to the OpenGLES engine as a texture. An EGL extension which allows the video and graphics stacks to cooperate in this way (EGL_NOK_image_endpoint2) will be added in a future Symbian release. Once this is available, QtMultimediaKit will automatically make use of it (see QTMOBILITY-1818), and the demo will start working.

You can, however, run the app today on the Nokia N9 - as shown by the video below.

Further information

If you weren't already excited about the potential of shader effects, hopefully you are now. Below are some links to other examples of their use in conjunction with Qt/QML.

• Qt Labs Blog: Qt Graphical Effects in Qt Labs
• Qt Labs Blog: The convenient power of QML Scene Graph
• Qt Labs Blog: QML Scene Graph in Master
• YouTube: Raspberry Pi, Qt 5, QML, Shaders
• YouTube: Livecoding video effects with Qt5