Laszlo Agocs

Introducing video filters in Qt Multimedia

Published Friday March 20th, 2015
15 Comments on Introducing video filters in Qt Multimedia
Posted in Graphics, Multimedia, OpenGL, Qt Quick

Qt Multimedia makes it very easy to get a video stream from the camera or a video file rendered as part of your application’s Qt Quick scene. What is more, its modularized backend and video node plugin system allows to provide hardware accelerated, zero copy solutions on platforms where such an option is available. All this is hidden from the applications when using Qt Quick elements like Camera, MediaPlayer and VideoOutput, which is great. But what if you want to do some additional filtering or computations on the video frames before they are presented? For example because you want to transform the frame, or compute something from it, on the GPU using OpenCL or CUDA. Or because you want to run some algorithms provided by OpenCV. Before Qt 5.5 there was no easy way to do this in combination with the familiar Qt Quick elements. With Qt 5.5 this is going to change: say hello to QAbstractVideoFilter.

QAbstractVideoFilter serves as a base class for classes that are exposed to the QML world and are instantiated from there. They are then associated with a VideoOutput element. From that point on, every video frame the VideoOutput receives is run through the filter first. The filter can provide a new video frame, which is used in place of the original, calculate some results or both. The results of the computation are exposed to QML as arbitrary data structures and can be utilized from Javascript. For example, an OpenCV-based object detection algorithm can generate a list of rectangles that is exposed to QML. The corresponding Javascript code can then position some Rectangle elements at the indicated locations.

Let’s see some code

    import QtQuick 2.3
    import QtMultimedia 5.5
    import my.cool.stuff 1.0

    Item {
        Camera {
            id: camera
        }
        VideoOutput {
            source: camera
            anchors.fill: parent
            filters: [ faceRecognitionFilter ]
        }
        FaceRecognizer {
            id: faceRecognitionFilter
            property real scaleFactor: 1.1 // Define properties either in QML or in C++. Can be animated too.
            onFinished: {
                console.log("Found " + result.rects.length + " faces");
                ... // do something with the rectangle list
            }
        }
    }

The new filters property of VideoOutput allows to associate one or more QAbstractVideoFilter instances with it. These are then invoked in order for every incoming video frame.

The outline of the C++ implementation is like this:

QVideoFilterRunnable *FaceRecogFilter::createFilterRunnable()
{
    return new FaceRecogFilterRunnable(this);
}
...
QVideoFrame FaceRecogFilterRunnable::run(QVideoFrame *input, const QVideoSurfaceFormat &surfaceFormat, RunFlags flags)
{
    // Convert the input into a suitable OpenCV image format, then run e.g. cv::CascadeClassifier,
    // and finally store the list of rectangles into a QObject exposing a 'rects' property.
    ...
    emit m_filter->finished(result);
    return *input;
}   
...
int main(..)
{
    ...
    qmlRegisterType<FaceRecogFilter>("my.cool.stuff", 1, 0, "FaceRecognizer");
    ...
}

Here our filter implementation simply passes the input video frame through, while generating a list of rectangles. This can then be examined from QML, in the finished signal handler. Simple and flexible.

While the registration of our custom filter happens from the main() function in the example, the filter can also be provided from QML extension plugins, independently from the application.

The QAbstractVideoFilterQVideoFilterRunnable split mirrors the approach with QQuickItem – QSGNode. This is essential in order to support threaded rendering: when the Qt Quick scenegraph is using its threaded render loop, all rendering (the OpenGL operations) happen on a dedicated thread. This includes the filtering operations too. Therefore we have to ensure that the graphics and compute resources live and are only accessed on the render thread. A QVideoFilterRunnable always lives on the render thread and all its functions are guaranteed to be invoked on that thread, with the Qt Quick scenegraph’s OpenGL context bound. This makes creating filters relying on GPU compute APIs easy and painless, even when OpenGL interop is involved.

GPU compute and OpenGL interop

All this is very powerful when it comes to avoiding copies of the pixel data and utilizing the GPU as much as possible. The output video frame can be in any supported format and can differ from the input frame, for instance a GPU-accelerated filter can upload the image data received from the camera into an OpenGL texture, perform operations on that (using OpenCL – OpenGL interop for example) and provide the resulting OpenGL texture as its output. This means that after the initial texture upload, which is naturally in place even when not using any QAbstractVideoFilter at all, everything happens on the GPU. When doing video playback, the situation is even better on some platforms: in case the input is already an OpenGL texture, D3D texture, EGLImage or similar, we can potentially perform everything on the GPU without any readbacks or copies.

The OpenCL-based example that comes with Qt Multimedia demonstrates this well. Shown below running on OS X, the input frames from the video already contain OpenGL textures. All we need to do is to use OpenCL’s GL interop to get a CL image object. The output image object is also based on a GL texture, allowing us to pass it to VideoOutput and the Qt Quick scenegraph as-is.

OpenCL example on OS X

Real-time image transformation on the GPU with OpenCL, running on OS X

While the emboss effect is not really interesting and can also be done with OpenGL shaders using ShaderEffect items, the example proves that integrating OpenCL and similar APIs with Qt Multimedia does not have to be hard – in fact the code is surprisingly simple and yet so powerful.

It is worth pointing out that filters that do not result in a modified image and are not interested in staying in sync with the displayed frames do not have to block until the computation is finished: the implementation of run() can queue the necessary operations without waiting for them to finish. A signal indicating the availability of the computation results is then emitted later, for example from the associated event callback in case of OpenCL. All this is made possible by the thread-awareness of Qt’s signals: the signal emission will work equally well regardless of which thread the callback is invoked on.

CPU-based filtering

Not all uses of video filters will rely on the GPU. Today’s PCs and even many embedded devices are powerful enough to perform many algorithms on the CPU. Below is a screenshot from the finished version of the code snippet above. Instead of faces, we recognize something more exciting, namely Qt logos:

Qt logo recognition with OpenCV in a Qt Quick app

Qt logo recognition with OpenCV and a webcam in a Qt Quick application using Qt Multimedia and Qt Quick Controls.

The application’s user interface is fully QML-based, even the rectangles are actual Rectangle elements. It is shown here running on a desktop Linux system, where the video frames from the camera are provided as YUV image data in system memory. However, it functions identically well on Embedded Linux devices supported by Qt Multimedia, for example the i.MX6-based Sabre SD board. Here comes the proof:

Qt logo recognition with OpenCV on the Sabre SD

The same application, using the on-board MIPI camera

And it just works, with the added bonus of the touch-friendly controls from the Flat style.

Do you like this? Share it
Share on LinkedInGoogle+Share on FacebookTweet about this on Twitter

Posted in Graphics, Multimedia, OpenGL, Qt Quick

15 comments

Sylv says:

With Qt 5.5, will it be possible to screencast/save to a video file a whole QML interface, with animations (not only the video stream) ?

Ilya says:

Hi Sylv! Please check out my tutorial about video capturing https://software.intel.com/en-us/articles/intel-inde-media-pack-for-android-tutorials-video-capturing-for-qt-quick-applications

Laszlo Agocs Laszlo Agocs says:

Regarding the tutorial: That is what QQuickRenderControl is for, to redirect an entire Qt Quick scene, most commonly into a texture via a framebuffer object. Trying to trick the scenegraph into using an arbitrary FBO is not going to be safe without it and will break once the application starts using Qt Quick features that need additional FBOs. (since the custom capture FBO will not be rebound)

m][sko says:

If anybody interested opencv and prototyping in QML there is project “Live CV”
http://livecv.dinusv.com/

nice video example
https://www.youtube.com/watch?v=uEnJE6Jawfw

Scorp1us says:

*Expletive Deleted* Yes!

You have no idea how important this is to me. ๐Ÿ™‚ I’ve been mucking around with QVideoProbe and got so close, it was able to work, but not real time. This will get me real-time.

This.Is.Awesome!

Ionut Dediu says:

There is an error with posting new comments on this website: if the comment is too long (I don’t know how many chars) it will redirect me to a blank page and no comment will be posted! Why don’t you just notify the user of too many chars????

Ionut Dediu says:

Does this mean we will finally be able to easily and simply access the video frame data in CPU on Android? I just want to get the YUV/RGB frames from the camera viewfinder, compress them with MJPEG and send the data over bluetooth/WiFi. ( http://sourceforge.net/projects/smartcam/ )

Ionut Dediu says:

I tried this in Qt 5.4 for Android but it just didn’t work. Although the QML camera examples worked great, trying to get access to the frame data in c++ just didn’t work, and I tried all the possible ways the internet could find: adding a custom QAbstractVideoSurface, another Qt method which I forgot and finally accessing the frames in Java and exposing the data back to Qt through JNI. The last method I used before on the days of necessitas/ministro and Bogdan Vatra and it worked, however it looks like in the latest versions of Android frames can not be accessed in Java if a video surface is not displayed.

Ionut Dediu says:

The only method I didn’t try was using OpenCV.

Anyway it just didn’t work; I hope it will work in Qt 5.5 with this new class. Please confirm if this is true!

Thanks,
Ionut Dediu

Laszlo Agocs Laszlo Agocs says:

The new classes and properties described here are 5.5 only.

Nick says:

Hi! I can access video frames on android in C++. I use QVideoProbe (so it has little delay from real frames). but it works. The only issue it works only with VideoOutput (i did’nt found why it do not work, but I think it possible to use pure QCamera, because QML declarative camera besed on it)

Laszlo Agocs Laszlo Agocs says:

Yes, QVideoProbe is an alternative to some extent, however it always implies a certain delay and it provides no means to return a modified or totally different frame in place of the input.

Laszlo Agocs Laszlo Agocs says:

It always depends on the platform and multimedia backends in use what the QVideoFrame will contain. You will have access to the QVideoFrame as-is.

I am not that familiar with the Android backends but a quick peek at the sources reveal that frames from the camera will be in YUV (NV21) format as regular CPU-accessible data, while frames from videos will be provided as OpenGL textures.

Ionut Dediu says:

Hi,

Thanks for the reply. I tried the QVideoProbe approach and that didn’t work either. This was actual the method that I have forgotten about. I did my testing on Galaxy Nexus. The issue with Android is fragmentation. It would be great if working examples could be provided for this new video filters feature and tested on a fair amount of Android devices.

Cheers,
Ionut

Nick says:

It is very good news!!!
Very useful option for Camera

Commenting closed.

Get started today with Qt Download now