Inside QImage (with a touch of Qt Quick and OpenGL)

QImage is the most common container for image data in Qt applications.

Getting a QImage that contains the pixel data of an image loaded from a file is fairly self-explanatory:


QImage img;
img.load("some_image.png");

This can then be used with QPainter, can be passed to widgets, and can also be utilized in Qt Quick scenes via custom image provider implementations. (although the latter would obviously be a massive overkill for image data coming from a file since that's what the Image element provides out of the box anyway).

So far so good. Now, what if the image data is coming from somewhere else? For example a custom drawing made via QPainter, or an image that comes from some external engine, like a camera, scanner or computer vision framework?

The answer lies in some of the 9 constructors. The interesting ones for our purposes are the following:


QImage(int width, int height, Format format);
QImage(uchar *data, int width, int height, Format format, ...);
QImage(uchar *data, int width, int height, int bytesPerLine, Format format, …);
QImage(const uchar *data, int width, int height, ...);
QImage(const uchar *data, int width, int height, int bytesPerLine, Format format, …);

Let's now take a look at some of the common use cases and how these constructors serve the specific needs of each case.

Case #1: Image data owned by the QImage

The common case when generating images on-the-fly, is using QPainter and its raster paint engine to draw into a QImage:


QImage img(640, 480, QImage::Format_ARGB32_Premultiplied);
img.fill(Qt::transparent);
QPainter p(&img);
p.fillRect(10, 10, 50, 50, Qt::red);
p.end();
...

Here the underlying image data is allocated and owned by img itself.

When it comes to the format for images that will be passed to Qt Quick, the basic recommendations (as of Qt 5.8) are the following:

  • In general, the first choice should be Format_ARGB32_Premultiplied. This is good because it is one of the preferred, fast formats for the raster paint engine, which QPainter uses under the hood when targeting a QImage, and the premultiplied format fits Qt Quick well since the scenegraph renderer and its materials rely on premultiplied alpha for blending. When creating textures for our image, the default, OpenGL-based Qt Quick scenegraph will avoid any potentially expensive QImage format conversion for ARGB32_Premultiplied images.
  • When there is no need for an alpha channel at all, due to our image being completely opaque, Format_RGB32 is a good alternative. This comes with the same benefit: no format conversion when creating an OpenGL scenegraph texture from such a QImage.

Other formats will lead to a convertToFormat() call at some point, which is not necessarily ideal. It is better to get the format right from the start.

It is worth noting that in order to get a proper no-conversion-by-Qt-on-CPU path, the OpenGL implementation must support GL_EXT_bgra, GL_EXT_texture_format_BGRA8888, or some of the vendor-specific variants. This can be relevant on older, OpenGL ES 2.0 only systems where BGRA support is not mandated by the GLES spec. In the absence of these (A)RGB32 image data will internally need either an additional swizzle step (this is what Quick and pretty much all old Qt 4 code does) or a conversion to a byte ordered QImage format (preferred by some of the newer code in QtGui and elsewhere), because the (A)RGB32 formats are not byte ordered.

What are byte ordered however are the Format_RGB(A|X)8888[_Premultiplied] formats introduced in Qt 5.2. These are nice because when read as bytes, the order is R, G, B, A on both little and big endian systems, meaning the image data can be passed to a glTex(Sub)Image2D call using a GL_RGBA format with a GL_UNSIGNED_BYTE data type as-is.

  • When targeting older OpenGL ES 2.0 systems is a must, custom OpenGL rendering code (for example, inside a QOpenGLWindow, QOpenGLWidget, QQuickFramebufferObject, etc.) can benefit from using QImages with a byte ordered format like Format_RGBA8888, simply because there is less code to write. (no manual swizzling or extra QImage format conversion is needed when BGRA support is missing)
  • Some of QtGui’s OpenGL helpers that may be used by such code, most notably QOpenGLTexture, also prefer Format_RGBA8888 when working with QImage, and will kick off a conversion for other formats.

Therefore, the older default recommendation of using Format_ARGB32_Premultiplied or Format_RGB32 is not necessarily valid always. In the context of Qt Quick however, sticking with these formats will typically still be the right choice.

Case #2: Wrapping external, read-only image data

Now, what if the image data is readily available from an external engine? The common solution here is to use the QImage constructor taking a const uchar pointer.


void *data = ...
QImage wrapper(static_cast<const uchar *>(data), 640, 480, QImage::Format_RGB32);
// 'wrapper' does not own the data.
// 'data' must stay valid until 'wrapper' is alive.

There are no memory allocations and copies made here. It is up to the application to ensure the width, height, format, and optionally the bytes per line reflect the raw image data received from the other framework or engine.

Like many other container classes in Qt, QImage is using implicit sharing. This is handy because this way a QImage can be passed or returned by value without having to worry avoid expensive copies of the actual image data.

Instances created from a const uchar * are special in the sense that any attempt to modify the QImage (via a non-const function) detaches (makes a copy) regardless of the reference count. Hence attempts to modify to original, external data are futile:


const uchar *data = ...
QImage wrapper(data, ...);
wrapper.setPixelColor(5, 5, Qt::green);
// 'wrapper' is not a wrapper anymore,
// it made a full copy and got detached

Note that while this is all nice in theory, and the zero-copy approach is great in some cases, in practice making copies of the data is often unavoidable due to the need for format conversions in order to match the needs of the various frameworks. For example when interfacing with OpenCV, the code (taken from here) to convert a CV_8UC4 image into a QImage could look like this:


QImage mat8ToImage(const cv::Mat &mat)
{
    switch (mat.type()) {
    ...
    case CV_8UC4: {
        QImage wrapper(static_cast<const uchar *>(mat.data), mat.cols, mat.rows, int(mat.step), QImage::Format_RGB32);
        return wrapper.rgbSwapped();
    }
    ...

(NB! It does not matter if we use the const or non-const pointer variant of the constructor here. A copy will be made either way due to the semantics of rgbSwapped())

Case #3: Wrapping and modifying external image data

What about the constructor taking an uchar pointer? As the non-const argument suggests, this allows modifying the external, non-owned data via a wrapping QImage, without making copies of the image data. Modification often means opening a QPainter on the QImage:


void *data = ...
QImage img(static_cast<uchar *>(data), 640, 480, QImage::Format_RGB32);
QPainter p(&img);
p.fillRect(10, 10, 50, 50, Qt::red);
p.end();
... // 'data' must stay valid as long as 'img' is alive

Here we paint a red rectangle on top of the existing content.

This is made possible by a perhaps at first confusing feature of QImage: with non-owned, non-read-only data modification attempts will not detach when the reference count is 1 (i.e. the image data is not shared between multiple QImage instances).

Compare this behavior with something like QString::fromRawData() which explicitly says: Any attempts to modify the QString or copies of it will cause it to create a deep copy of the data, ensuring that the raw data isn’t modified.

There a container associated with external data always makes a copy, and does not support modifying the external data. With QImage this is not acceptable, since the ability to change the pixels of an image from an arbitrary source without making a copy is a must have.

See this little example project for a demonstration of painting into a manually allocated image. It also demonstrates one potential issue one may run into when getting started with such a QImage:


void drawStuff_Wrong(QImage image)
{
    QPainter p(&image); // oops
    p.fillRect(10, 10, 50, 50, Qt::red);
}

QImage wrapper(data, 640, 480, QImage::Format_RGB32); drawStuff_Wrong(wrapper);

This is clearly broken. The red rectangle will not appear in the original image pointed to by ‘data’. Due to passing to the function by value the image gets its refcount increased to 2, and so the modification attempt when opening the QPainter has to detach with a full copy. The solution here is to pass by pointer or reference. (or yet better do not pass at all; QImage instances referencing external data should ideally be isolated as much as possible, in order to avoid issues like the one above)

Accessing the image data

And last but not least, let’s talk about accessing the bytes of a QImage.

For read access, something like QImage::pixel() clearly does not scale when having to examining a larger part of the image. Instead, use constBits() or constScanLine().


QImage img(640, 480, QImage::Format_ARGB32_Premultiplied);
const uchar *p = img.constBits();
...
// 'p' is valid as long as 'img' is alive

Read-write access is done via bits() and scanLine(). These exist in both const and non-const version. The const version are in effect same as constBits() and constScanLine(). In practice it is strongly recommended to use constBits() and constScanLine() whenever read-only access is desired, in order to avoid accidentally invoking the non-const bits() or scanLine(), and making expensive and totally unnecessary copies of the image data.


QImage img1(640, 480, QImage::Format_RGB32);
QImage img2 = img1;
const uchar *p = img1.bits();
// ouch! that's the non-const bits(). img1 detaches.
// we only wanted read access and yet wasted time with a copy.

Due to img1 not being const this should have been:


QImage img1(640, 480, QImage::Format_RGB32);
QImage img2 = img1;
const uchar *p = img1.constBits();
// no copy, img1 and img2 still share the same data

The non-const versions detach when multiple instances share the same data or when the image wraps read-only, external data (via the const uchar * constructor).

That is all for now. Hope this clears up some of the uncertainties that may arise when looking at the QImage API for the first time, and helps to avoid some of the pitfalls when working with external image data. Happy hacking with QImage!


Blog Topics:

Comments