Qt Graphics and Performance - OpenVG

In previous posts in this series, Gunnar has described the design and performance characteristics of the painting system in Qt, and explored Raster in greater depth.  In this post, I'm going to talk about the unique features of the OpenVG graphics system.

Paint Engine

Unlike the other engines, the OpenVG paint engine was much easier to implement because the OpenVG API itself is very close in functionality to QPainter.  You can read all about the specification on Khronos' OpenVG Page, but here are the high points:

  • VGPath objects represent geometry made up of MoveTo, LineTo, and CubicTo elements.  This is a very close match for Qt's QPainterPath and QVectorPath abstractions.
  • VGPaint objects represent brushes and pens for filling paths with pixel patterns.  Solid colors, linear gradients, radial gradients, and pattern brushes are supported, but not conical gradients.
  • VGImage objects represent pixmaps in a large variety of pixel formats.  OpenVG supports a lot more formats than OpenGL/ES which makes it a lot easier to convert QImage's into VGImage's.
  • VGFont objects (OpenVG 1.1 only) store glyphs represented as VGImage's or VGPath's for quicker rendering of text items.  Under OpenVG 1.0, we fall back to path drawing at present.
  • Scissor for rectangle-based clipping.
  • Alpha mask for clipping to arbitrary shapes.
  • Affine transformation matrices for path and glyph drawing, affine and projective transformation matrices for image drawing.

Transformation Matrices

OpenVG does not support projective transformation matrices for path drawing, which is annoying because QPainter allows any affine or projective QTransform to be used for any drawing operation.  There is a registered OpenVG extension called VG_NDS_projective_geometry but none of the OpenVG engines we have come across support it.  The reason why OpenVG doesn't support it is because generating paint pixels in perspective can be quite difficult.  Projective matrices are supported for image drawing because drawing a simple image in perspective is a well-understood problem that OpenGL/ES systems do all the time.

When a projective transformation matrix is used for path drawing, we convert the path point-by-point using the QTransform and then draw it as a normal affine path using a default transformation for the window surface.  But what about the paint pixels?  Unfortunately, they won't be perspective-correct.  In practice this isn't a big problem because most paths are drawn with a solid color brush, and a solid color looks the same in perspective.

In general however, we discourage people from using projective transformations with paths.  If you really want to draw a scene in perspective, first draw it into a QPixmap and then draw the pixmap using a projective transformation.  You'll probably want to do this anyway because perspective transformations mostly occur during "flip" animation effects - drawing every tiny path in perspective every frame during the flip would be too slow.

Path Transformation and Drawing

Most of the path transformation logic is done in vectorPathToVGPath() and painterPathToVGPath() in qpaintengine_vg.cpp. We detect the presence of affine vs projective transformation matrices and use an appropriate conversion. We convert both QVectorPath and QPainterPath using specialized routines. The other paint engines typically convert everything into a QVectorPath first. The QPainterPath conversion can improve performance slightly when arbitrary paths are drawn during SVG rendering and the like - there's no point creating a QVectorPath if it is going to be quickly thrown away.

Path drawing takes a lazy update approach, attempting to minimize the number of OpenVG state changes from request to request:

  • If the draw requires a pen, then the penPaint object is updated with the current QPen if it was different from last time.
  • If the draw requires a brush, then the brushPaint object is updated with the current QBrush if it was different from last time.
  • The path transformation matrix is updated if it has changed since the last path drawing operation.
  • The path is drawn using vgDrawPath().

Most of the OpenVG state persists across paint events so if the same pen is used from one frame to the next, then it will be set once and never changed.  The state is also shared between all windows because there is only one OpenVG context for the entire system.

In an earlier version of the OpenVG paint engine, I just uploaded the state changes whenever they were made without trying to be lazy about it.  That was a mistake!  Applications that use QPainter, particularly those using QGraphicsView, can be very chatty - constantly saving and restoring the painter state.  It was quite common for brushes, pens, and transformation matrices to be changed, then changed again, without anything being drawn.  Now, it will only update the OpenVG state at the point where an actual drawing operation is about to happen.  This house-keeping does have a cost though, so if you can avoid unnecessary QPainter state changes in your application, then please do so.

Preallocated Paths

Rectangles, lines, points, and rounded rectangles feature quite heavily in many applications, with constantly changing co-ordinates.  OpenVG makes us create and destroy a VGPath every time.  To alleviate this, we've provided some pre-allocated paths for simple drawing operations, which we update with vgModifyPathCoords() rather than allocate GPU memory for a new path.  However, some chipsets can be slower at modifying a path than just making a new one!  On those chipsets, compile Qt with the QVG_NO_MODIFY_PATH macro.
Image Drawing

The best image drawing will be achieved with QPixmap rather than QImage.  With QPixmap, the image is converted into a VGImage once and then drawn multiple times.  With QImage,the image must be converted into a VGImage every time it is drawn.

The OpenVG drawing primitive vgDrawImage() is very primitive - it draws the selected VGImage at the origin of the current transformation.  There is no in-built support for sub-rectangle drawing. Fortunately, OpenVG has vgChildImage() which allows a sub-region to be quickly extracted, with the pixel data shared with the parent.  However, "quickly" is a very relative term - I've seen frame rates almost halve when using vgChildImage() compared to drawing a full image.  So if you can, draw entire QPixmap's when using OpenVG and limit the use of sub-rectangles.

Another source of slowdown is drawing images with opacity.  OpenVG has a way to multiply a VGPaint object with a VGImage to produce a destination image.  This is a very cheap way to achieve opacity effects and is quite fast.  Except!  And there is always an Except!  Except when the image is drawn with a projective transformation matrix.  Remember - paint pixels cannot be generated in perspective - so we cannot use a paint object to generate an "opacity color" even though the solid opacity color will be the same in perspective from all angles!  This is very annoying - the OpenVG committee could have made a special exception for solid color VGPaint objects.

When the OpenVG paint engine draws an image with opacity, and a projective transformation is in effect, we have to generate a copy of the VGImage and use vgColorMatrix() to adjust the opacity.  This isn't too bad if you are drawing the same image over and over with the same opacity, but it is very inefficient if you are animating the opacity.  So avoid opacity animations with OpenVG if you can.

Painting into a QPixmap isn't currently accelerated with OpenVG - it uses the raster paint engine instead, so we recommend painting pixmaps once rather than constantly updating them.  We will be addressing this in future versions.  Even when we do implement painting into a pixmap, there will be a cost: switching rendering surfaces from a window to a VGImage and back again is not cheap - on some chipsets it can be as heavy as a full EGL context switch.  So try to avoid switching painting surfaces if you can.

Clipping

Clipping is the bane of my existence!  It seems so easy to application writers - set a clip rectangle and it will be efficient, drawing less pixels!  If only!

There are three techniques that can be used to achieve clipping with OpenVG:

  • Scissor rectangle list.
  • Alpha mask for arbitrary clip shapes.
  • Scissor rectangle for simple clips and alpha mask for complex clips.

The last is the default in the OpenVG paint engine, and there are #define's that can be used to enable the other modes.  However, on some PowerVR chipset versions there is a bug where if the scissor is combined with the alpha mask, performance drops off a cliff - down to 2 frames per second in some cases!  So on such devices you may want to turn on scissor-only or mask-only clipping.

Better is to not use clipping at all if you can avoid it.  Draw everything in your scene in bottom-up order and let the GPU do the heavy lifting.  Remember, modern OpenGL/ES GPU's can crank out thousands of triangles per second, with clever algorithms for hidden-surface removal that are much cleverer than anything you can do by setting a clip.  OpenVG uses the same GPU in many cases.  If you set a clip, you may end up confusing the GPU into taking a slower path internally than it would otherwise.

If you must clip, try to use single-rectangle regions that can be set via the scissor.

Window Surfaces

Below the OpenVG paint engine is the window surface logic in the graphics system.  This is usually where platform-specific customizations are required to get pixels onto the screen as fast as possible.  The QVGWindowSurface class wraps a QVGEGLWindowSurfacePrivate object, which provides the heavy lifting.  The default EGL implementation is QVGEGLWindowSurfaceDirect which writes pixels into the window back buffer and calls eglSwapBuffers() to transfer it to the screen.  It is possible to enabled single-buffered operation with QVG_DIRECT_TO_WINDOW, but the cost may be tearing artifacts on-screen.

If your platform has some clever EGL extension mechanism for getting pixels onto the screen, then you will need to write a new graphics system plugin and implement your own QVGEGLWindowSurfacePrivate subclass.  The QtOpenVG module has been structured to make it relatively easy to do this without touching the core Qt code.

Memory Usage

Everything you do with graphics uses memory - in the CPU and in the GPU.  Window surfaces, VG rendering contexts, VGPath objects, VGImage objects, and so on.  It can get quite tight in the GPU on embedded systems.  We've taken some steps to manage this; e.g. destroying older VGImage objects when trying to upload a new QPixmap, and destroying all OpenVG objects when an application goes into the background to free up memory for foreground applications.

The more complex your application, the more likely it is that you'll hit the GPU memory limit.  There's only so much the QtOpenVG module can do for you.  We can take emergency measures to recover, but that's about it.  So keep an eye on how many pixmaps and windows you have in use and see if you can simplify your application a little.  Definitely avoid uploading very large jpeg photographs as a single QPixmap - split them up into smaller "tiles" that can be released when GPU memory gets tight.

Summary

The following tips summarise the performance suggestions from the previous section:

  • Avoid projective transformation matrices with drawing paths.
  • Minimize state changes on pens, brushes, transforms, etc.
  • Use QPixmap in preference to QImage where possible.
  • Avoid drawing images using sub-rectangles.
  • When drawing images with opacity, use an affine transformation matrix, or only a single opacity level.
  • Avoid switching painting surfaces, particularly between windows and pixmaps.
  • Don't use clipping if you can paint your scene in bottom-up order instead.
  • Split large images up into smaller pieces to avoid overloading GPU memory.

What's Next?

There's always more that can be done to improve any software system.  QtOpenVG is no different:

  • Painting into QPixmap's using OpenVG.
  • Smarter VGImage pooling to deal with out of GPU memory conditions.
  • Qt/Embedded and Lighthouse screen drivers.

Blog Topics:

Comments