Qt Sensor Gestures

Published Friday July 20th, 2012
13 Comments on Qt Sensor Gestures
Posted in Gesture Recognizers, Qt, Qt Sensors, QtMobility

Here in sunny Brisvegas, ‘stralia, we have been doing more than not wearing shoes all year, we have been working all kinds of hard on QtDeclarative, QtMultimedia, QtLocation, QtSystems and QtSensors… oh ya, can’t forget Quality Assurance/CI System.

Of course, the most exciting one of these is QSensorGestures

With QtDeclarative, Qt3d, QtMultimedia and QtSensorGestures, you can have an awesome cross platform gaming platform with massive potential. Crikey Mate, That one is a ripper!

Apparently a hot topic for console games, tablets and mobile devices (as well as robots)  is – gesture recognition. Since I work in the Qt Sensors team, I decided to investigate gesture recognition using sensors such as accelerometer but also incorporating the light and proximity sensors and share with the world of Qt.

There are heaps of research papers detailing gesture recognition out there. Mostly using an advanced approach and algorithms which need a large amount of training data and time to teach the software the gestures. These systems are usually complicated and very detailed about the gestures they can support.

The Qt Sensors library in Qt 5 was developed as part of Qt Mobility. The Qt Sensor Gesture API was developed subsequently after the sensors code was integrated into the Qt 5 development branch.

Qt’s Sensor Gestures uses a plugin system for recognizer integration. Which means if the system allows installing plugins, you can write your very own recognizers. You could even go so far as to use the more complicated and popular way of writing gesture recognizers that use machine learning and computer vision techniques.

I have developed a few simple gesture recognizers in the QtSensorGesture plugin – pickup, twist, cover, hover, whip, turnover, shake and slam. This plugin uses an ad-hoc method of detecting gestures, but recognizer plugin can also use more advanced methods as well. This has the advantage of speed, memory usage and removing the lengthy process of machine training, but it has the disadvantage of the user not being able to create their own gestures, and recognizer collisions – where one gesture will be recognized when another gets performed. But that is always a hazard with sensor gestures.

These gestures are documented here (until the Qt5 release, anyway):

http://doc-snapshot.qt-project.org/5.0/sensorgesture-plugins-topics.html#recognizer-plugins

The current state of QSensorGestures is that the current plugins are most likely device specific, which means they might have to be adapted to any real hardware that might want to use them.

In the future, as well as adding “attributes/properties” for device specific configuration, there could be QtSensor backends for certain game controllers which have accelerometers as well as a library to access them, as well as other devices. I have also been looking into using the image sensors as well as audio sensors for gesture recognition, which would require more advanced methods of gesture detection.

I don’t have any flashy cool videos as I had hoped for, so you all will have to do with words, documentation and a few images.

and of course, the source code is available in the Qt5 repo.

Qt 5 is looking to be the best version of Qt yet!

 

 

Do you like this? Share it
Share on LinkedInGoogle+Share on FacebookTweet about this on Twitter

Posted in Gesture Recognizers, Qt, Qt Sensors, QtMobility

13 comments

Dhi Aurrahman says:

Voila! Very cool! I wish there will be a “trainer” plugin, hence user could define and train their own gesture. Worked for simple gesture recognizer 2 years ago, hope there a way to chip in 🙂

Koh says:

Great.

abrahamchen says:

Very Good!

Thomas Senyk says:

They can’t deliver yet, but this is definitely a awesome use-case for this API:

Lúcio Corrêa says:

What´s the point of Nokia still develop Qt? Does it have any future products based on it yet?

qt expert says:

coooooooooooooooooooool

As a Qt user, I’d rather see support for *basic* gestures like swipe and pinch on *all* the supported platforms than research on advanced stuff like this. Currently we cannot have a two finger swipe working on Mac OS X.

Lorn Potter says:

Flavio: These are not touch sensor gestures. I would have liked to include the touch sensor, but that is taken care of by QGesture.
Perhaps in the future QGesture and QSensorGesture can be merged.

Timur Kristóf says:

Very cool stuff!

DA says:

This is really cool!!!

we can make games with a lot of interactive stuff,

but I´m confused, for what mobile will serve this? Symbian 3? meego? windows phone? other new operating system?

best regards and very good job!!

yyyy says:

Qt5 is very cool but I have the same question as DA–What are the platforms QML2.0 serve for?
Nokia want to sell Qt, that means it wouldn’t have any meanings on deploy Qt5 on the phones of nokia
So what are the platforms QML2.0 aim for?
android, ios, dead symbian and meego?
Or Desktops?

Alexander Lenhardt says:

Hey Lorn, pretty nice stuff. I’m also researching on this topic. I would have three additions for this kind of project (first 2 are already implemented in my local repo, 3rd one runs at least in Matlab so far 😉 ).
1. Gravity compensation mode
2. Orientation independent mode (acceleration data will be rotated back to the users coordinate frame)
3. A generic gesture trainer/recognizer based on dynamic time warping.

Since we are in a “special” situation in Brisbane, who would be a good contact point for integrating and discussing these additions? Would that be still Alex Blasche?

Lorn Potter says:

Alexander: The best contact would be me, as I am becoming maintainer for QSensors, but probably best to use JIRA for these issues, so we can track them, and interested people can join in.

I think 1 & 2 are being worked on or talked about by kdab folks for RIM. 2 would be tricky to not be dependent on gui libs.
#3 – I have had something like this in the back of my mind already. Looking forward for collaboration.

Commenting closed.

Get started today with Qt Download now