Interacting effectively with increasingly widespread and advanced systems is one of the most important challenges of our time. Most modern HMIs are based on mouse, keyboard or touch screen and allow controlling even very complex devices in a simple and intuitive way. However, in certain contexts, the user may be unable to have direct contact with a device, in this case, we are talking about hands-free interactions and often voice commands are used to interact. But controlling a system by voice, however natural, is not effective for all types of operations and in all environments. In fact, every technology has its peculiarities, that's why the HMI design and the UX are the subject of continuous research and aim to offer increasingly effective and natural interaction methods, also thanks to the combined use of more complementary technologies between them.
Eye tracking technology
Eye tracking is an innovative technology that allows you to accurately estimate where a person is looking, instant by instant, through the use of dedicated devices. Today this technology is spreading more and more thanks to the reduction of costs, the miniaturization of the devices and the high degree of reliability achieved.
Knowing the direction of the gaze allows a system to understand in an immediate way which element we are interested in and to which we want to issue commands, avoiding having to move a cursor or touch a surface. Knowing where we look, a system can also make a page or image scroll automatically, always offering the best frame with respect to the point of interest. The gaze is also related to our attention and attention monitoring is essential when we carry out particularly critical operations.
DeepGlance Quick
The DeepGlance team has over 15 years of experience with this technology and its applications in the main markets, including medical, healthcare, automotive, retail and entertainment.
The current barrier to integrating this technology is that it requires an important initial effort related to specific knowledge of how eye movements work, device management and how to transform low-level data returned by an eye tracker into high-level behaviors necessary to control a system.
This leads to DeepGlance Quick, a QML extension plugin that encapsulates the complexity linked to the use of eye tracking technology and that allows anyone to integrate it immediately into their own Qt application. In fact, through the use of the plugin within Qt Design Studio it is possible in a few minutes to create and test gaze-controlled applications or add eye tracking functionality to existing Qt projects.
DeepGlance Quick was presented for the first time at the SPS IPC Drive Nuremberg 2018, the most important European exhibition for industrial automation, gathering great enthusiasm among the insiders.
How to get started using Qt Design Studio
Before starting, make sure you have installed Qt Design Studio 1.0.0.
An eye tracking device is required to control the application, but If you do not have one, you can use the eye tracker simulator provided by the plugin.
As a first step, download the DeepGlance Quick package.
The Tobii Stream Engine library is a dependency and it is distributed in the plugin package. You must be sure that the Qt Design Studio and your application can find it by copying it to a folder in your system path.
The following example uses an EyeArea in a Rectangle that changes the Rectangle color to red when observed for a certain amount of time:
To get started, create a new Qt Design Studio project named HelloWorld and copy the DgQuick plugin folder in the “imports” directory.
Modify HelloWorld.qml adding the plugin types:
import QtQuick
import DgQuick
import HelloWorld
Item {
width: Constants.width
height: Constants.height
Screen01 {
}
EyeTracker {
model: EyeTracker.ModelTobii
Component.onCompleted: start()
}
ErrorHandler {
onError: {
if (error === ErrorHandler.ErrorEyeTrackerSoftwareNotInstalled) {
console.log("Tobii software not installed");
} else if (error === ErrorHandler.ErrorEyeTrackerNotConnected) {
console.log("Tobii eye tracker not connected");
}
}
}
EventHandler {
anchors.fill: parent
}
}
EyeTracker type is used to control the eye tracker device, ErrorHandler to handle the errors and the EventHandler to forward the mouse and keyboard events to the plugin.
To use the simulator just set the EyeTracker model property to EyeTracker.ModelSimulator.
Add EyeButton.qml:
import QtQuick 2.10
Import DgQuick 1.0
Rectangle {
width: 100
height: 100
color: "green"
EyeArea {
anchors.fill: parent
onAction: { parent.color = 'red' }
}
}
The EyeArea is an area that is gaze-sensitive, it is is the main component of the DeepGlance Quick plugin.
Finally, modify the Screen01.ui.qml adding the EyeButton:
import QtQuick 2.10
import HelloWorld 1.0
Rectangle {
width: Constants.width
height: Constants.height
EyeButton {
anchors.centerIn: parent
}
}
For more information and details please refer to the official documentation.
Hands-on experience
Eye tracking is one of those technologies whose potential is fully understood only by experiencing it firsthand.
For this reason, the DeepGlance Plugin was used to develop a demonstrator that collects the main interactions which may be carried out using eye tracking technology.
Take a look at the tutorial video.
You can download the demonstrator at the link below:
http://www.deepglance.com/deepglance-explorer-download
Interaction and Experience design
Technology is not enough to create natural interfaces and valuable user experience, it is necessary to create a design focused on the engagement and involvement of users. For that reason, DeepGlance has started a collaboration with the Interaction & Experience Design Research Lab of the Polytechnic University of Milan, whose design department is ranked 6th in the world. Recently, during the workshop "New paradigms for HC interaction and eye tracking driven interfaces" realized in collaboration with Qt, the students of Digital and Interaction Design have designed solutions that integrate eye tracking technology. And thanks to the integration of DeepGlance Quick in Qt Design Studio they quickly developed working prototypes of the projects.
Use cases
There are many applications where eye tracking technology can add value.
In the medical field, there is often the need to consult images, information or check systems in contexts where hands cannot be used. This is what happens for example for the operations of robotic surgery in which the surgeon has his hands engaged with the manipulators for the control of the robotic arms. In this case, the gaze is used to make adjustments, decide which robotic arm to associate with the right or left hand, automatically move the robotic endoscope camera based on the point of interest by providing always the best view and finally to make sure that the surgeon's attention is focused while he is controlling the system. In robotic surgery, the gaze is often used as a method to select and a single physical button mounted on the manipulator is used to confirm an action.
Eye tracking technology is widespread in the healthcare field to allow communication to patients in different clinical conditions such as Cerebral Palsy, ALS, Autism, Spinal Cord Injury. It is used to create personal communication and entertainment systems in a home or clinical / hospital environment controlled exclusively with the eyes and that allows the patient to write and vocalize messages, send emails, browse books, surf the Internet, control television, home automation systems and more.
In the field of retail, events, exhibitions and interactive museums entertainment, gamification and shopping experience systems can be developed to offer the visitor an engaging experience, contrast the "showrooming" effect, increase the time spent in the store / stand and ennobling the image of the brand with a "cool" and innovative experience.
For the creation of information points for hotels, airports, shopping malls, real estate agencies, shop windows to present content adapted to the user's interest, profile prospects in relation to the content consumed and use information in a confidential manner, that is without making no gesture or touch the display.
As an alternative method for interacting with vending machines for drinks, snacks, tickets, orders to avoid having to use hands exclusively to interact, for reasons of hygiene, comfort and to offer an innovative experience to the consumer.
In the marketing field to effectively present promotions and offers, increase the exposure of the prospect to the message that the brand wants to convey, reach an audience that is not currently interested in the message or product, and open a quality two-way communication channel with it also through cross-media marketing.
In the control rooms, it is possible to improve the efficiency of the operators through a multimodal interaction, that is by combining eye tracking with traditional inputs to show information in an adaptive and contextualized way and interact more quickly and naturally.
In the automotive sector, it is possible to monitor the attention of the driver detecting any distractions and allow to interact with the instrumentation on board in a more effective and safe way. For example, understanding the target of a voice command or automatically selecting the element or device with which to interact with the joystick on the steering wheel.
In short
DeepGlance Quick is a plugin that allows you to integrate and test eye tracking technology in your Qt application in an immediate and effortless way, transforming a traditional interface, into a gaze-controllable interface. Eye tracking can be used as an exclusive input method or in combination with other technologies, to make the interaction more effective and natural, taking the most of each of them in a complementary way. On the machine side, it can be used to monitor the attention of an operator and adapt the information shown in an interface based on the elements of interest to the user.
More info:
