In Search of the “Ideal” Human-Machine Interface
by Barry Manz for Mouser Electronics
The human-machine interface (HMI) used in commercial, industrial,
and consumer systems is evolving to incorporate modern technologies like touch screens,
voice activation, and gesturing while simultaneously attempting to accommodate diverse
functions and making the result intuitive and easy to use. Taken together, they
are enormous challenges combining hardware, software, psychology, and sometimes
cognitive neuroscience. Even consumer products that are at the forefront of user
interface design have yet to achieve a fully satisfactory solution. Fortunately,
bright minds are at work throughout the world in the quest to remedy the
shortcomings of current HMIs.
If the path to success was simple, TVs and entertainment
systems would not have remote controls crammed with tiny buttons and vehicle
infotainment consoles wouldn’t require thick manuals to describe how they work.
Even HMIs for demanding applications like commercial aircraft cockpits and
high-speed trains have only recently been able to simplify the bewildering maze
of controls and displays. But it can be done:
Figure 1 -- The modern and surprisingly simple HMI in the
driver's cabin of a German Intercity-Express High-Speed Train. (Credit: TobiToaster.
German Wikipedia, Public Domain)
Figure 2 – Boeing’s 787 Dreamliner is one of the most
advanced commercial aircraft in the world, and its HMI is
just one example of how complex our world has become. (Credit: Alex
Beltyukov - www.airliners.net)
There are many schools of thought about how modern HMI
design should best be achieved. Industrial systems lag behind consumer products
here, but it can be argued that only modest changes to the front panels of the
least complex systems are necessary. That is, if invoking only a few functions
or monitoring a minimal set of system performance parameters are all that’s
required, changing a well-known operator may actually have the opposite of the
desired effect. However, the number of “simple” operating environments is
decreasing as more sensors are employed to detect more parameters, all of which
are expected to be connected to local or remote host control systems.
Texas Instruments has developed an entire ecosystem for
creation of HMI that provides virtually all the components, software, and
support required to develop a variety of interfaces. TI’s portfolio of HMI
products incorporates a broad range of I/O, graphics processing, audio for
voice recognition, Power
over Ethernet to reduce wiring complexity, and even support for wireless
connectivity.
Figure 3 -- Texas Instruments’ solutions for HMI
development can serve most use-case scenarios, including those requiring wireless
connectivity. (Source: Human Machine Interface Guide by TI slyy049a)
Choosing the “Right” Approach
HMI designers have many tools to choose from that can be
employed either alone or together. For example, a rotary dial on which there is
also a button can be used to change modes and vary values, with the result presented
visually on a display. This works well enough, but requires the operator to
perform two or three things: turning the knob, pushing the button, and watching
the display. It’s the equivalent of scrolling through menus to reach the
desired function, then invoking that function, and choosing some other activity
or activities at lower levels in the menu structure.
Touchscreens,
today’s de facto choice in products ranging from smartphones to
point-of-sale terminals, are also being employed in industrial HMI. While this
approach may seem ideal, it has disadvantages; one of which is that it uses the
same layered (menu-driven) approach of yore, so touching a button usually
results in the appearance of another screen on which are other buttons, and so
on. Among the many capacitive touch controls available, the MTC86303
from Microchip Technology stands out for its precision control of interface
actions including zone, multi-finger scrolling, and swipes. It can be easily integrated
into any system.
Figure 4 -- Microchip Technology’s MTC86303 is an
advanced capacitive touchscreen controller. Block diagram of the MTC86303
shown.
A more futuristic approach is gesture control, through which
the user commands the system by using hand signals in the air in front of the
display. This approach, which is undergoing significant development, offers
considerable promise owing to the almost unlimited number of possible gestures
and corresponding actions. ON
Semiconductor’s ASX340AT CMOS image sensor SoC is a camera-on-a-chip
designed for capturing high-quality VGA resolution in automotive
applications, making it a candidate for gesture control. It uses advanced
active pixel CMOS technology to provide high sensitivity and includes color
recovery, programmable gamma correction, sharpening, auto exposure, and many
other functions.
Voice
recognition is an obvious choice in HMI, as an unlimited number of functions can
be invoked, but has always suffered from inaccuracy. It is fiendishly difficult
to recognize even one person’s voice let alone the many others who may use the
same system. The first voice recognition systems in smartphones became almost
comic as their results were all too frequently unintelligible. While merely
frustrating for a smartphone user, this is wholly unacceptable for the operator
of a mission-critical system.
Fortunately, thanks to advances in artificial
intelligence, digital signal processing, and high-performance general
processors, today’s best voice recognition systems are surprisingly accurate,
even when the speaker is less than articulate, and they do not require
extensive “learning” before they can be used. In fact, this very article was “written”
using Dragon Naturally Speaking from Nuance, a voice recognition pioneer whose
technology got a huge boost when Ford Motor Company began using it in its
MySync 2 infotainment
system and by Apple for Siri. Nuance gained much of its voice recognition
expertise through its merger with ScanSoft, founded by Raymond Kurzweil of
optical character recognition and text-to-speech recognition fame--and much
else. That said, voice recognition can be exceptionally frustrating when
implemented poorly.
Starting from Scratch
One of the greatest impediments to modern HMI design is the
belief in some quarters that HMI must attempt to re-create familiar analog
controls by digital means. Superficially speaking, this makes a lot of sense
but places strict limitations on what can be achieved, as well as the tools
that can be employed to realize it. Consequently, some designers have effectively
started with a “clean sheet of paper” and many come from people targeting
automotive telematics and infotainment systems that have some of the most
complex HMIs of all.
The HMI in an automobile meets anyone’s definition of a
mission-critical system. It must accommodate not just a few different functions,
but potentially dozens while making them easy to invoke without distracting
the driver. The auto industry has attempted to achieve these goals using multiple
techniques, usually together, and the results until very recently have mostly
been underwhelming. It is also an example of why trying to use advanced digital
user interface techniques to replicate functions formally orchestrated by
analog controls doesn’t work.
It’s unfair to be too harsh on this industry though, because
the users of its equipment run the gamut from the technically proficient (who
are probably already trying to figure out how to jailbreak their car’s
infotainment system) to downright antagonistic and fearful. Trying to serve such
a diverse user community would be a challenge for anyone. Consider also the
number of completely independent systems that an automotive HMI must control,
which include all entertainment functions, display of
vehicle status (speed, distance, engine RPM, tire pressure, door ajar and
many others), alerts for traction control, overheating, and other potentially serious
events, heating and air-conditioning, and navigation.
To this long list must now be added the increasing number of
functions related to vehicle safety, including lane departure warning, backup cameras,
active and passive forward collision monitoring, autonomous parking, and
ultimately situational awareness - once vehicle
autonomy becomes more thoroughly developed. It’s safe to say that few other
systems must control and display so many functions within the confines of a
relatively small area. And they need to do this in the easiest possible way.
That said, techniques are in development that may actually
satisfy everyone, or at least almost everyone. One such system, created by San
Francisco-based designer Matthaeus Krenn, initially presents the driver with a
blank touchscreen from which many system functions can be performed using only the
fingers of one hand. By touching the screen with different numbers of fingers
at different distances from each other, various control functions can be
performed. The HMI uses different levels of sensitivity depending on the
function. For example, setting the audio volume uses small movements while
changing a music source uses larger ones. The written word doesn’t really do
justice to this elegant approach, which is best understood by watching Krenn’s
video.
The Disney Research division of the Walt Disney Company has
created a multifunction solution that adds the haptic feedback missing from a
“flat” touchscreen display. Its rendering algorithm simulates three-dimensional
features on a touchscreen using the principle that sliding a finger on an
object creates tiny variations on the surface that can be detected by “friction-sensitive
mechanoreceptors” in the skin. Modulating the friction between the finger in
the screen can create the illusion of three-dimensional features, which Disney
calls “3D bumps.”
The head up display (HUD) is another approach for displaying
information that can be combined with voice activation or physical controls to
invoke functions. It’s origin dates back to the optical reflector sight first
described in 1900 and later used on fighter aircraft in World War II. Even in its
earliest state, it allowed the operator to concentrate on activity in the far-field
while looking through semi-reflecting glass in the near-field to see a point in
his or her field of view. Modern-day HUDs are standard equipment on modern
military aircraft, including synthetic vision systems that present information
such as navigation, altitude, attitude, and terrain maps with exceptionally
high resolution.
Figure 5 – Synthetic vision systems are high-resolution virtual
displays the combine mapping and key pilot information. (Provided by Honeywell
with permission, CC BY-SA 3.0,
https://en.wikipedia.org/w/index.php?curid=36819016)
Since operator fatigue is a significant cause of accidents
in the mining industry, Caterpillar Global Mining is working with Canberra,
Australia, company Seeing Machines Limited to develop a system that uses eye,
face, and facial tracking to monitor operator awareness. It’s based on Seeing
Machines’ proprietary technology to detect operator fatigue and distraction, and
alert administrators to the condition. The system uses a vehicle-mounted PC,
GPS receiver, accelerometer, camera, and infrared sensors combined with
computer vision algorithms. Head position is also evaluated, as statistics show
that someone’s head tends to droop about six or seven seconds after the operator’s
eyes close. Seeing Machines has also developed solutions for automotive Advance
Driver Assistance Systems (ADAS), avionics, and rail transportation
systems.
Face detection is a critical component of the systems and Omron
Electronic Components B5T HVC Face Detection Sensor Module includes a Human
Vision Components (HVC) Sensor that incorporates 10 algorithms based on the
company’s OKAO™ vision image sensing technology. They include face recognition,
expression estimation, and hand detection and operate at speeds as fast as 1.1s
with a high level of precision. The HVC sensors even include age and gender
estimation, gaze estimation, blink estimation, and other factors.
One Solution Does Not Fit All
Every HMI application is different and so too, will be the
solutions that ultimately modernize them. In each case though, the best
approach will be the one that not just performs every required function but
does so in a way that is highly intuitive, fast, and most important, easy to
use. While advanced technologies such as face and voice recognition, large
touchscreens, and gesturing may represent many of the leading-edge technologies
being applied to HMI, in some cases they simply may not be necessary.
That’s because some systems simply don’t need more than a
reasonably large display, a few buttons, and perhaps a few knobs. But even in
these cases it’s possible to make them more useful even without applying “technology
at the edge”, although there’s no doubt that every HMI will be brought into the
world of advanced connectivity, which holds much potential. For most everyone,
advances in HMI will be most obvious in vehicles as the auto industry struggles
to simplify the increasingly complex array of functions presented to the
driver.
Barry Manz is president of Manz Communications, Inc. He has worked with over 100 companies in RF, microwave, defense, test and measurement, semiconductor, embedded systems, lightwave, and other markets. He edits for the Journal of Electronic Defense, Military Microwave Digest, and was chief editor of Microwaves & RF magazine.
