It's a Bentley! A guided tour of the new QNX technology concept car

"Bend it, shape it, any way you want it"
— Headline from a QNX advertisement, circa 1987

I’m about to show you some pictures of a car. Not just any car, but a powerful, luxurious, and stunningly beautiful car. A car that has undergone a technological transformation.

If you’re like me, you'll be fascinated by the car’s features, some of which have never been seen in a vehicle — until now. But if you can, remember that it isn’t just about the cool features. It’s also about the platform that enabled them.

I’m speaking, of course, of the QNX CAR application platform.

We created the new QNX technology concept car — a modified Bentley Continental GT — to demonstrate that flexibility and customization form the very DNA of the QNX CAR platform. If you’ve seen the QNX reference vehicle, you already know that the platform provides an extremely rich environment for in-car infotainment, complete with HMI frameworks, smartphone integration, an HTML5 engine, a mobile device gateway, and a host of pre-integrated partner technologies — everything to kickstart our customers' projects. But in the automotive world, differentiation is everything. So it’s just as important that the platform enables customers to add their own branding, features, and sizzle. And to do it quickly.

Ease of branding and
personalization is just one

of the capabilities of the
QNX CAR platform.

Which is where the new concept car comes in. To create it, we used the same base QNX CAR platform that we offer our customers. But when you compare the Bentley to the Jeep, which uses a stock version of QNX CAR, the differences are dramatic: different features, different branding, and a different look-and-feel. In effect, the Jeep shows what QNX CAR can do out of the box, while the Bentley shows what QNX CAR lets you do once you start bending it to your imagination. One platform, many possibilities.

Which brings me to the slogan at the top of this post. It's amazing to think that a core value of QNX technology in the 1980s — giving customers the flexibility to achieve what they want to do — remains just as true today. Some values, it seems, are worth keeping.

And now, the car…
I know that you’re anxious to peek inside the car and see what we’ve done. But before we go any further, take a moment to savor the car’s beautifully sculpted exterior. This is one classy set of wheels. In fact, if you ask me, the wheels alone are worth the price of admission:



The awesome (and full HD) center stack
Okay, time to hop in — but get ready to prop up your jaw. Because the first thing you’ll notice is the jaw-droppingly beautiful center stack. This immense touchscreen features a gracefully curved surface, full HD graphics, and TI’s optical touch input technology, which allows a physical control knob to be mounted directly on the screen — a feature that’s cool and useful. (In the photo below, the clock display appears within the knob.)

The center stack supports a host of applications, including a 3D navigation system from Elektrobit that makes full use of the display. Just check out this bird’s-eye view of the Las Vegas Strip:



So how big is the display? Big enough to provide access to other functions, such as the car’s media player or virtual mechanic, and still have plenty room for navigation. Check it out:



The awesome (and very polite) voice rec system
Time to talk to the car. Just say “Hello Bentley,” and the car’s voice recognition system immediately comes to life and begins to interact with you — in a British accent, no less. You can now tell the media player what you’d like to hear and the navigation system where you’d like to go.

To provide natural language speech recognition, the system uses the cloud-based AT&T Watson speech engine, as well as an “intent framework” from QNX. It also uses keyword spotting technology from Sensory so you can start the system simply by talking to it.

The awesome (and nicely integrated) smartphone support
The Bentley also showcases how the QNX CAR platform enables automakers to offer advanced integration with popular smartphones. For instance, the car can communicate with a smartphone to stream music, or to provide notifications of incoming email, news feeds, and other real-time information — all displayed in a manner appropriate to the automotive context. Here's an example:



The awesome (and just plain fun) web app
I know, I know: the car looks cool, but you’re not at CES this week to see it first-hand. But how about the next best thing? Just connect to the web app and keep tabs on the Bentley in real time. (Note: The car will go online later this morning.) The app lets you view a variety of data that the car publishes to the cloud, such as what song the infotainment system is playing and whether someone has just opened a door. It also displays information that would be extremely helpful if this were your personal car, such as fluid levels and tire pressure. (This is a preliminary screen for the app, so I'm not sure if the tire pressures are realistic.)



UPDATE: The web app is now live, and the desktop version features a live camera feed of the Bentley and Jeep. Check it out!



The awesome (and very configurable) digital instrument cluster
The instrument cluster is implemented entirely in software, though you would hardly know it — the virtual gauges are impressively realistic. But more impressive still is the cluster’s ability to morph itself on the fly. Put the car in Drive, and the cluster will display a tach, gas gauge, temperature gauge, and turn-by-turn directions — the cluster pulls these directions from the center stack’s navigation system (cool, that). Put the car in Reverse, and the cluster will display a video feed from the car’s backup camera.



There are other options as well. For instance, the cluster can display information from the media player or display the current weather:



The awesome everything else
I’ve only scratched the surface of what the car can do. For instance, it also provides:

• Advanced multimedia system — Offers direct support for Pandora radio and the first embedded in-car implementation of the Shazam music discovery service.
   
• Video conferencing with realistic telepresence — Separate cameras for the driver and passenger provide independent video streams, while high-definition voice technology from QNX offers expanded bandwidth for greater realism, as well as stereo telepresence for making the remote caller sound as if they’re sitting right next to you.
   
• LTE connectivity — The car features an LTE radio modem, as well as a Wi-Fi hotspot for devices you bring into the car.

Super size those images
Want to see the center stack and instrument cluster in all their high-resolution glory? Just check out our QNX Flickr account.

That's all for now, but stay tuned: We’ll have plenty more news for you today and all through this week.

New QNX framework promises to bring power of AT&T Watson to greater variety of in-car systems

This just in: QNX has announced a new framework that will help speech recognition systems in cars understand a speaker’s intent. The framework extracts meaning from the driver’s spoken words, enabling in-car systems to set complex navigation destinations, create calendar appointments, dictate email or text messages, or even perform general Internet searches.

The framework, which is a component of the QNX CAR application platform, will enable in-car systems to take advantage of AT&T Watson, a multilingual speech engine that runs on a cloud-based server to provide high-quality, low-latency voice recognition.

Determination of the driver's intent starts on the server, where the Watson engine begins to analyze the driver's words and fits them to known patterns. The results are then handed off to the car, where the intent engine from QNX performs further speech analysis to determine how to act.

According to my colleague Andy Gryc, "the server-side analysis provided by AT&T Watson is optimized for complex scenarios, such as a navigation application in which the driver may verbalize destinations in hundreds of different ways. The QNX client-side analysis grants car makers greater flexibility, enabling them to adapt the AT&T Watson results to a variety of in-car applications, regional aspects, or personal tastes.”

The intent system will be offered as a component of the QNX CAR application platform in 2013. For more information, read the press release.

Find me a Starbucks!
QNX and AT&T have already done a lot work to bring the Watson speech engine to cars. For an example, check out this Engadget video of the QNX concept car (a modified Porsche 911), filmed at an AT&T event this past April:

What if…

Imagine if your car could help you become more connected to friends and family — and to the road ahead. Enter a new video that peers into the not-so-distant future.

It blows my mind, but some people still see connectivity in the car as the enemy. They think that, the more connected the car, the more distracting and dangerous it will be. But you know what? Responding to their concerns is easy. I simply ask them what if.

For instance, what if connectivity helped you drive with greater situational awareness? What if it helped you sidestep traffic jams and axle-busting pot holes? What if it helped you detect a stop sign hidden behind a tree? And what if it helped you become more connected to the people important to you, as well as to the road and the cars around you?

When we talk connectivity at QNX, that’s the kind of connectivity we envision. It isn’t just about Bluetooth or Wi-Fi or LTE — that’s only the plumbing. Rather, it’s about keeping you in tune and in sync with your car, your environment, your business, your friends. Your life.

Am I crazy for talking to my car?

Earlier this afternoon, I participated in a connected car panel at SpeechTEK 2012, hosted by our friend Mazin Gilbert from AT&T. The other panelists included Greg Bielby of VoltDelta, Thomas Schalk of Agero, and Hakan Kostepen of Panasonic.

Even though Mazin did a fantastic job, not every panelist had a chance to answer every question. I was itching to answer some, so here are my responses to the questions that I didn't get to answer, or where I feel I could have provided a more complete response.

Have speech technologies matured to the point where they can be used robustly in the car? The general answer to this question from the panel was yes, but I think the real answer is a qualified yes. The technologies exist, but often aren't applied or may need auto-specific adaptations to handle in-cabin noise or other issues. Natural language recognition was an oft-stated driving technology, but a missing piece to the puzzle is hybrid recognition. I don't mean pushing recognition wholesale to the cloud, like Siri does. I mean a true split of the recognition effort, where each part does what it’s best at. Put the front half of acoustic processing in the vehicle to clean up the audio and convert the waveform to frequency-domain data, then send the data to the cloud-based server. The cloud server can then parse and interpret the data, and send back the result.

Hybrid speech rec solves three problems at once: better audio signals (the car can improve audio specific to the in-cabin environment), better cost (frequency data is far more compressed than raw audio, so you pay less for data transfer), and better responsiveness (hybrid rec gives the server time to start working on the response while it's coming in instead of waiting for the whole utterance to finish before starting).

Is driver distraction a major business driver, or is it the "Siri effect"? Currently, the car industry seems to use driver distraction as a reason to push a lot of features into speech. Many of those uses are gimmicky. Personally, I don't care if I can set my climate control system with voice — why would I when I can simply turn a dial? I once had someone ask me about the feasibility of adding voice recognition commands for rolling down the windows. I asked him, "Yes, but wouldn't people just push the window button?"

We shouldn’t implement speech commands just because we can. They may have contributed to excitement in the early adopter crowd, but we're beyond that now. Mind you, there are some seriously useful ways to use voice. For instance, any time you need to pick from a huge number of choices, voice recognition is the natural way to go. Calling contacts ("Call Sarah Potter"), entering destinations ("Go to 3121 South Park Street"), or picking music ("Play Audioslave") are all much easier than using an HMI to enter the same information, and safer to boot. It just has to work consistently and accurately.

Will car makers see more speech moving to the cloud, or will it be a hybrid of cloud and embedded? I disagree with the majority of the panel on this one, and, I think, the majority of people in the industry. Most auto people believe a hybrid between embedded and cloud allows the best of both worlds — good recognition and updatability when connected, and consistent reliability when not. My colleague Andrew Poliak also champions this view with a memorable catch phrase: Zombie Apocalypse. That is, you still want the system to work, albeit partially, when the infrastructure isn't available.

But if you ask me, everyone is missing the point — theirs is a technology-centric point of view. Everyday customer acceptance of a particular technology is notoriously harsh: if it doesn't work well, it gets rejected out of hand. Good cloud solutions beat an embedded solution hands-down; they just need some improvements (see my hybrid bullet above). Once a customer experiences a good solution, they will become frustrated with one that performs poorly. In my opinion, it's better not to offer the service at all, than to try a graceful degradation of capability, because most customers won't understand or care. Spend the effort instead on making sure you always have an acceptable cloud connection — either through multiple redundant mechanisms or a car-based powerful antenna — and you'll be better off. Even when the car knows some data that the cloud doesn't (like a mobile's contact list or music selection), there's no need to handle that on the embedded side. The cloud recognition server is powerful enough to not require the data set a priori. And I think we can predict an eventual migration of phone data to cloud-based data (or cloud-synchronized data) that makes the car's knowledge either easily transferrable or less relevant.

Who makes money, and how, from voice-enabled agents or voice services? This was one of the best questions of the panel, because nobody really knows the exact model, but everybody agreed that customer tolerance is very low. The most likely candidate is ad-based revenue. This doesn't mean reading ads aloud to the driver, but rather, positively influencing search results for either active or temporary situation-based points of interest (POIs). Depending on how valuable the service is to the driver, there will still be an option for service-based payments and high-value apps.

Standards and building mobile apps — will it come? You need standards if you want to build an app platform that will promote application creation and adoption. That's what we're doing with the QNX CAR 2 application platform — creating a way for someone other than the car companies to join the ecosystem and to deploy their apps to the car in a controlled way. But don't forget, you need a standard way to deploy apps for the cloud half of the recognition, too.

To close, let me share two photos. One was taken outside the Marriott Marquis, the hotel hosting the conference just off of Times Square in NYC. The other is from our PR agency, Breakaway Communications. What do they have in common? Wooden water towers. Sorry, I couldn't help myself; I just love those things. They just look so quaint in a city full of glass and brick.

In-car displays you hear, rather than see

We still have a lot in common with our caveman ancestors. (Yes, I know, they didn't all live in caves. Some lived in forests, others in savannahs, and still others in jungles. But I'm trying to make a point, so bear with me!)

Take, for example, our sense of hearing. At one time, we used auditory cues to locate prey or, conversely, avoid becoming prey. If a cave bear growled, getting a fix on the location of the growl could mean the difference between life and death. At the very least, it helped you avoid running directly into the bear's mouth.

Kidding aside, the human auditory system has a serious ability to fix the location, direction, and trajectory of objects, be they cave bears or Buicks. And it's an ability that's been honed from time immemorial. So why not take advantage of it when creating user interfaces for cars?

Which brings us to spatial auditory displays. In a nutshell, these displays allow you to perceive sound as coming from various locations in a three-dimensional space. Deployed in a car, they can help you intuitively identify voices and sources of instructions, and help pinpoint the location and relative trajectory of danger. They can also improve reaction times to application prompts and potentially hazardous events.

Interested in this topic? Learn more in Scott Pennock's ECD article, "Spatial auditory displays: Reducing cognitive load and improving driver reaction times."

I know, that's a lot to take in. So let's look at an example.

Locating the emergency vehicle, without really trying
Have you ever been cruising along when, suddenly, you hear an ambulance siren? I don't know about you, but I often spend time figuring out where, exactly, the ambulance is coming from. And I don't always get it right. That's called a location error.

Such errors can occur for a variety of reasons. For example, if the ambulance is approaching from the right, but your left window is open and a building on the left is reflecting sound from the siren, you might make the mistake of thinking that the ambulance is approaching from the left. Your mind realizes, quite correctly, that the sound is coming from the left, but the environment is conspiring to mask where the sound is actually coming from.

A spatial auditory display can help address this problem by controlling the acoustic cues you hear. The degree to which the display can do this depends, in part, on the hardware employed. For example, a display based on a large array of loudspeakers can provide more location information than one based on two loudspeakers.

In any case (and this is important), the display can help you determine the location more quickly and with less cognitive load — which means you may have more brain cycles to respond to the situation appropriately.


Helping the driver locate and track an emergency vehicle

A slight right, not a sharp right
I'm only scratching the surface here. Spatial auditory displays can, in fact, help improve all kinds of driving activities, from engaging in a handsfree call to using your navigation system.

For example, rather than simply say "turn right", the display could emit the instruction from the right side of the vehicle. It could even use apparent motion of the auditory prompt to convey a slight right as opposed to a sharp right.

But enough from me. To learn more about spatial auditory displays, check out a new article from my colleague Scott Pennock, whose knowledge of spatial auditory displays far surpasses mine. The article is called Spatial auditory displays: Reducing cognitive load and improving driver reaction times, and it has just been published by Embedded Computing Design magazine.
 

Living the Networked Life in New York City

Derek Kuhn

Last week, we demonstrated our latest concept car, built on a Porsche 911 Carrera S, at AT&T’s Living the Networked Life Event in New York City. The car has been on the road since we unveiled it at CES — it even went to Spain for MWC. We ran out of time to ship it to New York, so Mark Rigley, manager of the concept development group at QNX, and I drove it from Ottawa to SoHo. It was my first time driving the Porsche anywhere outside of a ball room, and it turned out to be a great drive.

Mark is super protective of the car, as he’s entitled to be, and he and the team have done a great job putting it together. But wouldn’t you know it? He let me drive and, as I’m pulling off Interstate 81 in Cortland NY, we get rear-ended. No — for real. I pull over, Mark jumps out, and I’m checking the side mirror to open the door safely in traffic but I’m mortified. Turns out that the Porsche’s rear valance is strong like a bull, and we emerge with just a series of scuff marks.

We drove through Binghamton, Scranton, and Poconos to get to the city. Traffic got faster and denser as we entered New Jersey, but Mark, to my surprise, let me drive again! We arrived at the hotel and handed the car off to a professional detailer before it was parked on site.

After setting up the car, we were ready. Soon we were joined by press, analysts, and AT&T partners and executives who all came out to see an important part of the car’s story — the use of AT&T’s WATSON speech engine. The Porsche is a great way to illustrate how helpful WATSON can be in an environment like the car.


Derek (seated in the car) with Mazin Gilbert of AT&T and Jordan Crook of TechCrunch.

The event was a big success and featured a presentation from Krish Prabhu, president and CEO of AT&T Labs Research; it also included demonstrations from AT&T Labs and AT&T Foundry. The demos included a smart door that recognizes you before you reach for your keys, and a smartphone application that performs language translations in real time.

The car will be on the road in coming weeks, including a stop at the CTIA Wireless show in New Orleans, from May 8 to 10. If you're there, be sure to check us out in the Solutions Showcase.
 

Techcrunch interview: Talking with (and about) AT&T's WATSON personal assistant

Yesterday, I shared an Engadget video highlighting how the QNX concept car takes advantage of WATSON, AT&T's cloud-based natural-language speech engine. Here's another video, from Jordan Crook of TechCrunch, in which AT&T's Mazin Gilbert describes what WATSON does and QNX's Derek Kuhn demonstrates how WATSON works.

The entire video is interesting, but jump to the 3:30 mark if you want to see the car. The video provides a very nice view of the QNX-equipped dash, btw.



The Engadget video and the TechCrunch video were both shot yesterday at AT&T's "Living the Networked Life" event in New York.

CES press: This just in!

OK well, maybe not 'just in' but these articles and releases were posted yesterday and are quite exciting if you are following QNX technology:

TeleCommunication Systems Supplies Advanced Navigation for QNX CAR 2 Application Platform

TI Takes The Driver’s Seat With An Unrivaled, Full System Solution For Connected Automotive Infotainment

AT&T's Watson coming to cars: won't mind if you ask it if you're there yet

Pimp your ride with augmented reality — Part II

Last week, I introduced you to some cool examples of augmented reality, or AR, and stated that AR can help drivers deal with the burgeoning amount of information in the car.

Now that we’ve covered the basics, let’s look at some use-cases for both drivers and passengers. Remember, though, that these examples are just a taste — the possibilities for integrating AR into the car are virtually endless.


AR for the driver
When it comes to drivers, AR will focus on providing information while reducing distraction. Already, some vehicles use AR to overlay the vehicle trajectory onto a backup camera display, allowing the driver to gauge where the car is headed. Some luxury cars go one step further and overlay lane markings or hazards in the vehicle display.

Expect even more functionality in the future. In the case of a backup camera, the display might take advantage of 3D technology, allowing you to see, for example, that a skateboard is closer than the post you are backing towards. And then there is GM's prototype heads-up system, which, in dark or foggy conditions, can project lane edges onto the windshield or highlight people crossing the road up ahead:



AR can be extremely powerful while keeping distraction to a minimum. Take destination search, for example. You could issue the verbal command, “Take me to a Starbucks on my route. I want to see their cool AR cups”. The nav system could then overlay a subtle route guidance over the road with a small Starbucks logo that gets bigger as you approach your destination. The logo could then hover over the building when you arrive.

You'll no longer have to wonder if your destination is on the right or left, or if your nav system is correct when it says, “You have arrived at your destination.” The answer will be right in front of you.

AR for the passenger
So what about the passenger? Well, you could easily apply AR to side windows and allow passengers to learn more about the world around them, a la Wikitude. Take, for example, this recent video from Toyota, which represents one of the best examples of how AR could make long road trips less tedious and more enjoyable:


 

Building a hands-free future

The end of my street is governed by a three-way stop. The other morning I was backing out of my driveway when someone rolled past the stop sign and came within inches of hitting me. I stopped, glared at him, and resumed driving. Two stop signs later, the same guy squeezed past my car (in the same lane), completely oblivious to what he was doing.

Why was he driving like this? Probably because he was deeply engrossed in a conversation on his cell phone.

Where I live, using a handset while driving has been illegal for over a year. You cannot talk, you cannot text, you cannot “Facebook”, you cannot Tweet — even if you're stopped at a red light. This makes perfect sense to me. As a driver, your primary responsibility is to control the vehicle. And yet I see people texting on the freeway, talking on their cell phones, and doing who knows what else on an alarmingly regular basis.

The QNX-powered BMW
ConnectedDrive system

Society has become obsessed with mobile devices, and it will take more than legislation to change its behavior. The answer, I think, is to embrace the behavior in a way that makes it possible to interact socially while maintaining control of the car. We’ve seen great progress in hands-free/phone integration, and BMW ConnectedDrive offers an example of how drivers can access email and other smartphone services more safely.

This is the tip of the iceberg. Integrating the handset with the infotainment unit is going to change the way you interact with your car. Intelligently designed apps, combined with multi-modal human machine interfaces, will let you Tweet or update Facebook using speech recognition, keeping your eyes on the road.

Without taking your hands of the wheel, you’ll be able to call a friend and decide that you want to go to dinner, do a local search to find out what’s available, check a restaurant review on Yelp, make a reservation, text your friend back with the time and place, and aim your navigation system at the restaurant. And you’ll be able to do it using natural language. None of this “please say a name” stuff.

Seems futuristic? It’s not. People are working on it today. In fact, QNX-based systems, such as Toyota Entune, already offer a taste of this hands-free and highly personalized future.
 

Speech interfaces: UI revolution or intelligent evolution?

Speech interfaces have received a lot of attention recently, especially with the marketing blitz for Siri, the new speech interface for the iPhone.

After watching some of the TV commercials you might conclude that you can simply talk to your phone as if it were your friend, and it will figure out what you want. For example, in one scenario the actor asks the phone, “Do I need a raincoat?”, and the phone responds with weather information.

A colleague commented that if he wanted weather information he would just ask for it. As in “What is the weather going to be like in Seattle?” or “Is it going to rain in Seattle?”.

Without more conversational context, if a friend were to ask me, “Do I need a raincoat?”, I would probably respond, “I don’t know, do you?” — jokingly, of course.

Evo or revo?

Are we ready to converse
with our phones and cars?

Kidding aside, systems like Siri raise an important question: Are we about to see a paradigm shift in user interfaces?

Possibly. But I think it will be more of a UI evolution than a UI revolution. In other words, speech interfaces will play a bigger role in UI designs, but that doesn't mean you're about to start talking to your phone — or any other device — as if it’s your best friend.

Currently, speech interfaces are underutilized. The reasons for this aren't yet clear, though they seem to encompass both technical and user issues. Traditionally, speech recognition accuracy rates have been less than perfect. Poor user interface design (for instance, reprompting strategies) has contributed to the overall problem and to increased user frustration.

Also, people simply aren't used to speech interfaces. For example, many phones support voice-dialing, yet most people don't use this feature. And user interface designers seem reluctant to leverage speech interfaces, possibly because of the additional cost and complexity, lack of awareness, or some other reason.


"Relying heavily on speech can lead
to a suboptimal user experience..."
As a further complication, relying heavily on speech as an interface can lead to a suboptimal user experience. Speech interfaces pose some real challenges, including recognition accuracy rates, natural language understanding, error recovery dialogs, UI design, and testing. They aren't the flawless wonders that some marketers would lead you to believe.

Still, I believe there is a happy medium for leveraging speech interfaces as part of a multi-modal interface — one that uses speech as an interface where it makes sense. Some tasks are better suited for a speech interface, while others are not. For example, speech provides an ideal way to provide input to an application when you can capitalize on information stored in the user’s head. But it’s much less successful when dealing with large lists of unfamiliar items.

Talkin' to your ride
Other factors, besides Apple, are driving the growing role of speech interfaces — particularly in automotive. Speech interfaces can, for example, help address the issue of driver distraction. They allow drivers to keep their “eyes on the road and hands on the wheel,” to quote an oft-used phrase.

So, will we see a paradigm shift towards speech interfaces? It's unlikely. I'm hoping, though, that we'll see a UI evolution that makes better use of them.

Think of it more as a paradigm nudge than a paradigm shift.

---

Recommended reading

Situation Awareness: a Holistic Approach to the Driver Distraction Problem
Wideband Speech Communications for Automotive: the Good, the Bad, and the Ugly