Android Innovation Labs: Augmented Reality in Dashboards

Automotive Services Connected Cars

December 12, 2016

Overview

Augmented Reality (AR) in Dashboards/Head Units is an area of significant interest in the connected car space. BMW, Tesla and several other OEMs have announced their intent to release an Augmented Reality dashboard in the near future.

Augmented Reality (AR) enhanced dashboards can enable several innovative use cases such as:

While using the navigation system, the head unit switches to live camera feed with AR overlays when you “almost” arrive at your destination. This allows for exact destination tracking (as opposed to current navigation systems that announce “you have arrived” but you don’t see your destination)
Using AR overlays whilst driving to show speed limits on routes, even when not using the navigation
Using smartphone capabilities to enhance dashboards that don’t have specific features, such as notifications via overlays about the color-coded distance between your car and the car in front

Implementing Augmented Reality in Dashboards

There are many ways to implement AR in the dashboard. We believe one of the early deployment technologies will be to leverage the power of the smartphone and its sensors/accessories to augment overlays on traditional head units. Standards such as CarPlay, MirrorLink, and Android Auto already exist that allow phones and dashboards to interact over well-defined protocols.

As part of our innovation labs prototyping, we wanted to see how a MirrorLink-based solution would perform today with AR overlays from the phone. We also wanted to use a stock Android tablet as an HU (head unit) and see how easy (or hard) it was to get MirrorLink working on a non-certified device (which we think will be a common requirement for aftermarket providers).

In our demo, we overlay live camera view with location information translated to names of buildings we are driving by. The MirrorLink implementation was provided by RealVNC.

Technology Concept Demo

Here is what we did for the technology concept demo:

To enable MirrorLink on a Nexus device, the AOSP Android kernel was recompiled with additional modules and several permissions in SELinux to build a customized Android image. HSC used that customized Android image in a Nexus tablet that served as a HeadUnit running Android Marshmallow (6.0)
HSC customized the Mirror Link Client provided by Real VNC
A MirrorLink signed app was developed that takes camera feed and displays the feed on the head unit via a MirrorLink connection
Performance optimizations and appropriate scaling had to be implemented for display on the HU.
GPS location was overlaid on the HU from the phone (again, a realistic scenario where a head unit may not have all the features)

This demo video is intended to be a technology showcase of integrating Augment Reality tools with MirrorLink and not necessarily a real-life use case. A real-life use case that leverages such technologies may be one of the use cases described above or one of many other innovative use cases.

We will get back to you!