Roadside support with Google Assistant and Android Automotive

Roadside support with Google Assistant and Android Automotive

This was an interesting but challenging project for me for a number of reasons. It was during September of 2020 and we had been working from home for several months due to the pandemic. I was also working across three time zones with stakeholders speaking English, Hindi and Japanese. I was brought in by Connected on a very short 3-4 week contract to work with a software development firm in India called Wipro, to explore and validate a new use case for Honda, that would beGoogle Assistant Honda in Japan.

We only had just over 3 weeks to work on this and I would be working mostly by myself as the lead researcher, conversation designer and product manager. I had a partner working with me to take a lead on managing stakeholder engagement and scheduling as we were working between different timezones and teams and it was very challenging to align everyone’s calendars.


The automaker had given Wipro a challenge that would require them to demonstrate their capability in value proposition and conversation design as well as VUI development. The project was to assist in the development of new car-specific actions for Google Assistant in the car. They challenged Wipro to find a great solution to answering driver questions regarding indicator lights that a driver may see while on the road, design the conversation flows, and develop a functional proof of concept and design rationale. The user research and conversation design are where my help was needed, in addition to informing any technical requirements through some early research and technical validation I would need to do.

Project plan

With a 3.5 week timeline, I choose the following activities to help me with the early validation of existing hypotheses, and run through two quick iterations of design, measure and learn cycles.


User Interviews


  • Conducted 1:1 interviews with 4 English-speaking drivers over a 60-minute zoom call.


  • Understand current user behaviours and identify opportunities for voice-assistant.
  • Research driver experiences of vehicle diagnostic/prognostics, user manual usage, and new feature onboarding to inform the UX for the first prototype.
  • Presented an image of a dashboard to participants with an indicator light and asked them how they might ask a voice assistant about it.


Key insights & themes

Providing assistance with information & troubleshooting

The indicator light does not provide enough actionable information to the user for them to feel safe and plan their course of action.

Our participants reported seeking this information from various sources, including; Google search, the owner's manual, and through calling the dealership.

“My Next Move after that is I would probably find an area to actually pull over the next, you know, the closest area with a new vehicle, I probably actually, once I did, I'd switch the visual representation on that, that square screen and just see what the temperature actually is at. But at that point, you know, if it doesn't seem like anything's wrong, I'd probably call the dealership or something.” - Kel

User manuals are inefficient and universally disliked

Paper manuals we derided by all of our interview participants. Users are accustomed to the quick results they can get from internet searches and mobile applications.

There is a risk in simply digitizing an experience that is already looked at unfavourably. Finding the most used sections of the user manual could potentially be starting place when deciding where a voice assistant could be a more efficient experience.

“The owner’s manual is too thick. Like, they should have it on an app or something where you can just look it up. Who uses a paper manual anyways. It's so much faster to search for it on Google, you know? I don't even know how to find it in the user manual. Right?” - Anne

Negative and limiting perceptions of voice assistants

Poor user experiences and dated technology have led to a general perception that automotive voice assistants are of limited utility and that Siri, Alexa, and Google Assistant are better options to bring with them into their vehicle.

Because of this, it is especially important to ensure that the addition of voice assistant capabilities is supporting a great user experience and does not detract from it. The automotive assistant needs to be very capable within its own domain.

“My car has a voice assistant, but to be honest I don't use it at all. I've tried to but it wasn't really easy or intuitive So, using a phone is always so much simpler. ” - Jeff

Customer Profile

We synthesized our learnings into a summary document which included customer profiles for both the show's audiences as well as the onscreen talent. These profiles outlined their core jobs-to-be-done, as well as their pains and gains.



  • Arrive at my destination safely
  • Gather information to assess risks
  • Maintain and repair the vehicle when needed
  • Plan how to address engine and system issues that arise


  • Not enough context provided with indicator lights
  • The sound UI is too monotonous and hard to decipher
  • Voice input requires exact phrasing and it’s not intuitive or natural
  • I don’t know what any of these symbols mean
  • Indicator lights make me feel anxious


  • I’m confident that my car will tell me when it needs maintenance or repair
  • I am given enough context and information to make decisions
  • I know if it’s safe to continue driving, or if I need roadside assistance.
  • I feel confident that my car will tell me when something is wrong

Opportunity framing

How Might We Help drivers

Wizard-of-Oz Prototype

TTS Voice synthesis prototypes that are linear in nature, and supported with visual stimuli. I presented these low-fidelity prototypes to our participants in a 1:1 setting during a zoom call.

Learning Objectives:

  • Elicit all the different ways drivers will enquire about indicator lights when they see them appear on the car dashboard regardless of the setting (while driving, while idle).
  • Validate our early hypothesis around the importance of contextual information and troubleshooting assistants
  • Extract utterances and intents from driver data.


Scenario 1 Testing both proactive and reactive prompts from the VA and documenting utterances from our driver to visual stimuli of a single indicator light turning on. (blue low temp coolant light).

Scenario 2 Testing troubleshooting concept features for desirability and usability.

Scenario 3 Testing utterances of different indicator combinations

Multimodal prototyping using Google Slides

Doing this research remotely brought new challenges. I knew that the context of the status of the car, and the GUI modality of the car's dash and indicator lights were both critical to prototyping this conversational interaction. I also wasn’t able to use simulation software because I wasn't able to control the participant's hardware.

My solution involved using Google Slides to provide the necessary context to the participant and a dash display. This multimodal experience is on track as progress the experience forward by tapping the arrow key, triggering animations. Assistant prompts are prerecorded and I listen to my participant's utterances and choose the appropriate response. I had a colleague demonstrate a test sequence in the video below.


demo of WoZ prototype using Google Slides

The “wizard” provides the scenario to the driver with visual aids. The participant will interact with the prototype using their voice. The “wizard” is operating the test by listening to the driver’s speech and manually feeding the appropriate responses to the driver to conduct the conversation. I set up transcription software to record the test sessions. The data will be used to identify additional utterances, intents, identify edge cases and other design changes.

Key insights & themes

Keep me safe & help me troubleshoot

Above all drivers see their most important jobs as staying safe. This involves monitoring the car status and making informed decisions when presented with potential problems. Our drivers really appreciated that the assistant was able to provide them with ways to troubleshoot the problem without needing to pull over and look at the user manual. For drivers; it’s important to know how critical is this issue and do they need to stop immediately, or can they complete their journey.

“I liked that I was given the option of a few different problem-solving techniques you know the turning up the heat wouldn't necessarily have occurred to me in many cases, but having some kind of reassurance as I'm driving through the middle of the desert is great.” - Kel

Just enough information at just the right time

It’s important for the drivers to know how exactly how critical the issue is. If there are ways to troubleshoot it, and what they should do next. However, they don’t need to know these things all at once, and doing can be frustrating for them because it causes anxiety, cognitive overload and difficulty with comprehension. The conversational exchange should focus on efficiency while balancing the driver's cognitive load.

“The voice assistant is doing a lot of talking and it's not very helpful, especially with the first light - the engine being cool and it not being a problem, that doesn't need to be too much information but when it's something serious, give me more information” - Lisa

Expectations of the assistant having contextual awareness

A majority of the drivers we spoke with made the assumption that the car assistant has information about which lights are on and other data points about the car status. This may be why drivers often don’t describe the indicator light icons, instead, they ask “What’s this light?”, or “What’s wrong?”.

“it would be hard to describe that, like, What's this? What's this wavy line? The whole indicator light would be a hard thing to communicate. And so the distinction between them. I know they're very standard indicators across all vehicles.” - Kel

A driving assistant is a guardian angel that protects and guides

Drivers reported feeling assured by the assistant's tone and language and described her as “a guardian angel”.

In stressful situations using a calming tone, plain language (avoid jargon) and providing helpful information kept our drivers calm and feeling safe and cared for.

“It’s kind of like a Guardian Angel, if you're faced with a problem and it's kind of there over your shoulder being like, no, don't worry this isn't a big deal, we'll try and fix it together. So it's comforting and reassuring” - Jeff

Asynchronous Utterance Gathering

Format: Crowd-sourced utterances across internal employees using a questionnaire in Google Forms.

Learning Objectives:

  • Gather a large sample of utterance data asynchronously to inform the design in a short time period
  • Understand if there are differences between how people ask about indicator lights when presented with more than one at a time.
  • While effective for this short engagement, for a longer and more complete research effort a survey with an audio recording may be a more effective input as it might provide a closer match to the natural language used by drivers.

Key insights & themes

“What does that light mean?”

A pattern that we saw in our initial WoZ testing continues to be validated. When presented with a single light, many drivers simply ask the assistant what the light is, or what’s happening to the car. Supporting this intent will require the car to have contextual awareness of telematics data and system parameters. This insight has significant implications for other domains in the car environment as well.


Context is critically important to successful & efficient interactions

Critically important is that only 10% (4 of 38) of the utterances captured in relation to a single indicator light contained enough information to correctly match to a specific indicator intent.

In a scenario where the assistant does not have access to indicator light status data a successful match a users utterance to an indicator light intent most, if not all of the following descriptors:

  • Light colour
  • Light blinking / solid
  • Shape description


1 light, 2 lights, 3+ lights

From this survey, it’s also apparent that the structure of utterances used by drivers varies greatly whether they are seeing 0,1, 2 or 3 lights.

Additional qualifiers were used when multiple lights were present and these point to important contextual data to provide to the assistant for disambiguation in cases of 2, 3+ indicator lights. These include:

  • Relative position (right, left, middle, top, bottom, below)
  • Temporal chronology “just appeared on my dash”

Wide range of utterance structures

With the large amount of utterance data we were able to gather, it was clear that our NLU interaction model needed to account for the diversity of ways that people may ask about indicator lights.

While intent matching for a single light is quite easy if the assistant has access to contextual data, the most difficult scenario is when users are asking about an indicator that is not being displayed. For this, we need extremely robust utterance data and a structure for entities and slot filling that accommodates that diversity.


Functional Prototyping

High-fidelity prototype with NLU intent matching, error handling, multipath flows, conversation repairs and scenario with evolving context (e.g. while driving/ whole idle) outlined.

To create a functional prototype I had to train the NLU engine (LUIS) with structured sample utterance data, defined entities and intent structure. I did this by collating and annotating the utterance data I received in the WoZ testing and utterance survey.

Learning Objectives

  • Validate our hypothesis around the user's desire for contextual information, proactivity and troubleshooting assistance.
  • Conduct user experience testing and improve the accuracy of the conversation design by logging any additional utterances, finding and addressing no_match intents (manually) and extending conversation flows based on driver data


Scenario 1 Testing a scenario where the car is experiencing 3 indicator lights in an increasingly critical issue with coolant temperature being below and above nominal temperature range.

Scenario 2 Testing to validate hypothesis around proactive/reactive. The assistant in this scenario is designed to be only reactive when addressing the non-critical low-temperature indicator however as the situation becomes more critical she is proactive in her updates.

Scenario 3 Testing the balance of contextual information based on troubleshooting strategies, and car status information like idling, driving, and ignition on/off. This helps us to validate hypotheses regarding cognitive load.

Interaction Model

The interaction model is the structured data that defines the intents, entities and variables. An entity such as the shape descriptor can include many synonyms of the same slot value. A coolant temperature warning might be described as a boat, a ladder, or a flagpole.

It’s important to account for the diversity of ways a driver may describe this symbol. The utterance samples used to train our NLU model are taken from our user testing, and survey and can be improved over time.

The variables I defined are intended to provide our assistant with the necessary context required to provide one-shot intent matching, efficient information as well as cover edge cases when lights are not displayed or multiple lights are displayed at once.



thermometer, stick, waves, ocean, water, coolant, cooling water, sail, yacht, a yacht on waves, comb, flagpole, flag, ladder, ladder in the water, flag in the water, shape, boat, sailboat, ship, stick, pole, boat in the water, ladder, coolant water, coolant, coolant fluid, a boat in the water,


{indicator_light} Different ways that people might refer to “indicator light”

{indicatorlight_colour} Lights can be red, blue or orange, and defines synonyms

{indicatorlight_flashing} Descriptive words to define if the light is flashing, or solid, blinking or solid etc.

{indicatorlight_position} When more than one light, the user sometimes refers to the position of the light

{indicatorlight_shape} The 28 different indicator symbols and the many different ways people might describe them.



Is the car driving or idling?


Is the ignition on or off?


Are any indicator lights on, and how many?


What is the light colour?


What is the symbol id of the light?

Contextual design

Within the prototype, I’m representing the context of the car’s status, and changing conditions with visuals, but also by updating variables at the moments we would expect these data points to be exposed to the assistant. In total there are 18 possible answers to the question of ‘What is this light’ because of the different contextual scenarios regarding the 3 engine coolant indicator combinations. The assistant’s response will vary depending on the contexual parameters of the symbol’s colour, whether the car is driving or idling, or whether an indicator light is even being displayed. These changing conditions will inherently change which dialogue is given to the driver.



Key insights & themes

Proactive, reactive, agentive.

Drivers had a wide range of preferences and opinions about if and when the assistant should be proactive and initiate interactions, or in some cases agentive and make non-vital decisions to help ease the cognitive load of the driver.

This may indicate that personalization settings may play an important role in dictating whether in the assistant’s reactive vs proactive nature.

Understanding how to define these thresholds of proactive, reactive and agentive behaviour would require additional research but may result in a safer and more helpful assistant.

“when the engine light is red, and then asks me if I want to turn the heat on. I feel like rather than asking me if I should turn it on, it should automatically turn on. It should tell me, Hey, your engine is overheating. I'm going to turn on the heater for you, as is the best course of action. Let me know if you want to turn it off. ” - Lisa

Keep me safe & help me troubleshoot

This second prototype further validated the desirability of timely and concise troubleshooting tips.

It’s important however that this advice is always given contextually, and delivered at the right moment in a way that addresses the emotional and informational needs of the driver.

These informational needs include:

Can I continue driving safely or do I need to stop?

Can I troubleshoot this and how?

Is the situation under control and being monitored?

“I wouldn’t turn [the voice assistant] off in stressful situations.  I like having updates since it gives me reassurance that I’m taking the necessary steps to resolve this issue with my car.” - Anne

Final Conversation Designs

Our designs take into account the research findings and aim to provide a frictionless user experience:

  • All the ways drivers talk to the car in real life.
  • The expectation is for guidance and help with troubleshooting.
  • Information quantity is balanced in order to reduce the cognitive load on the driver.


Hypervisor contextual data

I built into the interaction model a set of variables that provide context to the assistant. The assistant knows if there are any indicator lights being displayed, how many lights are displayed, and if the car is driving, or idling. Having access to this context allows the assistant to reliably answer questions about indicator lights with a single shot. It also allows the agent to tailor the information it provides to the given context. Lastly, the agent is able to determine which light a driver is referencing, when more than one is displayed, or disambiguate if required.

Disambiguation flow

There are edge cases when a driver may want to ask about an intermittent indicator light that may not be lit up. The car assistant should be able to answer questions within this domain and will seek out information that it needs to find the right answer. I also used this flow to demonstrate how much information is required by the assistant in order to know which indicator light the driver is speaking about, which is why contextual awareness is so important.


By providing contextually relevant troubleshooting tips the car assistant can guide drivers like a guardian angel. With the driver's permission, the car assistant can initiate a number of protocols to help resolve engine trouble, and support with calling roadside service if necessary.

Final Value Proposition Canvas

Throughout this short engagement, I continued to iterate on the initial value proposition for the indicator light Action on Google Assistant (or a custom VA power by Google Assistant). By gaining a deeper understanding of our customer segments’ needs we’re able to identify and validate high-value features that create a more direct product-market fit with their most pressing needs, by reducing their pains and amplifying their gains.


Above all, drivers need to feel safe. Indicator lights can be ambiguous, and cause anxiety. For the driver, it often means they need to gather information and occasionally may require them to troubleshoot. A car assistant should be able to address both of these needs, by providing necessary information efficiently and safely.


Our proof of concept demonstrates the validity of our design approach, which allows us to move fast and ensure that we deliver a validated and successful user experience to our stakeholder's customers. Our low-fidelity iterative prototyping approach allows us to uncover and fix conversation design flaws early before a big investment of engineering time and budget is made.

We recommended to our stakeholders that they apply the same approach to the next phase of delivery and by having a conversation design team run ahead for the engineering team and iteratively ship validated conversation flows to them for implementation.