Format:

Objectives:

The indicator light does not provide enough actionable information to the user for them to feel safe and plan their course of action.

Our participants reported seeking this information from various sources, including; Google search, the owner's manual, and through calling the dealership.

Paper manuals we derided by all of our interview participants. Users are accustomed to the quick results they can get from internet searches and mobile applications.

There is a risk in simply digitizing an experience that is already looked at unfavourably. Finding the most used sections of the user manual could potentially be starting place when deciding where a voice assistant could be a more efficient experience.

Poor user experiences and dated technology have led to a general perception that automotive voice assistants are of limited utility and that Siri, Alexa, and Google Assistant are better options to bring with them into their vehicle.

Because of this, it is especially important to ensure that the addition of voice assistant capabilities is supporting a great user experience and does not detract from it. The automotive assistant needs to be very capable within its own domain.

We synthesized our learnings into a summary document which included customer profiles for both the show's audiences as well as the onscreen talent. These profiles outlined their core jobs-to-be-done, as well as their pains and gains.

Jobs

Pains

Gains

TTS Voice synthesis prototypes that are linear in nature, and supported with visual stimuli. I presented these low-fidelity prototypes to our participants in a 1:1 setting during a zoom call.

Scenario 1 Testing both proactive and reactive prompts from the VA and documenting utterances from our driver to visual stimuli of a single indicator light turning on. (blue low temp coolant light).

Scenario 2 Testing troubleshooting concept features for desirability and usability.

Scenario 3 Testing utterances of different indicator combinations

Doing this research remotely brought new challenges. I knew that the context of the status of the car, and the GUI modality of the car's dash and indicator lights were both critical to prototyping this conversational interaction. I also wasn’t able to use simulation software because I wasn't able to control the participant's hardware.

My solution involved using Google Slides to provide the necessary context to the participant and a dash display. This multimodal experience is on track as progress the experience forward by tapping the arrow key, triggering animations. Assistant prompts are prerecorded and I listen to my participant's utterances and choose the appropriate response. I had a colleague demonstrate a test sequence in the video below.

Above all drivers see their most important jobs as staying safe. This involves monitoring the car status and making informed decisions when presented with potential problems. Our drivers really appreciated that the assistant was able to provide them with ways to troubleshoot the problem without needing to pull over and look at the user manual. For drivers; it’s important to know how critical is this issue and do they need to stop immediately, or can they complete their journey.

It’s important for the drivers to know how exactly how critical the issue is. If there are ways to troubleshoot it, and what they should do next. However, they don’t need to know these things all at once, and doing can be frustrating for them because it causes anxiety, cognitive overload and difficulty with comprehension. The conversational exchange should focus on efficiency while balancing the driver's cognitive load.

A majority of the drivers we spoke with made the assumption that the car assistant has information about which lights are on and other data points about the car status. This may be why drivers often don’t describe the indicator light icons, instead, they ask “What’s this light?”, or “What’s wrong?”.

Drivers reported feeling assured by the assistant's tone and language and described her as “a guardian angel”.

In stressful situations using a calming tone, plain language (avoid jargon) and providing helpful information kept our drivers calm and feeling safe and cared for.

Format: Crowd-sourced utterances across internal employees using a questionnaire in Google Forms.

Learning Objectives:

A pattern that we saw in our initial WoZ testing continues to be validated. When presented with a single light, many drivers simply ask the assistant what the light is, or what’s happening to the car. Supporting this intent will require the car to have contextual awareness of telematics data and system parameters. This insight has significant implications for other domains in the car environment as well.

Critically important is that only 10% (4 of 38) of the utterances captured in relation to a single indicator light contained enough information to correctly match to a specific indicator intent.

In a scenario where the assistant does not have access to indicator light status data a successful match a users utterance to an indicator light intent most, if not all of the following descriptors:

From this survey, it’s also apparent that the structure of utterances used by drivers varies greatly whether they are seeing 0,1, 2 or 3 lights.

Additional qualifiers were used when multiple lights were present and these point to important contextual data to provide to the assistant for disambiguation in cases of 2, 3+ indicator lights. These include:

With the large amount of utterance data we were able to gather, it was clear that our NLU interaction model needed to account for the diversity of ways that people may ask about indicator lights.

While intent matching for a single light is quite easy if the assistant has access to contextual data, the most difficult scenario is when users are asking about an indicator that is not being displayed. For this, we need extremely robust utterance data and a structure for entities and slot filling that accommodates that diversity.

High-fidelity prototype with NLU intent matching, error handling, multipath flows, conversation repairs and scenario with evolving context (e.g. while driving/ whole idle) outlined.

To create a functional prototype I had to train the NLU engine (LUIS) with structured sample utterance data, defined entities and intent structure. I did this by collating and annotating the utterance data I received in the WoZ testing and utterance survey.

Scenario 1 Testing a scenario where the car is experiencing 3 indicator lights in an increasingly critical issue with coolant temperature being below and above nominal temperature range.

Scenario 2 Testing to validate hypothesis around proactive/reactive. The assistant in this scenario is designed to be only reactive when addressing the non-critical low-temperature indicator however as the situation becomes more critical she is proactive in her updates.

Scenario 3 Testing the balance of contextual information based on troubleshooting strategies, and car status information like idling, driving, and ignition on/off. This helps us to validate hypotheses regarding cognitive load.

The interaction model is the structured data that defines the intents, entities and variables. An entity such as the shape descriptor can include many synonyms of the same slot value. A coolant temperature warning might be described as a boat, a ladder, or a flagpole.

It’s important to account for the diversity of ways a driver may describe this symbol. The utterance samples used to train our NLU model are taken from our user testing, and survey and can be improved over time.

The variables I defined are intended to provide our assistant with the necessary context required to provide one-shot intent matching, efficient information as well as cover edge cases when lights are not displayed or multiple lights are displayed at once.

{indicatorlight_shape}

thermometer, stick, waves, ocean, water, coolant, cooling water, sail, yacht, a yacht on waves, comb, flagpole, flag, ladder, ladder in the water, flag in the water, shape, boat, sailboat, ship, stick, pole, boat in the water, ladder, coolant water, coolant, coolant fluid, a boat in the water,

Entities

{indicator_light} Different ways that people might refer to “indicator light”

{indicatorlight_colour} Lights can be red, blue or orange, and defines synonyms

{indicatorlight_flashing} Descriptive words to define if the light is flashing, or solid, blinking or solid etc.

{indicatorlight_position} When more than one light, the user sometimes refers to the position of the light

{indicatorlight_shape} The 28 different indicator symbols and the many different ways people might describe them.

Variables

{sys_driving_idling}

Is the car driving or idling?

{sys_ignition_on_off}

Is the ignition on or off?

{sys_num_of_lights}

Are any indicator lights on, and how many?

{sys_light_colour}

What is the light colour?

{sys_symbol_id}

What is the symbol id of the light?

Contextual design

Within the prototype, I’m representing the context of the car’s status, and changing conditions with visuals, but also by updating variables at the moments we would expect these data points to be exposed to the assistant. In total there are 18 possible answers to the question of ‘What is this light’ because of the different contextual scenarios regarding the 3 engine coolant indicator combinations. The assistant’s response will vary depending on the contexual parameters of the symbol’s colour, whether the car is driving or idling, or whether an indicator light is even being displayed. These changing conditions will inherently change which dialogue is given to the driver.

Drivers had a wide range of preferences and opinions about if and when the assistant should be proactive and initiate interactions, or in some cases agentive and make non-vital decisions to help ease the cognitive load of the driver.

This may indicate that personalization settings may play an important role in dictating whether in the assistant’s reactive vs proactive nature.

Understanding how to define these thresholds of proactive, reactive and agentive behaviour would require additional research but may result in a safer and more helpful assistant.

This second prototype further validated the desirability of timely and concise troubleshooting tips.

It’s important however that this advice is always given contextually, and delivered at the right moment in a way that addresses the emotional and informational needs of the driver.

These informational needs include:

Can I continue driving safely or do I need to stop?

Can I troubleshoot this and how?

Is the situation under control and being monitored?

Our designs take into account the research findings and aim to provide a frictionless user experience: