This workshop was incredible! I’m so grateful I was able to attend. I learned so much here, and the people I met were awesome.
The workshop was from 9am – 12pm in the Punchcut office near Embarcadero. The workshop was run by Vicky Knoop (Director of Immersive Design) and Emilio Passi (Principle Design Technologist). There were 12 participants – the students were me, Kendra, and a Punchcut visual design intern who was sitting beside me.
The workshop covered 4 main areas, as shown in the photo above. Along with the following notes I captured, I’ll briefly explain what we learned/did for each area.
Knowing when to use VUI
Why use VUI? We learned about the “superpowers” of VUI:
- Dexterity – perform tasks with full hands, at a distance
- Efficiency – certain tasks can be done faster with VUI rather than GUI
- Adaptability – VUI can understand context and provide suggestions on the user’s situation
- Inclusivity – people who have trouble using GUI have another option
- Empathy – conversation is very human
But why not apply VUI to everything?
- It can be hard to explore – usually VUI is focused, it’s harder to see all the available actions at once
- It can be hard to use in public – privacy/personal concerns
- It’s not always accurate – the technology still has much room for improvement
- It’s a little slow – can’t process information quickly, unlike GUI
Try to avoid creating an entirely novel experience – the objective is to use VUI to improve an experience and compliment other interfaces.
Designing for conversation
Create a relatable agent!
- Write for the ear.
- Stay in character.
- Don’t state the obvious.
- Don’t overpromise.
- Be discreet.
- Remember things.
Make your agent relatable using the cooperative principle. We discussed Brice’s Maxims, which are four areas to keep in mind:
- Quality: what information is being provided?
- Quantity: how much information are you providing?
- Relation: is the information being discussed contextually relevant?
- Manner: is the tone of the agent cooperative?
If the user is being uncooperative, then be gracious. Move the conversation forward and try to understand the user’s intent in order to help them.
For this part of the workshop, we also did a VUI design activity in the groups we were sitting in. Kendra, Aastha, Sarah and I worked on “Lyft Loft”, a futuristic air travel taxi service. We wrote sample dialogues with the best practices we had learned.
When we read out these dialogues, one very nice touch the designers at Punchcut use are “agent cards” – the person who is the agent holds one so people don’t get confused during the read out as to who is the human and who is the agent.
We then dissected the dialogues to learn about the following 3 areas:
- intent – the purpose of the interaction (“global navigation”)
- utterances – what the user says, and the system translates to the intent (synonyms)
- entities – variables and components the user and system input/output
This exercise brought up an interesting point about building trust with a VUI. How should we design to balance stating the obvious with providing enough information so it doesn’t feel creepy?
Another thing to keep in mind are cultural considerations. These include:
- Different accents
- Different words different place use for the same thing (e.g. athroom, washroom, restroom, loo, etc)
- Preference for tone (e.g. Americans are more personable, Germans prefer straightforward and no fluff)
- Age range preferences (level of trust younger vs older people have for VUI)
Preparing for development
In this part, we learned about what we can create and what terms to use when communicating with developers to build the agent. First, we discussed that AI is not magic – there are limitations and things to think about when designing conversation flows. The flow in how it works is: automated speech recognition > intent classifier > fulfillment > speech synthesis.
A couple important terms:
- turns – how the user and agent interact
- utterances – what the user says, which the system uses to determine the intent
- intent – the purpose of the interaction
- invocation – wake word
- entity – variables
- barge-in – when the user can interrupt the agent
- escalation – increasing the amount of information being provided to the user as the conversation progresses
The prototyping tools Vicky and Emilio suggested using were (from simplest to most complex):
- Adobe Xd
- If This Then That
- Alexa Skills
Looking toward the future
Finally, our workshop ended with what to expect for VUI in the future. This part was especially exciting for me, as I am already exploring most, if not all, of these points in my senior project.
- Natural conversations
- Easier, more casual conversation
- Better understanding of user intent
- Using previous knowledge
- Considering emotions
- Proactive agents
- Taking actions before being invoked
- Learning user behaviours and adjusting to preferences
- Contextual awareness
- Connecting to sensors
- Being aware of who and what is in the room
- Identifying user actions and activities
- Providing responses that react to user actions and environments
- Eye contact
- Coordinated services
- Using multiple sources of information to understand context
- Using multiple sources of information to provide better responses
- One agent across the complete journey – all services/devices
Overall, this was an incredibly fun and educational workshop. I think that VUI is a field that I am interested in exploring in my career, and this workshop made me realize how much more potential (and fun!) is in this area.