Often the design and development of conversational interfaces are given insufficient or no user testing is done in the early stages. Features aren’t being validated. At the time we began this project, best practices for user testing might include two people to sit in a room and have one pretend to be a computer. For high fidelity prototyping, there was a high technical barrier to entry making them inaccessible.
Our team had started working on an increasing amount of conversational AI projects for The Alexa and Google Assistant-enabled devices that were coming onto the market and to help us with these projects, and out of necessity, we ended up developing Simili, a design tool that allows conversation designers to quickly prototype, test and iterate on dialogue, which helps designers to check to validate their hypothesis and test for usability and desirability.
Simili
We designed Simili to be platform-agnostic, and allow for prototyping, testing, and analytics for both voice and text chat. Fundamentally lean, enabling designers to surface user insights and validate hypotheses and designs as quickly as possible.
Build
Prototype your project configuration in the dialogue flow editor.
Measure
Perform live user testing with the operator console and companion test client.
Learn
Synthesize with analytics Collate & export utterance data
High experiential fidelity
Wizard of Oz testing can present a much higher degree of experiential fidelity than other types of low fidelity prototyping, such as paper prototypes. When we developed Simili there was only two low-code conversation design tools available, and neither were nearly as fast and efficient as using Simili. Even today there is a gap in tools available for WoZ testing.
Key Features
Dialogue design
Quickly design and prototype your dialogue flows within the dialogue editor. Conversation designers can quickly mock up dialog flows, and establish position and context by nesting responses. Simple keyword matching is used to suggest matches during WoZ testing. Because of this, no NLU training is required. Essentially, the operator is using their own natural language understanding, but Simili helps.
Testing & transcripts
Test your prototype with real users as a chatbot or voice agent. A testing console allows the operator to view the real time transcript, and select agent responses from their dialog, including an automatic surfacing of likely responses based on simple keyword matching. Transcripts are also generated and exportable and highlight when a no match response is used, or the operator needs to manually respond via improvised text input.
Measuring & exporting data
Synthesize your insights with analytics and exportable transcripts. It's easy to see which intents were most popular, and a list of utterances which resulted in no-match and fallback conversation repair. A csv export feature also allows us to conduct further analysis in Google Sheets.
Internal & External testing
We designed Simili to be platform-agnostic, and allow for prototyping, testing, and analytics for both voice and text chat. To test our product-market fit and usability we tested Simili extensively as well as externally. .
Demo
Some of our clients were interested in having their internal teams us Simili, and so we refined and added several features in response. Having external teams use Simili provided us with an opportunity to learn a lot.
Login & account management
Most importantly was login and account management, essential to allow for privacy and account management.
Flow categories & nested context
To help manage more complex or lengthly conversation flows we added categories and nested flows. During tests, responses are tracked, and nested responses will be prioritized in the WoZ suggestions by that position and conversational context.
Transcripts
For our own purposes we were able to access transcripts in our cloud storage but our clients didn't have this access. We build a transcript manager and the ability to export the transcripts as a downloadable PDF or CSV file.
Promotional video
We produced a short video to communicate its value proposition to potential users as well as potential investors.
This effort began in 2015 and was used and developed whenever we were working on CAI products. We would also jump back on it when we had unallocated team members and were able to create a small project team. Simili was used internally on several projects involving conversational AI and internal development continued on an as-needs basis over a two-year span. Eventually, we developed account management so we could provide secure access for several of our clients who used it for their own user testing needs.