Training the AI models for autonomous vehicles is a laborious process hampered by traditional 2D tooling. We were asked to prototype a solution for labelling training data using a Virtual Reality (VR) interface. Our concept and prototype addressed challenges in VR sickness and was designed to increase the speed and accuracy of tagging training data for AI models. The goal was to enable annotators to get have a more intuitive and efficient method of annotating LiDAR datasets. Our team kicked off this project with two main objectives: render and navigate within a LiDAR scene through virtual reality and create and edit bounding boxes within a LiDAR scene. I worked with another designer in the first few weeks of research and ideation and then worked with a software engineer over the last 3 weeks of design and development. The project was completed in 5 weeks.
Benchmarking & Technical Research
We did some benchmarking of a wide range of VR applications to understand which design patterns were best to incorporate.
We also looked at software for lidar annotation to better understand what would be helpful to data scientists.
In addition, we researched a wide range of Point Cloud data formats to find one that worked well in three.js, A-Frame, and webxr.
Concept Development
We used a software called Sketchbox to translate the most promising concepts from our ideation into low fidelity prototypes in VR which we reviewed with our stakeholder.
User Flows
We began further defining the specific interaction patterns and user flows of our functional prototype with storyboarding of our customer journey.
Prototype Development
Our prototype allows users to navigate in different ways through an AV lidar point cloud scene, as well as basic data annotation and editing functionality. We built this prototype to help validate the desirability and usability of VR visualization for data scientists and our assumption that it would help to increase productivity. Included Features:
- Teleport navigation
- On-the-fly scene scaling
- Palette Menu for annotation
Features
Teleport navigation
Navigating around the point cloud data is done with room-based tracking as well as a teleport interaction, allowing the user to cover wide distances with a point and click teleport beam.
Scene scaling
Viewing a scene from multiple perspectives and scales allows for faster interpretation. Users can quickly adjust the scale of the scene with the right joystick by toggling left and right.
Annotation
A laser pointer allows user to annotate their point cloud scene with labelled bounding boxes. They can just as easily delete.
Palette menu
The user's left hand carries a tabbed palette menu that they can select labels from and also view a list of their current annotations.
Hover highlighting
Bounding boxes are labelled with their annotation but the user can also hover with the pointer to see a label from a distance.
Outcome & Next Steps
We completed the project by developing a web-based prototype that allows users to navigate in different ways through an AV LiDAR point cloud scene, as well as basic data annotation and editing functionality.
In testing we found that the laser pointer was dropping the framerate, resulting in some motion sickness. We mitigated this by having the pointer timeout after several seconds of not being inactive, and reactivating when the trigger was pressed.
We successfully validated our prototype with users and used what we learn in our interviews and testing sessions to help us outline a roadmap of future features.
If you have a VR headset you can try our prototype out here: https://prj-holodeck-prototype.glitch.me
Next steps
- Streaming of large point cloud datasets using octree point cloud formats
- Integration with existing tools and pipelines
- Incorporation of camera data, and timelines
- Timeline scrubbing
- Multi-user collaboration
- Automatically sizing of bounding boxes, with manual adjustment
- Mobile AR exploration