Multi-Modal User Interaction: Gesture + Speech using Augmented Reality Headsets

Multi-modal interaction, in particular gesture and speech, for augmented reality headsets is essential as this technology becomes the future of interactive computing. It is possible that in the near future, augmented reality glasses become pervasive and the preferred device. This talk will concentrate on the motivation behind gesture and speech user interaction, a recent study, and future work. The first part of the talk will describe a study where we demonstrated early and essential findings of gesture and speech user interaction. Findings include types of the gestures performed, the timing between gesturing and speech when used in multi-modality (130 milliseconds), workload (using NASA TLX), and a series of design guidelines resulting from this study. The second talk will describe future work, in particular collaborative multi-modal gesture interaction.

Dr. Francisco R. Ortega is an Assistant Professor at Colorado State University and Director of the natural user interaction lab (NUILAB). Dr. Ortega earned his Ph.D. in Computer Science (CS) in the field of Human-Computer Interaction (HCI) and 3D User Interfaces (3DUI) from Florida International University (FIU). He also held a position of Post-Doc and Visiting Assistant Professor at FIU between February 2015 to July 2018. Broadly speaking, his research has focused on gesture interaction, which includes gesture recognition and elicitation. His main research area focuses on improving user interaction by (a) eliciting (hand and full-body) gesture sets by user elicitation, and (b) developing interactive gesture-recognition algorithms. His secondary research aims to discover how to increase interest for CS in non-CS entry-level college students via virtual and augmented reality games. His research has resulted in multiple peer-reviewed publications in venues such as ACM ISS, ACM SUI, and IEEE 3DUI, among others. He is the first-author of Interaction Design for 3D User Interfaces: The World of Modern Input Devices for Research, Applications, and Game Development book by CRC Press. Dr. Ortega serves as Vertically Integrated Projects coordinator that promotes applied research for undergraduate students across disciplines.