Please join us in ECCR 245 and on Zoom.

Abstract: Deep learning has driven remarkable progress in robotics, largely through the use of large foundational models. Despite their success, these models demand vast amounts of data, extensive computational resources, and lack the robust reasoning needed for intelligent robots to operate in human-centric environments. In this talk, I will show how neuro-symbolic approaches bridge this gap by integrating the expressiveness of neural networks with the interpretable reasoning capabilities of symbolic approaches to advance robotic intelligence. First, I show how augmenting neural models with knowledge graphs enables short-context action anticipation in human-robot collaboration by reasoning over contextual clues like object affordances and relationships. I then discuss strategies to quickly adapt these graphs, introducing an approach for few-shot object recognition and how large language models can be leveraged to generate hypotheses for zero-shot object manipulation. Finally, I will introduce an approach that utilizes neural networks to generate task-specific symbolic controllers for tabletop manipulation from language and vision, enabling efficient real-time robot control. Together, these methods pave the way for intelligent robots in complex, real-world applications - from in-home assistance to industrial automation - while addressing key challenges in life-long learning, interpretability, and efficiency.

Bio: Simon Stepputtis is a Postdoc at Carnegie Mellon University’s Robotics Institute, where his research focuses on leveraging neuro-symbolic approaches to enhance the efficiency, flexibility, and interpretability of vision and embodied systems. His work involves developing innovative methods to integrate domain knowledge and symbolic reasoning into neural networks, enabling these systems to interact more effectively with dynamic environments and the humans within them. Before joining CMU, Simon earned his Ph.D. from Arizona State University, where he specialized in physical human-robot interaction, utilizing language to condition robot behavior. He has also contributed to industrial robot manipulation projects at Bosch and X, The Moonshot Factory. His research has been presented at conferences such as NeurIPS and CoLLAs, where he received spotlight presentations. Additionally, he was awarded Nvidia’s Best Poster Award at the Southwest Robotics Symposium.