MARCER: Multimodal Augmented Reality for Composing and Executing Robot Tasks
Abstract
Multimodal Augmented Reality for Composing and Executing Robot Tasks (MARCER) is a multimodal end-user robot programming system that combines large language models, trigger-action programming, and augmented reality to enable users to create reactive robot programs using verbal commands. This was published at HRI 2025.
Date
March 4, 2025
Type
Conference
Name
HRI 2025
In this work, we combine the strengths of humans and robots by developing Multimodal Augmented Reality for Composing and Executing Robot Tasks (MARCER), a novel interactive and multimodal end-user robot programming system. MARCER utilizes a Large Language Model to translate users’ natural language task descriptions and environmental context into Action Plans for robot execution, based on a trigger-action programming paradigm that facilitates authoring reactive robot behaviors. MARCER also affords interaction via augmented reality to help users parameterize and validate robot programs and provide real-time, visual previews and feedback directly in the context of the robot’s operating environment. We present the design, implementation, and evaluation of MARCER to explore the usability of such systems and demonstrate how trigger-action programming, Large Language Models, and augmented reality hold deep-seated synergies that, when combined, empower users to program general-purpose robots to perform everyday tasks.
- Posted on:
- March 4, 2025
- Length:
- 1 minute read, 192 words
- Categories:
- Conference HRI
- Tags:
- Conference HRI
- See Also:
- Overlapping Social Navigation Principles: A Framework for Social Robot Navigation
- ARCADE: Scalable Demonstration Collection and Generation via Augmented Reality for Imitation Learning
- 🏆 The Cyber-Physical Control Room: A Mixed Reality Interface for Mobile Robot Teleoperation and Human-Robot Teaming