MARCER: Multimodal Augmented Reality for Composing and Executing Robot Tasks

Abstract

Multimodal Augmented Reality for Composing and Executing Robot Tasks (MARCER) is a multimodal end-user robot programming system that combines large language models, trigger-action programming, and augmented reality to enable users to create reactive robot programs using verbal commands. This was published at HRI 2025.

Date

March 4, 2025

Type

Conference

Name

HRI 2025

In this work, we combine the strengths of humans and robots by developing Multimodal Augmented Reality for Composing and Executing Robot Tasks (MARCER), a novel interactive and multimodal end-user robot programming system. MARCER utilizes a Large Language Model to translate users’ natural language task descriptions and environmental context into Action Plans for robot execution, based on a trigger-action programming paradigm that facilitates authoring reactive robot behaviors. MARCER also affords interaction via augmented reality to help users parameterize and validate robot programs and provide real-time, visual previews and feedback directly in the context of the robot’s operating environment. We present the design, implementation, and evaluation of MARCER to explore the usability of such systems and demonstrate how trigger-action programming, Large Language Models, and augmented reality hold deep-seated synergies that, when combined, empower users to program general-purpose robots to perform everyday tasks.



Posted on:
March 4, 2025
Length:
1 minute read, 192 words
Categories:
Conference HRI
Tags:
Conference HRI
See Also:
Overlapping Social Navigation Principles: A Framework for Social Robot Navigation
ARCADE: Scalable Demonstration Collection and Generation via Augmented Reality for Imitation Learning
🏆 The Cyber-Physical Control Room: A Mixed Reality Interface for Mobile Robot Teleoperation and Human-Robot Teaming