Abstract:

Industrial robot deployments are typically scripted for specific machines/tasks, an approach that does not scale to wide floors where robots must operate multiple machines with various procedures. Moreover, fixed scripts cannot readily adapt when a machine deviates from the expected procedure, whereas human operators can flexibly follow manuals that specify ordered steps, preconditions, and recovery actions. We present an agentic framework that grounds robotic machine operation in such manuals. The framework parses a manual into a memory representation and synthesizes a Behavior Tree whose condition nodes evaluate procedural preconditions against live machine state and whose action nodes invoke robot primitives. This representation enables the robot to detect failed preconditions and execute the corresponding manual-prescribed recovery actions while preserving the remaining procedure. We evaluate the approach in a factory environment containing three machines with physically actuated control panels, distinct operating procedures, and faults requiring physical recovery actions. The proposed method outperforms an ungrounded language-model baseline.