TR2026-148

ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies

    •  Hu, H., Huang, C.-T., Liu, J., Wang, Y., Suzuki, K., Brand, M., Koike-Akino, T., "ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies", IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Workshop, September 2026.
      BibTeX TR2026-148 PDF
      • @inproceedings{Hu2026sep,
      • author = {Hu, Haodi and Huang, Chung-Ta and Liu, Jing and Wang, Ye and Suzuki, Kei and Brand, Matthew and Koike-Akino, Toshiaki},
      • title = {{ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies}},
      • booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Workshop},
      • year = 2026,
      • month = sep,
      • url = {https://www.merl.com/publications/TR2026-148}
      • }
  • MERL Contacts:
  • Research Areas:

    Artificial Intelligence, Machine Learning

Abstract:

Vision-language-action (VLA) policies provide strong priors for language-conditioned manipulation, but remain brittle in off-nominal states requiring targeted recovery. We propose ReCoVLA—a failure-conditioned residual recovery framework that keeps a pretrained VLA policy frozen, uses an external vision-language model (VLM) to infer the failure mode and recovery stage, and compiles a structured reward from task-relevant components. Rather than using the VLM to generate actions or rewards directly, ReCoVLA uses it as a semantic reward selector: it predicts a recovery descriptor and reward mask for in-simulation residual-policy training, followed by zero-shot sim-to-real deployment of the trained recovery policies. This decouples high-level failure understanding from low-level corrective control to support different VLAs. Experiments across short-horizon, long-horizon, and contact-rich manipulation tasks show that ReCoVLA outperforms the tested baselines on average. In simulation, our reward compiler improves average success from 36.7% for the finetuned pi 0.5 baseline to 66.7%. In physical zero-shot sim-to-real experiments, ReCoVLA achieves the best average performance, with 61.7% success.

 

  • Related News & Events

    •  NEWS    MERL contributes to IROS 2026
      Date: September 27, 2026 - October 1, 2026
      Where: The IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
      MERL Contacts: Siddarth Jain; Toshiaki Koike-Akino; Jing Liu; Daniel N. Nikovski; Arvind Raghunathan; Alexander Schperberg; Kei Suzuki; Ye Wang
      Research Areas: Artificial Intelligence, Computer Vision, Control, Machine Learning, Optimization, Robotics, Signal Processing
      Brief
      • MERL made broad contributions to the technical program and robotics community at the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026), held in Pittsburgh, Pennsylvania. MERL researchers presented one main-conference paper and five workshop papers, participated in an editorial board meeting, competed in the Humanoid IKEA Assembly Challenge, and contributed to the organization of an IROS workshop.

        Main Conference Paper
        • ORIGAMI: Object Representation Inferred Geometrically for Articulated ManIpulation, Yunfu Deng and Daniel N. Nikovski (TR2026-141)


          • The work introduces a geometric representation for articulated-object manipulation, enabling robots to infer compact representations of previously unknown articulated mechanisms for downstream learning and control.

        Workshop Papers
        • Deliberate Practice: Learning Robot Skills under a Budget Shivam Vats, Sudarshan Harithas, Mete Akbulut, Arvind Raghunathan, and George Konidaris (TR2026-147)


        • ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies, Haodi Hu, Chung-Ta Huang, Jing Liu, Ye Wang, Kei Suzuki, Matthew Brand, and Toshiaki Koike-Akino (TR2026-148)


        • Test-Time Attention: Can Robots Better Follow Commands? Jing Liu, Ye Wang, Kei Suzuki, and Toshiaki Koike-Akino (TR2026-142)


        • Read the Manual: Grounding Behavior Tree Synthesis for Multi-Machine Factory Operation, Chak Lam Shek, Ye Wang, Jing Liu, Kei Suzuki, Pratap Tokekar, and Toshiaki Koike-Akino (TR2026-149)


        • DYNAMIT: Dynamic Zero-Shot Manipulation via Instruction-Grounded Visual Tracking and Interception in Industrial Conveyor Environments, Harsh Singh, Kei Suzuki, Ye Wang, Jing Liu, Paola Cascante-Bonilla, and Toshiaki Koike-Akino (TR2026-146)

          Together, these works address a range of challenges in modern robotics, including articulated-object manipulation, efficient robot skill learning, vision-language-action policies, test-time adaptation, autonomous factory operation, and dynamic manipulation. The paper on ReCoVLA was nominated as a spotlight talk.


        Robotics Community Contributions

        MERL Principal Research Scientist Dr. Siddarth Jain participated in the IEEE Robotics and Automation Letters (RA-L) editorial board meeting at IROS. Dr. Jain serves as an Associate Editor of RA-L, contributing to the peer-review and editorial activities of the robotics research community.

        Former MERL scientist Dr. Diego Romeres was also among the organizers of the 1st International Workshop on Industrial Applications of Robot Learning (IARL 2026). The workshop brought together researchers from academia and industry to discuss how advances in robot learning can be translated into reliable and scalable real-world industrial robotic systems.

        Humanoid IKEA Assembly Challenge

        MERL also participated in the IROS 2026 Humanoid IKEA Assembly Challenge with Team MEL-Craft. The team achieved first place in the competition, demonstrating autonomous humanoid manipulation capabilities for a challenging furniture-assembly task. The MEL-Craft team included MERL researchers Kei Suzuki, Jing Liu, Alexander Schperberg, Toshiaki Koike-Akino, and Ye Wang, with contributions from MERL interns Haodi Hu, Harsh Singh, Chak Lam Shek, and Maxwell Asselmeier. The competition achievement is highlighted separately in MERL's related award announcement.

        About IROS

        The IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) is a major international robotics conference bringing together researchers, engineers, and industry practitioners working across intelligent robots and systems. IROS 2026 took place in Pittsburgh from September 27 to October 1, 2026.

    •  

  • Related Research Highlights

  • Related Publication

  •  Hu, H., Huang, C.-T., Liu, J., Wang, Y., Suzuki, K., Brand, M., Koike-Akino, T., "ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies", arXiv, June 2026.
    BibTeX arXiv
    • @article{Hu2026jun,
    • author = {Hu, Haodi and Huang, Chung-Ta and Liu, Jing and Wang, Ye and Suzuki, Kei and Brand, Matthew and Koike-Akino, Toshiaki},
    • title = {{ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies}},
    • journal = {arXiv},
    • year = 2026,
    • month = jun,
    • url = {https://arxiv.org/abs/2606.09630}
    • }