TR2026-146

DYNAMIT: Dynamic Zero-Shot Manipulation via Instruction-Grounded Visual Tracking and Interception in Industrial Conveyor Environments

    •  Singh, H., Suzuki, K., Wang, Y., Liu, J., Cascante-Bonilla, P., Koike-Akino, T., "DYNAMIT: Dynamic Zero-Shot Manipulation via Instruction-Grounded Visual Tracking and Interception in Industrial Conveyor Environments", IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Workshop, September 2026.
      BibTeX TR2026-146 PDF
      • @inproceedings{Singh2026sep,
      • author = {Singh, Harsh and Suzuki, Kei and Wang, Ye and Liu, Jing and Cascante-Bonilla, Paola and Koike-Akino, Toshiaki},
      • title = {{DYNAMIT: Dynamic Zero-Shot Manipulation via Instruction-Grounded Visual Tracking and Interception in Industrial Conveyor Environments}},
      • booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Workshop},
      • year = 2026,
      • month = sep,
      • url = {https://www.merl.com/publications/TR2026-146}
      • }
  • MERL Contacts:
  • Research Areas:

    Artificial Intelligence, Machine Learning

Abstract:

Manipulation on industrial conveyors is inherently time-sensitive. Conveyor motion can shift the required grasp or placement location before the robot’s end effector reaches it. Existing approaches may require task-specific demonstrations, policy training, instance-specific object models, or configured conveyor motion, which can limit transfer to new objects and motion conditions. We introduce DYNAMIT, an instruction-conditioned, zero-shot framework for conveyor pick-and-place that operates without these requirements. The language instruction specifies the objects and receiving receptacles, while memory-based visual tracking preserves their identities in RGB-D observations. Depth measurements and track histories provide estimates of each instance’s position, dimensions, and velocity. A reachability-aware planner uses these estimates and robot proprioception to plan when and where each grasp or placement should occur, accounting for the end effector’s travel time. The planner updates this timing and location during execution, allowing the same pipeline to grasp moving objects and place objects into moving receptacles. We evaluate two conveyor tasks in simulation at different conveyor speeds and test transfer to industrial objects. DYNAMIT achieves the highest aggregate success rate and Q-score among the evaluated methods in each conveyor study. Evaluation on the Dynamic Object Manipulation (DOM) benchmark further demonstrates transfer to new scenes and task conditions without DOM demonstrations or policy training.

 

  • Related News & Events

    •  NEWS    MERL contributes to IROS 2026
      Date: September 27, 2026 - October 1, 2026
      Where: The IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
      MERL Contacts: Siddarth Jain; Toshiaki Koike-Akino; Jing Liu; Daniel N. Nikovski; Arvind Raghunathan; Alexander Schperberg; Kei Suzuki; Ye Wang
      Research Areas: Artificial Intelligence, Computer Vision, Control, Machine Learning, Optimization, Robotics, Signal Processing
      Brief
      • MERL made broad contributions to the technical program and robotics community at the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026), held in Pittsburgh, Pennsylvania. MERL researchers presented one main-conference paper and five workshop papers, participated in an editorial board meeting, competed in the Humanoid IKEA Assembly Challenge, and contributed to the organization of an IROS workshop.

        Main Conference Paper
        • ORIGAMI: Object Representation Inferred Geometrically for Articulated ManIpulation, Yunfu Deng and Daniel N. Nikovski (TR2026-141)


          • The work introduces a geometric representation for articulated-object manipulation, enabling robots to infer compact representations of previously unknown articulated mechanisms for downstream learning and control.

        Workshop Papers
        • Deliberate Practice: Learning Robot Skills under a Budget Shivam Vats, Sudarshan Harithas, Mete Akbulut, Arvind Raghunathan, and George Konidaris (TR2026-147)


        • ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies, Haodi Hu, Chung-Ta Huang, Jing Liu, Ye Wang, Kei Suzuki, Matthew Brand, and Toshiaki Koike-Akino (TR2026-148)


        • Test-Time Attention: Can Robots Better Follow Commands? Jing Liu, Ye Wang, Kei Suzuki, and Toshiaki Koike-Akino (TR2026-142)


        • Read the Manual: Grounding Behavior Tree Synthesis for Multi-Machine Factory Operation, Chak Lam Shek, Ye Wang, Jing Liu, Kei Suzuki, Pratap Tokekar, and Toshiaki Koike-Akino (TR2026-149)


        • DYNAMIT: Dynamic Zero-Shot Manipulation via Instruction-Grounded Visual Tracking and Interception in Industrial Conveyor Environments, Harsh Singh, Kei Suzuki, Ye Wang, Jing Liu, Paola Cascante-Bonilla, and Toshiaki Koike-Akino (TR2026-146)

          Together, these works address a range of challenges in modern robotics, including articulated-object manipulation, efficient robot skill learning, vision-language-action policies, test-time adaptation, autonomous factory operation, and dynamic manipulation. The paper on ReCoVLA was nominated as a spotlight talk.


        Robotics Community Contributions

        MERL Principal Research Scientist Dr. Siddarth Jain participated in the IEEE Robotics and Automation Letters (RA-L) editorial board meeting at IROS. Dr. Jain serves as an Associate Editor of RA-L, contributing to the peer-review and editorial activities of the robotics research community.

        Former MERL scientist Dr. Diego Romeres was also among the organizers of the 1st International Workshop on Industrial Applications of Robot Learning (IARL 2026). The workshop brought together researchers from academia and industry to discuss how advances in robot learning can be translated into reliable and scalable real-world industrial robotic systems.

        Humanoid IKEA Assembly Challenge

        MERL also participated in the IROS 2026 Humanoid IKEA Assembly Challenge with Team MEL-Craft. The team achieved first place in the competition, demonstrating autonomous humanoid manipulation capabilities for a challenging furniture-assembly task. The MEL-Craft team included MERL researchers Kei Suzuki, Jing Liu, Alexander Schperberg, Toshiaki Koike-Akino, and Ye Wang, with contributions from MERL interns Haodi Hu, Harsh Singh, Chak Lam Shek, and Maxwell Asselmeier. The competition achievement is highlighted separately in MERL's related award announcement.

        About IROS

        The IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) is a major international robotics conference bringing together researchers, engineers, and industry practitioners working across intelligent robots and systems. IROS 2026 took place in Pittsburgh from September 27 to October 1, 2026.

    •