Scene-aware Interaction

    •  Hori, C., Tsuchiya, M., Chen, S., Cherian, A., Hori, T., Harsham, B.A., Marks, T.K., Le Roux, J., Sullivan, A., Vetro, A., "マルチモーダルセンシング情報に基づくScene-aware Interaction 技術", Society of Automotive Engineers of Japan, Vol. 75, No. 5, pp. 66-71, May 2021.
      BibTeX TR2021-042 PDF Video
      • @article{Hori2021may,
      • author = {Hori, Chiori and Tsuchiya, Masato and Chen, Siheng and Cherian, Anoop and Hori, Takaaki and Harsham, Bret A. and Marks, Tim K. and Le Roux, Jonathan and Sullivan, Alan and Vetro, Anthony},
      • title = {マルチモーダルセンシング情報に基づくScene-aware Interaction 技術},
      • journal = {Society of Automotive Engineers of Japan},
      • year = 2021,
      • volume = 75,
      • number = 5,
      • pages = {66--71},
      • month = may,
      • url = {}
      • }
  • MERL Contacts:
  • Research Areas:

    Artificial Intelligence, Computer Vision, Human-Computer Interaction, Machine Learning, Speech & Audio


The recent artificial intelligence (AI) boom and intelligent use of data acquired from various sensors has accelerated the development of technologies needed to realize advanced human-like capabilities in machines. AI technologies have come a long way in accurately perceiving visual scenes and understanding speech. However, one important piece of technology is still missing: natural and context-aware human-machine interaction, where machines understand their surrounding scene from the human perspective and are able to share their understanding with humans using natural language. To bridge this communication gap, we have developed and built a new AI system, called scene-aware interaction, that enables machines to translate their perception and understanding of a scene and respond to it using natural language to interact more effectively with humans. This paper introduces an example application of scene-aware interaction to car navigation systems, which will provide drivers with intuitive route guidance.


  • Related Video