Instance-Level Semantic Maps for Vision Language Navigation — SI Maps

1KCIS, International Institute of Information Technology, Hyderabad, India; 2CSAIL, MIT, Cambridge, United States; 3Hasan Kalyoncu University, Turkey
Note - Published at the conference IEEE RO-MAN 2023
SI-Maps Teaser

Teaser

Abstract

Humans have a natural ability to perform semantic associations with the surrounding objects in the environment. This allows them to create a mental map of the environment, allowing them to navigate on-demand when given linguistic instructions. A natural goal in Vision Language Navigation (VLN) research is to impart autonomous agents with similar capabilities. Recent works take a step towards this goal by creating a semantic spatial map representation of the environment without any labeled data. However, their representations are limited for practical applicability as they do not distinguish between different instances of the same object. In this work, we address this limitation by integrating instance-level information into spatial map representation using a community detection algorithm and utilizing word ontology learned by large language models (LLMs) to perform open-set semantic associations in the mapping representation. The resulting map representation improves the navigation performance by two-fold (233%) on realistic language commands with instance-specific descriptions compared to the baseline. We validate the practicality and effectiveness of our approach through extensive qualitative and quantitative experiments.

Language Navigation Demo With SI Maps

BibTeX

@INPROCEEDINGS{10309534,
  author={Nanwani, Laksh and Agarwal, Anmol and Jain, Kanishk and Prabhakar, Raghav and Monis, Aaron and Mathur, Aditya and Jatavallabhula, Krishna Murthy and Abdul Hafez, A. H. and Gandhi, Vineet and Krishna, K. Madhava},
  booktitle={2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)}, 
  title={Instance-Level Semantic Maps for Vision Language Navigation}, 
  year={2023},
  volume={},
  number={},
  pages={507-512},
  keywords={Measurement;Visualization;Three-dimensional displays;Navigation;Semantics;Linguistics;Ontologies},
  doi={10.1109/RO-MAN57019.2023.10309534}}