The Challenge of Multi-Perspective in Artificial Intelligence: Language Understanding in Complex Environments

The interaction between humans and machines has made enormous progress in recent years. Natural Language Processing by Artificial Intelligence (AI) in particular has developed rapidly. However, one area that remains a challenge is the so-called "Grounding" of language, i.e. linking words and sentences to real or virtual environments. This is particularly complex when multiple perspectives have to be considered, as is the case, for example, in communication between multiple agents in a shared environment.

A research team has addressed this challenge and presented a new task and associated dataset for the generation and understanding of referring expressions in multi-agent environments. In this task, two agents sharing a scene must consider each other's visual perspective in order to produce and understand references to objects in the scene and their spatial relationships to each other.

The Importance of Perspective

Imagine trying to describe to a friend over the phone where to find a particular object in a room. You describe the position of the object from your point of view, but your friend sees the room from a different perspective. In this case, you need to be able to take your friend's perspective in order to provide them with an understandable description. The same is true for AI agents interacting in multi-agent environments.

To address this challenge, the researchers have created a dataset of 2,970 human-written referring expressions, each paired with human comprehension ratings. This dataset was used to evaluate the performance of automated models as speakers and listeners in interaction with human partners.

AI vs. Human: Room for Improvement

The results show that the models' performance in both generating and understanding references lags behind that of human agent pairs. This suggests that there is still much room for improvement in developing AI systems that can understand and use language in complex, multi-perspective environments.

One promising approach explored in the study is training an open-weight speaker model with cues to communicative success in combination with a listener model. This approach led to an improvement in communicative success from 58.9% to 69.3%, even outperforming the strongest proprietary model.

The Way Forward: Multimodal Models and Grounding

The research presented highlights the importance of developing AI systems that are able to understand language in its entirety, including the ability to consider different perspectives and link language to real or virtual environments. Future research in this area could focus on the development of multimodal models that combine different types of information, such as visual and language data, to enable a deeper understanding of language and its relation to the world.

The development of such AI systems is an important step towards more natural and effective human-machine interaction and opens up new possibilities in areas such as robotics, virtual assistants and autonomous systems.

Bibliography

Harnad, S. (1990). The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3), 335–346. Knoeferle, P., & Crocker, M. W. (2006). The coordinated interplay of scene, utterance, and common ground in real-time reference resolution. Psychological Science, 17(6), 452–459. Tan, H., & Bansal, M. (2019). LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 5099–5110). Yu, L., Poesio, M., & Traum, D. (2017). Incremental grounding of referring expressions in interactive dialogue. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) (pp. 20–29). Tang, Z., Mao, L., & Suhr, A. (2024). Grounding Language in Multi-Perspective Referential Communication. arXiv preprint arXiv:2410.03959.