Projects

---

Recent advances in Transformer-based Large Language Models (LLMs) have proven to be revolutionary in generating coherent conversations. However, their lack of symbol grounding and their sample inefficiency limits their applicability in robotics where adaptation to physical and social environments is essential. This project aims to address these limitations by grounding LLMs in an intrinsically motivated robotic agent that interacts socially, drawing inspiration from human language acquisition. Our approach is built on a dual model of symbol grounding that integrates individual sensorimotor grounding with social grounding of concepts. By combining our prior work, which grounded symbols in intrinsic motivations, with an LLM, we hope to create a system that learns from social cues (facial expressions, gaze, etc). Our four-fold work plan intends to: (1) integrate our current model with an LLM to cater to abstract language structures; (2) develop an intrinsically motivated social reward mechanism to adapt robot language output to the social interaction context; (3) deeply analyze the embedding space aspect of our model to better align them with human language learning mechanisms, using typical infant-caregiver strategies for sample efficiency and continuous learning and (4) evaluation that will comprise a comparison of our grounded model against traditional LLMs in both simulated and in a real-world environments and reproduction of cognitive studies.

---

This research challenges the traditional passive-learner approach in computational models of language development, where artificial agents are taught language as an isolated goal. Instead, it proposes a functional perspective, viewing communication as a means to achieve broader objectives, such as satisfying needs or fostering social bonds, reflecting observations in infant language development. While current models capture different aspects of language acquisition, they largely overlook the role of affect, emotions, and social interaction. This project aims to bridge this gap by integrating affective and motivational systems into language acquisition models for robots, enabling them to learn language in a socially interactive and goal-directed manner, similar to human infants. The research further seeks to explore how caregiver responses and emotional signals can shape language development, proposing an experimental framework to model and test these functions using reinforcement learning.

The funding is for a PhD student, Zakaria Lemhaouri and the environment.

---