CAREER: Integrating Interaction, Embodiment, and Emotion to Transform Language Models

When children learn language, they incorporate information from many sources including what they see, touch, smell, hear, and how they feel. Before children can speak, they feel emotions such as anger, joy, curiosity, and frustration and these emotions act as internal and external signals as they learn what words mean. For example, a child might show frustration when a caregiver does not understand what the child is saying, which in turn helps the child change what they understand a word or phrase to mean or how it is pronounced. The setting in which child language learning takes place is co-located where children are in the same physical location as the people they are learning language from, and the learning is through the communicative medium of spoken interaction. The information sources, emotions, and spoken interaction setting for children are in direct contrast to how machines learn language, which usually involves some kind of computational model that is given large amounts of text to learn from. This project aims to take inspiration from how children learn language in order to understand how to improve the models and methods of machines that learn language. Improved language learning in machines will enable machines to communicate with people more quickly, clearly, safely, and in ways that lower barriers for people to use complex technology with a natural spoken language interface.

This CAREER project examines a novel approach for computer systems to learn spoken language that will advance a theoretical model and improve how people and systems communicate. Research will improve language modeling in natural language processing by taking inspiration from how children learn language: they interact with others to learn words that denote physical entities and events, and, like all humans, often respond emotionally and embody how they feel in their behavior, whereas researchers currently largely train language models only on static text. The research team will use two robotic platforms for the research and significantly enrich model efficacy and will add knowledge of emotion by modeling it based on human perceptions of robot behaviors. The team will add embodied knowledge by grounding into vision and robot states, and finally, the team will train and evaluate a robot that uses the language model as it interacts with humans to learn language from them. The study will also result in two important datasets: robot behaviors with accompanying descriptions of those behaviors and emotion labels, and longitudinal data of robots interacting and learning language from humans. The objectives are to (1) Model emotion; test, and refine through interaction, (2) Develop a unified language model, and (3) Engage people and robots that learn language with emotional content.

This project is jointly funded by the CISE/IIS/Robust Intelligence Program and the NSF Established Program to Stimulate Competitive Research (EPSCoR).

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.