Collaborative Research: CCRI: New: A Research News Recommender Infrastructure with Live Users for Algorithm and Interface Experimentation

Machine learning recommender systems personalize users’ experiences online by ranking and selecting items to present based on users’ past behavior. For example, when a user visits an online retailer, the products shown are selected by a recommender system designed to help one find things to buy and to increase the vendors sales. Recommender systems are also behind most online news sources, and they can shape which news people see. Given the importance of recommender systems to individual choice, it is critical for researchers to be able to carry out studies to evaluate different designs and their impact on the users of the system. But conducting such studies is beyond the resources of most researchers. To get meaningful results requires building and sustaining a community of willing users who have given their permission to be studied. As a result, the amount of experimental research – and specifically experimental research on long-term users of a system – has plummeted. Almost all such studies are conducted by commercial recommendation platforms and their results are rarely made known to the public. This project is designed to develop a shared news recommender system specifically to enable researchers nationwide to be able to carry out experiments and learn just how different algorithms and interfaces affect users. This should create the knowledge that will allow the community to fully understand the impact of these systems and design new recommender systems that can enhance fairness and equity. When complete, this research infrastructure will support researchers in answering critical questions about how complex and often opaque recommender systems affect user behavior and to test new systems that can improve these systems and their outcomes. This community-centered project will design and build an experimental news recommender community infrastructure to support research in personalization and recommender systems, AI and machine learning, natural language processing, human-computer interaction, social computing, and other fields that would benefit from the ability to carry out online field experiments with long-term users of a system. The cloud-based software infrastructure includes a pluggable recommendation architecture in which researchers can deploy custom algorithms and interfaces, a feed of news articles starting with those obtained through a partnership with the Associated Press, experiment-support modules including consent, payment, and surveying of subjects, and support for two news interfaces—first a news digest and then a progressive web news browser. The infrastructure will maintain a set of long-term consented users, provide extensive support to researchers including overarching IRB protocols, training, sample experiments, datasets and metrics, and live support through a researcher support team. It will be governed by a community advisory board drawn from the researcher community with representatives of the content providers and end-users and charged with allocating experiment slots and steering the development and management of the infrastructure. By developing and deploying this research infrastructure, the investigators seek to empower individuals and small groups to study important questions in recommender systems, including questions about how different algorithms and interfaces can alter the diversity of sources and viewpoints represented and provide users with greater understanding and control over the content they explore. The investigators come from five institutions spread across the country and will in turn assemble and train a diverse team to take on this technically challenging and important work. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.