Understanding News Consumers' Perceptions of Believability: A Study of Real and Fake News

Social media have become essential in daily life, serving as platforms for public opinion, personal growth, and news consumption. This shift in how people access news has led to an increase in misinformation, including fake news intended to deceive. Various psychological and social factors, such as emotional appeal, cognitive biases, and social influence, drive individuals' susceptibility to believing false information. While prior studies have examined the believability of news content, large-scale analyses on real social media platforms remain limited, as well as studying factors influencing the believability of real and fake news separately and analyzing similarities and differences. This study introduces a new dataset of 14,535 Twitter user comments, annotated to measure user believability in real versus fake news. Using this dataset, we address the problem of predicting news believability and study which user or news characteristics predict believability in real and fake news, as well as checking whether believability enhances fake news detection. We employ machine learning models incorporating news style, emotional content, and user traits to predict believability and apply explainability methods to clarify key characteristics influencing user belief. Overall, the models achieved significant results in detecting news believability and several news and user-based features such as writing style, emotion, personality, and psychology have been individuated as strong predictors of believability in news. We further integrate believability insights into advanced fake news detectors, demonstrating improved performance. To our knowledge, this is the first large-scale human-annotated English dataset designed for studying news believability, which we have released for use in future research.