Expressing Uncertainty in Information Systems Analytics Research: A Demonstration of Bayesian Analysis Applied to Binary Classification Problems

The measures typically used to assess binary classification problems fail to incorporate the uncertainty inherent to many contexts into the results. We propose using a Bayesian model to express the uncertainty in binary classification problems. This study identified 10 previous studies that provided sufficient data to demonstrate the use of Bayesian analysis in Information Systems (IS) contexts with varying levels of uncertainty. The analysis and user study show that the addition of Bayesian analysis is most useful in high uncertainty contexts with a wide interval for positive predictive value. Such an interval will lead to high uncertainty, even with very certain sensitivity and specificity. The usefulness of Bayesian analysis in conditions of medium uncertainty depends on the context. In conditions of low uncertainty, Bayesian analysis does not add much value. The user study showed that presenting models with uncertainty changed researcher perception of which model performed the best with 18 of 21 researchers changing their opinion. We recommend that authors estimate the uncertainty in their models and provide confusion matrices and prevalence estimates in their results to enable Bayesian analysis as research in a domain matures.