TY - JOUR
T1 - Advancing the Assessment of Automated Deception Detection Systems
T2 - Incorporating Base Rate and Cost into System Evaluation
AU - Twitchell, Douglas P.
AU - Fuller, Christie M.
N1 - Publisher Copyright:
© 2018 John Wiley & Sons Ltd
PY - 2019/5
Y1 - 2019/5
N2 - In the last two decades, there has been an increased interest in automated deception detection systems (ADDs) for use in screening, although little attention has been paid to the usefulness of these systems. ADDs use various means, both invasive and non-invasive, to ascertain individual intent to deceive or engage in malicious behaviour. Many papers introducing ADDs use signal detection theory to compare a technique's ability to detect malicious intent with other techniques, but in doing so, they do not include contextual information such as base rate and cost. In this paper, we aim to improve future research by showing how the inclusion of contextual information provides a more realistic picture of the research. Through both theoretical arguments and a real-data example, we show that especially for those contexts where malicious intent is infrequent (ie, with low base rates of deception) that not factoring in the base rate overestimates the accuracy and therefore usefulness. We conclude with recommendations for how future research should provide a fuller picture of the accuracy and usefulness of ADDs.
AB - In the last two decades, there has been an increased interest in automated deception detection systems (ADDs) for use in screening, although little attention has been paid to the usefulness of these systems. ADDs use various means, both invasive and non-invasive, to ascertain individual intent to deceive or engage in malicious behaviour. Many papers introducing ADDs use signal detection theory to compare a technique's ability to detect malicious intent with other techniques, but in doing so, they do not include contextual information such as base rate and cost. In this paper, we aim to improve future research by showing how the inclusion of contextual information provides a more realistic picture of the research. Through both theoretical arguments and a real-data example, we show that especially for those contexts where malicious intent is infrequent (ie, with low base rates of deception) that not factoring in the base rate overestimates the accuracy and therefore usefulness. We conclude with recommendations for how future research should provide a fuller picture of the accuracy and usefulness of ADDs.
KW - automated deception detection systems
KW - base rate
KW - Bayes' theorem
KW - signal detection theory
KW - usefulness
UR - http://www.scopus.com/inward/record.url?scp=85057029937&partnerID=8YFLogxK
UR - https://scholarworks.boisestate.edu/itscm_facpubs/95
U2 - 10.1111/isj.12231
DO - 10.1111/isj.12231
M3 - Article
SN - 1350-1917
VL - 29
SP - 738
EP - 761
JO - Information Systems Journal
JF - Information Systems Journal
IS - 3
ER -