TY - CHAP
T1 - Resolving References to Objects in Photographs using the Words-As-Classifiers Model
AU - Schlangen, David
AU - Zarrieß, Sina
AU - Kennington, Casey
N1 - David Schlangen, Sina Zarrieß, Casey Kennington. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2016.
PY - 2016
Y1 - 2016
N2 - A common use of language is to refer to visually present objects. Modelling it in computers requires modelling the link between language and perception. The “words as classifiers” model of grounded semantics views words as classifiers of perceptual contexts, and composes the meaning of a phrase through composition of the denotations of its component words. It was recently shown to perform well in a game-playing scenario with a small number of object types. We apply it to two large sets of real-world photographs that contain a much larger variety of object types and for which referring expressions are available. Using a pre-trained convolutional neural network to extract image region features, and augmenting these with positional information, we show that the model achieves performance competitive with the state of the art in a reference resolution task ( given expression, find bounding box of its referent ), while, as we argue, being conceptually simpler and more flexible.
AB - A common use of language is to refer to visually present objects. Modelling it in computers requires modelling the link between language and perception. The “words as classifiers” model of grounded semantics views words as classifiers of perceptual contexts, and composes the meaning of a phrase through composition of the denotations of its component words. It was recently shown to perform well in a game-playing scenario with a small number of object types. We apply it to two large sets of real-world photographs that contain a much larger variety of object types and for which referring expressions are available. Using a pre-trained convolutional neural network to extract image region features, and augmenting these with positional information, we show that the model achieves performance competitive with the state of the art in a reference resolution task ( given expression, find bounding box of its referent ), while, as we argue, being conceptually simpler and more flexible.
UR - https://dx.doi.org/10.18653/v1/P16-1115
U2 - 10.18653/v1/P16-1115
DO - 10.18653/v1/P16-1115
M3 - Chapter
BT - Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
ER -