The Community for Technology Leaders
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Honolulu, Hawaii, USA
July 21, 2017 to July 26, 2017
ISSN: 1063-6919
ISBN: 978-1-5386-0457-1
pp: 6478-6487
ABSTRACT
Visual narrative is often a combination of explicit information and judicious omissions, relying on the viewer to supply missing details. In comics, most movements in time and space are hidden in the gutters between panels. To follow the story, readers logically connect panels together by inferring unseen actions through a process called closure. While computers can now describe the content of natural images, in this paper we examine whether they can understand the closure-driven narratives conveyed by stylized artwork and dialogue in comic book panels. We collect a dataset, COMICS, that consists of over 1.2 million panels (120 GB) paired with automatic textbox transcriptions. An in-depth analysis of COMICS demonstrates that neither text nor image alone can tell a comic book story, so a computer must understand both modalities to keep up with the plot. We introduce three cloze-style tasks that ask models to predict narrative and character-centric aspects of a panel given n preceding panels as context. Various deep neural architectures underperform human baselines on these tasks, suggesting that COMICS contains fundamental challenges for both vision and language.
INDEX TERMS
art, image colour analysis, inference mechanisms, interactive systems, learning (artificial intelligence), neural nets, text detection
CITATION

M. Iyyer et al., "The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, Hawaii, USA, 2017, pp. 6478-6487.
doi:10.1109/CVPR.2017.686
92 ms
(Ver 3.3 (11022016))