This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
BabyTalk: Understanding and Generating Simple Image Descriptions
Dec. 2013 (vol. 35 no. 12)
pp. 2891-2903
Girish Kulkarni, Comput. Sci. Dept., Stony Brook Univ., Stony Brook, NY, USA
Visruth Premraj, Comput. Sci. Dept., Stony Brook Univ., Stony Brook, NY, USA
Vicente Ordonez, Comput. Sci. Dept., Stony Brook Univ., Stony Brook, NY, USA
Sagnik Dhar, Comput. Sci. Dept., Stony Brook Univ., Stony Brook, NY, USA
Siming Li, Comput. Sci. Dept., Stony Brook Univ., Stony Brook, NY, USA
Yejin Choi, Comput. Sci. Dept., Stony Brook Univ., Stony Brook, NY, USA
Alexander C. Berg, Comput. Sci. Dept., Stony Brook Univ., Stony Brook, NY, USA
Tamara L. Berg, Comput. Sci. Dept., Stony Brook Univ., Stony Brook, NY, USA
We present a system to automatically generate natural language descriptions from images. This system consists of two parts. The first part, content planning, smooths the output of computer vision-based detection and recognition algorithms with statistics mined from large pools of visually descriptive text to determine the best content words to use to describe an image. The second step, surface realization, chooses words to construct natural language sentences based on the predicted content and general statistics from natural language. We present multiple approaches for the surface realization step and evaluate each using automatic measures of similarity to human generated reference descriptions. We also collect forced choice human evaluations between descriptions from the proposed generation system and descriptions from competing approaches. The proposed system is very effective at producing relevant sentences for images. It also generates descriptions that are notably more true to the specific image content than previous work.
Index Terms:
text analysis,computer vision,data mining,image recognition,natural language processing,forced choice human evaluations,BabyTalk,image descriptions,natural language description generation,content planning,computer vision-based detection algorithm,computer vision-based recognition algorithm,statistics mining,visually descriptive text,best content words,natural language sentences,surface realization step,Computer vision,Image segmentation,Natural language processing,Context awareness,Information analysis,Computer vision,image description generation
Citation:
Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C. Berg, Tamara L. Berg, "BabyTalk: Understanding and Generating Simple Image Descriptions," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 12, pp. 2891-2903, Dec. 2013, doi:10.1109/TPAMI.2012.162
Usage of this product signifies your acceptance of the Terms of Use.