The Community for Technology Leaders
Green Image
Issue No. 02 - April-June (2018 vol. 9)
ISSN: 1949-3045
pp: 255-270
Baohan Xu , School of Computer Science, Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai, China
Yanwei Fu , School of Data Science, Fudan University, Shanghai, China
Yu-Gang Jiang , School of Computer Science, Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai, China
Boyang Li , Disney Research, Pittsburgh, PA
Leonid Sigal , Disney Research, Pittsburgh, PA
ABSTRACT
Emotion is a key element in user-generated video. However, it is difficult to understand emotions conveyed in such videos due to the complex and unstructured nature of user-generated content and the sparsity of video frames expressing emotion. In this paper, for the first time, we propose a technique for transferring knowledge from heterogeneous external sources, including image and textual data, to facilitate three related tasks in understanding video emotion: emotion recognition, emotion attribution and emotion-oriented summarization. Specifically, our framework (1) learns a video encoding from an auxiliary emotional image dataset in order to improve supervised video emotion recognition, and (2) transfers knowledge from an auxiliary textual corpora for zero-shot recognition of emotion classes unseen during training. The proposed technique for knowledge transfer facilitates novel applications of emotion attribution and emotion-oriented summarization. A comprehensive set of experiments on multiple datasets demonstrate the effectiveness of our framework.
INDEX TERMS
Emotion recognition, Training, Semantics, Image recognition, Feature extraction, Visualization, Knowledge transfer
CITATION

B. Xu, Y. Fu, Y. Jiang, B. Li and L. Sigal, "Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization," in IEEE Transactions on Affective Computing, vol. 9, no. 2, pp. 255-270, 2018.
doi:10.1109/TAFFC.2016.2622690
180 ms
(Ver 3.3 (11022016))