The Community for Technology Leaders
Green Image
Issue No. 03 - July-Sept. (2018 vol. 9)
ISSN: 1949-3045
pp: 316-329
Yagmur Gucluturk , Radboud University, Nijmegen, HR, The Netherlands
Umut Guclu , Radboud University, Nijmegen, HR, The Netherlands
Xavier Baro , Open University of Catalonia, Barcelona, Spain
Hugo Jair Escalante , Instituto Nacional de Astrofísica, Puebla, Mexico
Isabelle Guyon , UPSud/INRIA Université Paris-Saclay, Saint-Aubin, France
Sergio Escalera , University of Barcelona, Barcelona, Spain
Marcel A. J. van Gerven , Radboud University, Nijmegen, HR, The Netherlands
Rob van Lier , Radboud University, Nijmegen, HR, The Netherlands
ABSTRACT
People form first impressions about the personalities of unfamiliar individuals even after very brief interactions with them. In this study we present and evaluate several models that mimic this automatic social behavior. Specifically, we present several models trained on a large dataset of short YouTube video blog posts for predicting apparent Big Five personality traits of people and whether they seem suitable to be recommended to a job interview. Along with presenting our audiovisual approach and results that won the third place in the ChaLearn First Impressions Challenge, we investigate modeling in different modalities including audio only, visual only, language only, audiovisual, and combination of audiovisual and language. Our results demonstrate that the best performance could be obtained using a fusion of all data modalities. Finally, in order to promote explainability in machine learning and to provide an example for the upcoming ChaLearn challenges, we present a simple approach for explaining the predictions for job interview recommendations.
INDEX TERMS
Interviews, Electronic mail, Feature extraction, Predictive models, Machine learning, Face, YouTube
CITATION

Y. Gucluturk et al., "Multimodal First Impression Analysis with Deep Residual Networks," in IEEE Transactions on Affective Computing, vol. 9, no. 3, pp. 316-329, 2018.
doi:10.1109/TAFFC.2017.2751469
92 ms
(Ver 3.3 (11022016))