Shaojing Fan
Mohan S. Kankanhalli
ABSTRACT
Automated assessment of visual sentiment has many applications, such as monitoring social media and facilitating online advertising. In current research on automated visual sentiment assessment, images are mainly input and processed as a whole. However, human attention is biased, and a focal region with high acuity can disproportionately influence visual sentiment. To investigate how attention influences visual sentiment, we conducted experiments that reveal critical insights into human perception. We discover that negative sentiments are elicited by the focal region without a notable influence of contextual information, whereas positive sentiments are influenced by both focal and contextual information. Building on these insights, we create new deep convolutional neural networks for sentiment prediction that have additional channels devoted to encoding focal information. On two benchmark datasets, the proposed models demonstrate superior performance compared with the state-of-the-art methods. Extensive visualizations and statistical analyses indicate that the focal channels are more effective on images with focal objects, especially for images that also elicit negative sentiments.
- Charu C Aggarwal and ChengXiang Zhai Mining text data Springer Science & Business Media, 2012. Google Scholar
Digital Library
- Xavier Alameda-Pineda, Elisa Ricci, Yan Yan, and Nicu Sebe Recognizing emotions from abstract paintings using non-linear matrix completion CVPR, 2016.Google Scholar
- Joel Aronoff How we recognize angry and happy emotion in people, places, and things Cross-cultural research, 2006.Google Scholar
- Rosemary A Bailey Design of comparative experiments. In Cambridge University Press, 2008, Vol. Vol. 25.Google ScholarCross Ref
- Damian Borth, Rongrong Ji, Tao Chen, Thomas Breuel, and Shih-Fu Chang Large-scale visual sentiment ontology and detectors using adjective noun pairs ACM MM, 2013. Google ScholarDigital Library
- Tobias Brosch, Gilles Pourtois, and David Sander The perception and categorisation of emotional stimuli: A review Cognition and Emotion, 2010.Google Scholar
- Neil DB Bruce and John K Tsotsos Saliency, attention, and visual search: An information theoretic approach Journal of vision, 2009.Google Scholar
- Zoya Bylinskii, Tilke Judd, Ali Borji, Laurent Itti, Frédo Durand, Aude Oliva, and Antonio Torralba MIT Saliency Benchmark. MIT, 2017.Google Scholar
- Zoya Bylinskii, Adrià Recasens, Ali Borji, Aude Oliva, Antonio Torralba, and Frédo Durand Where Should Saliency Models Look Next?. In ECCV, 2016. Springer.Google Scholar
- Victor Campos, Brendan Jou, and Xavier Giro-i Nieto From pixels to sentiment: Fine-tuning cnns for visual sentiment prediction Image and Vision Computing, 2017. Google ScholarDigital Library
- Moran Cerf, E Paxon Frady, and Christof Koch Faces and text attract gaze independent of the task: Experimental data and computer model Journal of vision, 2009.Google Scholar
- Tao Chen, Damian Borth, Trevor Darrell, and Shih-Fu Chang Deepsentibank: Visual sentiment concept classification with deep convolutional neural networks. In arXiv preprint arXiv:1410.8586, 2014.Google Scholar
- Tao Chen, Felix X Yu, Jiawei Chen, Yin Cui, Yan-Ying Chen, and Shih-Fu Chang Object-based visual sentiment concept analysis and application ACM MM, 2014. Google ScholarDigital Library
- Wolfgang Einh"auser, Merrielle Spain, and Pietro Perona Objects predict fixations better than early saliency Journal of Vision, 2008.Google Scholar
- Paul Ekman An argument for basic emotions. In Cognition & emotion, 1992.Google Scholar
- Shaojing Fan, Tian-Tsong Ng, Jonathan S Herberg, Bryan L Koenig, Cheston Y-C Tan, and Rangding Wang An automated estimator of image visual realism based on human cognition CVPR, 2014. Google ScholarDigital Library
- Shaojing Fan, Tian-Tsong Ng, Bryan L Koenig, Ming Jiang, and Qi Zhao A paradigm for building generalized models of human image perception through data fusion CVPR, 2016.Google Scholar
- Anastasia Giachanou and Fabio Crestani Like it or not: A survey of twitter sentiment analysis methods ACM Computing Surveys (CSUR), 2016. Google ScholarDigital Library
- Carlos FA Gomes, Charles J Brainerd, and Lilian M Stein Effects of emotional valence and arousal on recollective and nonrecollective recall. Journal of Experimental Psychology: Learning, Memory, and Cognition, 2013.Google Scholar
- Michael Gygli, Helmut Grabner, Hayko Riemenschneider, Fabian Nater, and Luc Van Gool The interestingness of images. In ICCV, 2013. Google ScholarDigital Library
- Xun Huang, Chengyao Shen, Xavier Boix, and Qi Zhao SALICON: Reducing the semantic gap in saliency prediction by adapting deep neural networks ICCV, 2015. Google ScholarDigital Library
- Laurent Itti and Pierre F Baldi Bayesian surprise attracts human attention. In NIPS, 2005. Google ScholarDigital Library
- Brendan Jou, Subhabrata Bhattacharya, and Shih-Fu Chang Predicting viewer perceived emotions in animated GIFs ACM MM, 2014. Google ScholarDigital Library
- Elizabeth A Kensinger Remembering the details: Effects of emotion. In Emotion review, 2009.Google Scholar
- Ita GG Kreft, Ita Kreft, and Jan de Leeuw Introducing multilevel modeling. In Sage Publication,1998.Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton Imagenet classification with deep convolutional neural networks NIPS, 2012. Google ScholarDigital Library
- Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories CVPR, 2006. Google ScholarDigital Library
- Yin Li, Xiaodi Hou, Christof Koch, James M Rehg, and Alan L Yuille The secrets of salient object segmentation. In CVPR, 2014. Google ScholarDigital Library
- Tie Liu, Zejian Yuan, Jian Sun, Jingdong Wang, Nanning Zheng, Xiaoou Tang, and Heung-Yeung Shum Learning to detect a salient object. In IEEE Transactions on Pattern analysis and machine intelligence, 2011. Google ScholarDigital Library
- Jana Machajdik and Allan Hanbury Affective image classification using features inspired by psychology and art theory ACM MM, 2010. Google ScholarDigital Library
- Alexander Mathews, Lexing Xie, and Xuming He SentiCap: generating image descriptions with sentiments arXiv preprint arXiv:1510.01431, 2015.Google Scholar
- Joseph A Mikels, Barbara L Fredrickson, Gregory R Larkin, Casey M Lindberg, Sam J Maglio, and Patricia A Reuter-Lorenz Emotional category data on images from the International Affective Picture System Behavior research methods, 2005.Google Scholar
- Tirin Moore and Marc Zirnsak Neural mechanisms of selective visual attention. Annual Review of Psychology, 2015.Google Scholar
- Ken Nakayama, Julian S Joseph, and R Parasuraman Attention, pattern recognition and popout in visual search The attentive brain, 1998.Google Scholar
- Aude Oliva and Antonio Torralba Building the gist of a scene: The role of global image features in recognition Progress in brain research, 2006.Google Scholar
- Stephen E Palmer. 1999. Vision science: Photons to phenomenology. Vol. Vol. 1. MIT press Cambridge, MA.Google Scholar
- Gabriele Paolacci, Jesse Chandler, and Panagiotis Ipeirotis Running experiments on amazon mechanical turk. In Judgment and Decision Making, 2010.Google Scholar
- Jane E Raymond, Mark J Fenske, and Nader T Tavassoli Selective attention determines emotional responses to novel visual stimuli Psychological science, 2013.Google Scholar
- Ulrike Rimmele, Lila Davachi, Radoslav Petrov, Sonya Dougal, and Elizabeth A Phelps Emotion enhances the subjective feeling of remembering, despite lower accuracy for contextual details.. In Emotion, 2011.Google Scholar
- Michael Rubinstein, Diego Gutierrez, Olga Sorkine, and Ariel Shamir A comparative study of image retargeting. In ACM transactions on graphics, 2010. Google ScholarDigital Library
- Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, and others Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015. Google ScholarDigital Library
- Bryan C Russell, Antonio Torralba, Kevin P Murphy, and William T Freeman LabelMe: a database and web-based tool for image annotation International journal of computer vision, 2008. Google ScholarDigital Library
- Harald T Schupp, Jessica Stockburger, Maurizio Codispoti, Markus Junghöfer, Almut I Weike, and Alfons O Hamm Selective visual attention to emotion. In Journal of neuroscience, 2007.Google Scholar
- Karen Simonyan and Andrew Zisserman Very deep convolutional networks for large-scale image recognition arXiv preprint arXiv:1409.1556, 2014.Google Scholar
- Nathan Sprague and Dana Ballard Eye movements for reward maximization. In NIPS, 2003. Google ScholarDigital Library
- Patrik Vuilleumier How brains beware: neural mechanisms of emotional attention Trends in cognitive sciences, 2005.Google Scholar
- Patrik Vuilleumier, Jorge L Armony, Jon Driver, and Raymond J Dolan Effects of attention and emotion on face processing in the human brain: an event-related fMRI study Neuron, 2001.Google Scholar
- Adrian Wells and Gerald Matthews. 2014. Attention and Emotion (Classic Edition): A Clinical Perspective. Psychology Press.Google Scholar
- Juan Xu, Ming Jiang, Shuo Wang, Mohan S Kankanhalli, and Qi Zhao Predicting human gaze beyond pixels. In Journal of vision, 2014.Google Scholar
- Quanzeng You, Liangliang Cao, Hailin Jin, and Jiebo Luo Robust Visual-Textual Sentiment Analysis: When Attention meets Tree-structured Recursive Neural Networks. In ACM MM, 2016. Google ScholarDigital Library
- Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang Robust image sentiment analysis using progressively trained and domain transferred deep networks AAAI, 2015. Google ScholarDigital Library
Index Terms
- The Role of Visual Attention in Sentiment Prediction
Recommendations
Large-scale visual sentiment ontology and detectors using adjective noun pairs
MM '13: Proceedings of the 21st ACM international conference on MultimediaWe address the challenge of sentiment analysis from visual content. In contrast to existing methods which infer sentiment or emotion directly from visual low-level features, we propose a novel approach based on understanding of the visual concepts that ...
Diving Deep into Sentiment: Understanding Fine-tuned CNNs for Visual Sentiment Prediction
ASM '15: Proceedings of the 1st International Workshop on Affect & Sentiment in MultimediaVisual media are powerful means of expressing emotions and sentiments. The constant generation of new content in social networks highlights the need of automated visual sentiment analysis tools. While Convolutional Neural Networks (CNNs) have ...
Salient object based visual sentiment analysis by combining deep features and handcrafted features
AbstractWith the rapid growth of social networks, the visual sentiment analysis has quickly emerged for opinion mining. Recent study reveals that the sentiments conveyed by some images are related to salient objects in them, we propose a scheme for visual ...
Comments