Affective Image Retrieval via Multi-Graph Learning

ABSTRACT

Images can convey rich emotions to viewers. Recent research on image emotion analysis mainly focused on affective image classification, trying to find features that can classify emotions better. We concentrate on affective image retrieval and investigate the performance of different features on different kinds of images in a multi-graph learning framework. Firstly, we extract commonly used features of different levels for each image. Generic features and features derived from elements-of-art are extracted as low-level features. Attributes and interpretable principles-of-art based features are viewed as mid-level features, while semantic concepts described by adjective noun pairs and facial expressions are extracted as high-level features. Secondly, we construct single graph for each kind of feature to test the retrieval performance. Finally, we combine the multiple graphs together in a regularization framework to learn the optimized weights of each graph to efficiently explore the complementation of different features. Extensive experiments are conducted on five datasets and the results demonstrate the effectiveness of the proposed method.

References

1. D. Borth et al. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In MM, 2013.
2. E. S. Dan-Glauser and K. R. Scherer. The geneva affective picture database (gaped): a new 730-picture database focusing on valence and normative significance. Behavior research methods, 43(2):468--477, 2011.
3. Y. Gao et al. 3-d object retrieval and recognition with hypergraph analysis. IEEE TIP, 21(9):4290--4303, 2012.
4. A. Hanjalic. Extracting moods from pictures and sounds: Towards truly personalized tv. IEEE Signal Processing Magazine, 23(2):90--100, 2006.
5. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM TOIS, 20(4):422--446, 2002.
6. J. Jia et al. Can we understand van gogh's mood? learning to infer affects from images in social networks. In MM, 2012.
7. P. Lang et al. International affective picture system (IAPS): Affective ratings of pictures and instruction manual. NIMH, Center for the Study of Emotion & Attention, 2005.
8. X. Lu et al. On shape and the computability of emotions. In MM, 2012.
9. P. Lucey et al. The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In CVPR Workshops, 2010.
10. J. Machajdik and A. Hanbury. Affective image classification using features inspired by psychology and art theory. In MM, 2010.
11. J. Mikels et al. Emotional category data on images from the international affective picture system. Behavior research methods, 37(4):626--630, 2005.
12. B. Pang and L. Lee. Opinion mining and sentiment analysis. Information Retrieval, 2(1--2):1--135, 2008.
13. G. Patterson and J. Hays. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In CVPR, 2012.
14. P. Viola et al. Robust real-time face detection. IJCV, 57(2):137--154, 2004.
15. M. Wang et al. Unified video annotation via multigraph learning. IEEE TCSVT, 19(5):733--746, 2009.
16. W. Wang et al. A survey on emotional semantic image retrieval. In ICIP, 2008.
17. J. Xiao et al. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
18. P. Yang et al. Exploring facial expressions with compositional features. In CVPR, 2010.
19. J. Yuan et al. Sentribute: image sentiment analysis from a mid-level perspective. In WISDOM, 2013.
20. S. Zhao et al. Video indexing and recommendation based on affective analysis of viewers. In ACM MM, 2011.
21. S. Zhao et al. Exploring principles-of-art features for image emotion recognition. In ACM MM, 2014.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

  • Computer graphics

    • Image manipulation

• Information systems

  • Information retrieval