skip to main content
10.1145/3126686.3126776acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Multimodal Classification of Violent Online Political Extremism Content with Graph Convolutional Networks

Published: 23 October 2017 Publication History

Abstract

In this paper we present a multimodal approach to categorizing user posts based on their discussion topic. To integrate heterogeneous information extracted from the posts, i.e. text, visual content and the information about user interactions with the online platform, we deploy graph convolutional networks that were recently proven effective in classification tasks on knowledge graphs. As the case study we use the analysis of violent online political extremism content, a challenging task due to a particularly high semantic level at which extremist ideas are discussed. Here we demonstrate the potential of using neural networks on graphs for classifying multimedia content and, perhaps more importantly, the effectiveness of multimedia analysis techniques in aiding the domain experts performing qualitative data analysis. Our conclusions are supported by extensive experiments on a large collection of extremist posts.

References

[1]
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, and others. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016).
[2]
Kobus Barnard, Pinar Duygulu, David Forsyth, Nando de Freitas, David M. Blei, and Michael I. Jordan. 2003. Matching Words and Pictures. J. Mach. Learn. Res. 3 (March 2003), 1107--1135.
[3]
Don Black. 1996-2017. Stormfront - a white nationalist, white supremacist and neo-Nazi Internet forum. https://www.stormfront.org/. (1996-2017). Online. Accessed on March, 2017.
[4]
David M. Blei. 2012. Probabilistic Topic Models. Commun. ACM 55, 4 (April 2012), 77--84.
[5]
Marc Bron, Jasmijn van Gorp, Frank Nack, Lotte Belice Baltussen, and Maarten de Rijke. 2013. Aggregated Search Interface Preferences in Multi-session Search Tasks. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '13). ACM, New York, NY, USA, 123--132.
[6]
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2013. Spectral Networks and Locally Connected Networks on Graphs. CoRR abs/1312.6203 (2013). http://arxiv.org/abs/1312.6203
[7]
Jiajun Bu, Shulong Tan, Chun Chen, Can Wang, Hao Wu, Lijun Zhang, and Xiaofei He. 2010. Music Recommendation by Unified Hypergraph: Combining Social Media Information and Music Content. In Proceedings of the 18th ACM International Conference on Multimedia (MM '10). ACM, New York, NY, USA, 391--400.
[8]
Maarten Clements, Arjen P. De Vries, and Marcel J. T Reinders. 2010. The task-dependent effect of tags and ratings on social media access. ACM Transactions on Information Systems (TOIS) 28, 4 (2010), 21.
[9]
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). Curran Associates, Inc., 3844--3852.
[10]
J. Deng, W. Dong, R. Socher, L. J. Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. 248--255.
[11]
David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gómez-Bombarelli, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. 2015. Convolutional Networks on Graphs for Learning Molecular Fingerprints. In Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS'15). MIT Press, Cambridge, MA, USA, 2224--2232.
[12]
Pierre Geurts, Damien Ernst, and Louis Wehenkel. 2006. Extremely randomized trees. Machine Learning 63, 1 (2006), 3--42.
[13]
Amirhossein Habibian, Thomas Mensink, and Cees G. M. Snoek. 2014. VideoStory: A New Multimedia Embedding for Few-Example Recognition and Translation of Events. In Proceedings of the 22Nd ACM International Conference on Multimedia (MM '14). ACM, New York, NY, USA, 17--26.
[14]
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014). http://arxiv.org/abs/1412.6980 Published as a conference paper at ICLR 2015.
[15]
Thomas N. Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016). Published as a conference paper at ICLR 2017.
[16]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 1097--1105.
[17]
Hugo Larochelle and Stanislas Lauly. 2012. A Neural Autoregressive Topic Model. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 2708--2716.
[18]
Dong Liu, Shuicheng Yan, Yong Rui, and Hong-Jiang Zhang. 2010. Unified Tag Analysis with Multi-edge Graph. In Proceedings of the 18th ACM International Conference on Multimedia (MM '10). ACM, New York, NY, USA, 25--34.
[19]
Edgar Meij, Wouter Weerkamp, and Maarten de Rijke. 2012. Adding Semantics to Microblog Posts. In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining (WSDM '12). ACM, New York, NY, USA, 563--572.
[20]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems 26, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 3111--3119.
[21]
David Milne and Ian H. Witten. 2008. Learning to Link with Wikipedia. In Proceedings of the 17th ACM Conference on Information and Knowledge Management (CIKM '08). ACM, New York, NY, USA, 509--518.
[22]
Facebook Newsroom. 2016. Partnering to Help Curb Spread of Online Terrorist Content. http://newsroom.fb.com/news/2016/12/partnering-to-help-curb-spread-of-online-terrorist-content/. (2016). Online. Accessed on March, 2017.
[23]
Daan Odijk. 2012. UvA Semanticizer Web API. https://github.com/semanticize/semanticizer. (2012).
[24]
Daan Odijk, Edgar Meij, and Maarten de Rijke. 2013. Feeding the Second Screen: Semantic Linking Based on Subtitles. In Proceedings of the 10th Conference on Open Research Areas in Information Retrieval (OAIR '13). LE CENTRE DE HAUTES ETUDES INTERNATIONALES D'INFORMATIQUE DOCUMENTAIRE, Paris, France, France, 9--16.
[25]
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The PageRank citation ranking: Bringing order to the web. Technical Report. Stanford InfoLab.
[26]
Jia-Yu Pan, Hyung-Jeong Yang, Christos Faloutsos, and Pinar Duygulu. 2004. Automatic multimedia cross-modal correlation discovery. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 653--658.
[27]
Jia-Yu Pan, Hyung-Jeong Yang, C. Faloutsos, and P. Duygulu. 2004. GCap: Graphbased Automatic Image Captioning. In 2004 Conference on Computer Vision and Pattern Recognition Workshop. 146--146.
[28]
D. Putthividhy, H. T. Attias, and S. S. Nagarajan. 2010. Topic regression multimodal Latent Dirichlet Allocation for image annotation. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 3408--3415.
[29]
Shengsheng Qian, Tianzhu Zhang, and Changsheng Xu. 2016. Multi-modal Multi-view Topic-opinion Mining for Social Event Analysis. In Proceedings of the 2016 ACM on Multimedia Conference (MM '16). ACM, New York, NY, USA, 2--11.
[30]
Stevan Rudinac, Alan Hanjalic, and Martha Larson. 2013. Generating visual summaries of geographic areas using community-contributed images. IEEE Transactions on Multimedia 15, 4 (2013), 921--932.
[31]
Stevan Rudinac, Martha Larson, and Alan Hanjalic. 2012. Leveraging visual concepts and query performance prediction for semantic-theme-based video retrieval. International Journal of Multimedia Information Retrieval 1, 4 (2012), 263--280.
[32]
Manos Schinas, Symeon Papadopoulos, Georgios Petkos, Yiannis Kompatsiaris, and Pericles A. Mitkas. 2015. Multimodal Graph-based Event Detection and Summarization in Social Media Streams. In Proceedings of the 23rd ACM International Conference on Multimedia (MM '15). ACM, New York, NY, USA, 189--192.
[33]
Karen Simonyan and Andrew Zisserman. 2014. Two-Stream Convolutional Networks for Action Recognition in Videos. In Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 568--576.
[34]
C. G. M. Snoek, K. E. A. van de Sande, D. Fontijne, A. Habibian, M. Jain, S. Kordumova, Z. Li, M. Mazloom, S. L. Pintea, R. Tao, D. C. Koelma, and A. W. M. Smeulders. 2013. MediaMill at TRECVID 2013: Searching Concepts, Objects, Instances and Events in Video. In TRECVID Workshop.
[35]
S. Sunderrajan and B. S. Manjunath. 2016. Context-Aware Hypergraph Modeling for Re-identification and Summarization. IEEE Transactions on Multimedia 18, 1 (Jan 2016), 51--63.
[36]
Jiajun Wang, Yu-Gang Jiang, Qiang Wang, Kuiyuan Yang, and Chong-Wah Ngo. 2014. Organizing Video Search Results to Adapted Semantic Hierarchies for Topic-based Browsing. In Proceedings of the 22Nd ACM International Conference on Multimedia (MM '14). ACM, New York, NY, USA, 845--848.
[37]
T. Yao, C. W. Ngo, and T. Mei. 2013. Circular Reranking for Visual Search. IEEE Transactions on Image Processing 22, 4 (April 2013), 1644--1655.
[38]
Soh Yoshida, Takahiro Ogawa, and Miki Haseyama. 2015. Heterogeneous Graphbased Video Search Reranking Using Web Knowledge via Social Media Network. In Proceedings of the 23rd ACM International Conference on Multimedia (MM '15). ACM, New York, NY, USA, 871--874.
[39]
Y. Zheng, Y. J. Zhang, and H. Larochelle. 2016. A Deep and Autoregressive Approach for Topic Modeling of Multimodal Data. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 6 (June 2016), 1056--1069.
[40]
Lei Zhu, Jialie Shen, and Liang Xie. 2015. Topic Hypergraph Hashing for Mobile Image Retrieval. In Proceedings of the 23rd ACM International Conference on Multimedia (MM '15). ACM, New York, NY, USA, 843--846.

Cited By

View all
  • (2024)High-performance computing in healthcare: An automatic literature analysis perspectiveJournal of Big Data10.1186/s40537-024-00929-211:1Online publication date: 2-May-2024
  • (2023)Down the Rabbit Hole: Detecting Online Extremism, Radicalisation, and Politicised Hate SpeechACM Computing Surveys10.1145/358306755:14s(1-35)Online publication date: 7-Feb-2023
  • (2023)Affect-GCN: a multimodal graph convolutional network for multi-emotion with intensity recognition and sentiment analysis in dialoguesMultimedia Tools and Applications10.1007/s11042-023-14885-182:28(43251-43272)Online publication date: 27-Apr-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
Thematic Workshops '17: Proceedings of the on Thematic Workshops of ACM Multimedia 2017
October 2017
558 pages
ISBN:9781450354165
DOI:10.1145/3126686
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 October 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. entity linking
  2. graph convolutional networks
  3. multimedia classification
  4. semantic concepts
  5. violent online political extremism

Qualifiers

  • Research-article

Funding Sources

  • European Union's Seventh Framework Programme

Conference

MM '17
Sponsor:
MM '17: ACM Multimedia Conference
October 23 - 27, 2017
California, Mountain View, USA

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)27
  • Downloads (Last 6 weeks)2
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)High-performance computing in healthcare: An automatic literature analysis perspectiveJournal of Big Data10.1186/s40537-024-00929-211:1Online publication date: 2-May-2024
  • (2023)Down the Rabbit Hole: Detecting Online Extremism, Radicalisation, and Politicised Hate SpeechACM Computing Surveys10.1145/358306755:14s(1-35)Online publication date: 7-Feb-2023
  • (2023)Affect-GCN: a multimodal graph convolutional network for multi-emotion with intensity recognition and sentiment analysis in dialoguesMultimedia Tools and Applications10.1007/s11042-023-14885-182:28(43251-43272)Online publication date: 27-Apr-2023
  • (2022)Improving Abusive Language Detection with online interaction networkInformation Processing & Management10.1016/j.ipm.2022.10300959:5(103009)Online publication date: Sep-2022
  • (2020)Countering Extremists on Social Media: Challenges for Strategic Communication and Content ModerationPolicy & Internet10.1002/poi3.23612:1(6-19)Online publication date: 16-Mar-2020
  • (2019)HyperLearnProceedings of the 27th ACM International Conference on Multimedia10.1145/3343031.3350572(2245-2253)Online publication date: 15-Oct-2019
  • (2019)Predicting Behavioural Patterns in Discussion Forums using Deep Learning on Hypergraphs2019 International Conference on Content-Based Multimedia Indexing (CBMI)10.1109/CBMI.2019.8877384(1-6)Online publication date: Sep-2019
  • (2019)Interactive Search and Exploration in Discussion Forums Using Multimodal EmbeddingsMultiMedia Modeling10.1007/978-3-030-37734-2_32(388-399)Online publication date: 24-Dec-2019
  • (2018)Exploiting Relational Information in Social Networks using Geometric Deep Learning on HypergraphsProceedings of the 2018 ACM on International Conference on Multimedia Retrieval10.1145/3206025.3206062(117-125)Online publication date: 5-Jun-2018
  • (2018)Automatic Classification and Linguistic Analysis of Extremist Online MaterialMultiMedia Modeling10.1007/978-3-030-05716-9_49(577-582)Online publication date: 11-Dec-2018

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media