ABSTRACT
With the growing ubiquity of the Internet and access to media-based social media platforms, the risks associated with media content sharing on social media and the need for safety measures against such risks have grown paramount. At the same time, risk is highly contextualized, especially when it comes to media content youth share privately on social media. In this work, we conducted qualitative content analyses on risky media content flagged by youth participants and research assistants of similar ages to explore contextual dimensions of youth online risks. The contextual risk dimensions were then used to inform semi- and self-supervised state-of-the-art vision transformers to automate the process of identifying risky images shared by youth. We found that vision transformers are capable of learning complex image features for use in automated risk detection and classification. The results of our study serve as a foundation for designing contextualized and youth-centered machine-learning methods for automated online risk detection.
- Shiza Ali, Afsaneh Razi, Kim Seunghyun, Ashwaq Alsoubai, Joshua Gracie, Munmun De Choudhury, Pamela J Wisniewski, and Gianluca Stringhini. 2022. Understanding the Digital Lives of Youth: Analyzing Media Shared within Safe Versus Unsafe Private Conversations on Instagram. (2022).Google Scholar
- Monica Anderson and Jingjing Jiang. 2018. Teens, Social Media and Technology 2018. https://www.pewresearch.org/internet/2018/05/31/teens-social-media-technology-2018/Google Scholar
- Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. 2021. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 6836–6846.Google ScholarCross Ref
- Karla Badillo-Urquiola, Zachary Shea, Zainab Agha, Irina Lediaeva, and Pamela Wisniewski. 2021. Conducting risky research with teens: co-designing for the ethical treatment and protection of adolescents. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3 (2021), 1–46.Google ScholarDigital Library
- Fanni Bányai, Ágnes Zsila, Orsolya Király, Aniko Maraz, Zsuzsanna Elekes, Mark D Griffiths, Cecilie Schou Andreassen, and Zsolt Demetrovics. 2017. Problematic social media use: Results from a large-scale nationally representative adolescent sample. PloS one 12, 1 (2017), e0169839.Google ScholarCross Ref
- Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021. Is space-time attention all you need for video understanding?. In ICML, Vol. 2. 4.Google Scholar
- Rubel Biswas, Victor González-Castro, E Fidalgo, and Deisy Chaves. 2019. Boosting child abuse victim identification in Forensic Tools with hashing techniques. V Jornadas Nacionales de Investigación en Ciberseguridad 1 (2019), 344–345.Google Scholar
- Virginia Braun and Victoria Clarke. 2012. Thematic analysis.American Psychological Association.Google Scholar
- Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In European conference on computer vision. Springer, 213–229.Google ScholarDigital Library
- Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9650–9660.Google ScholarCross Ref
- Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 1597–1607.Google Scholar
- Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 (2020).Google Scholar
- Arijit Ghosh Chowdhury, Ramit Sawhney, Puneet Mathur, Debanjan Mahata, and Rajiv Ratn Shah. 2019. Speak up, fight back! detection of social media disclosures of sexual harassment. In Proceedings of the 2019 conference of the North American chapter of the Association for Computational Linguistics: Student research workshop. 136–146.Google ScholarCross Ref
- Jody Clay-Warner. 2003. The context of sexual violence: Situational predictors of self-protective actions. Violence and victims 18, 5 (2003), 543–556.Google Scholar
- Patricia De Santisteban and Manuel Gámez-Guadix. 2018. Prevalence and risk factors among minors for online sexual solicitations and interactions with adults. The Journal of Sex Research 55, 7 (2018), 939–950.Google ScholarCross Ref
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).Google Scholar
- James W Drisko and Tina Maschi. 2016. Content analysis. Pocket Guide to Social Work Re.Google Scholar
- Sindhu Kiranmai Ernala, Michael L Birnbaum, Kristin A Candan, Asra F Rizvi, William A Sterling, John M Kane, and Munmun De Choudhury. 2019. Methodological gaps in predicting mental health states from social media: triangulating diagnostic signals. In Proceedings of the 2019 chi conference on human factors in computing systems. 1–16.Google ScholarDigital Library
- facebookresearch/dino. 2022. facebookresearch/dino. https://github.com/facebookresearch/dinoGoogle Scholar
- David Finkelhor, Kerryann Walsh, Lisa Jones, Kimberly Mitchell, and Anne Collier. 2021. Youth internet safety education: Aligning programs with the evidence base. Trauma, violence, & abuse 22, 5 (2021), 1233–1247.Google Scholar
- Abhishek Gangwar, Eduardo Fidalgo, Enrique Alegre, and Víctor González-Castro. 2017. Pornography and child sexual abuse detection in image and video: A comparative evaluation. (2017).Google Scholar
- Abhishek Gangwar, Víctor González-Castro, Enrique Alegre, and Eduardo Fidalgo. 2021. AttM-CNN: Attention and metric learning based CNN for pornography, age and Child Sexual Abuse (CSA) Detection in images. Neurocomputing 445 (2021), 81–104.Google ScholarCross Ref
- Christine Greenhow. 2011. Youth, learning, and social media. Journal of Educational Computing Research 45, 2 (2011), 139–146.Google ScholarCross Ref
- Joel W Grube and Elizabeth Waiters. 2005. Alcohol in the media: content and effects on drinking beliefs and behaviors among youth. Adolescent medicine clinics 16, 2 (2005), 327.Google Scholar
- Naeemul Hassan, Amrit Poudel, Jason Hale, Claire Hubacek, Khandaker Tasnim Huq, Shubhra Kanti Karmaker Santu, and Syed Ishtiaque Ahmed. 2020. Towards automated sexual violence report tracking. In Proceedings of the international AAAI conference on web and social media, Vol. 14. 250–259.Google ScholarCross Ref
- Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9729–9738.Google ScholarCross Ref
- Homa Hosseinmardi, Sabrina Arredondo Mattson, Rahat Ibn Rafiq, Richard Han, Qin Lv, and Shivakant Mishra. 2015. Analyzing labeled cyberbullying incidents on the instagram social network. In International conference on social informatics. Springer, 49–66.Google ScholarCross Ref
- ImageNet. 2021. ImageNet. https://www.image-net.org/Google Scholar
- Instagram. 2022. Abuse and Spam. https://help.instagram.com/165828726894770Google Scholar
- Instagram. 2022. How do I send a message to someone on Instagram?https://help.instagram.com/155540431448273Google Scholar
- Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. 2021. Perceiver: General perception with iterative attention. In International conference on machine learning. PMLR, 4651–4664.Google Scholar
- Lisa M Jones, Kimberly J Mitchell, and David Finkelhor. 2012. Trends in youth internet victimization: Findings from three youth internet safety surveys 2000–2010. Journal of adolescent Health 50, 2 (2012), 179–186.Google ScholarCross Ref
- Aparup Khatua, Erik Cambria, and Apalak Khatua. 2018. Sounds of silence breakers: Exploring sexual violence on twitter. In 2018 IEEE/ACM international conference on advances in social networks analysis and mining (ASONAM). IEEE, 397–400.Google ScholarCross Ref
- Seunghyun Kim, Afsaneh Razi, Gianluca Stringhini, Pamela J Wisniewski, and Munmun De Choudhury. 2021. A Human-Centered Systematic Literature Review of Cyberbullying Detection Algorithms. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–34.Google ScholarDigital Library
- Robin M Kowalski, Susan P Limber, and Annie McCord. 2019. A developmental approach to cyberbullying: Prevalence and protective factors. Aggression and Violent Behavior 45 (2019), 20–32.Google ScholarCross Ref
- Dong-Hyun Lee 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, Vol. 3. 896.Google Scholar
- Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 10012–10022.Google ScholarCross Ref
- Megan K Maas, Kyla M Cary, Elizabeth M Clancy, Bianca Klettke, Heather L McCauley, and Jeff R Temple. 2021. Slutpage use among US college students: the secret and social platforms of image-based sexual abuse. Archives of sexual behavior 50, 5 (2021), 2203–2214.Google Scholar
- Joao Macedo, Filipe Costa, and Jefersson A dos Santos. 2018. A benchmark methodology for child pornography detection. In 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 455–462.Google ScholarCross Ref
- Sheri Madigan, Vanessa Villani, Corry Azzopardi, Danae Laut, Tanya Smith, Jeff R Temple, Dillon Browne, and Gina Dimitropoulos. 2018. The prevalence of unwanted online sexual exposure and solicitation among youth: A meta-analysis. Journal of Adolescent Health 63, 2 (2018), 133–141.Google ScholarCross Ref
- Rose Maghsoudi, Jennifer Shapka, and Pamela Wisniewski. 2020. Examining how online risk exposure and online social capital influence adolescent psychological stress. Computers in Human Behavior 113 (2020), 106488. https://doi.org/10.1016/j.chb.2020.106488Google ScholarCross Ref
- Felix Mayer and Martin Steinebach. 2017. Forensic image inspection assisted by deep learning. In Proceedings of the 12th International Conference on Availability, Reliability and Security. 1–9.Google ScholarDigital Library
- Ersilia Menesini, Annalaura Nocentini, and Pamela Calussi. 2011. The measurement of cyberbullying: Dimensional structure and relative item severity and discrimination. Cyberpsychology, behavior, and social networking 14, 5 (2011), 267–274.Google Scholar
- Kimberly J Mitchell, David Finkelhor, and Janis Wolak. 2007. Youth Internet users at risk for the most serious online sexual solicitations. American Journal of Preventive Medicine 32, 6 (2007), 532–537.Google ScholarCross Ref
- Subhabrata Mukherjee and Ahmed Awadallah. 2020. Uncertainty-aware self-training for few-shot text classification. Advances in Neural Information Processing Systems 33 (2020), 21199–21212.Google Scholar
- Fudong Nian, Teng Li, Yan Wang, Mingliang Xu, and Jun Wu. 2016. Pornographic image detection utilizing deep convolutional neural networks. Neurocomputing 210 (2016), 283–293.Google ScholarDigital Library
- Gwenn Schurgin O’Keeffe, Kathleen Clarke-Pearson, Council on Communications, and Media. 2011. The impact of social media on children, adolescents, and families. Pediatrics 127, 4 (2011), 800–804.Google ScholarCross Ref
- Claudia Peersman, Christian Schulze, Awais Rashid, Margaret Brennan, and Carl Fischer. 2016. iCOP: Live forensics to reveal previously unknown criminal media on P2P networks. Digital Investigation 18 (2016), 50–64.Google ScholarDigital Library
- Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. 2019. Stand-alone self-attention in vision models. Advances in Neural Information Processing Systems 32 (2019).Google Scholar
- Afsaneh Razi, Seunghyun Kim, Ashwaq Alsoubai, Gianluca Stringhini, Thamar Solorio, Munmun De Choudhury, and Pamela J Wisniewski. 2021. A Human-Centered Systematic Literature Review of the Computational Approaches for Online Sexual Risk Detection. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–38.Google ScholarDigital Library
- Andrew G Reece and Christopher M Danforth. 2017. Instagram photos reveal predictive markers of depression. EPJ Data Science 6 (2017), 1–12.Google Scholar
- Lauren A Reed, Margaret P Boyer, Haley Meskunas, Richard M Tolman, and L Monique Ward. 2020. How do adolescents experience sexting in dating relationships? Motivations to sext and responses to sexting requests from dating partners. Children and Youth Services Review 109 (2020), 104696.Google ScholarCross Ref
- Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. 2021. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. arXiv preprint arXiv:2101.06329 (2021).Google Scholar
- Tara L Rutkowski, Heidi Hartikainen, Kirsten E Richards, and Pamela J Wisniewski. 2021. Family Communication: Examining the Differing Perceptions of Parents and Teens Regarding Online Safety Communication. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–23.Google ScholarDigital Library
- Sarita Schoenebeck, Carol F. Scott, Emma Grace Hurley, Tammy Chang, and Ellen Selkie. 2021. Youth Trust in Social Media Companies and Expectations of Justice: Accountability and Repair After Online Harassment. Proc. ACM Hum.-Comput. Interact. 5, CSCW1, Article 2 (apr 2021), 18 pages. https://doi.org/10.1145/3449076Google ScholarDigital Library
- Weiwei Shi, Yihong Gong, Chris Ding, Zhiheng MaXiaoyu Tao, and Nanning Zheng. 2018. Transductive semi-supervised deep learning using min-max features. In Proceedings of the European Conference on Computer Vision (ECCV). 299–315.Google ScholarDigital Library
- Thiago H Silva, Pedro OS Vaz De Melo, Jussara M Almeida, Juliana Salles, and Antonio AF Loureiro. 2013. A picture of Instagram is worth more than a thousand words: Workload characterization and application. In 2013 IEEE International Conference on Distributed Computing in Sensor Systems. IEEE, 123–132.Google ScholarDigital Library
- Vivek K Singh, Souvick Ghosh, and Christin Jose. 2017. Toward multimodal cyberbullying detection. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 2090–2099.Google ScholarDigital Library
- Devin Soni and Vivek K Singh. 2018. See no evil, hear no evil: Audio-visual-textual cyberbullying detection. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 1–26.Google ScholarDigital Library
- Catherine Cheng Stahl and Ioana Literat. 2022. # GenZ on TikTok: the collective online self-Portrait of the social media generation. Journal of Youth Studies (2022), 1–22.Google Scholar
- Kurt Thomas, Devdatta Akhawe, Michael Bailey, Dan Boneh, Elie Bursztein, Sunny Consolvo, Nicola Dell, Zakir Durumeric, Patrick Gage Kelley, Deepak Kumar, 2021. Sok: Hate, harassment, and the changing landscape of online abuse. In 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 247–267.Google ScholarCross Ref
- Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning. PMLR, 10347–10357.Google Scholar
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).Google Scholar
- Nishant Vishwamitra, Hongxin Hu, Feng Luo, and Long Cheng. 2021. Towards understanding and detecting cyberbullying in real-world images. In 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA).Google Scholar
- Emily A Vogels. 2022. Teens and cyberbullying 2022. https://www.pewresearch.org/internet/2022/12/15/teens-and-cyberbullying-2022/Google Scholar
- Emily A Vogels, Risa Gelles-Watnick, and Navid Massarat. 2022. Teens, social media and technology 2022. https://www.pewresearch.org/internet/2022/08/10/teens-social-media-and-technology-2022/Google Scholar
- Kate Walker and Emma Sleath. 2017. A systematic review of the current knowledge regarding revenge pornography and non-consensual sharing of sexually explicit media. Aggression and violent behavior 36 (2017), 9–24.Google Scholar
- Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. 2021. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 568–578.Google ScholarCross Ref
- Yilin Wang and Baoxin Li. 2015. Sentiment analysis for social media images. In 2015 IEEE international conference on data mining workshop (ICDMW). IEEE, 1584–1591.Google ScholarDigital Library
- Pamela Wisniewski, Heng Xu, Mary Beth Rosson, Daniel F Perkins, and John M Carroll. 2016. Dear diary: Teens reflect on their weekly online risk experiences. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 3919–3930.Google ScholarDigital Library
- Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3733–3742.Google ScholarCross Ref
- Emilios Yiallourou, Rafaella Demetriou, and Andreas Lanitis. 2017. On the detection of images containing child-pornographic material. In 2017 24th International Conference on Telecommunications (ICT). IEEE, 1–5.Google ScholarCross Ref
- Lequan Yu, Shujun Wang, Xiaomeng Li, Chi-Wing Fu, and Pheng-Ann Heng. 2019. Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 605–613.Google ScholarDigital Library
- Michele Zappavigna. 2016. Social media photography: construing subjectivity in Instagram images. Visual Communication 15, 3 (2016), 271–292.Google ScholarCross Ref
- Zhedong Zheng and Yi Yang. 2021. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. International Journal of Computer Vision 129, 4 (2021), 1106–1120.Google ScholarDigital Library
- Haoti Zhong, Hao Li, Anna Cinzia Squicciarini, Sarah Michele Rajtmajer, Christopher Griffin, David J Miller, and Cornelia Caragea. 2016. Content-Driven Detection of Cyberbullying on the Instagram Social Network.. In IJCAI, Vol. 16. 3952–3958.Google Scholar
- Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2020. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 (2020).Google Scholar
Index Terms
Towards Automated Detection of Risky Images Shared by Youth on Social Media
Recommendations
Social media and loneliness
Social media use continues to grow and is especially prevalent among young adults. It is surprising then that, in spite of this enhanced interconnectivity, young adults may be lonelier than other age groups, and that the current generation may be the ...
Social Media Content Analytics beyond the Text: A Case Study of University Branding in Instagram
ACM SE '19: Proceedings of the 2019 ACM Southeast ConferenceSocial media is unarguably one of the wealthiest sources of information. The opinions shared on social platform have an immense influence towards the brands equity. Social media has flourished with platforms like Facebook, Twitter, and Snapchat, etc. ...
Social Media Marketing in Luxury Retail
This study examines the potentials of social media marketing for luxury retailers. Social media marketing tactics of three luxury retail brands Barneys New York, Net-a-Porter.com, and Saks Fifth Avenue were examined across three major social media sites ...
Comments