Automated image analysis for studying online behaviour

Schwemmer, Carsten; Unger, Saïd; Heiberger, Raphael

Abstract

Digitization led to an enormous increase in the availability of visual data. As images are an important aspect of human communication, decades of social science research have analysed images, yet in mostly manual fashion with limited scaling capacities. In this work, we outline how recent advances in computer vision enable automated image analysis, allowing researchers to further unlock the potential of digital behavioural data. We introduce the field of computational social science and conduct a literature review of early studies using image recognition. We also highlight important aspects to be considered, such as computational demands and biases of computer vision models. Furthermore, in a case study, we examine the online behaviour of US Members of Congress during the early COVID-19 pandemic in 2020. In particular, we focus on sharing images showing face masks as they are a crucial aspect of health and safety measures during the pandemic. Using Instagram data and models for detecting face masks, we find that temporal dynamics and party affiliation play a substantial role in the likelihood of sharing images of people wearing face masks: images with masks are more often posted after the introduction of mask mandates and Democratic party members are more likely to share images with masks. In addition, we find somewhat weaker to no differences regarding the age and gender of politicians.