Towards Multimodal Campaign Detection: Including Image Information in Stream Clustering to Detect Social Media Campaigns

Stampe, Lucas; Pohl, Janina; Grimme, Christian

Abstract

This work explores the potential to include visual informa- tion from images in social media campaign recognition. The diverse con- tent shared on social media platforms, including text, photos, videos, and links, necessitates a multimodal analysis approach. With the emer- gence of Large Language Models (LLMs), there is now an opportunity to convert image content into textual descriptions, enabling the incorpo- ration of previously text-based methods into a multimodal analysis. We evaluate this approach by conducting a parameter study to assess the resulting differences in image captions and a case study to examine the contribution of textualized image information to campaign recognition. The results indicate that, using image captions separate from or along- side tweet texts, connections between campaigns can be identified, and new campaigns detected.