The Role of Images in Social Media Analytics: A Multimodal Digital Humanities Approach

Abstract

The analysis of social media data is an emerging research field that aims to study the dynamics of urban life. In this study, we adopt a multimodal digital humanities approach to combine the analysis of text-based social media data with visual social media data in an interactive map to investigate urban life in Singapore from a social semiotic perspective. Twitter is used as a source of user-defined localised textual data, Instagram as a source of localised user-generated images and Foursquare as a source of user-defined location-based information where is semantically organised according to Wikipedia's classification tree. In this way, we track the multimodal content of social media according to semantically organised location-based sources. This study suggests that users of Twitter express emotion about their own lives and the world around them, but these linguistic resources are differentially deployed according to venue. However, this is less variation in the use of photos to construe personal relationships, suggesting that photos fulfil and intrinsic need to be observed which transcends the nature of the social practice which is taking place. It is envisaged that the role of the visual will continue to expand as digital technologies refashion and transform out semiotic world.