Multimodal analytics: Software and visualization techniques for analyzing and interpreting multimodal data

Abstract

Multimodal analysis, the study of semantic patterns arising from the integration of language, images and audio resouces in multimodal texts, presents considerable challenges which are both theoretical and practical in nature. First, frameworks for modeling the semantic contributions of the different resources and the meaning arising from the interaction of choices in multimodal phenomena are required. Such frameworks also need to account for resemioticization of mulitmodal phenomena to investigate social practices over space and time (Iedema, 2001, 2003). In this repect, the social semiotic approach (Halliday, 1978; Halliday and Matthiese4n, 2004) developed by Kress and Van Leeuwen (2005 [1996] and O'Toole (2011 [1994]) has provided a powerful theoretical platform for investigating multimodal phenomena (eg. see this volume)