Intersemiotic Translation as Resemiotisation: A Multimodal Perspective

Abstract

Intersemiotic translation is viewed as the basis of cultural communication through which ideas are circulated, translated and explained using language, images and other semiotic resources. From this perspective, intersemiotic translation is conceptualised through the principle of resemiotisation which is concerned with how semiotic choices are translated over time as social practices. Key issues which arise from viewing intersemiotic translation as resemiotisation are investigated in this paper: (a) how can shifts of meaning be conceptualised across semiotic resources which are fundamentally different in nature?; (b) what meanings are retained and changed as a result of resemiotisation?; and (c) how can such meanings be modelled theoretically and tracked analytically? In order to explore these issues, a multimodal social semiotic approach based on Michael Halliday’s systemic functional theory is presented. In this approach, semiotic resources are conceptualised as systems of meaning which differentially fulfil various functions in society: to structure experience and make logical connections in the world, to enact social relations and create a stance towards the world, and to organise these meanings into multimodal messages. The multimodal approach to intersemiotic translation is illustrated through examples which explore shifts of meaning which take place across theses strands of meaning as semiotic choices are resemiotised within and across multimodal texts. The focus of this discussion is intersemiotic translation across language (spoken and written), images (photographs, film tracks and graphs) and mathematical symbolism in a range of different text types (i.e. a news report, an infographic, a video and a mathematical graph). These examples illustrate the principles of intersemiotic translation through which semantic expansions occur as semiotic choices are resemiotised using different semiotic resources, which result in access to a new meaning potential.