Multimodal, multicodal, and multisensorial semiosis
Interacting sign processes in different codes and sense modalities
Prof. Roland Posner, Technische Universität Berlin (posner@kgw.tu-berlin.de)
Dr. Martin Siefkes, Universität Bremen (martin@siefkes.de)
Dr. Janina Wildfeuer, Universität Bremen (wildfeuer@uni-bremen.de)
This section will focus on interactions between the modalities of semiosis, which have recently become a topic of interest in sociosemiotics (Kress 2010), film studies (Bateman/Schmidt 2011), and the general analysis of multimodal documents (Bateman 2008, Jewitt 2009).
The term “mode” or “modality”, as well as the derived term “multimodal” describing the integration of different modalities, can be used in two senses (cf. Fricke 2012: 47ff): In some disciplines (e.g. psychology), modality is primarily used in the sense of perceptual modality, and multimodality describes sign processes where various sense modalities such as visual, auditory, and haptic perception are involved. Furthermore, “modality” is also used in the wider sense of semiotic modality. Multimodality then describes sign processes where various codes are involved (e.g. music and speech in a radio feature which are both auditorily perceived; or images and written text in a comic, or pictures and text in a book, which are both visually perceived).
In multimodal contexts, semiotic interactions and influences occur which cannot simply be explained as combination of several unimodal contexts, and have to be understood in their semiotic properties (Hess-Lüttich/Wenz 2006). If we refer to the terms “medium” or “multimedial semiosis”, we need to distinguish carefully between a number of different uses of the term “medium” (Posner 2004: 60-4), which includes sense modalities and codes, but also technical and social media.
Interactions between modalities (in both senses) will be investigated in the proposed section. Semiotics and aesthetics have usually investigated semiotic codes as well as sense modalities separately, which is no longer sufficient because of the rapidly rising frequency of multimodal artworks and artefacts. Specifically, different types of interaction of codes and/or sense modalities are of interest, e.g. code use that is in opposition or in congruence, that supports or weakens the message conveyed in the other code, that is syntactically integrated across different codes, as well as gestalt effects (complex semiotic processes which cannot be analyzed as a combination of the contributing codes and/or sense modalities).
In this section, we welcome contributions which investigate semiotic processes combining more than one code, sense modality, or technical medium, and which reflect the specific conditions and semiotic properties of such sign processes. Topics of interest include, but are not limited to, the combination and interaction of two or more codes or sense modalities, e.g.
– language, gesture, facial expression, and/or body posture in daily contexts, film, or video;
– pictures, language (spoken or written, e.g. as subtitles), and music in film or video;
– written language, pictures, and other codes (e.g. hyperlinks or music) in print or online media;
– kinesic, musical, and clothing codes in theater, opera, and ballet;
– aesthetic codes and other codes in aesthetic perception and communication;
– written language and graphemic codes in written texts; etc.
We welcome contributions from different theoretical and methodical backgrounds, but expect the exposition and reflection of the theories used and the methods applied, which might include theoretical investigations building on the semiotic classics or more recent professional literature, specific case studies, corpus studies, multimodal discourse analysis, as well as experimental approaches.
Bibliography:
Bateman, John (2008), Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents. New York: Palgrave Macmillan.
Bateman, John & Karl-Heinrich Schmidt (2011), Multimodal Film Analysis. How Films Mean. London: Routledge.
Fricke, Ellen (2012), Grammatik multimodal. Wie Wörter und Gesten zusammenwirken. Berlin u.a.: de Gruyter.
Hess-Lüttich, Ernest W.B. & Karin Wenz (ed.) (2006), Stile des Intermedialen. Zur Semiotik des Übergangs. (= Kodikas/Code. Ars Semeiotica 29, 1-3). Tübingen: Narr.
Jewitt, Carey (ed.) (2009), The Routledge Handbook of Multimodal Analysis. London: Routledge.
Kress, Gunther (2010), Multimodality: A Social Semiotic Approach to Contemporary Communication. New York: Routledge.
Posner, Roland (2004), „Basic Tasks of Cultural Semiotics“. In: Gloria Withalm and Josef Wallmannsberger (eds.), Signs of Power – Power of Signs. Essays in Honor of Jeff Bernard. Wien: INST: 56-89.
Deadline for abstract submission: June 30, 2014
Abstracts:
1) Silja Nikula, silja.nikula@ulapland.fi, University of Lapland, Faculty of Art and Design, +358 40 4844395
Graphic Abstraction. From mental to pictorial representation
I am interested in, how mental images are turned into visible form. When expressing ideas, we need some kind of language; here I concern pictures as representations. People in the same culture share the same mental representation of concepts. Often they share conventions in visual presentation styles when communicating with each other.
For discussing the matter, I show my data: graphic images that are small and simplified drawings. They are linked in defined mental concepts (ideas or concrete items). Due to the context, they are abstracted. The process of abstracting means reducing and selecting the characteristic points of the items to be presented. My question is: what kinds of visual means are used in presenting, when the purpose is to get people catch the idea?
In analysing my material, I pay attention to semantic and syntactic levels. The relations between mental and material images are defined through the semiotic signs of Charles Sanders Peirce. In the presentations based on iconicity, it is also interesting to see, which features are emphasized to carry out the resemblance. I also pay attention to the figurative tropes, metonymy and metaphor when conveying meanings. The syntactic level is analysed by using the criteria of depicting perspective and drawing style. When drawing a representative of a certain category, it is important to differ from the other categories. Clear conventions in presenting can be found: for instance presenting fire, home or happiness. In most cases, colours do not make the pictures easier to understand. It is interesting to point out some pictures that were not understood at all, also and some of them needed written words to complete the meaning.
My picture examples are from my own country, Finland, and so connected to our local culture. There might come discussion, to which degree the pictures are understood also in other countries.
2) Tania Letizia Gobbett, Art history and drawing teacher – PAT/B, International Ass. IASV-IASS-IAS, tanialetizia.gobbett@gmail.com
Among organic vision and intertextuality: paging and space between pure and impure
In my research, in Art History, developed on schematic values of the paging, from the primitive world of the Rock Art to the first Renaissance to the Contemporary Art History, space reaches its greatest values of reconfiguration in the view-capture of the objects of representation, their circles and schematic folding as well as in their own eidetic mise en page. We may consider that the study of the scheme in visual languages completes itself in the recognition of the role of abstraction, even granting inedited and spiral form returns, canonical in their anthropological becoming, next to an idea of plan, of counterpoint of plastic values, newly represented as sensible interfaces (Aristotle), where pure and impure emerge as ways to give a position to the ‘figural’, as much as to configure themselves as plastic prosodic modulation (Greimas, Fabbri): what express more a content than a container.
Keywords: I. Scheme or réseaux dynamique, II. The turn of the encadrage/figure, III. <Ratios> as RF vs RD and styles, IV. Among impression and representation: pure and impure – homology of the observation, V. Cézanne, Degas, Van Gogh, VI. Primitive and Universal
3) Ekaterina Velmezova (Lausanne University, Switzerland)
Once again about the “semiotics of interjections”: Pleading the variety of approaches
Within the framework of “traditional” language descriptions, very heterogeneous groups of words are often referred to as “interjections”, without any explanation of this particular designation and classification. On the contrary, a semiotically oriented research allows to distinguish several groups of these “problematic words”. Intentionality should be considered as the first criterion to distinguish interjections as elements of a particular language (Saussurean langue) from “natural” and “instinctive” utterances of human beings; like all other words in languages, interjections are words-symbols. Nevertheless, the degree of their arbitrainess can vary, which sometimes was used as an argument to state that interjections are iconic signs par excellence and to explain in this way the fact that they often contain sounds which are atypical of their corresponding languages. The presence or absence, in the interjectional “meanings”, of the so-called conceptual aspect constitutes another important parameter in the description of interjections, (semiotically) bringing them together with some types of pronouns – signs-indices, etc.
In a number of recent works, interjections have already been studied from a semiotic point of view (a brief overview of these researches will constitute a part of our presentation). However, a certain (implicit) faith of their authors in the “absolute explaining power” of one or another semiotic parameter of description (arbitrariness vs non-arbitrariness; one particular typology of signs, etc.) often prevented them from using various semiotic criteria in their studies of interjections. As we shall show in our paper, if oneparticular semiotic approach can hardly resolve the majority of theoretical problems in the study of interjections, their combination allows not only to explain the main difficulties in the “interjectional researches” conducted during several past centurries, but at the same time to propose some innovative approaches to the study of interjections.
4) Silvia Barbotto F.
Visual syncronic translation
Thinking about translation we immediately remind to a sort of transition from a language to another one. What could happen if that “another one” is visual instead of verbal transmitted by written/draw signs instead of words. Maybe we couldn’t speak about this process under the characterization of perception or interpretation, but even that we can think about a form of expression able to be in between of mining full aspects.
For one hand there is the need to sit, define or describe this practice understanding that its origin and its theorist’s characteristics come from disciplines as semiotic, anthropology, art and visual communication. In the other hand, trying to recognize his use and his potentiality is possible to approach it pragmatically and experimentally.
At the moment is just an intuition and an research, but as we know, especially in humanistic studies, the intuition can be a legitimate methodology based on empathy and presence, capacity to listen and recognize the imputs coming from the context.
Grosso modo we can say that the visual synchronic translation (VST) is a sort of understanding coming from the senses: the organs receptors pay attention to the situation in which are engaged and, after a process of elaboration very short that could be call instant (immediately) transform this experience to a visual/grafemical bidimension, as a form of register end fix of the circumstance.
Crossing from a multidimensional to a bidimensional space, is inevitable to charge with a change of structure even if the essence follow to be a similar one. The new structure is a composition set by a syncretism between a semical and figural density, sometimes reaching an isograph, some other focus on abstract or lessema signs but always we a need to be approach in a mereological attitude.
To be part of the context, as to broken the liminal zone of being informant and researcher is a typical viewpoint (and action point) of constructivism that try to understand the knowledge as something organic, build entrenched by the participants..but this is another point call Vitácora comunitaria, a second step.
In terms of VST is important to define a style, as a “specific regularity that organize different manifestations of a specific field” (Volli): the specificity could be think as a track/imprinting, a fil rouge that cross all the text, and maybe all the texts (that would mean the author style). So to learn VST is to learn to pick up (grasp) a specific manifestation and transmit it in a specific arquitecture, or narrative, build by signs. The objective for the semiotic is integral: fix and understand this new way of writing and recognizing the relation with primarial and fisical’s needs, fix it in a new functional/artistic/academic language.
5) Dr. Silke Betscher, University of Bremen, betscher@uni-bremen.de
Multimodality and Modal Resources in Visual Discourse Analysis of Cold War Imagery
For a long time discourse analysis has focused on language. Many disciplines developed their own method to analyze spoken and/or written language in a discursive way. The more surprising it is that other elements or layers of discourse (performativity, sound, visuality) remained as blind spots for such a long time. It is only for a few years that the question on the role of images or better visuality within discourses rose.
In this talk it is argued that as a first step we have to understand images not only as single events within literal contexts but as complex visual patterns with their own discursive lines and agglomerations. And these patterns are always on a move changing in synchronic or diachronic perspective. In other words images generate an own mode or layer within a multimodal discourse. And this mode requires an own method of analysis, which I call Visual Discourse Analysis.
Furthermore the visual mode is characterized by different types of images, which play different roles and lead to different ways of perception.
It is argued that on the one hand meaning is produced by the image itself and at the same time by a play of images and counter-images and a whole set of inter-discursive links. This play or network can only be analyzed by highlighting main discursive lines and patterns. Therefore, Visual Discourse Analysis has to be based on big picture corpora.
On the means of photographs and maps as visual representations of the United States and the USSR I will work out main groups and series of images and motives characterizing visual Cold War discourses in eastern and western Germany during the first postwar years. I will show how these images and their interaction helped to create a perception of a bipolar divided world. Photographs and maps use different sorts of signs and lead to different sorts of statements and meanings. The maps as allegedly objective representations of the world constructed specific perceptions of space and localities and theses spaces and localities were quasi filled with photographs showing the political or social system of USA and USSR. Since images do not stand alone in the investigated illustrated magazines the visual mode has to be analyzed in context of written language as headings, subtitles and text as well. Only the interplay of these different modes form the complete discourse and define potentials of meanings.
6) Guilherme Henrique de Oliveira Cestari, gui_cestari@hotmail.com
Jorge Luiz Vargas Prudêncio de Barros Pires, jorge.barros.pires@gmail.com
Miguel Luiz Contani, contani@sercomtel.com.br
Routes of the final logical interpretant on a Video jockey’s ambience
This study uses the interpretants theory to understand, with diagrammatic guides, the continuous development of a final logical interpretant elicited from the ambience set out by a VJ (video jockey) through live images and performances. The aim is to find evidence of how the signs articulated on an audiovisual presentation can determine future behaviors. The audiovisual performances conceived by VJs are kinds of semiosis that mobilize various types of language and technology; they share different ways of expression and fruition at the urban ambient. The VJ’s presentation can be considered an organism that fallibly combines and incorporate emotions, actions and habits; a thinking network on continuous adaptation, expansion and complexification. Based on the intersection of the interpretant grid, a spiral pyramid analysis diagram is the main component of the methodology. Each area of the spiral allow for some kind of reference to a semiosis, in a route that fluctuate according to each interpreter´s repertory. Within the principle that thinking is the interpretant’s movement, when such movement is performed by the human mind, a self-organizing process is constituted by establishing a path: the chaotic reign of qualities underpins increasing sophistication, organization and intelligibility. A different movement occurs simultaneously at the human mind; the route of the interpretants is unpredictable, however, each interpretant tends to converge to the final interpretant; the nature of the final interpretant is a law. The interpretant’s generation does not occur sequentially. Even so, the firstness influence underlies the secondness influence, which underlies the thirdness predominance. Normally, VJs inhabits city nights; the signs articulated by the VJs are part of the urban repertory; mixing images offered by the urban spaces, the VJ explores the city and produces thoughts about it. Each mixed image is expression of an expansive network of signs. The VJ’s dynamic is called vjing, an action that determines behaviors: because of being an action, vjing has the nature of semiosis. Understanding how semiosis work allows for a comprehension of the ambience generated by the multiple sensorial images mixed by the VJ. From the standpoint of logic, along with time, the interpretant produced by the VJ’s images tend to be more and more sophisticated, and provide the means to support the interpreter’s mind.
Keywords: VJ. Interpretants theory. Diagram. Spiral. Visual semiotics.
7) Dr. Ognyan Seizov University of Bremen, Germany (seizov@uni-bremen.de)
Multimodal semiosis in political communication online
The game-changing potential of the Internet for contemporary political communication online is often heralded, but the analytical methods for understanding its mechanics have lagged behind. Lacking a clear understanding of how political actors and citizens express their ideas and opinions through the World Wide Web’s wide gamut of multimodal presentation possibilities (e.g. Bateman, 2008) leaves out a crucial aspect of political information exchange today. The first step towards acquiring such understanding is updating our content-analytical approaches to include the various communication modes (visuals, texts, audio) and their meaningful interactions within web documents. A synergy between political communication and multimodal semiotics is, therefore, well warranted and sorely needed in order to capture the communication phenomena in their completeness.
This paper presents a pilot attempt at scrutinizing multimodal political communication online by categorizing its instances in terms of both structure and function. To this effect, it uses a five-level content-analytical annotation scheme which classifies visual, textual, and layout semantic relations within web documents (presented in detail in Seizov, 2014). The layered annotation approach is inspired by similar major efforts from SFL-based multimodal document analysis (cf. Bateman, 2008), multimodal discourse analysis (cf. O’Halloran, 2008, 2011), and intersemiotic complementarity (cf. Royce, 2007). The analysis uncovers specific multimodal expression patterns in different genres of political content online (such as news, campaigns, NGOs, and social movements) and clusters similar communication artifacts into multimodal types.
Keywords: multimodal content analysis, image-text relations, document design, political communication, online communication