Multimodality is a term that is used in Linguistics as a means of communication. It is a broad term, and there are multiple things(MODES) that we used for our conversation, e.g., speech, writing, videos, audios, pictorial, gesture, symbols, etc. Sometimes a term multiple literacies are used to describe the multimodality. Multimodality means various modes or literacies or means of communication that we can use to deliver our messages.
For example, to comprehend and understand football’s game, we need to know about the various things that are used in the game of football, e.g., rules of games, different terminologies, duration of the game, and broadcasting commentary, etc. These various modes and literacies play an essential role in the understanding of the audience. In ancient times, people only used writing and speech to deliver their messages, but in today’s world, we know that multiple modes are used for communication. In the development of the internet, website, social media, and many other modern means of communication, we use various things to deliver our messages. Multimodality tells us about these multi-practice communications in the form of textual, lingual, spatial, aural, or visual resources to describe our words.
When we observe various types of communication, we found that the use of multiple techniques in conversation as the visual and oral in the text make our communications more powerful and attractive. For example, if someone wants to describe a particular event in the form of writing or expression in words will not be effective as much as interpretation will be useful if having some visual, oral moods in a message. So, in simple terms, we can say that multimodality is the use of two or more modes of communication for the delivery of our message. Here patterns are culturally and socially shaped medium of expression.
Multimodality is called the theory of communication and social semiotics. Here transmission means the delivery and reception of the messages. How do we perceive different notes and how do we deliver messages to other this is called communication. On the other hand, term social semiotics is also used in multimodality. This term is applied to comprehend and understand the discussion of the people. It describes how people use various means of communication in multiple particular communication and social settings. Here modes are described as not a fixed set of rules and regulations but for what these are made culturally and socially in routine correspondence. Here is also an essential question on how people manage signs in institutional and interpersonal communication to deliver a strong message. It is a fundamentally important point since semiotic can shape social relations and society itself. A crucial aspect of social semiotics is that various modes of communication are used historically, socially, and culturally for communication. In the social semiotic or semiotic system, we find multiple levels. These are linguistic, audio, visual, gestural, and spatial levels. Here linguistic level means the use of oral or written language. In linguistic level or oral or written communication, we find vocabulary, punctuation, grammar, and paragraphing. In the audio level, we observe music and sound effects as silence, pause, volume, rhythm, pitch, etc. moreover Spatial scale describes layout and organization of objects and space, e.g., direction, position in the area. Visual level describes the presentation of the pictorial in communication. These pictures may be moving or still, images or simple colors, drawing, and foregrounding. While the gestural is the body language or facial expressions as movement, speed, body position, stillness, etc.
Read also: What is structuralism
The study of communication from this perspective seeks to identify and inventories the semiotic options that are available to communicators, and that they choose to make. These modes are observed and seen how they make a different type of communication more powerful and meaningful. But on the other hand, it is totally up to the user how he/she adjusts different modes in his/her communication. The meaning of these modes is carried and extracted within the context where these are going to be used. In the context of multimodality, the usage and implications are described and applied because of available choices for communicators. While on the social semiotic purpose, this shows how speakers or communicators use text, including various modes and technological use.
Mode and Medium
When we move forward in our discussion, we find that there two terms are used. The first one is the medium, and the second one is the mode. Mode and medium are not synonyms for each other as people consider them. At the first point, we will describe the term mode. According to Gunther Kress, modes are of two types. The first type of mode is the mode that is shaped culturally and socially for making or forming meaning; For example, the speech, layout, images, oral, recordings, writings, pictures, etc. And the second type of the mode is Semiotic mode, having the potentialities and intrinsic characteristics of the medium. It also has the histories and values of societies and culture.
So every mode has different resources. When we observe the mode of written text, we find that various things are involved in the writing process as grammar, noun, verbs, for lexis, syntactic and graphic resources. The visual resources also can be distributed into size, font, type, etc. According to Kress’s theory, mode having meanings. It implies on an ontological, social, and historical basis as we take the example of a movie where we find different mode orientations of mode in the shape of historical, social, and cultural foundation.
On the other hand, in social semiotic, the medium is the substance that has the meaning. Medium includes the text, audio, visual, etc. in it. Socially, sociocultural and technological practices exist in semiotic in that particular society or culture such as films, magazines, journals, radio, billboards, television shows, etc. Multimodality enables the methods of digital and electronic modes using the images, videos, writing, layout, speech, etc. So in simple words, we can say that medium is a source of delivery of the modes in various contexts as per the requirements. The sources that we use to deliver our messages as radio, television, newspapers, magazines, journals, books, websites, the internet, etc. are the medium of expression. On the other hand, a form of expression that is being used for the delivery of a message is a mode as audio, video, picture, signs, symbol, diagram, etc.
Multimodality throughout the History
The phenomenon of multimodality, though, had not proper or exact name as it has now, but is under the discussion and study since ancient time. In the 4th century BC, researchers find clue the focus of classicists was on the rhetoric of gesture, voice, intonation, and expression in public speech. But the proper term multimodality was introduced in the 20th century. With the development of technology and scientific instruments provides various modes of expressions of ideas or presentation. In the late 20th century, modes are going to be used to illustrate and deliver messages and thoughts.
In the 1960s, various writers tried to use movies, films, voice recordings, and pictures for their message delivery. Expressionism invites writers to place their visual, aural, spatial, or any other medium to express or deliver their messages. Donald Murray describes that we need to move forward from what we have within us as what we see, hear, smell, feel, and taste regarding the world around us. And this is the use of the experience that a writer always does, and we called it expressionism. Murray’s Instructions for Students: You feel and see yourself as a camera to capture everything and describe it through your writing. Every single visual observation that they make for their working hours. So in simple words, by the use of our five senses, what do we communicate, learn, deliver, and perceive is called expressionism.
Read also: What is applied linguistics
Afterward, the 1970s multimodality was further progressed under cognitive research for learning. Various analyses played a significant role in its development. Most prominent names under the domain of cognitive research worked for multimodality are Jason Palmeri, James Berlin, and Joseph Harris. They studied alphabetic writings and made their comparison to music, art, and other forms of creativity. This research had a cognitive approach and examined how writers thought about and planned their writing process.
With the development of internet and website data, multimodality was geared up after the 1990s. Personal computers, websites, and digital technologies changed the modes as well as a medium of communication. The communication improved from the traditional to simple techniques such as images, colors, sound, and spatial. The literary books, journals, and magazines also included the pictures, symbols, and sign for a better understanding of the learners as well as teachers. These new techniques helped a lot in the comprehension and understanding of the messages.
Multimodality and Language
Every writer has its audience, and text aims to improve itself to engage its audience. So the multimodality is playing a critical role in this regard. It has become more sophisticated and appealing for the engagement of the audience. It is more inspiring and inviting characteristics for the audience. It is something more than solicit an audience as well as very helpful for the perception of meaning from the text. It even changes the way of knowledge of the people. The association of symbols and words is called the semiotic, and it is beneficial for the understanding of the language. A text with multimodal features changes its semiotic effects by putting words with conceptualized meaning in a new context. That very context may be visual, audio, or digital. This turn creates a unique and different sense for the audience. According to Kress and Bezemer, students can understand various information in different ways when text is delivered with another medium in the form of a sound, image, or presented in an only alphanumeric way. It is only due to drawing the attention of viewers. Here meanings are moving from one medium to another, and also requires the audience to redefine their semiotic connections. The re-contextualizing of a text with another medium develops a sense of comprehension and understanding. This novel type of learning can be controlled by various media types that we are going to use. Multimodality can be used to describe a text with a particular argumentative goal, make a definition, a value judgment, or make any policy decision.
Applications of Multimodality
Multimodality also describes the concept of the genre by developing grey areas out of what was once black and white. Carolyn R. Miller, a prominent professor of rhetoric and technical communication, analyzed in her genre studies of web blogs. She describes how genre transformed a blog. Here she states that there is a strong relationship with the essential features that changes simple writing into a blog. Muller explains that the chronology, combination of links, and frequent updating are the vital qualities of a blog. But somehow central features of blogs are obscured when evaluating of studying multimodal texts. Here some points are missing, like the ability for posts must be independent of each publisher while others are present. It is a situation, and here the genre of multimodality is challenging to define although the style is ever-changing, dynamic, and evolutionary.
The effects of technology on multimodality are also very prominent. This evolution has created a new genre of writing that attaches the readers and writer in contact. The concept of reading has become different with the advancement of technology. With the help of technology, the transmission and communication of the information have become quick. With the help of websites, social media, radio, television, the communicator and receiver of the message are in permanent contact, and these ways are affecting communication methods very much.
In the educational field and learning process, multimodality is playing a significant role. With the rise of technology and scientific inventions, a study has become fun nowadays. Here in modern books, various modes of communication are being used. For example, here in books, pictures, diagrams, signs, and symbols are going to be used that are very helpful for the understanding and comprehension of topics and lessons. Here in education, various mediums are going to be used for learning. In modern schools, the use of TV Screens, Tablets, computers, and laptops are also prominent, so knowledge becomes an easy and convenient task with the help of these facilities.
Read also: What is language
Multiliteracy is also an extended-term that is going to be used with multimodality. With the help of these gadgets and services, people are learning unconsciously. Blogs, social media, websites, and many other resources are helping and assisting people in getting knowledge. These new technologies and e-learning plate forms have become the more prominent part of the multimodality theory.
Classroom management also has become a fun and convenient nowadays due to the use of multimodality. Here we find visual and computer-based literacy, use of images, sounds, and visuals in-class learning, teaching, and management has become an order of the day for education. It is the most powerful and attractive way of learning and very helpful for students. Students can understand various points with the help of audio, visual, graphics, diagrams, and signs. In the field of higher education, the use of multimodality is very vast. Now we observe teachers and students are using various modes and mediums for learning. Here we find online classes, video and audio recordings of the lecture, different pictures, and diagrams in the process of learning.
The use of multimodal techniques in various societies and communities has become the order of the day. Here we find, people use multiple technologies for the delivery of messages and communication in society. In different types of celebrations and rituals, they adopt audio, visuals, videos, recording, pictures, signs, and symbols. In this way, they transmit their message to others in a better way. Television, DVD, Dish, Google, Wikipedia, Facebook, etc. have provided great leisure and change the modes and mediums of communication of people socially and culturally.
Read also: Philosophy of Language
The advertisement of different objects has been changed from simple to multimodal. In the beginning, people use only writing for their publication and advertisements, but now in modern times, they use various modes of communication in a single ad. Here is text, audio, video, and pictures in the multiple ads. The advertiser also uses different mediums for his advertisement publications, like television, social media, blogging, etc.
We also observe the uses of multimodal in the field of traffic signals and communication. With the help of various symbols and signs, we can get the idea more clearly and precisely while traveling. Signs and symbols give a clear and quick direction with can not be achieved able with long narratives in writings. Drivers can judge the situation and condition of the road, traffic, weather, speed, and much more regarding driving on the road with the help of symbols and traffic signs. These are signs and uses of multimodal techniques that enable a driver about the crowded areas, mosques, schools, churches, petrol pumps, cantonment areas, speed breakers, and many other things on the road.
Read also: Corpus linguistics
So multimodality has changed every field of life and affected it very much. With the use of multimodal techniques, advertisements, communication, customs, education, traveling, transportation, driving, teaching, preaching, and even every field of life has been changed. Multimodal techniques are playing a very active role in understanding and conveying the messages. These techniques not only facilitating the users but also communicating the words more effectively and powerfully. So the multimodality has become the most prominent part of every type of communication in the modern world.
Kress, Gunther R., and Theo Van Leeuwen. Multimodal discourse: The modes and media of contemporary communication. Vol. 312. London: Arnold, 2001.
Kress, G. (2009). Multimodality: A social semiotic approach to contemporary communication. Routledge.
Kress, G. (2000). Multimodality. Multiliteracies: Literacy learning and the design of social futures, 182-202.
Jewitt, C., & Kress, G. R. (Eds.). (2003). Multimodal literacy. New York: Lang.
Written & Reviewed by: Muhammad Asad Kasra,
M.Phil Scholar, Riphah Institute of Language and Literature, Riphah International University Lahore, Pakistan.