In this articles we will make An Introduction to Multimodality but before going to Multimodality we will point out first Mode. The term “Mode” refers to the set of culturally and socially shaped resources used for meaning-making. Mode represents a channel of communication for which no overarching name had been introduced (Van Leeuwen, 1999). Examples of modes include image and writing on the page, moving sound and image on the screen, gesture, speech, and gaze is used in interaction with each other. It is not that these modes of communication had not been recognized early or studied; much research was done on the mode of gesture and different modes of communication. Introducing a variety of communication means has become a challenge for the predominance of spoken and written communication. Also, it opens up a way of studying the new types of communication in which people are making meaning and how these meanings are multimodality interlinked. Modes are not fixed but are created through social processes. These modes are not universal, depends upon the community and their and understanding of semiotic characters.
Since Plato’s dialogue “Cratylus,” the question of meaning-making has always been at the center of any theory of language and communication (Schiffer : 1972) Multimodality is a process of multiple modes in one medium. For example, to understand a weather forecast on television, we need to understands the spoken, written, weather-specific language, symbols, and geography. Multiple-way of understanding these things are called modes that contribute to the customer’s understanding of the composition. This includes the use of isolated text as a primary in communication to the images being used in the digital age to convey a meaning. Multi-modality discusses the communication practices concerning the textual, Oral, and visual resources used to compose the meaning.
While all sorts of communications were a part of composing practices and the academic fields and scientific field, this phenomenon of multimodality started in the 1960s. Ronald Barthes led to many disciplines in the field of multimodality. Above all, composition instructors and rhetoric included multimodality in their coursework. In order to understand the language, one must have the knowledge from the written language to the visual graphics, gesture, and speech, etc.
Kress’s Work on Multi-Modality:
Although the term of the multimodality involves both “medium and mode,” but they are not alike, it depends on how the author is using it as per his terms of tradition.
Gunther Kress’s knowledge about multimodality is acceptable within the social semiotics approach. Kress introduces two ways. Firstly he says mode is a culturally and socially developed resource of the meaning like moving images, speech, gestures are the examples of modes that are used and usually observed in today’s date. Secondly, semiotic modes are formed by natural characteristics of the method and the requirements, values, and histories of the societies and cultures.
Read also: What is language
Multimodality has different mediums that are culturally and historically situated and which later breaks down into many; every source has various limitations for meanings. For example, if we break down the writing into its modal resources, it will turn into lexical, graphical, grammatical, syntactical medium. Graphics resources are further broken down into type and font size. In Kress theory, it is clear that “Mode is meaningful in every perspective,” and it shaped the deep historical, social, and ontological orientations from the society and its culture, and every sign represents it. Modes are shaped by the system in which they participate. Modes are shaped overtimes into cultural forms, an example is being filmed, which includes visual, modes of sounds and music, modes of dramatics, and actions speech.
Concerning the social semiotics method, the meaning is conveyed, which is easy and available for others. These mediums include images, audio, text, videos, etc. Social mediums of multimodality include sociocultural, technological, semiotics, such as newspapers, billboards, film, television, theater, classroom, etc. Multimodality uses electronic medium by making digital modes with closely linked images, layouts, videos, etc. These mediums have become the modes of conveying that consider the present and futures modes.
Digital media is also observed from the perspective of multimodality; The fact is that multimodality is not just developing the print-text resources to the screen-based presentations, the relationship between the viewer and speakers is also evolving. Due to the increase in the use of digital media, the only method of description is no longer just text, now digital media is more prominent. It is not the current use of the internet-based compositions; the term multimodality is far more widespread, applied to the various forms of the text suppose literature, advertising, art and literature, etc. Communication is not only limited to reading the text but also to understand or read other modes such as gestures, sounds and images.
History of the Multi-Modality:
Multimodality has received its importance through the history of writing and communication. The method of Multimodality was first introduced in the 4th century. With the passage of time and an increase in technology, many news modes of communication were introduced. The term of multimodality became standard in the 21st century, as people observed many new techniques of communication other than just spoken and written. These new types of communication were advertising, social media, gestures, visual and fine art. Rhetoricians of the classical times implied gestures, expressions, voices, in their communication. However, this multimodality was not given importance until the 20th century. (John A Bateman:2008) says in his book of the multimodality and genre that, “Nowadays… text is just one strand in a complex presentational form that seamlessly incorporates visual aspect ‘around,’ and sometimes even instead of the text itself.” Multimodality has soon become the normal state of human communication.”
Read also: STYLISTIC ANALYSIS OF I HAVE A DREAM
Between 1960 and 1970s, Many writers observed the phenomena of photography, audiotape, film to develop new ideas about the composition. It was the expressionist method of communication. Expressionist way of the writer’s imagination encouraged the writers to search other modes of communication outside the language by relating it to visual, temporal medium, visual and oral, etc. Murray once told his students to observe as a camera and write down every observation they observed for an hour. As a writer, it is essential to move onward to something that is within us to our senses like hearing, smell, taste. A writer makes meaning through the use of his experience.
During the time of 1970 and 1980s, multimodality was further developed through a cognitive approach to learning. James Berlin and Joseph Harris were prominent figures in this development. As it is impossible to get direct access to cognition, eye-tracking data can serve as a kind of window to reception processes (Yarbus: 1967). They studied the use of alphabetic in writing and compared it to music, art, and other sources of creativity. This study included how writers think and plan their writing process.
Further, Joseph Harris said that alphabetic writing is due to multimodal cognition. Writers often visualize their meanings with the help of the non-alphabetic meaning through music and visual imagery, etc. The idea was best emphasized in the work of Neil Fleming, a renowned neurolinguistic, in which his three styles of auditory, visual, kinaesthetic to understand the modes in which the people learn the best, interpret, and create meanings.
In the 1990s, A new mode of communication was introduced with the help of the internet, digital technologies, and personal computers. The communication of the new generation changed, They become used to a text that circulated in pieces across several mediums of image, color, and sounds. Modes of communication evolved so that the students in the class could be acknowledgeable with the help of graphics, videos, computers but not just alphabetic writing. Teachers had to change their education and teaching practices to include multimodal designs and help the student achieve success.
Effects of Multi-Modality on the people’s mind:
Multimodality can affect and change the perception of the viewer. Bezemer and Kress, were the pioneers of multimodality and semiotics, says that students understand better when the text is delivered with the help of secondary medium suppose they are being taught with the help of image and text, that will help them to learn to understand more. This is because it draws more attention to the reader, meaning moves from one medium to another, which allows the audience to redefine their semiotic relation. The common understanding of the language comes with the help of semiotics, the relation between the symbols and words. Meaning is changed after placing it with the preconceived meaning in the new sources, whether those in the digital, visual, or audio. It creates a new meaning to understand for the readers.
Multimodality has evolved with the help of technology.. The idea of the reading is completely changed with innovation and quick transmission of the information. Anne Frances talks about how the text as an action changed from with the help of technology reform. Along with the interconnection of the media, computer-based technology made it possible and improved the rhetorical device and the audience. This evolution of technology has introduced a new concept to writing, thus collaborative use of modes and keeps the reader and the writer in a relationship
Functional Approach of Multimodality:
Multimodality has its significance in the community depends upon the educational, social private, and public communities. Due to the multimodality, the private domain is evolving into the public domain in which different sections of the community function.
Based upon the functional communities, decide through their social interaction and which mode should be used. In the same pattern, these assumptions help to determine how multimodality functions and create new social and cultural identities. For example, Bezemer defines it as “culturally and socially shaped sources for meaning-making. “According to them, something to be called a mode of interaction, there must be a shared cultural sense with the community that can help them to extract or realize the meaning. Cultures that have similar or different sources of the knowledge, representations, and understanding will communicate through the same or different modes of communication. For example, the visual mode of communication determined by our daily use.
Multimodality is used externally to increase customer satisfaction by proving multiple means of communication. With the popularity in the text, social media, chat in the 21st century, most of the businesses try to promote cross-channel management. The business aims to solve customer queries quickly with the help of different modes of communication. Companies aim for better communication in real-time to make customers’ services better.
The private sphere is another term of multimodality, where a person has a personal authority and distance from all the places around him. Family photographs are an example of the multimodality in this private sphere. The family take pictures and compile them in an album that is meant to be shown to the other members of the family. These private albums have now entered in the circle of internet due to the rapid increase in technology.
Applications of the Multi-Modality:
Information is presented through digital media and multimedia that offers a multimodal composition. Standard words and images are shown as moving speech and images to understand the meaning of the words better. (Joddy Murray: 2013) says both non-discursive and discursive modes should be studied to see how the modes and media are used to create the composition. Murray shows a new way of thinking about composition that allows the images to be “sensuous and emotional symbols of what they represent and does not focus on what is abstract. These new website platforms allow families to share photographs and add text, design, and photographs. By using all these modes, families create a story of their lives known as universal audiences. Pauwels states that these digitized family snapshots reveal their family culture, values, beliefs, and aspirations.
Multimedia advertising is also a type of communication used to provide services. There are many forms of advertising through DVD’S, VIDEOS, ONLINE Advertising, etc. These advertisements methods help companies to increase their customers through multimedia advertising. Modern technology has brought innovation in the field of advertising and a shift from the traditional ways of advertising that included texts only.
Hashtags and user tags are one of the most known sources of communication on social media that helps us find the trending topic over the social media site, which is used by various social media. These features are linked to the user and themes. Characteristics of the multimodal can be observed on the Facebook status update option on Facebook. The status update includes the affordance of twitter, instant messages, and text as a single feature. What’s on your mind feature on Facebook is a source of multimodality over Facebook. These multimodal features allow the user to add links, images, text, or tag other users. These new media platforms have revolutionized the generation with the latest communication technique over the internet and social media platforms.
Read also: Nazik al Malaika’s poetry
Multimodal texts are very beneficial in almost every aspect of life. It has a significant impact on education. As texts are no more restricted to print technology (Healy, 2008). Now people have a range of multimodal texts. Teachers can make learning interesting for the students by giving them access to the internet can be the most exciting phase of communication. It is widely used in business communication and advertisements. Now the advertisements are not just limited to the printed forms.
Kress, Gunther R. Multimodality: A Social Semiotic Approach to Contemporary Communication. London: Routledge, 2010. Print
Murray, Joddy (2013). Lutkewitte, Claire (ed.). “Composing Multimodality”. Multimodal Composition: A Critical Sourcebook. Boston: Bedford/St. Martin’s.
Van Leeuwen, Theo (1999). Speech, Music, Sound. London: Palgrave MacMillan.
Yarbus , A. L. ( 1967 ), Eye Movements and Vision , New York : Plenum Press .
Schiffer , S. R. ( 1972 ), Meaning , Oxford : Oxford University Press .