Mixing Visual Media in Comics

By Nancy Pedri

Comics are a complex unique narrative medium that practitioners and theorists alike struggle to define. “Defining comics,” Robert C. Harvey writes, “entails cutting a Gordian-knotted enigma wrapped in a mystery” (“How Comics Came to Be” 26). Several critics consider comics a sequential art form, emphasizing the relation between words and images, and the unique literacy such a combination demands. Two nonscholarly books, Comics and Sequential Art and Understanding Comics, by two practitioners, Will Eisner and Scott McCloud, respectively, continue to hold an important position in defining comics as a distinct medium that combines sequential images with verbal text to communicate a story. Eisner specifies that in comics, writing and image-making “are irrevocably interwoven” (Comics 122), an assertion echoed by Harvey, who sees comics as “a blending of visual and verbal content” (“Comedy at the Juncture” 76) or, more specifically, the “static blending of word and picture for narrative purpose” (The Art of the Comic Book 3). Art Spiegelman proposes the term “co-mix” to connote the cooperative mixing of words and pictures at the heart of comics narration (Co-mix). Dale Jacobs also stops on the formal properties of comics to specify that “what makes [comics] distinct from other kinds of texts is their blending of images and words, a combination of sequential art and text to create meaning, including narrative meaning, for the audience” (64).

McCloud, whose focus is the sequential nature of comics, defines them as “juxtaposed pictorial and other images in deliberate sequence, intended to convey information and/or to produce an aesthetic response in the viewer” (9). Although he accentuates the coming together of words and images in comics, he distances himself from the type of blending proposed by Eisner, Harvey, and others. McCloud suggests a possible disconnect between the two modes, explaining, “Our need for a unified language of comics sends us toward the center where words and picture are like two sides of one coin. But our need for sophistication in comics seems to lead us outward, where words and pictures are most separate” (49). Hilary Chute picks up on this disconnect between words and images to highlight that although a comic is “a hybrid word-and-image form in which two narrative tracks, one verbal and one visual, register temporality spatially,” it “doesn’t blend the visual and the verbal—or use one simply to illustrate the other—but is rather prone to present the two nonsynchronously” (452). As she reminds us, in the economy of comics, the verbal and the visual do not always (or often) blend seamlessly; although the two modes may coincide in what they tell, they may also narrate different stories or offer different perspectives on the same story detail. In comics, storyworld information is indeed communicated and processed across two modes—the visual and the verbal—but the ways in which these two modes interact to tell a story are varied, complex, and endlessly stimulating.¹

These definitions of comics that grapple to understand the formal and narrative properties of this multimodal form converge in their view that the co-presence and interplay of images and words gives rise to a complex storytelling practice that is “always characterized by a plurality of messages” (Hatfield, “An Art of Tensions” 132). Comics are a highly malleable, rich narrative form able to engage virtually endless combinations of words and images to communicate a story to readers. The diversity in creative formats, genres, voices, content, and word-and-image combinations that “inspires truly new forms” (Duncan and Smith 91) makes comics particularly suited to push the boundaries of multimodal storytelling.

Indeed, comics’ multimodal storytelling practices do not stop at the combining of words and images. In it, not only words and images comingle and interact to tell a story, but also different writing styles and fonts in its verbal track, as well as different visual styles and types of images in its visual track. Despite the multiplicity and integration of different visual semiotic modes in comics, on the whole, comics “have not been widely studied as a form of visual culture” (Beaty 44), and are rarely understood in relation to other visual or multimodal art forms, with the exception of film (Booker; Christiansen; Gardner; Gordon et al.; Groensteen “Du 7e au 9e Art;” Lacassin) and to a much lesser extent photography (Bukatman; Cook; Pedri; Smolderden 119-136) and painting (Smith). The scarcity of scholarship that considers the use of different types of images (cartoons, maps, charts, photographs, and paintings) in comics is surprising, especially since the visual track and visual storytelling practices of comics have received much critical attention.

Theorists examining the visual track of comics have returned again and again to a central question: How is meaning conveyed by a sequence of images? A number of critics have focused their attention on the semantic links between framed images or panels, postulating the gap that separates the sequentially ordered images as the site where meaning is made. Whereas Eisner specifies that the role of the comics artist is “to arrange the sequence of events (or pictures) so as to bridge the gaps in action” (38), McCloud speaks of “closure” and “aspect-to-aspect transitions” whereby readers fill in the gaps between images with meaning that connects the panels (72) and completes the narrative flow. Karin Kukkonen (Studying Comics and Graphic Novels), Barbara Postema (Narrative Structure in Comics), and Mario Saraceni (The Language of Comics), among others also consider the gap, gutter, or visual space that graphically separates framed panels from each other a primary means to engage readers. Faced with gaps in the visual presentation, readers work hard to create a coherent story by filling in the missing narrative information suggested, but not shown or told by the panels and their ordering.

Interest in the narrative capacities of the visual track of comics has also led to numerous studies that focus on page layout. Charles Hatfield, for instance, highlights the complexity of the comics page and its impact on reading: “The fractured surface of the comics page, with its patchwork of different images, shapes and symbols, presents the reader with a surfeit of interpretive options, creating an experience that is always decentered, unstable, and unfixable” (Alternative Comics xiii–xiv). Examining the central tension between the arrangement of framed images (panels) in sequences and the design of the whole comics page, Hatfield highlights the importance of the visual design of comics for the creation and communication of meaning (Alternative Comics 48). In her analysis of page design in Chris Ware’s Jimmy Corrigan, The Smartest Kid on Earth, Angela Szczepaniak compares each comic panel to “a brick in the whole construction site that is the page” (87). Postulating the comics page as construction site highlights how its complex structure, where “the frame-gutter ‘fractured’ page layout” (88) depends on the readers’ interpretative work for meaning, is never a fully contained unit, as Eisner deemed it to be (40). Instead, the page as construction site is “an interactive design space” (Szczepaniak 95) able to accommodate an unpredictable number of readings, precisely because “all images on the comic book page stand for more reality than they can depict” (Duncan and Smith 141). Those who study page layout generally agree that when words and images come together to structure the narrative and communicate its meaning, unique reading strategies are put into place. In her study of multimodal novels, Alison Gibbons calls these strategies “bistable reading strategies, whereby the surface and texture of the book’s pages also becomes a significant dimension of literary meaning” (208-209).

Similar thinking informs the work of several comics theorists who stop on the use of visual metaphor in graphic illness narratives, including Renata Lucena Dalmaso, Elisabeth El Refaie, Evita Lykou, Susan M. Squier (“Literature and Medicine;” “So Long as They Grow out of It”), Theresa Tensuan, and Ian C. M. Williams. Collectively, they propose that visual metaphor provides a means to grasp and articulate how illness affects the protagonist and other agents in the storyworld who are struggling towards comprehension of self and of its experience of illness. That images, panels, and page layout carry more meaning than they depict has also bolstered the study of the repetition of images in comics or what Thierry Groensteen calls “braiding” (The System of Comics 146-148). Through the particularly charged visual storytelling technique of braiding, every comic panel is placed “in a potential, if not actual, relation with every other, leading to a densification of detail that charges the layout with meaning” (Horstkotte and Pedri 336). The braiding of visual content is a medium-specific technique of comics that has the potential to extend the “information and organization of a panel, a tier, and a page […] across many pages, across an entire book, and sometimes even across many successive publications” (Fischer and Hatfield 82). Braiding’s impact on meaning and reader engagement, as well as its broader narrative function, continues to attract much critical attention.

Surprisingly, sustained critical attention to the visual practices and visual narrative techniques of comics has not led to the systematic consideration of the blending and collaboration of different types of images in comics. Several artists working in a variety of comics genres have incorporated maps, photographs, sketches, charts, and all sorts of other visual semiotic modes into the visual track. Graphic memoirists Brian Fies (Mom’s Cancer); Marisa Acocella Marchetto (Cancer Vixen); and Art Spiegelman (Maus I; Maus II; In the Shadow of No Towers); graphic biographers Ho Che Anderson (King: A Comics Biography); Ann Marie Fleming (The Magical Life of Long Tack Sam); and Rick Geary (The Beast of Chicago: An Account of the Life and Crimes of Herman W. Mudgett, Known to the World As H.H. Holmes); graphic journalists Guy Delisle (Pyongyang: A Journey in North Korea); Emmanuel Guibert, Didier Lefèvre, and Frederic Lemercier (The Photographer: Into War-torn Afghanistan with Doctors Without Borders); and Michael Hoerger, Mia Partlow, and Nate Powel (Edible Secrets: A Food Tour of Classified U.S. History); and writers of fictional comics Eddie Campbell (The Fate of the Artist); Sonny Liew (The Art of Charlie Chan Hock Chye); and Benoit Peeters, Stephen D. Smith and François Schuten (The Leaning Girl) include photographs, charts, maps, x-ray images, reprinted receipts and comics, pencil sketches, and much more visual material in their accounts. As this short and by far non-exhaustive list of comics suggests, there is ample evidence that comics is experiencing a new thrust in creative experimentation and innovation marked by the coming together of cartooning and different types of images. The increasing sophistication with which comics artists introduce different visuals into their cartoon storyworld is deserving and in need of critical consideration. Mixing Visual Media in Comics begins this overdue work.

In his examination of multimodal novels, Wolfgang Hallet draws note to how the visual additions to novels are “a semantic, cognitive, or epistemological surplus that multiplies aspects and dimensions of the storyworld that are accessible to the reader” (“The Rise of the Multimodal Novel” 153). A similar proliferation of expression and meaning unfolds when comics artists engage multiple visual modes of communication. Few would contest that in comics, images are more than mere aids for understanding and that variation in the visual track, especially the easily recognizable intrusions of non-cartoon images, impacts how meaning is created, received, and interpreted. As multimodal theorists Michael Halliday, Robert Hodge and Gunther Kress, and Gunther Kress and Theo van Leeuwen (Reading Images; Multimodal Discourse) stress, multimodal communication is comprised of “modes,” forms within various sign systems that carry the meanings that a social collective recognizes and understands. In this social multimodal perspective, photographs, for instance, are a form belonging to a visual sign system; their meaning is determined at the intersection between what is shown on the photograph, the context in which the photograph is used, the histories of photography, and the personal experiences of the reader/viewer with photography and photographic images. Since a multimodal text engages the “dynamic interplay of semiotic resources as they contribute to narrative meaning” (8), a net of rich cross-fertilizing influences is put into place in comics that utilize, adopt, and adapt, a plurality of visual modes in the communication of their narrative.

Observations about the proliferation of meaning through the transfer of pre-existing material in multimodal texts corroborate and, at the same time, extend Eisner’s insights into how the multimodality of comics impacts reading. He writes: “The format of the comic book presents a montage of both word and image, and the reader is thus required to exercise both visual and verbal interpretive skills. The regimens of art (e.g., perspective, symmetry, brush stroke) and the regimens of literature (e.g., grammar, plot, syntax) become superimposed upon each other” (8). As Eisner suggests, the combination of words and images characteristic of the comics form puts in place a reading practice that fluctuates between, but ultimately combines, two types of literacy. Comics, in other words, instate a multimodal literacy defined by Alice Leber-Cook and Roy Cook, as “involving multiple (often interwoven) modes of expression” (23) that require “the mastery of a number of different literacies” (27).

When, however, the visual track is comprised of different types of visual material, readers are encouraged to engage in a multifaceted visual literacy, one that takes into consideration the different information—social, historical, imaginary—each type of image communicates alone and in combination with other types of images in a comics universe. In these instances, readers are asked to engage multiple visual literacies, to integrate them in the construction of the storyworld, and to synthesize them into a coherent narrative meaning. Eisner’s visual interpretative skills, which are restricted to information communicated across aesthetic techniques, cannot fully account for visually complex narrative strategies where a cartoon-drawn visual narrative is interrupted by and complemented with different types of images. To create meaning from these comics requires an interpretive process described by Hallet as “multisemiotic reading,” whereby,

on the one hand, the reader must decode and comprehend each of the semiotic codes utilized and displayed in its own right. A map or a photograph or a diagram each employs its own codes and grammars and unfolds a meaning of its own. On the other hand, the reader must be able to understand the semiotic interplay of all these modes and often of visual modes and verbal discourse in particular, so that reading becomes a multi-literate act based on the capacity to integrate a range of single literacies (e.g., linguistic, visual, topographical, mathematical, and so forth) in an act of making meaning out of the narration. (“The Multimodal Novel” 168).

In comics that narrate across different types of images, readers join aesthetic considerations with considerations of how a particular type of image communicates and what meanings are implied in its mode of production, its history, and its familiarity to readers. When images are borrowed or quoted in the visual track of comics, their original context (real or imagined), as well as their re-presentation in the new context of the comics cartoon universe, also factor into the visual interpretative process. The appropriated images are made to exist in a different version, taking on new meanings with their new configuration and within the new comics context. The mixing of visual images in comics thus orchestrates a unique reading experience, one that draws on the preconceived notions of readers, accentuates the mechanics of visual storytelling, instates complex multimodal reading practices, and distinguishes comics as a highly malleable and experimental multimodal form. It asks readers to adopt an interpretative practice that respects, but also crosses boundaries, separating visual semiotic modes.

Comics that mix visual media can thus be said to partake openly in the post-media condition, which sees various media intermingling and mixing. Art and media theorist Peter Weibel proposes that for art in the post-media condition, “no single medium is dominant any longer; instead, all of the different media influence and determine each other.” Multimodal theorists remind us of the richness of all texts, suggesting that modes “rarely, if ever, occur alone” (Jewitt & Kress 2). And, visual theorist W.J.T. Mitchell stresses that “all arts are ‘composite’ arts (both text and image); all media are mixed media, combining different codes, discursive conventions, channels, sensory and cognitive modes” (94–95). Complex relationships, including aesthetic, narrative, semiotic, and semantic relationships, exist among and between modes in all texts. However, they may very well be more overtly and critically engaged in texts that rely on more than one mode to communicate meaning, such as comics, and particularly comics that disrupt the visual stylistic coherency of the visual track by introducing a different type of image.

Mixing Visual Media in Comics sets out to explore the mixing of cartooning with other types of images in comics in terms of its narrative power. It explores what stories (and histories) are evoked, told, and challenged by different images, how they are experienced, and what type of reading practices they instate. The result of an intensive three-day international workshop held in St. John’s, Newfoundland (Oct. 13-16, 2016),² the papers collected in this issue challenge the reader to consider the ways in which the convergence of different visual modes expands the multimodal potentialities for comics.

Although contributors adopt different methodologies and engage in close analysis of different comic books from different comics genres, several interrelated assumptions guide the research collected here:

1. The visual track of a comic book does not simply show, but also tells; visualization is an integral dimension of the comics’ narrative discourse.
2. A comics author makes strategic choices about what to show and how to show, thus ensuring that the assumptions and beliefs of readers factors into the reading and the making of the visual narrative.
3. The visual composition of the comics page and the work as a whole have a primarily narrative function; they impact tone and direct focus and meaning.
4. The multiplication of visual semiotic modes in comics leads to a meaning-making process that interrelates different semiotic elements and that is positioned at the intersection of multiple modal systems.

This multi-person investigation of how cartooning works alongside other types of images in comics to shape and communicate a story suggests novel ways of understanding the history and practice of visual modes of representation, especially for what concerns visual storytelling techniques and literacies. To determine the distinct and shared storytelling properties of different types of images, as Mixing Visual Media in Comics aims to do, is to take a significant step toward better grasping how images evoke emotions, drive reader engagement, and perhaps even foster changes in behavior. This volume is thus situated within the broad project of reaching a more comprehensive understanding of visual forms of storytelling and how they interact.

Notes

Works Cited