Multimodality: Documents and Film and Comics and ...

Definition
Early Work
Multimodal Genres and Multimodal Corpora
Regular Courses
Presentations
Publications and other activities

 

Definition

Multimodal documents involve combinations of information employing differing presentation media or modes. I am particularly interested in the kinds of meanings that such combinations create and in using methods drawn from linguistics for analysing these. This has been extended over the past 5 years to include dynamic artefacs, such as narrative films.

  • The main sources of inspiration for the approach to static documents ere: Gunther Kress and Theo van Leeuwen's treatments of multimodality, Jim Martin and Jay Lemke's socio-functional developments of genre, and Rob Waller's accounts of design.
  • For film, there is a combination of influences, including again Theo van Leeuwen and Christian Metz. For more information about some of our current film activities see below. We also ran a PhD programme on the "Textuality of Film" from 2009 to 2011 as well as third-party funded projects on film and visuals (in German).

A selected bibliography can be created by selecting the multimodality virtual sub-bibliography on my bibliography pages or by clicking here.

Early Work on Static Documents We started a detailed consideration of static multimodal document analysis and automatic generation in a combined project at the Institute for Integrated Publication and Information Systems (IPSI) in Darmstadt in the mid-90s. This involved the application and extension of notions of rhetorical structure to multimodal documents. This was first described at length in the paper:

Bateman, J. A., T. Kamps, Reichenberger, K. and Kleinz, J. (2001). "Constructive text, diagram and layout generation for information presentation: the DArt_bio system." Computational Linguistics 27(3): 409--449.

A pre-publication version of this paper can be downloaded here (1.7Mb, gzipped postscript) or here (2.5Mb pdf)

Multimodal Genres and Multimodal Corpora

A larger project for exploring static multimodality was funded by the British ESRC in 1999. This project was the GeM project (Genre and Multimodality), which started work on building an empirically motivated construct of the linguistic notion of genre extended to include realizations across modes. The home page for the GeM project is http://purl.org/net/gem.

An important result of the GeM project has been to establish a basis for an extensively annotated multilayer corpus of multimodal documents. This corpus is being extended and will form the basis of several further empirical studies. The corpus can be reached directly here.

A book length description of some of the main results of the project is shown below.

Regular Courses

Courses on multimodal document analysis and annotation, film, and multimodal methods applying linguistic techniques to ever broader notions of 'text', are offered regularly by myself and colleagues at the University of Bremen.

Presentations

 

 

I gave an invited plenary talk at the 2nd Conference on Applying (Putonghua/English) Language Arts held in Hong Kong, October 2-3 2012, entitled "The technical analysis of multimodal artefacts: medium, mode and genre in multimodal contexts". The overheads for this talk can be downloaded here (as zipped pptx: 23Mb). A paper describing the approach to medium and genre is about to appear in print.

I held a two-day introduction to static and dynamic 2D multimodal artefact analysis at the 2008 pre-ISFC Winter Institute, Sydney Australia. The overheads for these courses can be downloaded here:

  • Introduction to Basis Techniques for Analysis Multimodal documents (pdf; 7Mb)
  • Static 2D Artefacts (pdf; 10Mb)
  • Dynamic 2D Artefacts: film (pdf, 2.4Mb [film extracts excluded due to copyright restrictions})

I held two talks at the 2004 International Document Design Conference (University of Tilburg, January 22-24):

Reflecting our move to consider film, I held a further talk at the 2006 Bremen Conference on Meta-representation and (self-) consciousness: Emergence of higher levels of self-organization in biological and semiotic systems, (September 27-29):

  • Film as (meta)-representation? [pdf 648kb, password protected because of copyrighted materials contained: email me for access]

as well as presentations:

Since then there have been considerable talks and workshops on multimodal analysis; as well as publications in the area. Refer to the bibliography information for further pointers. Both theoretical and empirical research is ongoing on film, comics, instructional videos, visual narrative, mediality, modality and more.

Publications and other activities

Follow-up projects for GeM continue as funding and personnel allow. Expressions of interest in cooperations are also very welcome.

In 2004, Chiaoi Tseng started a PhD project in Bremen working towards a contrastive multimodal investigation of multimodal cohesion in film; see the PhDs page for more details. This work was finished in 2009; several publications have appeared reporting on this work. Janina Wildfeuer also finished a dissertation on filmic discourse structure early in 2012. These can be found in the multimodality bibliography visible here.

In 2014, Tuomo Hiippala of the University of Helsinki completed a PhD dissertation entitled "Modelling the structure of a multimodal artefact". This thesis offers detailed empirical investigation of tourist brochures in Helsinki and represents the most extensive corpus application of the GeM model to date. The work includes both empirical analysis, extensions to the theory and the model, and proposals and tools for working with corpora based on multilevel annotation schemes for multimodality such as the GeM framework. The thesis can be found here.

In addition, several book chapters have appeared describing further developments moving on from the GeM work on static documents. These include:

book chapter Bateman, J. and Delin, J. (2003) Genre and Multimodality: Expanding the context for comparison across languages. In: Willems, D., Defrancq, B., Colleman, T. and Noël, D. (eds.) Contrastive Analysis in Language: identifying linguistic units of comparison. Houndmills: Palgrave Macmillan.    
Amazon UK
book chapter Bateman, J., Delin, J. and Henschel, R. (2004) Multimodality and empiricism: preparing for a corpus-based approach to the study of multimodal meaning-making. In: Ventola, Eija, Charles, Cassily and Kaltenbacher, Martin (eds.) Perspectives on Multimodality, Amsterdam: John Benjamins. pp65-89.       Amazon UK
book chapter Bateman, J., Delin, J. and Henschel, R. (2006) Mapping the multimodal genres of traditional and electronic newspapers. In: Royce, T. & W. Bowcher. (eds.) New Directions in the Analysis of Multimodal Discourse. New Jersey: Lawrence Erlbaum Associates.       Amazon UK
book Bateman, J. (2008) Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents. London: Palgrave Macmillan.     Amazon UK
book

Bateman, J. and Schmidt, K-H. (2011) Multimodal Film Analysis: how films mean. London: Routledge.

    Amazon UK
Again, there are now many more books and publications, information can be found in the bibliography pages as pointed to above. For introductions to all areas of multimodality, see the following textbook.
book

Bateman, J., Wildfeuer, J. and Hiippala, T. (2017) Multimodality: Foundations, Research and Analysis - A Problem-Oriented Introduction. Berlin: de Gruyter..

    Amazon UK
           

 

 

John Bateman, January 2018