Multimodal documents involve combinations of information employing differing presentation media or modes. I am particularly interested in the kinds of meanings that such combinations create and in using methods drawn from linguistics for analysing these. This has been extended over the past 5 years to include dynamic artefacs, such as narrative films.
|Early Work on Static Documents||We started a detailed consideration
of static multimodal document analysis and automatic generation in a combined
project at the Institute for Integrated Publication and Information Systems
(IPSI) in Darmstadt in the mid-90s. This involved the application and extension
of notions of rhetorical structure to multimodal documents. This was first
described at length in the paper:
|Multimodal Genres and Multimodal Corpora||
A larger project for exploring static multimodality was funded by the British ESRC in 1999. This project was the GeM project (Genre and Multimodality), which started work on building an empirically motivated construct of the linguistic notion of genre extended to include realizations across modes. The home page for the GeM project is http://purl.org/net/gem.
An important result of the GeM project has been to establish a basis for an extensively annotated multilayer corpus of multimodal documents. This corpus is being extended and will form the basis of several further empirical studies. The corpus can be reached directly here.
A book length description of some of the main results of the project is shown below.
Courses on multimodal document analysis and annotation, film, and multimodal methods applying linguistic techniques to ever broader notions of 'text', are offered regularly by myself and colleagues at the University of Bremen.
I gave an invited plenary talk at the 2nd Conference on Applying (Putonghua/English) Language Arts held in Hong Kong, October 2-3 2012, entitled "The technical analysis of multimodal artefacts: medium, mode and genre in multimodal contexts". The overheads for this talk can be downloaded here (as zipped pptx: 23Mb). A paper describing the approach to medium and genre is about to appear in print.
I held a two-day introduction to static and dynamic 2D multimodal artefact analysis at the 2008 pre-ISFC Winter Institute, Sydney Australia. The overheads for these courses can be downloaded here:
I held two talks at the 2004 International Document Design Conference (University of Tilburg, January 22-24):
Reflecting our move to consider film, I held a further talk at the 2006 Bremen Conference on Meta-representation and (self-) consciousness: Emergence of higher levels of self-organization in biological and semiotic systems, (September 27-29):
as well as presentations:
Since then there have been considerable talks and workshops on multimodal analysis; as well as publications in the area. Refer to the bibliography information for further pointers. Both theoretical and empirical research is ongoing on film, comics, instructional videos, visual narrative, mediality, modality and more.
|Publications and other activities||
Follow-up projects for GeM continue as funding and personnel allow. Expressions of interest in cooperations are also very welcome.
In 2004, Chiaoi Tseng started a PhD project in Bremen working towards a contrastive multimodal investigation of multimodal cohesion in film; see the PhDs page for more details. This work was finished in 2009; several publications have appeared reporting on this work. Janina Wildfeuer also finished a dissertation on filmic discourse structure early in 2012. These can be found in the multimodality bibliography visible here.
In 2014, Tuomo Hiippala of the University of Helsinki completed a PhD dissertation entitled "Modelling the structure of a multimodal artefact". This thesis offers detailed empirical investigation of tourist brochures in Helsinki and represents the most extensive corpus application of the GeM model to date. The work includes both empirical analysis, extensions to the theory and the model, and proposals and tools for working with corpora based on multilevel annotation schemes for multimodality such as the GeM framework. The thesis can be found here.
In addition, several book chapters have appeared describing further developments moving on from the GeM work on static documents. These include:
John Bateman, January 2018