Project "Multimodal Interaction"

Language, Multimodality and Mediality

Team: Prof. Dr. Axel Schmidt

The project deals with the change of interaction through media use. The basic communicative function of media is to bridge space and time. This changes interaction in a systematic way. Face-to-face interactions are oral and physical, copresent, simultaneous and reciprocal. Depending on the medium, this applies only to a limited extent to mediated interactions: When people are talking on the phone, they cannot see the other person, when they watch television, there is no channel for their reactions, and in social media, participants often communicate in writing. In addition, many media communications - such as television programs, theatre plays or YouTube videos - are based on a previous production process. This leads to a separation of production, product and reception. The interactions shown to the viewers inthe product have been designed in interactions during the production process. Thus, in contrast to everyday interactions, those interactions within the product are ‘pre-composed’ interactions.

Video data from theatre, television, and YouTube productions are currently being collected. The linguistic-multimodal analyses focus on the following questions:

  • How is the construction of media products organized linguistically and interactively?
  • How do participants distinguish and connect different frames of meaning (e.g. discussion of a scene versus playing a scene at rehearsals)?
  • Which aesthetic demands exist and how are they implemented linguistically and interactively?
  • By means of which practices do the participants generate and secure production-specific knowledge?


  • Making Let’s Plays watchable: An interactional approach to visualization. In: Crispin Thurlow, Christa Dürscheid and Federica Diémoz (Ed.): Visualizing Digital Discourse. Berlin u.a.: deGruyter (i. pr.) (with K. Marx).
  • Multimodality as Challenge: YouTube Data in Linguistic Corpora. In: Wildfeuer, Janina/Pflaeging, Jana/Bateman, John/Tseng, Chiao-I/Seizov, Ognyan (Ed.): Multimodality. Towards New Discipline. Berlin u.a.: deGruyter ( (with K. Marx).
  • Prefiguring the future: Projections and preparations within theatrical rehearsal. In Deppermann, Arnulf/Streeck, Jürgen (Hg.): Time in Embodied Interaction. Synchronicity and sequentiality of multimodal resources. Amsterdam/Philadelphia: John Benjamins, 2018, S. 231-260.
  • How to deal methodologically with Entertaining Hatred and Aggressive Humor on the Web (and Television). In: Studies in Communication Sciences, 2011, 11, 2, pp. 133-166.


  • Embodying avatars: Evidence from Let’s Play practices. Vortrag, IMPEC, 5.7.2018, ENS Lyon (with K. Marx)
  • Instructions and negotiations in theater rehearsals – How and why they change. Vortrag, ICCA, University of Loughborough, 14.7.2018 (with Stefan Norrthon).
  • When spectators talk back – practices of audience involvement in broadcasted video games, Vortrag auf VALS-ASLA, Universität Basel, 06.06.2018 (with I. Neise).
  • „Elicited and Exploited Emotions – How Reality TV Demands the Body”, Conference „Affects – Media – Power”, 29.6.-1.7.2017, Freie Universität Berlin (with D. Klug)
  • “Doing Genre – how television editors meet genre claims”, IIEMCA Conference “A Half-Century of Studies”, 09.-13.07.2017, Otterbein University Westerville, Ohio (USA) (with S. Reineke).
  • “Multimodality as challenge: YouTubeData in Linguistic Corpora”, Third Bremen Conference on Mulitmodality, 20-22.09.2017, Universität Bremen (with M. K. Marx).

Kontakt: Prof. Dr. Axel Schmidt,