Högskolan i Skövde

Research in naturalistic event perception highlights the significance of visuospatial attributes pertaining to everyday embodied human interaction. This research focuses on developing a conceptual cognitive model to characterise the role of multimodality in human interaction, its influence on visuospatial representation, event segmentation, and high-level event prediction.

Our research aims to characterise the influence of modalities such as visual attention, speech, hand-action, body-pose, headmovement, spatial-position, motion, and gaze on judging event segments. Our particular focus is on visuoauditory narrative media. We select 25 movie scenes from a larger project concerning cognitive film/media studies and performing detailed multimodal analysis in the backdrop of an elaborate (formally specified) event analysis ontology. Corresponding to the semantic event analysis of each scene, we also perform high-level visual attention analysis (eye-tracking based) with 32 participants per scene. Correlating the features of each scene with visual attention constitutes the key method that we utilise in our analysis.

We hypothesise that the attentional performance on event segments reflects the influence exerted by multimodal cues on event segmentation and prediction, thereby enabling us to explicate the representational basis of events. The first results show trends of multiple viewing behaviours such as attentional synchrony, gaze pursuit and attentional saliency towards human faces.

Work is presently in progress, further investigating the role of visuospatial/auditory cues in high-level event perception, e.g., involving anticipatory gaze vis-a-vis event prediction. Applications and impact of this conceptual cognitive model and its behavioural outcomes are aplenty in domains such as (digital) narrative media design and social robotics.