Högskolan i Skövde

In our current research project we develop a critical framework to facilitate the design and the critique of audio-based interactive narrative experiences. We foreground deep material connections to sound and affect as well as to the rhetorical and narrative properties of story/voice as a vehicle for storytelling. We draw inspiration for our framework from a range of philosophical, theoretical and aesthetic influences including posthumanism, new material feminism, affect studies, interaction design, as well as sound design and interactive and traditional narrative research. Our framework is intended to aid in the design of such audio-based artifacts to support embodied user experiences where affect is a key principle to support innovative material storytelling/deep listening with sound as the primary vehicle for delivery. To facilitate deeper understanding of the framework, we apply it to three original audio-based narratives we created to assess them preliminarily in terms of the four key areas we identify to understand affective narrative audio-based media: Temporality, Mediation, Interaction, and Embodiment/Material Experience. In future research we hope to expand and apply our model to other sound based works and to undergo more intensive user-testing. 

In this paper, we reflect on the design of an interactive audio-based digital narrative experience called Sisters. This work is designed for a single interactor, constructed as a mobile AR experience using graphic illustrations on a deck of player cards in connection with abstract audio activated by trigger images on the cards. An interactor is asked to cluster series of cards together into different abstract environments, based on sounds associated with each card, meant to represent spaces in an interior/exterior domestic site, a house and its immediate surroundings. The work conveys experiences of 4 family members in a complex abusive household, mediating between scenes of normalcy, love, companionship, and violence. The core focus of the work is to explore fragmented and very personal states of being and memories derived from an outsider’s perspective (the interactor), who co-experiences the complexities of the domestic spaces at a ‘safe’ distance, while also gaining empathy and affective connections to the characters. Connecting the content of the work and its fragmentary and elusive material audio and narrative design to our design model, the New Material/Spectral Morphology Model, we share how it may be used for aesthetic composition. Our model is based on feminist new material perspectives and foundational work from electroacoustic production and audio experimentation. Sisters extends our previous work with sound-based narrative, and we demonstrate how this work affirms our design strategies for novel interactive audio experiences. 

In this article, we reflect on the design and implementation of an interactive transhistorical and transmedial web-based digital narrative audio experience, PATTER(n)INGS: Apt 3B, 2020 that we developed in 2020. This work is an immersive audio-only application, and it focuses on the complex, material living conditions during the COVID-19 pandemic. Drawing inspiration from PATTER(n)INGS and its complex, material audio and narrative design, we propose a model for creating the content and delivery for similar sound-based interactive digital narratives. Our proposed model focuses primarily on the creative process for designing such sound-based work. To construct our analytical model, the New Material/Spectral Morphology Design Model (or NM/SM Design Model), we draw on theoretical influences from critical posthumanism, feminist new materialism and non-human narrative that critique notions of stable subjectivity as sites for power and authority over semiotic meaning-making. We combine these views with foundational theoretical research in electroacoustic musical composition notation, and audio experimentation that complicate notions of sound, sound making, spatial perception, psychoacoustic phenomena, and listening practices. Together, this theoretical/compositional framework provides a unique method to consider how one can sustain and maximize sonic agents as core phenomena to create anti-cognitive worlds and stories.

In Composing New Audio Worlds: transcription and transgression, Lissa Holloway-Attaway and Jamie Fawcus examine how in Insect Media, Jussi Parikka frames his discussion of ‘media as insect’ as a way to foreground media as more-than-technology and more-than-mediation. Insect media do not exist as a site between the natural world and the constructed, built, or human world. They are in/out of both at once.

In their paper, which Holloway-Attaway and Fawcus have submitted in the form of an audio walk, they explore/perform what it means to inhabit, discover, and become such an insect body, an effective mediator transcribing experience from a place of distributed non-human identity. Their focus is to explore alternative modes for engaging with critical theoretical models such as posthumanism, new materialism, non-humanism, and experimental electroacoustic music composition. These perspectives resist stable, cognitive subject identities for processing the world and its natural orders and rhythms, and create new conceptual spaces for composition and creation. The embodied design-states they imagine are formulated through generation of distributed agencies/embodiments, fragmentation (literary and sonic), affective acoustic space-making, and psychoacoustic manipulation.

Their audio content includes narrative voice but also incorporates psychoacoustic phenomena such as auditory brainwave entrainment, binaural beats, and the fragmentation/granulation of sound materials that decentralise and deconstruct the sounding world. They work to create resonant and reflective acoustic space, meant to be engaged by a single listener using headphones in solitude while in motion. Their aim is to ‘discuss’ the theoretical material focused on embodied non-human sites for mediation, while also helping to create emergent and resonant sound-space that re-orients the listener to experience the world in novel ways. They also hope to find new ways to consider academic engagement with complex theoretical models via creative, performative practice. How, they ask, can we be more-than when we compose and create? And how can the ear offer unique affordances to support novel engagement?

In our chapter, we reflect on our design of PATTER(N)INGS: Apt 3B, 2020, an interactive audio experience expressing some of the anxieties and challenges of living in and through 2020+ during a global pandemic in lockdown state. Our web-based audio application simulates a domestic space and its embodied inhabitants (human, non-human, and other) as encountered by a single user (or “eavesdropper”). The pandemic world we evoke is both specific and timeless, located and transhi- storical, in its remixing of literary materials and other sonic agents that destabilize fixed subject identities and rational cognitive states in favor of affective, ontological ones. We draw on theore- tical influences from critical posthumanism, feminist new materialism and non-human narrative as well work in electroacoustic musical composition and audio experimentation. We document our process of generative dynamic world-building and interactive digital storytelling as formulated though distributed agencies/embodiments, fragmentation (literary and sonic), affective acoustic space-making, and psychoacoustic manipulation.

In our performance, we re-create a live version of an interactive web-based audio experience we previously designed to explore being(s) suspended in the emerging “new normal” spaces, patterns and troubled times of 2020 COVID existence: “PATTER(n)INGS: APT 3B.” In our original web-based piece, users/listeners were encouraged to explore the many rooms of a virtual domestic space, that is Apt. 3B, in order to eavesdrop on personal and global responses to pandemics and lockdown scenarios, new and old. With no identifiable graphics, other than a full-screen black square, the eavesdropper-user was asked to wear headphones and could only move a computer mouse blindly across the flat surface of a desk in front of a blackened computer monitor. Hidden sound files moved and shifted and had to be discovered by the eavesdropper-user to participate in the experience. Our aim in the performance is to consider how to re-create this experience in a new kind of live platform.

In both our designs (web-based and live) we incorporate voice narration, psychoacoustic phenomena such as auditory brainwave entrainment, binaural beats, and fragmentation/granulation of sound materials that de-centralize and deconstruct the sounding world. The boundaries between music, field recording, sound art, sound assembly and live-ness are blurred, requiring them to be reinterpreted by listeners and performers alike. We draw creative inspiration from personal historical accounts of plague and disease narratives, combined with original texts, recordings and contemporary (2020) news reporting focused on global destruction, recovery, resistance, and homage. 

Our live sound spaces, a reflection of our web-based ones, are re-created through two performers, Holloway-Attaway (voice acting/text production) and Fawcus (live electronics and signal processing) and a COVID-compliant interactor moving through the performance space, replaying the role of the eavesdropper. Narrative voice, psychoacoustic sound, and electroacoustic music will be dispersed and digitally re-ordered through patterns and rules evolving simultaneously through the mutual configurations of the performers and from the interactor, who will be cued to respond independently. In this way we hope to move listeners between listening and making states, while exploring cross-references and viral connections across platforms in the plague years.

This is a web-based audio experience documenting our troubled times. Here, routines and intimate space-time proximities are tested.Constructed as a web-based interface, but with no identifiable graphics, other than a black square, the eavesdropper wears headphones (and ideally a face mask and blindfold) and may only move a computer mouse blindly across the flat surface of a desk in front of the blackened computer monitor. Hidden sound files, which also move and shift, must be discovered by the eavesdropper-user, who accesses them through a further limited sense of human touch, mediated through the technology of the mouse, a necessary prosthetic arbitrating social intimacy. The sound files shift and are layered to create the manifold and multiplicitous spaces of Apartment 3B. A restriction of visual stimulus gives the ear space to paint its own pictures, to distort, illuminate, and amplify. Paranoia and the imagination creep forward into daily life as the usually ignored sounds of our two-metre sphere become ever alive, deeply present in new configurations, patterns of recognition. 

In Insect Media, Jussi Parikka frames his discussion of ‘media as insect’ as a way to foreground media as more-than-technology and more-than-mediation. Insect media do not exist as a site between the natural world and the constructed, built, or human world. They do not negotiate duality. Instead such ill-conceived binary spaces converge in embodied forms of bestial, non-human, interconnectedness within living/lived spaces. Insect media do not interpret, describe, or represent realities for us (that is us humans). For Parikka, insect media are “a contraction of forces of the world, specific resonating milieus: internal milieus with their resonation, external milieus affording their rhythms as part of that resonation. ”What does it mean then to inhabit, discover, and become such an insect media body? How might such intensive states of being be revealed in the act of encountering, resonating with, and moving through embodied spaces? How might one be both inside and outside bodies, subject and other? In an age where transmission and infection bring fear of the ‘foreign body ’and its impacts, and where human bodies inscribe their devastating impact on the geological and atmospheric forces of the earth, what can we learn from becoming with such non-human, inhumane, bodies? What are their fluid, multimodal methods of cross-disciplinary trans-mission and how might we receive them? In our audio paper/audio walk, we explore what it is to inhabit these resonant spaces. Reflecting on theoretical models from posthuman, non-human, and more-than-human perspectives, we design a narrated audio experience that incorporates psychoacoustic phenomena such as auditory brainwave entrainment, binaural beats, and fragmentation/granulation of sound materials that de-centralize and deconstruct the sounding world. The boundaries between music, field recording, sound art and sound assembly are blurred and reinterpreted in the listening/explorative space. Spaces convolve, disperse and digitally re-order through patterns and rules evolving simultaneously through the sounding expanse, developed and mediated by the sounding, resonating space that emerges.