Audio-Haptic Description in Movies

Audio-Haptic Description in Movies

Audio-Haptic Description in Movies

August, 2011

Contact

Members

In literature, such as novels, an elaborate account of a scene, in terms of the location, ambience, and presence of characters, is presented prior or during the conversation of those characters in the scene. In movies, similar information is portrayed, but through the use of visual cues. For individuals who are blind or visually impaired, the visual content of a film is largely inaccessible, which makes its interpretation difficult without additional aids. To assist with visualization, an additional audio track of a narrator, who describes the scene and events, may be incorporated. These narrations are most commonly known as audio descriptions or descriptive video service (DVS). Audio descriptions are added to the audio track of a movie after it is completed to avoid overlap with the audio of the original movie, such as conversations, certain musical scores and sound effects. Given the limited duration available for audio descriptions, they are often stunted and abridged. An important visual cue that audio descriptions often fail to describe is the position of a character in terms of (1) his or her relative position on the screen; (2) relative distance from the camera; and (3) movements in front of the camera. Positional information is important for appreciating and interpreting a visual scene given the social interaction dynamics it can convey, as well as the structure of a scene in terms of where characters are located and how they are moving. To deliver this cue, we propose the use of a vibrotactile belt. Vibrotactile belts consist of vibration motors placed around the waist (in the form factor of a belt), and have been successfully used for applications where users require positional information. Moreover, as natural interactions, both with our environment and those around us, are multimodal, extending film to incorporate a haptic communication channel may complement audio descriptions in that more visual cues may become accessible, potentially providing a richer viewing experience. As part of future work, we are exploring haptic communication of other visual cues found in movies including scene changes and facial expressions of characters.

Members

Dr. Troy L. McDaniel

Dr. Troy L. McDaniel

Assistant Professor, The Polytechnic School; Director, HAPT-X Laboratory; Director, Center for Cognitive Ubiquitous Computing (CUbiC); PI, NSF-NRT grant program, Citizen-Centered Smart Cities and Smart Living

Dr. Sethuraman "Panch" Panchanathan

Dr. Sethuraman "Panch" Panchanathan

Director, National Science Foundation

Lakshmie Narayan Viswanathan

Lakshmie Narayan Viswanathan

Masters Student Researcher