The Sound of the Smell (and Taste) of My Shoes Too: Mapping the Senses Using Emotion as a Medium

This work discusses basic human senses: sight; sound; touch; taste; and smell; and the way in which it may be possible to compensate for lack of one, or more, of these by explicitly representing stimuli using the remaining senses. There may be many situations or scenarios where not all five of these base senses are being stimulated, either because of an optional restriction or deficit or because of a physical or sensory impairment such as loss of sight or touch sensation. Related to this there are other scenarios where sensory matching problems may occur. For example: a user immersed in a virtual environment may have a sense of smell from the real world that is unconnected to the virtual world. In particular, this paper is concerned with how sound can be used to compensate for the lack of other sensory stimulation and vice-versa. As a link is well established already between the visual, touch, and auditory systems, more attention is given to taste and smell, and their relationship with sound. This work presents theoretical concepts, largely oriented around mapping other sensory qualities to sound, based upon existing work in the literature and emerging technologies, to discuss where particular gaps currently exist, how emotion could be a medium to cross-modal representations, and how these might be addressed in future research. It is postulated that descriptive qualities, such as timbre or emotion, are currently the most viable routes for further study and that this may be later integrated with the wider body of research into sensory augmentation.