Professional Documents
Culture Documents
3ry
ART AND THE
SEI{SES
ART AND THE
SEI{SES
Edited by
FRANCESCA BACCI
Researcher, Center for Mind/Brain Sciences,
University of Trento, Rovereto (TN), Italy
and
DAVID MELCHER
Associate Professor, Center for Mind/Brain Sciences,
University of Trento, Rovereto (TN), Italy
OXFORD
\JNIVERSITY PRESS
oxroRD
TTNIVERSITY PRESS
Great Clarendon Street, Oxford OX2 6DP
University of Oxford'
Oxford University Press is a department of the
rft. u"iversity's objective of excellence in research' scholarship'
l,-ñrifr".r
and education by publishing worldwide
in
Oxford New York
Auckland Cape Town Dar es Salaam Hong
Kong Karachi
i;;i;;;-p"; Madrid Melbourne Mexico citv Natuobi
::--¿-";i-¿-i;i:':-,
::!
- -- -*L
Ll1:i _-
- -'--
FonBwoRD
SIAN Eos
A book devoted to the Art and Science of anlthing immediately suggests a binary
attitude to the way we consider the workings of the world and of ourselves in it. The
categorizing of the five senses suggests a further attempt to take taxonomizing to
possibly reductive extremes. The 'Art and the Senses' symposium, held in Oxford in
zoo6, examined those interfaces between art and science and between the senses, and,
because a1l the speakers were exquisitely thoughtful academics, the level of interaction
and the complexity of the discourse was profound, with each side exceptionallywilling
to listen to the other perspective and even subsequently announcing new approaches
in forthcoming research. In this paper I look briefly at the way we communicate our
experience ofthe senses and at the poetic language necessaryto convey something of
the felt physiological experience.
The experience of actually living in the world is multiply layered. We translate the
input from discrete sense organs into a blur of embodied sensation and, because we
are more than unself-conscious diatoms or zoophytes, we bring or 'bind' a range of
associations acquired from memory to make sense of the prelinguistic experience,
translating it more reflectively into what emerges as thought, and eventually even into
self-conscious awareness of thought-and thence to the pronouncements we make in
art history and scientific discourses. In humans, language is the key to this translation
and by language I include art, besides the spoken and written word, symbolic systems
where meaning is attributed to forms not literally associated with their referents. This
is so even in science. The palaeontologist Steven Mithen has created a hlpothetical
framework for explaining the emergence of the human mind in laler Homo sapiens
(Mithen 1996). Over millions of years we developed four kinds of intelligence, he
suggests-a natural history intelligence, a technical intelligence, a social intelligence,
and more recently, the capacity for language. At some point in the evolution of the
mind these separate facilities began to interact. The evolutionarypsychologist Nicholas
Humphrey proposes that the significant factor in an increased cognitive fluidity was
the development of a capability to see into the minds of others, a consequence of
advanced social intelligence. This internalized the communication processes which
had taken place between individuals in a social group in the outside world and self-
reflexively enabled the different domains in the individual's private mind to 'talk to'
each other. An increasing facility with language may have been the critical factor: social
Vi FOREWORD
ofthe seventeend
non-social world available for consciousness to explore, with a consequently rich
the syrnbols of rtr
expansion of connective ideas (Mithen rgq6). The ability to use language, both as a
a cascade ofasso*
foim of external social communication and as a means of internal cognitive thought-
In addressi¡e
processing coincided with the compulsion to make art-genera\ thought to have
reader's brain at
tccurred between 30 ooo to 6o ooo years ago (although new archaeological finds keep
'millions of cells
pushing the dates back). Our responses to objects in the world, then, are processed
thtoogtr a sequence which begins with the sensory stimulus, are made sense of through
can cross i¡to th,
purely the tech¡
an internalized top-down/bottom-up feedback loop and may end with intelligent
are unavoidabtre-
awareness in a kind of prelinguistic state, followed by self-conscious reflective thought,
loaded. 'Cells'th,
expressions both internalizedandcommunicated to others through language and art'
scope and remi_n,
ihi, pro..r, has been beautifully illustrated in an essay by the writer and critic
their microscop'e
A. S. Byatt in The Cambridge Companion to John Donne (Byatt zoo6). In considering
to describe the pr
the impact of this great seventeenth-century metaphysical poet, Byatt points out that
her pleasure in Donne's complex poetic conceits comes not so much from their stimu-
lus to her senses, as from the effort of the mental activity required to experience the full
intellectualforce ofthe metaphorical constructs. Metaphysical poets, she says' 'describe
not images but image-making, nol sensations but the process of sensing, not concepts
but the ideaof therelations of concepts" [my italics]
On the one hand, Byatt's statement is surprising. Art' and' famously, Donne's
ability to give a vigorous representation ofthe sensually, not to say, erotically charged'
certainly directly communicate to our physical senses-and there are many equiva-
lents in classical and romantic art, and even, too' in abstract, non-representational
works which contain hints and disruptions. Why else do we sometimes feel that
shiver up the spine, that almostpalpable sense of 'qualia', the sense of being almost
in the here and now, almostbeingable to smell the rose, feel the arrows, cry the tears?
In a contemporary visual equivalent, iane Wildgoose presents a sensuous arrange-
ment of objects with the heightened clarity afforded by a skilful use of digital photog-
raphywhich has an immediate physical appeal. Art isn't the real thing, but its sensuous
imagery triggers a deeply felt physical response which stimulates the mind through
the emotions, almost below the level of consciousness.
But Byatt takes this physical response to art yet further and analyses the waythe mind
reconstructs our physical responses ínto mental imagery, emphasizing that the more
ioniflexttie metaphorical connections, the more deeply pleasurable they are. They are
ai their most exciting when inour minds we reconstrucl new images, making newlinks
b.yondihose alreadylaid down in memory' forming a brand-new conrlection, a con-
'Cept; oi 'conceit'. All the best artists, whether visual or poetic, invent newways of seeing,
fact that she is playing on imagery familiar in art history, from the vqnitas paintings
of the seventeenth century, which she has particularly studied (Fig. r), and also through
the syrnbols of death familiar to us from Victorian funeral ritual, all of which set up
a cascade of association-making.
In addressing the effect of Donne's disparate image creation, Byatt pictures the
reader's brain at work in bringing many layers of understanding together, describing
'millions of cells with connecting dendrites and long questing axons, some of which
can cross into the opposite hemisphere of the brain.'At first sight, this appears to be
purely the technical language of science-'cells', 'axons', 'brain'-but metaphors
are unavoidable. Her language is both physically representational and intellectually
loaded. 'cells' themselves were so-named by Robert Hooke peering down his micro-
scope and reminded of monks' separate enclaves. I'm not sure whether scientists at
their microscopes would use Byatt's anthropomorphized active participle 'questing'
to describe the presumably flailing axons, but that seems exactly what they are doing
Jifterent s'enses, Fig. 1 Jane Wildgoose, Vanitas Still Life at the Wildgoose Memorial Library,2OOS.
- -. r 1-.c
(See Colour plate 1)
::-.. n..nl rhought. The
-, -;: knorrledge of the (photo: courtesy of the artist)
vlil FOREWORD
in her description. And 'hemisphere' is not quite technically or semantically correct In unpackine t
if one considers the two walnut halves of the wrinkled cortex, but has a hint of associa- .one
or multiple >,
tion with that the other sphere we know, the earth itself-the brain/the world both lzing it as a sr.r¡rt
forming complete circles containing all we know. It is hard to escape poetic conceits then re-creati¡g ¿
even in science. is the consequen,
The word 'dendrite' refers to a mineral crystallization in a branching or tree-like juxtaposition and
form inside another mineral bearing such a crystal formation, in a nerve-cell' the fine talking to a phr-,
branch of a dendron. 'Dendrite' comes from New Latin, from the Greek for dendron solving an equati
or tree. The ramifications (literally) of this image are beautifully brought out in V. S. Ramachand
Andrew Carnie's artwork Magic Foresf, in which the viewer wanders through an a finite set of rule
ethereal woodland of lacy winter trees, at once dreamlike and familiar, and suffused an unfamiliar ohí
with the kind of poignant evanescence that is indicative of the artist's indiüdual style givén a senson- r(
(Fig. z). The 'trees' are actually images of living brain cells viewed through a laser- ioñiióliemotior
scanning confocal microscope, drawn with the aid of computer-imaging techniques' éxperience if rre i
stained with fluorescent dyes, and projected on to layers of fine fabric. We can sensu- internal image-m
ously enjoy the experience but also relish a new knowledge of the scientific processes All this conne(
and combine the two-our sensuous brain, our thinking brain operating together, senses in their ori
flickering from thought to sensation and back, to widen our understanding. artwe do not thi¡
This is cleverh-
sensual processin,
tled 'Seeing' and
Steve Smith and
imaging) digital c
looking and lister
processes into cr--rr
puter informatio
technology into a
could be seen anc
ding them in sol:
split-second imae
representationall.
own saccadic e'*'c
materials, so the i
reflecting the hr-t
on many ler.eis. rl
concept-rr-e ieel.
art-science-meiap
It is salutan- ti
through the rvork
r semantically correct In unpacking the multiple layers of image-making-perceiving the object through
ri h¿-' a hi¡t of associa- one or multiple sbnses, feeling it physically with an implicit potential to ac, internal-
br¿in the rvorld both ízingit as a symbol or metaphor, bringing remembered associations into play and
<:r¿pe poetic conceits then re-creating a new concept or conceit-I would further surmise that the pleasure -"
is the consequence of a release of endorphinl, a small reward for seeing an unusual
b;anching or tree-like juxtaposition and then being creative with it. This occurred to me recentlywhen I was
: ¿ nen-e-cell, the fine talking to a physicist who was describing the immense satisfaction he felt when
ne Greek tbr dendron solving an equation. And it ties in with research undertaken by the neuroscientist
ai¿li-, brought out in V. S. Ramachandran, who, in attempting to systemize (rather in vain, I tend to think)
: ',t ¿-r"trers through an a finite set of rules for what he calls 'neuroaesthetics', explains that when we identifi'
i¿niiiar. and suffused an unfamiliar object from background noise (essential in crude survival iéims) we áre
¿11;:'s indir.idual style given a sensory reward, because perceptual processes link to the Iimbic system which
:e-rreú úrough a laser- ¿tnirols em*i-o-1" (Ramáchandian and girstein 1999). How múch richer is this
.:- -;l1aging techniques, experience if we now consider the extra intellectual pleasure we gain from the act of
=
f¡b'rr¡. \\'e can sensu- internal image-making itself and our self-conscious awareness of this process?
---¡ scientihc processes All this connection-making seems rather dense when we come to regard the five
i:n L':er¿tixg together, senses in their original Aristotlean separateness, because in our experience of life and
r,Jer.:anJing. art we do not think and feel in a modular way.
This is cleverly illustrated in a work that looks quizzically at the material science of
sensual processing and turns it into art resonant with many meanings. In projects enti-
tled 'Seeing' and 'Hearing' the artist Annie Cattrell worked with neuroscientists
Steve Smith and Mark L¡hgoe to capture fMRI (functional magnetic resonance
imaging) digital data as brain images relayed while subjects were caught in the act of
looking and listening, and then used Rapid Prototlping to transform these isolated
processes into computerized ürtual models (Fig. ¡). Using three-dimensional com-
puter information, stereolithography, the images were transferred through laser
technology into a variety of different materials, such as wax, resin, and nylon, so they
could be seen and felt as 'real'. Out of these she made waxy resin sculptures, embed-
ding them in solid square 'brain-boxes' made of transparent hot-cure resin. These
split-second images of the brain in the act of seeing and hearing are more than simply
representational. They are the impenetrable made visual and tactile. As the viewer's
own saccadic eye movements shift about the light and shine of the transparent
materials, so the images continually change position and shape, the hi-tech imaging
reflecting the hi-tech imaging of the observer's brain.' Here is a work that operates
on many levels, then, using technology to yoke a sensual response to an intellectual
concept-we feel, imagine, think, and learn at the same time-a rewarding piece of
art-sciencámei;thñi¿4.'
It is salutary to realize that one of the best ways of understanding the senses is
through the work undertaken by neurologists and psychologists on people who have
' Head On: art with the braín in mind, catalogue for Wellcome Trust exhibition at the Science
\fuseum, (London, The Wellcome Trust, zooz) curated by Caterina Albano, Ken Arnold, and
: Colour \'larina Wallace p. 38. Dr Mark L¡hgoe assisted by suggesting ways of using the technology to
realize the concept; Dr Steve Smith provided the fMRI scans, 3D Systems the Rapid Prototlping,
:':;, :=:': .r:isi) and Hobarts the encapsulation of the models.
FOREWORD
This is a gLr.;
applying a'tLr:-i
brought into p,i
perceptions ir ,_,:
Kuusisto grre .
His experien:.
focused arouni ¡
continuous s1i;.
from the retinopathy of immaturity and also nystagmus, 'darting eyes'' He is not voices in it. St-,;::= -,
completely blind but sees the world through a blur of colour and motion' His book' that you r-asu¡l',' :.
The planet of the Blind,describes his refusal to acknowledge his disability and his
risk-
sense ofa cockt¡ r :
taking struggle to maintain a normal life from childhood through to adulthood, even cruising com¡¿:, - :
ridinj a bicycle and negotiating new cities on his own (Kuusisto rgqS). The book is as Where'd I pur -l:
much about failure and stubbornness as it is about depicting the world of the visually
_
remember. .
impaired, but it leaves a strong sensuous impression on the reader. Kuusisto is famil-
.
iar with the visual arts through sporadic close-up studies of books, or through visiting
FOREWORD
galieries with friends, and it is interesting how he speaks of his world-view in terms of
paintings:
Faces loom, adults are rising from the vista like sensate stones. These are gorgeous mirages
of not-seeing, moments red and green and black as the studios of Matisse. The moments are
seriocomic: a white horse stands at the circus, no, it's a bed-sheet in the wind . . .
(Kuusisto r998)
Dear fackson Pollock, I've entered your Autumn Rhythm. The irregular or sometimes certain
flight of colour and shape is a wild skein, a tassei of sudden blue here, a wash of red. The
very air has turned to hand-blown giass with its imperfect bubbles of amethyst or hazel blue.
I stand on the ordinary street corner as if I've awakened at the bottom of a stemware vase. The
glassblower's molten rose has landed in my eyes . . .
His experience of the world also involves a compound of other sensory intelligences
focused around spatial orientation:
Delicate, skinny, inordinately active, I was sharpening a sixth sense that fostered the impres-
: ::,-:::, :':'e artlst) sion in my parents that I could see far better than I really could. Such acting requires a capa-
cious memory; in the gaudy nets of pastel colours where I lived, every inch of terrain had to
be accurately remembered. In the heart of every blooming and buzzing confusion, I found
a signpost, something to guide me back along my untutored path. . . I suppose this plum-
:.i::h: into the l'orkings meting through the world involves the same inexplicable faith known to skydivers. Fast blind
-:::::lule the language people have exceptional memories and superior spatial orientation. By the age of five, I was
':: : ..: ;lagc of science. a dynamo. Wanting to see me run, my mother saw me run and guessed that I must be seeing
: *:::.',' Sescribing visual' more than I really could. And so I landed iike the bee who sees poorly but understands destina-
- - ::icn.cs rthich rever- tion by motion and light and temperature.
:. :::r : gination. Literary
The brain quickly reads sensory stimuli and brings ready-made meanings to the fore
:;: L¡: the senses, because to aissist inttrpiéiátioñ. The desiie to make sense ofthiñgs seéms uiiavoidable, so much
!-: :s 13-e\-ocations are so that given a few hints, we can imagine things that are not actually there. An auditory
.: :,-- ;r-rnvev exactly what
equivalent of this top-down/bottom-up processing is amusingly described in f onathan
i¿--;:t rn one sense relies
Raban's sailing book, Coasting
p:irraiureh' and suffers The engine, the engine. Its thump and clatter, all mixed up with the smell of diesel oil and the
d::ring eles'. He is not continuous slight motion of the sea, is so regular and monotonous that you keep on hearing
voices in it. Sometimes when the revs are low, there's a man under the boards reciting poems
: :rJ n.lotion. His book,
that you vaguely remember in a resonant bass. Sometimes the noise rises to the bright non-
-s ¡isabilin'and his risk- sense of a cocktail party in the flat downstairs. At present, though, you're stuck with your usual
¡ uü to adulthood, even cruising companion at sixteen hundred revs, an indignant old fool grumbling in the cellar.
..:: 1998 ). The book is as
\r'here'd I put it? Can't remember. Gerroff you, blast and damn you. Where'd I put it? Can't
-:¡ irorld of the visually remember. . .
';:ier. Kuusisto is famil-
(Raban rq86)
,: ¡,L:. or through visiting
xl1 FOREWORD
have arranged the beads. Miss Sullivan touched my forehead and spelled with decided particular places
emphasis, 'Think.' The smell of a ilü
In a flash I knew that the word was the name of the process that was going on in my head. That a tobacco factor ]
was my first conscious perception of an abstract idea. lemonade OOR
a pub SMOLBEE
We can how the senses are a route not simply to an animal sensation
see here
but form part of a chain of intelligent and self-aware construction of the out-
side world.
The artist Alexa Wright effectively illustrates the brain at work where the body
sense ought effectively to be absent altogether in a project she undertook with neuro- ' Wellcome Tm¡;
scientists and patients with phantom limb syndrome, which occurs where people who zooz: also conters¿:l
FOREWORD xlll
:-,'-: ¿t the age of two, have had amputations continue to experience sensations in the non-existent limb.'
Neurologists understand that sensations arise as a result of a dynamic plasticity in
the brain which allows it to re-map bodily awareness, often very soon after the limb
;:,: r'-hite darkness shut
has been removed. Areas on the cortex which formerly received sensory input from the
i :::: shore rvith plummet
: -, ia:¡en? I rvas iike the amputated limb continue to be activated from parts of the brain on adjacent areas of
-:i::g-iine, and had no the cortex linked to parts of the body surface close to the amputated limb, so it feels
as if the limb is still there, though not always in a normal state. 'One man had a phan-
(Keller r9o3) tom arm fixed at right angles which he had to accommodate when passing through
. a door,' writes neuro-psychologist Peter Halligan, who with his colleague John Kew
-: ire lost ship to give a worked with Wright, 'another was plagued by his phantom arm floating up through
- ¡ :¿r situation is more
the bedclothes when he was trying to sleep.'
: :e: teelilgs. Her first
The subjects of Wright's photographs were gratified to see their non-existent sensa-
;. ¡:e*through. Her tions made visible, and the neurologists claimed that this helped them to be more
.'-¡r: b','getting her to
therapeutically accurate. Exhibited in conventional gallery spaces as art, the images are
-,¿
moving because of their paradoxical ordinariness, enabling viewers to share an inti-
:.:;¡ ,:l the honeysuckle mate identification with the subjects' unconventional self-perceptions (Figs a and 5).
:.: :,¡;ed mv hand under That the brain can compensate for loss, even if in strange ways, that it can continue
---.. -,-j--:r the svord water,
to alter and re-programme itself, is somehow both reassuring and surprising too.
-r: :-:r-,i!rr of her fingers. But though we are talking here of actual material plasticity with real change in brain
::-- ,-: re,:urning thought;
tissue and the brain's capacity to invent and reinvent experience, it almost explains the
-i: -jl:i',.--a-t-e-f' meant
flexibility and scope of the imagination.
In a work shown at Liverpool Biennial 2006 called Outsidein, the Norwegian artist
r::. :::r: ear-e birth to a
Sissel Tolaas walked with Liverpuddlians from the north, west, south, and east
of the city, and using high-end technology collected smells from the streets. She then
1 :r understanding of took the smells back to her laboratory in Berlin where she used micro-capsulation to
::: l: her understand- infuse micro-eggs with oils, which respond to touch or friction to be diffused into the
:::-:::rl the nature o[ atmosphere. This might be sensuous enough but Tolaas recognized the need to
stimulate a higher, more introspective smell awareness through the use of a language
which could be communicated further. For this part of the project she worked with
- -.::¿ be¿ds, three small
linguist Andrew Hamer, who specializes in Liverpool's Scouse dialect, and asked the
::,-:nted them out again
:l::::r ihe sequence and participants to describe the smells of their city. The resulting vocabulary is delightfully
,: t.. nin: horv I should pungent, funny, and somehow accurately suggestive of Liverpuddlian character and of
::: .¡elied rt'ith decided particular places in the city:
The smell of a filthy channel, for example is MOODO
n-,-ri on i¡r mr- head. That
a tobacco factory NEEK
lemonade OOR
:'-, ¿n animal sensation a pub SMOUBEE
: rrlLJrion of the out-
rubbish is BIISH
leather SKENN beyond analogi,'
take-away food TCKOO original source Ltr
senses. This short pathway has direct links to the limbic system, which deals with vigi-
lance and emotions, and there are also links to the hippocampus where memory r Marcel Pruri::. ,
is stored. However, there is no contact with the parts of the brain responsible for described r¡ Ch;::":
language or speech. This might explain why we have a limited vocabulary for smeli as Yott Get Oli¿" :'
FOREWORD
beyond analogy (smells 'like') and why, like Proust, we often find it hard to locate the
original source of a sudden waft of familiar perfume.3
Proust is a master in evoking the sensuous filtered through the intellect, so the plea-
sure is doubled. But it takes him four pages of close introspection to chase the original
stimulus and the ripples it then sets up.
Here is Proust's initial moment:
,----jil,l-t iettrring to the
I raised to my lips a spoonful of the tea in which I had soaked a morsel of the cake. No sooner
- - :¿a l¡ i''hich had been had the warm liquid mixed with the crumbs touched my palate than a shiver ran through me
..;isrone tbr explaining and I stopped, intent upon the extraordinary thing that was happening to me. An exquisite
,'i .-n memorv, evoking pleasure had invaded my senses, something isolated, detached, with no suggestion of its origin.
r::lc :e nse of time past. -{¡d at once the vicissitudes of life had become indifferent to me, its disasters innocuous, its
¡e:iLrrs transmit signals brevity illusory - this new sensation having had the effect, which love has, of filling me with
!.ss s1-stem ofany ofthe a precious essence or rather this essence was not in me, it wasme.
He goes on to remark:
onedoesn,thavetobediagnosedas.synaesthetictorecognizehowtheinputfrom associa-
andis-mádeiense of tñróügh multiple
different sensual stimuli quicklyblurs to rmmldiate-
metaphor there')
tion. Language is second-ha1d (note.the bodily lhe
nqe'
pir.r*T.?fli,i"uUliii t !l io mb ine an intelle ctual and gr e
lJivé?-ésp
o
or imag-
--Tñ-coñidéring simultarrt oorly the Art and science of the senses, the felt
the
ined experience and the material
pro."rr., ir,oolved, our minds cannot reconcile
together physical
bot th.r. i, pl"uror. in blending
ambiguity of such "'j;;;n;-u.h,
understandingandani,,t.lt..t,,ulupp,".i*io,,ofhowitworks.Theheightened
the second-hand vigour of
language ofpoetry dú' exploration' U'i"gi"g together
and analogies to
""i;
sensuons ."p.ri"n." *i r, un
inr"li.ctual plásure in making links
deep for words'
.ommunicate feelings normally too
RnrsnBNcns
The
Feeling thought Donne and
the embodied mind' In A Guibbory Gd')
t*;"6). Donle'll :+z:,tit:.T1,:idge University Press'
Cambridge'
Cambridge Comparuon to lohn
Life' Doubleday' New York'
Keller H (tgol). The Story of my
FOREWORD
:-. :':.iust takes a couple Kuusisto S (rqq8). The Planet ofthe Blind. Faber and Faber, London.
,:-: ::lemorv still elusive. Mithen S (rqq6). The Prehistory of the Mind. Thames and Hudson Ltd, London.
. r :rl ;11¿ilttain a con- Raban J Q986). Coasting. Collins Harvill, London.
Ramachandran VS and Hirstein W (rSSS).The science of art: a neurological theory of aesthetic
..:-- : r(\t b\ dlstraCtlng
experience. J ournal of Consciousness Studies, 6-7, 15-51.
- - :: sensation in terms
:;e oi madeleine
J-pping it first in
Editors' Intr
1. Making Sen
Fn¡xcrsc-r
Acnrrrr Bc
) The Science
Nrcsor-rs T
J. The,{,rt ot I
Gr,n-l.rrrxr
+. The Ifultise
Csenrrs Sr
). Aesthetic l[r
Roslrrs D
6. Rendering t
Erro Fn rsz
Sculprure a
Fn-rscrsc-{,
E. Touch an*j
Jrssrrrn \l
9. Hearing Sc.
\fultimod¿
D-rvlo Ho't
CoNTENT S
Foreword V
Editors' Introduction 1
Cn,tnrBs Sp¡Ncn
Euo FnaNzrNr
10. The Science of Taste and Smell 183 22. Mirror Ner
Trm Jecor Vrrronro (
11. 'Sound Bites': Auditory Contributions to the Perception and 23. Pictorial -d
Consumption of Food and Drink 207 Blind Pain:
CnenrBs SpsNcn, Mave U. SH¡NreR) AND HBsroN AnrrnArrr
BrurrpNttr¡r Arv¡Ro P-,
395
lndex 6t9
435
Amir Amedi Ph
of ferusalem, fen
Francesca Bacci
Palazzo Fedrigot
fennifer M. Bark
State Universin-.
Heston Blumend
Beal'riz Calvo-\l
Universidad Ctrr
IJniversity Lonoi
Patrick Cavanag
University, 3-r Ki:
Rosalyn Driscoll
Ivar Hagendoon
Patrick Haggarr
Alexandra Hous,
Paul Hekkert IL
CE Delft, Nether
David Hon'es t
Maisonneuve. br
Timfacob Scho
Avenue, Cardid
Geraldine A. Joh
OXr rPT, UK
Lrsr or CoNTRTBUToRS
Heston Blumenthal The Fat Duck Restaurant, High Street, Bra¡ Berks SL6 2AQ, UK
Carol L. Krumhansl Department of Psychology, Cornell University, zr4 Uris Hall, Cretien van Car
Ithaca NY 14853-76or, USA The Netherland.
Fred Lerdahl Department of Music, Columbia University, 6zr Dodge Hall, z96o Nicholas J. \\-ac
Broadway, New York, NY roozT USA Dundee DDr +H
GuyMadison Department of Psychology, Social Sciences Building, Umeá University, famie Ward Sctr
Laura U. Marks Art and Culture Studies, Simon Fraser University, 8888 University Massimiliano Z¿
Drive, Burnaby BC, Canada V5A 156 Bettini 31, Ror er,
Lofti B. Merabet Laboratory for Magnetic Brain Stimulation, Beth Israel Deaconess
Hospital, Department of Neurology,33o Brookline Avenue, Boston MA ozzr5, USA
Vik Muniz 47 Lexington Avenue, New York, NY u238, IJSA
Noa Tal Physiology Department, Faculty of Medicine, PO Box rzz7z, The Hebrew
University of ferusalem, Ierusalem 9rzzo, Israel
LIST OF CONTRIBI]TORS XXV
-:--t,.-
: '.1:;lonald Harrison
:i.:. C¿nada
---:r.i'e:sin.
of Oxford,
-
-- -:' \ lcdia, Birkbeck
-:H,-,PD
--::-. ¡rsirl' of Oxford,
world, as well ¿s
ffi{r., in terrns
it yields,fttea--h
-44etrd1"'3
ourcultu
$ltory,
In this book, as i¡
from the other ser
iust as there ar
the senses. The ar
investigating the
of these different
ists, scientists. a¡l
for multi- and rn
tant, the divergen
period and from ,
This collection
Oxford, Englan-1
have been uritcer
disciplinan irorl
unusually open ar
across the variou:
the same volume
gist, musicians. v
have easily resutrt,
there are commo
new connections
EDITORS'
INTRODUCTION
THB senses are our source for vital knowledge about the objects and events in the
world, as well as for insights into our private sensations and feelings. F,ach sense is
ür*iq.t", in terms of subjective feelings and regarding the array of information that
it flelds,]et each sense interacts with the others to p-roduce experiencg¡ that cannot
Erydg9.-.-!*t-%irq&.r¿¡!€.-In addition, our sensorT qglcgption reflects our personal
history, our culture, and the way that our brain has learned to combine sensory inputs.
In this book, as in life, no sense is privileged over another. No sense exists in isolation
from the other senses, the body, and one's personal experience in a specific culture.
Iust there are multiple senses, there are also multiple approaches to the study of
as
the senses. The arts, humanities, and the sciences provide different methodologies for
investigating the world and our relationship to it. In order to reflect the contribution
of these different fields, this book includes an approximately equal number of art-
ists, scientists, and humanist scholars. These essays provide a compelling argument
for multi- and interdisciplinary research. Both the convergence and, equally impor-
tant, the divergence found in these essays provide a snapshot-from a particular time
period and from specific points of view-of sensory research in each discipline.
This collection of essays explores some of the issues raised in a zoo6 workshop in
Oxford, England, entitled 'Art and the Senses'. Approximately half of the chapters
have been written by speakers or participants at that workshop. While not all inter-
disciplinary workshops are successful, the participants at the Oxford meeting were
unusually open and engaged in learning together about the issues of common interest
across the various disciplines. This book builds on that openness, daring to include in
the same volume essays by scientists, architects, art historians, a cuitural anthropolo-
gist, musicians, visual artists, a chel a choreographer, and many others. This could
have easily resulted in a confused mess. Instead, we hope that the reader will see that
there are common threads that weave in and out of many of the essays and will find
new connections beyond those envisaged by the contributors ofthis book. The aim
EDTTORS- INTRODUCTION
of this project, in addition to providing a unique multidisciplinary resource on the is no longer srr,
senses, is to inspire future cross-boundary interactions. As we hope these essays make would still lead to
clear, multidisciplinarity, like multisensoriality, can offer a richer perception than The focused etto:
that available from any single point of view. I_nter_national \lu
It is impossible, in one volume, to fully do justice to both the breadth and depth of Hqndbooko.f ''fr'i
sensory studies. Instead, this book offers a collection of examples on how scientists, the way that üe :,
artists, and scholars in the humanities are investigating and commenting on the senses. A second then
Every book is necessarily limited in scope, and this is no exception. Perhaps foremost is ways, the nature r
the lack of a clear definition of 'art', which means that different contributors will bring ngl!-qf qur pers,-
different conceptions of what is or is not art, so that certain boundaries in scope are emergence of the'
necessarily arbitrary. Undoubtedly, there are unavoidable exclusions in this collection, perceived flan in
such as the limited coverage of literature (addressed only in Sián Ede's preface). Any Johnson r9991 \\-t
exclusion, in terms of subject matter or methodology, should not be interpreted as a cognition that go
theoretical statement about art or the senses in general, but as a pragmatic editorial or an abstract. sr:
choice. In the case of literature, for example, there are aheady numerous resources movement rrithi¡
exploring the topic of literary representations of perceptions. Starting with the semi- A third mart r
nal statement by Horatio, ut pictura, poésis, history is replete with theories of the ing in shaping tlt
ongoing debate on the relationship between the written and spoken word and its in the humanite:
representation, all the way to Lessing, Nietsche, and to contemporary scholars such as period, and culru
W.I.T.Mitchell. in shaping our s:l
experience, is rec
the importance t-.
actively used b,,- :r
ry7zbv art histLr:.
THEMEs tion of .,'isual ar¡s
\re see is influen.
-CouvroN underpinned bt t
During the 'Art and the Senses' workshop, a number of common themes that cut t-leübilin'-i¡ te-
across the disciplines emerged. These connections surfaced despite many obstacles, strengths ofthe h
such as significant differences in how certain terms are used across academic fields, A fourth the:r
and became the focus point of discussions and, in several cases, interdisciplinary col- cussed bv man',' ¡
laborations which continued beyond the final day of the workshop. We offer a few of
iI-4, to guide a;
these themes here, but hope that readers will discover new connections and ideas from -\¡t can pror-ide ;
their own personal reading of this collection. :r.on:s_$e¡qglr es,
The first common theme, which motivated the initial organization of the work- ¿ctiYe and ternF,a l
shop, was a perceived consonance across various fields in reconsidering the value ofthe of the deveL',F¡r¡
body and the senses of taste, touch, and smell. Contemporary art practice has already Smell percentlc::
explored and exploited a multisensory approach. Art history, on the other hand, has .,',hile taste requi:
largely tended towards privileging the visual, both thiough the more recent inclu- -:-lterplav bell,, e:r
slori of 'visual culture' as a field of study and through its traditional, if criticized, ¿nists such a: "'u-:.
oiularcentic ápprqach. Among the humanistic disciplines, s-ocial sciences, critical i-rsual recognitl.¡:
studies and anthropology have been open to a diverse approach, which include a ¡rterience rnore ,
sénsoiy-oriented reading of their subject (see, for example, The Empíi'e of the Senses: -lar 'art tor ¿r:'s r
The Sensual Culture Reader, edited by David Howes). In science as well, perception rrough the erlle:
EDITORS INTRODUCTION
.::-::tan- resource on the is no longer synonymous with vision, even if reading most psychology textbooks
- :Ll:e these essays make rvould still lead to the idea that humans have onlyeyes or, occasionally, little-used ears.
. :i.her perception than The focused effort to investigate multisensory perception is eüdenced by birth of the
International Multisensory Research Forum (IMRF) inry99 and the publishing of The
---,. i-re¿dth and depth of Handbook of Multisensory Processesin2o04. Such research has led to a major revision in
::::,es on how scientists, the way that the senses are conceived of and studied in the mind/brain sciences.
:::entrng on the senses. A second theme is embodiment. Contemporary artists are exploring, in various
:. ... Perhaps foremost is rvays, the nature of the bod¡as a physical object, a cultural construct, and as a compo-
,: ; :'rt¡ibutors will bring nent of our personal self.Cognitive psychology and n.uroscience have also seen the
: .,;rdaries in scope are emergence of theories that challenge, via an 'embodied' and/or'situated' cognition, a
, *:-,-,ls in this collection, perceived flaw in traditional cognitive science (for reviews, see Clark ry97;Lakoff and
>-:: lde's Preface). Any Johnson 1999; Weiss and Haber 1999). Currently, scientists are considering aspects of
: :, r: be interpreted as a cognition that go beyond earlier metaphors of a'brain in avat' (no body, no emotions)
:: r :raqmatic editorial or an abstract, symbolic system for computation. Many of the essays here reflect this
::., :-Llnterous resources movement within the cognitive sciences.
. i.:ri:rg rtith the semi- A third major theme, across the collection, is the role of learning and experienc-
,-:. -,ittr theories of the rng in shaping the way that people use their senses. This is not a particularly new idea
:,: s;r.ken rvord and its in the humanities, which unfailingly start their investigations from the specific place,
:lf L-:irr- scholars such as period, and culture ofpertinence, but in the cognitive sciences the role ofexperience
in shaping our sensory perception, as opposed to the role ofour senses in shaping our
erperience, is receiving increasing interest. Many of the essays in this book examine
the importance of flexibility and experience in influencing the way that the senses are
actively used by individuals in a particular cultural context. The concept pioneered in
ry7zby art historian Michael Baxandall, who discussed the reception and interpreta-
tion of visual arts as affected by what he called the 'period eye', summarizes how what
\{e see is influenced by context and previous experiences, an intuition now further
underpinned by the result of the scientific studies mentioned earlier. One can view this
, :rlLr:r themes that cut tlexibility-in terms of sensitivity to social cues and experience-as one of the great
::.pite ntan,v obstacles, strengths of the human mind.
r :-:lrss academic fields, A fourth theme is the role of art in making us more aware of our senses. As dis-
¡¡.. :l:erdisciplinary col- ;ussed by manyof the authors, sensory perception is typically used as a means to an-
",,,_-
-,¡. \\'e ofrer a few of 9,$-to guide action, to recognizé ótjeats, to enable social interaction, and so on.'
:.:r.¡:ions and ideas from -\rt can provide a context-a defined space and time-i¡ ytrrch to sense our sensa-
rions them-selves, rather than just the object of the sensations. Touch, given its inter-.
-.::t,¡¿llon of the work- active and temporally extended nature, is one sense that lends itself to the experience
: :r -e:ing the value of the of the development of sensation-in-time, but the other senses can be active as well.
-, ::, r:actice has already Smell perception incorporates the active seeking (sniffing) of odorous molecules,
,. :r ::e other hand, has ,,rhile taste requires putting something in our mouth. Auditory art, in particular the
: i¡¡ rore recent inclu- interplaybetween performer/composer and audience, evolves over shared time. Visual
.::.r:ional, if criticized, artists such as Vik Muniz deliberately present challenges and ambiguities to our fast
. s,,,.:al sciences, critical r-isual recognition system, in order to make us 'feel' our visual sense. Art allows us to
::. ¡ch. n-hich include a e-rperience more directly our sensations without being distracted by function: rather
'-:': ,
',:pire of the Senses: than 'art for art's sake', we might think more broadly in terms of 'sense for sense's sake',
.::1r3 as rrell, perception through the experience ofart.
EDITORS'INTRODUCTION
aesthetics, and even ethics: areas in which verbal and written/pictured information
have traditionally been considered dominant. It re-examines the 'sensory hierarchy'
that has devalued the proximal senses. While exalting the relatively neglected values of
:r)-. Lrr senses, of interest. the proximal senses, her essayalso considers the hidden dangers ofuncriticallyembrac-
j-f ses of taste and smell, ing a reversed sensory pyramid with smell and taste now at the top. For example, the
, i.. t.ruch on synaesthesia re-emergence of smell and taste also reflects marketing strategies to sell, for example,
an artificially scented dish-soap or expensive coffee drink. Such products provide quick
";;rn sculpture and touch
":, .id
economic interests doses of sensory pleasure so that today's busy consumer can fulfil this need outside of
..: .ss¿\-, Rosalyn Driscoll his/her sensorially sanitized workplace.
- ::1.ii. oicreating sculp- The link between science and the arts might be most fully explored in the realm of
:: r-iden behind a glass music. Simon Shaw-Miller's essay considers the interactions between auditory and
: r-sr:!rrical analysis of the r-isual aspects of performance. David Melcher and Massimiliano Zampini review
.-!:¿r.e.,,vhen theoretical research on how certain features of sound and sight are more likely to be combined
:- ¿nd on the hierarchyof than others, and this finding has interesting implications for how auditory stimuli are
;:tlar:r of Touch', Charles used in the arts. The chapter entitled 'Improvisation in Time: The Art of Iazz' combines
::;:s that our experience i¡terviews with two musicians working within the tradition of iazz and, more gen-
. :ir:nq our haptic inter- erally, creative improvisational music: drummer Skip Haden and saxophonist Greg
rision' of the senses, Osby. These two musicians, in addition to performing at the highest level in their field,
s[Lrwing that, in many are dedicated teachers. Thus, in addition to sharing personal experiences and anec-
:.',: plaled a greater role dotes from smoky jazzclubs and high-tech recording studios, both musicians contrib-
::-.'.-,-s the history of aes- ute insights from their years of theoretical investigations into the nature of music and
,:;:s about the aesthetic creativity. Scientists Carol Krumhansl and Fred Lehrdahl report on recent experiments
-,r-hich test a computational model of musical tension. In addition to providing a case
.: Sarker notes, touch can
:::.,,' hich artificiallv limit study on how cognitive psychology has developed theories that predict and explain
.:,.:nines touch and the aspects of music, this essay emphasizes the way that composers and musicians have
r :s a case study on the developed strategies to take advantage of our expectations of what will come next in
¿ musical piece. Guy Madison's essay tackles the fundamental question: why is music
.:; .rd deeply embodied such a popular and powerful medium? Music would seem, at fi.rst glance, a surprising
:ri\'n past. David Howes
r
:nd costly way to spend one's time, rather than caring for more pressing needs which
: ienscs can interact in
-. - .horv clear 'adaptive'value (in terms of evolution). Madison critically reviews evidence
:r: .::e11 are reviewed by .or the main theories, ranging from suggestions that music is just 'auditory cheesecake'
:-. t¡: .rut understanding sen.ing no real evolutionary purpose, to other theories which attribute an important
:.::'i¿.n. The perfect mar- :¡le to musical sensitivity in language development, in consolidating social bonds,
,-.: -.r¿tive rvork between r in choosing a mate.
;:e. Heston Blumenthal. Like music, visual art is ubiquitous across cultures yet offers a bewildering diversity
:- :mong the best in the -r sn4e. This combination of cultural specifrcity and cross-culturality provides a real
,.::.rce of flavour is influ- :hallenge to scholars investigating the arts. Artist Vik Muniz answers questions about
"::¡:: unbeknownst to us, :is investigations into the nature of representation and vision itself. David Melcher
' .::d Patrick Cavanagh provide a review of the visual 'cues' which can be used by art-
:'.:erinlentation, both in
--re c,- lhese recent studies, ..ls in creating two-dimensional images. In his essay, Ruggero Pierantoni surveys the
: .,'"-:erience of eating, are :--sion. of pictorial cues to depth. From a fantasy encounter with a deceased scientist,
::lu-:l.enso4. interactions : - an erudite excursion through Roman and Renaissance artworks, Pierantoni follows
-.-:c. ,,iláctory stimuli as :.-s reasoning like a detective, seeking to recompose lhe puzzle of depth perception
-,: ::.rm a series of historical and visual cues.
:,-
'l;h in epistemology,
EDITORS'INTRODUCTION
Indeed, there is more to understandingüsion than the static, pictorial marks used in
drawing or painting: our real environment is d1'namic. As Jeff Zacks and |oe Magliano
describe in their essay, when viewing a movie our brain interprets the flow of images
in terms of meaningful events. Their work, which brings together textual analysis The editors than,
and neuroimaging, provides a fascinating case study on neuroscience and the arts. An workshop and ar:
interview with Vittorio Gallese examines the role of 'mirror neurons' in the under- and the Institu::
standing, via vision, of the bodily actions of others. This much-publicized discovery Many thanks t.-r
needs a careful contextualization in order to fully illuminate its possible consequences Charles Spence. ;
on the cultural world. Here Gallese explores how a sense of shared embodiment can book.
form the basis of a common understanding of art as a universal human expression. The Francesca Ba;,
role of art in understanding the nature of perception is further illustrated by research the Histon- o: -ri
examining the brain actiüty of a blind painter, described by scientists Amir Amedi, Universin' of T:e
Lofti Merabet, Noa Tal and Alvaro Pascual-Leone. Through their work, they provide the writing ani e
a thoughtful and entertaining look into the mind (and brain) of a visual artist who
happens to have been blind from birth. David lleich::
The synaesthetic nature of perception is explored in more detail in essays by ofTrento, the Br:
famie Ward and Cretien van Campen. In the first of these, Ward gives an overview Physiologie de 1a
of scientific research on synaesthesia, including recent, cutting-edge work showing Psychologl at :-r-:
experimental and neural evidence for this fascinating condition. True synaesthesia and much neede.
involves a perceptual experience of a feature which is not actually carried bythat stim-
ulus, such as 'tasting' a particular shape or hearing sounds when viewing a colour. Thanks to Lr,r
Ward and van Campen argue for the importance of slmaesthesia as a source of evi- ilspiration.
dence for understanding the mechanisms of perception in all persons, including non-
synaesthetes. Multisensoriality is also an important feature of dance, which involves
the artful organization of body movements. Across cultures, people have reacted to
REpEREwcrs
music or even basic patterns of percussion by moving their bodies. Dance as an art
form, in addition to as a popular pastime, is the subject of two essays that link dance
Calvert G-\. Sprn;
to scientific studies of the human motor system. In the first essay, Ivar Hagendoorn IfIT Press. C:r
describes how his choreographic work is influenced by theories from cognitive neu- Clark A i199- . --.
roscience. The linkbetween dance and the brain is further explored by neuroscientists Cambridee. I,Li
Bealriz Calvo-Merino and Patrick Haggard. Horr-s5 P r¿¿. ::
The final essays in this collection examine the role of senses in design and archi- L¿koft G and T".h::
tecture. Rick Shifferstein and Paul Hekkert guide us through a process of design lo Il ¿-.¡¿r: l;:,-;,
t"r-erss
that is motivated by the senses and not just functionality. Likewise, planning a build-
G an¿ H:-
-\-';l¡¿r¿ ¡",,,; l'.,.:
ing, from the exterior to the interior, involves a set of decisions that can work with,
or against, the ways in which people tend to interact with their surrounding space.
Influential architectural historian Alberto Pérez-Gómez and archi-star Juhani
Pallasmaa have each contributed an essay on multisensory architecture. The final
paper in the collection, by Elena Pasquinelli, brings the study of the senses to the
present, and indeed the future, by considering the digital sensorium: our physical body
in a virtual environment.
EDITORS'INTRODUCTION
: E.torialmarksusedin AcTNowLEDGEMENT s
Z¡;ks and |oe Magliano
::its the flow of images
L-,E.iher textual analysis The editors thank the following for funding and assistance with the 'Art and the Senses'
r:.rence and the arts. An workshop andartexhibitions: theWellcome Trust, Quest International, Science Oxford
: i -rons in the under- and the Institute for Historical and Cultural Research-Oxford Brookes University.
;l - rublicized discovery Many thanks to the commissioning editor Martin Baum, and to Martin Kemp,
:¡ :u ssible consequences Charles Spence, and Marius Kwint for useful discussion in planning the workshop and
¡::red embodiment can book.
, *u:::an expression. The
Francesca Bacci would like to thank the Centre for Visual Studies (Department of
:: r-lstrated by research
the History of Art, oxford university), the center for Mind/Brain studies (clMec,
j.l.ntists Amir Amedi,
university of Trento, Italy), and the MART museum of Rovereto for support during
,1.-r rt ¡¡l¡, they provide
the writing and editing process.
r ¡- a usual artist who
DavidMelcherthanks the Center for Mind/Brain Sciences (CIMeC) atthe University
-':e .letaii in essays by
ofTrento, the Brain and Cognition Laboratory (OxfordUniversity), the Laboratoire de
',\'lrú qi\-es
an overview
Physiologie de la Perception et de I'Action (college de France), and the Department of
-r:-edge rtork showing Psychology at Harvard University for providing various resources (particularly space
. -r ¡ n. True s1'naesthesia
and much needed time) to the editing and revising of the essays in the book.
. . ;¿:ried bv that stim-
'...^.::l \iewing a colour. Thanks to our families for their support, and to chiara Sophie, especially, for
.-::.: d: a SOUfCe Of eVi- inspiration.
r: r-i FS. including non-
- ¿::ce. irhich involves
:-::le har-e reacted to
-- - ;res. Dance as an art
RErsnENcEs
L ::i¿-,-S that link dance
Calvert GA, Spence C, and Stein BE, eds (zoo4). The Handbook of Multisensory Processes.
-!>:'''.,r,-¿r Hagendoorn
'.:- MIT Press, Cambridge, MA.
l-:: ;.rgnitire neu- clark A Q997). Being There: Putting Brain, Body and world Together Agaln. MIT press,
,, ::¡ :-,- ne u¡oscientists Cambridge, MA.
Howes D (ed.) (zoo5). The Empire of the senses: The sensual culture Reader. Berg, oxford.
,:: : l:.ig,n and archi- Lakoff G and |ohnson M (rsss). Philosophy In The Flesh: the Embodied Mind and its Challenge
.i , l f :Lrcess of design to Western Thought. Basic Books, New York.
: .--.. :l:rnine a build- Weiss G and Haber HF, eds Q99). Perspectiyes on Embodiment: The Intersections of
Nature and Culture. Routledge, New York.
:.- :.-.:l ;¿n riork rvith,
:.;-: :ur:Lr unding SpaCe.
,:: :r;h:-st¿r Juhani
::J:r:i..iure. The finai
.:', ,: ¡ :re senses to the
r -:r . -r:hrsicalbody
M,
AI
WIT
FnaNc¡sc,l Bac;l
Clocksin, titled l-
in Oxford (Fig. r.
polyester threai. I
sensitive intertace
cannot yet male :
ofthe fact thar r,:
to emphasize ou:
the same time e,,-1,
nature is insepar;.i
Using this as a s:
lvhich artists or :r
MAKING SENSE OF
ART, MAKING ART
OF SENSE
FRANCESCA BACCI
IN CONVERSATION
WITH ACHILLE BONITO OLIVA
Fn¡Ncnsc¡ Bacci (F.8.): The image that opens this conversation is a work by Ella
Clocksin, titled Touch together (zoo6), featured in the 'Art and the senses' exhibition
in Oxford (Fig. r.r).It looks remarkably like skin but is made of plastics, nylon, and
polyester thread. In the words of the artist, 'skin's tactile intelligence provides a highly
sensitive interface for non-verbal communication, especially so for new babies who
cannot yet make sense of what they hear or see.'Besides being a powerful reminder
of the fact that we wear our most expansive sense organ at all times, Clocksin chose
to emphasize our nature as feeling creatures, by excluding the wearer's body, and at
the same time evoking it through the mother and infant-sized garments. Our sentient
nature is inseparable from-and in fact defines-who we are.
Using this as a starting point, I would like to ask when I say the word
lmullisensoryl,
which artists or specific artworks do you think about?
Achi'lle Bonito Oliva (A.B.O.): I think about Eeuy,s_,-who worked with a multiplicity
of languages, I_t!r!nk about works by the group Gutai, and then by artists belonging to
the movement Fiuxüs. In general I think abourartiits wh'ó-activáted pra-tices oftrés-
fras5ing, I ilóuld saf, lingering in the passage from a contamination of materials, tech-
niques, and languages, such as Nam June Paik, and more recently Claus Rinke, Marina
10 FRANCESCA BACCI
Abramovic, Vito Acconci. All these artists have worked on this 'art of trespassing', invest-
ing their own body in a performative yet personal way, to the point of mutilation or
masturbation. Vito Acconci did a gallery performance where he masturbated; Marina
Abramoüc cut herself with knives. Naturally also Viennese Actionism, with its protago-
nists Günter Brus, Hermann Nitsch, and Rudolf Schwarzkogler, comes to mind.
F.B.: I am often intrigued by artists who choose to address mainly the most neglected
senses, such as taste and smell, because often the use of edible material as an art medium
'opefates a sort of displacement. I am thinking, for example, about another piece that Fig.1.2 Helen G;
was exhibited in the Oxford show'Art and the Senses'. It was The Radcliffe Camero as a Cake and its Wor
and touched-and challenged the aura of grandeur and eminence characteristic of to do it, at le¿s: ;
the real building. At the same time, the case around it hinted at its inaccessibility, like sugar cubes. Ol:'-,
the horizon in the landscape. Due to its perishable nature, this work provided also a possible to evrrk. -
brilliant commentary on the special attention commanded by objects on the brink of as, for exampl:. :::
disappearance, and on the fact that stone, too, is destined to turn into dust one day. in Naples. The."'-r:
of smell and t¿s--
A.B.O.: There have been terrific artists who worked with chocolut" (tlt ttt"r"t with Mario Ifer¡.
"
Oppenheim inthe U.S.), which stimulate olfactoryandtouch senses, and also indirectly fruit. He then l--
MAKING SENSE OF ART, MAKING ART OF SENSE 11
.-.::'..': i a C ocksin.)
Kounellis worked with meat, starting from the example of Rembrandt's painting with
the quarter of an ox. In his case, though, the quarter of beef was really exhibited.
F.B.: It seems to me that, in thepast few years, contemporary art -h4s undergone a
A.B.O.: After the'anorexia of the image'causedbythe digital era and the internet, after
concept;al art, now we are witnessing a polysensory comeback, in order to give real
foundations to what we see through its materiality, its technical explicitness, and also
with elements that reach beyond the traditional limits of the medium (for example,
trompe-l'oeil). Now w_e have thepres-entation of rather than their represen-
laterials
t{ign=real mate{qlq,
F.B.: True, but yet, interestingly, parallel to the sensory impoverishment caused by the
digitalization of the image and the consequent loss of its physicality, there is an increas-
ing physical distance between some artists and their works. Many sculptors today do
not come in contact with their own work, rather they are detached from it since its
conception, preferring to entrust'fabricators'with the realizationof their pieces.
A.B.O.: But that is fine. That is because they present themselves as designers, project-
makers, so they build a üsual machine that then gets delivered, to reveal itself to the
life of the spectators. There is project but not execution-but effectively the result is
three- dimensional, material.
F.B. Sculpture especially, it seems to me, is well suited to encapsulate an atray of sen-
sory stimuli in one object. I remember one group of sculptures by Carole Andrews, A.B.O.: I think ir'
called Sentinels (zoo5) (Fig. r.:). life, a complete ar
Shaped as a group of tall conifers, the work was constructed in roofing felt, finely I think that conre
pleated and worked into a complicated surface using an embroidery technique known it has come incre;
as smocking. A noticeable odour ofbitumen exuded from the surface, creating a mental familiar iconosra:
oxymoronbetween the industrial and natural attributes of the piece. To one's surprise, riality is strateeica
an unexpected sound of quiet breathing could be heard when hugging these tree-like a higher rate ofct
black forms, which made them feel uncannily alive. to percolate in a .-,
This taps into a well-established trend of today. In the past, the senses were used by
perceivers to capture as much of a percept as possible, since their survival depended F.B.: How rrou.,i
on the amount of information gathered and on their accuracy. Today, we are sub- artistic technique
jected to a continuous sensory'attack', with information competing for our attention multisenson'disc,
all the time-different realities, including art, strive to be perceived, almost throwing others)?
themselves in the public's face. For the purpose of 'visibility', it seems that artworks
that feature multisensory stimuli may have a greater chance of being noticed by the A.B.O.: In terrns r
public. Do you share my impression that embedding multisensoriality as a characteris- forget that Borro
tic of the work is a conscious goal pursued by contemporary artists (as synaesthesia or through it r.ou r¡
Gesamtkunstwerk used to be in Romantic times)? Let's not forget. a
MAKING SENSE OF ART, MAKING ART OF SENSE r3
: - :- Jesigners. project-
;i.. ir¡ r€\'€ál itself to the
: .i:-cti1-eh' the result is Fig. 1.3 Carole Andrews, Sentinels, 2O05.
(Photograph : Keith Barnes.)
chemicals emplole,
illusory depth-that is to say, non-existent up to the point of anamorphosis, which is
point, sion of 'contempoi
the intentional deformation of a shape that one can look at only from one specific
long march, development from the two paradox of preser,-i
often laterally, in order to see it. Art history is a a
up to lhe tabul7 rasa indistinguishable ¿i
dimensions of the surface to the three dimensions of perspective,
three-
of Impressionism, which restarted this march towards the outside, towards the
not illusory but material, through the collage' Picasso F.B.: And let us nLrr :
dimensionality of the spectators,
painting was a prelude comes at the cost oi:
is the father of all this. His collage with the train ticket glued to the
cially touch), rr-hic:r
to adding other elements, all the way to the ready-made-an object which substitutes
way in
the pictorial shape. Then there is Lucio, Fontana, who, with a cut, opened a
A.B.O.: That is par:
through the canvas, which thus became a space rather than a surface, which then
leads
public can physically move, the nomadic countries, such as lr
to installations. These offer a space where the
counts. In Chi¡a rr-:
nature of the spectator is excited, and this by itself implies-how shall I say it?-a
poly-
is this fetish of the ¿:
sensory dimension. This is the context where the new media came on to
the scene.
difficult to mediate between the need to make the work F.B.: This leads us r,-,
F.B.: As a curator, I often find it
Culturally Walter Benjami:r .
available to the public and the legitimate concern about its preservation' -
A.B.O.: Yes, they reckon that it is fine to replace the old with the new, for the
purpose F.B.: This sounds i
of maintaining a full sensory experience of the art'
\\rhat, then, dis¡nl,
peruse) to use ¿l c,i:
F.B.: This makes one wonder whether it is sensible to carry out what is commonly
A.B.O.: \\'hat is ci-.
referred to as 'philological' or 'scientific' restoration (in Cesare Brandi's terms) of
contemporary artworks-ultimately, whether it makes sense at all to stubbornly
Duchamp's teacl::r,
pursue ihe eternal physical integrity of art, since all that is material is headed towards
object on a plinth r:r
make art be an. uLt,--r
dissolution anryay.
A.B.O.: This is an open debate, which I addressed zo years ago with a series of articles in
F.B.: It is a u ell-k¡¡ '
which I wrote: how can you restore a work by Alberto Burri, where he used plastic that
of science. Histo:i;'
is destined to waste an1'way? Do you knowwhat solution I came up with?
I proposed to optics, perceptir-¡n :
this trend in conren
bury artists with their artworks,like pharaohs. Do you like this idea?
nentioned. is pro-u:i
F.B.: Well, I may like it, but I suppose that this proposal has not found much support
This installatior.
a particular afteru:
among collectors and institutions. . .
effect is a r;si
"rsual
:¿te of projection. T
A.B.O.: This need for the survival of artworks is born both from the desire of immortal-
ity of the artist, and also from the desire of the collector to preserve the work in which cimensional ligü: s-
s/he has invested money, so really it is the art system that requires this continuity-
i dilterent colou:¡i
'-,
hence the restoration that can also be carried out on contempolary artworks, although ¡rrrtrait, unachie','¿b
MAKING SENSE OF ART, MAKING ART OF SENSE r5
rc-idean geometry, of an often these works are very delicate, requiring deep knowledge of the materials and
ar:amorphosis, which is chemicals employed, coupled with a conceptual knowledge of the work. The profes-
.r nr one specific point, sion of 'contemporary art restorer' is still very 'futuristic', as one works on an inner
i ¿,¡rpment from the two paradox of preserving contemporary artefacts that could be easily replaced by other
.'''e . up fo fhe tabule rasa indistinguishable and identical contemporary artefacts.
:.ll:. rorvards the three-
r *- the collage. Picasso F.B.: And let us not forget that the pursuit of the physical integrity of the artwork often
-,: ::hting rvas a prelude comes at the cost of forbidding the full apprehension of the art through all senses (espe-
t,'a'e.1\rhich substitutes cially touch), which is to say at the expense of the work's full expressive power.
:
;r*i. opened a way in
:J:ce, rlhich then ieads A.B.O.: That is part of the European tradition, it is a Western characteristic. In Eastern
;; -,'mor-e, the nomadic countries, such as India for example, everfthing is left to decay because it is time that
...rjl Isar it?-apoly- counts. In China what counts is the experience. Here it is the object that counts; there
.r:r: .)n to the scene. is this fetish of the artwork tied to the concept of the unicum.
e reed to make the work F.B.: This leads us to consider the long-standing debate between original and copy, and
F :i sen'ation. Culturally Walter Benjamin's idea that the aura may be lost when the work is reproduced a great
r-:[ Lrf afil\'orks, such as number of times.
:,:,: ust at presen'ing the
:a:-r \\ e -,ritness instances A.B.O.: Well, while contemporary artworks may be more democratic because
i,r lriS€ñ-€ the object in of their reproducibility, the fact remains that, when a work is unique, it retains a
greater value.
::l. 1e1.,'. tbr the purpose F.B.: This sounds dangerously similar to the economic law of supply and demand.
What, then, distinguishes contemporary art from any other object of consumption? To
peruse, to use all of one's senses on an artwork, means to consume it.
-,;i ithar: is commonly
:s¿:; Brandi's terms) of A.B.O.: What is crucial is the frame in which the object is placed. This is, again,
r.¿ a: ¿[ to stubbornly Duchamp's teaching. A sanitary device placed in a toilet is just that, while the same
:;;::- is headed towards object on a plinth in a museum gets recognized as a'Fountain'. It is the frame that can
make art be art, differentiating it from any other object.
-,,,----: a .eries of articles in F.B.: It is a well-known fact that artists' curiosity responds quickly to the advancement
;i:-:. :r3 used plastic that of science. Historically, one can trace a trail of artworks that incorporate notions from
r: i: '",rih? I proposed to optics, perception psychology, and now also vision science. A stunning example of
\ Ji:, this trend in contemporary practice, which effectively summarizes the three fields just
mentioned, is provided by Georgia Chatzivasileiadl's Aftertrace (zoo6) (Fig. r.a).
:,: 'L-rurd much support This installation, shaped as a dark room with an entrance and an exit, is based on
a particular afterimage phenomenon, which is known as 'the rainbow effect'. The
visual effect is a result of the combination of the eye movement ('saccade') and the
:: ::: iesire oiimmortal- rate of projection. This creates the illusion of a two-dimensional image from a one-
:he rr-ork in which dimensional light source. When a viewer moves in front of the light beam, a stream
"s.:-'.e
:-...:r i rh js continuity- of different coloured afterimages of the moving person follows their figure, as a live
: :::.; arn. orks, although portrait, unachievable in any other medium. The new decomposed image expands into
16 FRANCESCA BACCI
A.B.O.: If these sc
Schródinger's qua:
stood that cont€m
tion to find its cor
public, which is nt
Rnrnnn¡qcr
Freedberg D and G;
Trends in Cogri:;:;
the third dimension: it is detached from the two-dimensional surface, and spreads into
the entire projection space, thus becoming, in the words of the artist, an'experiential
perception of the projected image'.
As a consequence of, or parallel to, such growing neuroscientifically oriented artis-
tic production, one can notice an increased interdisciplinary trend merging the fields
of art criticism and neuroscience. Let us discuss the recent article by Freedberg and
Gallese (zoo7), where the idea of mirror neuron system has been considered in the
context of the history of art. Basically, these authors claim that by observing a repre-
sentation of a real action we can experience, at the level of neural activation, the same
sensation as if we were protagonists of that action. This seems to put all figurative art
in a new context.
A.B.O.: This process of substitution is one that one can suspect in all phenomena, even
in collecting. A collector, in exchange for his money, 'lives' a work and manages it with
a sense of possession as if s/he was the artist. The spectator who goes to the museum
establishes a rapport, solitary even if temporary, with the painting, puts her/himself in
the presence of the work, thus achieving an emotion that should correspond to that of
the artist. Quite frankly, these are scientific hlpotheses that do not concern me intel-
lectually. can you explain why one has to raise this problem? why do I have to care if
the neurons activated through contemplation are the same as the ones activated
through action?
F.B.: Because it means that in the observer there is a physical involvement that goes well
beyond the simple observation, into the domain of empathy.
MAKING SENSE OF ART, MAKING ART OF SENSE 17
RErnRnNcn
Freedberg D and Gallese Y (zoo7). Motion, emotion and empathy in aesthetic experience.
T rends in Cogn it i ve Sciences, tt, tg7 *2o¡'.
:: ,1 : ::lanomena, eVen
, . :.. .r¿ tnanaqes it with
-
:,,:- the mUSeUm
.L:,aS tLr
a:,:-:. :u:s her,/himself in
*, I : '- ::espond to that of
: , :r : :Ll:icefn me intel-
' ''. ;1', :¡-r I har.'e to care if
:-. :. rh: ones activated
NICHOLAS I. WADE
INrnoDUcrroN
M¡Ny artists have sought to represent the pleasures of the senses and the genre has
ancient origins. However, studies of the senses display paradoxes associated with sci-
ence as well as art. In both, the senses tend to be limited to the classical five listed and
restricted by Aristotle: 'There is no sixth sense in addition to the five enumerated -
sight, heaiing, r-"11, taste, touch' (WD Ross 193r, p. 424b).In the context of art, one
sense-vision-is t1pically used to represent the others, often in allegorical form. The
sensuousness of touch, taste, smell, and hearing are depicted in a medium that denies
experience of them. For science, the paradoxes have been otherwise. Experimental
emphasis has certainly focused on vision, but theoretically touch has been accorded
dominance. This has been so despite the problems, evident to Aristotle, associated
with defining this sense.
The present chapter will examine the history of the science of the senses, with
flourishes of art, and will examine the factors that extended their number from the
Aristotelian five. The main contributors to this scientific history will be depicted as
'perceptual portraits'. The faces themselves are not always easy to discern-readers
will need to apply the power of their own perception to extract the facial features from
the designs that carries them. They can usually be seen more easily by squinting the
eyes, shaking the head from side to side, viewing the page from a distance, removing
7
NICHOLAS J. WADE
your spectacles if you wear them-in fact any procedure that reduces the sharpness
of the fine lines in the pattern that make up the motif. The aim of perceptual
portraits is both artistic and historical. They generally consist of two elements-the
portrait and some appropriate motif. The nature of the latter depends upon the endea-
vours for which the portrayed person was known. In some cases the motif is drawn
specificallyto displayaphenomenon associatedwiththeindividual, inothersitisderived
from a figure or text in one of their books, or apparatus which they invented. The por-
traits and motifs have themselves been manipulated in a variety of ways, using graphi-
cal, photographical, and computer graphical procedures. It could be the case that such
perceptual portraits both attract attention and engage the spectator's interest to a
greater degree than do conventional paintings, prints, or photographs. It is hoped that
this visual intrigue will enhance the reader's desire to discover why particular motifs
have been adopted, and in turn to learn more about the persons portrayed: it is intended
to be an instance of art serving science. Further examples can be found in Wade (r99o,
t995, zoo6, zoog).
The senses have always been at the heart of the sciences. They were not only studied
in their own right but they provided the evidence for assessing other natural phenom-
ena. Indeed, for Aristotle (c38+-zzz BC) (Fig. z.r), sensation was housed in the heart
because of its perceptible warmth in contrast to the cool of the exposed brain to touch.
At the beginning of the nineteenth century, with the growing knowledge of nerve
function, the relation of the senses to the brain was explored in greater detail, and
one question was repeatedly addressed: how many senses are there? The five senses of Fig.2.1 Nichola
sight, hearing, smell, taste, and touch are rooted in our culture. The prominence from an engravir
of eyes, ears, nose, and tongue on the head, and the specific experiences associated Vernor, Hood, an
üth them, have acted in the past, as well as in the present, to fix these four senses. taste, and smelf
Touch presents more problems because its sensitiüty is not localized to a particular which he various
sense organ, and the experiences derived from the skin are many and varied. Aristotle
confronted these aspects of anatomy and experience and reached similar conclusions.
Nonetheless, touch, requiring contact in order to experience it, was often taken as the and pressure I ',r r
most important sense, and the one relative to which others could be related: 'The pri- received experim
mary form of sense is touch, which belongs to all animals' (WD Ross 1937, p.4r3b). It behavioural and ,
is perhaps for this reason that Aristotle maintained that touch is a single sense, that the
number of senses is restricted to five. Boring's conclusion about this dogma was clear:
'It was certainly Aristotle who so long delayed the recognition of a sixth sense by his
doctrine that there are but five senses' (1942,p.525). For Boring, as for most other his-
torians of the senses, the additional one that emerged in the early nineteenth century
was the muscle sense.
The analysis of experiences deriving from stimulation of the senses has often been
placed in a philosophical context. However, the anatomy of the senses and the phe- Aristotle's sun'ei
nomena associated with them has guided the ways in which they can be classified. (see Beare 19o6 .
In what follows, the anatomical and phenomenological aspects of the senses will be In associating tho
emphasized. it will be argued that the separation of a muscle sense from touch was of Alcmaeon ilX.
given empirical support in the late eighteenth century, as was the evidence for a context of touch
movement or vestibular sense. The multiple dimensions of touch (like temperature and Democritus
THE SCIENCE AND ART OF THE SIXTH SENSE
PIalo (427-347 BC) wrote that touch distinguished between hot and cold, hard and
soft, heary and light, as well as rough and smooth. The approach by Galen (c.r3o-zoo)
to the senses displayed the advantages of anatomical dissection. He berated Aristotle
for denying that all the senses do not have connections with the brain: 'Hence all the
instruments of the senses - if we are to believe our eyes that see and our hands that
touch them - communicate with the encephalon' (May 1968, p. 39r). Galen's theory
of the senses was physiological, and it was based on the concept of pneuma advo-
cated by Empedocles (c.+gZ-+lZ BC): 'Unless the alteration in each sense instrument
comes from the encephalon and returns to it, the animal will still remain without sense
perception' (May 1968, p. 4c4). Galen restricted his discussion to the 'four sense instru-
ments in the head, namely, the eyes, ears, nose, and tongue, all of which take the source
oftheir sensation from the encephalon' (p.+oo).
The situation remained relatively unchanged through the medieval period:
'Aristotle's account of sensation and perception was held in great esteem in the Middle
Ages, and his systematic approach and many of his specific doctrines were widely
copied' (Kemp r99o, p. :S). Attention was directed principally at interpretations of
vision, with much less heed paid to the other senses. Developments did occur in fusing
Aristotle's account of the senses with Galen's pneumatic physiology, and the medical
tradition of describing diseases ofthe senses became more refined.
When Galileo Galilei Q56z-t642; ró23) (Fig. z.z) considered the senses and sensa-
tion he did so in the context of this medieval tradition. Nonetheless, he did not share
Aristotle's faith in the veracity of the signals provided by the senses, and his approach to
sensation canbe seen as breakingwith the medieval tradition (Piccolino zooZ; Piccolino
and Wade zoo8a,b). He clearly appreciated the indirectness of sensory experiences:
'I think that tastes, odors, colors and so on are no more than mere names so far as
the object in which we place them is concerned, and that they reside only in the con- Fig.2.2 Nichol;
sciousness' (Drake 1957, p.zfi.The deceits of these senses were matters for the proper embedded in thr
sensibles (such as colour, tone, and saltiness) rather than the common sensibles provided an ana
(like size, shape, location, motion, and number). From the üewpoint of sensory
physiology, Galileo's statement can be taken as a forerunner of Müller's doctrine of
specific nerve energies, formulated in the early nineteenth century. fohannes Müller my entire bodr. .;
(r8or-r858) argued that we only have available to us the signals that are sent to the sen- whose means rn e s
sorium, and this sentiment is clearly evident in Galileo's writing: by other parts ot :
To excite in us tastes, odors and sounds I believe that nothing is required in external bodies
except shapes, numbers, and slow or rapid movements. I think that if ears, tongues, and Note that spe.
noses were removed, shapes and numbers and motions would remain, but not odors or tastes
assigned to the 1
or sounds.
Galileo stated:
(Drake 1957, pp. 276-77)
Perhaps the on*l
In expanding on the relation of stimulus to
sensation, Galileo devoted more dissolve into rn.:r,
attention to the senses of hearing, taste, smell, and touch than to vision. For touch are lighter and n:
he stated: more sensitir-e th:
the upper surtice
A body which is solid and, so to speak, quite material, when moved in contact with any part etrating its mois:-.
of my person produces in me the sensation which we call touch. This, though it exists over various shapes. n-
THE SCIENCE AND ART OF THE SIXTH SENSE 23
,: senstrn- experiences:
:r ::rere names so far as
'-=.. t..rnlt in the con- Fig.2.2 Nicholas Wade, Ihe sensory messenger. A portrait of Galileo has been
; ::r:l!ers tbr the proper embedded in the title page of his book tt Saggiatore (Galileo 1623), in which he
provided an analysis of the senses.
-jli -LlIllITlorl sensibles (oNicholas Wade)
; -"--:r,.prrint of sensory
, - I'lu;'er's doctrine,of
:;-:-,.. liohannes Müller my entire body, seems to reside principally in the palms of the hands and in the fingertips, by
-.:'. lrq .cnt tO the Sen-
1
whose means we sense the most minute differences of texture that are not easily distinguished
by other parts ofour bodies.
t=
(Drake t957,p.275)
::.--::: ,n erternal bOdieS
-:.- -: :.rs. tOngUeS, and Note that specialization within the sense was also described, with greater sensitivity
. r ::: n¡t odors or tastes assigned to the fingertips than to other parts of the skin surface. For taste and smell
Galileo stated:
I :,:i: :9;;. pp. 276-77)
Perhaps the origin of two other senses lies in the fact that there are bodies which constantly
-''r-eo deYoted more dissolve into minute particies, some of which are heavier than air and descend, while others
r-. --- ','lsion. For touch are lighter and rise up. The former may strike upon a certain part of our bodies that is much
more sensitive than the skin, which does not feel the invasion of such subtle matter. This is
the upper surface of the tongue; here the tiny particles are received, and mixing with and pen-
: -:t ,.rnlact rvith any part etrating its moisture, they give rise to tastes, which are sweet or unsavoury according to the
":r-¡. lh.ruqh it exists over various shapes, numbers and speeds of the particles. And those minute particles that rise up
24 NICHOLAS J. WADE
may enter by our nostrils and strike upon some small protuberances which are the instrument
Contact and:
of smelling; here likewise their touch and passage is received by our like or dislike according as
1664119o9) to be
they have this or that shape, are fast or slow, and are numerous or few. The tongue and násal
passages are providently arranged for these things, as the one extends from below to receive feeling his l ar- t
descending particles, and the other is adapted to those which ascend. Perhaps the excitation for hearing and r
oftastes may be given a certain analogy to fluids, which descend through the áir, and odors to was traced back
fires, which ascend. importance oito
(Drake 1957, pp. 275-6) (1685-1753 t7o9
ing sight with tor
with regard to sound, Galileo adopted a similar mechanistic interpretation:
verging the er-es
Then there remains the air itsell and element available for sounds, which come to us indif- with sight for p
ferently from below, above, and all sides - for we reside in the air and its movements dis- argued that rre it
place it equally in ail directions. The location of the ear is most fittingly accommodated to all
the muscles con'
positions in space. Sounds are made and heard by us when the air without any special prop-
- to provide dire;
erty of 'sonority' or 'transonority' - is ruffled by a rapid tremor into very minute waves and
vision the dimer
moves certain cartilages of a tyrnpanum in our ear. External means capable of thus ruffling
the air are very numerous, but for the most part they may be reduced to the trembling of some fundamental sar:
body which pushes the air and disturbs it. Waves are propagated very rapidly in this way, and order in rvhich t
high tones are produced by frequent waves and low tones by sparse ones. sight, and hearir
(Drake ry57, p. 276) Touch' (rZ+S.p.
not thor-rght tc'r b
Touch, taste, smell, and hearing were related to the elements of earth, water, fire,
Representatir-e'
and air. The analysis of these four senses was mechanical, both in terms of the stimulus
Thedominan"
and the response to it. Essentially, Galileo was following the Aristotelian path of treat-
even when thel i
ing touch, a patently mechanical sense) as the yardstick against which taste, smell, and
uity, but \\'as nL-)1
hearing should be considered. By contrast, Galileo was much more enigmatic in his
BC) commented
consideration of vision:
and the üsual n:r,
And as these four senses are related to the four elements, so I believe that vision, the sense referred to the r:
eminent above all others in the proportion of the finite to the infinite, the temporal to the (c.S8-¡i BC ) ga,
instantaneous, the quantitative to the indivisible, the illuminated to the obscure that vision, 'The room seem
-
I sa¡ is related to light itself. But of this sensation and the things pertaining to it I pretend to
they themseh'es
understand but little, and since even a long time would not suffice to explain that trile, or even
building is not ti
to hint at an explanation, I pass this over in silence.
was able to indu¡
(Drake D57, p. 277)
to the movemenl
Both Aristotle and Galileo felt secure in their analyses of four of the putative five fl. 68o) noted rh¿
senses' but they were not the same four! Touch posed a problem for Aristotle because
Vertigo is occasicr
of its multiple dimensions, its wide distribution over the skin surface, and its primacy are ready to fall dl
as the quintessential contact sense. The other senses were clearlylocalized to special- nal object'w.hich r
ized sense organs. Galileo was more concerned with the stimulus than with sensation. or when their h¡.
Touch, taste, smell, and hearing could all be interpreted in terms of the response to in motion.
mechanical stimulation; vision did not fit easily with this scheme, as little was then
known about its physical basis. Accordingly, Galileo did not embrace it in the manner
Generating,,-is
he adopted for the other senses and he wrote very little about the process of vision
stopping, lrasasr
(Wade zooTa).
Wade zoooa . \-,
THE SCIENCE AND ART OF THE SIXTH SENSE 25
. it'hrlh are the instrument Contact and movement were considered by René Descartes i596-t65o; t637ltgoz,
o¡ dislike according as
-:: t66+lryog) to be the touchstone of all the senses. He used the analogy of a blind man
:-r,;. The tongue and nasal
feeling his way through space by means of sticks held in each hand as the paradigm
::: ::om be]ow to receivc
*;. Perhaps the excitation for hearing and vision as well as touch. In his philosophy of the senses, all perception
,',r¡h rhe air, and odors to was traced back to movement, and all stimuli were thought to be mechanical. The
importance of touch and muscular activitywas also a cornerstone of George Berkeley's
(t685_lZSl; r7o9) empiricism. He proposed that we learn to see distance by associat-
ing sight with touch. Moreover, the degrees of muscular contraction involved in con-
. -:1tefpretation: verging the eyes are also correlated with distance, and provide a source of association
''..r -h ¡ome Lo us indif- with sight for perceiving distance. In order to perceive distance visually, Berkeley
- .:l irr morements dis- argued that we learn the relationship between the visual stimulation and the states of
.,1'.' .r..ommodated to ail the muscles controlling the eyes. The muscular and touch systems were considered
. --. 'Lr ¿nv special prop- to provide direct and undistorted spatial information that could be used to teach
: - ',4:-!' minute tvaves and
vision the dimensions of space. Like Berkeley, Hartley (r7a9) held that touch was the
-¡ :::¿ble of thus rutfling
fundamental source of information to integrate the senses. This was evident from the
i :- .1: iiembling of some
-''..jt", in this rvay, and order in which the senses were surveyed: feeling was first, followed by taste, smell,
sight, and hearing. If there is conflict between touch and sight 'we always depend upon
Drake 1957, p. 276)
Touch' (V49, p.r3S). This was considered to be so, in large Part' because touch was
not thought to be as readily deceived as vision: 'we call Touch the Reality, Light the
::s i,i earth, rvater, fire, Representative' (p. r38).
-:. .r:¡rs olthe stimulus The dominance oftouchwas such as to obscure theviewofother sensoryexperiences,
' :. rc-:-rn path of treat- even when they involved movement. One common experience discussed since antiq-
: ,.":l;h :aste) smell, and uity, but was not incorporated into the senses, was vertigo. Theophrastus (c37o-286
. ::-.:e eniematic in his BC) commented on vertigo or dizziness (as when looking down from a great height)
and the visual motion that accompanies it (see StrattorrrgTTl Sharples zoo3). Aristotle
:; : ::t¿; ., tsiOn. the Sense
referred to the visual vertigo that follows drinking too much wine, and later Lucretius
: .: :ui lemporal to the (c.S8j¡ BC) gave a graphic description of vertigo following rotation of the body:
:,. i..-rre - that vision, 'The room seems to children to be turning round and the columns revolving when
:::-:t--,: :.r it I pretend to they themselves have ceased to turn, so much so that they can hardly believe all the
::,--: lt --hat t¡ifle, or even building is not threatening to fall in upon them' (1975,pp.307-9). Ptolemy (c'85-ró5)
was able to induce vertigo by üsual means alone (see Smith rgg6). Vertigo was related
D'ra;e ry57, p. 277) to the movements of the animal spirit in the head. For example, Paulus,lEgineta (1844,
, *: -,- ::e putative five fl. 6so) noted that:
:r :, : -l,ristotle because Vertigo is occasioned by a cold and viscid humour seizing upon the brain, whence the patients
-:i.;. and its primacy are ready to fall down from a very siight cause, such as sometimes from looking at any exter-
:-'" - - ¡¿iized to special- nal object which turns round, as a wheel or top, or when they themselves are whirled round,
*: :L,:n ''rith sensation. or when their head has been heated, by which means the humours or spirit in it are set
:::rs ,-'f the response to in motion.
(tg+c,p.y+)
.i:--.. as iittle rvas then
:tr¿!3:¡ i¡ the manner Generating visual vertigo, even in healthy individuals, by rotating the body and then
.: :h; process of vision stopping, was a source of regular reflection by medieval medical and optical writers (see
Wade zoooa). Vertigo accompanies many diseases and has been described frequently
26 NICHOLAS J. WADE
over the centuries in medical texts. It will be argued that the experimental study of The link beno
vertigo in the eighteenth century heralded the appreciation of a sixth sense before Be17 (ry2+-t8+:
experimental evidence for a muscle sense or the fractionation of touch was available. roots carry mL')t
was in specihirg
were described :
specific senses:
In this inquin-it r.
Cr¡.ssrFYrNG THE sENsES a capacity for rec,
incapable of recei
very remarkabtre i
The sources of evidence available to Aristotle (and to those who followed him over same instrumen:.
the next zooo years) for distinguishing between the senses were perception and relation to the or5
anatomy. They could report on their experiences when stimulated, and they could
relate them to their body parts. For example, sight ceased when the eyes were closed,
In the conter-rt
and hearing diminished when the ears were blocked. Additional inferences could be
pressure to the e',
drawn from disease or injury. At a theoretical level, recourse was made to philosophy,
Hagner r99o: Pa:
usually linking the senses to the elements-fire, earth, water, and air-which perme-
ated perception. The classical accounts ofthe senses drewprincipallyupon psychologi-
ofelectricitr-to :
cal (or behavioural) evidence for their independence. In contrast, developments in the If light, pressurr.
last few centuries have relied increasingly on anatomical and physiological indications the mind is üe r:
of separate senses, and the behavioural dimension has been given less prominence.
While emphasizing the veridicality of sensing in general, Aristotle did entertain
the possibility of errors (illusions) entering into a particular sense. The examples he
mentioned were those of colour or sound confusion and errors in spatial localization
of colours or sounds. Illusions are often considered to be a modern preoccupation,
based on specific theories of perception, but their origins are ancient and illusions
can be investigated with little in the way of theory (see Wade r99o, 1998, zoo5a). If
there is an assumption of object permanence, then an illusion occurs when the same
object appears to have different properties (of colour, position, size, shape, motion,
etc.) under different circumstances. Thus, the existence of an illusion might provide
evidence of a sensory system. For example, we normally feel stable and still when we
stand upright; however, an illusion of body motion can occur when we are standing
upright-if we have previously rotated the body rapidly.
The situation regarding the senses was radically revised in the nineteenth century,
with developments in physics, anatomy, and physiology. sources of stimulation could
be specified and controlled more precisely. This had already occurred in the context of
colour, with Isaac Newton's Q64z-t727; r7o4) methods of spectral separation of white
,ffi
,.4 +
light and mixing components of it. Thomas You ng Q77-r829; rSoz) applied the colour ,'./. .;
mixing method and found that all colours could be produced by appropriately com-
pounding three primaries; he suggested that the eye was selectively sensitive to each. Fig. 2.3Nichol
Young QSoTlzooz) also introduced the term'energy'in the context of weight, and diagram of the
this concept was related by others to different dimensions of sensitiüty, like light and portrait is prest
sound. of correspondin
THE SCIENCE AND ART OF THE SIXTH SENSE
. e\-perimental study of The link between energy and sense organs was forged soon thereafter. Charles
a si,xth sense before
t-.1 Bell (ry74-t8+z) (Fig. z3,left) is noted for discovering that the anterior spinal nerve
r,f touch was available. roots carry motor nerves (see Cranefield t'g14. His principal concern, however,
was in specifying the senses and their nerve pathways to the brain. His experiments
were described in a privately published pamphlet which also related stimulation to
specific senses:
In this inquiry it is most essential to observe, that while each organ of sense is provided with
ES a capacity for receiving certain changes to be played upon it, as it were, yet each is utterly
incapable of receiving the impression destined for another organ of sensation. It is also
very remarkable that an impression made on two different nerves of sense, though with the
..,'h.-, ¡ollorved him over same instrument, will produce two distinct sensations; and the ideas resulting will only have
s !r':re perception and relation to the organ affected.
.';-¿red. and they could (Bell r8u/zooo, pp. 8-9)
:r t,:e e\-es rvere cl0sed,
.' In the context ofvision, the demonstration of this fact had been known to Alcmaeon:
:- rnterences could be
pressure to the eye, even in darkness, produced the experience oflight (see Grüsser and
.s :rade to philosophy,
Hagner r99o; Park iggz). Bell was able to bolster this observation with the application
i¡:¡ ¿l¡-1yf¡ich perme-
ofelectricity to the eye:
,:: i'uponpsychologi-
i.:. c:r-elopments in the If light, pressure, galvanism, or electricity produce vision, we must conclude that the idea in
:,'r,-- loqical indications the mind is the result of an action excited in the eye or in the brain, not any thing received,
:r -:ss prominence.
-r-:r.:otie did entertain
¡¡:.". The examples he
": -: s:atial localization
1'-i3rl preoccupation,
. .:;rant and illusions
; _i.1_r. 1998,:.oo5a). If
:;;:s rrhen the same
:r. si:e. shape, motion,
" .s:,i¡:i mieht provide
::,- ¡ ard still rvhen we
:',!i:-:1 i\'¿ are standing
though caused by an impression from without. The operations of the mind are confined not
by the limited nature of things created, but by the limited number of our organs of sense.
(r8rr/zooo, p. rz)
A similar sentiment, voiced with primary reference to the nerves and their path-
ways, was written a fewdecades earlier bylohn Hunt er eTzg-r793;17g6).The examples
he gave to support this contention were the referred sensations arising after damage
or amputation. The seeds of this idea can be found in antiquity, although it was based
on philosophical rather than physiological speculation. Galileo sought to link the
stimulus with sensation for what he saw as the mechanical senses, but the seeds blos-
somed in the nineteenth century. The doctrine of specific nerve energies, as it became
called, was given further support by Müller (Fig. 2.3, right), in a monograph on com-
parative physiology and on eye movements (Müller 18z6) and it was amplified in his
influential handbook of human physiology (Müller 1838, 1843, zoo3). Although the
doctrine was framed in terms of differences between the senses, it was used increas-
ingly to determine qualitative distinctions within them (see Finger and Wade zooz;
Wade zoo3b). Müller considered the criteria that would need to be satisfied in order to
increment the number of senses: Fig.2.a Nicholas
The essential attribute ofa new sense is, not the perception ofexternal objects or influences a portrait, said tc
which ordinarily do not act upon the senses, but that external causes should excite in it a new (Hooke 1 665). Ri
and peculiar kind of sensation different from all the sensations of our five senses. (1684), is combir
(t843lzoo3, p. ro87)
Despite occasii
Anatomy tively little anent
the microscopes
The gross structures of the nervous system could be examined with the naked eye,
optical componel
but a new anatomical world was exposed by the microscope (see wilson 1995; Harris
powerful achrorn
1999; Schickore zooT). However, this formerly unseen territory was not sharply
after (Pol1,ak 19;
resolved by its seventeenth-century explorers due to the optical aberrations inherent
clearly articulate*
in the relatively simple devices and the lack of techniques for preparing specimens
r8¡8) (Fig. 2.5 a:
for observation. while the microscopes of Robert Hooke (1635-17o:) (Fis. z.4,left) and
the biological scr,
Antoni van Leeuwenhoek (1632-rzz) (Fig. z.4,right) bared the cell before the human
teenth centun'd:
eye' they did little to expose its fine structure. This was to await the development of
widely accepted
achromatic microscopes in the early nineteenth century.
In 1832, Jan E"
Hooke (1665) gave the name 'cell' to the structures üsible in sections of cork. His
an achromatic nr
compound microscope was capable of magnif ingby +o times, but he also described
cells in the cerel
a superior instrument of greater optical power. He did not make such a micro-
observations rrer
scope, but Leeuwenhoek 06zd did (see Ford r99r, zooT). He gave accounts of sev-
Purkinje used alc
eral animal cells, including optic nerve fibres, when he directed his simple magnifier
could be exarnin
to animal tissue. His major achievement was to refute the long-held view that nerve
able to resoh'e th
fibres were hollow. Moreover, Leeuwenhoek G6z) suggested that signals were propa-
time Santiago R;
gated mechanically along the neryes (in the fluids they contained) from eye
to brain scope at neurai s:
(Wade zoosb).
specimens. A sin
THE SCIENCE AND ART OF THE SIXTH SENSE 29
Despite occasional forays into this field throughout the eighteenth century, rela-
tively little attention was paid to these observations because the resolving power of
the microscopes was poor as a consequence of the aberrations introduced by their
.¡j -,'"-tlh the naked eye, optical components. The microscopic world was transformed by the introduction of
.t; i"'ii5e¡ 1995; Harris powerful achromatic instruments in the r8zos, and rapid advances were made there-
:-:i,.,' h-as nOt Sharply after (Polyak ry57;Harris 1999; Schickore 2oo1, zooZ). The cell doctrine was most
,' :5e¡:ations inherent clearly articulated at the end of the next decade by Matthias Schleiden (r8o4-r88r;
': :r:parine specimens r8¡8) (Fig. 2.5) and Theodor Schwann (r8ro-r882; 1839; figure z.) and its impact on
- _r r1g. r.+, left) and the biological sciences was dramatic. However, it was not until the end of the nine-
¿ . ¿ll belore the human teenth century that Wilhelm von Waldeyer's Q$6-t9zr) neuron doctrine began to be
:ir :jte delelopment of widely accepted (Spillane r98r; Shepherdrygt; Finger r994).
In 1832, Jan Evangelista Purkinje or Purkyné GzSz-t86g) (Fig. 2.6, left) obtained
-t i:itit¡nS of cork. His an achromatic microscope made by Plóssl of Vienna. Purkinje directed it at the large
.. : I he also described cells in the cerebellum, thereby identif ing his eponyrnous cells' His microscopic
: :r¿-ke such a micro- observations were made before any adequate staining methods had been developed.
::¡,,a accountS Of Sev- Purkinje used alcohol to fix his preparations, and he made thin sections so that they
:i :ris simple magnifier could be examined microscopically. It is an irony of science that Purkinje was not
:-r¡lc rierr- that nerve able to resolve the amazingly detailed structure of the cells bearing his name. By the
:rrt )tgnals \fere propa- time Santiago Ramón y Cajal (úSz-tgZ+) (Fig. 2.6, right) was directing his micro-
:-: irom eye to brain scope at neural structures great strides had been made in both sectioning and staining
specimens. A single Purkinje cell sprouted arborizations of astonishing complexity.
NICHOLAS J. WADE
(1 918) embeddr
nera; the black
and was extens
cell. Right, Pu,rl
drawing of a slr
Fig. 2.5 Nicholas Wade, Ce// doctrine; Hooke's
(1 665) diagram of the cell structure of cork has had
incorporated within it the portraits of Schleiden The isolation ¡
(left) and Schwann (right). environment¿- ,
(oNicholas Wade)
Cajal expressed his artistry not only in observing the fine detail ofneural structures
Physiologl
but also in their depiction. He used the silver staining methods, which was developed The manner ir:
by camillo Golgi (1843-1926) (Fig. 2.6, centre) inú7. The method enabled both Golgi (tp7-r798;t,9t
and Cajal to see detailed neural structures that had otherwise been invisible or over- a discharge trc,:
looked (Mazzarello 1999;Mazzarello et al. zoo6). cajal examined the microanatomy leg and it nr-rt;l
of the retina, the cerebellum and the spinal cord, and made some improvements to fluid that accui
the Golgi technique. He used his histological studies of the brain to lend support Alessandro \-,¡-
to the emerging neuron doctrine, which was enunciated by waldeyer in rg9r. Two were flalred. r:r
views were entertained about the structure of the nervous system. On the one hand, Volta had inte:
Golgi argued that the nerves were connected in a vast reticulum, on the other that studies of ga1';:
the nerves were independent units which were not physically connected (see wade tion producec
and Piccolino zoo6). ing the pile or :
The structure of the senses soon came under the gazeof the microscopists and there the eyes, ears. i
was surprise at the variety of cells they beheld. The microscopes revealed specialized mouth and c..:
cells, called receptors, and they could be related to the stimuli that excited them. Those dark room. \Ic'
located in well-defined sense organs were named on the basis of their morphology and offset r¡f "l
(rods, cones, hair cells, etc.), whereas the receptors in or beneath the skin1a¡"r" g"r"i-
by rapid alter:r
ally named after those who first described them (e.g. Golgi tendon organs, Krause end When he applrt
bulbs, Meissner corpuscles, Merkel discs, pacinian corpuscles, and Ruffini cylinders). completed I l.i
THE SCIENCE AND ART OF THE SIXTH SENSE 3r
Fig.2.6 Nicholas wade, Left, Purkinie ce//s; Purkinje after a portrait in Purkinje
(1918) embedded in his representation of cells in the cerebellum. Centre, Reazione
nerai the black reaction transformed histological studies of the nervous system,
and was extensively used by Golgi, who is dimly discernable within a Purkinje
cell. Right, Purkinje ce//; Cajal after a portrait in Finger (1994) combined with his
drawing of a single Purkinje cell.
(oNicholas Wade)
s
as.ad
tf' The isolation of receptors that were specialized to respond to specific forms of
environmental energy was adopted as a criterion for defining the senses (Neff tg6o ) '
Physiology
:-- -: neu¡al structures
,. -.,.'-ici rras developed The manner in which the nerves themselves worked was hinted at by Luigi Galvani
j
: - -rabled both Goigi (tzll_lzg8;r79r) (Fig. z'7,|eft)when he made a case for.animal electricity'. He applied
:;"r ln,,-isible or over- a discharge from a Leyden jar to the exposed crural nerve or muscle of
an isolated frog's
i;ú 'lta microanatomy leg and ii t*itched. Galvani suggested that this was due to a special type of electrical
. :::: i:r:¡ror.ements to fluid that accumulates in the muscles of animals (see Piccolin o Lg97; Bresadola 1998).
:::-n :o iend support Alessandro Yolta Gz+s-r827) (Fig. 2.7, right) suggested that Galvani's experiments
"'
:,¡e', er in r89r, Two were flawed, maintaining that animal tissue was not necessary for a current to pass.
:i::1. r,ln the one hand, Volta had interests in the effects ofelectrical discharges on the senses; he carried out
..i:1. tr]] the other that studies of galvanic light figures in the r79os, and also found that intermittent stimula-
: :,:r::ected ( see Wade tion produced longer lasting effects than constant stimulation. In his letter describ-
ing the pile or battery, Volta (r8oo) described how he applied electrical stimulation to
:: ;:,-,scopists and there the eyes, ears, nose, and tongue. He connected the wires from a battery between the
.. :.-.'e¿led specialized mouth and conjunctiva of the eye, which resulted in the experience of light, even in a
¿: =-'¡;ired them. Those dark room. Moreover, he noted that the visual sensation was associated with the onset
¡ -: ::¡rr morphology and offset of the current, and a continuous impression of light could be produced
: .:-e ssjn \vere gener- by rapid alternation of polarity (see Piccolino 2ooo; Piccolino and Bresadola zoo3).
,l ar¡qans) Krause end When he applied current to the two ears he reported: 'At the moment the circuit was
a
r:¿ R.'¡nr¡i cylinders). completelifelt a shaking in the head' (Volta r8oo, p. +z).This shaking did not last
32 NICHOLAS I. WADE
It is in this cor
vestibular stim'¡
The abilin'to c.
history (Hamil:;
the eighteenth c,
For example. tr'-
Fig.2.7 Nicholas Wade, Left, Galvani,s legst a diagram from Galvani (1797) passive dimens-',
showing a wire connecting to a frog's reg has been multiplied to partially in philosoph.,'. ¡,
hide
Galvani's portrait. Right, volta's p/es; volta can be seen in the stacked philosophers. ,,-."
piles
of his battery. sense, and aske;':
(oNicholas Wade)
To what org¿1. -rl
long; when the current was continued. he experienced sound and
then noise. The sen- feelings of resisr:::
sations were so disagreeable that he thought them potentially dangerous, Our feeling o: ::.,
and he did
not wish to repeat the experiments. solidity, liquiii:,.
A few years earlier, volta had appried a current to his tongue and noted our muscular l:¡::
an acidic
taste (Piccolino ry97). Volta's pile did much to hasten experimental modifications .-i.,
studies of the
senses' Electricity was a common stimulus that could be
apptled to different sensory
organs' inducing different sensations. Müller used the effecti to
support his doctrine: It was the pe','
'The stimulus of electricity may serve as a second
example, of u .rrrjfor¡¡ cause giving to nominate hir
rise in different nerves ofsense to different sensations'ltg43lzoo3,p.
ro63). The first erior spinal nen'r
example was mechanical stimulation, as had been alluded
to by Galiieo. signals. lrloreoi'.
The action of nerves on muscles led first Carlo Matteucci (tárt_rSOz)
and later Emil muscles to the b¡,
du Bois Reymond (r8rg-rg96) to propose the ways in which nerves
propagate impulses 1863). In fact, Bill
(Brazier 1959, 1988) Experimental evidence of action
' potentials ús to await techno- earlier, in the c¡,:
logical advances in recording and amplifiring small eleclrical
signals; this was provided
byAdrian (1928) who was able to record action potentials (Finger zooo). There is an insep":
when record_
ings of nerve impulses could be made from individual
cells inlhe visual pathwaytheir
ofthe voluntar, r
adequate stimuli could be determined. Adrian coined the impression upr,r :.
term 'receptive field, to refer the office of t:e :¡
to this, and it was applied to other senses, too.
voluntan- mus¡-...
with developments in the microanatomy of the senses and the physiology the eve bv the r':-.
of their
responses to stimulation attention moved away from retina, rr.e shaL :.-
gross anatomy and phenomenol_
ogy' The criteria applied to separating the senses becÁe the quality the eve-ball be ::-,
of the experience,
the nature of the stimulus, the gross and microanatomy of the
receptor system, and
irithout, ¡re sh¿,- :
the pathways to and representation on the cortex. The psychologicaliimension
is the
oldest of these, and yet less weight has been attached to behaüoural
eüdence for distin- That is, the i::
guishing and adding to the senses than to that derived from
anatomy and physiology. alone, but alsc. i:r'
THE SCIENCE AND ART OF THE SIXTH SENSE 1t
It is in this context that a claim can be made that the systematic consequences of
vestibular stimulation preceded those for the muscle and skin senses.
MuscrE sENSE
The ability to detect the position of the limbs, even with the eyes closed' has a long
history (Hamilton 1846). The term 'Muskelsinn'had been used by German writers in
the eighteenth century, and it was suggested that the idea was described even earlier.
For example, ]ulius Casar Scaliger (r+8+-r5S8; t557) distinguished between active and
'Galvani (1i97) passive dimensions of touch. Appeals to muscular sensitivityhave been commonplace
ed to partially hide in philosophy, particularly among the empiricists. It proved central to common sense
:l'e stacked piles philosophers,too.Forexample,ThomasBrown (ry18-t8zo) suggesteditwasaseparate
sense, and asked:
(:Nicholas Wade)
To what organ, then, are we to ascribe the external influences, which give occasion to these
:.: ::'ien norse. Ihe sen_ feeiings ofresistance and extension? It is not touch, as I conceive, that either ofthese be traced.
' ¿:rE.rous, and
he did Our feeling of resistance, in all its varieties of hardness, softness' roughness, smoothness,
solidity, liquidity, &c. I consider as the result of organic affections, not tactual, but muscular;
our muscuiar frame being truly an organ of sense, that is affected in various ways, by various
"*e ¡nd noted an acidic modifications of external resistance to the effort of contraction.
:-:---.¡:tai studies of the
(Brown r8zo, PP. 7B-9)
,.::¡ dilterent sensory
:, ir:port his doctrine: It was the physiological dimension in a paper by Bell (1826) that led Boring (1942)
, ..nriirrm cause giving to nominate him as the founder of this new muscle sense. Bell argued that the ant-
: ,-r. :. ro6_¡). The first erior spinal nerve roots, which are involved in muscular contraction, also carrysensory
, ,r:l,ieo. signals. Moreover, a nelve circuit was proposed, which passed from the voluntary
,- : t': and later Emil
--- muscles to the brain. Muscle spindles were not isolated until four decades later (Kühne
-, :j rro:aqate impulses r86¡) . In fact, Bell had provided behavioural evidence for the muscular sense three years
-! ,u ¿a t.-t arrait techno- earlier, in the context of determining the visual direction of afterimages:
::.is: tnis rt-as provided
There is an inseparabie connection between the exercise ofthe sense ofvision and the exercise
.: r,- :, .\\-henrecord-
of the voluntary muscles of the eye. When an object is seen, we enjoy two senses; there is an
:-¡ -, ,su¿l pathl ay their impression upon the retina; but we receive also the idea of position or relation which it is not
::;-rtir,-e fi eld' to refer the office of the retina to give. It is by the consciousness of the degree of effort put upon the
voluntary muscles, that we know the relative position of an object to ourselves . . . If we move
:::. :¡l-,-siologv of their the eye by the voluntary muscles, while the impression [of an afterimage] continues on the
.;r'".:rd phenomenol- retina, we shall have the notion of place or relation raised in the mind; but if the motion of
.: ::-, lri the erperience, the eye-ball be produced by any other cause, by the involuntary muscles, or by pressure from
without, we shall have no corresponding change of sensation.
: ::a-ltor sr.stem, and
(Bell 1823, PP. 178-9)
- ¡:¡¿i dimension is the
-: - :ridence for distin- That is, the visual direction of an object is not determined by visual stimulation
::,, :n,,' and physiology. alone, but also involves information about the position of the eyes-otherwise objects
34 NICHOLAS I. WADE
would appear to move with every movement of the eyes. Helmholtz (rg67) also the heart of Ernst
referred to this distinction between what have become called outflow and inflow first monograph u
theories. The former refers to deriving the eye movement information from efferent judging weights h"
(centrally generated) impulses to the eye muscles, whereas the latter reflects use of
The weight of an .-i
afferent (sensory) signals from the eye muscles themselves.
by the special se:s.
It is surprising that Bell did not refer to the earlier experiments by William Charles of the muscle r,i:e:
wells (r757-r8r7) on this topic, because the monograph by wells (1792, reprinted weights of obie;r. .
in wade zoo3a) was referred to by Bell (r8o3) in the context of vertigo. wells had of touch, rvhile --L,
performed essentially the same experiment as Bell (rgz3) later did and reached a assumes, of cour...
similar conclusion: a certain press'lr;
when we seem :¡ ;i
When we have looked steadily for some time at the flame of a candle, or any other luminous
body, a coloured spot Iafterimage] will appear upon every object, to which we shortly after
direct our eyes, accompanying them in all their motions, and exactly covering the point, which In makins ;h-.
we desire to see the most accurately . . . The apparent situation of the spot being . . . at the his reliance on ::r
same time affected by the voluntarl, motions of the eye, it must, I think, be necessarily owing
iology (HE Rc''ss .;
to fhe action of the muscles by which these motions are performed . . . the apparent direction
sation being subr.
ofan object, which sends its picture to any given point ofthe retina, depends upon the state
of action existing at the same time in the muscles of the eye, and consequently that it cannot The combir¿:r
be altered, except by a change in the state ofthat action. energies doctri:¿
The muscles have no connection with each other, they are combined by the nerves; but these
nerves' instead of passing betwixt the muscles, interchange their fibres before their distribu-
tion to them, and by this means combine the muscles into classes. The question therefore may
thus be stated: why are nerves, whose office is to convey sensation, profusely given to muscles
in addition to those motor nerves which are given to excite their motions? and why do both
classes of muscular netves form plexus? To solve this question, we must determine whether
muscles have any other purpose to serve than merely to contract under the impulse of the
motor nerves . . . That we have a sense of the condition of the muscles, appears from this:
that we feel the effects of over exertion and weariness, and are excruciated by spasms, and feel
the irksomeness of continued position. We possess a power of weighing in the hand: what is
this but estimating the muscular force? We are sensible of the most minute changes of mus-
cular exertion, by which we know the position of the body and iimbs, when there is no other
means of knowledge open to us.
" IJelmholtz (1867) also the heart of Ernst Heinrich Weber's (vgs-t8ls) (Fig. 2.8, left) psychophysics. In his
il¿d outflow and inflow first monograph devoted to the sense of touch, Weber (1834) distinguished between
:,:,:*.,totr from efferent judging weights by touch alone or by the additional action of the muscle sense:
--:e latter reflects use
of The weight of an object is perceived in two ways: first by the touch-sense in the skin, and then
by the special sense of the voluntary muscles. The latter sense tells us the degree of tension
:.:tt-. br' \\'illiam Charles of the muscle when lifting weights and other objects. These two methods of discovering the
,- ¡",-eils (r792, reprinted weights of objects are very different: the former method depends upon the objective sense
:r: :i i-ertigo. Wells had of touch, while the latter depends on the subjective sense of muscular kinaesthesis. This
=i:i did and reached a assumes, of course, that we call a sense 'objective' when we use it to perceive objects that have
a certain pressure on our organs and produces some effect; and that we call it 'subjective'
when we seem to perceive only the effect of the objects and not the objects themselves.
:,". :: anr-other luminous (HE Ross and Murray 1978, p. 55)
-,- -,',1:i¡h rve shortly after
: -,', .¡lrg the point, which In making this distinction between objective and subjective, Weber is displaying
' -:.- ,;ot being . . . at the his reliance on the philosophy of Aristotle, rather than the then contemporary phys-
b¡ lecessarily owing
.-,,:-. .
iology (HE Ross 1999). Müller's doctrine of specific nerve energies was based on all sen-
-.lt: :lparent direction
sation being subjective, that is, not in perfect accord with the stimulus giving rise to it.
:. :.::rds upon the state
The combination of Bell's tentative hypothesis of a nervous circle, the specific nerve
-!: --r.ntl\' that it cannot
energies doctrine, and psychophysical studies of lifted weights confirmed for many
-,",':r's
r-9:, pp. 65,7o_r)
the force of the muscle sense as the sixth sense. By the end of the century, Charles
Sherrington Qs57_t952) (Fig. 2.8, right) was able to devote a chapter of a textbook to
.i.:. -r-'.'ed in the main-
- ..--: :inic of muscular
=;-'',: - the position of the
* ¡Ls ri9:) and Bell
- ..'-:::inents, but these
..... .nli"al disseclions
The excitation of the receptors of the proprio-ceptive field in contradistinction from those
of the extero-ceptive is related only secondarily to the agencies of the environment. The pro-
prio-ceptive receive their stimulation by some action, e.g. a muscular contraction, which was
itself a primary reaction to excitation of a surface receptor by the environment.
(Sherrington ryo6 I zooo, p. 4o)
Thus, the contention that the muscle sense is the sixth sense was reasonably
Fig.2.9 Nicholas
well supported by phenomenology, physiology, and psychophysics in the nine-
superimposed on c
teenth century.
Lichfield. Centre, I
instruments for m
enclosed in a patl
se \\'as reasonably Fig. 2.9 Nicholas Wade, Left, Doctor Darwini Portrait of Erasmus Darwin
,'sics in the nine- superimposed on one of the paving stones in the garden at Erasmus Darwin House,
Lichfield. Centre, Temperature sensitivitq, Blix is shown together with some of his
instruments for measuring sensitivity to temperature. Right, Sensory spofs;von Frey is
enclosed in a pattern derived from the distribution of pain spots on the skin surface.
(oNichoias Wade)
tongue and silver on the gums, that on the tongue feels very clearly warm, but it feels
cold with silver in the same arrangement' (p. +is). That is, stimulation by the positive
pole produced the sensation of warmth whereas the negative pole resulted in experi-
-:-- :ereated up to the encing cold. Slightly earlier in the same year,Pfaff (r8or) had described the sensation
..- ,:pport (see Finger of coldness when he applied a current to his finger. Ritter (t8ol) extended the studies
-,*.: 1\-e perceive not on temperature sensitivity on the tongue as well as the finger; he found that the sen-
:: ;- rrf them are heat sation could vary according to the intensity and duration of the current. His general
-:-:¿. solidit\., motion, conclusion was that: 'one must consider the sense of temperature (for warmth and
:: i-stinction between cold) as essentially different from the common sense, and as a special sense' (p. ro)'
:i'. tr t-,jt-18o2, t794) Galvanic stimulation resulted in a short shock well as the particular sensation. In
as
..,rp:essure but did the case of temperature sensitivity, Ritter reported that the shock remained constant
:- ..¡:
and temperature even when the sensation changed from warm to cold' Rather than merely speculating
:---;. --r. lerr'r'ears later, that warmth and cold are separate sensory qualities, Ritter afforded experimental evi-
dence for this via his studies of galvanic stimulation'
.-:,.:.::- Weber (18+6) also followed Volta's lead in applying electric currents to the sense
^r¡ds: hardnesS,
. :: - -:i reat be a qual- organs, although he was critical of Ritter's work. He added little to what was known at
r :',. .ti¿Í and cold are that time about galvanic stimulation, but he did conduct experiments that supported
: ,.:,,- , I :..Uch iS said tO the existence of a temperature sense: 'The sensations of warmth and cold are not like
the sensations of brightness and darkness, for the former are positive and negative
@. +zz) quantities between which lies a null point determined by the source of heat within
.- - - : 1 l us' (HE Ross and Murray 1978,p. zro). Weber's great contribution was the introduc-
:--:: - using galvanic tion of experimental methods, like determining two-point thresholds, which enabled
:.,-::u and its general
'. quantification of sensitivity over the skin surface (Weber r8:+). These could then be
--: . -:tt.\t of German applied to establish acuity differences over the skin surface, and interpreted in terms
t::., -'t¡r¡]¡, bUt he did of regions of receptiveness (Weber 1846). Furthermore' Weber suggested that the
:::; hts sense organs. sensory circles could be related to the underlying nerve supply:
: .:a: aontrast in sen- But no matter how the elementary nerves do extend to cover the skin, the suggestion may be
.:.:i : zinc pole on the put for-ward that the skin is divided into small sensory circles, i.e., into small subdivisions each
t 38 NICHOLAS J. WADE
-:r ;I\- lltv€stigations have The phenomenological distinctions between the dimensions of touch, voiced since
:is ',\ithin a single sensory
antiquity, were given some empirical support from the late eighteenth century and
. !it¿: and moreover, that
integrated with cutaneous anatomy and physiology in the late nineteenth century.
: :n accurate touch_sense
provoke an unequal, tumultuous and disorderly flow of humors and pneuma. Therefore it is conditions that ¡a
only natural that people subject to skotoma are on guard against any motion of this kind. andWillis interpi:
(Siegel r97o, p. r3B) produced the a:p
behind that of tr¡
Galen was probably describing a disorder that is now called Méniére's disease,
continues after t ¡
after it was linked to disease of the vestibular system by Prosper Méniére (rygg-
of the animal spr:
186Z; 186r). Developments did occur in fusing Aristotle's account of the senses with
of it in his Oxfbr¿
Galen's pneumatic physiology, and the medical tradition of describing diseases of the
senses became more refined. Conditions like visual vertigo were related to the move- Vertigo arises tr¡::
ments of the animal spirit in the head. Generating visual vertigo, even in healthy brain and its m¡c:-
individuals, by rotating the body and then stopping, was a source of regular reflection phial are sent int. l
the smoke or \',1r, *
by medieval medical and optical writers.
therefore fleribi: .
Two great anatomists of the brain, Felix Platter (t96-t6al (Fig' z.lo, left) and .:
sensation persis:s
Thomas Willis (16zr-t67) $ig. z.ro, right) heralded the early modern era of research
on the movement sense. They each suggested a mechanistic interpretation for vertigo
in terms of motion of the animal spirit in the brain. Platter (1583) observed that: So little rr¿s :::,
was long heli. :',
An intense, uniform, and extended movement of the head transfers itself in a similar way to
the spiritus. Despite holding the head still afterwards, it appears to continue moving for a of the explana:r--:
while, before it eventually feels still. This is the basis for dizziness, if one rotates the head and the interpretati,--:.
body in a circle for a long time. rather than anii:.
(Koelbing ry67,p.89) In the eight¿::
r.ertigo in his ci¿..
Willis defined vertigo as 'an affection in which visible objects appear to rotate'
(t672,p.353), and devoted a chapter ofhis book to describing its pathology and the an hallucinatit r: '.",
aroundus...T:-:
equivalent to th¿. .
that membran;.
He drerv paraL.
and suggested tir
ments of blor.i -:r
and the possib:,-:,
Ar alternaii"'¿
phenomenon ..- -,
cal, and most c'f ..
11774-7766, incl;;
Fig.2.1O Nicholas Wade, Left, Platter's brairythe diagram of the brain is taken
from Platter (tSAg) and the portrait is derived from Freher (1688). Right, Circle
of Willis; Christopher Wren made the illustration of the base of the brain printed
in Willis (1672), and the portrait was derived from Birch (1743).
(oNicholas Wade)
THE SCIENCE AND ART OF THE SIXTH SENSE 41
:il;uma. Therefore it is
conditions that can induce it, including body rotation in healthy individuals. Platter
:-,-,trLrn of this kind.
and Willis interpreted vertigo in Galenic terms: motion of the animal spirit in the head
Siegel r97o, p. r38)
produced the apparent motion during rotation, rather like smoke in a flask lagging
-¿ \Ieniére's disease, behind that of the rotating vessel. Moreover, Willis described the visual motion that
:per \féniére (ry99- continues after body rotation ceases, and this was attributed to the continued motions
*:-t trf the senses with of the animal spirit relative to the stationary head. Willis gave a graphic description
¡:r--ine diseases of the of it in his Oxford lectures:
: ::,ated to the move- Vertigo arises from the circular motion of the spirits, and, as it were, their rotations in the
:ri.,. e\-en in healthy brain and its medullary part. it takes place just as smoke and vapour contained in a glass or
:: -: reqular reflection phial are sent into similar motion if you spin the vessel round. This motion lasts longer in
the smoke or vapour than in the vessel. Thus we find people whose spirits are very thin, and
:iq. :.i0, left) and therefore flexible and weak, pass into vertigo as soon as the body or head is rotated and this
: :;::r era ofresearch sensation persists after the body has ceased its turning motion.
He drew parallels between vertigo and visual persistence with rapidly moving lights,
and suggested that the sensitivity of the retina was changed by the retrograde move-
ments of blood in the vessels supplying it. He did discuss the effects of body rotation,
and the possibility of unconscious eye movements was entertained.
An alternative to speculating on processes in the retina or brain was to study the
phenomenon of vertigo itself. Eighteenth-century interest in it was principally medi-
cal, and most observations on it were made in that context. For example, Robert Wh¡t
(t7t4-t766) included giddiness amongst the symptoms for nervous diseases:
Many people of a delicate, nerr'ous, and vascular system, after stooping and suddenly raising
their head, are apt to be seized with a wrtigo, which is sometimes accompanied by faintness.
In this case, the vessels of the braiir being too weak, seem to yield more than usual to the
weight of the blood, when the head is inclined; and after-wards, when it is suddenly raised, and
the blood at once descends towards the heart, those vessels do not contract tast enough, so as
.r: brain is taken to accommodate themselves to the quantity of blood remaining in them: At the same time
ie8l, Right, Circle the brain, on account of its too great sensibility, is more affected than usual, by any sudden
f :ne brain printed change in the motion of the fluids through its vessels.
lt, (Whitt 1765, p. 3o9)
: N cholas Wade)
42 NICHOLAS I. WADE
Diseases of the inner ear were discussed by Bell (r8o3/zooo), but their association body rotation re
with vertigo was not explicitly entertained. While he mentioned that 'Of the diseases these aspects tha:
of the labyrinth, there is little on record' (p. +ir), he did observe that inflammation this regard that t
around the auditory nerve was accompanied by an increased sensitivity to slight mentally. The ir
head movements and to vertigo. The paradox of these investigations is that the gross were not relatei
anatomy of the labyrinthine organs was reasonably well known at that time. they recognized :
At the beginning of the nineteenth century, galvanic stimulation was applied to the
The histon' o¡',' i
regions around the ears and provided some indication that more than hearing was
tion, giddiness. i,
involved in the structures of the inner ear. As was noted earlier, Volta (t8oo) reported is voluminous ¡r
that his head seemed to be shaking when current was applied to his ears. Ritter (r8or) it: in rgzz Grri-i:
described the dizziness generated by experiments on applying galvanic stimulation to about Purkinie .
'
the head. A similar account was given by Augustin (r8o3): 'If one surrounds the ears semicircular ;.n ,.
with wire . . . one becomes dizzy and sees electrical lights' (p.pg). Purkinje (r8zo) r83o), the \fac:,-l
(r87-t875). ¿i--i -.i
carried out further studies on galvanic stimulation of the ear and the subsequent ver-
(r9rB), Griflr:: ::
tigo that it induced. He constructed a voltaic pile from twenty zinc and copper pairs
and applied the current to the ear. The immediate sensations were of light flashes and
a metallic taste, and then he reported feeling dizzy. It was like a motion from ear to ear, The histon,'r.
and its direction depended on the polarity of stimulation. He felt nauseous following who was al'¿:l¡
ten minutes continuous stimulation, and experienced after effects for the following its histon'. He :¡
two hours. These sensations could only be produced when the current was applied to tions on ver-tiq¡
the ears; similar application elsewhere on the head did not produce vertigo. Purkinje dismissed as ¿c¡
extended his observations in a later article:
The realitv is ri^..:
The direction of the rotary motion from vertigo goes from right to left if the copper pole is in fully accustonec.
the right ear, and the zinc pole is in the left, and in the opposite direction from left to right, if his predecessc.:' .
the copper pole is applied to the left and the zinc pole to the right ear. As often as the galvanic regarding the r:-:,
current is alternated, the vertigo is experienced in the opposite direction and lasts for a longer whether it is e;;:,:
or shorter time according to the longer or shorter application. rotate themseir ¡.
(Purkinje ú27, p. 297) to rotate and trb.¡
at the writing :a':,
More systematic investigations were conducted by Eduard Hitzig (r838-r9o7). In in 1825.
examining the effects ofvestibular stimulation, Hitzig(r87r) applied electrical currents
between the mastoid bones and recorded not only the direction of apparent visual
motion but of actual body and eye movements. When the head moved in one direction Both Bárar:','
the eyes moved in the opposite direction. The actual and apparent movements of the Pierre Fioure::s
body were in the same direction. Hitzig found that the effects of galvanic stimulation (Fig. z.u,lelt . r:
were more pronounced when they were applied with the head tilted, and that it was (r838-r9zz t Fi:
difficult to maintain balance under these conditions. Two blind subjects felt that their accuracy of G:::
bodies were rotating when the current was applied, as did sighted subjects with their 1974). Perhaps :
eyes closed. amplified in h-.
commenced:
The reality is that they all say much the same thing. In the Middle Ages one had become
: ,: -.-. ..'pper pole is in fully accustomed to the complete description. Whoever wrote a book studied the texts of
- - -:-,:r lelt to right, if his predecessors and wrote more or less the same thing with small variations. For example,
--.; ,:i:: ar rhe gah'anic regarding the interesting question of vertigo from rotation, many authors have speculated
- .:: -:.ts ibr a longer whether it is accompanied by unconscious eye movements. It did not occur to any of them to
rotate themselves a few times and to feel if their eyes were moving, or to ask his good friend
: :.,nle iS:;, p. 297) to rotate and observe his eyes. The often insightful considerations would only be carried out
at the writing table. The first to make the observations that will be discussed here was Purkinje
i-::-¡ i3_r8-t9o7). In in 1825.
-. i :-etrical currents (Bár ány t 9t3, pp. 39 6-7)
:: -l:lparent visual
. , .i -r rrne direction Both Bárány and Boring were correct in citing the physiological experiments of
:: ::-t-',r,-aments of the Pierre Flourens (ry94-r867), and the hydrodpamic theory of Ernst Mach (1838-rg16)
stimulation (Fig. z.u, left), Josef Breuer (1842-1925) (Fig.2.r1, centre), andAlexander Crum Brown
=.--" i11¿
'':.1. :nd that it was (ú38-r9zz) (Fig. z.n, right). However, Boring placed undue reliance on the historical
"i'.¡t s tllt that their accuracy of Griffith's (r9zz) monograph, as have others (see Wendt r95r; Kornhuber
; ,ub;;cts uith their ry74). Perhaps all of them were in thrall of Mach's historical authority, which was
amplified in his book on movement perception (Mach 1875; Young et al. zoor).Mach
commenced:
The work before you attempts, for the first time, to present a complete chapter of physi-
ology, to which the incontestable Purkinje (Purkyné), Flourens and Goltz have laid the
foundations . . . The elder Darwin and Purkyné have studied the remarkable subjective sensa-
.¡ :: seeinq or hear- tions of rotation that take place if one rotates rapidly several times and then stops suddenly.
: i,rr-rsir¡ns based on (Mach 1825, pp. iii and r)
44 NICHOLAS ]. WADE
Objects about
ues, not onh' rthe r
continues for
absolute Rest.
The evidence i
was not conscirru
remained station
he distrusted úe
science, and he :
of their incre¿s¿
Darwin to deli¡e
dispute rvith Dar
Fig.2.11 Nicholas Wade, Left, Mach's chair; Mach used the rotating chair to following bod''' ¡¡
examine vertigo. Centre, Breuer's pigeons; the influence of semicircular canals on Wells tu9:. --
the posture of pigeons was confirmed by Breuer. Right, Crum Brown's nystagmusi tigo and nlsiaErl
Crum Brown was able to record the slow movements of the eyes and their rapid be considered :i
return (nystagmus) during body rotation, as well as the adaptation that occurs; His experimen;.
the sawtooth patterns are derived from Crum Brown's (1878b) diagrams. of a ner,v sense is-
(oNicholas Wade)
do not act upLrt :
liar kind of sers-i
external cause> :l
Crum Brown (r878a) similarly surveyed the past in Purkinje's favour. With regard to
both of the bod-'
the aftereffects of body rotation he wrote:
A common -¿
Purkinje studied the conditions under which this apparent rotation occurs, and arrived at images to asse:s
the following conclusions, which have been confirmed by all succeeding observers: r. That describe alter:r
the direction of apparent motion of surrounding objects depends upon the direction of the son and the ta:ht
preceding real motion of our body, and is always opposite to it. z. That the axis about which
likely that the R:
the apparent motion takes place is always that line in the head which was the axis of the pre-
Erasmus (\\-aJ:
ceding real rotation.
during postrrrl¿:
(p. 6:+)
a slight fit ot g-;
In a second article by Crum Brown (r878b), mention is made of Erasmus Darwin's occasionedbr L¡,
investigations of body rotation; post-rotational nystagmus is both described and moment to b<::l
illustrated, but again its initial observation is credited to Purkinje. Darwin was also of the positirlns
mentioned by Griffith Qgzz), Boring (ry42), and Cohen (rg8+), but they did not on this happr a;
recount the reasons why he chose to carry out his studies. It was William Porterfield mO\'€ITI€ntS ttr il
(cú96-tzzt) whose speculations regarding the link between eye movements and fast and siort ::
post-rotational visual motion stimulated renewed interest in the visual dimension of time (Tatler ¿ni
vertigo in the late eighteenth century (see Wad e zooTb) . Motion was the last of the phe- ¿.r..i$sd htl-'r :
nomena of vision described in the second volume of Porterfield's Treatise on the Eye, head positior i
the Manner and Phcenomena of Vision, and his analysis of it was subtle. Vertigo was the rotation and s':
final phenomenon discussed in the final section: see,that. aitht-,u
sockets. ñrst :,:-,
But, before I dismiss this Subject, I shall endeavour to explain another Phcenomenon of
Motion, which, tho' very common, and well known, yet, so far as I know, has not as yet space of a lerr :
had any Solution given to it. If a Person turns swiftly round, without changing his Place, all as the\'?rB €\lli¡
THE SCIENCE AND ART OF THE SIXTH SENSE 45
Objects about will seem to move in a Circle to the contrary Way, and the Deception contin-
ues, not only when the Person himself moves round, but, which is more surprising, it also
continues for some time after he stops moving, when the Eye, as well as the Objects, are at
absolute Rest.
(Porterfield 7759, pp. 424-5)
The evidence that the eyes do not move following rotation was subjective. Porterfield
was not conscious of any movements of his eyes and so he was convinced that they
remained stationary following rotation. The situation was clarified by Wells Gzgz);
he distrusted the recourse to subjective experience in deciding upon a matter of
science, and he found that experiments with afterimages were preferable because
of their increased objectivity. It was Wells's monograph that galvanized Erasmus
Darwin to deliberate further on vertigo, and it was Wells who engaged in a public
dispute with Darwin concerning the involvement of eye movements in visual vertigo
'olat ng chair to
following body rotation (see Wade zoqa).
ricircular canals on
Wells (1792, t794a,b) conducted sophisticated experiments on postrotational ver-
Bra¡vn's nystagmus;
tigo and nystagmus long before Purkinje's studies. Wells's analysis of vertigo should
Its and their rapid be considered as heralding the first clear behavioural evidence for the vestibular sense.
:a: on that occurs;
His experiments satisfied Müller's G8+llzoo) requirement: 'The essential attribute
lJ ciagrams.
of a new sense is, not the perception of external objects or influences which ordinarily
! Nicholas Wade)
do not act upon the senses, but that external causes should excite in it a new and pecu-
liar kind of sensation different from all the sensations of our five senses' (p. ro87). The
:¿', - Li:. \\-ith regard to external causes are linear and angular accelerations, and the sensation is one ofrotation
both of the body and the üsual scene.
A common feature of many of Wells' experiments on vision was the use of after-
. ,:...:s. ¿nd a¡rived at images to assess the manner in which the eyes moved. He used the term 'spectra' to
:'sen'ers: r. That
describe afterimages; they were so called by Robert Darwin (t166-t8+8), Erasmus's
. drrection of the
son and the father of Charles, in an article a fewyears earlier (R. Darwin 1786). It seems
likely that the Robert's article received more that a little assistance in its writing from
Erasmus (Wade zooz)l Wells enlisted afterimages to determine how the eyes move
t.P, 6l+)
during postrotational vertigo, although his initial observation was accidental: 'During
a slight fit of giddiness I was accidentally seized with,
a coloured spot [afterimage],
i::smus Danvin's occasioned by looking steadily at a luminous body, and upon which I happened at that
:-r iescribed and moment to be making an experiment, was moved in a manner altogether independent
l¿nrrn rr-as also of the positions I conceived my eyes to possess' (Wells 1792, P.qr). Wells capitalized
.-rr thev did not on this huppy accident and provided experimental evidence to link the pattern of eye
movements to the direction of visual vertigo. He gave the first clear description of the
fast and slow phases of postrotational nystagmus, and its decreasing amplitude with
time (Tatler and Wade zoo3; Wade and Tatler zoo5; Wade zooTc). Furthermore, he
described how the direction of postrotational afterimage motion was dependent on
head position during rotation. Wells was not aware of feeling his eyes moving after
rotation and so he asked another person to rotate and then stop 'and I could plainly
see, that, although he thought his eyes were fixed, they were in reality moving in their
----:: Pl',,trtotttenon of sockets, first toward one side, and then toward the other' (Wells 7792, P.97). In the
.:- r,. has not as yet space of a few pages, Wells encapsulated the essential features of vestibular function
:-.rE-n¡ his Piace, all as they are expressed through eye movements and postrotational vertigo.
46 NICHOLAS I. WADE
Purkinje (Fig. z.tz, left) unknowingly repeated many of Wells' experiments on influence exertei
body rotation, although he was able to add a mechanically rotating device to study ment that was sc'¡
vertigo. Initially Purkinje examined the introspective aspects of postrotational vertigo movements in tl:
and made many experimental manipulations of it. He described rolary and postrota- Purkinje later r,,-:
tional eye movements and suggested that 'visual vertigo is a consequence of the conflict not change subst
between unconscious involuntary muscular actions and voluntary conscious ones in added to Wells .
the opposite direction' (Purkinje r8zo, p. gl ) . Among the few sources of earlier research
he cited were a translation into German of Erasmus Darwin's Zoonomia (Darwin 1795)
and Herz's (1786) medical text on dizziness and its treatment. Purkinje deduced a gen-
eral principle from his experiments: 'that the midpoint of the head (considered as a Vestibular s¡
sphere), around which the initial rotation was performed, invariably determined the
The significanc:
direction of apparent motion regardless of the subsequent position of the head' (r8zo,
slowly emerged a
p. 86). Kruta (1964) referred to this as 'Purkinje's law of vertigo'. There was no clear
and later on the s
indication of how such motions in the head could be detected, and his initial interpre-
bookwas publs:
tation was that motion ofthe brain itself lagged behind that of the head, with particular
On r5 Novemb:: -
immediatelr'J¡, .
the turning oi th.
In later expe:r
circular canal :-:
and equilibriu::r
tion ofthe seve:.
this experimen:;
function and i::,
Breuer, and Cr:r
head rotation :r
ling angular acc.
Mach (r8:-r. :
could also rotate
and also r-isua. :
he concluded ¡:
tion. Breuer r!-
dogs; he also c.s
vestibular sr-s:e:
(1874, 1875 ) ba.:
ing stool; the -,r:
Fig.2.12 Nicholas Wade, Left, Purkinje's schaukel; Purkinje and a rotating device semicircular '¿:
like that used in his experiments on the effects of body rotation. Right, Bárány's in the conter: ¡ -
chairi Bárány devised a rotating chair for studying semicircular canal function.
It was very similar to the chairs devised for the treatment of maniacs early in But at times sc,:-,
skth sense e¡,is:.,
the nineteenth century; the one shown here was derived from an illustration in
there rvas a tir-:.
Hallaran (1818). any one rvill la. r:
(oNicholas Wade)
we have been c¡,:
THF, SCIENCE AND ART OF THE SIXTH SENSE 47
ll'e11s' experiments on
influence exerted by the cerebellum. Purkinje concluded his first article with a state-
rtaring device to study ment that was soon to be realized: 'It remains for a future work to establish the possible
itrostrotational vertigo
movements in the brain which measure its structure and organization' (r8zo, p. 125).
:C rtrtan' and postrota-
Purkinje later wrote several briefer articles on vertigo, but his interpretation of it did
!ecuence ofthe conflict
not change substantially (see Wade and BroZek zooi). The dimension that Purkinje
:an- conscious ones in
added to Wells' studies was the application of galvanic stimulation to the ears.
l:¡es of earlier research
: t,iú ttt i a (Darwin 1795)
the name disappears. It was further said to me that animals exist which have no labyrinth, but clinical index or i
which can yet orientate themselves, and that consequently the labpinth has nothing to do chair. This is pu;
with orientation. We do not walk forsooth with our legs, because snakes can propel themselves earlier in order t
without them! But if the promulgator of a new idea cannot hope for any great pleasure from refined a calori. I
its publication, yet the critical process which his views undergo is extremely helpful to the ments they intl:i
subject-matter of them.
relationship of ',-.
(p.zgz) ofresearch on l:¡
Mach, Breuer, and Crum Brown continued to investigate the consequences of though Mach r,-;
the hydrodynamic theory of semicircular canal function, but Crum Brown, in 1878, ments by Pur[r
made a particularly astute prediction: if deaf-mutes have defects in all the parts of the Mach, Breuer. :;
inner ear, then they will not be able to experience vertigo nor have the eye movements (186r). He sum::r
: ::,": no labyrinth, but clinical index ofvestibular function. Indeed, the rotating chair is now called the Bárány
" -.:: ias nothing to do chair. This is puzzling because essentially similar devices were employed a century
', -::l :ropel themselves earlier in order to treat the mentally ill (Wade zoo5c; Wade et al. zoo5).In 19o6 he
::.1 ;:e¿, pleasure from
refined a caloric method of stimulating the semicircular canals so that the eye move-
..:::::t;-..' helpful to the
ments they induce could be easily observed. Bárány (r9r3) wrote two articles on the
relationship of vertigo to the semicircular canals, and these also suweyed the history
Q.zgz) of research on the topics. His history fell short of the standard set byMach (1875), even
:jt. ,onseqltences of though Mach was cited! Nonetheless, in addition to the descriptions of the experi-
-:*r: B¡orr-n, in r87g, ments by Purkinje and Flourens, as well as outlining the hydrodynamic theory of
.-r :-l the parts of the Mach, Breuer, and Crum Brown, Bárány integrated the clinical studies of Méniére
': ::a el-e mo\.ements (t86r). He summarized their endeavours with a musical allusion:
PurkinjeandFlourens(1825-1828)havedevisedaningeniousprelude, Méniére(186r)hasintro-
.':-::: rte delicacy and duced the first phrase, Goltz QSTo) addressed a new theme, and the theory of Mach, Breuer,
and Crum Brown concfuded the movement, with a powerful, full-bodied accord. The second
= :. : -.::l| the cochlea,
- : -:--rr :stablishments movement is like a fugue, where many voices whirl around one another, at one time with har-
:,: l::r,;n made upon monic combination, and a moment later they are in conflict.
..=..... :l::nt, and if the (r9r3, p. 399, original italics)
_.-- -]a¡Jde¿t_mute,
.:- ',, -'d do more to Bárány standardizedaform of stimulating the vestibular system that did not involve
.:-:.. -,1:¿n
anv number rotation-irrigation of the with warm and cold water. The procedure was known
ear
about and had been applied for many years; Goltz (r87o) referred to it as common
: 'r :. ,-,:iginal italics) knowledge. This is now called caloric stimulation and it is a standard clinical test for
vestibular function.
--. :, -.i:ucted device Vertigo was considered to provide insights into motion illusions generally by those
' ;::-il'. HOrt.er.ef, Of
with a psychological interest. For physicians it was seen as a useful tool for determining
.. ]: results were
the integrity ofvestibular function. These two activities became more closely integrated
.-:. r-,. ceai individu-
with the perceptual demands of first aircraft and then space flight in the twentieth
::: : :1\ Stagmic eye
_)
century. That is, there were practical applications of the knowledge derived from
:. :r--,',-rdedagen_
.,
both experimental and medical research on vertigo.
. l{e related
--::--. -,_ai
-¡j:-;::a n\-Stagmus.
, -;:t¿n'because'it
',',
:1ls r;g+b) had Visual orientation
:- '::-tis related the Mach extended his own research to examine visual orientation during body tilt, as well
. .:.i., sis of motion as visual motion following body rotation. He was able to use his rotating chair and to
: t-:g : ob.iects are exclude the visibility of the surround. His research on orientation was stimulated by
- --:túer3d to be still an experience of visual disorientation when travelling in a vehicle:
---. ._r :Io\-e. ,ames
Thus my attention was drawn to this point by the sensation of falling and subsequently by
=. l.:aiir'.
another singular occurrence. I was rounding a sharp raiiway curve once when I suddenly saw
all the trees, houses, and factory chimneys along the track swerve from the vertical and assume
a strikingly inclined position. What had hitherto appeared to me perfectly natural, namely,
the fact that we distinguish the vertical so perfectly and sharply from every other direction,
now struck me as enigmatical. Why is it that the same direction caú now appear vertical to me
t:,:::rite d the medi- and now cannot? By what is the vertical distinguished for us?
:'' r :loYements aS a (Mach r9ro, pp.286-7)
5o NICHOLAS I. WADE
Mach appreciated that judgements of orientation are made with respect to frames search for a sL¡
of reference. Normally those available from the senses correspond with the cardinal senses and mu
directions defined by gravity, but occasionally this accord is disrupted. Mach did have gradually emer
recourse to the structures within the inner ear-the 'secret ¡sfs¡s¡6s'-that could external stimr¡l
'indicate these positions of the body', and he conducted experiments with his tilting senses offeelin
chair to confirm it. the dependenc,
Wells (1792) was similarly searching for the sixth sense that occupied many scholars tion provided ,
throughout the nineteenth century, and he also discussed the manner in which we touch from ter
can orient ourbodieswith respectto gravity, andhowour judgements ofvisual orienta- vanic stimulaii
tion are influenced by changes in bodyposture. He was aware, on theoretical grounds, The success o:
that there must be some system that registers the position of the body with respect to physiologr-. a:
gravity: 'In the estimates we make by sight of the situation of external objects, we have guishing ben' t
always some secret reference to the position of our own bodies, with respect to the to include rnic
plane of the horizon' (ry92,p.85). Although he asked'What is there within us, to indi- stimuiation a¡
cate these positions of the body?' (p. 86), he could not answer to his satisfaction, since ioural measure
he could only draw on the actions of the voluntary muscles. Thus, the orientational anatomv ancl p
aspects of otolithic function were examined behaviourally, as well as the rotational Studies of tl
consequences of semicircular canal stimulation. our understar
The receptors that mediate vestibular sensitivity are closely linked to those for
hearing. Hair cells in the cochlea were first observed in the r85os, and they were later
identified in the vestibular system (see Finger ry94).In the twentieth century, the fine
detail of the hair-cell receptors could be observed with electron microscopes and a w
cortical projection from the vestibular nuclei was demonstrated. The vestibular sense sf
xl
is unusual in various respects. First, the sensory experiences following stimulation \dtl
are not localized as they are with the other senses; we feel giddy or see the world spin SE
rather than have a single sensation like sight or hearing. Secondly, the gross anatomy K.T
of the vestibular system was known long before its function was appreciated. Thirdly,
w
st
systematic evidence indicating the action of the semicircular canals (in vertigo) was KT
available from the late eighteenth century. Nonetheless, behavioural studies which pro- VI
vided support for a new sense were not accorded the status given to isolating specific SE
x.t
receptors or establishing projections to the brain. It was the behavioural dimension YI
that encouraged Crum Brown (1895) to state: 'I am not sure whether in this account st
of the sense of rotation, of its organ, and of the use of it, I have carried all my hear- xl
YT
ers with me, and convinced you of the real existence and the real practical use of this
SE
sense'(p. z8). KI
VT
SE
KC
v¡
st
CoNcrusroN K.n
vt
sÉ
Kf
Aristotle restricted the number of senses to five for theoretical rather than empirical
reasons, although the sense of touch or feeling remained an enigma. The subsequent Fig.
THE SCIENCE AND ART OF THE SIXTH SENSE 57
- i:e subsequent
Fi1.2.13 Nicholas Wade, Ihe sixth sense.
[oNicholas Wade)
52 NICHOLAS ]. WADE
The gross anatomy of the vestibular system was described before its function was Bell C (rsz6). On the :
appreciated; the visual consequences of vestibular stimulation were subjected to obser- Philo soPhical T r,i'' :.;
Seil C (rs++). The at;'¡:'
vation and experiment between these two events. Rotating the body to induce ver-
|ohn Murra,v, Lonc:
tigo resulted in postrotational nystagmus and apparent visual motion, the directions
Berkeiey G (r7o9 ). -{'; .
of which were dependent upon head orientation during rotation. Applying electric Birch T (ry43'). The i;:.;
currents to the outer ears produced feelings of dizziness, as well as movements of Blix M (1884). Eraerir
the body and of the eyes. These responses to rotation and galvanic stimulation could Hautnerven. Z.'i ¡-'¡ ;:
be understood when the hydrodynamic theory of semicircular canal function was Boissier de Sauvages F
advanced in the r87os. Moreover, the absence of these responses in many who were en genres et eil ¿:;'.:
Lyon.
deaf and dumb added to the evidence for a separate sense.
Boring EG (r94zl. -';':-'
In an historical sense, it is difficult to imagine originating a classification of the
Century, Nel'Yoii
senses that did not depend on anatomical and perceptual distinctions. Such a clas- Bnzier MAB (rg;q .Il
sification would have preceded others based on energy because the characteristics of VE Hall (eds ;:.;
perception were described long before there was an adequate understanding ofenergy Physiological Str;1.:
sources in the environment. These categories were later reinforced by evidence from .Brazier MAB (tg8S 'r.
neuroanatomy and neurophysiology: specialized receptors respond to features ofthe BresadolaM(rg9S..
stimulus and these are analysed in discrete regions of the brain. Research Bullerir, l'
Breuer ] (r87+). L're :
The number of senses has always been arbitrary, depending on the criteria that
Jahrbuch, 4,;)-1' '
are applied. Adding one or more senses to Aristotle's five reflected the advances that
Brown T (r8zo ). -sr¡¿:-';
took place in nineteenth-century neuroscience. Despite these advances in science, the & Miller, and \\ -r:
restriction of the senses remains fixed at five in both our language and our art. Perhaps CohenB (1984). Era.i:
this is an instance in which art has dominated science! 3, tzt-9.
Cranefield PF 1r9-: .
Beli C (r8rr). Idea of a new anatomy of the brain; submitted for the observations of hk friends. Vol. VL Cert-. F¡:
Published by the author; printed by Strahan and Preston, London. (Reprinted in NJ Wade Descartes R (166::;
(ed.) (zooo). The emergence of neuroscience in the nineteenth century. Vol. l. Routledge/ Descartes, \'oi. -\I
Thoemmes, London.) Dewhurst K (r9Ec
Bell C (1823). On the motions of the eye, in illustration of the uses of the muscles and of the Donaldson HH t::'
orbit. Philosophical Transactions of the Royal Society, u3, 166-86. Drake S (1957). Dr-''-'
THE SCIENCE AND ART OF THE SIXTH SENSE 53
::, :: lts funCtiOn WaS Bell C (t826). On the nervous circle which connects the voluntary muscles with the brain.
::; i:bjected to obser- Philosophical Transactions of the Royal Society, n6, 163-73.
. :"-,.ir- to induce ver- Bell C (rs++). The anatomy and philosophy of expression as connected with the fine arts,3'd edn.
:r '-L.r:l, the directions fohn Murray, London
::. -{pplr'rng electric Berkeley G (r7o9). An essay towards a new theory of vision. Pepyat, Dublin.
,.-i :s mor.ements of Birch T (ry+). The heads of illustrious persons of Great Britain. VoL t. Knapton, London.
Blix M (r88+). Experimentelle Beitráge zur Lósung der Frage über die specifische Energie der
- r. srimulation could
Hautnerven. Zeitschrift für Biologie, zo, L4r-56.
: :rlal function was Boissier de Sauvages F (tZlz). Nosologie méthodique, ou dístribution des maladies en classes,
:i -it inan\- who were en genres et en especes, suivant I' de Sydenham, 6la méthode des botanistes. VoI. 4. Bruyset,
Lyon.
, : .:sthcation of the Boring EG (rg+z). Sensation and perception in the history of experimental psychology. Appleton-
-.--',--ns. Such a clas_ Century, New York.
:l:; -.-l¿:acteristics of Brazier MAB (rqiq). The historical development of neurophysiology. In I Field, HW Magoun,
:.:.:::rdinq ofenergy VE Hall (eds.) Handbook of physiology. Neurophysiology. Vol. t., pp. 1-58. American
Physiologicai Society, Washington DC.
:: : I -,' 3., idence from Brazíer MAB (rq88). Ahistory of neurophysiology in the ry't' century. Raven, New York.
: J : --r ¡!¿f¡¡ss Of the Bresadola M (rqq8). Medicine and science in the life of Luigi Galvani (v3z-t798). Brain
Research Bulletin, 46, 367-8o.
: ihe criteria that Breuer | (rs74). Über die Funktion der Bogengánge des Ohrlabpinthes. Wiener medizinische
.: :.-r< ¿dvances that Jahrbuch, 4,72-L24.
: . t: [t :ctence, the Brown T QSzo) . Sketch of a system of the philosophy of the human mind. Bell, Bradfute, Manners
:r-: I ji art. Perhaps & Miller, and Waugh & Innes, Edinburgh.
Cohen B (1984). Erasmus Darwin's observations on rotation and vertigo. Human Neurobioloy,
3, rzr-8.
Cranefield PF (tgz). The way in and the way out: Franqois Magendie, Charles Bell and the roots
of the spinal nerves.Ftttrra Mount Kisco, NY.
Crum Brown A (1874). Preliminary note on the sense of rotation and the function of the semi-
circular canals ofthe internal ear. Proceedings ofthe Royal Society ofEdinburgh,8,255'7.
Crum Brown A (l8Zf ). On the sense of rotation and the anatomy and physiology of the semi-
circular canals of the internal ear. Journal of Anatomy and Physiology, S, 327-3t.
-.-t- )\-úenham Crum Brown A (rS7Ba). Cyon's researches on the ear. l. Nature, r8, 633-5.
Crum Brown A (r8Z8b). Cyon's researches on the ear. II. Nature, fi, 657-9.
,t , -t.a' ':.r.lt r'tttischen Crum Brown A G8g). The relation between the movements of the eyes and the movements of the
head.Frowde, London.
::=::l!t:fa¡at deS Dallenbach KM (rq¡q). Pain: History and present sta:u:s. American lournal of Psychology,52,
- .:.:.,¡nq. Eigene
337-47.
Darwin E (vs). Zoonomia; or, the laws of organic life. VoL r. Johnson, London.
.. "'.
-, : : .. :. Clarendon, Darwin E (Vg). Zoonomia, oder, Gesetze des organischen Lebens (iD von Brandis trans.).
Hahn, Hannover.
; .: :, :.-:1',:. lOngman, Darwin RW (rz86). New experiments on the ocular spectra of light and colours. Philosophical
- ;:¡ :,,t:rgence of Transactions of the Royal Society,76,38-48.
.-,:. Descartes R QgT hgoz) . La dioptrique. In C Adam and P Tannery @ds.) Oeuvres de Descartes,
- .::.''.: .-.'t¡:s fiiends. Vol. VI. Ceri Paris.
:l: :::: in \J \\¡ade Descartes R (1664119o9). Traite de i'homme. In C Adam and P Tannerf (eds.) Oeuvres de
- .'. .. - Routledge/
Descartes, Vol. XI. Cerf, Paris.
Dewhurst K (r98o). Thomas Willis's Oxford lectures. Sandford, Oxford.
: _:_-q¡ Jnd Ot the Donaldson HH (r88¡). On the temperature-sense. Mínd,rc,399-416.
Drake S Q957). Discoveries and opinions of Galileo. Doubleday, New York.
54 NICHOLAS J. WADE
Finger S Q991. Origins of neuroscience. A history of explorations into brain function. Oxford HunterJ(t;¡6.t:
University Press, New York. London.
Finger S (zooo). Minds behind the brain. Oxford University Press, New york. |amesW(r88:.l
Finger S and Wade Nf (zooz). The neuroscience of Helmholtz and the theories of Johannes James W (r89¡ :
Müller. Part e. Sensation and perception. Journal of the History of the Neurosciences, rt, Jardine L (:.oc-¡ .
234-54. KempS(r99o.,"
Flourens P (1824). Recherches expérimentales sur les propriétés et les fonctions du systéme neryeux Koelbing Hlf r+
dans les animaux vertébrés. Bailliére, Paris. Kornhuber HH
Flourens P (r8¡o). Expériences sur les canaux semi-circulaires de l'orielle. Mémoire de l' Royale physiolog',. '''.
des Sciences de Paris,9, 455-66. New York.
Flourens P (r8+z). Recherches eL?érimentales sur les propriétés et les fonctions du systéme neryeux KreidlA(r89t.i
dans les animaux vertébrés, z"d edn. Bailliére, Paris. Taubstunn--¡r
Ford BJ ft99r). The Leeuwenhoek legacy. Biopress, Bristol. KrutaV (196+ . -1,
Ford BI Q.oo7). Enlightening neuroscience: Microscopes and microscopy in the eighteenth l'équilibre. !.::
century. In HA Whitaker, CUM Smith, and S Finger, eds. Brain, mind, and medicine: essays Kühne \\r ,r¡¡-:
in úth century neuroscience, pp. 29-41. Springer, New York. klinische l'Í¿.;:
Freher P Q688). Theatrum yirorum eruditione clarorum. Hofmann, Nuremburg. Leeuwenhoek -j',
Frey M von (1895): Beitráge zur Sinnesphysiologie der Haut. Scichsische Akademie der nerve. Pl¡:;¡-..'
Wissenschafte zu Leipzig 47, 166-84. Leeuwenhoek i
Galileo G (16z:). Il saggiatore. Mascardi, Rome. the optick r:e :
Galvani L (vgr). De viribus electricitatis in motu musculari. De Bononiensi Scientiarum et Leeuwenhoek -r"
Goldscheider A (r88+). Die specifische Energie der Temperaturenerven. Monatsschrift für MA.
praktische Dermatologie,3, 198-2o8. Mach E iiE-_r
Goltz F (r87o). Ueber die physiologische Bedeutung der Bogengánge des Ohrenlabyrinths. Sitzwtgsl.r'.:)
Archiv für die gesammte Physiologie,3, 172-92. MachE(r8.;.'.
Griffith CR (r9zz). An historical survey of vestibular equilibratlon. University of Illinois Press, Mach E (r9.c
Urbana, IL. Chicago. IL
Grüsser OJ and Hagner M (rggo). On the history of deformation phosphenes and the idea of May MT (r9ó!
internal light generated in the eye for the purpose of üsion. Documenta Ophthalmologica, Ithaca, \l.
74,57-85. Mazzarello P :;
Gudden B von (r87o). Experimentaluntersuchungen über das peripherische und centraie and A Badi:-.r
Nervensystem. Archiv fLr Psychiatrie und Neryenkrankheit, z, 693. Mazzarello P. iJ
Hallaran WS (tSr8). An enquiry into the causes producing the extraordinary addition to the Cento atrii .;.
number of insane, together with extended obseryations on the cure of insanity: with hints as to Melzack R r: -
the better management of public asylums znd edn. Edwards & Savage, Cork. r378-8:..
Hamiltonw(18+6). TheworksofThomasReid,D.D.Maclachlan,stewart,Edinburgh. Melzack R and i
Harris H (rygg). Thebirth of the cell.Yale UniversityPress, NewHaven, CT. Ménidre P r:¡-
Hartley D Gr+g). observations on man, his frame, his duty, and his expectations. Leake and gestion cer;:
Frederick, Bath. Múller I (18:6 .
Heimholtz H
(1867). Handbuch der physiologischen oprlft. In G Karsten ed,. Allgemeine nebst eir.¿"t'.',
Encyklopiidie der Physik. Vol. 9. Voss, Lerpzig. Leipzig.
Herz M (1786). Versuch über den Schwinde. Voss, Berlin. Müller J ,rS,:¡
Hrtzig E (r87r). Ueber die bei Galvanisiren des Kopfes entstehenden Stórungen des Bonn.
Muskelinnervation und der Vorstellungen vom Verhalten im Raume. Archiv Müller I (r8+-.
für Anatomie, .
, :'-,ii,l
.lunction. Oxford Hunter J Q786) . Obseryations on certain parts of the animal economy. published by the author,
London.
r! L:r:k.
fames W (1882). The sense of dizziness in deaf-mules. American lournal of Otology, 4,239-54.
i. ::ieorres of /ohannes
James W Q89o). Principles of psychology. Holt, New York.
:: .)'.: \eurosciences, Lt, jardine L (zoo3). The curious life of Robert Hooke.Harper Collins, London.
Kemp S Gggo). Medieval psychology. Greenwood, New York.
;: --':-i s|st¿me neryeLx
rrjt¿
Koelbing HM (rg6z). Renaissance der Augenheilkunde. t54o-t6jo. Huber, Bern.
Kornhuber HH (g24. Introduction. In HH Kornhuber, ed. Handbook of sensory
-:. -',!¿tloire de I' Royale physiology. Volume VI/t. Vestibular system. Part r
Basic mechanism; pp. 3-14' Springer,
New York.
::,-':,.,;i; :t'stétne neryeux Kreidl A (r89r). Beitráge zur Physiologie des Ohrenlabyrinthes auf Grund von Versuchen an
Taubstummen. Archil für die gesammte Physiologie,5l' 119-50.
Kruta V (ry6+). M.-|.-P. Flourens, J.-E. Purkyne et les débuts de la physiologte de la posture et de
the eighteenth l' ér1uilibre. Alenqonnaise, Paris.
tr;edicine: essays Kühne W (186¡). Die Muskelspindeln. Archiv für pathologische Anatomie, Physiologie und
klinis che M edizin, 28, 528-38.
Leeuwenhoek Avan (1674). More observations from Mr. Leewenhoeck, concerning the optick
--::.:-.-';rr- -\ktttlemie d.er nerve. Philosophical Transactions of the Royal Society, g, t78-82.
Leeuwenhoek A van (1625). Microscopical observations from Mr. Leewenhoeck, concerning
the optick nerve. Philo sophical Transactions of the Royal Society, 9, 378-Bo.
'.
-,::."'.:: Scietftiarum et Leeuwenhoek A van (t68+). Ontledingen en beschouwingen der onsigtbare geschapene
waarheden. Leiden.
-.. :t)IO9tIá. Lucretius G975). De rerum natura (WHD Rouse trans.). Harvard University Press, Cambridge,
i :: _1!I!-¡¡tr?¡-i_schrift MA.
für
Mach E (1873). Physiologische Versuche über den Gleichgewichtssinn des Menschen.
:: ;,-l: renlab)'rinths. Sitzungsberichte der Wiener Akademie der Wissenschaften,68, t24-4o.
Mach E (ú7). Grundlinien der Lehre yon den Bewegungsempfndungen. Engelmann, Leipzig.
i:i-:. .ri Iliinois Press, Mach E (r9rc). Popular scientifc lectures,4'h edn. (TJ McCormack trans.). Open Court,
Chicago, IL.
::.:t¿s :nd the idea of May MT Q96B). Galen. On the usefulness of the parts of the body. Cornell University Press,
:. : :.; -i ¡ i., :i al tn ol o gi ca,
t
Ithaca, NY.
Mazzarello P (rSSS). The hidden structure. A scientifcbiography of Camillo Golgi. (HA Buchtel
r::-:;j]e und centrale and A Badiani trans.). Oxford University Press, Oxford.
Mazzarello P, Calligaro A, Garbarino C andVannini V (zoo6). Golgi, architetto del cervello.
::".":,*. ,¡"iiitiotl to the Cento anni dal primo Nobel italiano. Skira, Milan.
i¡il lúnts as to
,...'::.'. : i' Melzack R (zoor). Pain and the neuromatrix in the brain. Journal of Dental Education, 65,
r378-Bz.
--i.r.:urg Melzack R and Wall PD (rq6r). Pain mechanisms: A new theory. Science,L5o' t77-9'
Méniére P (rS6r). Mémoire sur des lésions de I' interne donnant lieu á des symptoms de con-
:.,..;. ia,:-.. Leake and gestion cérébrale apoplectiform e. Gazette médicales de Paris, 16, 597-6ot'
Müller I (18z6). Zur wrgleichenden Physiologie des Gesichtssinnes des Menschen und der Thiere,
: :.:..
.- --_
iü.
-i rrl
-lttgemetne nebst einemVersuchüber die Bewegungder Augenundüber denmenschlichenBlick. Cnobloch,
Lerpzig.
Müller J (rB¡B). Handbuch der Physiologie des Menschen für Voilesungen. VoI. z. Hólscher,
Stórungen des Bonn.
'.
.i!¡r Anatomie, Müller I (r8+¡). Elements of physiology,2nd edn. (w Baly trans.). Tayior and walton, London.
Müller | (zoo3). Müller's Elements of Physiologl. Thoemmes, Bristol.
:-, _;;¡_. r,to.de by mag_ Natanson LN (rS++). Analyse der Functionen des Nervensystems. Archiv filr Physiologie und
-::¡;. London. Heilkunde,3, 515-35.
56 NICHOLAS ]. WADE
Neff WD (196o). Sensory discrimination. In I Field, HW Magoun, and VE Hall, eds. Scaliger IC (u;, . i
Handbook of physiology. Neurophysiology. Vol. 3., pp. r447-7o. American Physiological Hieronymwrt C':,'.;.
Society, Washington DC. SchickoreJ(zoor.:'',
Newton I GZo+). Opticks: or, a treatise of the reflections, refractíons, inJlections and colours of copy. Perspecti'.,: :
'.:-. atd YE Hall, eds. Scaliger JC OSSil.Exotericarum exercitationum liber quintus decimus, de subtilitate, ad
.il::¡:ican Physiological Hieronymum Cardanum. Vascosani, Paris.
Schickore J (zoor). Ever present impediments: exploring instruments and methods of micros-
':-l:.-::0ru-s and colours of copy. Perspectives in Science, g, t26-46.
Schickore | (zoo). The microscope and the eye. A history of reflections 1740-1870. University of
:--.ii. phvsiological, and Chicago Press, Chicago, IL.
Schleiden M (r8¡8). Beitráge zur Phltogenesís. Müller's Archiv für Anatomie, Physiologie und
s ::d :he 'larv of specific wissenschaftliche M edizin, pp. 86-7 6.
-: :, -.:;: r. 48, 457-65. Schwann T (r8¡q). Mikroskopische (Jntersuchungen über die Übereinstimmung in der Struktur
l :... \f. und dem Wachsthum der Tiere und Pflanzen. Reimer, Berlin.
--.; E :t: rt D e tltl N eurology, sharples RW (zoo3). on dizziness. In \NW Forrenbaugh, RW sharples, and MG Sollenberger,
eds. Theophrastus of Eresus on sweat, on dizziness, and on fatigue. pp. 169-25o. Briii,
i ,r:.É: the tbundation of Leiden.
Shepherd G (r99r). Foundations of the neuron doctrine. Oxford, Oxford University Press.
: :ie artificial eiectric Sherrington CS (r9oo). The muscular sense. In AE Scháfer, ed. Text-book of physiology VoI. z.
pp. tooz-zs. Pentland, Edinburgh.
. -; | :,,2:,i t t tt, 4, 2 45-7 8. Sherrington CS (rqo6). The integratite action of the nervous system, Scribner, New York.
.;..:. ',,-tita e I' animale. (Reprinted in Nl Wade, ed. (zooo). The emergence of neuroscience in the nineteenth century.
Vol. 8. Routledge/Thoemmes, London.)
: ::tls¿s in the tvork of Siegei RE Q97o). Galen on sense percepfioa. Karger, Basel.
Sinclair D (tg6). Cutaneous sensation. Oxford University Press' London.
l' :,:.j. : : t¡ \ euro sciences, Smith AM j996) . Ptolemy's theory of visual perception: an English translation of the Optics with
introduction and commentary. TheAmerican Philosophical Society, Philadelphia' PA.
-::=r ::-lit Voltaischen Spillane JD (rg8r). The doctrine of the nen,es. Oxford University Press, Oxford.
Stratton GM (r9r7). Theophrastus and the Greek physiological psychology before Aristotle.
Macmillan, New York.
:...i. (-it.;ego, IL, Tatler BW and Wade N| (zoo3). On nystagmus, saccades, and fixations. PercEtion,32,t67-84.
''
:,.;,-,:. 1 ¡1. -:. Hamiiton vieth GUA (r8r8). l_Iber die Richtung der Augen. Annalen der Physik und chemie,58,43-55.
Volta A (rBoo). On the electricity excited by the mere contact of conducting substances of dif-
, -' l:¿¿utoqnostischen ferent species. Philosophical Transactions of the Royal Society, 9o, 403-37.
'-::-t'.:::i:¿l Staates, 6, Wade NJ Q99o). Visual allusions: pictures of perception. Lawrence Erlbaum Associates' Hove,
East Sussex.
,:,:: _'i¿:¿.' Beitrage zur Wade NJ Q9g). Psychologists in word and image. MIT Press, Cambridge' MA'
" Wade Nl (rqqs). A naturalhistory of visiorz. MIT Press, Cambridge' MA.
= ¡ :r:¿ die Beziehung Wade NJ (zoooa). William Charles Wells (r757-r8ry) and vestibular research before Purkinje
;,;.;--:": .r;ir iie gesammte and Flourens. J o ur n al of V e s tib ular Re s e ar ch' to' 727 -37.
Wade NJ (zooob). Porterfield and Wells on the motions of our eyes. Perception,29,227-39.
: :'r..lcians, Prague. Wade NJ Qooz). Erasmus Darwin (r73r-r8oz). Perception,3t,643-50.
: : "'. ":,: ti -.ens¿. Millar, Wade NJ (zoo¡a). Destined for dktinguished oblivion: the scientific vision of William Charles
WeIIs (ry57-t|tz). Kluwer/Plenum, New York.
:: "' Wade NJ (zoo3b). The search for a sixth sense: The cases for vestibular, muscle, and tempera-
i-t.rischen Batterie.
ture senses. Journal of the History of the Neurosciences, rz, 175-202.
::r-!mus. .\nnalen der Wade NJ (zoo5a). Perception and illusion. Historical perspectites. Springer, New York.
Wade Nl (zooSb). Vision and the dimensions of nerve fibers. Journal of the History of the
: :rd \t'R Uttal, eds. N euro s ciences, l^4, 28t-9 4.
¿ -nternational Society Wade NJ (zoo5c). The original spin doctors - the meeting of perception and insanity.
Pe rcep ion,34, 251-óo.
t
:r¡ Press, London. Wade NJ (zoo6). Perceptual portraits. In F Clifford Rose, ed. Neurobiology of painting.
pp. rZ-38. Elsevier, San Diego, CA.
t8 NICHOLAS I. WADE
Wade NJ QooTa). Galileo and the senses: vision and the art of deception. Galilreana,4,
259-88.
wade NJ QooTb). The vision of william Porterfield. In H whitaker, cuM Smith, and s Finger,
eds. Brain, mind, and medicine: essays in tl't' century neuroscience,pp.tg-76. Springer, New
York.
Wade NJ QooTc). Scanning the seen: vision and the origins of eye movement research. In RPG
van Gompel, MH Fischer, WS Murray, and RL Hill, eds. Eye movements: a window on Mind
and Brain, pp. 31-61. Elsevier, Oxford.
Wade NI (zooq). Allusions to visual representation. In M Skov and O Vartanian, eds.
Neuroaesthetics. pp. t7t-22o. Ba¡vood Publishing, Amiqville, NY.
Wade N) and BroZek | (zoor). Purkinje's vision. The dawning of neuroscience.LawrenceErlbaum
Associates, Mahwah, NJ.
Wade NJ, Norrsell U, and Presly A (zoo5). Cox's chair: 'a moral and a medical mean in the
TO
treatment of maniacs'. History of Psychiatry,r6,73-88.
Wade N] and Piccolino M (zoo6). Nobel stains. Perception,35, 1-8.
Wade Nl and Tatler BW (zoo5). The moting tablet of the eye: the origins of modern eye moye-
Mt
ment research. Oxford University Press, Oxford.
Weber EH (rS:+). De pulsu, resorptione, auditu et tactu.Koehler,Leipzig.
Weber EH (rsa6). Der Tastsinn und das Gemeingefühl. In R Wagner, ed. Handwórterbuch, der
Physiologie, Vol. j, pp.48r-588. Vieweg, Braunschweig. GE
Wells WC (ry92). An essay upon single vision with two eyes: together with experiments and obser-
yations on seyeral other subjects in optics. Cadeii, London. (Reprinted in Wade zoo3a.)
Wells WC (tzg+a).Reply to Dr. Darwin on vision. The Gentleman's Magazine and Historical
Chronicle, 64, lg +-2. (Reprinted in Wade zoqa.)
Wells WC (vg+b).Rep1y to Dr. Darwin on vision. The Gentleman's Magazine and Historical
Chronicle, 64, 9o5-7. (Reprinted in Wade zoqa.)
SrNc¡ the earh- 19¡ -
Wendt GR (rqfr). Vestibular functions. In SS Stevens, ed. Handbook of experimental psychol-
from its origins rn --'
o g, pp. t1g1-223. Wi1ey, New York.
Wh¡t R Q76). Observations on the nature, causes, and cure of those disorders which have historical and theor,
commonly been called nervous hypochondriac, or hysteric. Becket, Du Hondt, and Balfour, reception-orienteo ;
Edinburgh. the ocular, on ho'"'
Willis T Q67z). De anima brutorurr. Wells & Scott, London. Clearly, the visual:e,
Wilson C (rsss). The invisible world. Early modern philosophy and the inyention of the micro- example of Earlr- \f i
scope. Princeton University Press, Princeton, N].
demonstrates ihe 'ur-
Young LR, Henn V, and Scherberger H (zoor). Fundamentals of the theory of moyement percep-
insights into horr r:::
tion by Dr. Ernst Mach. Kluwer/Plenum, New York.
Young T (r8oz). On the theory oflights and colours. Philosophical Transactions of the Royal their contempora.;
Society,92, t2-48. In Early N{odein
Young T GBoz). A course oflectures on natural philosophy and the mechanical arfs. |ohnson, believed that touci-
London. (Reprinted by Thoemmes Press, Bristol zooz.) objects. Not o¡1-'' ru:
Zotterman Y (1959). Thermal sensations. In J Field, HW Magoun, and vE Hall, eds. Handbook
of physiology. Neurophysiology. vol. 4 pp. 431-5B. American physiological Society,
' This essav l'as ¡¡:-'
Washington, DC.
' Reader-response :::
scholars such as Ha:.. .'
Michael Fried, Hars 1..
Kemp rg8:).
' In this chapter. F::
refers to the perioci s::t.
Gdlilreana, 4,
:_ _j -. r-
r_r \ ¿It¿n1an, eds.
THE ART OF
TOUCH IN EARLY
:.-. _¿..rTence Erlbaum
: ,i,¡d Historical
SrNcn the early 198os, reception theory (also known as 'reader-response criticism'
','".t1íúl psychol-
from its origins in literary studies) has become an increasingly important tool in the
; .,t'ltich have historical and theoretical analysis of visual and material culture.' However, in most
and Balfour, reception-oriented art historical research, the focus has almost exclusively been on
the ocular, on how beholders, both real and imagined, have looked at art objects'
Clearly, the visual reception of art is crucial. But as this chapter will suggest through the
,: :.i rhe micro-
example of Early Modern Italy, a careful consideration of other senses such as touch
demonstrates the ways in which non-optical models of reception can provide new
percep-
":11:¿tlt insights into how images and objects were actually made, engaged' and understood by
''.: .-,í tlrc Royal their contemporary beholders.'
In Early Modern Italy, texts on sculpture in particular confirm that many writers
,;r;-.. ]ohnson, believed that touch was indeed an important way of negotiating encounters with art
objects. Not only writers, but fifteenth-, sixteenth-, and seventeenth-century painters
¡-. ¿ds. Hnndbook
,--:sical Society, ' This essay was completed during a Leverhulme Trust Research Fellowship'
' Reader-response criticism was first formulated in theoretical terms in the r97os by literary
scholars such as Hans Robert Jauss and Wolfgang Iser, and then taken up by art historians such as
Michael Fried, Hans Belting, and Wolfgang Kemp (Belting r98r; Fried r98o; Iser ry78; Jauss t977;
Kemp 1983).
, In this chapter, 'Early Modern' (often used interchangeably with 'Renaissance and Baroque')
refers to the period stretching from the early fifteenth through the seventeenth century.
6o GERALDINE A. JOHNSON
and sculptors working in Italy also explored the notion of tactility in some of their Some philosoPhe
projects. By considering a wide range of texts, images, and objects, one can begin to centric assumPtions
appreciate the importance of attending to the sense of touch in this period. More inpassing. The fen:
significantly, by considering tactility in the context of Early Modern Italy, one can takes pleasure mor€
begin to explore the possibility of writing a more general history of art-or rather, a oftouch as a possibl
history of the senses used to apprehend art-that goes beyond the ocularcentric and trism' of contemPor;
instead considers other modes of experience and forms of attention, such as those ioural psychologrsts
made available by touch-a theme that runs through many of the other chapters in the contributors to the
present volume.s vision in their studi
Given the evident interest displayed by Early Modern writers and artists in the ars of Early Moder
sense of touch, it is surprising that most scholars working on this period have tended and contemPoran'
to ignore issues related to tactility in their studies. Three important exceptions are touched on the sens
Michael Baxandall, Suzanne Butters, and Jennifer Montagu, art historians who have
considered the materiality of Early Modern sculpture, if not explicitly its tactile recep-
tion (Baxandall r98o; Butters 1996; I Montagu 1996). Authors who have explored the
importance of the sense of touch itself in this era have often done so only in passing
or in abstract or metaphorical ways, for instance, as a textual theme or as the subject
of works of art, rather than as an actual physical phenomenon involving beholders
T
literally handling an object (Barolsky 1995; Bolland zooo; Boyle 1998; Harvey zoo3;
Hecht r984; Sherman zooo; Summers r98t, r987).
Tactility as an abstract concept has also interested a number of influential art his-
torians working on periods other than the Early Modern one. Indeed, already in the Ever since antíqui;
late nineteenth and early twentieth century, key figures such as Adolf von Hildebrand, about the hierarch-
Alois Riegl, and Heinrich Wolfflin investigated the tactile qualities of sculpture in r99r). Plato, for eri
particular, although generally from a theoretical rather than from an historical or touchthe most ani:
somatic point of view (von Hildebrand 1893; Iversen rgg:; Olin ry92; Podro ry82; Ages. In the Earlr-
Riegl rgor; Wólfflin r9r5; Wood 1998). Bernard Berenson likewise explored what he vision, especialh-ir
called 'tactile values' in paintings produced in Renaissance Italy, but not touch as saint, Antoninus. ¿
an actual physical practice (Berenson 1897,pp. ¡¡-+). More recently, a few scholars tect and art theoris
have considered the role touch plays in the work of Modern artists such as Cézanne, an1'thing. . . It is .
Kandinsky, and the Surrealists (Olin rg8g; Powell D97;Shiff ry9r). As with Berenson, PP.36-8). Srmilar-
however, the sense of touch is once again considered in relation to the subject, compo- the baser, more c¿
sition, or sty'e ofpaintings, rather than to the actual physical production and reception associated u'ith rrs
of art objects, whether sculpted or painted. mind; lwhiler th¿t
rg82, p. 6r). Leo¡l
embraces the b'e¿"
asserted that tou-h
3 'Ocularcentric'refers here to theoretical or historical approaches to art objects that privitege
the
visual. see especially lay r9 g, p. 3 and passim. on touch and the senses see my forthcoming book
r97L,P.637; Sunri
entitled The Sound of Marble: The Sensory Reception of Art in Renaissance ltalT and also, among Since antiquin-
others: Classen zoo5; Hall 1999, pp.8o-ro3; Howes 1991, 2oo3, zoo5; Johnson zooza; Marks zooz; edge is gained a;r
A Montagu 1986; Nordenfalk 1985, r99o; Petrelli 1994; Rivlin and Gravelle 1984; Stewart 2oo2; Synnott to seeing a Pictu
1993; Vasseleu 1998; and the journal The Senses and Society (first issue published in March zoo6).
Growing interest in tactiliry is also demonstrated by events such as the 'sculpture and Touch'
symposium in May zoo8, organized by Peter Dent at the Courtauld Institute of Art, London. a See n. ¡ for sei¡;
MODERN ITALY 6't
THE ART OF TOUCH IN EARLY
-
6z GERALDINE A. ]OHNSON
intellectual contemplation (Miles 1983; summers r9g7, pp. g9, zoo). The thirteenth- Modern period as
century philosopher Roger Bacon went so far as to claim that all knowledge of the era as demonstrate(
world can only be obtained from sight, and thus a blind man could learn 'nothing that thematize tacii
that is worthy in this world' (summers ry87,pp.36). In the sixteenth century, the
artist and art historian Giorgio Vasari asserted that knowledge and judgements about
art in particular were best obtained through vision rather than touch: 'it is neces-
sary to have the compasses in the eyes and not in the hand, because the hand works
and the eye judges' (summers r98r, p. 37r). More generally in the Early Modern
period, the ocular practice of linear perspective became a model not only for vision,
l
but for subjectivity itself, that is, for how one formulates a 'point of view' about
the world in which one lives, as discussed by Erwin panofsky in his influen-
tial essay on perspective as symbolic form, as well as by other more recent authors
Some ofthe first p..,
(Panofsky 19z7).
tising artists u-orli¡
The primacy of sight in Ancient, Medieval, and Early Modern thought is, however,
recipes and pracic;
neither absolute, nor uncontested. For instance, visual anxiety is a common theme
Throughout the E¡
in classical mythology, as in the stories of Narcissus, orpheus, and the Medusa,
physical, and r-er,' :.
while touch is seen as a positive, life-giving force in the m¡hs of pygmalion and
expect, the fopc-. .'-
Prometheus. Likewise, for Plato, it was sight's perilous powers of illusion that were
found in aLmost e'''¡
most worrying, while St Augustine warned of the dangers associated with ocular
that considered a:t .
:- - . The thirteenth- Modern period as well.5 However, as we shall see, touch did indeed matter in this
. ' gorrledge of the efa as demonstrated by numerous texts on sculpture, as well as by images and objects
¡L-, lc
iearn 'nothing that thematize tactilityboth implicitly and explicitly.
t..ai:ith century, the
r¿ :-rdEements about
: ::uch: 'it is neces-
:-!É :ne hand works
- ::e Earlr- ir,Iodern TncrrrrTY AND THE ART oF
- ,--:: ..rlr- tb¡ r'ision,
,,-
-:rt rrl |ierv' about EARLY MODERN IT¡.TY
ir'r -¡ hlS lntiuen-
: -:a reaent authors Some of the first post-Classical writings on art were technical manuals composed by prac-
-,-r
tising artists working in Italy (Cennini 2oo4). These volumes were filled primarily with
: -sli: is. horl'ever,
recipes and practical adüce intended for use in the workshops of craftsmen and artisans'
.: i theme
-Llr]llll-1O11
Throughout the Early Modern period, Italian wdters continued to emphasize the active,
: :r. the \{edusa, physicáI, and very tactile process of hand-crafting a statue or relief' Indeed' as one might
, -,: P','¡:malion and the topos of the sculptor's manual prowess in making and handling sculpture is
-::.'-r¡ton that were "*p..t,
found in almost every Early Modern art treatise and technical manual. The first treatises
,,-:l::-i r.ith OCUlar
that considered art from a more theoretical perspective were written in the first half ofthe
.. .:l: :hilosophers fifteenth century by Alberti. However, while his famous book De picturatrealedpaínt-
: li :e tnlv model ing as a practice worthy of serious humanistic and scientific consideration, his much less
..: : ,¡¡tstotle, rvhile well-known treatise on sculpture, De statua,essentia\ continued the earlier tradition of
:r:, :ta ner-ertheless writing technical manuals for craftsmen (ñbertttgZz;Aurenhammer zoor). Alberti's dis-
::¡---;l irom all the
tain for the manual labour involved in creating sculpture, which he opposed to painting's
-:::u:-,- SummerS
more cerebral demands, was a theme that would resurface repeatedly in later art treatises'
:::- *a:t e5 a means It is perhaps not surprising that the first treatise demonstrating an explicit concern
. i - re:hing rvith a with raising the status of sculpture to that of a liberal art was written around the middle
:: -r,': .',f Descartes's
of the fifteenth century by a practising sculptor with intellectual ambitions, Lorenzo
Ghiberti (Ghiberti r9+Z). Rather than primarily giving technical advice to its readers,
,. r'iSual
":,;;-:Sir-elr' Ghiberti's text instead displays the sculptor's humanist pretensions and, by extension,
. .. il¿r:. COmpared
the aspirations of the art he practised. At the same time, Ghiberti's concern with
the tac-
::.:. r-! , Srmilarly,
tile values of sculpture foreshadow important aspects of later writings on sculpture'
:::::.'' lrtbedded in
When ,p.ukirrg in the abstract about sculpture, Ghiberti stresses the importance of
,--. r:-nd of a true
vision, opii.r, u.rá hghtit g effects. When he begins to discuss specific statues he has
"-.:-:-¿ s claim that
personaliy encountered, however, Ghiberti often introduces tactility as a key element
.: ,rt,-riano Bruno in the reception of sculpture. In one passage' Ghiberti describes a recently unearthed
. :r,:ro Yates 1966,
Classical female statue. Ghiberti states that this 'marvelous' figure had'so much refine-
',1:-Lr:1,
ment ldolcezze], which the face lil viso, i.e. looking with one's eyes] could not appre-
, in .{ncient,
ciate by either strong or moderate light, only [by] the hand touching it could this be
',' .;.c l¿rge, scholars
- * t:ñicu1ar, espe- 5 One of the few widely acknowledged exceptions is the late seventeenth- and eighteenth-century
:,.:specti\-e. Indeed, philosophical debate known as Molyneux's problem, which focused on whether touch, rather than
s,.h an important ,ririo.r, *u, the most important sense for gaining empirical knowiedge about the world' See Degenaar
":,i; ibr the Early 1996, johnson 2oo2a, pp. 63-4, and Morgaí 1977'
64 GERALDINE A. JOHNSON
I have seen by diffused light the most perfect sculptures . . . made with the greatest skill and
care, including . . . a statue of an hermaphrodite the size of a r3-year-old girl, which statue had
been made with admirable skill . . . It is not possible in words to tell of this statue's perfection . . .
In this fstatue] there was the greatest refinement [dolcezze), which the face lil viso,i.e. looking
with one's eyes] would not have seen, had not the touch [ofl the hand sought it out.
(Ghiberti ry47, pp. 54-5)
This passage suggests that Ghiberti may well have discovered or, at the very least,
confirmed the male sexual attributes of the hermaphrodite only after touching it with
his hand, especially since his initial assumption seems to have been that the figure
was female, in his own words, 'a r3-year-old girl'. Ghiberti's manual encounters with
female and bi-gendered statues also suggest that sculpture's tactile accessibility can
at times be profoundly intertwined with questions of sexual desire and differen-
tiation. Unlike a painting, a touchable sculpture always has another side to explore,
which allows for realms of bodily experience to exist that are unimaginable in two-
dimensional works of art. I
There were, of course, many different types of touch associated with sculpture in I
the Early Modern period. For instance, touch could have a talismanic or devotional :
quality, as when wooden figures of Christ were taken down from supporting crucifixes i
d
for ceremonies associated with Holy week, or when pilgrims eagerly tried to touch
carved reliquary caskets or saints tombs (Fig. 3.r). Another religious practice was the F'
ritual handling of life-size statues of the Christ Child, often in wood or terracotta. Pl
Documents describe nuns in particular cradling such objects, with the statues occa-
sionally appearing to come to life in the women's arms (Hale r99z; Klapisch-Zuber
1985). Perhaps not surprisingly, later Church reformers worried about the dangers
ofidolatry inherent in these and related devotional practices that involved physically
handling and worshipping sacred sculpture. describe a blind r::
The tactile encounter with sculpture was also seen in the inquisitive, admiring, and Mazenta et'en ;r¿r
proprietary hands of the collector and connoisseur. Indeed, Early Modern behold- himself (Baroccn:
ers often describe and depict themselves touching three-dimensional art objects, as When consici¿r
in Titian's Portrqit of lacopo Strada (vienna, Kunsthistorisches Museum) in which edge that sculPi::
a Venetian art dealer and collector is shown holding a Classicizing female statuette scale statuettes :r:
(Fig. 3.2). Other texts and images confirm the importance of the sense of touch to the and his follor. e:
evaluation and appreciation of art. For instance, an apocr-yphal anecdote in Cesare manipulated b.' :
Ripa's rconologia of 16o3 claimed that Michelangelo relied on touch alone to judge a beholder's abr-:
the merits of Antique and Modern statues as he gradually grew blind in old age (Hall life-size marbie '
1999, p.87). A number of other sixteenth- and seventeenth-century texts and images Signoria, rvhich .
likewise focus on the topos of the blind man's tactile handling of works of art.
lusepe
da Ribera and Luca Giordano, for instance, both painted versions of this subject, while
6 The theme t': L:
the writers vincenzio Borghini, Raffaele Borghini, and Giovan'Ambrogio Mazenta all
feature of the ph::..,
THE ART OF TOUCH IN EARLY MODERN ITALY 6S
üat onlr.bytouching
;sible to r-ision alone,
.l ','ith sculpture in
::ric or der.otional
::roting crucifixes
.:<r1i rried to tou.ch
: --. rra!-tice rvas the Fig. 3.1 Master of St Sebastian (Josse Lifferin?)'
lt:r'¡[1 L]r tellacotta. Pitgrims visiting a saint's shrine, oil painting'
--::ú¡ statues occa- c.1500' Rome, Galleria nazionale d'Arte antica'
i::: Klapisch-Zuber (Photograph: Geraldine A Johnson )
The inherent truthfulness of touch is also presented by other writers as a key virtue
of sculpture. Varchi himself in his lecture on the paragonestafedthat vision:
is not the most reliable sense, indeed it often deceives . . . lwhile] the most reliable sense is
touch . . . lW]hen we see something and we are doubtful about it, . . . we use touch to verify
it. Everyone thus knows that touching a statue confirms everl'thing the eye sees . . . therefore
sculptors say that their art is truthful and painting is Inot] .
(Barocchi ryv,pp.5T-4)
qualities characterize much of the theoretical writing on the porogone in sixteenth- chromatic adr-ant¿i
and seventeenth-century ltaly. Many of the terms of this debate, however, had already different viervs at ú
been set by the late fifteenth century thanks to an influential collection of writings claimed that, unlilt
by Leonardo (Farago r99z). Although Leonardo was a practising sculptor as well as view ofa figure. ind
a painter, in his writings he came down firmly in favour of painting, which he saw as a single sculpture ci
an intellectual actiüty pursued by proper gentlemen, an art that could use colours to
imitate even nature's most ephemeral effects. Leonardo may well have sought to allude
to such effects in lhe Portrait of Ginevra de' Benci (Washington, DC, National Gallery 8 Iacobs 1988. p. :r;
of Art), which uses paint to show reflections on the surface of the lake to the right of the views of their sitter' -,:
sitter andthe glint ofsunlightonthewoman's carefullycombed hair (Fig.3.6). By depict- Museum) and Giori.
ing a fictive speckied marble surface on the back of this panel, Leonardo further attests Farago r992, pp. 11.-+i
THE ART OF TOUCH IN EARLY MODERN ITALY 71
s argument in favour
',n¡iters as a keyvirtue
--:at tision:
- : siatue (Barocchi
:i;-;uled the tactile
-,-:u-.tr rrho
sexually visually to painting's power to imitate the very materials of sculpture itself (Butters
i.:r to the tempting t996,I, p.ro9, and II, p. XXXI) (Fig.¡.2).
:r:ied ir-ith handling "
The paragone debafe seems to have inspired several other painters to demonstrate
r --¿;ome objects of
visually the advantages of their chosen medium as well. Titian's I a Schiavona (London,
National Gallery), for example, depicts the same women twice, once in full colour
:tc ¿salnst its tactile
and a second time in profile in a monochromatic marble relief, thereby proving the
-,;{..¡ri in sixteenth-
chromatic advantages of painting (Fig. ¡.8). By showing the same figure from two
,l'-\'erel. had already
different views at the same time, Titian was also rebutting advocates of sculpture who
, l¿:rion of rvritings
claimed that, unlike sculpture, two-dimensional paintings could only provide a single
: sculptor as well as
view ofa figure. Indeed, the sculptor Benvenuto Cellini had gone as far as asserting that
* ¡. it hich he saw as
a single sculpture could consist of more than 4o different views.8
:r- ¡lC use colours to
¿-.; sousht to allude
'[. \ational Gallery 8
.:. :.- the right of the Jacobs 1988, p. r49, n. 5o. Other paintings that seem to rebut such claims by having multiple
views of their sitters include Gian Girolamo Savoldo's so-ca1led Gaston de Foix (Paris, Louvre
Frg. _i.6 ). By depict-
Museum) and Giorgione's iost Sf George, praised by both Vasari and Pino (Barocchi :-977, pp. 552-3i
:¿:úLr lurther attests Farago 1992, pp. 27, 49; Hecht 1984, p. rz6).
72 GERALDINE A. JOHNSON
Thematizing the paragone not just verbally but visually also seems to have intrigued advocate of painti:rs
Leonardo's teacher, the painter-sculptor Andrea del Verrocchio. Before Leonardo's perhaps be taken'.'-i
Ginevra de' Benci was cut down along its bottom edge, the sitter's now-lost hands In fact, althouch
(which are documented in a drawing) may well have echoed those seen in Verrocchio's a certain point in h
marble Bust of a Lady with Flowers (Florence, Bargello Museum) (Butterfield 1997, Incredulity ofS¡ Ii;¡
pp. 90 ffl (Fig. 3.9). Indeed, the sitter in Leonardo's painting could even be the same of Or San Michele. ;
woman depicted in the bust, a figure who, incidentally, highlights the erotic potential which Thomas ref¡,
of sculptural tactility through her fingers, which seem to strain to touch her thinly- touching the risen t
clothed breast. Although the chronology of these two works in relation to one another nails, and thrust m.'-
remains unclear, it is possible that the student, Leonardo, produced his full-colour precisely sculpture'
portrait in order to demonstrate to his teacher the supremacy of painting over sculp- a key theme oi n¿:
ture, thereby giving visible form to the pro-painting arguments he later presented in Although the subje
his theoretical writings. If we believe the testimony of Vasari, Leonardo's virtuosity in Florence's Mercat::
painting such images convinced Verrocchio to stop painting altogether and instead sioned the sculptur
devote himself only to sculpture (Vasari ry65, p.236). Given that Vasari was a key oftouch. Indeed. u
THE ART OF TOUCH IN EARLY MODERN ITALY 73
' ,ícflye
' Bencr],
1a
Fig. 3.8 Titian, La Schiavona, oil on canvas, c.1510.
London, National Gallery.
(Photograph : The Yorck Project (http://commons.wikimedia.org)-)
:r-¡ tLr hal-e intrigued advocate of painting's superiority in the paragonedebate, however, his account should
F¡:ore Leonardo's perhaps be taken with a grain of salt.
:-:'s ;rOrr'-lOSt hands In fact, although Verrocchio does indeed seem to have stopped painting after
..en in \-errocchio's a certain point in his career, he may have tried to meet Leonardo's challenge in his
i Butterfie\d ry97, Incredulity of StThomas,a two-figure bronze sculpture group executed for the exterior
:-: e-,-en be the same of Or San Michele, Florence's guild church. The project illustrates the Gospel story in
-: the erotic potential which Thomas refused to accept Jesus's miraculous resurrection without first literally
:, iüuch her thinly- touching the risen Christ's wounds: until'I shall . . . put my finger into the print of the
:i-ilr to one another nails, and thrust my hand into his side, I will not believe' (John zo:zl) (Fig. 3.ro). It was
-L;e,i his full-colour precisely sculpture's ability to give tactile proof to doubtful beholders that became
:¿rirting over sculp- a key theme of many pro-sculpture arguments presented by Early Modern writers.
re -ater presented in Although the subject of Thomas's incredulity had been assigned to Verrocchio by
::¿rdo's virtuosity in Florence's Mercanzia, the chamber of commerce-like organization that had commis-
',-.qelher and instead sioned the sculpture, his design nevertheless thematizes the much-contested sense
r¿: \-asari was a key of touch. Indeed, in some ways, one could argue that touch itself is the real subject
74 GERALDINE A. JOHNSON
i'l
jt
txI
Fig. 3.10 An
bronze, I 461
ambitious sculpt,-,:
Fig. 3.9 Andrea del Verrocchio, Bust of a Lady with
the art thel'pract-..
Flowers, marble, c. mid-1470s. Florence, Bargello Museum.
Similarlv, a rt¡, :
(Photograph : Geraldine A. ..lohnson.)
Fanti in 15z6 tric:
Michelangelo hi:::s
of this sculpture. In their writings, later advocates of sculpture such as Varchi and the process oi c¿l-,.
Il Tribolo repeatedly verbalized what Verrocchio had already implied visually in his At the same ti:r..
statue group, namely, that sculpture's tactile qualities were proof of its greater truthful- labour invoh-ed r::
ness to nature, the ultimate goal of art in the eyes of manypre-Modern artists, patrons, work, swinging L-'
and theorists. ure's abdomen. s--
Tactility was also an important issue in discussions about the relative social status of pro-painting aci-' .-, :
painters versus sculptors. From the middle decades of the fifteenth century onwards,
The sculptor in .:.¡
sculptors with social and intellectual aspirations used a variety of strategies in their writ-
hammer...amc:::
ings to try to downplay the manual labour involved in producing sculpture, thereby is smeared ¿ll ¡r.::
_.,,_
confirming their status as gentlemen-artists rather than as mere artisans. Although and filled u,ith il:.:.'
sculptor-writers such as Ghiberti and Gauricus strived to include in their texts
numerous Classical references and suggested that the cerebral process of design or
disegno was crucial to the production of sculpture, nevertheless their treatises also This descriptic r
inevitably included digressions on technical matters more closely associated with as an educated s.-
artisan traditions than with the concerns of gentlemen and scholars-something that contact associa:e:
happens much less frequently in the theoretical writings of painters (Gauricus 1969; How different
':. --
Ghiberti 1947). This suggests that, in their writings, even intellectually and socially ease. He is rvell d:.'
THE ART OF TOUCH IN EARLY MODERN ITALY 75
!
Fig. 3.10 Andrea del Verrocchio, lncredulity of St Thomas (detail),
bronze,'1467 -83. Florence, Orsanmichele.
(Photograph : Stefan Bauer (http://commons.wikimedia.org).)
ambitious sculptors could not completely avoid the manual and material aspects of
l,r.r*. the art they practised.
Similarly, a woodcut in a popular fortune-telling book compiled by Sigismondo
Fanti in t5z6 tried to put a humanistic veneer on the scene by depicting the great
Michelangelo himself in the guise of a loincloth-wearing Classical sculptor while in
i*--:'r ¿s Varchi and
the process of carving his Dawn for the Medici Chapel in Florence (lohnson zoozb).
r:--e ; ,,rsuallf in his
At the same time, however, the print unavoidably highlights the intense physical
- -,s :i¿ater truthful- labour involved in producing sculpture. In fact, the image of Michelangelo hard at
1;:r :rtists, patrons, work, swinging his hammer and with his knee wedged uncomfortably into the fig-
ure's abdomen, seems to confirm visually the negative views of sculptors presented by
l:'::-,-¿ srrcial status of pro-painting advocates such as Leonardo:
:l ;entun'onwards,
:=: ¿Eres in their writ- The sculptor in creating his work does so by the strength of his arm and the strokes of his
hammer...amostmechanicalexerciseoftenaccompaniedbymuchperspiration...Hisface
: s:.i1pture, thereby
is smeared all over with marble powder so that he looks like a baker . . . and his house is dirty
r ::tisans. Although
and filled with flakes and dust of stone.
:-*.1¿ in theirtexts
(Leonardo 952, p. yo)
r:,-^;ess of design or
i :reir treatises also This description stands in sharp contrast to Leonardo's vision of the ideal painter
¡¡-.,' ¿ssociated with as an educated gentleman who avoids the manual toil and all-too-intimate physical
,:.-something that contact associated with the grimy work of the sweating sculptor:
:¿is Gauricus 1969;
How different [is] the painter's lot . . . for the painter sits in front of his work at perfect
.-:-l¿llr- and socially ease. He is well dressed and moves a very light brush dipped in delicate color . . . his home
76 GERALDINE A. JOHNSON
is clean . . . and he often is accompanied by music or by the reading of various beautiful simply can't keep h
works to which he can listen with great pleasure without the interference of hammering and practises is inadr-e¡
other noises. female statuette.
(Leonardo 1952, p. 33o) A number of sL.
literally man-hanC
One of the most influential texts on the relative status of painters and sculptors
should attend care
was Baldassarre Castiglione's Book of the Courtier of ryz8.In this book, a debate on the
encounters with scl
paragone takes place, with painting emerging as the proper art of the ideal gentle-
tioned earlier, the t,
man-courtier (Castiglione 1967, pp. 98-ror). The sculptor Baccio Bandinelli strived living womafl, ráttlr
to embody Castiglione's ideal of the gentleman-artist in both his life and his sculp-
ished bronze (see F:
tural practices. Bandinelli's social ambitions, for instance, are revealed by his almost
visually the sculpte
inordinate pride in having been made a knight of Sant'Iago. In an engraving of his
and the engrar.ing t
studio made just three years after Castiglione's book appeared, Bandinelli also seems
the touched seerTrs'
to have succeeded in banishing the sweat and dust of the working-class artisan's
artist or beholder c.
shop from his sculpture academy (Fig. ¡.n).Instead, the well-dressed apprentices that
comparison to an ¿
surround him are shown sketching Classicizing statuettes. However, in an uninten-
tively reciprocal an,
tionally telling detail, Bandinelli himself, despite his social and academic pretensions,
between object an,
asserted, the act of:
ing" and being "to;
The compleritie,
fact that some imas
in Paolo Veronese'
Museum of Art c
(Vienna, Kunsthist
homoerotic aspecrii
sion about sculptu:
portraits has been
(Currie 1997, pp. 1r
sculptors are olten
could also be depii
believed to be the ir
Unlike the imag
produced paintiee
objects. Instead. p
maintain an intel,
art objects (Currie
of Pierantonio B'tr:
Canada), do not er
beside them (Fie. -r
cable, hower-er, L-
painting in the .r:r,
Fig. 3.11 Agostino Veneziano (after Baccio Bandinelli), The Academy Bronzino unambie
of Baccio Bandinelli, engraving, 1531. Oxford, Ashmolean Museum. two fsides of this c
(Photograph : Ashmolean [/]useum, with permíssion.) side, that is u-ith ri
THE ART OF TOUCH IN EARLY MODERN ITALY 77
:ri .rf \-arious beautiful simply can't keep his hands off of sculpture: the inevitable tactile allure of the art he
,::¡e of hammering and practises is inadvertently demonstrated here by his two-handed handling of a nude
female statuette.
Leona¡do t952, p. yo) A number of sixteenth-century paintings also depict male artists or collectors
¿ln:ers and sculptors
literally man-handling nude female sculptures, something that suggests that one
itre¡k, á debate on the
should attend carefully to the gendered power relations implicit in such manual
'l rrf the ideal gentle- encounters with sculpture. In the case of the Portrait of lacopo Stradaby Titian men-
::c, Bandinelli strived
tioned earlier, the female statuette held by the sitter is painted in the flesh tones of a
':i-e living woman, rather than in the colours of white marble, plaster, terracotta, or pol-
::s and his sculp-
ished bronze (see Fig. 3.2).In other words, Titian's chromatic palette serves to equate
ei e¿led b). his almost
visually the sculpted female body with the body of a living woman. In this portrait
: in engraving of his
and the engraving of Bandinelli's academy, the relationship between the toucher and
3¿rdl¡relli also seems
the touched seems to remain essentially hierarchical and heterosexual, with the male
,-,rklnq-class artisan's
artist or beholder clearly in control of the sculpted female body. At the same time, in
:'s sed apprentices that
r ¿i'er. i¡ an uninten-
comparison to an act of ocular scrutiny, this type of tactile relationship is compara-
tively reciprocal and thus retains the potential to subvert the hierarchical relationship
;:¿enic pretensions,
between object and beholder. Indeed, as the philosopher Maurice Merleau-Ponty
asserted, the act of touching is 'an ambiguous set-up in which . . . the róles of "touch-
ing" and being "touched"' can alternate (Merleau-Ponty ry62,p. y).
The complexities of tactile engagements with sculpture are also demonstrated by the
fact that some images depict male sitters holding sculpted male figures as, for instance,
in Paolo Veronese's Mqn with a Stqtuette of St Sebastian (New York, Metropolitan
Museum of Art) or Giovanni Battista Moroni's Man with an Antique Male Torso
(Vienna, Kunsthistorisches Museum) . Although it would be interesting to consider the
homoerotic aspects of these same-sex manual encounters, in the context of a discus-
sion about sculptural tactility it is perhaps most relevant to note that the sitter in both
portraits has been tentatively identified as the Venetian sculptor Alessandro Vittoria
(Curriet997,pp.tzz1).If this is true, one could suggest that, while male collectors and
sculptors are often shown possessively holding sculpted female bodies, a male sculptor
could also be depicted gripping a male body, which Early Modern culture in general
believed to be the ideal and normative human form.
Unlike the images encountered thus far, one sixteenth-century artist in particular
produced paintings in which sitters resolutely avoid manual contact with sculpted
objects. Instead, portraits by the painter Agnolo Bronzino often depict sitters who
maintain an intellectual, emotional, and physical distance from three-dimensional
art objects (Currie 1997). Significantly, Bronzino's haughty sitters, as in his Portrqit
of Pierantonio Bandini with a Venus Pudica Statuette (Ottawa, National Gallery of
Canada), do not even deign to look at, let alone touch, the sculpted figures displayed
beside them (Fig.3.rz). This determined antitactilityvis-á-ús sculpture is readilyexpli-
cable, however, if one realizes that Bronzino was one of the strongest advocates of
painting in the ongoing parogone debate. In his response to Varchi's questionnaire,
Tke Academy Bronzino unambiguously stated his position: 'I intend to align myself with one of the
ln ¡r'luseum. two [sides of this debate], because I honestly feel I am aligning myself with the truer
,- /, :' oermission.) side, that is with the side of painting' (Barocchi 1977, p.loo). It is thus not surprising
78 GERALDINE A, JOHNSON
the beloved's inner emotions. In other sonnets, the sculptor's god-like hand is seen
as a powerful, life-giving force that is capable of animating carved figures. For exam-
ple, in one of his best-known poems,'Non ha l'artista...' (Sonnet r5r), Michelangelo
describes a sculptor bringing a statue to life with 'the hand that obeys the mind'
(Michelangelo 1975, p. zrz) .
Michelangelo also thematized hands in many of his statues, as in the autoerotic,
probing hand of the so-called Dying Slave (Paris, Louvre Museum), which recalls the
hand of the sculptor-as-lover described in some of his poems, or the dramatically
oversized hands of his famous marble David. Likewise, Michelangelo's Moses (Rome,
San Pietro in Vincoli), who insistently fingers his flowing beard, alludes visually to the
important role played by touch in the enjoyment and evaluation of sculpted forms
(Fig. ¡.r:). Even in the design he provided for Jacopo Pontormo's painting of Noli me
Tangere (Florence, Casa Buonarroti), the focus is on the figures'hands: Christ, whose
index fi.nger seems just to touch Mary Magdalene's breast, and the Magdalene herself,
'!1 0
i cod,
,
r'r..ts de-emphasize,
ri.r-:o imitate coolly
I: :¿ct. according to
:elium bv definition
:::s tr¡ ¿rt are the lines
:; three-dimensional
s--,ltors themselves
:;. In \fichelangelo's
:',; f'tO-dimensional
',\'as best the more it
: r: resembled paint-
¡ t¿ctili¡- is particu- Fig. 3.13 Michelangelo Buonarroti, Moses (detail),
¡..1 .abour inr.olved in marble, c.1506-16. Rome, San Pietro a Vincoli.
:: s lesire to uncover (Photograph: Geraldine A. Johnson.)
8o GERALDINE A. IOHNSON
who unusually for this subject appears actually to be grasping lesus's back (Wallace
1988) . The prominence of hands in these and other works associated with Michelangelo
suggests that, at some level, this member may have functioned for the sculptor as a
visual syrrecdoche, that is, as a part symbolically representing the sculptor-or perhaps
the art of sculpture-as a whole.
However, some of Michelangelo's contemporaries who were avowed advocates
ofpainting repeatedlytriedto denythe physicalityofhis sculptures. Vasari, for example,
praised the beard of Michelangelo's Mosesforbeing so well made that it almost looked
painted, the highest compliment a painter could give, but one that seems wilfully to
ignore the material reality of the three-dimensional medium (Barolsky ry95, p. ú).
Similarly, another painter, Francisco da Hollanda, referred to Michelangelo's statues
in the Medici Chapel as paintings in marble (Summers r98r, p. 263). Such a response to
Michelangelo's insistent tactility can best be explained in light of the ongoing p aragone
debate in which painters and their supporters were eager to claim Michelangelo, the
great universal artist of the sixteenth century, for their own camp.
In the seventeenth century, Gian Lorenzo Bernini thematized tactility in some of his
sculptural projects as well. In the Apollo and Daphne (Rome, Galleria Borghese), for
example, Bernini depicted the very moment when the god's touch begins to transform Fig.3.1a G
the nyrnph into a laurel tree (Bolland zooo). Ovid's description of this instant may Albertoni{a
even have suggested to Bernini that Daphne was turning into a white marble maiden,
for just at the very moment that Apollo nearly embraces her, his breath already touch-
ing her hair, the nymph 'grew pale' (Ovid rq:¡, p. 43). Tactility is alluded to not only
in the narrative moment chosen, but also in Bernini's virtuoso sculptural technique
RErsnENcrs
itsell which depicts Daphne's delicate fingers being transformed into wafer-thin laurel
leaves and her smooth flesh metamorphosing into the rough bark of a tree, thereby
Alberti LB (r9;: . tl,
reminding beholders of the transformative powers of the sculptor's own god-like (C Grayson trani
hands. The sculptor's almost divine ability to use his hands to transform a marble block Aurenhammer HH
into a maiden is alluded to by showing the maiden herself mutating into a different und Skulptur in .
sculptural material, namely, wood. Like Michelangelo, Bernini also focuses particular 355-410.
visual attention on the hand itself in a number of his works. For instance, in his monu- Bann S (1989). I,;i ,
Universiry Presr.
ment for the Blessed Lodovica Albertoni (Rome, San Francesco a Ripa), the power
Barocchi P (ed. i;
traditionallyassociatedwith the talismanic touch ofthe saint is linked to an autoerotic,
editore, irdilan ::
manual exploration of the female body. Indeed, Bernini's sophisticated manipulation BarolskyP (r99¡ . T1
of light and shadow in his design serves to highlight Lodovica's probing marble fingers and Industn' tt ::
against her decorously covered breast (Fig. 3.r4). Athens, GA.
one could cite many more examples, visual as well as textual, that would confirm Baxandall NI (r9S -
the importance Early Modern culture accorded to issues related to touch and tactility, New Haven. CT
especially in relation to sculpture. By reconsidering fifteenth-, sixteenth-, and seven- BeltingH (r98r . f,.
der Passion. Geb:
teenth-century texts, images, and objects in tight of these concerns, important new
Berenson B (169-
insights can be gained into the reception of art in Italy in this period. perhaps more
New York.
importantly, by attending to tactility in one of art history's paradigmatic moments, we BollandA (zooo . -l
can also embark on a more general exploration of the sense of touch in relation to past Daphne. Art B:,:"
and present artistic practices, as well as can begin to consider more broadly why the Boyle MO (1998 . ,r;
reception of art objects should not be limited to optical experiences alone. EJ Brill, Leiden.
THE ART OF TOUCH IN EARLY MODERN ITALY 8r
Merleau-Ponn'\l :;
(Originally publ. 15o4.)
Paul, London. Or
Ghiberti L (ts+). I commentari (o Morisani ed.). Riccardo Ricciardi editore,
Naples'
Michelangelo Buoi.:
(Originally written c. late r44os.)
Spirituaiity" (Originalh' urir¿r:
Hale Ró e99z).'Imitatio Mariae:Motherhood Motifs in Late Medieval German
Miles M (1983). \'is.,:
Unpublished Harvard University Ph.D., Cambridge, MA'
trinitate and C¡'::-
Ha[ I ?rqqq). Theworld as sculpture: The changing status of sculpture from the Renaissance to
MontaguA(rgE6.I
the Present Day' Chatto & Windus, London'
of New York. (t'- ¡:¡:
Harvey ED (ed.) (zoo3). sensible Flesh: on Touch in Early Modern culture. university
PA. MontaguJ(rqq6.G
Pennsylvania Press, Philadelphia'
Universifr- Press. !
Hecht P (rq3+). The ParagoneDebate: Ten Illustrations and a Commenl. Simiolus,t4,725-36' -
Morgan MI (tq.
Hildebrand A von (rsg¡).ba s Problem der Form in der bildenden Kunst' JHE Heitz, Strasbourg'
.
Cambridge Unr-,:r
Hirst M (1996). Michelangelo and his First Biographers. Proceedings of the British Academy,
Nordenfalk C r9S;
94'q-84. Warburg ar:.i C::"
Uowes D, ed. (r99r). The Varieties of Sensory Experience: A Sourcebook in the Anthropology
of
Nordenfalk C 'i99,
the Senses. University of Toronto Press, Toronto'
University and the Vl-.1i.;i: :,
Howes D (zoo:). Sensual Relations: Engagingthe Senses in Culture and SocialTheory.
New York.
of Michigan Press, Ann Arbor, MI.
Olin M (1989). \--r;
Howes D, ed. (zoo5). Empire of the senses: The sensual culture Reader. Berg, oxford.
144-72.
Irigaray L (1985). This ser which is Not one (c Porter trans. with c Burke.). cornell university
NY. (Originally ptbl. tgll.) Olin M (r99:.,. ,r:'"
Press, Ithaca,
Universitv Pr¿ss.
IserW(rqZ8). TheActofReadingATheoryofAestheticResponse.TheJohnsHopkinsUniversity
MD' Ovid (rqst). Tltt )'i:
Press, Baltimore,
PanofskyE(rg:;.1
Iversen M (rss¡). Alois Riegl: Art History and Theory' MIT Press, Cambridge' MA'
and the Paragone. artibus et 1924-1925,Pr.:'
Jacobs FA (rqSS). An Assessment of Contour Line: Vasari, Cellini
PanofskyE (r95: . t
historiae, 9 (r8), r39-5o.
w Fink' Munich. Petrelli M (1994 r,'-
Iauss HR (tg7). Asthetische Erfahrung und literarische Hermeneutik
THE ART OF TOUCH IN EARLY MODERN ITALY 8¡
'-'. )'--: Prtttce in Ducal M (rgq¡). Downcast Eyes: The Denigration of Vision in Twentieth-Century French Thought.
Jay
University of California Press, Berkeley, CA.
, : --::ir ersirv Press, fohnson GA (zooza). Touch, Tactility, and the Reception of Sculpture in Early Modern Italy.
In P Smith and C Wilde (eds.) The Blackwell Companion to Art Theory, pp. 6r-74. Blackwell
:-i. n¿rmondsworth. Publishing, Oxford.
Johnson GA (zoozb). Michelangelo, Canon Formation and Fortune Telling in Fantr'sTriompho
:.:, House, Ferrara. di Fortuna.In L Jones and L Matthew (eds.) Coming About: A Festschrift for John Shearman,
pp. rg9-2o5. Har-vard University Art Museums, Cambridge, MA.
Kemp W G9q6). Der Anteil des Betrachters: Rezeptionsiisthetische Studien zur Malerei des 19.
".
: l''.,'. :: : ¿ ntlt C entury. J ahrhunderts. Máander Verlag, Munich.
Klapisch-Zuber C (1985). Holy dolls: play and piety in Florence in the Quattrocento.ln idem,
:r.:;.-.Íraits: Apre- Women, Family, andRitualinRenaissance Italy (LG Cochrane trans.), pp.3ro-29. University
-:..; I r-,.-;¡ 1100-1700, ofChicago Press, Chicago, IL.
Koerner IL (rSS:). The Moment of Self-Portraiture in German Renaissance Arr. University of
: .. l-:':-::iittn o.f Forms Chicago Press, Chicago, IL.
. ', :*bl. r99:.) Leonardo da Vinci (t952). The Notebooks of Leonardo da Vinci (IA Richter ed.). Oxford
'., .:, :''. :ite Ere of the University Press, Oxford. (Originaily written c. late r5'h-early r6th century.)
Lindberg DC (rgZ6). Theories of Vision from Al-Kindi to Kepler. University of Chicago Press,
::r: .; -'\-.1' Edition of Chicago, IL.
Marks LU (zooz). Touch: Sensuous theory and multisensory media. University of Minnesota
r .::. ,,,¡i of Diderot. Press, Minneapolis, MN.
Mendelsohn L (rq8z). Paragoni: Benedetto Varchi'sDueLezzioni and Cinquecento Art Criticísm.
:.:¡ D:oz. Geneva. UMI Research Press, Ann Arbor, MI.
Merleau-Ponty M (1962). Phenomenology of Perception (C Smith trans.). Routledge and Kegan
..::, ::iiore, Naples. Paul, London. (Originally publ. 1945.)
Michelangelo Buonarroti (r97). Rime (G Testori intro. and E Barelli ed.). Rizzoli, Milan.
- -': :::r::r Spirituality.' ( Originaily written r6'h century. )
Miles M (1983). Vision: the eye of the body and the eye of the mind in Saint Augustine's De
t- , . .j'"; R¿tnissance to trinitate and Cont'essions. Journal of Religton,63, rz5-42.
Montagu A (rq86). Touching The Human Signifcance of the Skin,3'd edn. Perennial Library,
-. :'..':. University of New York. (r" edn. publ. r97r.)
Montagu I GSS6). GoId, silyer, and bronze: Metal sculpture of the Roman baroc1ue. Princeton
. :,': :.', :r-., 14, 12516. University Press, Princeton, NJ.
--¡ H:irz, Strasbourg. " Morgan MI GSZil. Molyneux's Question: Vision, Touch and the Philosophy of Perception.
- .. S-i:ish Academy, Cambridge University Press, Cambridge.
Nordenfalk C (rq8s). The Five Senses in Late Medieval and Renaissance Art. lournal of the
' . .,'.¿ .t,l1rlffopology of Warburg and Courtauld Institutes,4S, t-22.
Nordenfalk C (r99o). The Sense of Touch in Art. In K-L Selig and E Sears (eds.) The Verbal
-:.,. , ::¡¡r,r'. Llniversity and the Visual: Essays in Honor of William Sebastian Heckscher, pp. ro9-32. Italica Press,
New York.
:.::. ,,lrlbrd. Olin M (rg3g). Vaiidation by Touch in Kandinsky's Early Abstracf Arf. Critical Inc1uiry, t6,
: . rl.rrneil University 144-72.
Olin M Q99z). Forms of Representation in Alois Riegl's Theory of Art. Pennsylvania State
. --;¡kinsUniversity Universiry Press, University Park, PA.
Ovid (rgSS). The Metamorphoses (MM Innes trans.). Penguin Books, Harnrondsworth.
Panofsky E j9z7). Dte Perspektive als 'symbolische Form'. ln Vortriige der Bibliothek Warburg
_- l_ \ I \
-:.,',t.¡0,,r.
.- artibus et 1924-1925, pp. 258-33o. GB Teubner, Leipzig.
Panofsky E (rSS+). Galileo as a Critic of the Arts. Matinus Nijhoff' The Hague.
: r... \iunich. Petrelli M (rss+). Valori tattili e arte del sensibile. Alltnea editrice, Florence.
8q GERALDINE A. JOHNSON
Podro M Q98z). The Critical Historians of Art. Yale University Press, New Haven, CT.
Powell KH (1997). Hands-On Surrealism. Art History,zo,5l.6-33'
Riegl A (t9ol. Die spAftijmische Kunst-Industrie, nach den Funden in Ósterreich-(Jngarn, L
KK Hof- und Staatsdruckerei, Vienna.
Riviin R and Gravelle K (rq8+). Deciphering the Senses: The Expandíng World of Human
Perception. Simon and Schuster, NewYork.
Schulz J (1975). Michelangelo's Unñnished Works. Art Bulletin,57,366-73'
Sherman CR, ed. (zooo). Writing on Hands: Memory and Knowledge in Early Modern Europe.
University of Washington Press, Seattle' WA.
ShiffR (rqqr). Cézanne's physicality: The politics of touch. In S Kemal and I Gaskell (eds.) The
Language of Art History, pp. lz9-8o. Cambridge University Press, Cambridge'
Stewari S (zooz). Facing, Touch, and Vertigo. In S Stewart (ed.) Poetry and the Fate of the
pp. 145-95. University of Chicago Press, Chicago, IL'
Senses,
Summers D (rqSr). Michelangelo and the Language of Art. Princeton University Press'
Princeton, Nl.
P]
Summers D (tg9l). The ludgment of Sense: Renaissance Naturalism and the Rise of Aesthetics.
Cambridge University Press, Cambridge
Synnott A (r99r). Puzzling over the Senses: From Plato to Marx' In D Howes (ed.) TheVarieties
of Sensory Experience: A Sourcebook in the Anthropology of the Senses, pp. 6r-7ó. Universiry
of Toronto Press, Toronto.
Synnott A (:9y) . The Body Social: Symbolism, SeIf and Society. Routledge, New York'
The Senses and Society (zoo6-). Berg, Oxford.
Thomas B (tqq6). 'The paragone debate and sixteenth-century Italian art, I-IL' Unpublished
D.Phil., University of Oxford.
Vasari G (t96). Lives of the Artists (G Bull trans.). Penguin Books, Harmondsworth.
Vasseleu C (rqq8). Textures of Light: Vision and Touch in lrigaray, Levinas and Meileau-Ponty.
Routledge, London.
Wallace WE (rq88). ll NoIi me Tangere di Michelangelo: tra sacro e profano. Arte Cristiana,
76, 443-50.
White J Q9g). Paragona Aspects of the Relationship between Painting and Sculpture.
ln idem, Studies in Renaissance Art, pp. t-67.Pindar Press, London'
Wólfflin H (r9r5). Kunstgeschichtliche Grundbegrffi: Das Problem der Stilentwicklung in der TuE majorin' of t.
neuer en Kunst. F Bruckmann, Munich. vision, hearing, to-
Wood CS (1998). Germany's Blind Renaissance. In M Reinhart (ed.) InfiniteBoundaries: Order, entirely separate ¿l
Disorder, and Reorder in Early Modern German Cuhure, pp. 225-44. Sixteenth Century segmentation oilh,
f ournai Publishers, Kirksville. well-being and the
Yates FA (ry66). The Art of Memory. Routledge and K Paul, London. senses (Baillr'Dur
(Lindstrom zoo5 r
entlyalways has be¡
anythought gir-er:
however, oúf s€lSr
and events. \\trat's
sorially determinec
2oo2; Calvert et el.
hear, and smeli c¿r
An extensir.e bt-r"
olfactory cues can
CHAPTER 4
THE
MULTISENSORY
' ::. :;-i.' of Aesthetics.
PERCEPTION OF
., ::. Tlte \rarieties
, :: :t--6. University
TOUCH
.: .'" ;r', \ ork.
(such as the texture) ofhaptically explored (i.e. explored through active touch) objects For example, Gib
and surfaces. Visual cues have also been shown to dominate over the sense of touch a straight metre sli*
when people evaluate an object's structural properties (e.g. Gibson 1933; Rock and through lenses th¿t
Victor t964), and applied researchers have demonstrated that the information avail- their eyes, horter-er
able to the various senses can all contribute to an object's perceived pleasantness and as, of course, i¡ re¿
functionality (see Spence andZampini zoo6i Schifferstein and Spence, zooS). When 1964; Rock and Hr
taken together, this rapidly-growing body of empirical research from the fields of to rate their impre:
experimental psychology, psychophysics, cognitive neuroscience, and multisensory or both see and ree
product design all converges on the conclusion that whenever people think about the a square object t-or
factors that affect people's touching and handling of objects, they really need to con- made the square Lr,
sider (and be made aware of) the variety of sensory inputs that may be contributing ticipants were un¿1'
to people's evaluation and appreciation ofthose objects. This article provides a brief such conditions, ",i
overview of the literature on multisensory perception that is designed to demonstrate Further experimer
just how dramatically the various senses can influence people's 'tactile' perception instead gave rise :c'
of many different objects and surface qualities, no matter whether people realize il these led Rock anc
or not. touch and even sha
The two articles
journals Science ar,'
thinking (and erae
(see Warren and R,
studies replicated :
SnNsoRY DOMINANCE AND VISUAL CAPTURE
According to the traditional view, touch was thought to educate vision (see Berkeley
t7o9lt957; Arnold zoo3; Classen zoo5). However, early empirical research came to
exactly the opposite conclusion concerning the interaction between vision and touch.
Perhaps the earliest data to be reported on this question was highlighted by Sir David
Brewster (1832) in his book Narrz ral magic (see Epstein r97r). There, Brewster describes
how optically left-right reversing an indented object (an engraved watch seal) results
t;
I
elongated (B)) bv
' Welch and Warren ( 1980) have suggested that the term 'discrepancy' (or 'discordance') might be perception of the
more appropriate than 'conflict', since, according to them, the latter term connotes an awareness that the participa
of, and possibly an aversive response toward, the stimulus situation (cf. Derrick and Dewar 1970).
However, 'intersensory conflict' is the term that has been used more frequently in the literature over
it to be visually [i
the years. when the two ser
THE MULTISENSORY PERCEPTION OF TOUCH 8l
,::ri'e touch) objects For example, Gibson (1933) showed that when people ran their fingers up and down
:: ,ii. sense of touch a straight metre stick they perceived it as being curved if they simultaneously looked
ri - r 19,13; Rock and through lenses that made the stick appear curved. As soon as the participants closed
" :rl.¡rmation avail- their eyes, however, or turned their gaze away, the edge of the rule felt straight again,
:J p-easantness and as, of course, in reality, it was. Meanwhile, Rock and his colleagues (Rock and Victor
:.i--;.. :oo8). When 1964; Rock and Harris 1967) reporled a number of experiments in which people had
- :: -,1-] the fields of to rate their impression of the size of a small object which they could either see, feel,
: :¡,d multisensory or both see and feel at the same time. In one experiment, the participants looked at
. :-= ::ink about the a square object for five seconds through a pair of distorting (reducing) lenses which
'" :¡:lli' need to con- made the square look as if it was only half as large as it actually was (although the par-
:-,:-, b¿ contributing ticipants were unaware of this fact), and then indicate how large the square was. Under
-.:,: :rrrvides a brief such conditions, vision was found to dominate completely over touch (see Fig. +.rA).
r:-:: :.r demonstrate Further experiments in which distorting lenses that made the square look rectangular
, :.;-r-e' perception instead gave rise to exactly the same pattern of results (see Fig. +.rB). Findings such as
--,., :¡-.ple realize il these led Rock and Harris Qg6Z,p.96) to conclude that'vision completely dominates
touch and even shapes it'.
The two articles published by Rock and his colleagues in the prestigious scientific
journals ScienceandscientiJicAmericanduringtherg6oshadaprofoundimpactonthe
thinking (and experimentation) of manypsychologists over the decades that followed
(see Warren and Rossano 1991, for a review). While the majority of these subsequent
CAPTURE studies replicated the basic finding that vision can modulate tactile/haptic perception
.
-:r..r see Berkeley A) B)
.- :..r::Ch Came tO Vision Touch Vision Vision Touch Vision
:: ', -)tL-1n and touCh. alone alone + touch alone alone + touch
of the size, volume, and shape (i.e. structural properties) of an object when the sensory Converging evider
cues relating to that object were put into conflict, it is important to note that not all fine surface texture
of the studies showed a complete dominance of vision over touch (e.g. Over 1966; touch was shown ttr
Kinney and Luria r97o; McDonnell and Duffett ry72; Mlller rqzz; Fishkin et al. ry75; smooth JaPanese a:
Power and Graham t976;Power r98o, r98r; Heller r99z). For example, Fishkin et al. have shown recenti''
reported an experiment in which they observed strong, but incomplete, visual cap- ment of fine surfac: :
ture in a slant judgement task in which the participants had to orient a bar until it was concerning the rcL-¡
horizontal, an approximate compromise between the senses of vision and touch in a (i.e. fabric samPle. ---
length judgement task, while a trend toward tactile capture was observed in a texture abrasion), discrin-r:
matching task (see Fig. a.z) . Lederman and Abbott (r98r) also found that participants or both senses to:¿::
tended to made compromise judgements (weighing the two modalities about equally) pilled fabric sam:-:,
when vision and touch were put into conflict in a task in which they had to match the accurately usins '- *
texture of two abrasive sandpaper samples. Meanwhile, a long series of experiments having slighth- d'-:-
--
by Lederman, Thorne, and fones (rg86) showed that the extent to which one sensory their fingerttPS n: ':
input is preferred over another may depend on the nature of the task that partici- . participants' iu.l-. :'
pants have to perform. They were able to demonstrate that certain texture discrim-
ination tasks appear better suited (or more appropriate) to vision (e.g. determining
the spatial density of texture elements), while others appear more appropriate to OPaque ci.-
touch (e.g. determining the roughness of fine textures). 0.45-
o
L
CL ._'----..-------
^^
o
G
.A 40
\o (a)
Fig.4.3 A) Sche-
(2007) in order:¡
Slant judgemenl Length judgement Texture judgement perception. SePar;
Task upon individua ','I
designed so that P
Fig. a.2 Results of a study by Fishkin et al. (1975) highlighting the task- feeling the same'
dependency of the visual dominance (or capture) of touch. A variety of different from the same sc¿
distorting lenses were used in order to introduce a conflict between what when incongrueni
participants saw and what they felt. The percentage of visual capture represents were presented lt:
the extent to which participants' judgements were consistent with what they dominates oV€r ,' :
saw visually. lf the participants had weighted the input from the two modalities on the surface cf
equally, then 500/o visual capture should have been observed (dashed line in that touch can Cc
'.:i .,.,'hen the sensory Converging evidence for the dominance of touch over vision in the perception of
ll iL-¡ llote that not all frne surface texture comes from a study reported by Heller (lg8S), in which
,*J: e .g. Ot er 1966; touch was shown to be superior to vision in the judgement of the texture of very
:r Iishkin et al. ry75; smooth fapanese abrasive sharpening stones. Finally, Guest and Spence (zooZ)
.:::::-e. Fishkin et al. have shown recently that touch can dominate completely over vision in the judge-
:, ::f .ete, r'isual cap- ment of fine surface texture. The participants in their study had to make judgements
"--:l a bar until it was concerning the roughness of the surface texture of a series of pilled fabric samples
,r,:,:r and touch in a (i.e. fabric samples that had been roughened to varying degrees through a process of
:.:r-ed in a texture abrasion), discriminating between pairs of pilled fabric samples using touch, vision,
-:r: :rat participants or both senses together. Participants were able to discriminate befween the pairs of
.-::res about equally) pilled fabric samples (in terms of which sample was more pilled, of rougher) more
-.',' r¿d to match the accurately using touch alone than using just vision. However, when fabric samples
:ri:: r-ri experiments having slightly different pill values were presented to the participants' eyes and to
r,ci one sensory their fingertips at the same time, touch dominated completely over vision in terms of
.,,
.-.: :l:. that partici- participants' judgements of how pilled the samples were (see Fig. +.¡). These results
::-,r ':t\iure diSCrim-
, :.S. determining
'. :- :Fpropriate to Opaque cloth
t.3
Ol
o
o u.J
9o
o Touch Vision+
.2
o -nc
-" Touch
o
ór
-1.5
Sensory Modality
Fig. 4.3 A) Schematic view of the experimental set-up used by Guest and Spence
(2OO7) in order to assess the relative contributions of vision and touch to texture
_: - ram^ñl
! -.sJJv rrvrrr perception. Separate pilled fabric samples were presented to touch and vision
upon individually controlled motorized wheels. Note that the apparatus was
designed so that participants were given the impression that they were seeing and
+^-1,
-^
- i^ Ld)K- feeling the same fabric (i.e. the two sensory impressions appeared to originate
;r ew of different from the same spatial position). B) Bar graph highlighting the results obtained
\ e3n 'rVhat when incongruent stimuli (i.e. fabric swatches having different pilling values)
¡3ILl:e represents were presented to vision and to touch. These results show that touch completely
:¡ ,¡,'hat they dominates over visual in the perception of the roughness (or amount of pilling)
e :¡;o modalities on the surface of these fabric swatches. These results are consistent with the view
;s'ed Iine in that touch can dominate over vision in the perception of very fine surface texture
(i.e. microgeometric surface features), presumably because by itself touch is better
- :: ¿ . 1975, Table 1.)
at this task than is vision.
9o CHARLES SPENCE
therefore demonstrate that tactile cues can dominate completely over vision, at least specialized for ditler<
when people have to evaluate fine pilled fabrics (see also Guest and Spence zoo3a). accurate modali¡ ¡'ri
Researchers have observed that the influence of vision on tactile perception tends ments under condit--'
to be more pronounced when the stimuli in the two modalities are presented (or modality appropriat,
appear to originate) from the same spatial position (Miller, r97z; Gephstein et al. zoo5; dominate over tactil.
Congedo et al. zoo6; though see also Helbig and Ernst zooT). This may, in part, be sory conflict is slmP
because people are more likely to believe that various different sensory impressions accurate, or aPProPrl
belong together (i.e. they are more likely to make an 'assumption of unity'; Spence (at least for the kin,l,
zooT; Welch and Warren r98o) when the stimuli originate from the same (rather than participants to m¿k¡
different) spatial location (and if they occur synchronously in time). The dominance brains tend to rteigl;
of vision over touch is larger when people assume that what they are seeing and feeling the task at hand n.';
actually refer to the same object or event (see Miller ry72;Welch r97z; Heineken and to make judgements
Schulte zooT; Spence zooT). Consequently, the majority of psychologists have tried (i.e. judgements con
to keep the participants in their studies naive to any intersensory conflict that may macrogeometric Prc
have been present within their experimental set-ups (e.g. Rock and Victor r964; Miller "
have to make iuCee ;
r97z; Ernst and Banks zooz). Although multisensory interactions can still take place if (i.e. when peoPle h;
people notice (or are made aware ofl the presence of intersensory conflict (e.g. Over Several researchers i
1966; Welch ry72;Heineken and Schulte zooT;Helbig and Ernst zooT),lhe magnitude sensory modalin ill
of any visual influence on tactile perception tends to be much weaker. What's more, ing a particular Per*
top-down factors and individual differences come to play a more important role in Lederman et a1. lgSt
determining any intersensory interactions that are observed (see Miller ry72;Welch the dominance oi i:
r,\-
and Warren r98o; Heller r99z). That is, once people become aware of the discrepancy aware of (see ¿|5¡r
between their senses, they then have the choice of responding either on the basis of
what they are seeing or on the basis of what they are feeling.
TgE n'r¡.
Eenrv THEoRTES oF sENSoRy DoMTNANcE
Posner, Nissen, and Klein Q976) put forward one of the first theories that tried to Ernst and Banks -
account for research on sensory dominance. They suggested that people may direct rently very poPul:
their attention more toward the visual sense due to the poor alerting, or arousing, experiments in r"-hi
qualities of visual stimuli. Posner et al. argued that this attentional bias often results accurate sensor,-:Tl
in (attended) visual inputs (i.e. those sensory impressions that people are concentrat- sensory attribute sl
ing on) having alarger influence on people's judgements than those inputs from the modality estirn.r:.:
other relatively 'less attended' senses. However, the emergence of a growing number theirworkrvas ihrt
of studies demonstrating a compromise between the senses, or demonstrating that the dominance l ould ,
other senses (such as touch) could apparently dominate vision under the appropriate estimation accoLl:'1.'
conditions (e.g. Fishkin et aL. t975; Lederman and Abbott r98r; Lederman et al. 1986; a manner that is "'
Welch et al. 1986; Guest and Spence zooT),led to the development of an alternative (see Ernst and Bui
account of sensory dominance, modality appropriateness (Welch and Warren r98o). tion of disparate :
Building on Freides's (ry74) earlier observation that different sensory modalities are (or variance assoc:
THE MULTISENSORY PERCEPTION OF TOUCH 91
specialized for different tasks, Welch and Warren went on to suggest that the most
accurate modality might actually be the one that will dominate over perceptual judge-
ments under conditions of conflict (see also Becker-Carus, 19n). According to the
modality appropriateness hypothesis, the reason why visual information tends to
dominate over tactile information so frequently in the published studies of intersen-
sory conflict is simply because vision is the sense that normally provides the most
accurate, or appropriate, source of information concerning the judgement being made
(at least for the kinds of perceptual judgements that psychologists tlpically ask their
participants to make). According to the modality appropriateness hlpothesis' our
brains tend to weight information in the sensory modality that is most appropriate to
the task at hand more heavily. That is, while vision may dominate when people have
to make judgements regarding the macrogeometric properties of an object or surface
(i.e. judgements concerning the structural properties of the object, or concerning the
macrogeometric properties of the surface texture), touch may dominate when they
have to make judgements concerning the microgeometric surface properties instead
(i.e. when people have to try and discriminate between very fine surface textures)'
Several researchers have supported the view that people may choose to attend to the
sensory modality that happens to provide the most accurate information concern-
ing a particular perceptual attribute (or judgement; e.g. Lederman and Abbott r98r;
Lederman et al. 1986; Heller r99z; Guest and Spence zoo3a,b, zooT), hence enhancing
the dominance of that modality over the multisensory percept that people become
aware of (see also Warren and Rossano r99r).
--..:ies that tried to Ernst and Banks (zooz) put forward the theory of sensory dominance that is cur-
; ¡3¡rple may direct rently very popular amongst researchers. They conducted an influential series of
;:'ii¡q. Or arOUSing, experiments in which they were able to show that the human brain weights the most
:i ¡ias often results accurate sensory modality (i.e. the modality in which people's estimates of a particular
-:l; are concentrat- sensory attribute shows the least variance) more heavily than the inputs from those
,.- s: rnputs from the modality estimates having more variance. The particularly novel contribution of
: : irorring number their work was that it allowed researchers to predict mathematically how much sensory
:rL- :tstrating that the dominance would occur in any given situation. According to the maximum-likelihood
.i;: ihe appropriate estimation account of sensory dominance, the human brain may actually operate in
.:iriran et ai. t986; a manner that is very close to that of a statistically optimal multisensory integrator
:r:: rrl an aiternative (see Ernst and Biilthoff zoo4, fot further details). That is, the multisensory integra-
. ard \\'arren r98o), tion of disparate unisensory inputs appears to maximally reduce the uncertainty of
:.,:,¡,' modalities are (or variance associated with) our multisensory estimates of external stimulus qualities
92 CHARLES SPENCE
A
I
I
(given that all of our sensory estimates are intrinsically noisy; see Fig. 4.4). In the i
experiments reported by Ernst and Banks, participants had to try and judge the height :c:
=
of a bar which could either be seen on a screen or else felt between the participant's (ú
-o
finger and thumb (using two force-feedback devices). Using this set-up, bars of either ol
the same or different height could be presented to vision and touch/haptics. Ernst and
Banks were able to show that whether people demonstrated visual dominance, tactile
dominance, or some intermediate weighting of vision and touch/haptics was criti- (a)
cally dependent on the variance associated with each of the sensory estimates (see also
Becker-Carus 1923; though see Derrick and Dewar rylo;Mlller tgzz).
Although Ernst and Banks's (zooz) original study used a virtual realityhaptic display
(thus raising some concern regarding the ecological validity of the stimulus displays
used), Helbig and Ernst (zoo7) have recently demonstrated a very similar pattern of !--
results under conditions where participants had to evaluate the shape of a number of (ó
real objects that were held in their hands. In fact, the maximum-likelihood estimation o
o-
account of sensory dominance has now been shown to provide a remarkably good
account of the relative contributions of the senses (audition, vision, touch, and prop-
rioception) to multisensory perception in a variety of different experimental settings
(e.g. see Alais and Burr zoo4; Andersen et al. zoo5; see Ernst and Btilthoff zoo4, for a
review). It should, however, be noted that some residual role for directed attention
may still be necessary (see Battaglia ef al. zoo3; though see also Witten and Knudsen
zoo5).
Fig.4.4 SchematLc
of sensory dominar
situations of inters
of sensory dominal
AuprroRY coNTRTBUTToNS To property (such as Í
functions for unim
TACTILE PERCEPTION
functions). Note tt
conflict. The two e
Researchers have demonstrated that manipulating the sounds that people hear when perceptual estimat
they touch a surface can have a dramatic effect on the perceived roughness of that is, we were to m
if
surface (Jousmáki and Hari 1998; Guest et aL. zooz). What's more, people's percep- object, those judgt
tion of the pleasantness, powerfulness, and forcefulness of many different products has to the maximum-
also been shown to be influenced by the sounds that they make when used (Zampini each sense to oul'
et al. zoo3; Susini et al. zoo4; Spence andZampinizooT). For example, Guest et al. associated with ea
highlighted a particularly dramatic demonstration of the auditory modulation of of variability is as:
tactile perception, known as the 'parchment-skin' illusion (Iousmáki and Hari 1998). multisensory perct
They showed that people's perception of the 'feel' of the palmar skin of their own represents a weiEi
hands could be changed simply by changing the sounds that they heard when they capture). However
rubbed them together. will be less noisy (
The participants in Guest et al.'s (zooz) study either heard the actual sound of their hence the person
hands being rubbed together (which was picked up by a microphone and played back (i.e. they will expt
over headphones), or else the sound was manipulated in order to reduce the overall with the multisen
unimodal perceptr
t:: Fig. 4.4). In the
. ,: 'udge the height
=
=
::-1 -1i participant's
-o
::i- --r. bars of either o
fL
-- ..::tics. Ernst and
., : ::inance, tactile
..- :t:tics \\¡as cliti- (a)
l- ;:llllláteS (See alSO
__
. ..- . i
estimation
,-
-o
o_
: - r'.::-r::iab1r' good
- : --;h. ar-rd prop-
: -:--:t::tal SettingS (b)
: --.:-r, .. :oo4, for a
: - :.:-.:d attention
.,.:.l:rd Knudsen Multisensory
sound intensity level (by either zo or +odB) or to amplifi' or attenuate just the high- (e.g. Freed r99o; Care
frequency components of the sounds (i.e. those above zkHz) by rzdB. The partici- Giordano and trfc-l,;
pants made separate ratings concerning how rough and how moist their hands felt. can determine the :¿
The participants rated their hands as feeling significantly dryer when listening to the sounds that thel n-ia
louder hand-rubbing sounds or when just the high-frequency sounds were boosted. which to discrimin¿t
Changing the auditory feedback also had a small but significant effect on participants' tory cues aPParentl-'
ratings of the roughness of their hands as well. Guest et al.'s results therefore highlight Gordon and \\/ebb, r'
the significant role that auditory cues can play in influencing people's evaluation of ulate these auditon'i
the'feel'of a surface, even for one that is as familiar to them as the palmar skin of their when particiPants -i
own hands. whether they rvoull
Kitagawa and Igarashi (zool) have even shown that simply hearing the sound were simultaneous'-,
of someone else's ear (in this case the ear on a dummy head) being gently stroked area for future resea:
with a brush can give rise to the illusion of being touched, provided that the sound in nature).
(which was recorded by a tiny microphone placed inside the dummy head's ear
canal) was played back from close to the participant's own ear (the participants
also saw a video recording of the ear of the dummy head being stroked). Many of
the participants in this experiment reported that they felt as if they themselves had
actually been touched (or rather tickled) on their own ear. These results therefore
demonstrate that touch-related sounds can, by themselves, induce illusory tactile
o
experiences (see Kitagawa and Spence zoo6). The auditory modulation of tactile expe-
rience is not, however, restricted to the perception of our own skin/bodies: for Guest
et al. (zooz) have shown that peoples'perception ofthe texture (roughness) ofsand-
papers varying in roughness (or grit) value can also be modified simplybyvarying the To date. far less res
auditory feedback that they heard when touching them. In this case, the amplification modal influence o:
ofthe high frequency sounds appeared to result in an increased perception ofsandpa- et al. zoo6). Lai¡d. r:
per roughness, while the attenuation of these sounds made the touched surfaces feel each ofz5o house.u-
smoother instead. to judge which par:
A growing body of more applied research has now demonstrated that auditory 'Feel them in ¡'oi,r ,..,
cues also play a veryimportant role in people's perception of many everyday products you would or¿¡¡i';r:i
(e.g. Wolkomir t996; Spence andZampini, zoo6). For example,Zampini et al. (zoo:) stockings were ¿ct:
have shown that people's perception of the pleasantness and powerfulness of an elec- fragrance, another
tric toothbrush is influenced by the particular sound that it makes when people brush fourth pair had a 's;
their teeth. Similarly, the perceived forcefulness and pleasantness of an aerosol spray ferred the narciss -s
can also be systematically influenced by changing the particular spectral profile of the sachet scent, and :l-
spraying sounds it makes when used (Spence andZampinízoo7). Results such as these, ofthe houseuives ,
when taken together with a large number of other studies (see Spence and Zampini (which onll'slr ol::
2c,c,6, for a review), demonstrate just how important auditory cues are to our percep- of the stockinss ii,
tion of many everyday products and surfaces. It is, however, important to note that the that were presun;i
particular auditory manipulations that are required to change our tactile perception see McKenzie r9:-::
appear to depend on the particular surface, or object, that is being judged or evaluated Dematté et ai. :
(see Spence and ZampinizooT, on this point). tigate whether oi r
Finally in this section it is worth noting that, by themselves, auditory cues often the sofness of fabr:
provide sufficient information for people to be able to judge the approximate (or rela- participants in th:.
tive) size of the sound-making objects, as well as certain of their material properties ent swatches of t¿i
THE MULTISENSORY PERCEPTION OF TOUCH 95
nrete iust the high- (e.g. Freed r99o; Carello et al. 1998; Klatzky et al. zooo; Kunkler-Peck and Turvey 2ooo;
,'r:dB. The partici- Giordano and McAdams 2006). For example, Freed (rggo) has shown that people
::si :hei¡ hands felt. can determine the relative hardness of percussive mallets on the basis of the impact
':en listening to the sounds that they make. Auditory cues can also provide sufficient information with
riJs rtere boosted. which to discriminate an object's length (Carello et al. i998). The one thing that audi-
1-ü on participants' tory cues apparently cannot be used to determine is the shape of a drum (Cipra t99z;
ii:¿reiore highlight Gordon and Webb ry96;Kac 1966). As yet, however' no one has attempted to manip-
;,rie's evaluation of ulate these auditory cues (e.g. in a manner equivalent to the üsual distortions induced
:¡-nar skin of their when participants look at an object through a distorting lens) in order to determine
whether they would influence the perceived size/material properties of objects that
::Erütq the sound were simultaneously being explored haptically. This clearly represents an interesting
r::rs qentlt'stroked area for future research (cf. Bee et al. zooo, for a possible example of this phenomenon
-lel rhat the sound in nature).
i-imm\- head's ear
,: ie participants
.::¡<edt. \,lany of
rr',' tl-i¿-t.1,res had
s; iesults therefore
:':;¿ illuson. tactile OrrecroRY coNTRTBUTToNS
,::,:,n of tactile expe-
'--¡,,fies: TO TACTILE PERCEPTION
* for Guest
:ilness) ofsand-
:::l', b., r.arving the To date, far less research has been directed at investigating the nature of any cross-
.- ::e amplification modal influence of olfaction on the sense of touch (see Laird l93z; Cox 1967; Dematté
:c;::jon ofsandpa- et al. zoo6). Laird, in a now classic study, described a house-to-house survey in which
r-:ie,i surtaces feel each of z5o housewives was presented with four pairs of silk stockings and then asked
to judge which pair had the best/highest quality. The housewives were instructed to:
'Feel them in your fingers, look through them, stretch them, look at the seqms. Do anything
you would ordinarily do to pick out the best for your own use.' (Laitd ry32, p. z4l. The
stockings were actually identical except for the fact that one pair had a slightly rancid
fragrance, another had a fruity fragrance, a third pair had a synthetic scent, while the
fourth pair had a 'sachet' scent. Fifty per cent of the housewives reported that they pre-
ferred the narcissus-scented hosiery (z4o/o preferred the fruity scent, r8%o preferred the
sachet scent, and the remainin g 8o/o preferred the slightly rancid scent). The majority
of the housewives attributed their preferences not to the smell of the scented hosiery
(which only six of them noticed), but to the texture, durability, sheen, weight, or weave
of the stockings instead (i.e. to their tactile and/or visual material qualities, qualities
that were presumably much easier to articulate verbally than are olfactory descriptors;
see McKenzie r9z3; Richardson and Zucco t989).
Dematté et al. (zoo6) recently followed up on Laird's (tyz) eatly research to inves-
tigate whether or not the presence of an odour would modulate people's perception of
iuliton'cues often the softness of fabric samples under rather more controlled laboratory conditions. The
:ir-r\]mate (or rela- participants in this study sat in front of a rotating wheel on which a number of differ-
::::erial properties ent swatches of fabric were hung (to allow for a naturalistic rubbing of the materials
96 CHARLES SPENCE
between the participant's thumb and forefinger). On each trial, the participants felt
one of the swatches (the participants were not allowed to see the fabric samples) and CoNsEeuE¡
had to rate its perceived softness. While the participants were evaluating the feel of AND
a particular fabric swatch, a lemon or an animal-like odour was occasionally deliv-
ered direct to the participant's nostrils by means of a custom-built olfactometer. On
each trial, the participants first had to classify any odour as either pleasant or unpleas- So far, the focus ot
ant, or else respond that no odour had been presented. For the majority of partici- ences on tactile/h¿P'
pants, the lemon odour was judged as pleasant while the animal odour was judged as However, a large btr
being unpleasant. Next, the participants indicated how soft the fabric felt. The partici- of how the residual :
pants rated the fabrics as feeling significantly softer when presented at the same time complete loss, or l:r
as the 'pleasant' lemon odour than when presented together with the 'unpleasant' see Galton 1883; Gri,
animal odour. Althoughthe er-iden.
In a subsequent experiment, Dematté et al. (zoo6) went on to show that olfactory ofresearchers har-e :
cues (in this case, the smell of lavender and/or animal) can also influence people's can result in an r¡r-r¡:
judgements of fabric softness when the fragrance was applied directly to the fabric . Brigitte Róder and r
swatches themselves (i.e. using a somewhat more ecologically-valid experimental find it much easier;.
design). It is, however, at present unclear whether the olfactory modulation of tactile stimuli are presenlc:
perception highlighted by previous research is restricted to the perception offabrics et al. zoo4). \\rhat's :
(i.e. to surfaces/materials that are commonly scented), or whether instead such cross- confused about thc i
modal influences can also influence people's perception of other surfaces (such as, for hands when ther. ¡:
example, sandpaper) that maynot typicallybe associated with anyparticular fragrance. make any such n-i:.t
Once again, this represents an important question for future research. uncrossed hands p,:,,
Many studies have demonstrated the significant effect that olfactory cues have on such dramatic intrr
people's perception of (and responses to) various different products (e.g. see B1'rne- Grant et ¿].. 299¡r . -i
Quinn 1988; Bone and Jantraniatgg2t Gulas and Bloch 1995; Neff zooo; Ebster and in tactile percepiiür
Kirk-Smith zoo5; Dematté et al. zooT). Pairing a product with an appropriate scent Heller r989r F1.r
(see
has been shown to enhance people's evaluation of the product, while the addition The congenital,r
of an inappropriate scent tends to lower people's product evaluations. For example, the sighted (Róder ,
a major European electronics goods manufacturer recently reported that they had orchestra condu¡lc
become aware of the fact that people's perception of their newly purchased (and often responses to peri::r,
very expensive) electronic consumer goods was sometimes being adversely affected anecdotai evidenct
by the unpleasant smells emanating from the packaging when it was initially opened more recent ianC ir
after purchase. In this case, the company was actually considering the introduction evidence for enh¿r¡
of a pleasant fragrance to the packaging materials themselves in order to avoid this is some evidence :"-
negative (and incongruent) initial impression of their products. It should be noted the sighted (see \1r
that people's judgements can be influenced by ambient odours even when they are Currentlr', tlle ¿.
presented at such low concentrations that they are not perceived consciously, or else the conclusion thal
when people simply fail to notice the odour any more because they have adapted to perform the sighteJ
it (see Spence zooz). Thus, the failure to discern the presence ofan odour does not ties. Howeve¡. i¡',';,
guarantee that one is not unwittingly being led by one's nosel In summary, the stud- p.3o9) claim (baseC
ies reviewed in this section highlight the important, if often unrecognized, role that deprived of one t i
olfactory cues can play in determining our reactions to a variety of different materials the remaining ones
and products. those for which the
THE MULTISENSORY PERCEPTION OF TOUCH 97
Although the evidence on this issue is not entirely clear, it is certainly true that a number
, j:1., !r that olfactory
of researchers have now shown that the loss of one sense (tlpically vision or hearing)
¡-, :::luence people's can result in an improved acuity in one or more of the residual senses. For example,
:-:-¡tL',- to the fabric Brigitte Róder and her colleagues in Hamburg have shown that the congenitally blind
, -'. ¡-rd esperimental finá it much easier to judge the temporal order in which two near-simultaneous tactile
(Róder
:-- -, : rlation of tactile stimuli are presented to their hands than do either the late blind or the sighted
:.:;eprion offabrics et al. zoo4). What's more, they also found that while the sighted and late blind often get
t: -1.:ead such cross- confused about the temporal order in which two tactile stimuli are presented to their
. -::¡¡es I such as, for hands when they are crossed over the midline, congenitally blind individuals rarely
:.-:cular tiagrance. make any such mistakes, even when matched on their baseline performance in the
¿:; i.. uncrossed hands posture. It should, however, be noted that not everyone has
observed
see Galton 1883;
such dramatic improvements in tactile perception in the blind
::---,-':-u' cUeS haVe On (e.g.
-:;:s e.g. see Byrne- Grant et al. zooo), and some researchers have even failed to find any such differences
;:, : -rrloi EbSter and in tactile perception between the blind and sighted across a range of different tasks
:.:- .rpropriate scent (seeHeller rgSg; Hollins rg8g)'
: -,,
1¡l¡ the addition The congenitally blind also appear to hear more acutely in the periphery than do
professional
the sighted (Róder et al. 1999; see also Lessard et al. 1998). Interestingly,
,"rLr:rS. For example,
:', ::;j that they had orchestra conductors have been shown to exhibit a similarly enhanced pattern of
:.:;r¿sed ( and often " responses to peripheral sounds (Mtinte et al. zoor). Furthermore, while there is some
^-: :l''-erselv afrected (e.g. Gault r9z3),
anecdotal evidence for the blind having an enhanced ability to smell
no
"'.:nitiall1. opened
=s more recent (and better-controlled) laboratory studies have provided absolutely
r'-¡ r]:e introduction in the blind, although there
evidence for enhanced olfactory (or gustatory) sensitivity
: ::ie¡ to avoid this is some evidence to suggest that the blind may find it easier to label/name odours
than
-: .lould be noted the sighted (see Murphy and cain rg86; smith et al. 1993; Rosenbluth et al. zooo).
;,,'¿n rr'hen they are Currently, the available empirical evidence would therefore appear to support
: ;.''nsciousiy, or else the conclusion that those deprived ofinput from one sense, such as vision' can
out-
.:r:-,- har e adapted to perform the sighted on certain other perceptual tasks in their residual sensory modali-
, .l odour does not ties. However, it would seem most parsimonious to agree with Grant et al''s
(zooo,
'people
i::-tntan', the stud-
s p. 3o9) claim (based on both their own data, and previous tactile research) that:
::.Lrgnized, role that áeprived of one or more senses do not develop truly supernormal sensitivities of
r f :llterent materials the remaining ones but rather learn to use them proficiently for purposes other
than
those for which they are normally used'. This conclusion would seem to fit well
with
98 CHARLES SPENCE
interesting in futu:< :
individuals have been
the results of those studies in which 'experienced' sighted also be less influencec
(e.g. Smith el al. t993; Róder et al' ry99;
shown to perform just as well as the blind |ousmáki and Harr ttc
extensive or specialized
Münte et al. zoor). Thus, it would appear that it is their One imPortani isi'r
(though note that training can result in
training, rather than cortical plasticity per se hasn't actuallr' re¿.1'' i
sensory/perceptual abili-
cortical reorganization), that i, ,"rponrible for the enhanced Howes zoo6), cona¿:r
in certain tasks'
ties shown byblind people (and others with sensoryloss) uni- and multi.senscr'
to show that the part of the brain that
Cognitive ,r.rrror.i.ntists have been able zooz). While it seen.
what they see (known as the
is normally used by sighted people in order to
process
(such as suPeradditr.'-
occipital cortex) upp.J^ to be used in congenitally
blind individuals in order to aug-
multisenson' integ:¡t
taking place in their somatosensory cortex
-.rrt the tactile information processing the visual cortex also appears
shown to aPPh- ac:':'
(e.g. see Sadato et al. tgg6;coh..r et al. t99). similarly, up to humansi e.s. s(
tasks as well (Kujala et al' 1995)'
ro be recruited when Uíind people perform auditory senses may be erae ;:
sighted individuals begin to
In fact, the latest research hu, ,Lo*n that even normally ago, Wober (196ó '-
after a few days without vision
show the first signs of such cortical reorganization .patterns of imPon::
(Merabetetal.zoo4).BrainscansoftheparticipantsinMerabetetal.'sstudyshowed different cultures. Il
that their visual cortices had started to be recruited
to help them perform purely
might differ frc-¡m rtr
spending as little as an hour and a
tactile tasks after five days of blindfolding. In fact, or propriocePtioi -.
half in the dark has ,ro* t.".r shown to result in
significant (if short-lasting) improve-
research is urgen:''' :
ments in both tactile and auditory perception in
the sighted (Facchini and Aglioti
cross-culturai dlre:¿
zoo3; Lewald 2oo7; see also Boroojerdi et al' zooo)'
Givenourgrowingunderstandingoftheneuralmechanismsbehind,andthe
performance that have now been
behavioural .or.r"qrrá."s of, the variáus changes in 1Q-
following the loss (or congenital
documented in each of the senses that remain intact c)
zoo4), researchers are now begin-
absence) of a particular sense (Roder and RÓsler 0)
(-)
(rather than just unisensory) per- l.o -
ning to investigate whether the rules of multisensory CD
t:-,-l.rals have been interesting in future research to investigate whether the congenitally blind would
:-.: Roder el aI. ry99; also be less influenced by sound in the parchment skin illusion described earlier (see
::l:li i oI specialized Iousmáki and Hari 1998; Guest et al. zooz).
::,1::inq can result in One important issue that hasn't been dealt with here, and which, for that matter,
i :-,' perceptual abili- hasn't actually really been studied much by experimental psychologists to date (see
*:.:sks. Howes 2¡,¡16),concerns the role that cultural factors may play in the various aspects of
=:. -: :he brain that uni- and multisensoryperception (e.g. see wober 1966,r99r; Howes l99r; MacDonald
. .;= krorvn as the zooz). While it seems likely that the same general rules of multisensory integration
: --.-, ::t ordef to aug- (such as superadditivity and the importance of spatial and temporal coincidence to
.:1:-,-':enSOry COfteX multisensory integration) are likely to apply the world over (as they have already been
i, -- :'c\ also appears shown to apply across the animal kingdom, from unicellular organisms all the way
-- !,.-i':,a et al. 1995). up to humans; e.g. see Calvert et al. zoo4b), the particular weighting of the various
-1-'' i¡uals begin to senses may be expected to exhibit significant cultural difference. In fact, many years
:.' ' .,'ithout vision ago, Wober (1966) introduced the notion of the'sensotype'to refer to the differing
.: -..s:udvshowed patterns of importance of the various senses that he speculated might be found in
r.:r :::iornt purely different cultures. He argued that the predominance of vision in Western cultures
.:-, . .r ¡n hour and a might differ from the putatively greater importance associated with audition (and/
, ::--:.lrng imprOve- or proprioception) in certain other (e.g. African) cultures. Further interdisciplinary
'
::: --:--rr and Aglioti research is urgently needed on this topic in order to explore more adequately any such
cross-cultural differences in human perception. Note that Wober's notion of different
-r, :::ind, and the
: :::,:Leve nOWbeen
-.. -:.. Or Congenital
E
,-.::: lia nOf,u, begin- o
*,: :r:isensory) per- o
Io 1.6
-: ,::l Róder (zoo4) o-
o
o
...,:.: perception in 55 1.4
--4- Blindfolded sighted
participants
:l I l:lr¡It hOrv many
--'tr- - - CongenitallY blind
, :,-,;ches delivered ?
c)
t.z padicipants
; .-r -, lresented With" -o
E
-: ,:l:: r.\'ele pfeSented f,
c1
c(u
--*'-: -'- t¿ctile stimuli C)
: --:-:r irhen multiple 0.8
'..:r,r,us. the partici- 23
r::- :he participants Number of sounds
..:::nd concentrate
::, -:ect ofsound on Fig. a.5 Graph highlighting the results of Hótting and Róder's (2004) recent study
..:
-,
¡h¿n in the con- comparing the multisensory ¡nfluence of sound on touch in the congenitally blind
:.: . :esults therefore and sighted. The results show that the congenitally blind were overall better able
.-lrrS of the residual to perceive the number of touches presented to their fingertip than the sighted.
More importantly, however, the results also show that the sighted were more
-.¡-e tn terms of the
:-:
1t-+; \Velch and influenced by what they heard under conditions of sensory conflict (where the
number of felt and heard stimuli differed) than are the congenitally blind.
=::,'::i-ied the sighted
. . '^-
j be particularly
(Redrawn from Hóttinq and Róder 2004, Figure 1.)
100 CHARLES SPENCE
sensoq?es can also be used to describe the consequences of sensory loss (such as occurrence outside o
congenital blindness) that were documented earlier. Finally, even within Western Connor zooo). Inste
cultures, it is important to note that there appear to be some marked individual dif- event, or environrne
ferences in the extent to which one sense dominates over another (e.g. Binns 1937; perceptual experient
McDonnell and Duffett r97z; Samuel r98r; Heller r99z). The growing und'
the rules governins :
another (a phenomenon known as 'visual capture', when visual inputs completely Bone PF and lantr:r.
override the inputs from one or more of the other senses) is probably a fairly rare z\9-q6.
THE MULTISENSORY PERCEPTION OF TOUCH 101
i::rrrrn- loss (such as occufrence outside of the psychologist's laboratory (though see Kinney and LutiaryTo;
r',':n rrithin Western Connor zooo). Instead, each of the senses that are stimulated by a particular object,
:r.¿d individual dif- event, or environment will likely contribute meaningfully to our overall multisensory
,:rii e.q. Binns 1937; perceptual experiences.
The growing understanding of the multisensory nature of human perception, and
the rules governing it, that has emerged from recent cognitive neuroscience research
(e.g. Calvert et al. zoo4a) is now beginning to offer a principled means of enhancing/
optimizing the multisensory nature of our tactile experience of the objects we inter-
act with and the environments in which those objects are placed (e.g. Spence zooz;
Spence and Zampini zoo6; Schifferstein and Spence 2oo8). What's more' the multi-
sensory design of objects and environments that more effectively stimulate the senses
seems to constitute a particularly worthwhile endeavour given that multisensory
:¡: ll..re to the sense experiences are qpically far richer, more pleasurable, and more memorable than
: t '¿., Lrf e\perimen- the individual unisensory experiences from which they are constituted (e.g' Knasko
:-r:i1trn of both the t993; Stein and Meredith rgg¡; Aggleton and Waskett 1999; Spence 2oo2; Howes 2oo5;
;:, 'a.tionality, can Schifferstein and Spence, 20o8).
-:i Lr¡ssano r99r), as
:: ::.,,-smell (Spence
,: ::rlduct, or even
.:; l:preciation of RnrnnENcBs
., :" erperience (see
* >:-nce zooS). It is Aggleton JP and Waskett L (rqgq). The ability of odours to serve as state-dependent cues for
':,,¡s to tactile per- real-world memories: Can Viking smells aid the recall of Viking experiences? British Journal
of Psychology, go, r-7.
":. :rd. rrhether they Aiais D and Burr D (zoo$.The ventriloquist effect results from near-optimal bimodai integra-
--. r lntormation is
Iion. Current Biologlr, t4, 257-62.
Andersen TS, Tiippana K, and Sams M (zoo5). Maximum likelihood integration of rapid
.: :a:lSaS tO OUf OVef- flashes and beeps. Neuroscience Letters,38o, r55-6o.
:, .¡; rierv that each Arnold K (zoo3). Skulls, mummies and unicorns' horns: Medicinal chemistry in early English
::,: -:. ¡o our overall museums. In RGW Anderson, ML Caygill, AG MacGegor, and L Syson' eds. Enlightening
the British: Knowledge, Discowry and the Museum in the Eighteenth Century, pp. 74-8o. The
: -:r¡¡ gir-en to each-
British Museum Press, London.
.:-;:tual sYstem that
Bailly Dunne C and Sears M (rqq8). Interior Designingfor all Five Sens¿s. St. Martin's Press,
-- : r--1. for a review),
New York.
.:. :,ti¡rates that are Battaglia PW, Jacobs RA, and Aslin RN (zoo3). Bayesian integration of visual and auditory
:
" c,.¡rinated by the signals for spatial localiz ation. ]ournal of the Optical Society of America A, zo, r39t-7 .
:-:l::red so forcefully Becker-Carus C (rSZ¡). Verándert die Wahrnehmungsschwelle die dominanz zwichen sehen
:-::lres of an object und tasten? lDoes the perception threshold change the dominance of vision and touch?]
i,- ',,.'ei'er. it is just as Z eit s chrift für Exp erim ent ell e u n d An gew an dt e P sy ch oI o gi e, zo, 3 47 -6 5'
. Bee MA, Perrill SA, and Owen PC (zooo). Male green frogs lower the pitch of acoustic
-,:her senses) under
signals in defense of territories: A possible dishonest signal of síze?. Behavioral Ecologlr, rr,
:--- 'rate, least noisy
169.-77.
- ':inate over vision Berkeley G (vogltgsl). Anew theory of úsion. |.M. Dent & Sons' London.
.: ind Spence zooT). Binns H. (1937). Visual and tactual'judgement' as illustrated in a practical experiment. Brirlsh
:r:; oi one sense by Journal of Psychology,2T, 4o4-to.
.:, -::puts completely Bone PF and Jantrania S (rggz). Olfaction as a cue for product quality. MarketingLetters,3,
:: - -.:blr- a fairly rare 289-96.
1c)2 CHARLES SPENCE
Boroojerdi B, Bushara KO, Corwell B, et al. (zooo). Enhanced excitability of the human visual Gault RH (r92.-¡ . .{r:
cortex induced by short-term light deprivation. Cerebral Cortex,to' 529-34. Psycholog', 17. -rer-
Brewster D (r8¡z). Natural Magic. fohn Murray, London. Gephstein S, Burge 1.
Byrne-Quinn | (r98S). Perfume, people, perceptions and products. In S Van Toller and depends on sp;t:''
G Todd, eds. Perfumery. The Psychology and Biology of Fragrance, pp. zo5-t6. Chapman and Gibson JJ (1931 ). -\c¡r
Hall, New York. of Experiment,ti P:'
Calvert GA, Spence C, and Stein BE, eds. (zoo4a). The Handbook of Multisensory Processes. Giordano BL anC \1¡
MIT Press, Cambridge, MA. size variation in ':
Calvert G, Spence C, and Stein BE (zoo4b). Introduction. In GA Calvert, C Spence, and BE America,1r9, 1r-1-
Stein, eds. The Handbook of Multisensory Processingpp. xi-xvii. MIT Press, Cambridge, MA. Gordon C and \\-etr-.
Carello C, Anderson KL, and Kunkler-Peck AJ (rgg8). Perception ofobject length by sound. Grant AC, Thiagar:.ia
Psychological Science, g' 211-14. A psychophvsica- .
drum. Science, 225' 1642-3. (z Psychophtsí;:.6.
Cipra B (1992). You can't hear the shape of a
Classen C (zooS). Touch in the museum. In C Classen, ed. The Book of Touch, pp. 275-86. Griesbach H trE99 .
Dematté ML, Sanabria D, Sugarman R, and Spence C (zoo6). Cross-modal interactions Hay IC, Pick HL Tr
uct evaluation . P sychology and Marketing, 22, 739-49. Perception, z.r. 6<,
Epstein W (rgZr). David Brewster's observations on perception when touch and vision Heineken E and Scr,
conflict: An historical note. Perception 6 Psychophysics, ro, 97. natural and rir.r:
Ernst MO and Banks MS (zooz). Humans integrate visuai and haptic information in a statisti- HollinsM(1989. --'
cally optimal fashion. Nature,4r5, 429-33. Hótting K and R¡¿.
Ernst MO and Butthoff HH (zoo4). Merging the senses into a robust percept. Trends in sighted indirr,i::..
C o gnitit e S ciences, 8, t 6z-9. Howes D, ed. tr99r .
Facchini S and Aglioti SM (zoo:). Short term light deprivation increases tactile spatial acuity the Senses. Unir e:
in humans. Neurology, 60, 1998-9. Howes D (zoo5 . --
Fishkin SM, Pishkin V, and Stahl ML (rqzS). Factors involved in visual capture. Perceptual and Empire of the s;r.s
Motor Skills,
4o, 427-34. HowesD(zoo6.C:
Freed DJ (r99o). Auditory correlates ofperceived mallet hardness for a set ofrecorded percus- Jousmáki V and H¡-
sive sound events. Journal of the Acoustical Society of America,87, 3tr-zz. 8,869-72.
Freides D Oslj.Human information processing and sensory modality: Cross-modal func- Kac M (1966). Can :,
tions, information complexity, memory, and deficit. Pq'chological Bulletin,St,284-3ro. Kinney JAS and L;
Galton F (r8s:). Inquiries Into Human Faculty And Its Developmenr. MacMillan, London. Perception i- P:',:
THE MULTISENSORY PERCEPTION OF TOUCH 103
- r, .¡f the human visuai Gault RH (1923). An unusual case of olfactory and tactile sensitivity. Journal of Abnormal
.:::--1+. P sy c h ol ogy, t7, 395- 4ol.
Gephstein S, Burge J, Ernst MO, and Banks MS (zoo5). The combination of vision and touch
. --. : \-an Toller and depends on spatial proximity. lournal of Vision,5,1ot3-23.
:: ;-16. Chapman and Gibson f) (1933). Adaptation, after-effect and contrast in the perception of curved lines. Journal
of Experimental Psychology, 16, r-3r.
t,', ! :;.:i
:t rt s o ry Pro cesses. Giordano BL and McAdams S (zoo6). Material identification of real impact sounds: Effect of
size variation in steel, glass, wood, and plexiglass plates. Journal of the Acoustical Society of
r.::. f Spence, and BE America, rr9, u7r-8t.
: -.'::. C¿mbridge, MA. Gordon C and Webb D (rqq6). You can't hear the shape of a drtm. American Scientist,84, 46.
: ;:t i::rsth by sound. Grant AC, Thiagarajah MC, and Sathian K (zooo). Tactile perception in blind braille readers:
A psychophysical study ofacuity and hlperacuity using gratings and dot patterns. Perception
': ú P sychophysics, 62, 3or-r2.
,. .-.- l,-i,;ii. pp. 275-g6. Griesbach H
(l8qq). Vergleichende Untersuchungen über die Sinnesschárfe Blinder und
Sehender fComparative studies of perceptuai acuiry in the blind and the sighted]. Pflügers
: -. ,- : ¡¡,:,ss-modai plas- Archiv,74, 577-q8.
Guest S and Spence C (zoo3a). What roie does multisensory integration play in the visuotactiie
.:- -.itla:1 On perCeptual perception oftexture? InternationalJournal ofPsychophysiology,5o, 63-8o.
: ".r.:,;:..,i ¡_i! tj, 353_7. Guest S and Spence C (zoo¡b). Tactile dominance in speeded discrimination of pilled fabric
,: - ::'--ni,.'ersity Press, samples. Experimental Brain Research, t5o, 2ot-7.
Guest S and Spence C (zooZ). Tactile dominance in fabric texture perception. Manuscript
--:: -: piLrcess. In Rlsk submitted for publication.
-- ':::uate School of Guest S, Catmur C, Lloyd D, and Spence C (zooz). Audiotactile interactions in roughness
perception. Experimental Brain Research, 146, 16]-7t.
".,.. . t \lainstream
.-.'. r':a-. Gulas CS and Bloch PH (rqqi). Right under our noses: Ambient scent and consumer responses.
Journal of Business and Psychology, lo, 87-98.
-.-::t :,c;l interactions Hay IC, Pick HL Jr, and Ikeda K (1965). Visual capture produced by prism spectacles.
nomic S cience, 2, 215-16.
P sy cho
- .:. ' :Jgments of facial Helbig HB and Ernst MO (zoo7). Optimal integration of shape information from vision and
touch. Exp erim ent al B r ain Re s e ar ch, ry 9, 59 5-6 o 6.
: =. . :¡rction of accu- HellerMA (1989). Texture perception in sighted andblind observers. PercePtion 6Psychophysics,
45,49-54.
,. .::;rOSienOl On prOd-, Heller MA (1992). Haptic dominance in form perception: Vision versus proprioception.
P erce pt ion, 2r, 655-6o.
::=:: :OuCh and ViSiOn Heineken E and Schulte FP (zoo7). Seeing size and feeling weight: The size-weight illusion in
natural and virtual reality. Human Factors, 49,86-44'
.-,: :::ration in a statisti- Hollins M (rgSq). UnderstandingBlindness. Erlbaum, Hillsdale, NI.
Hótting K and Róder B (zoo4). Hearing cheats touch, but less in congenitally blind than in
:*:: :ercept. Trends in sighted individuals. Psychological Science, l5, 60-4.
Howes D, ed. (r99r). The Varieties of Sensory Experience: A Sourcebook in the Anthropologt of
.i:j ::.file spatial acuity the Senses. University of Toronto Press, Toronto.
Howes D (zoo5). Hlperesthesia, or, the sensual logic of late capitalism. Irl D Howes, ed.
::::ire. Perceptual and Empire of the senses: The Sensual Cubure Reader, pp. 281-303. Berg, Oxford-
Howes D (zoo6). Cross-talk between the senses. Tfte Senses and Society,l, 381-90.
t tt to totd Percus - Jousrnáki V and Hari R (rqgS). Parchment-skin illusion: sound-biased totch. Current Biology,
,_*.t.t 8,869-72.
. :.-: Cross-modal func- Kac M (1966). Can one hear the shape of a drum?. Mathematics Monthly, rr3,21-3.
:'....: :1,., 8r, 284-3ro. Kinney JAS and Luria SM (r97o). Conflicting visual and tactual-kinesthetic stimulation.
¿:l,,lii1an, London. P erception dt P sychophysics, 8, 189-92.
704 CHARLES SPENCE
Kitagawa N and Igarashi Y (zool). Tickle sensation induced by hearing a sound. lapanese Power RP (r98o). Th: :
Journal of Psychonomic Science,24, t2t-2. 457-66.
Kitagawa N and Spence C (zoo6). Audiotactile multisensory interactions in information pro- Power RP (r98r). The ;;
cessing. Japanese Psychological Research, 48, 158-73. Lo,29-33.
Klatzky RL, Pai DK, and Krotkov EP (zooo). Perception of material from contact sounds. Power RP and Grah¿n: -
:,:s e sound. lapanese Power RP (r98o). The dominance of touch by vision: sometimes incomplete. Perception, 9,
457-66.
:' i: intbrmation pro- Power RP (lg8r). The dominance of touch by vision: occurs with familiar objects. Perception,
lo, 29-33.
::L¡i]l contact SOUndS. Power RP and Graham L Ggl 6). Dominance of touch by vision: generalization of the hlpoth-
esis to a tactually experienced population. Perception, 5,76t-6.
:-¿:li: and incongruent Richardson JTE and Zucco GM (1989). Cognition and olfaction: a review. Psychological
Bulletin, rc5,352-60.
'::rL:n in blind humans Rock I and Harris CS (1962). Vision and touch. Scientifc American, zt6 (rZ May), 96-704.
Rock I and Victor J (rg64). Vision and touch: an experimentally created conflict between the
\: :1-i. tital P sychology: two senses. Science,143, j94-596.
Róder B and Rósler F (zoo4). Compensatory plasticity as a consequence of sensory loss. In GA
i i¿i:son- impressions: Calvert, C Spence, and BE Stein, eds. The Handbook of Multisensory Processes, Pp. 719-47.
-.'-.r. 16.:-1r-6. MIT Press, Cambridge, MA.
t :r:.:sanson' ofganis- Róder B, Rósler F, and Spence C (zoo4). Early vision impairs tactile percePtion in the blind.
;:-,:l"'sics. Jottrnal of Current Biologlt, r.4, 12t-1,24.
-'a Róder B, Teder-saiájar-vi w, Sterr A, Rósler F, Hillyard sA, and Nevilie HI (rsss). Improved
::- :. in GA Calvert, auditory spatial tuning in blind humans. Nature,4oo,t6z-6.
'i-.ji-..
FF. 10/-22. MIT Rosenbluth R, Grossman ES, and Kaitz M (zooo). Performance of early-blind and sighted
children on olfactory tasks. Perception,29' tor-to.
t; i:siLrn and touch: Sadato N, Pascual-Leone A, Grafman J, et al. (1996). Activation of the primary visual cortex by
.:,.."::!i:i1l Psychology: Braille reading in blind subjects. Nature,38o, 526-8.
Samuel JMF (rq8r). Individual differences in the interaction of vision and proprioception. In
'
:-- :: slbiects localize RD Walk and HL Pick, lr, eds. Intersensory Perception and Sensory Integratíon, pp. 375-98.
Plenum, New York.
--. ::: -rSht deprivation. Schifferstein HNJ and Spence C (zoo8). Multisensory product exPerience. In P Hekkert and
HNJ Schifferstein, eds. Product Experience,pp.133-161, Amsterdam: Elsevier'
-..-.:¡. *{rir¿li, Sight and Schmidt RF, ed. (D86). Fundamentals of Sensory Physiology. Sprínger-Verlag, Berlin.
Smith RS, Doty RL, Burlingame GK, and McKeown DA (rqq¡). Smell and taste function in the
rr -:-rutrrS in sensory- visually impaired. Perception (t' Psychophysics' 54, 649-55.
: :l: -r",.,ir¿rT-s, pp. 113-23. Spence C (zooz). The ICI Report on the Secret of the Senses. The Communication Group'
London.
-- :. ,:.ont-lict between Spence C (zoo). Audiovisual multisensory integration. Acoustical Science and Technology, 28,
6r-7o.
I.:::::tann, LOndOn. Spence C and Zampini M (zoo6). Auditory contributions to multisensory product perception.
,¡:*'l-Leone A (zoo4). Acta Acustica united with Acustica,92, Loog-25.
'::a>. it,Jnnl of Neuro- Spence C and Zampini M (zooz). Affective design: modulating the pleasantness and force-
fulness of aerosol sprays by manipulating aerosol spraying sounds. CoDesign,3(Suppl r)'
- :-:--.: ¡orm perception 109-23.
Stein BE and Meredith MA (rgq¡). The Merging of the Senses. MIT Press, Cambridge, MA.
: *:r:on' spatiai tuning Susini P, McAdams S, Winsberg S, Perry I, Vieillard S, and Rodet X (zoo4). Characterizing the
sound quality of air-conditioning noise. Applied Acoustics, 65, 7q-9o'
'::ner.
Pltysiology and Warren DH and Rossano MJ (t99r). Intermodality relations: Vision and touch. In MA
Heller and w Schiff, eds. The Psychology of Touch, pp. 119-37. Lawrence Erlbaum,
":-.- -..¡. February 27,22. Hillsdale, NI.
: Lr:no.eption. British Welch RB Q97z). The effect of experienced limb identity upon adaptation to simulated dis-
placement of the visuai fr'eId. P er cep ti o n d' P sy cho phy s í c s, l.2, 453-6.
: :narion-Processing Welch RB, DuttonHurt LD, and Warren DH (rg86). Contributions of audition and vision to
1- temporal rate perception. Perception (t Psychophysics'39' 294-3oo'
106 CHARLES SPENCE
CHAPTER 5
::: :.i S;t5e¡y P.aperience:
:::; oi TorontO Press,
ROSALYN DRISCOLL
A vouNc man leans forward and lightly touches his iips to a sculpture. This act is not
a transgression but an exploration: the young man is blind, the sculpture is meant to
be touched, and lips happen to be exquisitely sensitive.
Touching of any sort is taboo for most artworks in most museums. As a visual
artist, I accepted this prohibition for my finished work. That is, until I began to think
about ways to create access to my art for people with visual impairments. At the time
I was making handmade paper, which is an immensely tactile process. Making books
with the paper, I noticed that books have to be touched to be known. could touch,
I wondered, be a way for people with visual limitations to know art? There was no
consensus on this question in r99o. I was forging into unknown territory. To make
sculptures that would accommodate the touch of üewers required a radical reorien-
tation, a retooling of my working process, the use of new materials, and research into
tactile perception. I had to learn to think and make art through a new perceptual,
expressive language.
Intheprocessldiscoveredthattouchisawayfor allofustoknowart.Bymakingart
that people of all abilities can touch consciously, I am developing a little-used aspect
of the sense of touch. Most of the touching we do in our daily lives is manipulative,
functional, and largely unconscious. Aesthetic touch is conscious, inquiring touch
that explores forms, materials, and spaces for their qualities, their effects, and their
meanings. It's like aesthetic looking, which we readily understand as a special appli-
cation of sight-primarily for the arts, but also occasionally for ordinary objects and
situations. Like aesthetic sight, aesthetic touch involves a departure from habitual
recognition and functional use; attention to formal elements such as shape, space,
and pattern; transformation of the object or situation into alternative structures,
concepts, or meanings; and openness to the emotional implications of what we per-
ceive. An enormous body of writing and thinking exists about aesthetic sight and how
1c)8 ROSALYN DRISCOLL
it works. Very little is written or known about aesthetic touch. Although it resembles leave me relativel-,
aesthetic sight in the ways just described, aesthetic touch differs in other ways, which the reality of thing
enrich the aesthetic experience. wood, sand, and s{
My understanding of aesthetic touch has come through several avenues: through than sand, and u-a:
the observations of people who had trained as visual artists before becoming blind, Furthermore, a
and textures I co¡
who were willing to share with me their experiences as they touched sculpture of all
kinds; through the process of learning to make and exhibit my own tactile sculptures; about the location
and through documenting the accounts of people who touch my work, especially motions-fast, slo
the world and our
those who were blindfolded, a condition which reveals the possibilities and limitations
of touch without the interference of sight. I have extended my research into reading
Ourbodyimaer
about tactile perception and exploring the phenomenology of touch with neurosci- but are constantlr-
entists, psychologists, engineers, art historians, curatols, artists, anthropologists, My sense of mv ortr
and philosophers. ous streams of into:
Although touch is increasingly being allowed and advocated in art museums for about mybodv. . .'
people who are blind or visually impaired, I would like to see touch move out of the and fuller sense of s,
ghetto of disability into the realm of possibility for everyone. Touch can be a satisfiring,
appropriate, even profound way for people of all abilities to know sculpture. This sense of s,
Our colloquial use of the word touch usually means contact with something by ence occurs betrn-e
the skin, especially the hand. My use of the word is more expansive' encompassing one another. \\-r
kinesthesia, proprioception, balance, temperature, páin, and pleasure-indeed the something balan;
whole body. The haptic sense (which includes movement and proprioception) is the stability and caln
oldest, most comprehensive and complex of the senses, with receptors embedded Perceptual ps'''.
throughout the body from the skin down into the joints and muscles. Aesthetic touch operate anr.rr'here
draws from this deep well of sensory input to inform us about what we are touching In perceptual :er
as wellabout the self who does the touching.
as
percepL ob,ieoiri
Touch is reciprocal, a two-way exchange. Every time I touch, I am also touched. The processes t:
I affect the world and the world affects me, unlike sight and hearing, which seem to able sensations. J
perception oiit. I
sensations \\'e aan
When touching.
impression; rre c;
anr,vhere on thr "
Expanding or I
vide us with the '
perature, ma55. il
with the 'subjeü;,
associations. an'j
context of art is s
brings us back t"'
and selves eralon
Fig. 5.1 Rosalyn Driscoll, Limen,2OO8. Wood, copper, steel, resin. Art is produ;e
(Photograph : David Stansbury.) tions, and e\perii
AESTHETIC TOUCH 109
touch). The use of touch to explore art amplifies and intensifies our awareness of
this complex interplay of self and object. The shared capacity of art and touch to
fuse inner and outer conditions makes touch a deeply effective and affecting way
to know art.
People who touch my sculptures often discover this. Knowing a work of art through
enriches the sensory range and therefore the scope of meaning. mind. It provides ;
Those who explore a sculpture first by touching (by wearing a blindfold) and later body in explorini
by looking, often generate two different versions of the artwork: a haptic one and a themselves, in aiir
visual one. Awoman touching a sculpture while blindfolded thought she was in a forest carry aesthetic ani
with trees and moss; when she took off her blindfold she was astonished to see flat, ing from mo\.em.r:
rectilinear, bronze surfaces. When people experience multiple but distinctly different centre and penpl:e
versions of a sculpture with touch and then with sight, they realize that the sensory and square. This ls
mode determines the nature of what they perceive. This is a profound insight. We dis- tap, caress. These n
cover that what we see is not the whole story. perceptual meanl¡
Knowing artworks only through a visual lens can distort or limit their meanings. Everymemon'.:
The art critic Michael Brenson says of Giacometti's figures that to the eye they seem ing an artwork rn¿
ravaged, but: ment, caressing a ,
When you engage Giacometti's figures with your hands, it is apparent that they have every for one's memor!'
bit as much to do with tenderness as they do with violence. To hold their shoulders, backs associations and l¡:
and heads is to ease the violence committed by the eye.
(Brenson r995) ' See Francesca B:.:.
AESTHETIC TOUCH 111
Brenson r995) ' See Francesca Bacci's essay'sculpture and Touch' in this volume (Chapter 7).
712 ROSAI,YN DRISCOLI-
experienced before
rement'. Touchine
more attracted to ti
These spaces mal i
I found them inter,
categories ofpositi
egories of figure ar
they become more
figure to my inqui:
that figure. Seeing
hierarchies of salie:
non-hierarchical. a
ended sense ofan ¿
experienced before because it was not made finite and given limits by visual measu-
rement'. Touching one of David Smith's Cubi-tllted, stacked metal cubes-I was
more attracted to the triangular spaces between the cubes than to the cubes themselves.
These spaces may have been negative to my eye but they were positive to my hand;
I found them interesting in their shape and size, fitting around my hand. The visual
categories of positive and negative prove to be less useful in haptics. The visual cat-
egories of figure and ground also assume different meanings for the sense of touch;
they become more interchangeable, mobile, transient. Whatever I am touching is
figure to my inquiring hand; everything I have already touched becomes ground to
that figure. Seeing seeks to identifi' and categorize an image and its parts, to create
hierarchies of salience, form, space, and colour. Haptic knowing is more horizontal,
non-hierarchical, and democratic. It produces a more fluid, changing' evolving, open-
ended sense of an artwork, supporting the transient nature of reality and the multiva-
lent nature ofart.
This fluidity includes our neurological relationship with a sculpture. We now
know that the brain is not only constantly mapping our bodies' but is also mapping
the space around us, including things near us oI that we are touching. So when I touch
a sculpture, as far as my nervous system is concerned it becomes part of my map of
i, my body. I enter its space and it enters mine, so that the boundary between inside
and outside quite literally dissolves (Blakeslee and Blakesle e zooT) '
At times the skin's function as a boundary reasserts itself. Because touch is so
critical to survival, we are quick to detect risk in touching even a sculpture. Touch can
evoke a sense ofvulnerability and danger as well as intimacy and affection. Its closeness
to visceral feelings and to emotions, which are bodily sensations too, makes touch-
' :n s artwork by ing a more likely trigger for emotions. As one person said' 'This experience is very
S;ee1, copper, fundamental, basic, primitive, more direct than intellectual experiences and therefore
stronger'. Touching often raises memories of childhood, or evokes a sense of play,
'' :-...'.3" . El en Augarten.)
because as children we openly explore the world through touch and movement' As we
grow up) conscious, exploratory touch becomes socially prescribed and often limited
to sex, making it easyto confuse sensuous and sexual. These unconscious connections
.::;e er-ery gesture,
to danger, r-ulnerability, emotion, childhood, play, and sexuality make haptic experi-
.: ,-.: the body. In the ence of art especially powerful. Add the taboo against touching art, and you have the
.. ::r: hand, can draw potential for a ttigftty charged encounter. Deep, unconscious currents run through
:rr:ll-as if my hand
even the most cursorY of touches.
than sight; scale
-:¿:a-1 Haptic experience of art is powerful at another, even less conscious level. Sensory
:J a place inside the
and motor functions normally operate below the level of awareness' Most of our
,:-'' :nd I rvas standing
daily haptic experience remains unconscious. When we bring touch into conscious
awareness and even into the realm of aesthetics and meaning, as we do in touching
i'' The moving hand
sculpture, we join lower and higher functions of the brain, generating a more integra-
rI-t-rnS €\'er if there is
tive, holistic experience.
,: - space I have never The unconscious aspect of haptics is operating when we encounter the vast number
of artworks that are not touchable. The connection between inner and outer realms
that is true of the haptic experience of art also holds in the visual encounter with
::- - Jels of space
::r¿. i,'hich is a cast of
art. Haptic perception underlies and informs visual perception, both developmentally
t74 ROSALYN DRISCOLL
and experientially, throughout our lives. Even when we are onlylooking, tactile percep-
tion is subliminally implicated in the appreciation of art. When we look at a sculpture,
we draw on our haptic memories of textures, forms, spaces, motion, and materials to
inform our understanding of what we see. We engage haptically when seeing by pro-
jecting ourselves into an artwork, by identi$ring with elements in it, and by attributing
movement to what is static. We imagine ourselves interacting with the artwork or with
the forces represented in it. We imagine ourselves being lhe forces or the figures in
an artwork. We inhabit landscapes, rooms, fields of colour-whether in a painting, a
sculpture, a photograph, or a building. At a subtle muscular, proprioceptive level, we
RI
feel in our bodies the attitudes and tensions within an artwork, subliminally mimick-
ing those forces. We feel in ourselves-and therefore know-the thrust of a figure's SE
gesture, the arc ofa brushstroke, the expansiveness ofa space.
Because we are multisensory beings, our senses inform and integrate with each
other. Haptic experience enriches and enhances visual perception. We can learn to
see more deeply by engaging in haptic exploration. Touching sculpture embodies and
amplifies our visual tendencies to identi$r with, project into, and energize sculpture.
This merging and blending of self and sculpture, whether through actual or imagined
touch, gives us the experience ofentering another reality, a shared space that enlarges
our sense of ourselves and the world. This complex response ofbody and mind is noth-
ing less than empathy. And empathy is both the seed and the fruit of art.
Tns meaning of an
RnrnnBNcns
critical, iconoloeic¿
Brenson M (ISSS).Memory of the Hand. Sculpture,4$). Here we try a difere
Blakeslee S and Blakeslee M (zoo7). The Body Has a Mind of its Own. Random House, nature of üsibili¡ ¿
NewYork. oped in Western ¡:r
Juhan D (t987). Job's Body: A Handbookfor Bodywork. Station Hill Press, Barrytown, NY. sensed. It is translat¡
Katz D, ed. (rq8q). The Workl of Touch (LE Krueger trans.). Lawrence Erlbaum Associates, resentations. In tac.
Hillsdale, NJ. nature of all things.
LakoffG and lohnson M (rq8o). Metaphors We Live By. University of Chicago Press, Chicago, IL.
intuitively.
The knowledee tl
paradigm, but rathe
idea of form, r*ric:
to an imitation of ¡e
bolic and expressi,,'e
sensory realig'-at i
Art is space, it is t
seeing and touchinE
doxically the senst--;
Diderot to Lessins.
nomenological tradi
act. Not only do the-u
.:!;-¡9. tactile percep-
'; ,,¡ ¡k at a sculpture,
Tna meaning of art can be revealed in rnany ways. One can carry out a historical'
critical, iconological, or iconographic analysis. One can analyse forms and styles.
Here we try a different way, in which the meaning of art is defined by reflecting on the
" : i.::dom House, nature of visibility and of the senses, on the model of knowledge itself, as it has devel-
oped in Western civilization. Here we examine art as a reality that can be seen and
: _____.r \\/
sensed. It is translated into images and figures, without being exhausted by those rep-
, : :- :.t l:l -{SSOciateS, resentations. In fact, art always alludes to its original invisibility, which belongs to the
nature of all things, ill-represented by words and that images can allow us to grasp
: :::..- C:ricago, IL,
intuitively.
The knowledge that is within images cannot be fully understood through a classicist
paradigm, but rather by unearthing their symbolic value. In art one grasps a symbolic
idea of form, which mediates between visible and invisible. In sum, art is not reduced
to an imitation of reality: the artistic form is not a mimetic image. It is rather the sym-
bolic and expressive meaning of a sensory reality, which has our body-that is also a
sensory reality-at its very centre.
Art is space, it is time that we can touch and see-but it is irreducible to the act of
seeing and touching. The senses are essential to our understanding of form, but para-
doxically the sensory form stands beyond what our senses can apprehend. From
Diderot to Lessing, Winckelmann, Baumgarten, Kant and Herder through the phe-
nomenological tradition, we will show how the sense of the body and that of art inter-
act. Not only do they contribute to an abstract idea of form, they also enrich each other
116 ELIO FRANZINI
through exchanges of meanings which unfold, in their figural expression, into the There is no dol
historical reality of artworks. expression (from t
senses', has been i¡
gives the priorin-
particulars of the r
certainty. It is not ,
duction of visible things offers immediately the reality of what's represented through a by Gregory the Gr
direct image. Nonetheless there is a basic difference that marks the relationship between writes that the firs
the different areas of knowledge: things that are sensible-that one can see, touch, and 'ideal'meaning. Le,
hear with their specific qualities that elicit emotions, passions, pleasure and sorrow- all the other arts.
have a different relationship with the image represented than those of the realm of all other senses. L,
thoughts. Knowledge is the link between these two processes and much of philosophy regarding the harn
can be viewed in terms of the history of attempts to find connections of knowledge beliefthat, rather tl
between thinking and sensing, which are situated at the origins of aesthetics and logic. p.96). This deba¡e.
Obviously, logic and aesthetics cannot be entirely separated. The epistemological of which we rvill a:r
problem of how we acquire knowledge assumes their connection, even if in different The superiorir',-
ways. But if we consider aesthetics, which is the study of the different aspects of theological reasons
sensing such as feelings and sensations, not as merely preliminary to the science of to summarise and
thinking, but as an autonomous territory inaugurating a specific forefront of knowl- Leonardo is to loca:
edge, then their relationship can be reconsidered in search for a, perhaps paradoxical, line the intrinsic i'.
aesthetic woild logos.t lf, as Aristotle teaches, there cannot be a science of particulars but judgements becau:
only of universals, then the aesthetic judgement, which refers to single things falling with the pictured rr
under our senses in a contingent way, will not be able to offer absolute knowledge: its Moreover, paintm;
images will have to be mediated through a categorical synthesis, i.e. every single object interpretation of r
will have to find its place into a system of objects. Imagination, or representation, apperception. Thr)
involves two moments: the passive given image and its active conceptual processing. that poetry and p:
The first operation pertains to the world of sensation (aesthetics), the second to the reaching a perfe.r :
world of thou ght (logic) . Representation, even if it has to encompass sensation , aisthesis, tive root of thines I
finds its truth and its universality only in the logos, in its translation into categorical and that makes paintin
conceptual terms. just the names oi::
is visible, as accord
attack on classicisr¡
' The search for a'logos of the aesthetic worid'is a characteristic of Edmund Husserl's of the idea thar pic,
phenomenology. tion of nature are t,'
RENDERING THE SENSORY WORLD SEMANTIC 1\7
. {pression, into the There is no doubt that the Aristotelian model, summarized for centuries in the
expression (from the De anima)'nothing is in the intellect that was not first in the
senses', has been influential on subsequent thought. The double status of the image
gives the priority of immediacy to aesthetics and to its sensory rapport with the
particulars of the world, but it gives to logic the possibility to reach knowledge with
certainty. It is not our intention here to discuss Aristotle's view, but only to presume
that the Aristotelian distinction between aistheta and noeta, between sensory things
and intellectual things, between sensory images and logical figures (from which we
started), may leave space for a series of questions from which aesthetics has developed
,*:;rg. the 'theory of in the multiplicity of its views.
:.:"; ¿ld of the acts of For example, in Leonardo's thoughts one finds two different directions that
:::.: .tlte¡ themselves appear to continue, on several levels and in a more pragmatic way, the debate on
: ,-,!:ects of sFnbolic the nature of the image and of the representation that started in classic Greek
:::-; ¿ unihed canon philosophy. Besides the common idea in sixteenth-century treatises that painting is
:: ¡e considered as an a 'high' mimesis of the real that is geared towards an ideal dimension, there is also
-:- . -r:ltrast the repro- an insistence that this noble imitation continues the metaphorical tradition started
:::;sented through a by Gregory the Great and by the Carolingian books. In fact, Leon Battista Alberti
:.,:.::.,1ship between writes that the first task of a painter is to represent a 'history', but to give to it an
l;.f,il See. tOUCh, and 'ideal' meaning. Leonardo adds one element to these theories: painting is superior over
-:!L:a ¿¡d 5g¡¡g\^/- all the other arts, including even poetry, since the sense of sight is superior to
:- ¡¿ .-,f the realm of all other senses. Leonardo, 'overcoming the monotonous unanimity of the critics
::;;: trf philosophy regarding the harmonious sisterhood of poetry and painting, manifests the vivifying
: -:- :, 1S .-,f knorvledge belief that, rather than twins, these arts are totally different in many respecls' (Lee ry74,
- "; ¿-. ¡i;¡¡l;_i and logic. p. 96).This debate, as we have seen, is not at all new and will have wide consequences,
,,-; ¡iistemological of which we will analyse here some historical and theoretical aspects.
I ;i.in if in different The superiority of the sense of sight is affirmed not for physiological, mystical or
:-fl3rent aspects of theological reasons, but for its characteristic of being a 'mental eye' having the ability
::-, I.r the science of to summarise and formulate aesthetic judgements. It is obvious that the first goal of
: - - :eirLrnt of knowl- Leonardo is to locate painting among the liberal arts, but there is also the aim to under-
:::haps paradoxical,, line the intrinsic judgemental potential of vision. Pictorial representations are truly
-r;. Lraparticulars but judgements because, differently from poetry, they pose an immediate relationship
'-r:e things falling with the pictured objects. Painting is superior to poetry for its larger communicability.
;: l:te knorvledge: its Moreover, painting owes such a superiority to the fact that it manages to be a life-like
: ii en'single object interpretation of nature through figures, which are visible data of immediate visual
1 --,r representation, apperception. This is how Leonardo unhinges the classicist paradigm: it is not true
'.:e:tual processing. that poetry and painting are sister arts and that their rules are equally effective in
: . ',he second to the reaching a perfect formal measure. Painting is superior because it goes to the qualita-
.. s;nsation, aisthesis, tive root of things themselves. It is the presence of a technical capability to construct
i i:lrL] aategorical and that makes painting superior to poetry: unlike the latter, painting does not register
just the names of things, but also the universality itself of their shapes-a form which
is visible, as according to Leonardo the eye is the organ that gives shape. Leonardo's
attack on classicism will be resumed by some eighteenth-century authors: some echoes
- i L-, ..-1.- of the idea that pictorial art maybe an expressive and aesthetic/imaginative interpreta-
tion of nature are found in Bacon, Vico, or Diderot. There is a claim that art interprets
118 ELIO FRANZINI
nature by and thus makes visible what would otherwise remain hidden. In summary, the sensoryworld aii,
the classicist tradition holds that painting and poetry are sister arts. Nonetheless, sion ofspace and oft
according to classicism, painting finds its completion in the word: as the Carolingian 'theory of art', there
held, it is the word that can best come close to the purity and immediacy of truth. field o{ aisthesls (and
Painting is, by nature and unavoidably, something material. In the post-Renaissance Diderot, for one, tali;
art systems, painting does not always manage to maintain its equivalence with school ofthought tha
poetry. In other words, the symbolic layer is increasingly occupied by the word, who knew the r'r.riti¡
leaving analogy and metaphor-considered rhetorical forms of inferior epistemologi- them, since he difter
cal value-to the image. The symbol's power of allusion, made manifest in poetry, intuitive dimensions
establishes its superiority over the metaphorical materiality of painting, unless the which is deployed ir
latter can subjugate itself to ¿ '1¡¿n¡g¡'-to the rules of perfection, of iconicity and the body that is erlLii
immutability that characterize poefry. ibility. According to
Even if these are the dominant views, they are not the only ones and, as demon- At the sáfle tinr
strated by Leonardo, they will not rule the century uncontested. Beyond its historical Laocoon to hold tha;
limits, classicism is simply that view that tends, in a Platonic way, to bring the image . of touch. Highligh,i:
back to the logos, and the seeing to the eidos, temporarily blocking the aesthetic and goal, i.e. denying t r
material layer. The image is thus limited to performing only a 'minor' task, which tends and the invisible.Bea
to limit it to narration, denying the possibility of switching from the metaphor to the form that onlv an ;
symbol, which is so characteristic of the iconic painting. In order to make painting take the complexitl' of ui
on that role of sgnbolic mediation-as a sgnbol of thought itself-that the Nicaean aesthetic dimension.
had attributed to it, it is necessary to exit from classicist paradigms in order to recover art. From this one cal
the formative sense of the image, as a certain thought that goes through the body's tational rapports c¿r
sensitivity and holds the power to render the sensoryworld semantic, through a gesture This also entails, as I
of poiesis. having its main rete
beauty is profounC'
the eighteenth cenru
enon that happens ,
born in France i¡ ü
VrsroN AND ToucH parts ofEurope due t
tion that Lessing de'
aesthetic dimension:
To this purpose, we shall now trace the main stages of the debate regarding the of the models of rrs
nature of visibility that may seem to pertain only to figurative arts, but that instead natural qualities of h
provide the occasion to show in an emblematic way how knowledge is not limited to a Instead, we neei
logic-discursive, abstract and formal, representation. What emerges in this debate is address tactilih.. Thi
the key idea that sensory and visible reality, which is translated into images and figures, attitude towards t¡¿
always alludes to an origin ary invisibility,embedded in the nature of things, that cannot to grasp its senson-r
be reduced to the word and can be shown only through the images themselves. this origin, even r,,jtL
The 'knowledge' in images is graspable only by breaking free from a classicist to perceive beaun'. C
paradigm controlled by rules and by reinstating a symbolic idea of
form. This symbolic role that Condillac t-
idea of form has the üsible and the invisible always simultaneously present (and
mediated): not more than the word, but in a different way. The artistic 'form' is not
a mimetic image, but it is, as Kant will say, an aesthetic idea not reducible to the
' see G.b. Lessns ,
sculptural g.oup. It! tn.-
word. To see and to touch (as Herder will maintain: see Herder 1994) what is beyond Goethe and Winckelm¿:
RENDERING THE SENSORY WORLD SEMANTIC 119
-:den. In summary, the sensory world allows us to place art in a symbolic rapport with the aesthetic dimen-
: .::ts. \onetheless, sion of space and of time. At the same time, just as there is no single concept of 'art'or
1: ¿s lhe Carolingian 'theory of art', there is no single concept of aesthetics. Moreover, the link between the
:rnediacv of truth. freld of aisthesls (and of aestheta) and artistic objects has not always been considered.
-; ¡¡st-Renaissance Diderot, for one, talks about these links, but it is only in the wake of the Leibnitz-Wolff
:. equivalence with school of thought that this discourse can find some systematic premises. It was Lessing',
-::ed br- the word, who knew the writings of Winckelmann and Baumgarten, who explicitly theorized
:rr-,r: epistemologi- them, since he differentiated among figurative and poetic arts through the aesthetic-
::::n:tist in poetry, intuitive dimensions of space and time. in this context, the specificity of figurative art,
r:i:rin-g. unless the which is deployed in space and escapes the temporal dimension of action, is given to
:. ¡: iconicir,v and the bodythat is exhibited in it. The way of exhibiting bodies is, in aesthetic terms, vis-
ibility. According to Lessing, it is visibility that dictates the rules of painting.
:.. :rd. as demon- At the same time, it would be inappropriate and not justifiable by Lessing's
i:'.', 1l its historical Laocoon to hold that the founding role of visibility implies the rejection of the sense
- :nlq the image
: of touch. Highlighting the visible value of aesthetics in figurative arts has another
:a :it. aesthetic and goal, i.e. denying (or, even better, giving new meaning to) the link between painting
- : : , sk. u-hich tends and the invisible. Beauty, traditionally linked with seeing, has become a way to address
:::: ::etaphor to the form that only an artform meant for bodies can satisfr. This formulation rejects
::.11: raintingtake the complexity of ugliness. In order to be accepted, this complexity needs a futher
j-::tlt .he Nicaean aesthetic dimension, and that is the temporal dimension that is characteristic of poetic
' -:: ,--,rCer to recover art. From this one can deduce that, for Lessing, modernity and all its layers of represen-
-::--ris.h the body's tational rapports can find in poetry a more adequate and complete expressive form.
-. :it:rtugh a gesture This also entails, as the following generations will confirm, that a form of aesthetics
having its main reference point in the empathic connection between visibility and
beauty is profoundly limited. Obviously it is not just Lessing, in the second half of
the eighteenth century, who let go of the corporeal visibility of beauty. It is a phenom-
enon that happens wherever one questions the tradition of a rhetorical classicism,
born in France in the wake of Boileau, which garnered so much success in different
parts of Europe due to all its academic inheritances. Therefore, even the specific atten-
tion that Lessing devotes to 'visibility' need not be interpreted as a refusal of other
aesthetic dimensions in order to grasp the beauty of bodies, but only as an adoption
¡=-: resa¡ding the of the models of visibility that, since the Renaissance, associated the corporeal and
'-.ut
:r. that instead natural qualities ofbeautiful forms with the act of seeing.
; -: :tLr¡ iimited to a Instead, we need to consider as similar to Lessing those authors who explicitly
;.s n this debate is address tactility. The consideration of tactility can be viewed as the sign of a critical
r::rqes and figures, attitude towards the 'closed' and 'regulated' concept of an artistic beauty is unable
-:-:r:gs, that cannot to grasp its sensory meaning, its aesthetic origin. On the contrary, touch emphasizes
:r: ¡lselves. this origin, even without opposing üsibility, but rather supplying further possibilities
. ::¡'m a classiciSt to perceive beauty. Generally speaking (in philosophical terms), this is testified by the
, "'i;. This symbolic role that Condillac outlines for touch inhis Treatise of sensations. He associates touch
:s-r' present (and
::rslc 'tbrm'is not
,: ::ducible to the " See G.E. Lessing (1729-178I), Laocoon (1766). This text continues the debate on this marble
sculptural group. Its theme is told even in Virgil's Aeneid.This discussion was continued also by
:-j -lrhat is beyond
Goethe and Winckelmann.
120 ELIO FRANZINI
with sight as a sensation fundamental for knowledge, and defining it as a 'philo- Winckelmann. The
sophical and profound' organ., This idea is, at a different level, confirmed by Berkeley link between 'touch
and, more generally, in English eighteenth century culture, within which Burke's ers the treatises o¡:,
On the Sublime and Beautiful (the 'manifesto' of artistic anti-classicism in this period) represent the a_riolc
plays a fundamental role. In the context of a sensory vision of the aesthetic categories debate between the
(both 'beautiful' and 'sublime'), it gives to tactility a special attention, explicitly paired onlypainting and p,
up with sight in its ability to relate itself to form and to the formless. The alternatine
There is a passage in which Burke specifies that the description of beauty starting Paragone betn'een r
from sight could also be illustrated through touch, which produces a 'similar effect' brought within thr
(Burke :,992, p. 136): it provokes 'the same kind of pleasure' and demonstrates theorized the link',,,-
howthere is'alinkbetween all of our sensations', which'are nothingbut different t)?es arts (in the u'ake o:
of impressions'provoked by different kinds of objects, but'all in the same way'(Burke bases). In fact Herd
Lg92,pp.86-7). Moreover, not only does Burke go so far as to present the famous confronts us uith tl
paradox through which he formulates the hlpothesis of an enjoyrnent of colours the aesthetics ofscu_
through touch, but also, and more importantly, he designs his entire aesthetic theory - was forgotten bl hrs
of beauty around sensory qualities that are mainly tactile. In fact, the variability of ofa path that, throui
bodies (which is a crucial element for a non-classicist and non-model-ruled concept of ship between touch
beauty) can be best 'grasped' through touch. His definition ofbeauty, notoriously sen- tion ofthe trvenrier
sual and feminine, is totally tactile, since pleasure-the same phenomenon that, when Herder is at the
"-r
brought into the sensory world, grants the continuation of species-is not linked to aesthetic-philosopl::
geometric objects but rather to 'soft, smooth, sensual bodies', to 'beautiful' bodies, the aesthetic-artis:r,
evidently associated with femininity. In sum, touch is the true sense of beauty, the one century theorized is
that defines its kind ofpleasure and also, through sex, its social usefulness, since'touch definition of the rde,
receives pleasure from softness, which is not originarily an object for sight' (Burke that 'sight shorvs u
can be recognized l:
touch is absolutely necessary because it allows an empathic rapport with the object, and not even of hn¡
thus causing a 'relaxation' of bodies. Still, even the opacity of the sublime and the form- One needs to doub,.
lessness that comes with it can get excited by the tactile powers, where 'rough and sharp pertaining to the bo
bodies' cause an 'impression of pain', 'which consists of a violent tension or a contrac- 'body' (i.e. the obie;i
tion of the muscle fibres'. In conclusion, touch has a precise aesthetic role in the beauti- resented only br- s;
ful and in the sublime and, particularly for the beautiful, it is the primary cause of that But the 'bodv' of sc:
pleasure that defines its form and function. It reveals itself as an 'aesthetic' organ is to say, with a bt-c
because it does not grasp the unity ofbodies, but rather, against every form ofclassi- hardness, softness. s:
cism, the variety, which is necessary to beauty (Burke g9z, pp.t6z3) . general epistemolrrEl
When Herder, in his essay on sculpture, Sculpture, (1994, originally published in not compare and cu.:
German inry78), seems to attribute a special role to tactility, he is simply continuing taken together can il
the eighteenth-century tradition. This is confirmed, in the first place, by his cultural touch guides siglrt. :"
references who, other than the members of Wolffls school, are Condillac, Diderot, able to give, alone. th
Burke (and with him all of English empiricism), winckelmann and Lessing. Herder's century reasons for d
underlining of the'antique' and'past' aspects of figurative art, and of sculpture in par-
ticular, was already present in Lessing and, perhaps, could be deduced from
a In rgor Alois fuegi r:
the artistic forms relate:
3 EtienneBonnotdeCondillac
Q7ry-q8o) wrotehis Treatiseof Sensations,inspiredby drives his thinking and ::
Anglo-Saxon philosophy, in r754. As a consequence, he was accused of materialism. phenomenology of risir. :
RENDERING THE SENSORY WORLD SEMANTIC
rl.-i1i¡g it as a 'philo- Winckelmann. The true novelry of the Herderian discourse is elsewhere, in the explicit
c,:rltrmed by Berkeley link between 'touch' and 'sculpturs'-sys¡ if this is not completely new, if one consid-
,,irh¡ u,hich Burke's ers the treatises or iconography of the Renaissance-almost as if this art may come to
¡srci-sm in this period) represent the axiological reference ofthe aesthetic-corporeal sense, and re-define the
rÍ ¿esthetic categories debate between the visible and the invisible, which, until that moment, had involved
:t: :r. erplicitly paired only painting and poetry, image and logos.
:ia
The alternating fortune of tactility within the sixteenth-century debate on the
.,-,r of beauty starting Paragone between the arts had been surpassed in the eighteenth century or, at least,
:r;es a 'similar effect' brought within the context of knowledge. But never before Herder had anyone
; ¿nd demonstrates theorized the link with sculpture in a context meant to specif' an organic system of the
rJ rutdi.tferenttFpes arts (in the wake of Lessing and on aesthetic-intuitive bases, and not on rhetorical
:r- same x'ay'(Burke bases). In fact Herder, in an advance that is full ofconsequences, argues that tactility
-- :iesent the famous confronts us with the third dimension, with that spatial characteristic that identifies
:nri-,Tnent of colours the aesthetics of sculpture: the haptic moment, as Riegl will call it.a Even if this moment
::rre aesthetic theory was forgotten by his contemporaries and by the next generation, it is the starting point
.;. rhe i.ariability of of a path that, through Lipps or Fiedler, Ieads to multiple theorizations on the relation-
- i¡,-ruled concept of ship between touch and sculpture, which has been characteristic of the critical reflec-
r".:.'. irotoriously sen- tion of the twentieth century.
:aiIriflorl that, when Herder is at the origin of a problem that, even if present and made possible in the
;-.'-is not linked to aesthetic-philosophical thought of the eighteenth century, leads far from its premises:
,,: ':eautiful' bodies, the aesthetic-artistic collaboration between visual and tactile that the eighteenth
-..¡ ,¡: beauty, the one century theorized is here cast in doubt, with a resulting backlash against the general
ti¿-ness, since'touch definition of the idea of artistic form. Herder maintains, at the beginningof sculpture
::;: lLrr sight' (Burke that 'sight shows only figures, while touch only bodies: ever)'thing that is shape
:¡::r in the sublime), can be recognized only through the sense of touch, while through sight only surface,
'¡,-,r: ,,r-ith the object, and not even of bodies, but onlythe surface exposed to light'(Herder 1994,p.4r).
:b-:me and the form- One needs to doubt not only that (as Lessing said) all figurative arts may be arts
'rough and sharp pertaining to the body, but also that 'form' may be apprehended through sight. The
--ie
:::tst0n or a contrac- 'body' (i.e. the object that is present in space through its three dimensions) can be rep-
;i:,role in the beauti-, resented only by sculpture, while painting will have to make do with the figure.
::rman'cause of that But the 'body' of sculpture is such because it is in tactile contact with ourbody, fhat
.=: 'aesthetic'organ
is to say, with a body that can really make one 'feel' the form, its 'impenetrability,
r-. .n' tbrm of classi- hardness, softness, smoothness, form, figure, roundness'-and here one can see the
general epistemological concept that underlies the Herderian discourse. One need
:-¿11)-published in not compare and contrast touch and vision, but rather comprehend that only when
:s simplv continuing taken together can they lead to judgemenf. As a consequence, those artforms where
i,ace, b1'his cultural touch guides sight, i.e. sculpture, will have such an epistemological centrality as to be
: Londillac, Diderot, able to give, alone, the'form'of things. In Herder, therefore, the canonical eighteenth-
irJ. Lessing. Herder's century reasons for the epistemological association between the two senses are utilized
,"i.,risculpture in par-
i be deduced from a In rgor Alois Riegl published an important fext, Late Roffian art industry, in which he discussed
the artistic forms related to the optic and to the tactile. The principle of the 'will of art' (Kunstwollen)
. -:-':ired bv drives his thinking and it is very strictly linked to the historical relativity of the several ways of the
-:ii.n. phenomenology of vision.
122 ELIO FRANZINI
both to distinguish between the respective arts and to differentiate their epistemologi- Touch indicates the
cal roles. the form's endeavou
In addition to having an intrinsic phenomenological validity (in the proper sense touches completelr-.
of the word, since he explicitly alludes to Lambert: Herder ry94,p. +5), the Herderian is the shape of quiet ;
distinction also explains the double level on which the topic of touch needs to be in art, a senson/ \ra'\-
understood. On the one hand, by continuing the anti-classicist traditions of the eigh- illusory finitude of tl
teenth century and the debate on Paragone between the arts and their interpretative unitaryand indefinit
potential, Herder argues against the epistemological centrality of sight by adding to it darkness of the nigh
the necessary and complementary role of touch. On the other hand, though, he high- form, but not reduc:
Iights the rapport between sight and touch to further a philosophical discourse on the one single form. kee
arts which implicitly challenges the eighteenth-century'systems ofthe arts'. According formal temporalin- c
to Herder, these systems are built only on hearing and sight: to these senses one now Herder's conclusii
needs to add touch, since it does not limit itself to perceiving what is 'outside' of it mentioned, in eieht¡
(sight), and does not only put one object 'next' to the other, but can perceive tual'premises, thel r
them 'one in the other', thus offering not only surfaces or sounds but also forms. - canquestionrvhr-t¡r
If, implicitly referring to Alberti, painting is tabula and proceeds with a clear infinity in a form. ü
reference to rhetoric (in fact, Herder maintains that it always has to come back to mimetic and repettr:
invention and disposition, that is to say, to the two fundamental parts of the construc- theories of art rthi¡r
tion of the discourse), sculpture is a spatio-temporal form (he writes'it is here and issues have been nur
it lasts'), it is the existence of the corporeal presence of life itsell which aspires to eth centuries. Some
aesthetically present 'the soul of the body' (Herder 1994, p. 50).t tactility, be it touchr¡
What we have is not a dichotomy between 'sight' and 'touch', but rather, to and Worringeró or tl
better understand the superior formal completeness of the latter corporeal sense, theories are not suli
a partition between the respective artistic correlates, i.e. between painting and sculp- sophical) that Herde
ture. The first is'dream'and'enchantment', while the second, by contrast, is truth and should be touch that
presentation (Darstellung); the first is rhetorical or 'novel', the second a living pres- can maintain that tt
ence. Sight, by itself, is only'the hand's reason', while 'only the hand gives forms, con- between tactilitr ano
cepts of what the forms mean, of what lives in them' (Herder 7994,p.67). Moreover, the classicism of the
following Burke, if the hand prefers a rounded and sensual form, sculpture does not The Herderian p't
limit itself to it, to the beauty, but rather it tends toward the sublime which demands via touch, the Less:r
reverence. This is the paradox of a sublime form that 'creates its light' and 'creates Herder one can apF
its own space': exhibition of infinity that demonstrates the infinite nature of artistic aesthetics behreen 1
form, its ability to build a space that, in order to be complete, needs the other intuitive escapes isolation anc
form, i.e. time. This time is not linear and narrative but is that of a hand that actively perception, r'r.hich e
moves across the surface of the matter through touch, going beyond the limits of power behind the ¿:
visibility and pronounceability. One can thus grasp the epistemological possibilities
ofart starting from an originary aesthetic act that pursues the pre-categorical through u Wilhelm \\iorri¡q¿:
a spatio-temporal present form. In fact, Herder writes, even blpassing the issue theories of fuegl ani H:
of sculpture, what the hands touches 'seems bigger than what the eye sees in a flash the title Abstracfion ¡,:"i
as fast as lightning', and is thus appropriate for the sublime (Herder 1994, p. 98). Marc, Klee, and Kan¿r
; In 1896 Bernharc F,
¿ rheir epistemologi- Touch indicates the aesthetic tension that is in the form, in the sublime aim of art, in
the form's endeavour to be not only a representation, but also a figure: 'the hand never
touches completely, it cannot grasp any shape in one go, aside from the sphere, which
is the shape of quiet and of perfection resolved in itselfl (Herder ry94,p.99). Touch is,
in art, a sensory way to feel infinity, the sublime, without totally constraining it in the
illusory finitude of the visible representation. Sculpture is, for touch, at the same time
unitary and indefinite, present and overcoming its finitude: 'the sculptor stands in the
darkness of the night and goes searching for gods' figures' (Herder ry94, p.gq). It is
form, but not reducible to allegory or to abstaction, because it is never entirely given to
one single form, keeping its opacity as an invitation to deepen, time and again, the
formal temporality of space.
Herder's conclusions on tactility and sculpture find their first justification, as already
mentioned, in eighteenth-century culture. Nonetheless, on the bases of these'contex-
tual' premises, they raise a more general problem. In fact, starting from this point, one
can question why touch should embody the sublime opacity of art, the ability to grasp
infinity in a form, the same ability that allows us to overcome a conception of art as a
mimetic and repetitive illusion (connecting back to rhe fl rouge of the interpretative
theories of art which started in the Italian Renaissance). The consequences of these
issues have been numerous, particularly in art criticism of the nineteenth and twenti-
eth centuries. Some led towards the detemination of a specific 'value' connected to
tactility, be it touching with the eyes (Hildebrands), the gothic-barbaric haptics of Riegl
-;i'. but rather, to and Worringer6 or the 'tactile values' of Berenson.T Others have affirmed that these art
ll;: -trrPOr€ai Sense, theories are not sufficient to solve the'aesthetic problems' (epistemological and philo-
r:rliiinq and sculp- sophical) that Herder brings in (Merleau-Ponty 1979).8 The problem, in fact, is why it
;::r:ast. is truth and should be touch that indicates the limits of representation. Historically speaking, one
i,-;,rnú a living pres- can maintain that this happens because Herder is within a discourse where the link
:: j illes forms, con- between tactility and sculpture aims to show that sublimity of art which is unknown to
:+- l. 6; l. lvloreover, the classicism of the'sister arts'.
:. r;,-ilpture does not The Herderian position can be described as an attempt to raise to the sublime,
--::-r; .,,,hich demands " via touch, the Lessingian system of art. Instead, one can hypothesize that through
:s irght' and'creates Herder one can approach other questions which bring back the classical debates in
:i:: :rature of artistic aesthetics between visible and invisible. On the one hand, touch is the sense that
i.. :¡e other intuitive escapes isolation and opens to the totality of the aesthetic experience. It is an embodied
i;. i:and that actively perception, which goes beyond the clarity of 'visibility' to include also the hidden
:,t'''¡nd the limits of power behind the apparent transparency of the representation. Touch indicates the
.: - -'sical possibilities
-¡¿:egorical through u Wilhelm Worringer (188r-1965) was indebted to Lipps for the notion of 'empathy', and to the
;,-"passing the issue theories of Riegl and Hildebrand. He is known mainly for his dissertation, published in r9o7 with
i: ¡i'e sees in a flash the title Abstraction and empathy. This work gave impulse to the artistic theories of artists such as
develops the theme of 'tactile values'. He maintained that one cannot gather a precise sense of the
.-:. .,: -.alled pure
.:,r :i:.uenced greatly third dimension through sight alone.
s Maurice Merleau-Ponty (r9o8-r96r) is the main protagonist of phenomenology in France.
124 ELIO FRANZINI
possibility of reaping the hidden aspects of form, the inüsible, the 'unfinished' that, of painting as a pure
since the time of Leonardo, has been the best response to the exclusively narrative, the work is more th¿
metaphorical or rhetorical view of art. In this way, one can affirm that touch is an ulte- space. According to \
rior method of opening the symbolic dimension of art, which is precluded-as Diderot things, in a sort of co
taught-by its reduction to language or to only one of the senses. Touch is, instead, rg79,p.zz8-g).
bound to the ambiguity of a bodily gesture: for Herder, touch was irreconcilable with To argue that in p
any form ofallegory or rhetoric. at a crossroads' (trfe:
eye but a combinatit
in which it is imposs
1968, p. r74). The cen
enology ofperceptio:
MnnrnAU-PoNTY AND THE from the moment th,
tTorAt EXPERTENCE' perception, are inter
ultimately discernit'1'
The search for this
Herder's arguments or, perhaps more generally the enfire Settecento in its anti- the ontological sen,
classicist mode, find their theoretical reference outside art criticism (which, indeed, specificity of the rtorl
was perhaps not their intended destination) and therefore outside any particular embodied, and the s;
axiology ofthe various arts or any scientific, physiological claim about the capacity of always within a colru
the different senses. It is, in fact, in contemporary thought regarding aesthetics and If this rule is not cc
art that one finds links to the questions raised by Herder. The first and most evident Husserl called 'm'-
path leads to Merleau-Ponty, for whom painting has the task of offering a 'visible and Descartes. Scier
existence' to what 'the profane vision believes to be invisible', in such a way that experience ofsensins
the 'muscular sense' is not needed to perceive the 'voluminous nature of the world'. centre of graviti' of ,
Doubtlessly, for Merleau-Ponty, the eye and visibility were of supreme value, in that speaking feel in orc
painting'even when destined for other purposes always celebrates the enigma of visi- the physicist concei
bility'. However, visibility, the eye itself, is occupied with more than mere 'visual data': communication be¡
it is the intention, truly platonic, to seek the Being, its 'traces', that transforms vision viction that'the lbn
into a 'mirror or concentration of the universe', a 'metamorphosis of Being in relation to their spe
the vision of the painter' (Merleau-P onfy t979,p . zrl). The centrality of the gaze, there- (Merleau-Pont\- 19Él
fore, is not to the detriment of touch but rather focuses our attention not on the physi- We can then cor
ological, but rather on the ontological nature of perception: returning to the theme experience'ofour e:
imposed by Herder. Thus, when Merleau-Ponty writes about the eye and visibility, he sensory experienccs
shows how painting is an 'interrogation' of the invisible through the üsible: but this by science. Instea,i.
interrogation always begins with a unitary body, with the 'delirious and secret genesis a single object, r'-h,
of the aspects of our body'. Vision and the eye, privileged because paintings refer to the 'the body, which r']
act of seeing, are metaphors for a corporeal access to the hidden truth of Being. all means in its po,.' t
Additional evidence for this idea comes from Merleau-Ponty's suspicion of the We could cont-¡
optical-physical systems of vision, in particular perspective theories of vision. the 'precategorical'
Metaphors transport us beyond the 'illusionism' ofvision, back to the 'metaphysical' been ignored br-th'
significance of painting: to an invisible Being made of contingencies, reversibility, of these citations n oul
living and lived bodies. Consequentially painting is not complete either in 'fil
rouge', which, i
the visual or in touch's unveiling: it is not'adding a dimension to the two dimensions Herder) is defined ¿
of the canvas' but, instead, echoing Nicea, it supersedes entirely the consideration Like Herder, lrferle'
RENDERING THE SENSORY WORLD SEMANTIC 125
:::e 'unhnished'that, of painting as a pure and simple representation. As Herder had argued for sculpture,
i:':;-usi\-eiy narrative, the work is more than an object in space, rather an object that constructs its own
r -:r¿i touch is an ulte- space. According to Merleau-Ponty, in this space'it is the painter that is born in these
:.;iuded-as Diderot things, in a sort of concentration and returning to self of the visible' (Merleau-Ponty
.¡¡. Touch is, instead,
1979,p.228-9).
¿. :ireconcilable with To argue that in painting 'vision is the encounter of all the aspects of Being like
at a crossroads' (Merleau-Ponty 1979, p.46) gives rise to a body that is not just the
eye but a combination of tactile and visual, a protagonist in an 'integral experience
in which it is impossible to measure the contribution of each sense' (Merleau-Ponty
1968, p. ryl.The centrality of this'embodied synthesis' is undeniable in the phenom-
enology of perception and is not contradicted by the privileging of the act of 'seeing',
IE
from the moment that tactile data, even if often left out of 'scientific' consideration of
perception, are integrated into a total experience in which the separation is not
ultimately discernible.
The search for this 'total experience' that transports Merleau-Ponty to ontology and
i-;--.riirr in its anti- the ontological sense of painting, is distant from an analysis of the objective
:-¡:: -'rhich, indeed,
specificity of the work, of the 'canvas' as a perceived object. But this total reality remains
::'-l: anr- particular embodied, and the specificity of the sense or senses involved in the act of perception is
.i,-,u: -rhe capacity of always within a communicative context in which'synaesthetic perception is the rule''
::: :ls ¿esthetics and If this rule is not consciously explicit it is due to the 'cultural' intervention which
.:.: :rd most evident Husserl called 'modern objectivism' and which he traced back to Galileo
, : rte:ing a 'visible
': and Descartes. Scientific knowledge, according to Merleau-Ponty, has removed the
-r ruch a rvay that experience of sensing: 'we are unaware of it onlybecause scientific knowledge shifts the
:-::*re of the world'. centre of gravity of experience, so that we have unlearned to see, hear and generally
-::-r:Le r-alue, in that speaking feel in order to deduce, from our bodily organization and the world as
:. -,he enigma of visi- the physicist conceives it, what we are able to see, hear and feel'. It is, instead, the
::r r:-¡e 'r-isual data': communication between the senses that opens up the 'structure of things', in the con-
--:: :r¡nsforms vision
viction that 'the form of objects is not in their geometric shape: it stands in a certain
-ltL:,sis of Being in relation to their specific nature and appeals to all our other senses as well as sight'
-,:. ,r: the eaze, there- - (Merleau-Ponty r968, p. 3o8).
t, rt rlc-ri on the physi- We can then conclude that the 'form' of objects is inseparable from the 'total
:*:n:ns to the theme experience' of our embodied senses: our perception is the capacity to unite all of our
r-,'. ¿rd r-isibility, he sensory experiences into a 'single world'. This world differs from the one described
- -h¿ r-isible: but this by science. Instead, it is nearer to the way in which our binocular vision perceives
:s :id secret genesis a single object, where it is not the 'epistemological subject' that synthesizes but
:.rnrinqs refer to the 'the body, which when it escapes from dispersion, pulls itself together and tends by
ii.¿r truth of Being. all means in its power toward one single goal'.
','' :uspicion of the
We could continue with these citations from Merleau-Ponty, bringing us to
-¡¡ ¡,: r-ision. the'precategorical'sense that he attributes to our body schema, an aspect that has
r,:, :he'metaphysical' been ignored by the intellectualism against which he has always battled. However,
of
:;-¿s. rer-ersibility, these citations would bring us back to the central point: the unity of the senses, that
;--nplete either in 'fi1 rouge', which, as in the anti-classicist views of the settecento (from Burke to
,-i3 ¡1ro dimensions
Herder) is defined as an opposition to a purely'intellectual' form (and formation).
,'. :he consideration Like Herder, Merleau-Ponty wants to go beyond a purely'scientific' conception of
].26 ELIO FRANZINI
the body and form. He describes a dynamic 'common sense' in which the senses 'trans- sensation, one throu
late among themselves without needing an interpreter', without passing'through the encounterbrings neit
idea'. These observations, writes Merleau-Ponty, permit us to give full sense to for Deleuze, the figur
the expression of Herder: 'Man is a perpetual common sense, which now is touched form related to an ob
from one part and then touched by another.' This citation from Herder shows (the 'body synthesis'
that, beyond the differences in historical contexts, the road is the same: via touch multiple instincts. 1er
one reaches the totality of embodied experience to establish (with the aid of a phenom- For Deleuze, paint
enological stance) that with the notion of the body schema 'it is not only the unity of in the preceding pag
the body that is described in a new way, but also through this, the unity of the senses a lived body is 'a pali
and the unity ofthe object' (Merleau-Ponty 1968, p. 314). In this, 'the unity and identity able Power'. The unit
of the tactile phenomenon do not come about through any synthesis of recognition in plunges into chaos.
the concept, they are founded upon the unity and identity of the body as a slmergic are perpetuallv and
totality.' Thus, tactile experience is always a meeting between the organic totality and here, to the 'bodr- r,¡:
the embodied experience. sensation is the amp-
"beneath and ber.ond
In fact, this dark f,,c
the temporal and i..r
ible forces which. ac;
painting grasps, are :r
Dnrnuzg AND THE cRUEL BoDY ofthe genesis ofthe :
in painting it is the Being which manifests its significance, it is perhaps necessary to ask even ifnot directh-re
what role might be reserved for the specific experience of the artistic object as a spatio- noticed):'these man.
temporal aesthetic reality, with its capacity to bring forth precise thoughts. This is one world into the visu¿i
of the central themes of contemporary aesthetic thought, which harkens back to the The optical orean
relationship between splbol and metaphor, betweenvisible image andinvisible mean- linear perspectir.e. bl
ing, and which seems to be a fundamental axis of the entire history of aesthetics. For sensing body, is no.,.,'
Merleau-Ponty, painting is the figural expression of an ontological form that has its an operational diag:,
primary reference (genetic and interpretive) in the organic totality of the body and its toward abstraction. r
actions. In contrast Gilles Deleuze, despite being included in a tradition that is outlined not have to suboro:
and rooted in a tactile üsion of art, challenges any interpretation which ignores the Expressionism (or :':
cognitive sense of art. In his bookFrancis Bacon: TheLogic of Sensationthere are evident an exclusiveh¡ manu;
influences deriving from the phenomenological tradition, particularlyMerleau-Ponty: the hand subjuga:e.
the originality of the corporeal sensation, the centrality of the figure, the need to 'which is neither op:
supersede the false dilemma between figurative and non-figurative, the hermeneutic 2oo5, pp. \7o-r71 .
role of Cézanne, and the suspicion towards an illustrative or narrative conception of This is the I
the pictorial are all signs of this legacy. But here one also finds a new consideration of a haptic way rvhere ;
the relationship between sensation and figure, placed in an unintentional context, in tradition in paintin<.
which 'the figure is the sensible form related to a sensation'; 'it acts immediately upon a tactile-optic spac:'
the nervous system, which is made of flesh'. The reference to the flesh and to sensation can be defined as inr,
is explicitly connected to a phenomenological (Heideggerian) Being-in-the-World, tion to express itselt.
in virtue of which 'I become in the sensation and something happens through the only the optical fun;
RENDERING THE SENSORY WORLD SEMANTIC 727
icl.: the senses'trans- sensation, one through the other, one in the other' (Deleuze zoo5, p.85). But this
:passing 'through the encounter brings neither an intentional and descriptive stance nor an ontology. In fact,
',. sir-e fulI sense to for Deleuze, the figure-the form attributed to the sensation-'is the opposite of the
"hr;h norv is touched form related to an object'. This challenges the singular, global unity of Merleau-Ponty
::r¡¡n Herder shows (the'body synthesis') and insteadbringsusto fragmentation: to a sensation made up of
'rhe same: via touch multiple instincts, levels and intensities that lacks a defined centre.
r;ie ¿id of aphenom- For Deleuze, painting, particularly that of Cézanne, excludes the idea, often found
orh'the unity of
rt-¡r in the preceding pages, of an ontological base. Rather, he argues, a unity based on
:¿ unin-of the senses a lived body is 'a paltry thing in comparison with a more profound and almost unliv-
--:e unin-and identity able Power'. The unity of the bodily rhythm can be found only where 'the rh¡hm itself
::srs rrI recognition in plunges into chaos, into the night, at the point where the differences of level
:l¿ ttrdr as a slnergic are perpetually and violently mixed' (Deleuze 2oo5, p. ro¡). He makes reference,
¿ ¡,¡Senic totality and here, to the 'body without organs' of Artaud, which has only 'zones and levels' where
sensation is the amplitude of waves, a'hysterical'body facing the artwork's presence,
beneath and beyond the representation.
In fact, this darkbody, hysteric and cruel, has eyes everywhere and refuses to develop
the temporal and formal moment of the organic encounter with the Being. The inüs-
ible forces which, according to a phrase by Klee also used by Merleau-Ponty, the art of
ODY painting grasps, are not comprised within a formative dlmamics; they are not branches
of the genesis of the Being; they are not even icons of transcendence, but rather of an
intense and pulsive obscurity. They are screams that, unlike those of Laocoon, should
:1 a¡:rtmon sense and not to be confused with the visible spectacle that makes you scream. They are forces of
:. ::r \lerleau-Ponty, isolation, deformation and dissipation instead. In sum, these corporeal, obscure forces,
:i::rs necessaryto ask even if not directly referring to sculpture, are primarily manual actions (as Herder first
:::; .rbiect as a Spatio- noticed): 'these manual signs, almost blind, mean to testifr the intrusion of a different
'.:.',ughts. This is one world into the visual world of figuration' (Deleuze 2oo5, p. 168).
: :arkens back to the The optical organization, destabilized by Merleau-Ponty through his criticism of
.: ¿nd i¡risiblemean- linear perspective, but then restored on the basis of the organic unity of the living and
-,:-,- of aesthetics. For sensing body, is now definitely surpassed by defining painting not as visual data but as
.:;:.i lorm that has its' an operational diagram. In contemporary art, this diagram has taken different routes:
¡ ,rithe body and its toward abstraction, when shapes 'belonged to a new, purely optic, space, which does
il 'r.rn that is outlined not have to subordinate any manual or tactile elements to itself; toward Abstract
::: in-hich ignores the Expressionism (or formless art) where 'optical geometry falls apart to leave room for
; ;:., r there are evident an exclusively manual line' ; and moving on to the pure tactile Action Painting, in which
*::1"' \ Ierleau-Ponty: the hand subjugates the eye. Most importantly, there is also another way, Bacon's,
. ngure, the need to 'which is neither optic as abstract painting, nor manual as Action Painting' (Deleuze
:lre, the hermeneutic 2oo5, pp. t7o-ryr).
:r::ir-e conception of This is the Egyptian way, which means, according to a definition by Riegl,
,-',',' consideration of a haptic way where painting becomes strictly associated with sculpture. The Western
rtentional context, in tradition in painting, often ignoring this plastic aspect of figurative art, has substituted
,:s rmmediatelyupon a tactile-optic space for this primitive haptic vision and its spatiality. This new space
:-:sh and to sensation can be defined as intentional, since 'it is no longer the essence, but rather the connec-
3e
-g-in-the-World, tion to express itself, that is to say the human organic activity'. When the eye takes on
r¿:pens through the only the optical function, leaving behind the haptic, as one can see in Merleau-Ponty,
128 ELIO FRANZINI
(Deleuze
it ends up subordinating the tactile to itself, considering it a'second power' surpassed, as is even' :
is not history nor representation, that unfolds under an eye that is part ofthe un-coor- painting is onlr. a mu,
dinated sensorial system. It is not a symbol of the organic unity of the body' Lessing's and repetition (r99-r"
denial that figurative arts may be linkable to action and narration, and thus
of the body
to the space ofihe body, is reaffirmed, on the condition that the organicity
be shattered in the multiple tactilism of the Figure, in which'the hand, the touch, the
(Deleuze
grasp evoke this direct manual activity that traces the possibility of the fact'
zoo5,p. z3z)
Painting itself embodies the difference that is the fact itself, the paradoxical
Ivrp rr c
foundation of a third eye that, as Herder maintained, no allegory can express and no
ontology can embody: one haptic eye, which leads to a'new clarity'-evidently neither
Cartesán nor phenomenological-in which every dualism between tactile and optic
is
Starting from this e
Henri Focillon (r88r-r943) is the main exponent of the formalist stream of French art critique.
e one can arrive at a c
In 1934 he published a successful book, Life of Forms, and subsequentiy a short essay titled
Exrlogl' relationship benr-eer
The problem of rh¡hm is central to the investigations of French aesthetics at
the beginning of the in the 'philosophl r
twentieth century. For a general survey ofthese issues, see Franzini (1984)' talk about 'philost-'c
RENDERING THE SENSORY WORLD SEMANTIC L29
,-,rd power' (Deleuze surpassed, as is every form of their ambiguous fusion, to inaugurate a 'logic of sensa-
:b,e,i bv Wolff, which tion'. In this logic there is no room for representing, and the figurative semantics of
-"'hich takes place in
feeling are articulated in such a way that the centrifugal and centripetal rh¡hms of the
iruine art), andwhich body have dispersed the harmonic organicity of the form.
rrlq on one side the Deleuze not only redefines Herder's 'sensorium commune', but he also continues to
.l;:r leads to the praise advance toward illustrating the opacity of the presence' of the figure, that is to say,
rhese lrr'o having the of the elements that a living body will never be able to reconcile in an organic vision
¿urdng or, as Deleuze (constituent or ontological), and which therefore have to be accounted for by a
: -.s sic representation' 'cruel body'. However, at the opposite end of the scale from Herder, and closer to the
:,it did not convince problems that today's art world presents, the haptics of the body refuse a symbolic
.r:i complex correla- consideration of the tactile form. This presents itself, in its substantial sensory pres-
: ;u-,hentic history of ence, as indefinable, similarly to how it was described in Merleau-Ponty's ontology:
i¡:¡e: identifies with the different clarity pursued by Deleuze does not investigate the symbolic opacity of
::r; haptic function the work as form, as an aesthetic reality whose condition of being a representation
and an image holds an element of secret workings. Although this does not reduce it to
:; i
¡r.'ught by Bacon the bare aesthetics of the form, nonetheless it cannot do without such aesthetics and
: :rt:Lrrt behveen the without its frame.
r: lnterpreted along Deleuze, in his complex diagrams that deconstruct the optic and the tactile, under-
:..l-?onn', since 'it is stands that in painting there is an event of such strength that it cannot be explained
, :¡¿nrc substitutions' by simply linking it back to the Being (and perhaps this is the most violent break from
¿:rr;: rhe digital, the the tradition that started with Neoplatonism). This event is made up of entities
----1 :-le eve and among lacking a harmonic rapport with the body, so that the exchanges and reversible interac-
; . ir i''hich different tions that perception deploys are not immediately true options or tensions, but rather
:- -: i.11 anr. story (not primarily facts, where what is at srake is the body, with all its tactile physicality. Even
,-: represent anl.thing if h" do., understand clearly the difference of painting-or rather the thickness of
.,-:;s qei together in a this difference that characterizes painting-and its constant escape from allegorical
or metaphorical substrates (Ricoeur rg8r, p. 19), as in Herder-nonetheless he links
.,,::r. ¿ presence which the sublime opacity of the symbol to the ambivalences of the simulacrum, in which
-s :¡n
ofthe un-coor- painting is onlya multiplicityformedbydifferential elements, as expressedinDifference
üe body. Lessing's
,:,: and repetition(r994, originally published in French in 1968)'
:l:ration, and thus
: :s¿n jcin'of the body
=
::rd. the touch, the
; -: the tact' (Deleuze
:r;ieristics of 'making the image has produced also illusory'enlargements'-just think about the movie about
l: :iom their own aes- the image,'Blow up', by Michelangelo Antonioni-which offer only the surface
::i¡nl. A phiiosophical of things, in which the invisible blends into the illusion that destroys feeling and
-,¡. to be resolved by the sense of truth. Recently, Baudrillard observed that, in art, things fall into nothing,
'- t J oi deconstruction.
:- and iconoclasts seem to posthumously win, not by burning the images, but rather
: ::e able to show the by multiplying them infinitely without worrying about their aesthetic reality and their
:-J-cible, in its logic, spiritual meaning.
.:;:roqates the general
'-: ::r. emerging of an
- :: inr-estigation on RErsnnNcns
.--:,. on the aesthetic
il sublime. G. Sertoli and E Miglietta, eds. Aesthetica
Burke E (:992). Inchiesta sul bello e
: :.ir:esthetic system of
edizioni, Palermo ltaly.
- :eine intentional, Deleuze G (rSS+). Dffirence and Repetition Columbia University Press, New York.
:,:: rpon these bases Deleuze G (zool). Francis Bacon: The Logic of Sensation, DW Smith, trans. University of
-::r¿i
: reality of the Minnesota Press, Minneapolis, MN.
. - : definition or a Franzini E (rq8+). L'estetica francese del'900. Analisi delle teorie. Unicopli, Milan.
:: ."i '.ti,1 or a body) we Herder IG (rSS+). Plastica. G Maragliano, ed. Aesthetica edizioni, Palermo Itaiy. Originally
::-, ¡:: i¡s relationship published in German (1778). For a transiation in English, see Herder JG (zooz) Sculpture:
,:r :i:i reYert to form, Some obseryations on shape and t'orm from pygmalion's creative dream. Untversity of
Chicago Press, Chicago.
-..... Thrs may be the Lee RW (tgZ). Ut pictura poesis. La teoria umanística della pittura. Sansoni, Florence Italy.
: :,{:l€e[th century: Merleau-Ponty M GgZil. L'occhio e 1o spirito. In F Fergnani, ed. Il corpo vissuto.Il Saggiatore,
.. ..:,is to a question Milan.
--:: :.-r:m, its sublime Merleau-Ponty M (1q68). Fenomenologia della percezio¡¡e. A Bonomi, ed. Il Saggiatore, Milan.
:.-. .¡nses and of the Originally published in French in 1945.
::-,: -,1.c meaning that Ricoeur P (rq8r). La metafora viva.Jaca Books, Milan'
,\, ::
hat is coml-nLrrl',-
, .,rlinq from dr-t::¿:
", hich is the ieehng - :
CHAPTER 7
SCULPTURE AND
TOUCH
FRANCESCA BACCI
Turs essay revolves around the crucialbut deceptively simple question of whether a
sculpture is an object designed to be touched. The fact that it belongs to the realm of
three dimensions implies the physical possibility of touch. But many works, especially
contemporaÍy, are designed precisely to exclude touch, as we shall see later in this
chapter. Bearing this issue in mind, I propose to explore it through the lens of cogni-
tive neuroscience, motivated not by the pretence to provide an exhaustive answer, but
rather by the possibility to add one further approach to this critical debate, while at the
same time exposing the often unacknowledged complexity of touch and its intercon-
nectedness with the other senses. I hope, through this paper, to raise the awareness that
an answer to this question has repercussions on such diverse fields as, among others,
museum policies and cultural heritage preservation, interpretation and critical analy-
sis of artworks, didactic and cognitive approaches to sculpture' and, ultimately, on our
cultural identity.
. rbid.
See Charles Spence, Chapter 4 this volume, 'The Multisensory Perception of Touch', pp. 85-ro6'
SCULPTURE AND TOUCH 135
rl rouched that goes and knowledge derived from the past (von Helmholtz 1866). More recently, following
I ;¡ t-ur brain. this line of enquiry, neuroscientist Richard Gregory (1997) proposed that:
r';¡:s ¿re experienced aS knowledge is necessary for vision because retinal images are inherently ambiguous (for
'. :::rr¡ the body during example for size, shape and distance of objects), and because many properties that are ütal
-i=r:--ore,
touch may be for behaviour cannot be signalled by the eyes, such as hardness and weight, hot or cold, edible
t ; ,:.elr to peripersOnal or poisonous.
:. : :¿l and peripersonal
Moreover, 'perceptions are hypotheses, predicting unsensed characteristics of
objects, and predictingin time, to compensate neural signalling delay' (Gregory
.I¿n brain constructs
ry97).Perception, especially vision, is not a passive window on the world, but requires
,'.s. irhich include:
intelligent problem-solving based on knowledge. Instructive in this regard is the expe-
s :- lhe space occupied rience of Gregory with a patient, Mr. Bradford, who acquired sight via corneal grafts
':-, :"i:hin the reach of after a lifetime of blindness. To his surprise, the man was immediately able to read
" '-: the time on a wall-clock, despite having never seen one before-but having touched
l.l..,rn,rut,
et al. zoor) the clock's hands for a lifetime. Technically, the patient showed 'cross-modal transfer'
from touch to vision, so he was able to solve the problem ofthe meaning ofwhat he was
ta:
seeing. Gregory Qoo4) wrote:
-: :;rs located beyond
'-. '." be reached with- Vision was generally thought to be separate from the other senses and is still mainly studied
.: i;::se. then, that the in isolation; yet Bradford showed that exploratory touch - as well no doubt as taste, sound
.. :i-,::: and other sensory experiences - gives richness and meaning to retinal images. For optical
those in extrap-
images are but ghosts, materialized into objects by perceptual experience of the non-visual
properties of things.
-:-r-. :nd Spence zoo4)
These modern observations seem to partially confirm a very old hypothesis, dating
:.' a;: epistemological
from the thirteenth century. What Aristotle had called sensus communiswas the part of
--, -a;quiied percept. the psyche responsible for binding the inputs ofthe individual sense organs into a coher-
,: sr¿ht, based at least
ent and intelligible representation. According to the philosopher and theologian Thomas
i :¡ educate sight or Aquinas, the fundamental basis for the sensus communis is to be found in the sense of
:!rinq the history of touch, as he wrote in his commentaryto Aristotle's De anima (liber III, lect. 3. 6oz):
- ,,,' Bre\\.ster showed
:ie haptic sensations fTouch] is the first and in a way the root and foundation of all senses . . . This power is attrib-
r.n. a century later, uted to the sense of touch not as a proper sense, but because it is the foundation of all senses
and the closest to the fontal root of all senses, which is common sense.
-:e intluence of one
.: :i. 1965; Rock and A second major change in the scientific view of touch has come from new models
of how the senses interact. The idea of one sense dominating over the others has been
; iheon., which have superseded in favour of the more accurate view that our perceptual system combines
r. screntists have also the information coming from different sensory modalities in one unified percept. In
,,brects upon match- everyday life, in fact, we receive correlated information from different senses about
ienses, in particular the same object. 'An extensive body of empirical research now demonstrates that
. ','¡n Helmholtz had visual, auditory and olfactory cues can all influence people's tactile perception ofthe
"s rom sensory data substance properties (such as texture) of haptically explored objects and surfaces."
One popular theory, which aims to predict how touch and vision combine in deter-
mining perception, comes from Ernst and Banks (zooz). According to the so-called
'maximum-likelihood' estimation account of sensory dominance, the brain weights withstand this phrs:::
the most precise sensory modality (i.e. the one which presents the least variance) more media raises manl c-u
heavily than inputs from those modality estimates showing more variance.3 Given hands. Many conter::;
that our knowledge of the world is formed through the integration of signals coming haptic contact, a r-ir::.
from our different senses, it of tactile access to
appears undeniable that the lack volumes about the ri:;
sculpture determines an impovetishment of our percept, and thus of our aesthetic deprived socieh'. It l.
experience of the artwork. are designed to in;,-,:
(such as Gonzales-T-.:
the artist's dead par::,
sage of time (Andr C:
consumed by the r::'-
Do roucH THE ART . . . AT LEAST guaranteed in the icr::::
in digital format.
WITH YOUR EYES Even those sculpl-.:
tion primariil. thro', :.
consider photograp::'.'
As we have seen in the previous section, haptic information constitutes effectively
of the tactile relatr¡:r.
a layer of meaning that should be considered as indispensable to the sculpture's p¡::
photographl- has
expressivity as colours to a painting. Unless otherwise indicated by the artist or by the (ú3- t:.
artist's atelier
placement and context for which the artwork has been conceived, it could be reason-
the best-known e\¿:::
ably assumed that a sculpture's texture and surface treatment, but also temperature,
This choice of su'¡
hardness, and weight-all of which to be ascertained haptically-are rich sources of
tures did not move :
sensory stimuli and are vehicles for information that is important for the sculpture's
the fact that diftlre-,
aesthetic apperception. There are indeed concerns regarding the physical preservation
masterfully with ti: -
of unique works of art from the past that determine the current restrictive museum
precisely this faith- :.
policies, but the issue of tactile access to artworks is complicated by social and political
the image, ditl-eren,-:
factors as well, as Fiona Candlin, among others, has theorized in several of her papers.a
sculpture has bee:: :.:
The time has come to seriously consider whether to privilege the physical integrity
srylistic approaches. .:
of artworks to the cost of their expressive power, or whether to find ways to guar-
photographs are. ::..:
antee the fullness of the experience of the aesthetic encounter with art. There are
eloquently expose ::.
increasing numbers of institutions, such as the Victoria and Albert Museum in
One interestinq ::¡-
London, which strive to allow the public to touch selected exhibits in order to gain a
distance betrveen ::c
haptic experience capable of enriching the understanding of the collection.5 It is time
provides the opp.-:.;
to ask ourselves whether what we are preserving is our cultural identity, embodied by
distance betlveen pe::
the object's integrity, or its monetary value. While sculptures have historically always
tactor, largell' dete:r-,:
been fondled and thus created in materials, such as bronze and marble, that could
riork. There are ph'-'
the decorum ofthe ::
3 Ibid.,pp.9t-92.
a On this topic
see Fiona Candlin's Touch and üe Limits of the Rational Museum (zoo8). Senses and
pornographic for :r:,
Society,3 b), zll-gr; Museums, modernity and the class politics of touching objects. In Helen Chatterjee, :nasterworks oltir .-
ed. Touch in Museums. Policy and Practice in Object Handling.London: Berg, zoo8; The dubious
inheritance of touch: art history and museum access (zoo6). lournal of Visual Cubure, s b), nl-5+.
t An issue that is worth briefly mentioning is that of the possibility of producing copies of original ' \lary Bergsteii-i. L, .
artworks for tactile apprehension. Would it feel the same as the original, to our hands, to touch a +¡;-gt. ,
:,:.)i¿tín,Z+(l') .
carefuliy crafted copy? Could we acquire haptic knowledge that would enrich our visual experience ',l:nson (1999),
a Scu1:.:,-,
of the original sculpture?
SCULPTURE AND TOUCH r37
r:¿. the brain weights withstand this physical approach, the modern use of alternative, often ephemeral,
e -east variance) more media raises many questions regarding the willingness of the sculpture to meet our
'i¡re r-ariance.3 Given hands. Many contemporary works entrust their survival to their prescribed lack of
-;,n oi signals coming haptic contact, a virginity that takes on an anthropological connotation and speaks
¡l of tactile access to volumes about the fragility, fear, mistrust, and isolation that characterises our touch-
th'¡s of our aesthetic deprived society. It is meaningful that numerous artists are presenting works that
are designed to incorporate their physical disintegration as a metaphor for death
(such as Gonzales-Torres, Portrait of Ross in L.A., made of sweets weighing as much as
the artist's dead partner, which the public is invited to eat) or for the inevitable pas-
sage of time (Andy Goldsworthy's installations made of ice, wood, or leaves, left to be
consumed by the natural elements). The survival of these sculptural installations is
.EAST guaranteed in the form ofvideo and photographic recordings, both printed and stored
in digital format.
Even those sculptures built to endure the test of time, however, come to our atten-
tion primarily through the eyes, most often in the form of photographs. One can then
:,- ]]stitutes effectively consider photography of sculpture as one of the many possible ways to tell the story
:-; :o the scuipture's
of the tactile relationship between sculpture and its perceiver. Since its beginnings,
photography has portrayed sculptures as a favourite subject. Louis Daguerre's The
:-, :he artist or by the
,'t. r: could be reason- ortist's etelier (ú37) and Hippolite Bayard's Plaster casts (c.t8:q) (Fig.7.r) are probably
the best-known examples of this early production.
:'-: aiso temperature,
'-¡re rich sources of This choice of subject was not only motivated by the obvious reason that sculp-
::i i¡r th€ sculpture's tures did not move (a necessity determined by the long exposure times) but also by
the fact that different textures combined with the whiteness of the marble played
:ri:ical preservation
i: :eslrictive museum masterfully with the light, providing an astonishingly realistic representation. It is
precisely this faithful rendering of tactile values that conferred a life-like quality to
:', sr-'cial and political
i-'..:¡¿l of her papers.a
the image, differentiating it from its painted alternatives. In the last two centuries,
sculpture has been depicted through the camera according to numerous and different
-:e phvsical integrity
stylistic approaches, each indicative ofa specific critical, social and cultural attitude-
: -- ind ways to guar-
,,nth art. There are photographs are, after all,'woven in to the fabric of their own time'.6 These images
:
I lbert }fuseum in eloquently expose the different ways in which these artefacts have been perused.
-r
:iis in order to gain a One interesting fact is that, in photography, the image meaningfully registers the
distance between the photographer's eye (hence his/her body) and the object. This
: ;¡riection.; It is time
¿.ntit\-, embodied by provides the opportunity of reading photography of sculpture according to the
distance between perceiver and perceived. This distance, far from being an accidental
ir e ¡ristorically always
.J rrrarble, that could factor, largely determines our physical, hence intellectual, engagement with the art-
work. There are photographers, such as David Finn for example, who have rejected
the decorum of the arm's length view, pursuing inquisitive close-ups that are almost
pornographic for their exaggerated anatomical exactitude. His shots of Renaissance
-,".*-r: :oo8). Senses and
: =--.. In Helen Chatterjee, masterworks offer an intimate vision of the sculptures photographed, one that
. ,:: The dubious
,-.,:.,'t.5 |1),87 54.
,;-;ri copies of original 6 Mary Bergstein, Lonely Aphrodites: On the Documentary Photography of Sculpture. The Art
-: r;nds, to touch a Bulletin, T4b) 475-98.On photography of sculpture, see also the excellent book by Geraldine
- ::. .- .-r¡ r isual experience
Johnson $999), Sculpture and Photography: Ent isioning the Third Dimension. Cambridge University
Press.
138 FRANCESCA BACCI
Fig,7.2 A stereoscc:
Invented br'\\-h;.:
cally three-dimenslc:
Fig. 7.1 Hippolite Bayard, Plaster casfs, c.1 839. (source: http:// obtained by simulta,-,e
com mons.wikimed ia.org/) slightly differins ari,
the views of the trrtr :.,
presupposes emotion, possession, and sensual intimacy with the object, rather than because binocula¡;:¡.
historical perspective and documentation. At this distance all of the background especially in seeing :l;
disappears, and with it the context, thus the sculpture loses its temporal and mate- tion about the thre:-;
rial dimension, almost becoming liüng flesh palpitating in an eternal present, read- photographers ende:'
ily available for the observer's consumption.T Moreover, as detailed earlier, even the of depth were mrr:. -
cognitive relationship between perceiver and sculpture is quite different accord- didactic purposes l:t:
ing to the space occupied by the latter in the spatial map of the former (whether the struggle towards ob,.::
sculpture is in a person's peripersonal space or beyond it). In other words, objects that rng a tactile sense rri .
are located at a distance shorter or equivalent to arm's reach offer the characteristic tool enriched the erl¿
of being potentially touchable, and this alone makes them different to perceive than shape, but it also ;:.
other objects. visually tantalizins ¡:
One extremely popular photographic device that evoked the presence of an object space that, frustratir ¿l
in peripersonal space is the stereoscope (Fig. 7.2). Some monumelt:-
:o almost enforce the
:equires haptic veril.;.
7 The use ofzoom lenses can simulatenon-existent physical closeness between photographer
a
and object, but the observer of the photograph will sti1l perceive this physical closeness as if it was These works are sc,
!r\
/
unless one gathers a multitude of impressions by walking around and inside of them. act of creating scul:':ir:
This characteristic transforms the environment in which the sculpture is placed into a are most often the p:,-
qualitatively new space, one in which the vestibular discomfort is compensated for by the complex process .
the tactile sensation of massiveness and stability. As Alva Noé (zooo) aptly observed, the shaping will to r:¡
Serra's sculpture: bodily feedback th ro r: :l
enables us to catch ourselves in the act of perceiving and can allow us thus to catch hold
ofthe gesture there is .
of the fact that experience is not a passive interior state, but a mode of active engagement in the fourth centur, FL.
with the world (...) When one first encounters a piece, one is liable to be struck by the scale touch reaches in m¿r: '-l-.:
and the visually inscrutable orientations and distributions of weight. One not only notices other senses we fall t <l-:.,
these qualities, but one is disturbed by them. One puzzles: what stops these giants from falling
species in exactness r¡: c-,
over? To wander around or through a piece such as this can cause a loss ofbalance. In this way
the works make us reflect on how we feel, perceptually, in their presence.
In light of these observations, we could rephrase our curatorial question (whether If this tactile an¿ .
a sculpture is an object for touching or not) by asking whether sculpture provides
sculptor (that is, b., ;.:r
rvith clay, plaster, or :.
the affordance of 'touchability', in other words whether the possibility of being
touched is implicit in its sculpture-ness. The idea of 'affordance' was pioneered by action that the artist r., L,
When in use, a tool is a sort of extension of the hand, almost an attachment to it or a part of
the user's own body, and thus is no longer a part of the environment of the user. (...) This : She was a generou. .:*
capacity to attach something to the body suggests that the boundary between the animal and .lstruction manual for -,'c *-
the environment is not fixed at the surface of the skin but can shift. More generally it suggests i'ork, 1939.
SCULPTURE AND TOUCH r4L
that the absolute duality of 'objective' and 'subjective' is faise. When we consider the affor-
dances of things, we escape this philosophical dichotomy.
(Gibson ry79,p.4r)
-: ::+rons.wikimedia.org/)
Cnn¡.rrvg AND ExPLoRAToRY ToucH
In addition to the role oftouch in the perception and appreciation and sculpture, the
rl and inside of them. act of creating sculpture is, itself, deeply connected with touch. Sculptures, in fact,
iprure is placed into a are most often the product of an act of touch directed by the Kunstwollen through
: j .{-rinpensated for by the complex process of creation. In the act of sculpting, the gesture that imposes
r , rr ü aptly observed, the shaping will to the unformed matter is both performing an action and receiving
bodily feedback through the touch. In the reaction of the matter to the strength
sense of
of the gesture there is a great amount of information. As Aristotle had already noticed
' :' :hus to catch hold in the fourth century BC:
:. ,:': actÑ'e engagement
, -.¿ st¡uck by the scale touch reaches in man the maximum of discriminative accuracy. While in respect of all the
i. ¡ne not only notices other senses we fall below many species of animals, in respect of touch we far excel all other
-l:e se grants from falling species in exactness of discrimination. That is why man is the most intelligent of all animals.
In this way
(De Anima. Liber II, lect 9)
::.-1,0U""...
i:.r question (whether If this tactile and sensory-motor information is received by an experienced
;r sculpture provides sculptor (that is, by an individual with an extensive knowledge of past interaction
. :ossibility of being with clay, plaster, or stone), then it üll sensibly modify and regulate every further
r.. \\as pioneered by action that the artist will undertake upon the medium. It is this accumulation of hand
experiences with the material that, ultimately, influences greatlythe artist's style.
r f ' action possibilities'
¡¡': the individual. For This seldom-considered phenomenon appears evident to practising sculptors
r¿ head (but not if the
who work in traditional media such as wood, clay or marble, as American sculp-
;:e closely linked: tor Malvina Hoffman (t}Sl-tg66) exemplarily reported.8 One of the tenets of her
:---nent to it or a part of
¡:-- ¡i the user. (...) This E She was a generous sculpture teacher, pupil ofRodin, and author ofa useful and detailed
t't¡reen the animal and instruction manual for young artists, titled Sculpture inside and orf. W.W. Norton & Company, New
i : r: :enerally it suggests York, r939.
r42 FRANCESCA BACCI
teaching method was the insistence with the students on extensive hours of practice
u, ,r...rrury to achieve mastery over the medium, to 'train their hands to obey their
minds' (Hoffman t939,p.82). This was encouraged not so that the students would
become able to subjugate the matter to their own will, but in order to allow the
expression of one's own style to flow freely, unobstructed by the incompetence of the
hands. However, such style-and here comes the most surprising point-is largely
determined by the medium of sculpture itself. She wrote:
It would be well for students to understand why they should carve directly in stone while
they are studying the first principles of sculpture. (...) The resistance of the stone controls
their minds; the appearance of forms as they emerge in the stone gives the caryer a new
demonstration of why the stone demands a solid form. Details and personal attributes
are
automatically subordinated to the basic needs of the materiai. Training that is limited to mod-
He
elling in clay, for casting into bronze, is liable to lead the student into dangerous channels.
will f,robabiy become intrigued in the details of drapery, expression of the face, gestures of
action, etc. The plastic consistency of clay is a temptation to model unessentials' ("') The
discipline which stone and marble impose upon the artist, however, is unquestionably the
best ieacher that a student could have. Stone is a taskmaster that is unprejudiced,
that presents
the same obstacles to every student, old or young' archaic or modern' that commands undi-
vided attention and tenacity of purpose, strong steady hands, and infinite patience.
(Hoffman 1939, PP. r57-B)
rary sculptors as exemplars of how these scientific characteristics of touch are inter- structed tactile bc'::*
twined with current artistic researches and practices. The work of sculptor Rosalyn It¿rlian pedagoeue i,l-
Driscoll ties into the history of sculpture while, at the same time, posing entirely new :rescribed intensr.,.¿ ¡
questions.e Conceived specifically for tactile apprehension, these works recreate the and arithn;:
"-riting,
original haptic, and somewhat intimate, relationship between perceiver and sculpture. ;hildren perfbrmr;r:
fusfring Hoffman's thoughts to extreme consequences, the final visual appearance of :heir sense of tou¡:-. -
Driscoll's sculptures is determined by the tactile sensation that the artist intends to : blindfold to h¡st:-.
communicate, and it is a direct consequence of the material's specific haptic require- - his simple strat.{, :
ments. In so doing, these sculptures question the modern limits posed to our hands :¿thered throusir :- *
by our cultural institutions for preservation purposes. They also open up the rare er"::'
,,.r,r (zoo8), for
possibility of expanding the discerning abilities of the sense of touch through the ,-l- apparatus and :s ,
awareness of one's haptic exploration, either informed or uninformed by the sense
of sight. Some of Driscoll's sculptures are so big to require continual bodily adjust- F.T. trlarinettl -::-
ments in order to be investigated, adding an intimate dimension of proprioceptive : -:rr. eds. La Futti.::..; :
'- .:nd Design, Phi,e¡.-,
e See her essay in this volume, 'Aesthetic Touch' (Chapter 5)' l'titl.
SCULPTURE AND TOUCH r43
book) workprecise.-,
touch (Fig.7.6).
Made while blin,i
human head, and ir,
The artist could no, I
sively his sense oi :
resulting work is th:
concerned with the :
representation of t¡,
chart these stages i:t I
colour of the pen th:
his sight and vier.-::::
first. Since he cou,c i-
placed at the top tri -:
whichhewas tracir::
reference point t¡trr-
composing the ob:¡;
Fig. 7.5 Michael Petry, Party Number 1,2OO7. There is often a disiu- ¡
(Photograph: courtesy of the artist and Westbrook Gallery, London.) them and the phvs:r.-
confine me to rvh¿¡ i :-
The material encasing the copper and silver elements is rawhide. In touching it,
we feel a skin that once could have felt us. Its translucent appearance isa visual reminder
of its permeable and containing function-our body is a vessel encasing vessels (called
'lota' in Urdu and Hindi). Inside, a series of connecting tubes represents the pulsat-
ing motion and endless circulation of fluids necessary for life. Moreover, the chro-
matic contrast between these different materials takes on a symbolic meaning: 'inside
us there are things that shine unexpectedly'says the arrist,'Lotawill change over time
as people touch the metals, revealing the small, cumulative, and unconscious effects
of touch that may not be apparent at this time'.'.
Michael Petry's sculpture Party I'{umber one(Fig.7.5), which is meantto be handled,
addresses the sensual aspect of touching. The sexual connotations of the sculpture,
whose shape is evocative of human anatomical parts, suggest sensual manipulation
of a lover's body, thus providing a reminder of the importance of touch as a primary
physical and psychological site of non-verbal communication. In the artist's words,
'a seductively polished slice of Cedar of Lebanon is hanging in the space. The velvety
smooth, horizontal piece ofwood has many perfectly round holes cut into it that tempt
the viewer to touch it or other viewers through it in the darkly lit space'. if we imagine
stroking the wooden sculpture while looking at it or observing someone else perform-
ing this operation, this activity is liable to activate sensory-motor neuron signals in our Fig. 7.6 Claude Hea.
brains via our fondling eyes.
claude Heath's blindfold drawing series (some of which were presented in the
exhibition 'Art and the Senses' concurrently with the conference that inspired this
'r Andrew Patrizio _.
Heath. In A Kingston. e:.
" Personal communication with the author. London, October 31, 2oog.
n-riting, available onli:¡ :
SCULPTURE AND TOUCH r45
book) work precisely on exploring the gap and the sensory overlaps between sight and
touch (Fig.7.6).
Made while blindfolded, the artist had one hand engaged in touching a cast of a
human head, and the other simultaneously busy tracing this touch-journey on paper.
The artist could not look at the sculpture nor at the paper while working, using exclu-
sively his sense of touch, proprioception, and 'body-centred' maps to draw. The
resulting work is the graphic translation of an exploratory haptic strategy that is not
concerned with the final aesthetic result, but rather with the development of the mental
representation of the head as formed through the tactily-acquired information. To
chart these stages in time, in some of the drawing from this series, Heath changed the
colour of the pen that he was using in order to be able to understand, having regained
his sight and viewing his work for the first time, which part of the picture he made
first. Since he could not rely on the sense of sight, Heath used a small lump of BIu-Tak
placed at the top ofthe shape to feel and another one on the upper part ofthe sheet on
which he was tracing (which leaves a visible white spot on the drawing). This served as a
reference point from which he could feel the relative distances of the different elements
composing the object felt. Heath said:
There is often a disjunction between the images being shaped on the paper, the way I visualise
them and the physical reality ofthe object itself. The disparity is healthy because it does not
:::,'::. Ga ery, London.)
confine me to what I imagine, expect or desire to see.'3
) ::-r.ant to be handled,
.-,- rs of the sculpture,
.-rrual manipulation
-,: :.ruch as a primary
i: the artist's words,
:le space. The velvety
: s :u: i¡to it that tempt
- !p¿re . If we imagine
,r nreone else perform-
Fig. 7.6 Claude Heath, Plaster Cast of Head and Drawing 138, 1995.
f :riuron signals in our (Photograph : Peter Dent.)
This methodology of work betrays the artist's scientific intention. He empirically the paper. This tec:
recorded his perceptions in order to better understand and become increasingly allow a free-flol-ins
more sensitive to the acquisition of knowledge via haptic exploration, In this sense, used while sketchrn
these drawings are to be considered test results, maps charting the three-dimensional since the result is th:
movements of the hand on the flat surface of the sheet.
the hand goes rrhei:
An object seen always has a profile, a visible edge, the line of its shape as it stands out silhou- tated or loses a lin--
etted against the world behind it. An object felt has any number of contours running over unforeseen srvellinq..
its surface, but none of them has the special status of this profile. (...) Drawing by eye always each section had it. ¡'-
deais with these restricted visibilities. But drawing from touch has no such limits. It has no
prejudice in favour of near side over far side, or in front of over behind. It deals indiscrimi- Instead, he can b
nately with the whole tangible surface. On paper, al1 contours superimpose in confusion. placed on the paper
(Lubbock zooz) evidently requirei ¿
des Photographies :
The observer's difficulties in deciphering throughout the unfolding of this journey, to a precise spati¿- :
which presents some areas of opacity, are similar to those experienced by the scientist, of carefully contrc,L
when trying to understand how the different spatial maps interact in the human brain rather than the t¡e-
in order to give a unitary percept. with gouache anc i,
This use of drawing as an instrument to capture tactility has an illustrious prec- outlined, and tra c.ri
edent in the work of the nineteenth-century national champion of French sculpture, syrnbolize the ct¡n-.:
Auguste Rodin (r84o-r9r7). His working method was indeed different from that taught explored in this ¡,a;
in fine arts schools worldwide. In fact, he used to keep his eyes fixed on the model and
draw without looking at the paper and at the pencil that was leaving marks on it. He
never lifted his pencil, once he had begun drawing, until he had completed the outline
of the figure in one continuous sweep.1l This technique allowed Rodin to haptically
understand the shape that he would then render three-dimensionally in sculpture,
almost as if the pencil, moving on the paper's surface, was 'a surrogate finger passing
along the subject's contours' (Elsen zoo3, p. 6r:). Rodin explained this procedure in a
conversation with his secretary and friend Ludovici:
I have to incorporate the lines of the human body, and they must become part of myself,
deeply seated in my instincts. I must become permeated with the secrets of all its contours, all
the masses that it presents to the eye. I must feei them at the end of my fingers. All this must
flow naturally from my eye to my hand. (...) Not a thought about the technical problem of
representing it on paper could be allowed to arrest the flow of my feelings about it, from my
eye to my hand. The moment I drop my eyes that flow stops.,5
The drawing was a result of the hand motion rather than an attempt to perform a
üsually guided exercise. In 19o6, Rodin made numerous works with this technique,
portraying a group of Cambodian dancers who performed in Paris and Marseille. This
is an ideal series to study, since both the drawings and the photographs of the master
sketching are extant. In the latter, one can see that Rodin held the pencil in the correct
way, that is between thumb and index finger, from the back, with his hand lifted from
'a This working method is particularly well described in Anthony M. Ludovici, personal 'ó This quote ccl:-l-.
Reminiscences of Auguste Rodin,pp.t35-4o. John Murray, London, 1926. Ludovici served as Rodin's Les Dessins de Rod::. :
private secretary, and was an eyewitness of this methodology of drawing.
'- This picture car -.:
" Anthony M. Ludovici, Personal Reminiscences of Auguste Rodin, cit., pp. r3g-9. P nr adíse. Sculpto r' : -;'.-,'
SCULPTURE AND TOUCH r47
ai:trn. H€ empirically the paper. This technique is taught in art academies as the most convenient way to
become increasingly allow a free-flowing motion of the pencil when the work is visually controlled-if it is
,,r:iion. In this sense, used while sketching without looking, though, this fluidity becomes an impairment,
:e rhree-dimensional since the result is that:
the hand goes where it will: often the pencil goes off the page; the drawing is thus decapi-
=. l: stands out silhou- tated or loses a limb by amputation (...) It contains naturally some excessive deformations,
. - titrrrs running over unforeseen swellings, but (...) if the relation of proportions is destroyed, on the other hand,
I r¡iiinq bv eye always each section had its contours and the cursive schematic indication of its modelling.'o
- .u:h limits. It has no
Instead, he can be seen in another photo using the aid of his little finger, firmly
-::. i: deals indiscrimi-
-: -.: :¡ confusion. placed on the paper, when adding some visually guided strokes on a drawing, which
evidently required a finest degree ofprecision and motor control (inv.Ph.6398, Cabinet
Lubbock zooz)
des Photographies du Musée Rodin, Paris)." This way of anchoring the hand motion
:,;:ng of this journey, to a precise spatial point recalls Claude Heath's use of Blu-Tak, and expresses the need
.:.-d b\- the scientist, of carefully controlling the visual consistency and placement of the different marks,
;: Ln ihe human brain rather than the feel of producing them. Often, then, Rodin would tint his drawings
with gouache and watercolours, in order to extract the salient features of the figure
.s ¿n illustrious prec- outlined, and to disambiguate the overlapping and misplaced lines. These drawings
. -- i French sculpture, syrnbolize the complex relationship and interconnectedness between the two senses
::er:: Íom that taught explored in this paper.
r-¡i t¡:t the model and
1i',:lE narks on it. He
:::::pie ted the outline
: L¡din to haptically
.- :,:la-Llr- in sculpture,
--:riate hnger passing
.l :ris procedure in a
¿Íernpt to perform a
, ,,',ith this technique,
'-¡ ¿nd
\larseille. This
::raphs of the master
=
pencil in the correct
r hrs hand lifted from
P¿rsonal '6 This quote comes from the description of another eye-witness of Rodin's work. Clément-Janin,
sen-ed as Rodin's Les Dessins de Rodin, in Les Maítres du Dessin,pp.285-7. Paris, r5 October r9o3.
'r This picture can be seen on the back cover of the book by Claude Judrin, Rodin: Hell and
Paradise. Sculptor's drawings. Paris: Musée Rodin, zooz.
748 FRANCESCA BACCI
REpnRENcns
Aquinas T (rSfS). In Aristotelis Librum de anima III.Lecf.3. 602 (AM Pirotta ed.). Turin.
Elsen AE (zoq) . Rodin's Art: The Rodin Collection of Iris 6 B. Gerald Cantor Center of Visual
Arts at Stanford Unitersity. Oxford University Press, New York.
Ernst, MO and Banks MS (zooz). Humans integrate visual and haptic information in a statisti-
cally optimal fashion. Nature,4t5, 429-33.
Gibson IJ GSZZ) .The theory of affordances. In R Shaw and I Bransford, eds. Perceiting, Actíng,
and Knowing,pp. 67-82. Lawrence Eribaum, Hillsdale, NJ.
Gibson II Gglg) . The Ecological Approach to Visual Perception. Horghton Mifflin, Boston, MA.
Gregory RL (r997). Knowledge in perception and illusi on. Philosophical Transactions: Biologícal
Sciences, 352, r72t-7. TT
Gregory RL (zoo4). The blind leading the sighted. An eye-opening experience of the wonders
of perception. Nature, 43o, 836.
Hoffman M (rS:S). Sculpture Inside and Out.W. W. Norton & Company, New York.
Holmes NP and Spence C (zoo4). The body schema and the multisensory representation(s) of
peripersonal sp ace. C o gnitit e P r o c e s s in g, 5Q), 9 4-ro 5.
Lubbock T (zooz). Don't look now. The Independent, S October.
Montessori M (rqog). Il Metodo della Pedagogia ScientiJica applicato all'educazione infantile
nelle Case dei Bambini. Lapi, Cittá di Castello. (First published in English as The Montessori
Method,:-9tz.)
Noé A (zooo). Experience and experiment in art. Journal of Consciousness Studies, Z(B-S),
123-35.
Vaishnavi S, Calhoun J, and Chatterjee A (zoor). Binding personal and peripersonal space:
evidence f¡om tactile extinction. J ournal of Cognititte Neuroscience, r3(z), r8r-9.
von Helmholtz H (1866h962). Concerning the perceptions in general. In Treatise on WupN we sa\'\\-e ri
physiological optics, voI. III, 3'd edn. ( JPC Southall, trans. t9z5 Optical Society of America viscerally, we me::i
Section z6). Reprinted Dover, New York, 1962. tinctly tactile rela:i,-- -
and response to Ii.
cinema as an li iti',:-;
connection, ratire: l.
cinema as a pure-'.
experience of ciner::
one's percePtion r-- I
viewers and films. -'
shaped by a tacti e :
objects, subjects iir
The Quav Brorir
palpable corn€ctit-:
but is rarely madi :,
spaces and mor-e :¡.:
the film is an obie¡.
emerge in the flesh"",
:. :J:. Perceiving,
:eripersonal space:
. t.it_g.
::--. In Treatise on WHeN we say we are moved by a film, that it touches us' or that we respond to it
. -{¡¡ier,r' o.f America viscerally, we mean it in a more than metaphorical sense. These claims imply a dis-
tinctly tactile relationship between film and viewer that is a key factor in our attraction
and response to the movies. Exploring cinema's tactility opens up the possibility of
cinema as an intimafe experience and of our relationship with the cinema as a close
connection, rather than as a distant experience of obserwation, which the notion of
cinema as a purely visual medium plesumes. Vision and touch are married in the
experience of cinema, as are mind and body. To perceive the world and to express
one's perception of that world are not solely cognitive or emotional acts taken up by
viewers and films, but always already embodied ones that are enabled, inflected, and
shaped by a tactile engagement with and orientation toward others (things, bodies,
objects, subjects) in the world.
The euay Brothers' Street of Crocodiles (1986) dramatizes and literalizes the
palpable connection between film and spectator that exists in all cinematic experiences
Lut is rarely made so explicit. With its intense focus on material textures, miniature
spaces and movements, and the ambivalent temporality of
'stop-motion' animation,
the fi1m is an object lesson of sorts, revealing the way films' meaning and significance
emérge in the fleshy, muscular, and visceral encounter between films and viewers.
t This chapter is expanded on in The Tactile Eye: Touch and the Cinematic Experience@2}}9 by the
regents of the University of California, published by the University of California Press. Selections
from this book are reprinted with permission from the University of California Press.
r5o JENNIFER M. BARKER
The film begins with a prelude, entitled 'The Wooden Oesophagus', that blends particular styles of s *l
live-action cinematography with animation. In it, a museum caretaker enters a silent, portment (swar.ine. I
cavernous auditorium, its chairs empty and its stage populated with apparently pulsing, skipping a n,
lifeless machinery. He whistles as he steps on to the stage, carrying a stage light. He tor's body and the *i:
sets the light on the floor next to one particular machine and walks out of the frame. the viscera, which ¿:¿
A series of camera movements follows, in which the camera moves quickly down rality, respectir.elr-.
the surface of this machine, giving us glimpses (in black and white) of bits and pieces As a material n-ro,l¡
of the contraption: an eyepiece, a tiny crank, a small porcelain bowl of screws. itly to a single orgar: ,
The man returns and peers through the viewfinder. Another jerky camera move- asphenomenoloeisr 1,,
ment follows that takes us down into the machine's inner workings and then rapidly experience by a1l iis .
back up (this time in colour rather than black and white). The caretaker adjusts a certain tlpical stru;r *
magnifying glass that is attached to the machine and perched above a detailed map. He tactility occltrs rlot ,--:
stands upright and, after a pause, spits slowly into the innards of the machine. There spectator's bodv all
is a cut to a colour image of the machine's interior, where the saliva falls slowly from that it registers mo-":
the top of the frame and sets into frantic motion a series of gears, pulleys, and uniden- system and the inn.:
tifiable mechanisms. Tension, balance. e:¡
Amidst the mechanical whirrings, a little man-like puppet appears who will be though none mani:e ..
our protagonist. His hand is attached by a thread to a pulley hanging above a door- Cinematic tactrltr..
way, and he tugs at the thread but is unable to move. The film cuts to a black-and- cinema that the hui:.
white image of the caretaker's hand reaching slowly into an opening in the side of the face of the bodl'; hn..
machine, holding a pair of scissors. As the puppet-protagonist continues to pull his tendons, and bones i:
arm to no avail, the caretaker reaches his scissors further into the machine and snips. the murky recesses ,- -
Inside the machine, the scissors appear enormous now and in colour, emerging from receive, respond trr. ::
the top of the frame to cut the thread holding the protagonist. A quick camera move- toward the world ¿n¡
ment tracks up to the pulley as it spins out of control. Finally the protagonist moves played out across iis
jerkily out of the frame, free to wander the murky and maze-like space of this strange caress of shimmeiii:i
mechanical inner world. thetically, throush tr,
The tactile engagement between the caretaker and the contraption and its resi- human and mecha:::-
dent puppet is, in some ways, analogous to the relationship between spectator and rush through a proi;;'
filmr. The strange machine in this prelude invites the involvement of the caretaker's The skin, the mus:
body; indeed, it cannot move without him. He brings the machine to life not just they are stretched 'rc-,
by looking but by committing his entire body to its spectacle: he steps up to it, leans nomenological sign:r
close, bends over the eyepiece, reaches into it, manoeuvres his fingers to cut the strings terrain, but their bc u
of the pulley, and finally spits into its inner workings. His preliminary contact with constantly bleed. r-i : :,
the apparatus at the surface of the body (both his and the machine's body) parallels a dimension into the :::
contact that occurs even deeper within him and the machine, a contact involving the This essay rvili u¡
muscular, mechanicalworkings and internal rh¡hms of each. they apply to botir ., -
Likewise, the cinematic experience involves a mutual and distinctly tactile rela- of skin that we hnú r"
tionship between film and spectator that involves all dimensions of the body at once: perception:'Hapt:; -,-
surface, middle, and depth. Both films and spectators perform meaningful, embod- plunge into depth. r -:
ied acts of perception and expression that, upon closer study, reveal corresponding p.25). The muscui.:
tactile structures. These acts and their underlying structures manifest themselves as
' For an elaboratio: -
' For an extended discussion of the role of tactility in the cinematic experience, see Jennifer M. nodes ofperception a:- .
Barker, (zoo9). uier,ver, see Sobchack l::,
TOUCH AND THE CINEMATIC EXPERIENCE T51
:r:hagus', that blends particular styles of surface contact (caressing, scraping, pricking, piercing, etc.), com-
:e:aker enters a silent, portment (swaying, tugging, lurching, leaning, etc.), and internal rh¡hms (flowing,
::¿d rvith apparently pulsing, skipping a beat, etc.). They occur in and across three regions of the specta-
ir:rq a stage light. He tor's body and the film itself, regions that I refer to as the skin, the musculature, and
¡.ks out of the frame. the viscera, which are related (but not limited) to materiality, spatiality, and tempo-
:r-.r,, es quickly down rality, respectively.
,:. of bits and pieces As a material mode of perception and expression, touch need not be linked explic-
of scrervs.
: '*.,'- itly to a single organ such as the skin. It is enacted and felt throughout the body, for,
: ':¡h,- camera move- as phenomenologist Maurice Merleau-Pontywrote, 'the body is borne towards tactile
::Es and then rapidly experience by all its surfaces and all its organs simultaneously, and carries with it a
r¡ ¡¿retaker adjusts a certain tlpical structure of the tactile "world"' (Merleau-Po nly 7962,p.317). Cinematic
i'; ¿ Cetailed map. He tactility occurs not only at the skin or the screen, but traverses all the organs ofthe
- ::e machine. There spectator's body and the film's body. As J.J. Gibson wrote, 'vision is kinaesthetic in
ir','a iails slowly from that it registers movements of the body just as much as does the muscle-joint-skin
. :uilels. and uniden- system and the inner-ear system' (Gibson t979, P. 183, cited in Rutherford zooz).
Tension, balance, energy, inertia, languor, velocity, rh¡hm-all of these are'tactile',
¡:pears rvho will be though none manifests itself solely, or even primarily, at the surface of the body.
::1{i:rq above a dOOr- Cinematic tactility, then, is not just 'skin-deep'. It is a general engagement with
--l:i io a black-and- cinema that the human body enacts in particular ways: haptically, at the tender sur-
-::rg i:r the side of the face of the body; kinaesthetically and muscularly, in the middle dimension of muscles,
:,- rt:rues to pull his tendons, and bones that reach toward and through cinematic space; and viscerally, in
: ;::achine and snips. the murky recesses of the body, where heart, lungs, pulsing fluids' and firing synapses
, -,-, lr. emerging from receive, respond to, and re-enact the rh¡hms of cinema. The film's body also adopts
; *:.k camera move- toward the world and spectator a tactile 'attitude' of intimacy and reciprocity that is
ri protagonist moves played out across its non-human body: haptically, at the screen's surface, with the
: s:¿.e ofthis strange caress of shimmering nitrate and the scratch of dust and fibre on celluloid; kinaes-
thetically, through the contours of on- and off-screen space and of the bodies (both
human and mechanical) that inhabit or escape them; and viscerally, with the film's
rush through a projector's gate and the'breathing' of lenses.'
The skin, the musculature, and the viscera are not meant here metaphorically, but
they are stretched beyond their literal, biological meanings to encompass their phe-
nomenological significance. These categories helpfully lead us through a complicated
terrain, but their boundaries are pliable and permeable. Sensations and behaviours
constantly bleed, vibrate, dissolve, cut, infect, meander, or muscle their way from one
dimension into the next.
This essay will use Street of Crocodiles to elucidate those three categories as
they apply to both viewer's and film's embodied behaviour. It is in the dimension
::-;:rnctlv tactile rela- of skin that we find what Laura Marks has termed the 'haptic' qualities of cinematic
' ;: the body at once: perception: 'Haptic looking tends to rest on the surface of its object rather than to
:reaningful, embod- plunge into depth, not to distinguish form so much as to discern texiure' (Marks 1999,
:ei'e¿1 corresponding p.25). The musculature involves things like gripping, grasping, holding, clenching,
"r:ii!st themselves as
, For an elaboration of the 'film's body' as a concretely embodied entity whose visual and visible
:.':.. see Jennifer M. modes of perception and expression place it in a relationship of reciprocity and reversibility with its
viewer, see Sobchack (rqgz).
I52 JENNIFER M. BARKER
Street of Crocott:.¿
and what Michael Taussig describes as a kind of 'sentient contact that is another mode
grasping where the touch falters' (Taussig 1993, p.3o), while the novella of the sanr¡
of seeing, the gaze
tactility. In Schulz . .
turbulent, pulsing rh¡hms of the viscera constitute yet another register of tactile per-
ception and expression. Can you understa::,' --
papier máché, for d ...
such, for its flufñn¡::
its resistance, its ¡,u:r.
its heary effort, its :-'
S rrN
The scene in th. .
pet-protagonist::t.:
Street of Crocodiles obscures its objects in a way that makes vision difficult and porcelain dolls. -, --
invítes the viewer to feelrather than see the film. Objects and images are mysterious fondle their gues:. :'.
and murky, veiled by layers of dust and detritus and kept just out of visual range by bewildered-1ooHn.
camera movements that merely glance over their objects before the next shot. Film and replace ltit hs¿l r',,t
viewer press against each other in a skin-to-skin contact: rather than lookwith a precise from the empt\' :.:
and penetrat iÁggaze,the eye skitters along textures that 'make sense' only to the touch' At one point. ::..
Texture is something viewer and film engage in mutually' rather than something pre- sitely detailed nr: :
sented byfhe ñ\m fo a passive and anonyrnous viewer. It is in the meeting of the film's meat that he lar-s ¡., .
skin and the viewer's skin that this film becomes meaningful' the meat, u'hose ::-
'By engag-
The film thus invokes haptic visuality, in the sense that Marks describes: up the surface o: ::,
ingwithanobjectinahapticway, Icometothesurfaceof mysell (Marks ry99,p.t84). the tissue patte:: .
In letting ovf gaze wander over the surface of the image , we do come to the surface protagonist. the ::--
of ourselves, feeling ourselves more keenly in the touch of our skin against the film's coarse grev hair,
skin. This is a touch by which we and the film bring each other literally and sensuously The tailor in-,:r.,
into being. Indeed, as Marks writes, 'haptic images are erotic in that they construct an 5e|ond the tai't,: ¡
intersubjective relationship between beholder and image' (Marks 1999, p. r83).' rough tu-eed crrf,:. ::
If we take 'skin' to mean the literal fleshy covering of a human or animal body, then ..f the g1ass, anLr':..:
to say a film has a skin would be quite a stretch. But if, as Merleau-Ponty has said of ,: light bulb th¿i -. ,:
touch, 'skin' also denotes a general style of being in the world, and if skin is not merely :he child pulls a -'.
a biological object but also a mode of perception and expression that comprises the iie prorinin .-::.-,.
surface of a body and the site where it meets the world of others and objects, then r ¡lb, and the ¿.,1l i
film can indeed be said to have a skin. Through its skin, the film is caught up in a recip- ..rrror:ndit] g th: --::
rocal, intimate, and fundamentally erotic intersubjectivity with its viewer. rr-, f OfOLIS and i:-, ::
In a very particular way, stop-motion animation interacts with its viewer haptically, The tailor .i-':
by way of textures that beg to be touche d. Street of Crocodiles ts filled with objects that, :.:s fLlm, as rrei- ,,
beyond carrying any narrative or syrnbolic value (although some do), address üewers l:.-\\ een SefU¿- .:::
primarily as organic objects of the world' reminding us of our own embodied exis- :'.:er and the p-:::
tence in that world. Even the mechanical objects, and there are many, seem significant : s riorLd tecl,.--,
only by virtue of the dust particles and rust that accumulate on them, the saliva that . '. ,r,LSlr O\ eI e\-ir-. :
propels them, and the dirt that surrounds them. . ,-i*:-s here ".-n¡ ::
- :,-.uch as a ¡-.a::
: ::-- l :it¡nal e\pell : :
r For a discussion ofthe erotics ofhaptic visuality, see Marks (1998).
TOUCH AND THE CINEMATIC EXPERIENCE 153
ü that is another mode Street of Crocodiles takes very little in the way of narrative from Bruno Schulz's
i99,r, p. 3o), while the novella of the same name, but it does borrow the text's keen focus on texture and
r :eqister oftactile per- tactility. In Schulz's story, a character asks:
Can you understand the deep meaning of that weakness, that passion for colored tissue, for
papier máché,for distemper, for oakum and sawdust? This is . . . proof of our love for matter as
such, for its fluffiness or porosity, for its unique mystical consistency . . . We love its creaking,
its resistance, its clumsiness. We like to see behind each gesture, behind each move, its inertia,
its heary effort, its beariike awkwardness.
(Schulz ry77,p.62)
The scene in the film that corresponds to this passage revels in texture. The pup-
',l5ie¡ difñcult and pet-protagonist meets a tailor, a doll with a head of porcelain who, with three other
i's
porcelain dolls, welcomes the protagonist into his shop. The dolls fawn over and
::r:qes are mysterious
,-::: of r-isual range by fondle their guest, running their tiny porcelain hands over his wiry hair and cracked,
::. :re\t shot. Film and bewildered-looking features. After draping his body in rich, colourful fabrics, they
replace his head with one like their own, stuff it with cotton, and pull tufts of cotton
:.;: lurok rvith a precise
:: ;;' u¡n1r. to the touch. from the empty top, ears, and eye sockets.
:: :ran something pre-
At one point, the tailor stands at a table where he runs his hands over an exqui-
sitely detailed map of Poland and magically produces from his fingertips a piece of raw
; neeting of the film's
meat that he lays over the map. He smoothes a sheet of dress-pattern tissue paper over
the meat, whose moisture seeps through the tissue papel. An army of pins marches
-. :¡scribes:'By engag-
up the surface of the meat, sinking themselves into the glistening flesh as they fasten
i I'larks ry99,p.t84).
i-- ;trme to the surface
the tissue pattern to it. As the dolls continue to dress the now barely recognizable
protagonist, the tailor cradles his detached head, running his tiny fingers through the
...n against the film's
coarse grey hair.
:¿r:l1r and sensuously
--::¿: ihev construct an The tailor invites the protagonist to gaze through a window into a darker room
beyond the tailor shop. Flanked by two dolls who caress his backside through his
,,: 1jj!. p. 183).,
rough tweed coat, the protagonist peers through the grimywindow. On the other side
: ¡r animal body, then
of the glass, another porcelain doll-a child-sits on a dirt floor, cradling in his arms
;:.r-Ponty has said of
j r skin is not merely a light bulb that is nearly as big as he is. As the protagonist and dressmakers look on,
:
the child pulls a woollen hat from his head and gently covers the light bulb with it,
-':r that comprises the
the proximity of the camera emphasizing the wool fibres, the dirty surface of the glass
:::s ¿nd objects, then
bulb, and the doll's smudged features. In an extreme close-up, tiny screws on the floor
.:;¿rqht up in a recip-
surrounding the little boy raise themselves up, twirl deliriously through a layer of dirt
-:i Itetr er.
so porous and dry that one can feel it under one's fingernails, and scamper off.
: :ls rierver haptically,
The tailor shop scene is characteristic of the haptic quality of the experience of
ij¿J rrith objects that,
; lo . address viewers this film, as well as the plaful and perhaps disturbing way it probes the boundary
' --'-,rn embodied exis- between sexual and non-sexual touch and between pleasure and disgust. Like the care-
taker and the puppet-protagonist, the porcelain dolls in the tailor shop experience
:t:it\'. seem significant
iiem, the this world tactilely rather than visually or cognitively, running their fingers volup-
1 saliva that
tuously over every available surface, and the film inütes its viewers to do the same. The
textures here and throughout the film awaken the fingertips and call forth the sense
of touch as a meaningful and complex mode of perception, communication, and
emotional experience.
154 JENNIFER M. BARKER
In this way, street of crocodiles seems to exemplifl' the physicality of the film experi- This is not to sav tha
ence, which Siegfried Kracauer described this way: where between our bc
two places at once, e'. <l
Film not only records physical reality but reveals otherwise hidden provinces of it, includ-
We are invited anc
ing such spatial and temporal configurations as may be derived from the given data with the
well as our own, caui-i-
aid of cinematic techniques and deüces. The salient point here is that these discoveries ' . '
mean an increased demand on the spectator's physiological make-up. The unknown shapes he attests, we compreher:
encounters involve not so much his power of reasoning as his visceral facukies' Arousing his sensuously, then if rie .
innate curiosity, they lure him into dimensions where sense impressions are all-important. in two places lvhile '."'r
(Kracauer 1965, p. r58) writes in another co::;
wholly into the ne\r :;
(Merleau-Pont\' t9 6:. :
:¿lir¡ of the film experi- This is not to say that we aren't situated at all and are floating anchorless some-
where between our body and the screen. Rather, we are doubly situated: we exist in
r ::Lrlinces of it, includ- two places at once, even if we never literally leave our seats.
:, :i¿ siten data with the We are invited and encouraged, then, to commit ourselves to the film's space as
,l:: lhese discoveries . . . well as our own, caught up 'there' and 'here' at the same time. Ii as Merleau-Ponty
: -::¡ unlnown shapes he attests, we comprehend space not objectively and theoretically, but by living in it
";" .-,t;tlries. Arousing his sensuously, then if we are to comprehend filmic space at all, we must live quite literally
- ::. are all-important.
in two places while watching a movie: our own and the film's. What Merleau-Ponty
ñ:¿cauer 1965, p. r58) writes in another context may just as well apply to the film experience: 'I am borne
wholly into the new spectacle and, so to speak, transfer my centre of gravity into it'
(Merleau-Po nly tg6z, p. 25r) . Psychologist and early film theorist Hugo Münsterberg
described the sensual component of attention in the cinema in similar terms, saying,
'We feel that our body adjusts itself to the perception . . . We hold all our muscles
in tension in order to receive the fullest possible impression with our sense organs'
(Münsterber gryú, pp. 36-).
As Vivian Sobchack has made clear, the fi.lm/viewer relationship isn't anthro-
:: -c¿st as much as the pomorphic, for clearly my legs are not 'like' a dolly anymore than my eyes are 'like'
,-h.:;h I rvould argue is a camera lens. But, like our body, the film's body 'uses modes of embodied existence
:-,_i! on its muscular (seeing, hearing, physical and reflective movement) as the vehicle, the "'stuff', the
substance of its language' (Sobchacktggz, p. 5). We and the film exhibit likenesses
' :re. This 'lure' is an
-::: images but also to in behaviour and comportment and in the way we use the muscular body as a means
¿- ;e alin'but also actu- of perception and expression. Pushing, pulling, reaching, cowering, flinching, leaning
forward, and pulling back, for example, all express in muscular terms one's 'attitudes'
paradoxical in the about one's relationship to the world and others in it. All these gestures can be per-
':l.-
:rs do not correspond formed by human bodies and film bodies, though we may use arms, legs, and spine to
perform them, while the film might use tripod, crane, and zoom lens' In this way the
'-*:s oftinyobjects- film's body and the viewer's body are irrevocably related to one another, but neither
¡-¿ce-that swell to
,
:::i.on to the tiny fea- identical nor completely divergent.
,..cmes a thick, three- As with the definition of 'skin', then, the 'musculature' is not reducible to biology
:.iei \\'rote, the view* or mechanics; rather, a body's musculature is defined by its expressive and percep-
;-'.:p: it requires (and tive functions. In the case of both viewer and film, the musculature isn't a set of fleshy
or mechanical 'parts' that we possess, but something through which we live and
I ill clll i experience the world. If we can 'inhabit' cinematic space and experience movements
in (and ofl the film with our own theatre-bound bodies, as Sesonske described, this is
-i' irto the space of the
because the muscular movements of both viewer and film express a particular interest
: -::re speed, be rejected
-i saenes rr'here we seem in and attitude toward the world. They are counterparts, their muscular behaviours
, ¡:t. of the film. inspired, imitated, and sometimes resisted by one another.
The muscular relationship between film and viewer can be inflected in any
S¡sonske ryn,p.4c¡)
number of ways; it might be tentative, playful, erotic, hostile, etc. Always, though, this
;:ience, it is not only reciprocal relationship is lived out by means of specific gestures. In Street of Crocodiles,
:s' rn the film space, of perhaps the most dlmamic of the film's muscular 'gestures' are the animated'camera
.r:1rr: our seat, the chill movements'. I set the term in quotation marks to indicate that at least some of these
¡:le sensations can be movements are probably fabricated in the stop-motion mode, while others move in
:ar of the film's body. real time over their objects, and it is difficult to teil how any given movement is made.
156 JENNIFER M. BARKER
The camera movements are not only ontologically ambiguous, but also themati- activities, and regur:r
cally ambivalent: they express, in muscular terms, the simultaneous but contradictory attention or being ,:',
experiences of curiosity and claustrophobia, freedom and restriction, vitality and and enacted in a di::r
death. The pans and tilts and tracking shots are as tiny as the objects they explore: attention diverted tii
each one moves very quickly, sliding across intricate, richly textured surfaces like The work of the r-i-.;
fingertips across Braille. This rapid movement, combined with the camera's extreme our bodies' movem-:
proximity to the objects of its attention, expresses an intense curiosity, a desire to body actually consi.:,
explore the world of the film closely and intimately, and to 'take in' as much detail we make involves t:e
and information as possible with each fleeting instant. At the same time, though, the example, and'the l:iir
camera movements underscore a sense of restless and uneasy confinement. No camera open and shut, anc :.
movement lasts more than a second or two, and each one ends abruptly, so that the rh¡hmic actir.itres. -:
restless energy of the camera is constantly snuffed out. The camera moves almost (Sobchack 1990, p. :_:
obsessively in tiny increments as if struggling to work itself free from a cramped space. film speed, and or::.,
Ultimately, it fails: like the protagonist who is set free from his tether in the prelude camera and proje;t,: r
only to wander endlessly through the small and twisted spaces of this cave-like world, than intermittent. ir,l
the moving camera is paradoxically perpetually constrained. Whereas molemar
Sesonske writes that, in the film experience, 'The slightest invitation will persuade us of its smooth ...,
us to abandon our ordinarylives and live whollywithin the world of the film. Cinema smooth motion o: r:
space presents that invitation' (Sesonske 1973, pp. 399-4oo). These miniscule camera these images and t¡¡'¡
movements are the 'invitation' to inhabit the space of Street of Crocodiles, but at the stop-motion ¡1tr. r,r i
same time they are the means by which we are 'rejected or excluded from it' (Sesonske we simultaneousl,,- i:,
1973, p.4o3), as the space is made to feel inhospitable, too close and too small for
poral structures bo::
comfort. This ambivalence between inhabitability and inhospitality, and between irregularity of the i.;'
movement and constraint, perhaps parallels the one that functions at the haptic level Street of Cr o c tt,i :,. :.
between pleasure and disgust: both involve a simultaneous attraction and repulsion, tension between cr.:r:
an ambivalent push/pull dynamic. cinematic and hul.,.
Like the materiality of its objects and spaces, then, the miniature scale and meaningful becaus=
movements of Street of Crocodiles invite a tactile participation by the viewer that skin and the mus,u .
informs and shapes the viewer's intellectual and emotional experience of the film. We rial and muscula: r:''
feel for the puppet-protagonist precisely because we feel withhtm, muscularly caught implicated in its ter::
up in a not-yet-intellectualized but sensuous, muscular connection to the miniature connection in a ria-, r:
world of the puppets, dolls, meat, metal, ieather, and dust. Some viervers ¿r¿
makes sensible to :::¡
narily sensible. H or,';.'
l-hich perhaps ar:s;s
the intermittence ¡:
Vr scnn,t irotion animation.
Street of Cro;,..;:'..
:he complexities .:::
The tenuous, tactile connection between the film's and viewer's bodies exists as well
engagement bet\\'ea: :
in their depths, where a similar dynamic of attraction and repulsion is at play. The
,hat appear in th¡ ::
deeply felt but (ordinarily) scarcely noticed rh¡hms of the internal body can enact and
,he floorboards o: :
express attitudes and emotions as dramatically as the skin and musculature. Like the
:1ong its corridors. T.
skin and musculature, the viscera are here defined phenomenologically, not as indi-
:ocket watch. At tit¡
vidual inner organs but as that which keeps the body in motion, enables all conscious
:,\treme ciose-up ai-:
TOUCH AND THE CINEMATIC EXPERIENCE I57
than numbers, further destabilizing our ordinary sense of time)' The watch shuts tight New York.
and spins around on the table, then falls open to reveal its inner workings, which con- Marks LU (rgg8). \ idc'-:
sist entirely of glistening raw meat. The screws emerge again from these shockingly Marks LU 1r999). fll, ::
organic innards, wriggle their way out of the meat to fall on to the table again, and Universiry Press' D-: .
Merleau-PontY M r t"--
scurry away.
Münsterberg H (Lql" -
This remarkable sequence echoes the entire film in its attention to minuscule detail,
RutherfordA(¿oo:. .
its kinetic ambivalence, and the astonishing materiality of the watch's innards. More ema.com/contenl' j -'
significantly, however, this meaty sequence also reminds us as embodied viewers
':: --
SchulzB\t977)'1
oio.r. carnal embeddedness in time itself. The film engages us in two different tem- SesonskeA(19;3 ( -r* '
'-
poral modalities simultaneously, at once reminding us of our own discontinuous Martinus Niihn- l'-
nature as beings in motion and of our own lived perception of our movement as con- SobchackV1r99o .-- --
Film and Vrde,,. r:. : -,-
tinuous and smooth. By engaging our skin, our musculature, and our viscera with
Sob.chackV(r99:. -
its own, the film entices us into a deeper relationship with it, by which we come to
a
UniversitY Prer'. Pr'- -,'
palpable recognition that the inner clockwork of our bodies and the film's body are Taussig M {rqq.r .' : '
uniattnily similar. Our tactile, visceral body grasps the implication immediately: like New York.
the cinema itself, our own robust vitality is an illusion, and our vital rhythms
mark with
everybreath the perpetual possibility of stillness and death'
To say that we are touched by cinema indicates that it has signifi.cance for us, that
it comes close to us, and that it literally occupies our sphere and shares something with
us (materiality, muscular movements, rh¡hm), but the relationship between viewer
and film is not anthropomorphic. Instead, it is a matter of shared styles of touch, and
the differences between film's body and viewer's body in this respect are as impor-
tant as the similarities. Particular structures of human touch correspond to partic-
ular structures of the cinematic experience, so that forms of tactility that filmgoers
experience at the movies are taken up-in complex, not always comfortable ways-by
both spectator and film.
Cinematic tactility comes to mean not simply contact, but rather a profound
manner of being, a mode of perception and expression by which all parts of the body'
both cinematic and human, commit themselves to or are drawn into an intimate
and intersubjective contact with the other. Like the wriggling screws and the meaty
pocketwatch, film and viewer are present to one another not only in the press of skin
io skin, muscle to muscle, and viscera to viscera, but in a full-bodied opening on to
one another that goes beyond surface, middle' and depth. We do not'lose ourselves' in
the film, so much as we exist-emerge, really-in the contact between our body and
the film's body.
RspERnNcns
BarkerJM(2009). TheTactileEye:TouchandtheCinematicExperience.UniversityofCalifornia
Press, Berkeley CA.
TOUCH AND THE CINEMATIC EXPERIENCE r59
:-,r;efull)i almost vio- Gibson Il OgZil. The Ecological Approach to Visual Perception. Houghton Mifflin & Co.'
:-. -¡atures letters rather Boston, MA.
lhe
rr atch shuts tight KracauerS (rg6i). Theoryof Film: TheRedemptionof PhysicalReality. OxfordUniversityPress,
New York.
: ',rar:kinos, which con-
Marks LU (rqgs). Video haptics and erotics. Screen,39,331-47.
,:-,r these shockingly
MarksLU (tggg).TheSkinoftheFiIm:InterculturalCinema,Embodiment,andtheSenses.Dtke
:. r:re table again, and University Press, Durham, NC.
Merleau-Ponty M (rq6z). Phenomenology of Perception. Routledge, London.
::¡ minusculedetail, Münsterberg H (rqr6). The Photoplay: A Psychological Study. Dover, New York'
i.,.:i!:t s innards. More Rutherford A (zooz). Cínema and Embodied Affect, Senses of Cinema, http://wrt'w.sensesofcin-
:: ¿rnbodied viewers ema.com/con tents I q I z5 I embodied-affect.html
, -:t r,\'o different tem- Schulz B Gglil. The Street of Crocodlles. Penguin Books, New York'
Sesonske A (1973). Cinema Space. In D Carr and ES Casey, eds. Explorations in Phenomenology.
-r I'rr-n discontinuous
Martinus Nijhoff, The Hague.
, *r :]lur\ ement as con-
Sobchack V (r99o). The active eye: a phenomenology of cinematic vision. Quarterly Ret'iew of
::1l trrlr t'iscera with FiIm and Video, rz, zr-36.
" l:ch rr'e come to a
-,u
Sobchack V (rqqz). The Address of the Eye: A Phenomenology of FiIm Experience. Princeton
1: r:e ltLm's body are University Press, Princeton, NJ.
:- :- -nmediately: like Taussig M (rSS¡). Mimesis and Alterity: A Particular History of the Senses. Routledge,
:.,::-,llmS markwith New York.
-: :::her a profound
-- .,- :arts of the body;
: "",-:tnto an intimate
).:.,', s and the meaty
.--, :: :he press of skin
:. i:ed opening on to
:,, i -ose ourselves'in
;:-..,'3in our body and
:rsiñ-of California
HEA
TAS
CRC
HEARING SCENTS,
TASTING SIGHTS:
TOWARD A
CROSS-CULTURAL
MULTIMODAL
THEORY OF
AESTHETICS
DAVID HOWES
INrnopucrroN
Turs essay opens with a review of recent advances in neuropsychological research
on the multisensory organization of the brain. It goes on to critique that research
from the standpoint of the anthropology of the senses. Crucially missing from the
neuropsychological account of how the senses function is any recognition of the
role that culture plays in the modulation of perception, as shown in the first section.
The second section begins with an inquiry into the history of the Western concept of
'aesthetics'. This concept originally stood for the perception of 'the unity-in-multi-
plicity of sensible qualities' (Baumgarten), but the intersensoriality of this initial for-
mulation was quickly lost from view. The third section presents a sulvey of the role
L62 DAVID HOWES
of the senses in the aesthetic practices of a range of non-Western cultures. The tradi- receiver equivalent :.,
tions covered ail show a marked tendency to combine the senses (rather than sepa- those where the oi;:.
rate them) for artistic and other purposes, most notably healing. This survey in turn may also be made l¡
lays the groundwork for the articulation of a cross-cultural multimodal theory of tion. In such case.. .::
aesthetics in the final section, which recuperates sone of the elements of the original often quite cofirpl3.¡ -
definition. The essay concludes with a series of suggestions regarding how the disci- the researcher to sa:l
plines of art history and neuropsychology might reconceptualize their engagement nance, as in the cas; -
with the sensorium in a manner that is attuned to the historicity and interplay of the ventriloquist's d¡,iir: :t'
senses in art as in everyday life. or one of cottcrltt".,,..:'
tal fruit moths si.;: :
'
with her pherom..r-.,
and they need an =::.
wiil copulate' ( P¿:r-::,
is a third possibilir,'. .,
THn NEw MULTISENSORY PSYCHOLOGY Manyofthe stuú,ú j
OF PERCEPTION the multiple sites . ..
long thought to b; ::-
so-called higher i::::
It is commonly assumed that each sense has its proper sphere (e.g. sight is concerned the formation of u*:-
with colour, hearing with sound, and taste with flavour). This modular conception onstrating the tun¡i- .
of the sensorium is reflected in the anal1.tic orientation of most current research in ies point to therr :-.1-:
the psychology of perception with its sense-by-sense (or 'one sensory modality at that senson'-spec:..; .
a time') approach to the study of perceptual processes. In recent years, however, a
areas in situations , - ,.
more interactive, relational approach to the understanding of how the senses function
cortex ir-r bhnd ii;-', -:
has begun to take shape as a result of the growirrg body of evidence that poinls to the
auditory corte-\- ir.r ¿..:
'multisensory organization'.or 'integration' of the brain. As Calvert, Spence, and
Stein write in their introduction to The Handbook of Multísensory Processes (the most Oi r-rote, the c1ua,t.'. - - ,
duced and decoded along the pathways primarily devoted to that sense, but also how this ::rnciple that inful. :.:
information is modulatedby what is going on in the other sensory pathways. .:r-rt process the in¡¡:r -.,
(Calvert et al. zoo4, pp. xi-xii, emphasis mine) ::--,'en bl the naiu:. '
::m Lultures. The tradi- receiver equivalent to the response to each unisensory component (ec1uivalence), and
-.s-'s rather than sepa- those where the overall response is enhanced (superadditive). Multisensory signals
-:. This survey in turn may also be made up of stimuli that convey different (i.e. non-redundant) informa-
:ruitimodal theory of tion. In such cases, the relationships between the component parts of the signal are
--:rents of the original often quite complex, and it takes a great deal of painstaking attention on the part of
.r¡úlng how the disci- the researcher to sort them out.' For example, the relationship may be one of domi-
r-:; their engagement nance, as in the case of the ventriloquism effect (where the seen lip-movements of the
.:, :rd interplay of the ventriloquist's dummy alter or 'capture' the apparent location of the speech sounds),
or one of concetenation, as in the case of the reproductive behaviour of male orien-
tal fruit moths (such moths 'need the visual presence of the female in combination
with her pheromones before they will perform their most intricate courtship displays,
and they need an additional tactile stimulus of a touch on the abdomen before they
will copulate' (Partan zoo4,p. 45). Emergence, as exemplified by the McGurk effect,'
is a third possibility, and this typology could be extended further.
HOLOGY
Many of the studies in the Handbookuse modern neuroimaging techniques to reveal
the multiple sites of multisensory processing in the brain, including many regions
long thought to be modality-specific or 'primary sensory' areas as distinct from the
so-called higher order, 'associative' areas traditionally assumed to be responsible for
. :. sight is concerned
the formation of unified percepts out of the diversity of inputs. In addition to dem-
: :rrrdular conception
onstrating the functional interdependence of the modalities, a number of these stud-
:. a.rrrent research in
ies point to their functional equivalence or adaptability. For example, it is now clear
.; i.:rsLrn' modality at that sensory-specific areas can be 'recruited' or 'remapped'by other sensory-specific
a;a-: 1-eafs, hOWeVef, a
areas in situations of sensory deprivation or intensive perceptual training. Thus, visual
:',\ :re senses function
cortex in blind individuals has been found to show activation in auditory tasks while
.:r.. that points to the
auditory cortex in deaf individuals can be activated byvisual tasks.
r-:,,,'eri, spence, and
--'. Prir.e-s,ses (the most Of note, the quality of sensation associated with activating the visual cortex in congenitally
biind individuals, or the auditory cortex in congenitaily deaf individuals, appears to derive
from the nature of inputs. That is, visual inputs are perceived as visual even when auditory
=:,,t; are most likely to cortex is activated fin the case of the biind, while the reverse holds true in the case of the
- -: lack of arvareness of deafl . . . Furthermore, even in normal, nondeprived humans, there is evidence for extensií"e
:. :--rlch of sensory per- multisensory interactions whereby primary sensory areas of the cortex can be activated in
.:.,:,r,' modality is trans- a task-specific manner by stimuli of other modalities . . . Common to these findings is the
-.-l.e. but also how this principle that inputs recruit pathways, cortical areas, and networks within and between areas
that process the information, and the sensoriperceptual modality associated with the input is
: :,--r.ii, emphasis mine) driven by the nature ofthe input rather than by the cortical area activated per se.
(Sur zoo4, p. 690)
:¿j üct that, in noisy
:', ¡an be seen as well
:Jundancy hlpothesis
' Classic information theory remained blind to modulations of this sort due to the assumption
l ;h¿nnels' are better that such'interference' simply constituted so much'noise'.
' In the version of McGurk's experiment with which I am familiar, the research subject watches a
:'res deeper to explore dubbed video of an actor's face pronouncing syllables. The seen lip-movements alter which phoneme
is heard for a particular sound (e.g., a sound of/ba/ tends to be perceived as /da/ when it is coupled
: r-,,' sisnal (e.g. Partan
with a visual lip movement associated with lgal).In this instance, the response to the multisensory
ri:lication, redundant signal is new, qualitatively different from the response to either of the unisensory components, and
J:ce responses in the thus demonstrates emergence.
DAVID HOWES
Such evidence of adaptive processing, or 'cross-modal plasticity', poses a serious metaphor, saying tt,
challenge to the conventional model of the sensorium as consisting of five structur- thesia nor metaphc,: :
ally and functionally distinct modalities. In light of this challenge, some researchers p. 868). The authcr.
have proposed that the phenomenon of synaesthesia (i.e. the union or crossing of determine whethe r .::
the senses, e"g. hearing colours, tasting shapes) might provide a more productive conceptual (an int:
model for conceptualizing perceptual processes than the conventionai sense-by-sense only those in rvht,:::
approach that has dominated research on the senses and sensations to date. able), and then o..¿:
'cross-activation t-i :
ferent mechanisn:s. :
through an e\cess -
The model of synaesthesia activity in back-r: ,
The condition of synaesthesia is tlpically understood to be quite rare. Estimates of defective prunrn: ,--: -
its incidence vary from r in zoo to r in zooo to r in zo ooo people (Baron-Cohen and In the case of c,,- -.
Harrison 1997; Ramachandran et al. zoo4, p. 368). The most commonly documented ple-the brarn ¿:;-.
form is colour-grapheme synaesthesia in which written words or letters are per- other in the fusiti-:::r
ceived as having particular colours. To iirnit synaesthesia to a congenital condition, nectivityt as a resu,: . :
however, would be myopic.3 Synaesthetic connections can be learned. Take the case of rience this eftect r. - -.,
odour-taste synesthesia which, perhaps because it is such a common effect, has failed l¿r from being ,.r :'.:
to attract much popular attention or scientific documentation. Yet the evidence is clear: :ieuroscience and. a .:.
:resia is beginnin: :
t-!L1r3l,._i.tty of people appear to e4_perielge odor-taste synaesthesia. First, sweef is one of
--usiYe questions :-- - ,-.
íh;-";.o--on descriptors upfti.a to odori . . . [Fuiiñ¿amóie,fn¡heñlmelling an odor, tho*.:.
-:neuage and
ñosi people .utr -oi. easily recognlze a taste-like quality such as sweetnéis thán more specifi-
'qualities such as strawberry- or banana-likeness.
(Stevenson and Boakes zoo4,p.69)
There rs mu.:r i
This finding raises the question: when we speak of the odour of vanilla or straw- ,,:dv of percepti,;, :
berry as 'sweet', are we speaking in metaphor rather than reporting an actual olfactory l -.n unduh reii:r. '
drinking, patterns of retronasal odor stimulation co-occur with oral stimulation, notabiy
of the taste receptors, so that a unitary percept is produced by a process of either within-
r,.. to the seiec. ' ;
event associative learning or by a simple encoding as one event. Eatirlg s_wee!'11ni!q {la¡,o_r : . l-. ne\-er aSCa:]: -:.
',
ipe cqeqm yill en-sure thal thg retronasal odor of vanilla becomes associated with sweetness; ¡r¡rqht coltstttJ:; j
ó¡ some later occasion the smell of vanilla will seem sweet, even if no conscious recollecton of : ::r, e S COnae l:li-. i
eating ice cream comes to mind. . -.::::chandran t: . :
(Stevenson and Boakes zoo4, p. 8r)
3 The ethnomusicologist Steve Feld once remarked to me that limiting synaesthesia to those
who are congenitally susceptible to this effect would be like restricting music to those with perfect
pitch. It cannot be so confined.
HEARING SCENTS, TASTING SIGHTS 165
!,t:i¡-', poses a serious metaphor, saying that "synaesthesia is just metaphor" helps to explain neither synaes-
'r.i:n_U
of five structur- thesia nor metaphor and merely compounds the mystery' (Ramachandran et al. zoo4,
:l:i. Some reSearCherS p. 868). The authors go on to tout the experimental procedures they have devised to
. :rlon or crossing of determine whether an alleged synesthete's experiences are 'truly perceptual' or merely
:. a more productive conceptual (an important distinction when it comes to selecting test subjects since
: --,¡ nai sense-by-sense only those in whom the effect can be shown to be involuntary are considered desir-
-, -s ¡o date. able), and then offer a physiological explanation for the effect having to do with the
'gross-activation of brain maps.' Such cross-activation may come about by two dif-
t(r)
ferent mechanisms, namely: cross-r.i'üing between adjacent [brain] areas, either
through an excess of anatomical connections or defective pruning, or (2) excess
áctivity in back-projections between successive stages in the hierarchy (caused by
-.,:: :are. Estimates of defective pruning or by disinhibition)' (Ramachandran et al. zoo4,p.87z).
:-; Baron-Cohen and In the caseof colour-grapheme slmaesthesia-Ramachandran et al.'s chosen exam-
::-- :r !r n i\' documented ple-t!e-br.a* uráJi&.irp."dñrgló frulh.-.t und colours are right next to each
"::,,-,: letters are per- gt\er,in lhe fusiform gyrus, and the potential for excess cross-activation or'hlpercon-
¡ - nsenital condition, nectivity' as a result of some genetic mutation in those individuals who naturally expe-
.:::d. Take the case of rience this effect is therefore strongly indicated. Ramachandran et al. conclude that:
:non effect, has failed far from being a mere curiosity, synaesthesia deser.¡es to be brought into mainstream
;: the er-idence is clear: neuroscience and cognitive psychology. Indeed, fprecisely because the neural basis ofslmaes-
. J ¡::. is one of thesia is beginning to be understoodl it may provide a cruciai insight into some of the most
-sile¿f
-: -.':-Jñelling atr odor, elusive questions about the mind, such as the neural substrate (and evolution) of metaphor,
J- :htn more-specifi-c language and thought itself.
(zoo4, p. 88r)
.': Btrakes zoo4, p. 69) There is much to be said for Ramachandran et al's 'bottom-up' approach to the
--: --f ranilla or straw- study of perceptual processes, but I find the phy:igloAical reductionism 9-f thgir posi_
--= ::i actual olfactory [o." g"4qly restrictive from my own perspective as a cultural anthropologist, and
I thinkan equallyvalidcase couldbe madefora'top-down'approach. Such an approach
would start by examining the cultural organization of the sensorium and descend via
, . ::.ult of eating and
the psychologilal ro the physiological level of brain organization. In point of fact,
:.- ,. r:]ulation, notably
: .r>: Lrf either within- due to the selective focus of their discipline-neuropsychology-Ramachandran
: -.. .r1 :¡t et al. never ascend in what they call.'the hie¡aryir_y- as far as the culturlllevel-o This
_r_41!]lq-J1¡vcr1
, :=.-:d r,rith sweetness; oversight constitutes a serious lacuna,'ior álcultural pSychiatiist Laurence Kirmayer
: :r>:.L)us recollecton of_ observes concerning the hierarchical systems view of neural organization (which
Ramachandran et al. presumably share):
.r:3oakes zoo4, p.8r) Contemporary cognitive neuroscience unde¡stands mind and experience as phenomena
that emerge from neural networks at a certain level of complexity and organization. There
-. ¿1so rejected by is_ingry-asing recognition that this organization is not confined to the brain but also inclüdE6
:::rer on 'synesthsia, lggls t!19qgh the body and the environment, most crucially, through a sociai world that is
Thcir objection rests
:: ri¡ neural basis of
a Ramachandran et al. are not alone. There is but one reference in the whole The Handbook of
-:..:resia to those Multisensory Processeslo cross-cultural variation in the modulation of perception: apparently, the
: ::r!rse rr'ith perfect McGurk effect is significantly weaker in Japanese than in American perceivers (Bernstein et a1. zoo4,
P. :o7).
766 DAVID HOWES
culturally constructed. On this view, 'mind' is located not in the brain but in the relationship wings of her nose (C¡L
óf brain and body to the world. additive function of :r
l,:-..--.' '-----'* (Kirmayer zooT) reception and utteran,
bad-smelling ones-¿l
Ideally, Kirmayer states, 'we want to be able to trace the causal links up and down
late between soundai
this hierarchy in a seamless way'.
These findings of .
5 Or even earlier periods of Western culture. For example, a form of audio-grapheme synaesthesia 6 Thatis,proximin i,:
has been described for the Renaissance: 'In a person's handwriting, Erasmus claimed, he could hear Ramachandran et a]. r. :: -
that person's very voice' (Smith zoo+, z8). torrvard, and back-pror.;
HEARING SCENTS, TASTING SIGHTS 167
.,r ¡rut in the relatio-nship wings of her nose (Calame-Griaule 1986, pp. 3o8-1o). The nose-rings have the super-
additive function of modulating both her hearing and her speech by promoting the
(Kirmayer zooT) reception and utterance of 'good-smelling words' and the deflection or repression of
iu.a- links up and down bad-smelling ones-all in accordance with the intrinsic connection the Dogon postu-
late between sound and odour.
-i.';,tural Handbook of tft¿é-. findlngs oi audio-olfactory synaesthesia, familiar to scholars of sensory
r j3:tson'processes to be arrthropology, would likely come as a surprise to most scholars of neuropsychology.
-:,. ;ulturally patterned For exañple, Stevenson and Boakes claim that: '-O_{ors display taste properties but
-r. relationship of brain dq lot elicit auditory or visual sensations'.(zoo4, p.ZJ). On the contrary in Mali as in
Melanesia, sounds do elicit olfactory sensations, an_d üce versa. Thus, what Stevenson
--. .'i svnaesthetic per-
: c;ltural practices and and Boakes take to be a physiological given actually rests on certain culturally-contingent
;:r.ss different cultures (i.e. peculiarly Western) assumptions about the divisions of the sensorium and/or
..:i.,n rt-hether the inci- potential for cross-sensory activation. This example underscores the need for more
: an aural-oral society cro ss -talk b etween the disJiplilej,_o$9_919p sycholo gy an d anthrop
919
gy if co¡lem -
: ¡ocieñ'.i In the latter, co,gnitive neuroscience is to achieve a comprehensive understanding of the
.pollty
: .r: a computer screen, multipG possible forms of cross-talk be_twee¡*tfu se!-ses: To put the matter another
:i :orn-rer, words (being way, starting with cross-cultúrai examples such ur ih.r", which are practical(i.e. sup-
, :her rr-ould be felt or ported by cultural practices which form part of the 'loop' through which all sensations
-r Papua New Guinea, must pass) as well as metaphorical, neuroscientists could well be inspired to discover
,-.c ridn languages, such all sorts of heretofore unsuspected cross-linkages between the senses wherever they
..r .ri'hearing a smell,' may be localized in the brain.6
,-.: imel'. The reason
: .c. rr ithin olfactory
-,1. rhe body with oil,
. ) :resence and words
AnsrrrETrcs-THE scrENcE oF
:l
language ofMali),
.i
:s. speech 'has material
SENSE PERCEPTION
':r¡sl sound . . .
[It] has
::r origin, are so neár The concept of 'aes!h_e1-r!g' was coined by the philosopherAlexander von Baumgarten
r¡:ie-Griaule 1986). The
!¡1t-he=4id-eighleeqth century. It is derived from the Gréek-aj:thesls, meaning sense
:nd bad or impetuous p3rc-ep.!19!, For Baumgarten (r75o), aesthetics had to do with the perfection ofper"
, ,r\ened to the rectum' only secondarily with thb perception of perfection, or beauty.. His new
¿Ized in, for example,
5?ption
iscience -a
oI sense cognition' was to occupy an intermediary rung, as a 'science of
-:¡ ñ¡st step occurs at the lower cognitive power' (sense perception) in contradistinction to 'the higher
:.- ring in her lower lip cognitive power' (reason). By limiting aesthetics to the perception of the 'unity-in-
:,: :ier ears at age six to multiplicity of sensiblé qualities', as he put it, Baumgarten hoped to insulate it from
:¡tical mistakes and/or being reduced to 'arid' intellection. He believed that the intellect or 'reason' was the
::a ln the septum and poo,l, I'or the-fact that it trafficked exclusively in'distinct ideas', as opposed to the
:: - -.r:pheme synaesthesia ó That is, proximity of brain areas would no longer be the determinative criterion (pace
-. :l"r::red, he could hear Ramachandran et al.), as indeed it is not given all the evidence of cross-modal activation, feed
forward, and back-projection processes that has begun to emerge.
DAVID HOWES
'confused and indistinct ideas' which were the commerce of the senses. For Baum- silences any input r--
farten, therefore, the aesthetic was r6oted in the body:-specifically, in the disposition many distractions. -i,¡
to sense acutely-ind involved attending to'the natüre of sensory experience in itself demarcation and the .
rather than trying ro intellectualize sensation (Gregor 198j, pp. J64-651. to appreciate the 'crrq:
'- náumgárténts né*'science' was quickly appropriated and just as quickly subverted By way of illustratio:r.
by his contemporaries. They replaced his emphasis on the sensuous disposition of the 'The cultivation of se:.
aesthete with a taxonomy of 'the five arts' (architecture, sculpture, painting, music,
In the appreciation or :. .,
and poetry). The scope and criteria of tñá various arts were delimited in terms of the of the presented rtc.:l:
dualism of vision (epitomized by painting) 1nd hearlng (epitomized by either music of our neighbours .-- -,
or poetry). Thc 'dark' or 'lowetl5ensgq..o-f sme!, taste, a¡d toqc]¡y'ere deemed too When reading a po.:rl
base to hold any significance for the fine arts. Theatre and dance were also excluded and unmindful of se .-,..
on account of their hybrid character, since they played to more than one sense at once intrinsic qualities of :::. .
(see Rée zooo) and we are alert to ,:- :
sic qualities stand rc ..:
Baumgarten's worst fears concerning the intellectualization of aesthetic percep-
kernel oftruth in thc i::
tion were realized in Immanuel Kant's Critique of ludgment (r79o). Kant attempted to
transcend the dualism of vision and hearing and replace it with a fundamental division
between the'arts of space' (e.g. painting) and the'ar!9 of lime' (e.g. music), accessible It should be nc.:¡*
to'outer intuition' and'innerintiiition'respectively (Rée 2ooo, pp. fS-6o). It could only something that -.
be said that.Kant rarefied aesthetics by divorcing it from perception and s-ubs_titut- r.iewer from u.ithou: 'r'
.i¡g intuition. After Kant, aesthetic judgment would be properly neutral, passionless chatting, no perfuntt. -
and disinterested (see Eagleton r99o; Turner ry94). This definition of aesthetics such spaces for the ::
guaranteed the autonomy of the enclave now known as 'art' but at the expense of r998; see further, Dr :
s_qnlg-ryp19-n1ltd9,_ 'rvhite cube').
There have beer : *
teenth centun, to .i;.
tion of aesthetics pe::,
The disincarnation of aesthetics ¡-ather of the Sr-m:,. --
In its modern incarnation (or more accurately, disincarnation), uaesthetics has tg rather than their sc:.:
the appreciation of the formal relations intrinsic to a work of art, and is admirable hours. r-¡:-:.
$o_-with
divorced from that work's content. In one particularly influential characterization strrking sensations . .
bf tlie ptoper object of aesthetics, Robert Redfield offered the following analogy: perfumes tell us of ',,', :
Saudelaire's count-::,
Art . . . is like a window with !t. Ope 4ay focus on either the garden or the
a garden behind
recarre Symbolist -:. :
window. The common viewer of a Constable land?ape or a statue by St. Gaudens locuses
on thé garden. Not many people . . . 'are capable of adjusting their perceptive apparatus to the :cents through the r.:
windowpane and the transparency that is the work of art' -:rents, and exubera: - :
(Redfield ry77,p.46 quoting Ortega y Gasset) \larinetti and his f ,
.-.riality in their an-,:-
The implication is that it is only the uncommon viewer, capable of training his or her -:: Futurism w?S rrtlr:
gaze onthe windowpane itself, who is capable of enjoying a pure aesthetic experience, .rrq correspondenc¿s _
-¡a senses. For Baum- silences any input from the non-visual senses, which are deemed to constitute so
:¿-l';. in the disposition many distractions. Aesthetic perception thus depends on a dual process of sensory
,:-; erTeriencg in i1s,éli demarcation and the elision of non-intrinsic sensations so that the viewer may come
.ll-O\ l. to appreciate the'organic unity'of a given work of art, be it a painting or a sgnphony.
.: :s quickly subverted By way of illustration, consider the following passage from a Harold Osborne essay on
disposition of the
- --'us 'The cultivation of sensibility in art education':
'l:re, painting, music, In the appreciation ofa work ofart we concentrate attention exclusively upon a selected region
-::iied in terms of the of the presented world. When listening to music we shut out so far as possibie the sounds
::-rr¿ed bv either music of our neighbours' coughing, the rustle of programmes, even our own bodiiy sensations.
l!: l\ deemed too
ere When reading a poem, looking at a film or watching a stage play we tend to be imperceptive
ra- -!i-ere also excluded and unmindful of sensations from outside. But within the chosen sector we are alert to the
;:ar one sense at once intrinsic qualities of the sense-impressions imparted rather than to their practical implications
and we are alert to the patterned constructs formed by the relations in which these intrin-
sic qualities stand to each other . . . This is perception for its own sake, and represents the
: ¡l aesthetic percep-
kernel of truth in the traditional formula'disinterested interest'.
;r- . Kantattemptedto
(Osborne 1984, p.3z; see further Elkins zooo, p. xi)
i:.ndamental division
e.9. music), accessible It should be noted that the aesthetic sensibility of which Osborne speaks is not
-. pp.;8-6o). It could only something that is cultivated by the viewer from within, it is also instilled in the
: ¿rrion and,qu-b5!iqu! viewer from without by the architectural design and codes of conduct (no touching, no
'-,' reutral, passionless chatting, no perfume, etc.) that obtain within the concert hall or art museum and other
:irition of aesthetics such spaces for the production of 'single-sense epiphanies' (Kirshenblatt-Gimblett
i lt at the expense of 1998; see further, Drobnick 2oo4, on the sensory ideology behind the art museum as
'white cube').
There have been numerous attempts in the history of Western art since the eigh-
teenth century to escape the straitjacketing of the senses that followed the rarefica-
tion of aesthetics perpetrated by Kant and his contemporaries. The poet Baudelaire,
father of the Symbolist movement, celebrated the correspondence of the senses,
¡,: . aesthetics has to rather than their separition. 'Who has not known', Baudelaire proclaimed, 'those
¿ ',uárk of art, and'is admirable hours, veritable feasts of the brain, when one's heightened senses perceive
::ltial characterization striking sensations . . . when sounds ring out as in music, when colours speak, when
,l.rriing analogy: perfumes tell us of worlds of ideas?' (whence his famous poem,'Correspondences').
Baudelaire's counterpart in the visual arts was Gustave Moreau, whose paintings
.-rher the garden or the
became Sy'rnbolist icons on account of the way they evoked textures, sounds, and
--r -st. 6un¿.;s foCuses
scents through the representation of intricately patterned surfaces, musical instru-
:reptr\-e apparatus to the
ments, and exuberant flowers (see Classen 1998, ro9-r8).
Marinetti and his Futurist confréres also experimented extensively with intersen-
rroirng Ortega y Gasset)
soriality in their artistic works (see e.g. Marinetti 1989, zoo4), though the emphasis
: oi tlafning his or h91 in Futurism was more on producing a clash of sensations than the gently concatenat-
i ¿esthetic experience, ing correspondences of the Syrnbolists (see Classen 1998, pp. tz61t) Both of these
movements were, however, eclipsed by the relentless drive toward abstraction in art
It-,aLrs on intersensory that unfolded over the course of the twentieth century. Abstract or'non-representa-
s as the perceptionof tional' art is all windowpane, with no garden behind it-the logical culmination of
tdorrpane effectively the Kantian vision.
L7O DAVrD HOWES
Healing arts of the Shipibo-Conibo Indians of Peru time of his illness. T:-.
recovery. But horr'is : t
In many non-Western societies, the aesthetic does not constitute a realm apart, but
is rather an aspect of everyday and ritual practice, and the senses are not separated
from each other but rather combine in specific ways to achieve specific purposes, The Shipibo-Cr.n-l
such as healing. Consider the geometric designs of the Shipibo-Conibo Indians of is used to refer to i-:
Peru (see Fig. 9.r). These designs-which are said to originate in the markings of the foilow how the sh¿:
cosmic serpent, Ronin-are woven into textiles, incised on pots and houseposts, visual, auditon', and -
painted on faces, and even recorded in books (see Illius zooz). However, their fore- therapy of beautr . ;r*-
most use is in the context of Shipibo-Conibo healing rituals. At the start of e h: -
The Shipibo-Conibo understand medicine to be an art, literally, and their healing under the influence
practices place a premium on synaesthesia in contrast to contemporary mainstream if with an X-ra., :r
'as
Western medical practice which is geared to the anaesthetization of the patient. messy design', or l-'.
r'vhich is the mark
One important condition of fShipibo-Conibo] therapy is the aesthetically pleasing lquiquinl
ritual involves bo;h -.
environment into which the shaman and the family place the patient. He is carefully sur-
rounded by an ambience designed to appease both the senses and emotions. Visible and tion of the pathosc:::.
invisible geometric designs, melodious singing, and the fragrance from herbs and tobacco The shaman be:-:
smoke pervade the atmosphere, and ritual purity cha¡acterizes his food and each person with painted garment :::
whom he has contact. The patient is never left alone in the mosquito tent during the critical grant herbal bund-. .
'sheets'oflumines;¡:
in the air, which sr.:
the designs into s..:..
songs once again ::,
'settle down pernt:::
on to the bodl of t:.
srlelling anti-sonq:
products of the c,-,:
Gebhart-Sat'er 19:i :
lake up to five sessi¡,-,
:o be finished. ( If t]:¿
:o recover.)
Another strate{,' --
in their ti::.
.rr reside .
f Peru time of his illness. This setting induces in the patient the necessary emotional disposition for
recovery. But how is this indigenous concept of aesthetics lquic1uinl to be understood?
:r'J¡e a realm apart, but
(Gebhart-Sayer 1985, p. 16r)
srjas are not separated
:::i'e specific purposes, The Shipibo-Conibo lerm quiquin, which means both aesthetic and appropriate,
:¡¡-Conibo Indians of is used to refer to pleasant auditory and olfactory as well as visual sensations. Let us
: Ln the markings of the follow how the shaman operates with c1uiquin-ness on these three sensory levels-
. pots and houseposts, visual, auditory, and olfactory-and how they are 's).naesthetically combined to form a
. I{orvever, their fore- therapy ofbeauty, cultural relevance, and sophistication' (Gebhart-Sayer r985, p. 16z).
At the start of a healing session (there will be five such sessions in all), the shaman,
:¿-Jr-. and their healing under the influence of lhe ayahuasca hallucinogenic vine, sees the body of the patient
:::r-rporary mainstream 'as if with an X-ray machine'. A sick person's visual body pattern appears 'like a very
.: rrf the patient. messy design', or mixed-up pile of garbage, and its pathological allra has a vile stench
which is the mark of the attacking spirits (nihue) causing the illness. The healing
:-;:Jr pleasin g lquic1uin]
r::¡. He is carefuliy sur- ritual involves both the restoration ofa healthyvisual body pattern and the neutraliza-
i enotrons. Visible and tion ofthe pathogenic aura through life-enhancing fragrance.
:r.,:rt herbs and tobacco The shaman begins by brushing away the 'mess' on the patient's body with his
- d and each person with painted garment and fanning away the miasma of the attacking spirits with his fra-
- :en: during the critical grant herbal bundle, all the while blowing tobacco smoke. He then takes up his rattle
and beats a smelling rh¡hm: the air is now 'full of aromatic tobacco smoke and the
good scent of herbs'. Following this, the shaman, still hallucinating, perceives whole
'sheets' ofluminescent geometric designs, drawn bythe Hummingbird spirit, hovering
in the air, which gradually descend to his lips. On reaching his lips the shaman sings
the designs into songs. At the moment of coming into contact with the patient, the
songs once again turn into designs that penetrate the patient's body and, ideally,
'settle down permanently'. However, the whole time the healing design is being sung
on to the body of the patient, the nihue will 'try to ruin the pattern by singing evil-
smelling anti-songs dealing with the odor of gasoline, fish poison, dogs, certain
products of the cosmetic industry, menstrual blood, unclean people, and so on'
(Gebhart-Sayer 1985, p.\7r), and thereby smudge or contaminate it. This is why it may
take up to five sessions for the design to come out 'clear, neat, and complete,' and the cure
to be finished. (If the design does not settle down permanently, the patient is unlikely
to recover.)
Another strategy commonly employed by the evtl nihue to prevent the cure from
taking is to seek out the shaman's medicine vessel which contains all his design songs,
and pry the lid off it. This causes the therapeutic power of the songs to escape. 'This
power is imagined as the fragrance of the design songs or the aromatic gas frzzing
from fermenting 1'ucca beer' (Gebhart-Sayer 1985, p. ryz).The design songs thus have
an olfactory dimension, in addition to their visual one, as their power is understood
to reside in their fragrance.
The synaesthetic interrelationships of the designs, songs and fragrances used in
Shipibo-Conibo healing rituals are nicely brought out in the following lines from a
shamanic healing song:
The (harmful) spirit pneuma
swirling in your body's ultimate point.
772 DAVID HOWES
This gustaton' c:
le quick to note. :
matter does nLrt !:-
SENSORIAL INVESTIGATIONS lhe scent ofllar::
Let us now consider another non-Western aesthetic practice which gives the lie to
the assumption that 'Odors display taste properties but do not elicit auditory or
visual sensations' (Stevenson and Boakes zoo4)-namely, the fapanese incense-
guessing game or'way of incense' (kodo).2 Osborne (1984) refers to this game approv-
ingly as a way of cultivating the ability to make 'fine discriminations within a narrow
range of sensory quality,'which is so important to art appreciation. He characterizes it
as involving'the competitive discrimination of scents of the incense t)?e.'As we shall
see, however, this characterization is altogether too unisensory, since the trick to this
game lies in crossing sensory borders.
Hearing scents
The history of incense in Japan is as old as that of Buddhism, since the two were
introduced from China together in the sixth century. To this day, Buddhist monks
use incense to sacralize a space, as a vehicle of prayer, and as an aid to concentra-
tion. However, incense may also be enjoyed without any religious purpose, in which
case it is called 'empty burning', a pastime which became especially popular among
the aristocracy during the Heian period, who delighted in compounding incense and
then guessing the ingredients when burnt. The 'way of incense' proper was codified
in the fourteenth century, with different schools devising different rules. The tea cer-
emony (shoji) and the art of floral arrangement date from the same period. All three
constituted essential accomplishments of the courtly class. The ceremony was taken
I This account of the Japanese incense ceremony is based on personal experience and the
-, ,,-! !¿- -L:,.
: _ -- a-- - - *'
following key sources: Bedini 1994; Pybus zoor; Morita zoo7.
HEARING SCENTS, TASTING SIGHTS 173
over by other classes in subsequent centuries, but then fell into desuetude in 186r,
only to be revived as a Japanese family game in the twentieth century.
Originally the grading of incense was by country of origin, since jinko (aloeswood)
from various parts of the world differed in quality: Manaban (from the Malabar coast
of southern India) was the coarsest and Rakoku (from Thailand) among the finest.
There were six such categories in ali. In the fourteenth century a gustatory lexicon was
;.r-''. these designs as devised for grading purposes.
::::;es of intersensory
' Sweet honey or s).rup-like aroma
:--¿ lne
musical scores . Sour unripe fruit aroma
:". Ther-are not simply . Hot spicy aroma
. Salty marine, ozonic, or perspiration aroma
. Bitter medicinal aroma
This gustatory ordination of incense makes sense, as Stevenson and Boakes would
be quick to note, since the senses of taste and smell are so integrally linked. But the
matter does not stop there, since this categorization also added a class dimension.
)]is The scent of Manaban, for example, is considered sweet, unrefined, and rather gritty
whence the characterization of its demeanour as 'The Coarse Peasant', while the
',','rich gives the lie to aroma of Rakoku, which is pungent and bitter, has the demeanour of a warrior and is
:,,,t elicit auditory or known as'The Samurai'.
,--; Tapanese incense- Probing further, we discover that the Japanese do not smell or taste the woodsmoke
:: ¡his game approv-
:L-' but rather, as the saying goes, 'listen to the incense' (ko wo kiku). Various explana-
.:r -.ns rr ithin a narrow tions have been offered for this turn of phrase. One authority notes that it translates
r -- :. Ile characterizes it the original Chinese phrase wen xiang, the pictograph for which resembles a person
:-:lse ñpe.'As we shall kneeling to pray and meditate. Another suggests that 'listening' more aptly conveys
'. .nce the trick to this the concentration that is involved in kodo than a term such as 'smelling' (or merely
'hearing'). Yet another authority stresses the cosmological dimension: ever¡hing is
fragrant, like incense, in Buddha's world, including his words or teachings, which
are therefore to be scented as well as heard. Finally, it is significant to note that many
forms of the game involve the recitation or composition of poetry, and thus implicate
the verbal arts.
:r. since the two were The visual and haptic senses also play a role in odour-appreciation lapanese-style:
;ai-. Buddhist monks the former in the way calligraphy is used to record the contestants' guesses and the
. :n aid to concentra- latter in the way the master of ceremonies, the tabulator and all the contestants adopt
iS PLrrPos€, in which
,-, a kneeling posture for the duration of the game. The ostensibly restricted role which
-:iallv popular among the visual and haptic senses play in the practice of kodo (e.g. having to kneel still, not
:¡'unding incense and being able to see the jinko since it is concealed in envelopes) is counterbalanced by the
: rroPer was codified degree to which these senses are extended in the imagination. Consider the variant
::nr rules. The tea cer- of the game called 'Shirakawa Border Station,' where the poem by the same names
,:ne period. All three proüdes the ceremony with its structure-or rather, its map. The idea is to retrace the
. .eremony was taken long journey (37o miles) from Kyoto to Shirakawa which the poet Noin (who lived
during the late Heian period) undertook by foot, taking two full seasons to complete.
These life-giving colours form a primary set of sensory energies for the Desana.
Secondary sets consist of such phenomena as temperatures, odours, and flavours.
Odours are believed to result from a combination of colour and temperature, flavour,
in turn, arises from odour.
: rnree pieces ofincense The Desana understanding and use of these sensory phenomena or qualia is
. Border Station. Guests complex and extensive. As regards odour, the Desana are extremely attentive to the
::st. and Autumn Wind odours in their environment, calling themselves 'wira', which means, 'people who
:le tn'out phase.) smell'. They hold that each individual has a'signature' odour. This underlying odour
Lrne each of Mist in the can be altered by changes in emotional states, by lifecycle changes such as pregnancy,
or by changes in diet. Tribes or extended kin groups are said to share a similar char-
.,:ate their listening by acteristic odour that permeates the area they inhabit. Even when this living space
;.i \forita zooT). shows no sign of human habitation, the tribal scent is said to still be present. In fact,
when travelling through their rainforest enüronment, the Desana will continually sniff
r.É. one imagines a boat
¿srr .,\manohashidate
the air in order to detect the scent of the peoples inhabiting the region, as well as the
,,'.-:rh odours of different Amazonian animals and plants. As the Desana move through the
its famous Shinto
environment they say that they lay down what they call 'wind threads', or scent trails,
'h;ps it is because the which can, in turn, be discerned by other people and animals.
Odours for the Desana are not simply indicators of presence, however, or of emo-
' ;¡nnection is the fact
r:ien \-ou catch a whiff tional state or dietary preference. Like colours, odours are believed to embody key
values. One particular odour associated by the Desana with a range of items includ-
1::rLrn a scent to mind
ing deer and palm trees, for example) conveys the notion of male fertility. The odours
e.h-morvn hay). Kodo
:¡nrial ofthe of certain kinds of ants and worms, by contrast, are associated with feminine forces.
senses of
Such perceived olfactory similarities make possible a variety of ritual substitutions.
1.h rr'.1t detracting from
i'.':en he says that the During certain ceremonies, for example, ants or worms may symbolize women, whose
:li¿tions within a nar- odour they share. They also necessitate an utmost attention to olfactory interrela-
. correct to say that it tions, as it is considered highly dangerous to combine odours (or colours) embodying
contrasting forces indiscriminately. This is particularly emphasized in Desana cuisine
in which great care is taken to blend food odours harmoniously, not in order to create
pleasing dishes, but so as to preserve the proper order of the cosmos.
Desana artefacts, in turn, also contain multisensory meanings. The values associ-
Colombia ated with a Desana basket, for example, are manifested not only through its colours
and patterns, but also through its texture, its odour, and even though the flavour ofthe
. s..Iaesthesia' (Howes
-le Desana, all sensory particular vines of which it was made. (This, by the way, seryes as a good illustration
' ::tual ingestion ofhal- of how much sensory and aesthetic meaning may be suppressed when a multisensory
artefact is turned into a purely visual display in a Western museum (see Classen and
-r:e of the cosmos, the
Howes zoo6).)
:r crithese colour ener-
: erample, exemplifies The intensely multisensory nature of the Desana cosmic model means that every
l ;c'mbination of these floral aroma, every bird song, every flutter of a butterfly suggests a particular cosmic
value to the Desana and may be grouped together with other similar values in order
s- t-.r inr"isible, as in the
r:i'; the shaman is able
to create a sensorial classificatory system that cuts across species'boundaries. The
synaesthetic nature of the Desana cosmos means that one sensory phenomenon
¡ ¡:e cial rock crystals.)
readily suggests another phenomenon in a different perceptual field, if not a whole
train of sensations. An odour will bring to mind a colour; a colour, an odour,
:,:,r'.es preSented in along, perhaps, with a temperature and a vibration. A particular design or arrange-
ment of colours used in Desana art, for example, will evoke odours, flavours, and
\76 DAVID HOWES
other sensations customarily linked with those patterns and arouse a host of related
cosmological values. A good example of this is the synaesthetic associations the
Desana make with a certain kind of flute. The sound of the flute is said to be yellow in
colour, hot in temperature, and masculine in odour. The vibrations it produces are
said to remind people of correct child-rearing practices. I should emphasize that these
synaesthetic, multimodal, associations are culturally shared and not simply the idio-
syncratic perceptions of anyparticular individual. This form of synaesthesia transcends
the domain of the neuropsychologist and involves those loops through the body, the
environment, and the social world discussed earlier.
It must be noted, however, that the Desana do not themselves understand their
synaesthetic world üew as a cultural construct, but as plain reality. What is more, the
Desana have their own theories of neuropsychology, and their brain science is highly
aesthetic. The right hemisphere of the brain, they say, is concerned with practical and
biological matters. This hemisphere is known as 'existence-first'. The left hemisphere
has as its sphere divine, abstract ideals and is called 'abstract-first'. The right hemi-
sphere perceives different sensory phenomena, the left translates them into under-
lying moral values. In another image, the brain is conceptualized as a buzzing beehive,
with each hexagonal compartment containing honey of a different colour, flavour, or
texture and relating to a different area of human life. As with the slmaesthetic associa-
tions they reflect, such cerebral models are not the domain of specialized individuals,
but are generally shared among the Desana.
As regards art, its primary role among the Desana is to employ iconic images in
diverse sensory fields in order to assist individuals to be 'in tune' with the vibrating
energies of the cosmos and the values these represent. Art, in fact, is a means of tran-
scending the everyday realm of practical affairs and uniting oneself with the abstract
ideals of the divine realm. The more sensory channels are involved in the process, the Fig.9.2 Di
more complete the transcendence. Senses'co.
With the Desana we seem to have come as far afield as possible from Western
notions of aesthetics and perception and entered a completely foreign realm or 'differ-
me to think that. -.":.
ent reality'. Yet parallels may be found. One might think of the multisensory visions
rtell as for relations: :
of the medieval mystics or of nineteenth-century efforts to create an 'omniart', which
only across cultures .-
would engage all of the senses (see Classen 1998). Within the field of science there \\'est itself.
is the enigmatic 'pyschophysicist' Charles Henry who worked at the turn of the
twentieth century. Sometimes known as the 'Symbolist scientist' for his influence on
the Symbolist art movement (see Argüelles ry72), Henry argued that all of life was
governed and unified by underlying cosmic 'rhlthms'. Henry held that, as the cosmos
itself was a unified whole, divisions and oppositions between the fields of science
and art were artificial and counterproductive. Indeed, the highest goal ofart, accord- Toweno ¡
ing to Henry, is to employ iconic images which will put people in touch with the basic
T
rh¡hms of the universe and, by integrating all their sensory faculties, enable them to
achieve a state oftranscendence.
While the means by which Henry arrived at his 'scientific aesthetic', and the values In this essay I have :.
he associated with different sensory phenomena, would no doubt be alien to the Desana rendency Io separnt¿ -..
worldüew, nonetheless some correspondences are definitely there, Which leads \\-estern cultures, h,r _
HEARING SCENTS, TASTING SIGHTS
;.elr-esunderstand their
¿lin. \\-hat is more, the
: --rai¡ science is highly
:ned rr-ith practical and
=:'. The left hemisphere
-i¡st'. The right hemi-
iates them into under-
.ú as a buzzing beehive,
colour, flavour, or
':ent
h: srnaesthetic associa-
specialized individuals,
or transmutqtion of one modality into another. I have further suggested that any theory Kant sought to supp:.,
of aesthetics, if it is to hold cross-culturally and not merely pertain to the West, will curated by Frances,r :
necessarily be multimodal, and that the same goes for any scientific theory of percep- described in lules \-;::
tion. Recent advances in neuropsychological research on the multisensory organiza- reminiscent of the S' :
tion of the brain lend support to the latter proposition, up to a point. We have further mony; and, Helen G¿rl
seen that when a cross-cultural dimension is added to this theory, our comprehension znd lts Workbook, tr':,',:
of the multisensory organization of the brain stands to be increased a hundredfold, if ing sights which is s,:, ¡
not more, as a consequence of the revelation of all sorts of heretofore unsuspected of these installations ,
cross-linkages between the senses in cultural practice, such as hearing scents (kodo) to which I compare :i--.
and tasting sights (rasa). These practices represent so many'loops through the envi- ticular lifestyle or c',i-. -
ronment' (in Kirmayer's terms) or ritualized 'extensions of the senses' (to borrow sensoriality, and the:. :
Mcluhan's phrase) in the same way that television, for example, represents a techno- What is more. tl:¡ ',
logical extension ofthe senses ofsight and hearing. ferent way of perce i'.',:
Prior to this conference on 'Making Sense of Art, Making Art of Sense,' I held a conceive of themse ', ..
dim view of neuropsychologisis ever being able to imagine (never mind simulate their frame of referer:
experimentally) all of the possible interconnections of the senses that current research dimension. Our ap:::
in the anthropology of the senses has brought to light and thus suggests may be latent enhanced consider¿ r ',
in the human condition. This dim view was based on my perception of the absence shown. And e.,'en thr :
of any meaningful cross-talk between the two disciplines. Now, after having seen is best understood u:r .
Charles Spence perform on opening night, I have a much brighter view. In his pre- ofthe earliest (and s:- .
sentation together with Heston Blumenthal, as in the many ingenious experiments he of Wassily Kandins¡-,
has conducted in his Cross-Modal Research Lab, Spence has been systematically trying This brings me to rl ,
out every conceivable permutation in the kaleidoscope-or better, collideroscope lor painting or musr;
(to borrow another Mcluhan expression) of the sensorium. Indeed, Charles Spence tlvo senses, rather ti:
strikes me as the Futurist scientist of our day, just as Charles Henry was the Symbolist
scientist of his own time, with the difference that Spence is more interested in patterns
of interference (than correspondence) and in fhe modulation (as opposed to the har-
momization) of perception across the modalities.e The impressive scope of Spence's
research into the multisensory integration of the brain (see e.g. Spence and McDonald
zoo4; Spence and Zampini zoo6) leads me to think that he has brought us to the
brink of a revolution in the sensory sciences which is as radical as the revolution in lin-
guistic science inspired by de Saussure's Course in General Linguistics 1959, again with I rr'ish to thank mv r¡,,
the difference that Spence's research crosses modalities whereas de Saussure confined sensational sources. l:
his attention to the registration of discriminations in a single sensory field-nameiy, :omments on earlie :',.
the sonic. :nd David Melcher : ::
This paper has also attempted to recuperate the original idea behind the 'science ¡uch a stimulatins t,- :.
of sense perception' as defined by Baumgarten. We have seen how his notion of research made possil-.
aesthetics as the perception of 'the unity-in-multiplicity of sensible qualities' can be founcil ofCanada.
expanded through a consideration of the aesthetic practices of a range of non-Western
cultures, while in the West this concept has been whittled down to fit within the discrete
fields of painting, music, poetry, etc.. Here too, however, some of the more innova-
tive practices of contempo nary art show signs of recovering the sensory plenitude that ' Ifyou think the c:r::: -
-:smissal of any aesthe ::; , .
'Thehistoryofsln;;.,
e Post-Futurist would perhaps be a more fitting appellation. ' l:i-slc (2oot;t for a mu,::,.:'
HEARING SCENTS, TASTING SIGHTS 179
!-sested that anytheory Kant sought to suppress.'" I think of two works in the'Art and The Senses' exhibition
rr¿in to the West, will curated by Francesca Bacci: Oswaldo Macia's Smellscape, which translates the route
r:ihc theory ofpercep- described in Jules Vernes' Around the Woild in Eighty Days into scents in a manner
:r ultisenso ry or ganiza- reminiscent of the Shirakawa Border Station version of the Japanese incense cere-
:trint. \Ve have further mon¡ and, Helen Ganly and Heather Peers' The Radclffi Camera as a Celebration Cake
:t. Llur comprehension qnd ItsWorkbook,which, when properly savoured, recalls to mind the practice of tast-
hundredfold, if
a ing sights which is so central to the Indian theatrical tradition. It is true that neither
':-.ed
e:etotbre unsuspected of these installations is as interactive or transformative as the non-Western examples
:l¡a¡ins scents (kodo) to which I compare them here, nor are they rooted in social practice (that is, in a par-
'rr¡s through the envi- ticular lifestyle or culture), but they do open the perceiver to the experience of inter-
:e senses' (to borrow sensoriality, and therein lies their value.
e. :epresents a techno- What is more, the works by Macia and Ganly and Peers help attune us to a dif-
ferent way of perceiving art, of which art historians should take note. Rather than
-].rt of Sense,' I held a conceive of themselves as students of 'visual culture', art historians ought to expand
r.e,.-er mind simulate their frame of reference to include the 'sensual culture' of which the visual is but one
s :hat current research dimension. Our appreciation of the art of the Symbolists or Futurists stands to be
rugqests may be latent enhanced considerablyby adopting such a multimodal approach, as Classen (1998) has
..ption of the absence shown. And even the drive toward abstraction in the art of the later twentieth century
r--',i. atter having seen is best understood using a multimodal approach, given that the impetus behind many
iter rierv. In his pre- ofthe earliest (and still finest) instances ofnon-representational art (e.g. the paintings
iiiiL-lus exp eriments he of Wassily Kandinsky or Georgia O'Keefe) was the aspiration to create 'visual music'."
r s"'stematically trying This brings me to my final point: when doing art history, whether one's passion is
betrer, coliideroscope for painting or music or architecture, etc., it is surely advisable to cultivate (at least)
:,ieed, Charles Spence two senses, rather than one.
::r.,' \\ as the Syrnbolist
: -nterested in patterns
:-i rlPposed to the har-
s-i.e scope of Spence's
and McDonald
"pe nce
ras brought us to the
AcxuowLED GEMENT s
-; the revolution in lin-
l.:-iiid-i 1959, again with I wish to thank my research assistant, Elena Papadakis, for supplying me with many
i ie Saussure confined sensational sources, and Constance Classen and Andrew Irving for their helpful
;nson- field-namely, comments on earlier versions of this paper. I am especially indebted to Francesca Bacci
and David Melcher for organizing the 'Art and the Senses' conference, and creating
,.: behind the 'science such a stimulating forum for the exchange of ideas. This paper is based in part on
:n horv his notion of research made possible by a grant from the Social Sciences and Humanities Research
:s:b1e quaiities' can be Council of Canada.
r¿:ige of non-Western
:, ir 1\ithin the discrete
- o: the more innova-
lelsory plenitude that '" If you think the critique of Kant presented here is overly harsh, see his withering critique and
dismissal of any aesthetic vocation for the senses of smell and taste (Kant zoo5).
" The history of s1'naesthesia in art and music since rgoo is traced by Brougher et al. in Visual
Music (zoo5); for a multisensory critique of their argument see Howes zoo6b.
18o DAVID HOWES
RnpnnBNcns Kirshenblatt-Gimb-e
- l
Marinetti FT (r9S9 . -
Marinetti FT (zoo; . l
Argüe11es IA (rytz). Charles Henry and the formation of a psychophysical aestherlc. University of
Chicago Press, Chicago, IL. Morita K (zoo7). ¡;"-- :
Baron-Cohen S and Harrison J, eds. (t997). Synaesthesia: Classic and Contemporary Readings. OsborneH(r98-i.T,
Blackwell, Oxford. Educatíon, r8, _11-r
Bedini S Ggg+) . The Trail of Time. Cambridge University Press, Cambridge. Partan S (zoo4). ,\Í,r.:-
Bernstein L, Auer E Jr, and Moore | (zoofi. Audiovisual speech binding: convergence or asso- eds. The Handb:'¡;.
ciation. In G Calvert, C Spence, and BE Stein, eds. The Handbook of Multisensory Processes, PinardS(t99r).A,a':.
pp.202-23. The MIT Press, Cambridge, MA. ed. TheVarietie: .'
Brougher K, Strick J, Wiseman A, andZtlzcer J (zooS). Visual Music: Synaesthesía in Art and Pybus D (zoor). ñ...;,-
Music since ryoo. Thames & Hudson, London. Ramachandran VS. l_
Calame-Griaule G (1q86). Words and the Dogon WorlcJ,D La Pin trans. Institute for the Study and the foundati-'.
The Hsndbook ¡' l.!
of Human Values, Philadeiphia, PA.
Calvert G, Spence C, and Stein BE, eds. (zoo4). The Handbook of Multisensory Processes. The Redfield, Robert ry-,
Press, Nerv York.
MIT Press, Cambridge, MA.
Rée I (zooo). The 'l=.-
Classen C (rqqo). Sweet colors, fragrant songs: sensory models of the Andes and the Amazon.
A m eri ca n Et h n ol ogi s r, V, 7 zz-35.
Vlew. Serpent's ;.-,
Classen C (r9y). Worlds of Sense: Exploring the Senses in History and Across Cultures. Routledge,
(rg;g . _-
Saussure F de
Eagleton T (rqqo). The Ideology of the Aesthetic. Blackwell, Oxford. Sur M (zoo4). Rei.,'-:,
Elkins J (zooo). How To Use Your Eyes. Routiedge, New York. development anc _..
Gebhart-Sayer A (r985). The geometric designs of the Shipibo-Conibo in ritual context. Journal Multisens o ry Pr¡ ;, -.,,,
HowesD(zoo3). SensualRelations:EngagingtheSensesinCultureandSocialTheory.University
of Michigan Press, Ann Arbor, MI.
Howes D, ed. (zoo4). Empire of the Senses: The Sensual Culture Reader. Berg, Oxford.
Howes D (zoo6a). Scent, sound and synesthesia: intersensoriality and material culture theory.
In C Tilley, W Keane, S Küchler, M Rowlands, and P Spyer, eds. Handbook of Material
Cultur e, pp. t67-7 2. Sage, London.
Howes D (zoo6b). Cross talk between the senses. The Senses 6 Society, r, 381-90
Illius B (zooz). Ventana hacia el infnito; arte shípibo-conibo. lnstifttto Cultural Peruano
Nortamericano, Lima.
Kant I (zoo5). Objective and subjective senses: the sense of taste. In C Korsmeyer, ed. The Taste
Culture Reader, pp. 2c9-74. Berg Publishers, Oxford.
Kirmayer LJ Q.ool). On the cultural mediation of pain. In S Coakley and K Shelemay, eds.
Pain and its Transformations, pp. 363-4or. Harvard University Press, Cambridge, MA.
HEARING SCENTS, TASTING SIGHTS 181
'. ! .,i::sertsory Processes, Pinard S (r99r). A taste of India: on the role of gustation in the Hindu sensorium. In D Howes,
ed. The Varieties of Sensory Experience. University of Toronto Press, Toronto.
',':;.tillesia in Art and Pybus D (zoor). Kodo: The Way of Incense. Turtle Publishing, Boston, MA.
Ramachandran VS, Hubbard EM, and Butcher PA (zoo4). Slmaesthsia, cross-activation,
:.:i¡ute for the Study and the foundations of neuroepistemology. In G Calvert, C Spence, and BE Stein, eds.
The Handbook of Multisensory Processes. The MIT Press, Cambridge, MA.
, -:.. :,. rr, P r o cess¿s. The Redfield, Robert (t97t). Art and icon. In C Otten, ed. Anthropology and Arf. Natural History
Press, New York.
r:¡s ¿nd the Amazon. Rée J (zooo). The aesthetic theory of the arts. In P Osborne, ed. From an Aesthetic Point of
Vlew. Serpent's Tail, London.
Saussure F de (rqSq). Course in generallinguistics, W Baskin trans. McGraw-Hill, New York.
l'.,i tu r e s. Routledge,
Schechner R (zooi). Rasaesthetics. The Drama Review,45, 27-50.
Social Science
Smith BR (zoo$. Listening to the wild blue yonder: the challenges of acoustic ecology.
In V Erlmann, ed. Hearing Cubures: Essays on Sound, Listening and Modernity, pp. 2r-4t.
Berg, Oxford.
r-
:,.;i¡¿ric I nagination.
Spence C and McDonaIdI Qoo+). The cross-modal consequences of the exogenous spatial
orienting of attention. In G Calvert, C Spence, and BE Stein, eds. The Handbook of
'.:lsrbilities and indig- Multisensory Processes. The MIT Press, Cambridge, MA.
-i.':-.rl'le Obiects. Berg,
Spence C and Zampini M (zoo6). Auditory contributions to multisensory product perception.
Acta Acustica, 92, 7-L7.
: -. ::' Stnell. Routledge,
Stevenson RJ and Boakes R (zoo4). Sweet and sour smells: learned synesthesia between the
senses of taste and smeil. In G Calvert, C Spence, and BE Stein, eds. The Handbook of
-r,-i;ture. In D Howes,
Multisensory Processes. The MIT Press, Cambridge, MA.
Sur M (zool. Rewiring cortex: cross-modal plasticity and its implications for cortical
development and function. In G Calvert, C Spence, and BE Stein, eds. The Handbook of
Muhisensory Processes. The MIT Press, Cambridge, MA.
:, :ifu¿1 context. ]ournál
Turner B (rSS+). Introduction. In C Buci-Glucksmann, Baroc1ue Reason: The Aesthetics of
' --S5. Modernity. Sage, London.
-::
i , ::,;i Tlteory. University
-. 3;rg, Oxford.
I ::::rerial culture theory.
. ;i,'.,ldbook of Material
:. r. -rSr 9o
.-,-t. Cultural Peruano
INrno r
\\¡ny are taste anú s:--
and convince |ou r-. ::
Taste and smei, ':
chemicals in our ic ,,:
long time in evolu:r'-:
still retain SoIDe t¡f i:-¡
senses have access :,-
thev influence mtr...i -
encountered that s:r¿
:rigger involunta:', :.
..omething. The tr^;.- ..
ttr provide inforna:'
',tarning system. p:,-,:
nore to it than th¡i.
:¡metimes referrec l -
r:lgesting poisons.
CHAPTER 10
THE SCIENCE OF
TASTE
AND SMELL
TIM JACOB
Svrsrr Cribriforr :
What is smell?
Smells, or more scientifically'odorants', are airborne chemicals. We take them into
the nose as we breathe and they dissolve in the mucus at the roof of the nostrils. In
order to be detected by our olfactory system these odorants need to have certain prop-
erties. For instance, they must be small enough to be volatile (less than -3oo relative
molecular mass) so that they canvapoñze, get airborne, reach the nose, and then dis-
solve in the mucus. This tells us that smell, unlike taste, can signal over long distances,'
enabling it to function as an early warning device. The olfactory system is an early
evolutionary adaptation and hasn't changed much between the fruitfly (Drosophila)
and mammals. The fly has an olfactory system with a very similar organizationto that
of the mouse. This similarity between the olfactory systems suggests that a very effi- '\
cient solution to the problem of detecting and discriminating small molecules emerged
over 5oo million years ago, the evolutionary distance between these two species, and Fig. 10,1 i
has not required much modification since. of the ffc -
Of course, the human olfactory system does more with the information it receives epithelir -
than the fly. Our nose samples our environment for information, continuously test- weil as o:'
ing the quality of the air we breathe (this will alert us to potential dangers, e.g. smoke, the muc-s.
decomposition) as well as informing us of other relevant information, such as the the bore'i:
presence of food, the familiarity/novelty of the ambient smell and its congruencywith
the surroundings, or the identity of another individual.
Olfactory nerves
Olfactory epithelium
:: .. \\-e take them into
Mucus
: -, -.i oi the nostrils. In
:-: h¡recertainprop-
,;¡.
rhan -3oo relative
' -.: :!)ic, and
then dis-
:-.- -'i er long distances,'
.. r-, sl stem is an early
,.. :: *itrlr' (Drosophila)
-: :.¡nization to that
-r-:.:\ that a very effi-
''
-- :lolccules emerged
.:-;s: ¡fo Species, and Fig. 10.1 A section through the nose showing the elements
of the front end of the olfactory system. The olfactory
--: : :mation it receives epithelium contains olfactory receptor neurons (ORNs) as
. -.. .rrntinuously test- well as other cell types (see Fig. 10.3), which extend cilia into
-.- ;:rgers, e.g. smoke, the mucus. ORNs have an axon that projects through
- ::t:lion, such as the the bone (cribriform plate) to the olfactory bulb'
-: ris congruencywith {source: httpl/www.cf.ac.uk/biosi/staffinfo/jacob/teaching/sensory/olfactl.html)
to damage from environmental exposure (to oxidizing agents' pollutants) and are
replaced every 2 to 3 weeks. City-dwellers tend to have poorer senses of smell than
: i':¡teln 'taxi service' country folk because of the damage done by pollution (e.g. sulphur and nitrogen
:=rl binding proteins oxides from car exhausts).
::it:s to their correct
',,,3 ere ready
for the
--.;:on- epithelium at The importance of smell
i:: ,-rn
the membrane
l--:se cilia, which are The importance of smell can be appreciated by understanding what is Iost by someone
:. ::i cularly vuinerable who loses their sense of smell later in life.' Smell permeates every aspect of our lives,
itis the backdrop of our every waking experience and it contributes towards the plea-
sure of eating and drinking. The familiarity/novelty of smells in any situation relays
information to us. Ii for example, upon returning home you smell an unfamiliar
:::- ¡j in parts ofEngland
::- :]t: ilcross the smell you will be immediately alerted and investigate it in an attempt to identif. the
r :r:rreh'
. described as source. We look for congruency in smelis-things should smell as we expect them
: .:-:ireds of calls f¡om to. Pleasant, familiar smells are relaxing and, under the right conditions, this can be
: - ::t.. ad mOre than
',!:: :tiher agricultural or
' Anosmia is the medical term.
186 TIM JACOB
extend the 8-20 olfactory cilia. Basal cells are stem cells for ORNs as well as other
epithelial cells. Supporting cells contribute to the composition of the mucus layer as
well as supporting the ORNs metabolically. The odorant receptors are situated on the
cilia. The axon from the ORN projects to the olfactory bulb.
(Source: http://www.cf.ac.uk/biosi/staffinfo/jacob/teachingisensory/olfactl.html)
used therapeutically. Smell is associated with memory, particularly with our emo- Cingulate gy'.s
tional memory. It is integral to relationships-there are biological mechanisms at play l
which influence our choice of partner based on their smell, and smell is an impor-
tant factor in sex. A loss of libido can follow loss of smell. We gain information about Olfactory bulb ---_=-'
ourselves and others through the medium of smell. We communicate emotions
(fear, happiness) via smell and many aspects of health are related to smell. We tend to Septurr /
avoid smells we find unpleasant and we find them unpleasant for good reason-they
may be hazardous, poisonous, or warn us of death, decay, disease, and unhygienic Pyrifc'r :
conditions. Furthermore, there is the whole universe of perfume, fragrance, and the Álr¡ :
associated memories. Imagine a world without this olfactorv dimension.
Where the nose leads the rest of us follows. More than any other sense our sense
of smell determines the emotional tone of our every experience.
" : :ensory/olfactl.html)
Dentate gyrus
Fig. 10.3 A section through the brain showing the main structures of
the limbic system (gray-shaded regions)-amygdala, cingulate gyrus,
dentate gyrus, fornix, hippocampus, hypothalamus, mamillary body'
: r.ufoanatomy of the pyriform cortex (outside the plane of this section) and septum.
: ,,Jours, and attempt (o Tim Jacob 2007).
I88 TIM IACoB
first girl/boyfriend,a smell that can take you right back and evoke a surprising amount
utu>>-L
of detail about the person as well as the emotions you experienced at the time.
Circum,,
Tesrn trU:i,
Introduction to taste
Fungiforr:
We have particular taste sensitivities to those foods and minerals for which we have
an absolute requirement. Thus, we have a sweet receptor because we need carbohy-
drates for energy, an umami' receptor for amino acids (the building blocks for pro-
teins) for growth and repair, and a salt receptor because we must have salt (sodium
chloride). The appetite for a particular food, when satisfied, causes the brain to release
a reward signal and also to suppress the desire for that food. This can explain why
chocolate is so seductive. This reward system worked well in the past when Homo
sapiens had to work hard to find carbohydrates, but now we are surrounded by them
Fig. 10.4 A diagra'n o-
it encourages obesity.
of taste papillae. Fungif
The sour and bitter tastes evoke an aversive response and protect us by prevent-
appear as little red do:'
ing ingestion of acids (sour) and poisons (most poisons taste bitter).
back of the tongue anc
output from the papi a
nerves 7 and 9). Cran a
Taste mechanisms from the foliate and cl'
central region of the t3
The five tastes each have a receptor mechanism that has been identified genetically,
and the mechanisms for translating taste into nerve impulses for the brain to inter-
pret is becoming much better understood. Tastes are detected by taste cells within
taste buds and these taste buds are situated on papillae distributed over the surface rvarns of danger and :r
of the tongue (Fig. ro.4). Those little red dots scattered over the front of your tongue taste bitter. Horvever. ¿l'
are taste papillae. Each taste bud contains a number of taste receptor cells, each ple. you can taste ¿ . :
expressing different receptors and they will therefore be sensitive to a number of tbrm papillae, so-call. :
different tastes. Although the 'taste maps' one sees in text books and reproduced in mainly on the fronr , i :
numerous articles are wrong,a it is true that we are more sensitive to bitter taste at the predominantly sen:i:'. :
back of the tongue. It is for this reason that mums and dads try to spoon the unpleas- of the tongue with n.r r.
ant bitter-tasting medicine directly down the throats of their reluctant offspring- rte swallow, lood ir: r-
skipping the tongue altogether to avoid activating the bitter detectors. Bitter taste rrhere there áre firor.' rl:
r Umami (pronounced óó-má'mc) was first identified by Kikunae Ikeda at the Imperial University
of Tokyo in r9o9. It has only recentlybeen adopted in the West following identification of its
receptor. Supertasters
a This mlth originated from the misinterpretation of a PhD thesis written in German by Hanig in
r9or. His thesis included a graph oftongue regional sensitivity with no scale on the y-axis. Another -\bout z5olo of the pt p *
scientist later took the reciprocal of these values greatly exaggerating their differences. .o the chemical ri-pri:
THE SCIENCE OF TASTE AND SMELL 189
Circumvallate
Fungiform
:::1. ir
rvhich we have
---j: need carbohy-
\re
.: -rl biocks lor pro-
:r*.: h¿r-e salt (sodium
--,.. .he brain to release
- . ;an explain why
.- ::3 past tyhen Homo
.:. .r"-:rounded by them
Fig. 10.4 A diagram of the tongue showing the location of the three main types
of taste papillae. Fungiform papillae are located at the front of the tongue and
r: lril us by prevent-
appear as little red dots. Foliate and circumvallate papillae are situated at the
...;:
back of the tongue and this region is more sensitive to bitter and sour tastes. The
output from the papillae is carried to the brain by two separate nerves (cranial
nerves 7 and 9). Cranial nerve 9 carries information about bitter and sour taste
from the foliate and circumvallate papillae. There are no taste papillae in the
central region of the tongue leading to a taste 'blind spot'.
. ;¿ntiÍred genetically, (source: http://www.cf.ac.uk/biosi/staffinfo/jacob/teachingisensory/taste.html)
i : : rhe brain to inter-
:* ri-titste cells within
,:,,-¡ed over the surface warns of danger and most drugs are poisons (especially at high concentrations) and
:. -:ont ofyour tongue taste bitter. However, all taste buds are sensitive to all tastes to some extent; for exam-
,.: receptor cells, each ple, you can taste a coffee granule (bitter) on the front of your tongue. The fungi-
l, :1."'e to a nUmber Of form papillae, so-called because they resemble button mushrooms in shape, are found
:... and reproduced in mainly on the front of the tongue and, while they are sensitive to all tastes, they are
.,
: :o bitter taste at the predominantly sensitive to salt and sweet tastes. There is a bald patch in the centre
, i:, spoon the unpleas- of the tongue with no papillae. We are therefore 'taste blind' in this region but, when
: :;-uctant offspring- we swallow, food in this region is brought into contact with the roof of the mouth
' rii.ctors. Bitter taste
where there are more taste buds.
genetic basis and results in a greater density of fungiform taste papillae, and therefore
Smell and taste re
taste buds, on the tongue which can easily be seen with the aid of a magniS,ing glass.
Another z5o/o of the population cannot taste n-PROP at all. More recent studies We are sensitive ro ..:
(Lim et al. zooS) have shown that this sensitivity generalizes to other tastes (e.g. sweet) hundreds ol differer:r ---
and supertasters tend not to like green vegetables or fatty foods, and there have been even more funclion', :
suggestions that this might influence dietaryhabits. instance, a bloodhou: -
to know in which d,:.-
structure and sensir:', -
may be tailor-made : :
Central pathways
only five. These are tr: .
Information about taste must then be transmitted to the brain, in order for per- for which there ar< ,.-
ception of taste to occur. The first stop is the medulla, the lower part of the brain stem as [at, metallic, dr':- ...
in Fig. ro.3. From there the information is relayed to (r) the somatosensory cortex remain obscure.
(postcentral g¡rus in Fig. ro.5) for the conscious perception of taste, and (z) to the
hypothalamus, amygdala, and insula (a cortical region just below and in from the
postcentral gyrus in Fig. ro.5), giving the so-called 'affective' component of taste.
These latter determine the behavioural response, e.g. aversion, gastric secretion,
feeding behaviour. It is also these regions that are responsible for the protective func-
tion of taste-its 'last gatekeeper' role-and if we ingest poison a vomiting reflex is
Wner enl
triggered.
Smell and taste as
.al/{ /4¿
/
:otentially harnrrul - -.
'wI
-.ur ofbabies (Canc .:
r/, ----Lr-
:rpressions sugge)r.:.
l:fears to be an ir:::- -
- -
one of the cortical regions where taste information is processed, the i',nre r€seárch:u.--
other being the insula, just down and in from this region, which is .r.ir e smells. Or.-.:,
linked to the affective component of taste.
(oTim Jacob 2007. Source: Wikipedia, with permission_)
. dug detecr. rh- .
THE SCIENCE OF TASTE AND SMELL 191
. ::: ;, with permission.) t The dog detects the concentration of scent on the trail-the more recent, the more concentrated.
I92 TIM JACOB
when given fish or rotten egg odour (Steiner ry77).This is an area of some controversy Your unique smell
since it is difficuit to assess the meaning of a day-old baby's facial expressions and
they could just be responding to the intensity of the odour rather than the quality. Our olfactory sysrem ..
The alternative hlpothesis is that response to smell is learned-and there is much sup- possibilities, it serr cr ,
port for this idea including work that shows babies can learn odours in utero (Schaal and can recognize anc'¡-
et al. zooo). interactions. For a ta:,.-
Babies born to mothers eating 'anise' showed a preference for anise odour com- smell I recommend P:-:.
pared to those born to non-anise consuming mothers, which displayed aversion or was born with no snri
neutral responses. However, one has to ask the question of what use exactly is a warn- of course, to per:ona1 1.".
ing system in which one has to learn which are the hazardous odours? By the time by our immune s\ slin'.
one has learned it may be too late! by our HLA lhunran ; -
There are some cross-cultural differences in the appreciation of smell. Wintergreen an unique smell, r'itl: .-
is considered to be a pleasant smell in the United States whereas it is unpleasant in identical twins apart :: -.
the United Kingdom (Cain and fohnson 1978). These cross-cultural differences our personal smell. B.--,
extend to the odour of people; for instance Western Europeans, and in particular distinguish them from : ,
the British, are perceived by Hindus as smelling of butter/milk because of their large gender siblings and pa:::
intake of dairy products. The fapanese have very few apocrine glands in their axillae interesting that a conr-r
(Kuno r956, p.4r7) and the smell ofbodyodourwas, until recently, grounds for dismissal mented on the linl l-. ,
from the lapanese army. Europeans smell strongly of body odour to the Japanese Clara Ursitti exhibirc' :
although they arefar too polite to say so.
Flavour
Many people mistake flavour for taste, indeed one chef misguidedly insured his
taste buds for one million pounds. Flavour is mainly smell Although it does include
contributions from the sense of taste (as well as touch (mouth-feel), temperature,
pH,u and moisture) it is mainly smell (zl-goolo). Test this for yourself by eating or
drinking something while holding your nose.
Smell is vital to our enjoyrnent of food and drink. Without smell the experience
of eating and drinking is reduced to the five basic taste qualities, namely: sweet, salt, F-rn
sour, bitter, and umami (savoury).
Smell is important to our appetite-when we are hungry, the brain sends infor-
mation to the olfactory bulb, removing inhibition which acts to sensitize our noses
to smells, and food smells are particularly attractive. However, the situation changes
radically upon satiation. When our stomach is full, odours that we found pleasant a
short while before, are now unpleasant or even repulsive to us.
Fig. 10.6 C
6 pH is a measure of acidity-alkalinity. Strong acids have a low pH
THE SCIENCE OF TASTE AND SMELL 193
She also famously authored an installation called Blll, presented in the wake of the bodily scents before :;::',
Monica Lewinski scandai, featuring an empty room filled with the scent of semen. exhibited at MOtrf.\'s i
Mothers learn to recognize the smell of their babies; 9o7o of women tested iden- dating agency aimeJ .r
tified their babies by smeil alone after only ro minutes to r hour's exposure to their Olfactory communii;.-
baby (Kaitz et al. zoo4). All of the women tested recognized their babies' smell after
exposure periods greater than r hour. By as early as 3 days a baby can discriminate
between a gavze pad worn by his or her own breastfeeding mother and that worn
by another lactating mother used as a control. The smell of newborn babies always
features in the top ten of the most popular smells in surveys. Babies need to use all
means at their disposal to attract favourable attention from their mothers.
1..C in the wake of the bodily scents before they actually meet. According to the artist this protoq?e, recently
:i; scent of semen. exhibited at MOMA's 'Design and the Elastic Mind' show, is a'proposal is for a blind
,: ,,,,'omen tested iden- dating agency aimed at individuals wishing to meet a suitable partner for procreation.
, u: s e\posure to their Olfactory communication is given precedence over visual stimuli.'
:,r;:¡ babies' smell after
::bi can discriminate
:-:-.ther and that worn
:--.'.'t'orn babies always
E:ries need to use all
:-: I-trrtherS.
.,,
r-le others they find
: *:l; j\ Ste m. InCfeaSing
l,l-{C senes
-called
:- *: ¡nd mating pref-
i. ::o¡e dissimilar the
.... . Ir has been shown
,'. ¡¿-ated to their HLA
: :r,rr¡ated that women
--,',",'r. The smell from
..r:roie. the odour of
-: t¡eir orvn actual or
r::i :rat the HLA genes
Menstrual synchrony
One of the most often-quoted examples of smell signalling in humans is menstrual
synchrony. McClintock (r97r) found that female college students, who spent signifi-
cant amounts of time together, showed synchrony of their menstrual cycle, and
attributed this synchrony to odours (pheromones). A few years later this finding
was reinforced by another study (Stern and McClintock 1998). Sweat samples of five
women with regular z9-day cycles were taken daily. These donor samples were applied
to the upper lips of the female test subjects three times a week for 4 months. By the Fig. 10.8 Sissel Tolaas
end of the test period, test subjects menstruated significantly more often at the same
time as the donors than subjects in the control group. It was suggested that menstrual
synchrony, which also is indicative of ovulatory synchrony, was controlled by phero-
mones. In a parallel study, the influence of male odours on the menstrual cycle was Smell indicates rei'. .
tested. Odour samples of male axillary secretions were again applied to the upper lips
Humans are cap.rl-..
of female test subjects. Those who were not sexually active had irregular menstrual
¡ues alone. These ci.*-
cycles at the beginning of the experiment. After 4 months the mean cycle length was
:en and testosteronc. i:
29.5!3 days in a majority of the test subjects. This strongly suggested that male phero-
.ecretions may be : . .
mones have a regulatory effect on the menstrual cycle.
r.ould be highest in:-
Fear and happiness ¡leasant during or ul,.
ire)when they arc r :
Dogs and horses can smell fear in humans. Work by Chen and Haviland-lones
(rggg) has demonstrated the ability of underarm odour to influence mood in others.
:oor). In lower nla::::'.
properly timed m¿ri'.-
Karl Grammer and colleagues have recently demonstrated that the smell of fear can
rhe situation in hu¡::-
be detected (by women) in the armpit secretions of people who watched a terrifliing
film (Ackerl etal. zooz). The implication of this work is that a chemical signal is secreted
.
'rulation, is conc."-. -
in sweat which communicates the emotion. So far, fear and happiness are the only 'Jourchangesth¿r, ¡:-
lr-rctive signalling nr;, - -
two emotions to have been shown to be communicated in this way and women are
more sensitive than men at interpreting the chemical signal.
Fittingly, artist Sissel Tolaas' best known work is titled the FEAR of smell - the smell
of FEAR (zoo5 to present: Fig. ro.8). The installation presents synthesized human Changes in smell ¿
-ru:lans is menstrual
::s. iiho spent signifi-
renstrual cycle, and
:¿::i later this finding
lirrat samples of five
: s:nples were applied
-:,r + nonths. By the
--
- :; olten at the same
Fig. 10.8 Sissel Tolaas, the FEAR of smell - the smell of FEAR,2005 to present.
(Photograph: courtesy of the artist.)
:.¿::ed that menstrual
¡ ¡ -.ntrolled by phero-
: renstrual cycle was
Smell indicates reproductive status
:,:¡d to the upper lips
; -rreqular menstrual Humans are capable of discriminating between males and females by olfactory
:;:n o'c1e Iength was cues alone. These chemical cues are molecules derived from the sex hormones oestro-
gen and testosterone. Sex differences in the composition of human axillary (armpit)
-.,:d that male phero-
secretions may be the basis for such discrimination. In women, these compounds
would be highest in those of reproductive age. women's body odour is rated as more
pleasant during on¡lation and men tend to rate women more highly (i.e. more attrac-
::J Haviland-Jones tive) when they are exposed to secretions produced by ovulating women (Kohl et al.
:r;e mood in others., zoor). In lower mammals, one function of chemical communication is to produce
,re smell of fear can properly timed mating, and oestrus is signalled by visual and odour cues. However,
, ',,.'¿rched a terrifiring g: the situation in humans is more complicated and a woman's most fertile period,
:r:cal signal is secreted ovulation, is concealed. It has therefore been proposed that these secretions and
::piness are the only odour changes that occur at ovulation are part ofan elaborate and unconscious repro-
,','ar and women are ductive signalling mechanism (Kohl et al. zoor).
:,ii sntell
¡¡.i
- the smell
) si'nthesized human Changes in smell and smell sensitivity
r:rnt on the gallery
-:r nine men expe- Effect of hormones
r ..lours are released women's sense of smell peaks at ovulation, coinciding with a surge in the hormone
,. Tolaas intended to oestrogen. It has been suggested that this enables them to be at their most sensitive
it; Lr\ er-Valued visual to the smell of any potential mate (i.e. their suitability-see section 'Smell and mate
selection') when they are at their most fertile (Kohl et al. zoor).
198 TIM JACOB
There have been many stories about smell changing during pregnancy when hor- into every bodilv o¡.
mone levels are changing dramatically. Most pregnant mums have some recollection ened and peeled oll-. t
of strange alterations to smell and taste. Leslie Cameron (Cameron zooT) studied these smell'; diphtheria 1:a,
effects and found that, consistent with previous reports, 9oo/o of pregnant women and the sweet-smelir
reported that specific odours smelled less pleasant and 6o0/o reported that some was said to attend :.
odours smelled more pleasant. Although nearly two-thirds of pregnant women rated so now. This phenc,:
their olfactory sensitiüty to be enhanced during pregnancy and overall pregnant electronic noses h¡-
women's self-rated olfactory sensitivity was higher than that of controls, self-ratings the breath.
did not correlate with measured scores. The results suggest that her findings and those There have beer ,
of previous studies may reflect the fact that the effect of pregnancy on olfaction is now researchers ha'. ¡
small and inconsistent. dogs have been tra,:
ments verified the ;,.
Changes in sensitivity to certain hormones dogs with onlr' a ie ,'.
{¿ between breath
We can change our sensitiüty to certain hormones (derived from testosterone and s¿::r
oestrogen) found in our sweat by repetitive exposure. This was first demonstrated (McCulloch et ai. : -
by Wysocki et al. (i989) when one of the experimenters (C.].W.), who was anosmic trained to target r ):
(unable to smell) for androstenone at the beginning of one of their previous studies, BEES(zoo7),as.r,.
found himselfbecoming sensitive to it as the trial progressed. It was thought that certain fertility status thr tr ; I
people, who were unable to smell this compound, had a specific anosmia. However, diagnostic instrur.
they went on to demonstrate that this was a general phenomenon and people could aestheticaliy pleasi:.
'learn' to smell it by repetitive exposure (sniffing a sample for 5 minutes three times a
day). jacob and colleagues (Wang et aL. zoo1;Jacob et al. zoo6; Boulkroune et al. zooT)
have shown that this also occurs with other related compounds and that, as sensitiza- Smell and stan
tion occurs, the smell quality of the hormone changes and the electrical response of
the brain increases in a gender specific manner. These compounds are not Yery During stan'atit-rr. l.
volatile and we will only detect them on someone else by getting close to them. This down to acetone. Kr
will happen most often in between mothers and babies and couples. Because of this it of the energv ior ::
has been suggested that these compounds may play a role in bonding in some presently metabolism of k¿:'--i
unknown manner (Boulkroune ef aL zooT) . the breath can sm.-
starved themselle:
Specific anosmias tics smelled s\\e3t ¿l
Some people cannot smell skunks, others cannot smell fresias. These specific anosmias probably explairei
led Iohn Amoore on a wild goose chase in the r96os searching for the'primary' smells holiness as foui s¡r;
from which all other smells were formed, analogous to vision where the colours of the Saint Teresa of -\'' -':
visible spectrum are formed by the differential of activation of green, blue, and red \laravillas of Iesu. -
cones. We now know that this is not how smell works, but these interesting anoma- ¿manated from P¿;
lies remain. The current thinking is that smell receptors detect molecular features of r¡hen his bodr",',';¡
odorants. One odorant will therefore activate a number of receptors and one receptor nain ingredient. '-,:
will be activated by many different odorants. The perceived smell is derived from the santalol which h¿s :
pattern of receptor activation.
p:egnancy when hor- into every bodily opening, including the mouth and eyes. When these lesions black-
1¿1'e some recollection ened and peeled off, they emitted a sweet and pungent odour known as the'smallpox
o:r :oo7) studied these smell'; diphtheria has a mousy smell. An acid smell is indicative of acute rheumatism,
: pregnant women
r-rf and the sweet-smelling breath of diabetics is well known. Curiously a sweet smell
: reported that some was said to attend the plague, but what was counted as sweet then might not be
iieEnant lvomen rated so now. This phenomenon of disease-related odours is being used to advantage, and
.nd or-erall pregnant electronic noses have been developed that can smell tuberculosis bacteria on
: ¡.¡ntrols, self-ratings the breath.
-rcr ñndings and those There have been anecdotal reports of dogs spotting cancer in their owners, but
fr:r.\- on olfaction is now researchers have proved this phenomenon scientifically. In the United Kingdom,
dogs have been trained to sniff out bladder cancer and the first controlled experi-
ments verified the claim (Willis et al. zoo4). In the United States, ordinary household
dogs with only a few weeks of basic 'puppy training' learned to accurately distinguish
:,:r:r testosterone and between breath samples of lung- and breast-cancer patients and healthy subjects
;¿. ñ¡st demonstrated (McCulloch et al. zoo6). Portuguese designer Susana Soares studied how bees can be
'¡-- . ¡,rho rvas anosmic trained to target a specific odour very easily, using Pavlov's reflex. She then produced
::r;L: prer-ious studies, BEE S (zoo7), a series of glass objects that contain bees trained to detect a woman's
:s l:loueht that certain fertility status through smelling her breath, and several other health conditions. These
-; ¿:rosmia. However, diagnostic instruments use insects as biosensors, while also being easy to use and
l:n ¿nd people could aesthetically pleasing.
:r'-rutes three times a
,: ;lkroune elal.zooT)
¿rd that, as sensitiza-
Smell and starvation
e,¿.rrical response of
::ll:rUnds are not very During starvation, fat gets broken down to ketone bodies which, in turn' get broken
:: ;-ose to them. This down to acetone. Ketone bodies are produced mainly in the liver and provide much
.:-::. Because of this it of the energy for the heart and brain during starvation. Acetone, derived from the
;:ng in some presently metabolism of ketone bodies, has a fruity smell and a sweet taste. During starvation
the breath can smell fruity sweet due to the presence of acetone. Mediaeval mystics
starved themselves to reach a state of altered consciousness. Reports that the mys-
tics smelled sweet and experienced a sweet smell and taste comparable to honey are
:ere specific anosmias f: probably explained by this phenomenon. Pleasant smells have been associated with
-,: the'primary' smells holiness as foul smells with sin. 'Heavenly scents' were said to have been emitted by
i:":e the colours of the Saint Teresa of Avila, Saint Thérése de Lisieux (known as'the Little Flower'), and Saint
: E:een, blue, and red Maravillas of Jesus immediately after they had died. The smell of roses is said to have
,.; r¡tereSting anoma- emanated from Padre Pio's stigmata. St. Casimir, Patron of Poland, died in 1483, and
::.¡,lecular features of when his body was exhumed 12o years later, a sweet smell issued forth. One of the
:'r,:,:s and one receptor main ingredients of incense over the millennia has been sandalwood. This contains
.,, :s derir.ed from the santalol which has a shape very similar to sex hormones.
and are interconnected by many nerve fibres. Odour exposure during a learning of seizures in epilep.:;.
task has been shown to increase memory when the same odour is present at the recall sniffing the conditi,'--.
stage (Herz r997)-useful in exam revision! associations tend to b¿
It is a common experience to have a vivid recall, or déid vu, evoked by a particu- forced associations .r:r.
lar smell. The experience with smell-evoked memory is unique among the senses in once formed, can be ¿'.'
the extent of the emotional, as well as factual, recall. The phenomenon has become
known as 'autobiographical memory', or the Proust effect, after the acco unlin Swann's
Way of the memories conjured while savouring the taste (he didn't actually refer to Smell conditioni¡
smelll) of a petit madeleine dipped into his tea. He was transported back to his Aunt
Leonie's house in Combray many years before and all the attendant memories. Artist Smell conditionins --:'
Oswaldo Macia's Smellscape, presented at the 'Art and the Senses' exhibition (Oxford (Epple and Herz r99¡,
zoo6), featured a series ofscents that would have been encountered by the character the presence ol- an ":
Phileas Fogg during his circumnavigation of the globe in Jules Verne's novel Around p other tasks to conr:..-,
s
the World in Eighty Days (Fig. ro.g). The installation comprised also of a map of the than a comparablc. .
world where the place names were substituted by names of scents' whose memory behaviour. What a.l- -r'-
had a strong association with the soul-deep identity of each place' also feeling this scn.....
Smell can be used as a conditioned stimulus by reinforced association. This occurs the experience oi '.:..',
by repeatedly pairing an odour with a situation or object. For example, learning to ¡n ambient odou¡. '.''-
relax to a chosen smell (aromachemical) has been shown to reduce the frequency on other tasks rv¡r --:,
can be used to e\cr- : -
Smell loss
\r'e lose our nic r : .-
:e
'rders experience k is
Fig. 10.9 Oswaldo Macia, Smellscape,20O6. Jetect dangers such ;.¡
(Photoqraph : Francesca Bacci) onnecticut Chenro..:,
THE SCIENCE OF TASTE AND SMELL
u:e during a learning of seizures in epileptics (Betts zoo3), and the onset of seizures can be prevented by
: l: present at the recall sniffing the conditioned smell. This is different from the Proust effect: Proustian
associations tend to be few, idioslncratic, and last a lifetime. In contrast, with rein-
,- :r'oked by a particu- forced associations one has some choice of the conditioning odour, and the effect,
:e among the senses in once formed, can be extinguished.
romenon has become
--heaccount in Swann's
;: jn't
actually refer to
Smell conditioning
¡ned back to his Aunt
lCant memories. Artist Smell conditioning can be used to modify behaviour. Rachel Herz and colleagues
i¿s' erhibition (Oxford (Epple andHerz ry99) gave children an apparently simple, but insoluble, maze in
::e:ed br. the character the presence of an ambient odour. When the poor kids were later presented with
rr-cfir€'S novel Arounel
other tasks to complete in the presence of the same odour, they performed worse
:. aso of a map of the than a comparable control group, indicating that the odour had influenced their
i.cents. rr-hose memory behaüour. What a shame to use a smell to influence behaviour in a negative way! Perhaps
also feeling this sentiment, Simon Chu (zoo8) gave underachieüng schoolchildren
s:Lr.iation. This occurs the experience of unexpected success at a paper-and-pencil task in the presence of
: :rample, learning to an ambient odour. When they later experiencedthe same odour again, performance
ieduce the frequency on other tasks was superior to that of relevant control groups, showing that odours
can be used to exert positiveinfluences on human behaviour.
Smell loss
We lose our sense of smell temporarily sometimes during a bad cold. We can't taste
anlthing because smell is such an important part of flavour (see 'Flavour' section).
Our sense of smell can be also lost, sometimes permanently, from a traumatic head
injury severing the olfactory nerves or a severe infection of the upper respiratory
tract (e.g. bad cold or flu). A total loss of the sense of smell is referred to as 'anosmia' and
partial loss as 'hlposmia'. Recovery following either head injury or infection damage
occurs in approximately a third of cases and can take from one to five years.
As we age we lose smell sensitivity, men faster than women. On the basis of a smell
identification test, this begins to happen in the 6os in men and the same loss occurs
about a decade later in women. By the time we reach our 8os, three out of four people
have some smell dysfunction (Doty and Laing zoo3). However, this figure includes
patients suffering from a variety of dementias that will affect olfactory ability. It is
thought that if subjects are prescreened for nasal health and dementias, the differences
between older and younger subjects are not that great (Murphy et al. ry97).
and the Warwick Olfaction Research Group (WORG) in 1999 (Van Toller 1999), prob-
lems with libido and relationships were found. Twenty-six per cent of the CCCRC
respondents mentioned some disruption of their marital, sexual, and social relation-
ships. In the WORG study, anosmics (those without a sense of smell) often stated that
their interest in sex had dwindled.
The emotional consequences of smell loss are often underestimated, particularly
in the medical community. Anosmics show varying degrees of depressive s)..rnp-
toms, such as feelings of helplessness, isolation in their condition, sad mood, loss of
independence, and fatigue. Depression is common among patients with smell dys-
function. One-fifth of a sample of anosmics scored in the clinically depressed range on
a depression measure, and one-third of anosmia patients reported extreme helpless-
ness comparable to that seen among chronic illness samples such as arthritis patients
(Ossebaard et al. rgqg).
exercise because they have lost the pleasure associated with food and drink. Eating is no
longer a recreational activity for anosmics (Van Toller 1999).
Smell is the foundation of our waking lives and also continues to function as we
sleep, influencing our dreams. We communicate though smell, consciously and
unconsciously, choosing our partners by their smell, advertising our genes with our
RrrrRrNcr s
body odour as well as with applied perfume. Smells can relax us, help our memory,
stimulate us and warn us of danger. Smell is integral to sex and, with taste, it exposes
us to the vast array ofsensory delights in food and drink. Both senses protect us from
danger and warn us of disease, infection, poor hygiene, and health risks. And if you
really want to know how important these senses are, ask someone who has lost them.
I would like to conclude with a reflection elicited by another artwork by Clara
Ursitti. The artist made a video (Untitled, t995, presented at the show 'Odour Limits'
at Esther Klein Gallery in Philadelphia) where she is featured laying naked while
perfumer and olfactory researcher George Dodd describes in great detail her odour in
different body parts, while sniffing her from her feet to the genital areas, to her axillar-
ies, and relating these scents to food and human behaviours (Fig. ro.ro).
THE SCIENCE OF TASTE AND SMELL 203
:.irmated, particularly
s '-.: depressive symp-
--Lr,r. sdd mood, loss of
1::.:rrs rvith smell dys-
l,',' d epressed range on
:t:d errreme helpless-
-;:: as arthritis patients
ü
I
This footage eloquently exposes and summarizes the cultural taboos and lexical
limits of our approach to odours. Despite their increasingly recognized importance
to our well-being, as explained in this essay, scents are still an undervalued and some-
;
what unexploited resource to the average person.
:e: to function as we
:ei. consciously and
rg {-rur genes with our
RErnnrNcEs
:-.. help our memory,
Ackerl K, Atzmueller M, and Grammer K (zooz). The scent of fear. Neuroendocrinology Letters,
. r,,ith taste, it exposes 23,79-84.
enses protect us from Bartoshuk LM, Dufly VB, and Miller I| (r99$. PTCIPROP tasting: anatomy, psychophysics,
:¿ith risks. And if you and sex effecls. Physiology and Behavior,56, rr65-7t.
:¿ ',,, ho has lost them. Betts T (zoo3). Use of aromatherapy (with or without hlpnosis) in the treatment of intractable
r.: artwork by Clara epilepsy - a two-year follow-up stttdy. Seizure 12, fi4-8.
s:r.-,rt'Odour Limits' Boulkroune N, Wang L, March A, Walker N, and lacob TJC (zoo7). Repetitive olfactory
exposure to the biologically significant steroid androstadienone causes a hedonic shift and
; -aring naked while gender dimorphic changes in olfactory evoked potentials. Neuropsychopharmacology, 32,
:¿: detail her odour in
r9zz-9.
:- a¡eas, to her axillar- Cain WS and ]ohnson F Jr (rgz8). Lability of odor pleasantness: influence of mere exposure.
. - r.10). Perception 7, 459-65.
204 TIM IACOB
Cameron EL (zoo7). Measures of human olfactory perception during pregnancy. Chemical Ossebaard CA, Taver r,','
Eggert F, Muller-Ruchholtz W, and Ferstl R (iSSS). Olfactory cues associated with the major Steiner JE (i924). Inn:i.
the New Y ork,\c'¡,', ¿''
histocompatibility complex. Genetica, to4, 797-7.
Epple G and Herz RS (rqqq). Ambient odors associated to failure influence cognitive perfor-
Steiner IE (l9zz) Fa:r:-
mance in children. D ev elo p m ent al P sy chob i olo gy, 35, to3-7.
related chemical .:-:---
Sweet Preferen.c. ::. -
Ganchrow JR and Mennella JA (zoo3). The ontogeny of human flavour perception. In
Stern K and McClin -¡ :.
RL Doty, ed. The Handbook of Olfaction and Gustation, z"d edn., pp.84-46. Marcel Dekker,
New York. 392, 177-9.
Gent JF, Cain WS, and Bartoshuk LM (rq86). Taste and smell measurement in a clinical
Yan Toiler S (1999 . -.,,.
Chemical Ser;-.e,.. z-+. -
setting. In HL Meiselman and RS Rivlin, eds. Clinical Measurement of Taste and Smell,
pp. lo7-16. Macmillan, New York.
\\'ang L, Chen L, anc -.:
Herz RS (1997). The effects of cue distinctiveness on odor-based context-dependent memory. response. lourrn,.t$
\t'edekind C and F..:. :
M emo ry and C o gniti on, 25, 37 5-8o.
Herz RS and Inzlicht M (zooz). Sex differences in response to physical and social factors for specific irlHC ¡ .:.
involved in human mate selection. The importance of smell for women. Evolution and
(London) B, 264. --
:-..-
-'t'iLlis
Human B ehavior, n, 359-6 4. Sl.l .
CM, Church
by dogs: prooi r.i:
Jacob TlC, Wang L, laffer S, and McPhee S (zoo6). Changes in the odour quality of androsta-
-
::-E :iegnancy. Chemical Ossebaard CA, Tayer WG, Nicassio PM, and Cain WS (1999). A mediational model of depress-
ion in smell disordered patients. Chemical Senses,24, 6o7.
:.: ¡dors. Physiology and Penn DJ and Potts WK (rqqq). The evolution of mating preferences and major histocompat-
ibility complex genes. American Naturalist, r53, t45-i.64.
. *:::ns. Clrcmical Senses, Pfaffmann C (lqZ8). The vertebrate phylogeny of taste. In EC Carterette and MP Friedman,
eds. Handbook of Perception, VlA, Tasting and Smelling, pp. 51-123. Academic Press,
:,.:-:::: Tlrcrapy, pp. 1-ro. New York.
Porter RH and Winberg J Oggg).Unique salience of maternal breast odors for newborn
--*n-ran olfactory func- infants. N eur o s ci en ce d. B iob ehav ior al Revi ew s, 23, 439_ 49
.;,':.;:-..".1 o.f Olfaction and Schaal B, Mariier L, and Soussignan (zooo). Human foetuses learn odours from the pregnant
mother's diet. Chemical Senses, 25, 729-37.
,',, ;i¡ted rvith the major Steiner IE Ggl|. Innate discriminative human facial expressions to taste and smell. Annals of
the New York Academy of Science,237,229-33.
,-,
-:::;e cognitive perfor- Steiner IE GglZ) Facial expressions of the neonate infant indicating the hedonics of food-
related chemical stimuli. In |M Weiffenbach, ed. Taste and Development: the Genesis of
: :-.i'¡ur perception. In Sweet Preference,pp. t7-BB. US Department HEW Publications, Bethesda, MD.
' !:,:-16. \Iarcel Dekker, Stern K and McClintock MK (rqq8). Regulation of ovulation by human pheromones. Nature,
392, 177-9.
.:i.]::rnent in a clinical Van Tolier S (rsss). Assessing the impact of anosmia: review of a questionnaire's findings.
'' :,'.. :,i Tnste and Smell, Chemical S ens es, 24, 7 o 5-t2.
Wang L, Chen L, and Iacob TIC (zoo+). Evidence for peripheral plasticity in the human odour
..:':- :ependent memory. response. Journal of Physiology, 544, 46-44.
Wedekind C and Furi S (rssz). Body odour preferences in men and women: do they aim
' .-;.- :nd social factors for specific MHC combinations of simply heterozygosity? Proceedings of the Royal Society
: r. -,:tin. Et,olution and (London) B, 264, r47r-9.
Willis CM, Church SM, Guest CM, et al. (zoo4). Olfactory detection of human bladder cancer
r *: :-u¿liñ'of androsta- by dogs: proof of principle study. British Medical Journal,329, 7t2-t5.
il : : Wysocki CJ, Dorries KM, and Beauchamp GK (rq8q). Abiliry to perceive androstenone can
É. :.:ognition of their be acquired by ostensibly anosmic people. Proceedings of the National Academy of Sciences,
86,7976-8.
: it: :L:'¡tones: integrating
L--r:iirrrl.
1-:i trl t¿ste and creami:
:::iime preferences in
r: :asic neurobiology.
'-::-Sl.
\larcel Dekker,
coI
PERI
COI{
FOO
MAY,
HES
.SOUND BITES':
AUDITORY
CONTRIBUTIONS
TO THE
PERCEPTION AND
CONSUMPTION OF
FOOD AND DRINK
CHARLES SPENCE,
MAYA TJ. SHANKAR, AND
HESTON BLUMENTHAL
Eating is the only thing we do that involves all the senses. I don't think that
we realize just how much influence the senses actually have on the way that
we process information from mouth to brain.
Heston Blumenthal (Tasting Menu, http://vr,'w.fatduck.co.uk)
CHARLES SPENCE, MAYA U. SHANKAR, AND HESTON BLUMENTHAL
Interestingly, nLlme:,-
INrnooucrroN ,': . -
theorize the totalitr
tion with symbolic n:r,:
Wn¡N thinking about the sensory systems that contribute to our perception and Looking into the le.. r.,
enjoyrnent of foodstuffs, people tlpically mention the senses of taste and smell. works such as Duch;:-.:
\Mhile it is certainly true that gustatory and olfactory (as these senses are more for- food and sexl, Piero ), 1,:
mally known) information contribute to our everyday food and beverage experiences ously cooked and sitr - :
(e.g. Rozin r98z), they alone do not constitute the whole picture. In fact, over the past sumed by the pubJi. . -
spiritual food), ltlarro 1,1
Zo years or so) researchers have been able to demonstrate that our perception of
food and drink is far more multisensory than many people had realized (see Duncker perlection vs. dec¡r . ::. -
1939; Moir rg¡6; Hall 1958; for early work in this area). Indeed, many researchers packaged food, and t.-
now talk about flavour perception in terms of perceptual gestalts (i.e. unified multi- a final poignant coml:rr
sensory perceptual experiences), consisting of the combination of all of the available food itselfl.
unisensory experiences, be they gustatory, olfactory, auditory, tactile, visual, and/or Even among the :r-..-
trigeminal (e.g. Delwiche zoo4; see also Spence, Sanabria, and Soto-Faraco, zooT).' rvork of the follorrir¡ ::
As Ferran Adriá, head chef at El Bulli (located near Barcelona), an establishment that adopted [o explore .:. .,
is frequently rated as the world's best restaurant, puts it: 'Cooking is probably the most 'rvho has cooked mc...- :
multi-sensual art. I try to stimulate all the senses' ('Does it make sense', zoo7, p. ry). Sonja Alhauser r ianr. --
Indeed, some chefs are starting to create culinary sensations that are increasingly being ¿ certain time fran:. .
referred to-in the press at least-as art (e.g. see Brooks zoo8). Sterbak (her Varrlr.;..' -
Vision, specifically in terms of the colour of food and drink, has been shown to play tied into the percep: :
a dramatic role in modulating the perceived identity and intensity of flavour and end excretes, rellc;r.- -
aroma (e.g. Duncker 1939; DuBose et al. r98o; Morrot et al. zoor; Zampini el al. zooT; Bruguera Iin her P,,¡ :.'
Levitan et al. zooS; Shankar, Levitan, and Spence zoog). Visual cues have also been through thedisplar , "
shown to influence how rapidly we reach s$ety when eating (Rolls et al. r98z; see also Our understalrdi:.
Raghubir and Krishna 1999), and can modulate our appetitive responses (e.g. Birren ou r culinary experic: -:
1963; Wheatley ry:;).'Similarly, the tactile attributes of food and drink, whether it be Llavaneres, in Barce- -- :: :
their temperature, texture, or viscosity have a profound influence on our experience of ¡ombines his knor. ..---
food as well (Green zooz; see Verhagen and Engelen zoo6, for a recent review)-a con- l: a neuroScientiit . :...'
cept without which one can hardly appreciate the visual pun of Meret Oppenheim's ;hef and proprietcr ,
Fur Coyered Cup, t936. Finally, it is important to note that flavour perception is also Jining experiences :::.
modulated by the stimulation of the trigeminal sense (see Green 1996; Delwiche :oor ). Similarlr'. Hc-. :
zoo4; Verhagen and Engelen zoo6); here, one need only think ofthe tingle ofcarbon -onsumer's dining . ,.:
dioxide bubbles in a carbonated beverage, or the pleasantly painful burning sensation rspects offood and dr-:
associated with eating a chicken vindaloo at the local curry house. ,rf his dishes at The i::
'.'od that is intrin.i¡:. ',
:ifficult for a \\ e:t::: .
:udding. Even chang::.
' Marinetti's Futurist Cookbook (rq:o/rg8q) advised readers that to stimulate the senses during
::n affect the perceir . -
eating, people should try wearing pajamas made of differently textured materials such as cork,
sponge, and fe1t. Marinetti also suggested eating without the aid of knives and forks by simply ',t-ansink zooz).
burying one's face in the plate to enhance the tactile sensations. Our Focus in the p:t-..
' Indeed, those attendees who went to the conference dinner at the'Art and the Senses'meeting in ,.hen people think ¿r -
zoo6 were treated to a starter of blue soup (while listening to Miles Davis' 'A kind of blue') followed
by green chicken and mashed potatoes. As one might imagine, these were not always the easiest of
:rinivasan r955.). P<:h-:
dishes to consume. . . ¡erception and betrt e¿,
.TO\ BLUMENTHAL SOUND RITES 209
Interestingly, numerous artists have used food preparation as their art form or to
theorize the totality of their artistic movement; or they have charged food consump-
tion with symbolic meanings as an integral part of their work (see Goldbergry7il.
:Lr Lrur perception and Looking into the less recent history of modern art, one can think about some classic
--s:s oi taste and works such as Duchamp's chocolate grinder (used as a metaphor of the ties between
smell.
:-: iCr)S€S are mOre for- food and sex), Piero Manzoni's r96o performance (where he ate a series of eggs previ-
:- J'beverage experiences ously cooked and signed with fingerprints by the artist, to comment on how art is con-
-r;. In fact, over the past sumed by the public while being experienced, and on the equivalence of physical and
i:¿t our perception of spiritual food), Mario Merz's spiral tables with fruit (dealing with the dichotomy of
r¿ r¡alized (see Duncker perfection vs . decay, and using food as a metaphor for the stages of life), AndyWarhol's
i.ed. many researchers packaged food, and Claes Oldenburg's fictional giant food items (which provided
. :.:. i.e. unified multi- a final poignant comment on how packaging comes to represent the identity of the
:,: ¡f a1l of the available food itself).
-.'. .:ctile, r'isual, and/or Even among the many possible examples from contemporary art, considering the
* * Soto-Faraco, zooT)., work of the following artists may give a sense of the variety of approaches presently
, . .n establishment that adopted to explore the sensory and conceptual complexity of food: Rirkrit Tiravanija
.l.r is probabiythe most (who has cooked meals for gallery-goers as a form of installation art since the r99os);
:..re .enSe', 2OO7, p. f9). Sonja Alhauser (famous for her edible sculptures to be consumed by the public, within
:: "¡e increasingly being a certain time frame, exploring the issue of eating as an aesthetic experience); Iana
Sterbak (her Vanitas: Flesh dress for qlbino anorecflc expands on the idea of food as
. r:. been shown to play tied into the perception of self); Wim Delvoye (designed a machine that eats, digests,
:.:.nsilr- of flavour and and excretes, reflecting on mindless eating as a mechanical experience); Tania
,r: Zampini eIal.zoo7; Bruguera (in her Poetic Justice. zooz, she tackled issues ofexploitation and colonialism
':- cues have also been through the display of two extensive wall surfaces coated in used tea bags).
ii: Ils et al. ry82; see also Our understanding of food as a multisensory experience has made its way into
'; fesponses (e.g. Birren our culinary experiences through restaurants like the L'Esguard de Sant Andreu de
.- j irink, whether it be Llavaneres, in Barcelona, Spain. The proprietor and chef, Miguel Sánchez Romera,
1.: -rn our experience of combines his knowledge of cogniqie psychology (which he acquires by day in his job
: -:-cllt review)-a COn- as a neuroscientist), and his e*pe.iet ce of flavour (which he acquires by night as chief
- ir1¡¡s1 Oppenheim's
--
chef and proprietor of a very successful restaurant), in order to produce eating and
:'.'Lrur perception is also dining experiences that more effectively stimulates the senses (see Sánchez Romera
i,:.en 1996; Delwiche zoor). Similarly, Heston Blumenthal works to manipulate different aspects of the
the tingle of carbon
:. i-,: consumer's dining experience by modulating different sensory and contextual
,li-iul burning sensation aspects of food and drink, in order to optimize the overall likeability and enjoyment
_ s-. of his dishes at The Fat Duck restaurant in Bray, England. As he notes, there is no
food that is intrinsicaily disgusting, otherwise no one would eat it. Eating a live crab is
difficult for a Westerner, while a Japanese diner might be appalled at a sweet rice
pudding. Even changing a food's name, from 'frozen crab bisque' to 'crab ice cream'
-,:.: -he senses during
can affect the perceived saltiness and flavour of the dish (Yeomans et al. zooS; see also
..-:r:Is such as cork,
. : :orks by simply Wansink zooz).
Our focus in the present chapter is to examine the sense that is most often overlooked
::¿ .he Senses'meeting in when people think about their culinary experience, namely the sense of hearing (see
..-.-rd of blue') followed
*.- . ai.,r'avs the easiest of
Srinivasan 1955). Perhaps surprisingly, the relationship between hearing and flavour
perception and between hearing and culinary experiences/behaviours more generally
2ToCHARLESSPENCE,MAYAU.SHANKAR'ANDHESTONBLUMENTHAL
has, to date, remained relatively unexplored (as compared to other multisensory inter-
actions in the domain of food and drink). In fact, some researchers have even gone so Ii:l:iH:,lJfffi.
far as to assert that interactions between sound and the chemical senses are, a priori, various mechanisnr. ::. -
unlikely p. 142).Indeed, in their comprehensive recent review of
(see Delwich e 2oo4,
the multisensory contributions to flavour perception, Verhagen and Engelen (zoo6)
devote less than one page of their 37-page article to any consideration of multisensory
interactions involving the auditory system.
It is our aim, therefore, in this chapter to try and correct this imbalance; in par- Auorrc
ticular, we bring together and critically evaluate the emerging (if widely distributed)
SELEC
body of empirical evidence from research fields such as cognitive and environmental
psychology, food science, neuroimaging, marketing, and psychophysics that has
iecently started to illustrate just how important what we hear is to our everyday A large body ol er f: .
food and drink experiences. For example' researchers have now shown that what ment can play an im:. :'
people hear can influence ever¡hing from which foodstuffs they choose to purchase purchase, and hori:'.-
1".g. Xo.tn ef al. t997), how much money they spend on
the items that they choose
al. 1988: Rozin anJ . -
(e.g. Areni and Kim 1993; North and Hargreaves 1998), what these food items taste
al. zoo3). Conrpcll:-- .
like (e.g. Srinivasan rgll; Holt-Hansen 1968, 1976; Rudmin and cappelli rgs:; zampini from a study br B.. ,
and Spence zoo4; Reid zoo8), how rapidly people consume them (e.g. Roballey et al. possible to alter hc,'.',
rgg5; Guéguen et al. zooS), and how much they consume (e.g. Guéguen et al. zoo4). ing the foods thenr..
Taken together, these findings demonstrate just how profoundly what we hear, Bournemouth L tiir . :, -'
be it the sound of the environment in which we eat, the sound of the food as we are .lavs. For the first tr' . -'.
eating or drinking it, or even the sounds of the opening of the packaging in which that decorated norm.rilr. =
food is presented (see Spence and Zampini zoo6) affects people's eating/dining rhe restaurant rr;r; Si . : '
experiences. Interestingly, these crossmodal influences appear to occur no matter .n the menu u5ir.-
whether people are aware of them or not. In fact, the evidence actually points to the ;ondition VS. r?r¡?i,i' '
fact that people are very often unaware of them (see Milliman r98z; Roballey et al' 1985; tbod choices, the il:-t
North ef a|. t99).It is our belief that by gaining a better understanding of the mul- a:ked to rate the fe:-:
tisensory contributions to the perception of food and drink (in particular, here, the iust selected.
contributions of what people hear), we can start to design food and drink experiences
that will more effectively stimulate the senses of the consumer/diner.
In the sections that follow, we review the extant research investigating the role
of auditory cues on the various aspects of our food experiences/behaviours. In the Table 11.'1
first two sections we highlight the influence of background music on people's food restaurarl
and drink selections, and on the rate at which they are consumed. In the following (normalJ 1
two sections we focus on the influence of the sounds that people actually make when
consuming food and drink, and on the sounds made by certain kinds of food pack- Ita lia n
aging on their experience of flavour. In the last section, we report on the results of
Red-crer' t
several new experiments designed to show how auditory stimuli can be used to
enhance people's perception of the identity and pleasantness of various food items.3
Italian:¿;:
Chiantl ::,:
Italian -¡'
r It is also important to note, though, that certain of the effects of tempo, or musical type, and Italian ira ":
even loudness on people's behaviour might simply be explained by people's preference for the
particular music being played (Herrington and Capella 1996; Caldwell and Hibbert zooz). In fact, to Source:3:
áate, it seems far to say that many researchers have underestimated the effects of pleasantness.
\ BLLTMENTHAL 'souNp BITES' 271
-.. i: multisensory inter- The results of these experiments illustrate how changing what people hear can influ-
--l- ¡:s haye even gone so ence (or change) their perception of foods. Where appropriate, we also highlight the
:r- senses are, a priori, various mechanisms that have been put forward to account for these effects.
.:::r\ e recent review of
..:- :nd Engelen (zoo6)
t..:it¡n of multisensory
. .::.'alll points to the on the menu using 'ethnic names' (e.g. maccheroni al formaggio in the Italian
:: R¡r[¿]lgy condition vs. maceroni qnd cheese in the British condition). After having made their
" : et al. t985;
.:,r:::ding of the mul- food choices, the customers (N = r38) were given a questionnaire in which they were
asked to rate the perceived ethnicity and acceptability of the food items that they had
-:r ;::ticular, here, the
just selected.
. .1 lrink experiences
:-:-¿f.
-r:..';streating the role
:;. :ehar.iours. In the
r*sr. on people's food Table'11.1 Decorations used in the Grill Room
::-,=j. In the following restaurant in order to elicit either an ltalian or British
.: :;tuallr.make when {normal) theme (atmosphere)
,: krnds of food pack-
Italian British
::--:' tln the results of
r::-r'.ir can be used to Red-checked tablecloths White tablecloths
.: ',.:rious food items.3 Italian flags on walls No flags
Chianti bottles on tables No bottles
Italian menu names British menu names
. :-ilsical ilpe, and
I ::.:'-rance fbr the Italian travel posters No travel posters
: : r::i :ooz). In fact, to Source: Bell et al. [1994).
: , :.:leasantness.
272 CHARLES SPENCE, MAYA U. SHANKAR' AND HESTON BLUMENTHAI
Bell et al. Ggg+) found that when the restaurant was given an Italian theme, the of a supermarket. O::: :
customers were more likely to select pasta and eat Itaiian desserts (e.g. ice cream four shelves, and ea;.
and zabaglione), while theychose fewer fish items (such as trout; see Fig. rr.r). Theyalso On alternating dar s. .''-
found that the Italian restaurant theme significantly increased the customers' ratings played in the backer,r,.',
of the perceived Italian ethnicity of the pasta items and increased the perceived ethnic- played, customers [.u:- -
ity of the meal as a whole. These results therefore provide a powerful demonstration was Irue when Gernl"r -
of how the perceived ethnicity of a dish can be aitered merely by manipulating the were stopped and qLr.-.-
sensory attributes of the environment. While Bell and coiieagues' study focused on the till, only o o1'r-: -:.
the effects of changing the visual attributes of the environment, a number of other been influenced br ::.
researchers have shown that people's selection of food and drink can be modified by demonstrate that rrh¿. :
manipulating the auditory aspects of the environment as well. For example, research- they make, no nlatter ..'
ers have shown that altering the music that people hear can also have a profound It is worth nothir: -
effect on people's food choices (North et ai. zoo3). These findings are particularly effects of playing mi.s.,
important in the context of the service/retail setting, given that this aspect of the envir- lets (e.g. Smith and r- ..--:
onment is both easy and relatively inexpensive to control (Herrington and Capella Herrington and Ca1'. ,
r994; Caldwell and Hibbertzoo2 Treasure zooT). ing study, Areni ¿nJ I
For exampie, North, Hargreaves, and McKendrick Q997) have demonstrated that items and selected ::: ..
the music played in a store can have a marked influence on customers'wine purchas- Lhe background th:: . -
ing decisions. In their study, four French wines and four German wines (matched an average of S- rri-.:' .
between countries for cost and drymess/sweetness) were displayed in the drinks section -1o' hits (see al't, \ .. -.
shown that chans.'::- .:
people move dort n l.--. -
alfecting custonrcr: : .
Percentage of customers selecting ltalian dishes based the traffic flow in . -:
on theme of room rras played than r.'l:.: '
increase in sales r' hi-. .-
o
H' zo
c
o
o
b15
(L
Bot
Fre¡
Bot
Gerr
Spaghetti Macaroni Chicken Trout Veal I
Meal type
r :r Italian theme, the of a supermarket. One French wine and one German wine were placed on each of
::s:-its (e.g. ice cream four shelves, and each wine was accompanied with its corresponding national flag.
:: s:e Fiq. u.i). Theyalso On alternating days, either French accordion music or German Bierkeller music was
I :-: customers' ratings played in the background. North and his colleagues found that when French music was
=; ,:e perceived ethnic- played, customers purchased more French wine than German wine, while the reverse
,'... :rtui demonstration was true when German music was played (see Fig. u.z). Crucially, when the customers
-'. :-,' nanipulating the were stopped and questioned about their purchasing decisions shortly after leaving
.*.. studv focused on the till, only 6 of 44 shoppers asserted that their decisions may have in some way
.:-:.: nulnber of other been influenced by the music playing in the background. These results therefore
-=.. ;:n be rnodified by demonstrate that what people hear can contribute to the food and drink choices that
:: erample, research- they make, no matter whether they re ahze ít or not (see also North et al. rggg ).
. .i.,-' hale a Profound It is worth nothing here that a number of other studies have also looked at the
: :-:-rgs are particularly effects of playing music on people's behaviour in supermarkets and other retail out-
.' r: rspect of the envir- lets (e.g. Smith and Curnow rg66; Yalch and Spangenberg r99o; Areni and Kim 1993;
l.::i::qton and Capella Herrington and Capella 1996;Wilson zoo3). For example, in one particularly interest-
ing study, Areni and Kim reported that the customers in a wine store bought more
.'. : iemonstrated that items and selected more expensive drink items when classical music was played in
:. :rers rvine purchas- the background than when 'Top 4o' music was played instead: the customers spent
:::--:it \\-ines (matched an average of $Z when listening to classical music versus $z when listening to 'Top
": r: :he drinks section 4o' hits (see also Yalch and Spangenberg 1993). Meanwhile, other researchers have
shown that changing the tempo of the background music can affect how rapidly
people move down the aisles in the supermarket (e.g. Milliman r98z), hence indirectly
affecting customers' purchasing behaviour. More specifically, Milliman obsewed that
: -.-:s based the traffic flow in a grocery store was significantly slower when slow-tempo music
was played than when fast-tempo music was played. Milliman also reported a 38%o
increase in sales when the slow music was played. In another early study, Smith and
Curnow varied the loudness of the music in two supermarkets and reported that the
Bottles of 12
40
French wine (77%) (23%)
sold
Bottles of 22
8
German wine (27%) (73%'
sold
ple, McCarron and Tierney reported a laboratory-based study involving 3o students Taken together, th: ''
in which they demonstrated that increasing the loudness of the background music environmental (or rii *, ..
(from no music to music playin gaf 70 or qo dB) resulted in a significant increase in the ence on the rate of ct::, -
consumption of soft drinks during a z5-minute period' while students sat passively in people consume. Th= ..,
the laboratory. loudness, tempo. ¿nü :-:
Meanwhile, Guéguen et al. (zoo+) conducted a more ecologically valid study in decision-making ar.rd : ..:
two bars (one located in an urban area and the other in a rural area), in which the Researchers have put :- :
loudness of the background music (Top +o music, as normally played in the bars under how background m;: :
study) was also varied. Their results showed that the customers (N = rzo participants) suggestion has been :r.:'
ordered significantly more drinks when louder music (88-9rdB) was played than when perception o[ the pa-:,.
the music was played at its normal quieter level (lz-ZSdB). In their latest research, et al. 1996; see also \r -..'
Guéguen et al. izoo8) have reported that playing loud music can affect the drinking ¿nd Hibbert zooz . T: ,
behaviour of young men (in the r8-25-yeat age range; see also Smith zooS). The young people perceive th¿t ::=
men in Guéguen et al.'s study drank 3.5 drinks in venues with loud music vs. 2.6 drinks .pending Ionger i¡ r. -:-
in venues playing quieter music. They also drank their beer faster (u'5 vs' l4'5 minutes; cating, drinking. anJ -.-
see 'Loud music in the pub "makes you drink more"' zoo8)' A second accoun: . :
In another oft-cited early study, Milliman (1986) manipulated the tempo (i.e. the result in a change in ..-:
number of beats played per minute' or bpm) of the music played in a restaurant, Jrinking behaviour. : :
and assessed its effect on the patrons' consumption of food and drink' The customers ¿nd Curnow 1966). T|. :
in this study ate more quickly when fast music was played (M = io8 bpm) than when tal stimuli, such as st,:;'
slow background music was played (M = 56 bp-)' Although there was no effect of olfactory cues, can c - .
musical tempo on the amount of money spent on food, there was a marked differ- nance, which mav in . --
ence on the final bar bill, with those in the slow music condition spending 4oolo more. :ee Mehrabian and J'.'
Similarly, McElrea and Standin g Gggz) have reported that drinking time decreased :rr usic may serV€ t o € F. r. : l
significantly when background piano music was played at l3z bpm as compared to see also i-.
Alpert and .
*hen it was played at 54 bpm. Note that the participants (N = 4o) in this laboratory- :nusic may simplr a;: :.
based study were simply asked to drink a can of soda with the stated aim of rating rave been shown to sn-l;
its flavour. McElrea and Standing's results are consistent with those from a study by rnd Dalix (zoot) har. ,
Roballey et al. (1985) in which fast-tempo instrumental, non-classical background r¿dio consume 15oo t:: :
:i]\ BLUMENTHAL SOUND BITES 215
. -¡'uder music (though music (M = rzz bpm) resulted in consumers (N = u) taking significantly more bites
per minute in a cafeteria, as compared to when a slow-tempo piece (M = 56 bpm) was
played (+.+ rrs. 3.2 bites per minute, respectively).
More recently, Iacob (zoo6) has shown that it is not only the tempo, but also the
style of background music that is played that is important. She investigated whether
the music played in a bar in a seaside resort town in France influenced how long the
patrons spent there. The drinkers in her study (N = s¡) were found to spend signifi-
HE RATE AT cantly more time and money when 'drinking' music (e.g. 'The Drunken Sailor Song'
AliD D_RI*^ or any one of a number of other festive songs that had lyrics pertaining to food and
alcohol) was played (at 7o bB) than when 'Top 4o' music (the music usually played
in French bars), or'cartoon' music (e.g. 'Some day my Prince will come' or any other
'- pe,rple consume food type of music played in animated films) was played instead. It should, of course, be
; ri. ',rhat they happen noted that the most appropriatelliked music will also depend to a large extent on the
,- :9S51 \fcCarron and ageldemographic make-up of the customers in a given retail or service setting (see
:: i.. -:.oo-1). For exam- Herrington and Capella rgg+; Caldwell and Hibbertzooz).
; ing 3o students
-m,. r¡1r Taken together, the studies reported in this section therefore show that auditory
::. background music environmental (or situational; Edwards et al. zoo3) factors can have a significant influ-
.-r-::ant increase in the ence on the rate of consumption of food and drink and on the overall amount that
:-:.ltts sat passively in people consume. The research that has been published to date has shown that the
loudness, tempo, and t¡pe of music can all exert a significant influence on consumer
- --;¿iir- r-alid study in decision-making and purchasing behaviour in a variety of different everyday settings.
:¿r :rea r, in which the Researchers have put forward a number of different accounts to try and explain why/
'-:."'.d in the bars under howbackground music might modulate people's behaviour in this way. One popular
\ = t:o participants) suggestion has been that changing the background music may simply affect people's
',.,:s played
than when perception of the passage of time (see Linsen 1975; Kellaris and Kent l99z; Kellaris
* üeir iatest research, et al. 1996; see also Wansink r99z; Yalch and Spangenberg zooo; though see Caldwell
::r altect the drinking and Hibbert zooz). The idea here is that if a certain piece of music happens to make
r::th :oo8). The young people perceive that the time is passing more slowly then they might well end up
*i nusic vs. 2.6 drinks spending longer in whatever environment they happen to be in, and hence possibly
:: -1.5 \'S. r4.5 minuteg; eating, drinking, and ultimately, spending more.
A second account of how listening to music can modulate behaviour is that it may
.-: the tempo (i.e. the result in a change in the level of arousal, and it is this that influences people's eating/
'-.-,ed in a restaurant, drinking behaviours (e.g. Roballey et al. 1985; Guéguen et al. zoo4, zooS; cf. Smith
: lrink. The customers and Curnow 1966). Third, according to environmental psychologists, environmen-
= - -,S bpm) than when tal stimuli, such as sounds and music (see Guéguen et al. zoo8), as well as visual and
:i¡:e rr,as no effect of olfactory cues, can elicit emotional responses, such as pleasure, arousal, and domi-
: i!'¿s a marked differ- nance, which may in turn m.ediate a variety of 'approach-avoidance' behaviours (e.g.
- spending 4oo/o more. see Mehrabian and Russell 1974; Holbrook and Anand r99o; Koneéni zooS). Thus,
::l:nq time decreased music may serve to enhance a person's mood and/or the pleasantness ofthe atmosphere
rtrin as compared to (see also Alpert and Alpert r99o; Bruner r99o; Drews et al. ry92; Jacob zoo6). Fourth,
: -
in this laboratory- music may simply act as a form of distraction. Note here that both adults and children
; .tated aim of rating have been shown to snackmore when watching TV (Wansinkzooz). Similarly, Bellisle
::üse tiom a study by and Dalix (zoor) have shown that people who eat their lunch while listening to the
-:-assical background radio consume r5olo more food than those who eat in silence.
2].6 CHARLES SPENCE, MAYA U. SHANKAR, AND HESTON BLUMENTHAL
A final possibility here is that listening to pleasant music might actually affect More interestinel .
people's perception of the taste/flavour/pleasantness of the food or drink itself Adrian North (and --:.
(though see McCarron and Tierney 1989). Related to this latter suggestion, Kristian tigated whether li.tc:r- - :
Holt-Hansen Q968, t976) conducted studies in which the frequency at which people students at Herriot \\':..
experienced (s1'naesthetic) 'harmony' (or unity; see Vatakis and Spence zoo7, zooS) when 'heavy' backsr. -
between taste and sound (pure tones) while consuming beer (Carlsberg) was init- 'heavier' (see 'Music .,*
ially established. The participants reported that the beer tasted better (e.g. more full- a powerful. hearrp:.,.
bodied) when the pitch of the sound was'harmonized'with it (known as the 'pitch of was played. peollc :j :
fit'; see also Bronner et al. zoo8).4 By contrast, when the pitch of the sound was changed, robust than when ri..
the participants no longer experienced such 'harmony' and reported that the taste Dave Williams, the iJ .
and sound were now experienced separately (one might say that the 'unity assump- find 'music lists' in \1.-
tion' had been broken; see Vatakis and Spence zooT). lJnder such conditions, the par- tive restaurateurs Tr: :
ticipants tended to report that the beer tasted weak, thin, strong, watery, bitter, acrid, restaurants to the r' :r '- -
and/or nauseating! What is more, some of the participants also experienced rh¡hmic match specific rtinc- : :
sensations in their heads, as well as tickling sensations in their jaws and mouth. What is more. i-: ::
In a follow-up study, Holt-Hansen Q976) reported that three of the (nine) partic- that specific sounJ::,,.
ipants had a range of'extraordinary' experiences after having been trained to taste without having to \'. ::
one of six lager beers/ales while listening to a rh¡hmic sound, a sound whose pitch them indigestion arc ..,
was varied. These participants reported experiences that apparently resembled those possible using suci-. . ,
of people under the influence of drugs, such as mescaline, LSD, and cannabis (see own soundtrack c,','r -.:
also Beeli el aL. zoo5; Day zoo) . For example, one participant reported the following
experience at the 'pitch of harmony': 'My right hand with the glass of beer in it trem-
bled so violently that I was suddenly afraid of dropping the glass. I felt as if I was floating
in the air. The tone was intensified to such a degree that it sounded like a spnphony
orchestra and the room was filled with it. My jaws were moving in and out with the
rh¡hm of the tone.'5 Table 11.2 Music recc
In another laboratory-based study, Ferber and Cabanac j9S7) found that the varieties
hedonic valence ofsucrose (but not ofsodium chloride) solution could be elevated by
Cabernet Sauvignon
exposure to noise and loud music (McFadden et aL. ry7t). And, in what was perhaps
the first study to look specifically at the role of the ear in flavour perception, Srinivasan
(1955) observed that people's perception of the saltiness of common salt and the sweet-
Cha rdonnay
ness of cane sugar could be altered by putting the palms of their hands or their fingers
against their ears. When the participants closed their ears in this way many reported Syra h
either the enhancement or the suppression (depending on the participant) of the
odour of strongly-smelling substances such as asafoetida (see also Petit r958).6 Merlot
4 The pitch of harmony for Carlsberg lager was in the range of Source: Montes wines.
5ro-5zo Hz, while Carlsberg
Elephant beer was found to have a pitch of harmony in the range of 64o-67o Hz.
t Not bad on rz fluid ounces (i.e. 36o millilitres) of regular beerl More seriously, however, these
results presumably speak more of the dangers of relying too much on subjective report (see also
Srinivasan 1955), rather than of the miracuious effects of drinkingbeer per se (see also Rudmin and
Cappelli 19s3). \ote though. rh r. .
conclusions regarding the actual effect, ifany, ofclosing the ears on flavour perception across the : :. ing Gregorian i\.1:'.- .
-: right actually affect More interestingly, the latest (as yet unpublished) research to be conducted by
:. lood or drink itself Adrian North (and funded by Aurelio Montes, the Chilean wine producer), inves-
::;r suegestion, Kristian tigated whether listening to different kinds of music would affect people's (N = z8o
. .^3:r.1' at which people students at Herriot Watt University, Scotland) judgements ofwine. North showed that
j .- )ncnce .zoo7, zoog) when 'healy' background music was played, people judged certain wines as tasting
-: -arlsbergt was init- 'heavier' (see'Music can enhance wine taste', zooS; Reid zooS). So, for example, when
, rc:tcr /e.g. more full- a powerful, heary piece of music, such as 'Sweet Child O'Mine' by Guns 'N Roses,
. ..::.,rr n as the 'pitch of was played, people rated a cabernet sauvignon (a red wine) as tasting 6o%o more
t:::: .ound
lvas changed, robust than when the wine was tasted in silence (see Table u.z). These results led
.::,.rted that the taste Dave Williams, the editor of Wine and Spirit magazine, to suggest that we might soon
--.: .1( 'unity assump- find 'music lists' in Michelin-starred restaurants (Reid zoo8). Alternatively, innova-
- - - :.-nditions, the par- tive restaurateurs may soon start coordinating the background music played in their
:ri. ',retery', bitter, acrid, restaurants to the wines chosen by the patrons, in the same way that they currently
-,.f cricnced rh¡hmic match specific wines to particular dishes.T
':,... .tnd mouth. What is more, the recent development of hlper-directional loudspeakers means
:: rhe rnine) partic- that specific soundtracks could be directed to specific diners in a restaurant setting,
r r:.1 trained to taste without having to worry about any auditory information from the next table giving
- , ..'und whose pitch them indigestion and without the need for the use of headphones. In fact, it should be
.:: r:-',. resembled those possible using such loudspeakers for each person on a given table to have their very
.nJ cannabis lsee own soundtrack coordinated with their specific meal choices.
- -: -'¡ted the following
^80
.¡iated sounds people 3t
gF 70
;ttir¡rr of foodstuffs. It 60
"E'E 50
¡ .,tunds rvhen theyare
.:.. crisps, and cereals
:3
gE
40
30
: tne erpected flavour Eá 20
,, l:r-e norv shown that eC 10
-. :¡ intluenced 0
by the Attentuate Normal AmPlifY
.- :retr] (e.g. see Drake (A) Frequency ManiPulation
-- , -J Colas r99JJ. For
,. ¡¡,r¡1c's judgements Mean Responses for SoflOrisp Response Scales
for Three Attenuation Levels
-
-,.1 l. ;elery, turnips,
j: ..:. on the recorded
BO
70
-.-i: :o their judgments .¡C
íÁo
oF 60
.--: i,-ickers r98r). c6
otr
50
: ::.-r people')s percep-
-. !o
C)o
oq) 40
--,; ¡e atfected simply ()E
>f 30
".:.t:. The participants o'=
Vo 20
:: o(Ú
r-rsles potato chips o-E
10
::. tbr psychophysi-
0
r--, .:- its crispness and
odB
:: r.:-en. During each "iiloX?i' r-!ü'ilr
::-i r: L)\rn crisp-biting (B) Decibel Level
:l t:l ierrls ofits over-
::. .¡.¡nd that the par- Fig. 11.3 A) Participants'mean crispness ratings when listening to auditory
:: ¡tsnihcantiy fresher feedback incorporating one of three different high-frequency (i.e. those
.-.-:h tiequency sounds frequencies above 2kH) manipulations (attenuate, normal, or amplified high-
::: :articipants rated frequency components). B) Mean crispness responses with auditory feedback of
.. , ¡'itheir food-eating three different loudness attenuation levels (O dB, -20 dB, or -40 dB).
::: Source: Zampini and Spence (z0o+).
l:tenuated instead.
-:.-¡red biting sounds
.:-.renth.'noisy' foods.
: :::tporal qualities of sounds were once again manipulated in real-time. The participants judged the water
:-;an influence how samples in cups of carbonated water that they held in their hand as being significantly
: :-,:Lrn- ) influence that more carbonated when the overall level of the carbonated water sound was increased
.-. : -;O5, -zOO6; See alSO and/or when the high frequency components were amplified.
These findings are also supported by a nice anecdotal example reported by lon
: , ::r.estigate the role Prrnz, afood researcher based in the Netherlands. He describes a situation in which
:. i99r; Harper people were trained to bite rh¡hmically into a food (by trying to keep pace with the
:a"'-;kers
irr rate a SerieS Of sound of a metronome playing in the background). If, during the course of the biting
, . urd emitted by the action, the experimenter played the sound of glass breaking, people's jaws would
--, ;rnposition of the often freeze up-it was as if they could no longer eat'
22O CHARLES SPENCE, MAYA U. SHANKAR, AND HESTON BLUMENTHAL
Hashimoto and his colleagues in Japan have taken the idea of playing with the a packet of Kettle's ;: ,
sounds of food one stage further via their recently developed straw-like user interface strate); (¡) the popr-r.=
(see Hashimoto et al. zooT). Users of the device are instructed to place the straw over
chips (rZo-g cardhr ,::
a food of their choice (with different foods pictured on different plates) and then to andpolyethylene ::::-
suck ('drink') on the straw. The device gives a surprisingly convincing virtual impres- 1964; McFadden el .,
sion that one is sucking up one of a range of different foods through the straw. This random order. Th. :.' -
impression is achieved simply by combining tactile and auditory cues as the user ness of the potrto -. .: '
sucks on the straw. So, for example, juice appears to enter the straw with little resis- to be listening to j:: -
tance, while sucking up jelly through the straw gives rise to intermittent resistance. bit into when li-t.- --
The impression here is, of course, entirely illusory since no food is, in fact, actually both I as being i:.:-.- -
delivered. of Pringles cris;- . -
-
Lhe compari.,'- --:t - -
,'rc-errlJti,':.. . - . - -
It is, however, important to note that it is not just the sound that a food makes as --.h can be t::-:.- .
we consume it that modulates our perception of noisy foods (such as crisps and a: 1n f rrrLo-, s :,: ', :
carbonated beverages). In fact, even the sound ofthe packaging in which the food ::r¡r'iours :r:a : :::. -
happens to be presented (e.g. think of the noise of the rattling of the crisp packet :. --: i:e :.- -- i
whenever eating a bag ofcrisps) can have a subtle, but significant, effect on our percep-
tion of the food within.
Amanda Wong and Charles Spence recently conducted a study in which zo par-
ticipants had to bite into 4o Original Pringles potato chips, while either listening to
white noise or to the sounds associated with the opening/manipulation of one of
three different crisp packets. The participants in this experiment were given an assort-
ment of fresh potato chips (from a newly opened pack) and stale potato chips (from
a packet that had been opened and then left out for several days).8 Before starting the
experiment proper, the blindfolded participants were given a sample crisp to taste,
which theywere informed had a crispness rating of 5 on a ro-point crispness scale, with
ro being 'very crispy'. During the experiment itself, the participants were instructed
to bite into each potato chip with their front teeth and give a verbal estimate of the
'crispness' of the chip on the ro-point scale. During each trial, one of four different
sound clips was played to the participant over a pair of closed ear headphones. The
four auditory soundtracks consisted of: (r) the sound made by the rattling of a packet
of Walker's crisps (25-g bag composed of two layers of pollpropylene, one metallic
layer and the other with the print on the outside); (z) the rattling sounds made by
.-r ol playing with the a packet of Kettle's crisps (28-g, form-fi.11-seal bags made from a coated-paper sub-
:¿rt-like user interface
stiate); (¡) the popping sound associated with the opening of a tube of Pringles potato
, :
: .-- place the straw over chips (r7o-g cardboard tube with a paper liner and a foil liner made of aluminium foil
:.*: nlates) and then to anJ polyethylene); and (4) continuous white noise presented at 9r dB (cf. Kupfermann
- .: . jng vjrtual impres-
rqOa; N4cFadden et al. ryll. The various auditory conditions were presented in a
:-r: -9h the straw. This random order. The results showed that participants' perception of the crispness/stale-
: :,- :-!- cues aS the user
ness of the potato crisps was affected by the particular soundtrack that they happened
- . :i\\'rrith little resis_ to be listening to (see Fig. rt.4). In particular, the participants rated the crisps that they
-.:-rmittent resistance. bit into when listening to the rattling of a bag of Kettle's or Walker's crisps (M = 6 for
, l :s, in fact, actually both) as being significantly crispier than when they listened to the recording of a tube
of Pringles crisps being opened (M = l.l; t(V) = 3.48, p <o'oo5, by post hoc t-test for
the comparison between the Pringles soundtrack and both the Kettle's and Walker's
crisp rattling soundtracks).
That what we hear (even when it is the sound of a rattling crisp packet) can have a
significant effect on our perception of food should perhaps not come as such a surprise
TGING *h"n orr. considers the animal literature. For one need think only of Ivan Pavlov's
(t927) early demonstration that dogs can be conditioned to salivate in response to the
presentation of a sound that has previously indicated the arrival of food' In fact' even
: ::¿r a iood makes as fish can be trained to engage in food expectancy behaviours in response to sound,
:i .¡ch as crisps and as in Frolov's classic report of the various species of carp whose food-approach
: : ir rrhich the food behaviourshadbeen conditionedtothe sound ofatelephone ringing (Frolovr9z4lt937,
.: ,:f the crisp packet p. 66; see also 'The fish with a memory who knows when it's time to eat" zooS)'
:l.ct on ourpercep- Meanwhile, Kupfermann Q964) reported that a variety of eating behaviours in rats
result from the mere presentation of white noise to the animals' Given that the sound
.*i', :n rthich zo par-
- ... :irhe¡ listening to
,.:- :ulation of one of
very
: ,";Ia gl\'en an aSSOrt- crisp Mean crispness ratings
: : rrrtdto chips (from 7-t
,. Betore starting th,e 0
,-::,-tple crisp to taste, c
'E
(ú
- ::rspness scale, with cc6
IA
:::iS \\ere instfuCted o
o
',
c
::-.:1 estimate of the @
. ne of iour different O5
-.:headphones. The
(d
q)
: . : :a Ltling of a packet
".
¡','lene, one metallic 4+
--.-::. sounds made Nol Walkers Pringles White
by very Noise
crisp
(B)
. r:rstant gratification
:¿rs. The design firm
,-\1 \,-últte Meal: Design
i:i¿nne, France (6-14
:n:tion of this food,
ii:ned together, which
,:;¡ 'sar-es'
the wearer
r: )eme time making a
r n the o\-erconsump-
-,-- lev programmed a
in
(B)
. AL SOUNDS Fig. 11.5 A) Picture showing Heston and Charles at the front of the audience at
TIES AND the'Art and the senses'conference (oxford,2006). B) Picture showing Heston
Blumenthal in his kitchen'
(a) photograph: Francesca Bacci b) oHeston Brumenthal
farmyard soundtrack and a yellow plastic spoon (e.g. consistent with the colour of
rnents together with eggs) while the other sample was presented with a blue spoon.
r r¡i the 'Art and the The results showed that the audience rated the ice cream tasted in the presence of
. trr e\amine whether
the sizzling bacon soundtrack as having a significantly more bacony flavour than the
r-.lence people's per- ice cream sample that was tasted in the presence of the farmyard chicken sounds
-¡e hrst experiment, (t(¡S)= 4.o, p <o.oor, bypaired-samples t-test; see Fig. rt.7)' Interestingly, however,
:c egg'ice cream, one both scoops of ice-cream had come from the same batch-that is, the flavours were
rstructed to rate the actually identical. These results show that auditory cues can be used to bias people's
r a .cale provided on a perception of the relative strength of two competing flavours in a food.e
I in the background
:.iiTIS: One soundtrack
,Note that when the bacon and egg-flavoured ice cream is served at The Fat Duck restaurant, a
''u'hile the other con-
pieceoffried bread is also presented on the plate. Adding this crispy component to the dish really
r.r ,r¿s servedwith the makes the dish work. Informally, it seems that the fried bread provides the texture appropriate to the
224CHARLESSPENCE'MAYAU.SHANKAR'ANDHESTONBLUMENTHAL
Bacon flar¡o-'
is much strc-;'
Bacon flavour Egg flavour is
is much stronger much stronger
The bacon and egg Fig. 11.7 Results of
flavours are equally strong part¡cipants rated tl'(
bacon soundtrack as
the ice cream samp e
oysters in terms oi ::
shown in Fig. u.S. -
More generally, it should be noted that the disambiguation of the flavour of a food
dish can be achieved by a number of means: either visually, by changing the colour of
the food, verballybymeans oflabelling (Meiselman and Bellrggz; Wansink et al. zooo),
4'
bypresenting pictures or other cues on the packaging (see Dematté et al. zoog), and/or
by the presentation of auditory cues, as described in the present study. Furthermore,
t:
even saying the word 'cinnamon' has been shown to activate the olfactory cortex
(i.e. the part of the brain that plocesses smells; see González et al. zoo6)' Playing the Oyster l
sizzlingbacon soundtrack at the 'Art and the Senses' conference may therefore have
influenced the audience's perception of the bacon flavour in the ice cream simply by \_____t\
making them think of bacon. All provide putative explanations of how listening to
the sound of frying bacon might make bacon and egg ice-cream taste more bacony! Oyster 2
It is at present an open question as to whether simply writing the word bacon on the
screen in the front of the auditorium would have had the same effect (see Sakai et al.
zoo5; Dematté et al. zoog). The soundtracks were presented at a clearly audible level
in this study and some of the audience members could feel that their flavour percep-
tion was being changed by the soundtrack; a few even tried to override this form of
auditory manipulation by repeating the word'bacon, bacon, bacon. . .' to themselves
ff
l} Egg flavoirr is
LJ Bacon flavour The bacon and egg
bJ
is much stronger flavours are equally strong much stronger
-i ruchflavour
Egg is
stronger
Fig.11.7 Results of the bacon and egg cream ice cream experiment. The
participants rated the ice cream sample tasted in the presence of the sizzling
bacon soundtrack as having a significantly more pronounced bacon flavour than
the ice cream sample tasted in the presence of the farmyard chickens soundtrack.
Elg flavour is
-.]3h stronger
likemantra when listening to ihe farmyard chickens clucking away, but to little avail.
a
r: and the Senses' One suggestion is that environmental auditory stimuli may activate superordinate
r experiment. They knowledge structures and hence prime related stimuli (see Martindale and Moore
T: e, to indicate 1988; North eta7.rygZ).
tE flavour. In a second study, 33 members of the audience were asked to taste and rate two
oysters in terms of their pleasantness and intensity of their flavours using the scales
shown in Fig. n.8. One oyster was served in the shell from a wooden basket (of the
: ,r: ¡]avour of a food
,-':qinq the colour of
-,'r'-lnsink OYster-tasting
. et al. zooo),
A Please mark on the following scales (between the two
.:": :r al. zoog), and/or crayons) how pleasant you perceive the oyster to be, &
ll
: ,:..Jr'. Furthermore, H on the other scale how intense the flavour was.
, :he o1t-actory cortex
... :',,o6 l. Piaying the Oyster 1
r:-.-.. :] the mouth as Fig. 11.8 The two scales presented to audience members at the'Art and the
-- - ...1 to the texturally
Senses' conference (Qxford, 2006) during the oyster-tasting experiment. They
-: :: :r \\ hich has more were asked to mark a line through each scale after tasting each oyster, depending
on how 'pleasant' and 'intense' they found the flavour of each oyster to be.
CHARLES SPENCE, MAYA U. SHANKAR, AND HESTON BLUMENTHAL
tlpe that one corrmonly sees at the seaside). The other oyster had been removed Taken together, th. :.
from the shell and was served in a petri dish instead. The first oyster was served while the Senses' conferen,.
the audience listened to the 'sounds of the sea' soundtrack in the background (this influence (or bias' |-r - :
consisted of the sound of seagulls squawking and waves crashing gently on the beach), results led directlr r. . - ,
the second while they listened to the 'farmyard noises' used in our first experiment.'o tasting menu of The l ,
The results revealed that the audience rated the oyster that they had consumed while are presented r,r'ith ., :
listening to the 'sound of the sea'as tasting significantly more pleasant than the oyster weed, and sand all r -. .
that had been tasted while they listened to the farmyard noises instead (t$/1=2.r0, iPod hidden inside - -,
p(.05, paired sample t-test; see Fig. rt.9). Interestingly, however, no such effect was encouraged to pr.it {': -
found for the intensity ratings (see Fig. n.g). That is, changing the sound had no effect .oundtrack) belore .: ="
on people's perception of the intensity of the flavour of the oysters. o:l the tasting fircflL ' :.
In fact, it has becor-:,
.cveral levels: br g(....--
br helping to illu-tr:.. -
r' e believe, on thc l-.,.
ttccause the sott ndrr.,.'
h* flavour (see'F¿- . . .
Preasant
/
Oyster eaten when Oyster eaten when
listening to the listening to farmyard
'sound of the sea' sounds
lntense
No flavour
flavour
Fig. 11.9 Results ofthe oyster-tasting experiment. The participants rated the
oyster consumed while listening to the 'sound of the sea' as tasting significantly .t;:
more pleasant than the oyster tasted while listening to the farmyard sounds
instead. No such effects were seen on the ratings offlavour intensity.
"' Note that we counterbalanced the order of presentation of these two conditions, that is, one
group of audience mernbers hear the sound of the sea first, and the farmyard noises second, while the
order ofpresentation ofthe sounds with the oysters was reversed for the other group ofparticipants.
r.-) \ BLUMENTHAL 'souND BITES' ))a
):-: had been removed Taken together, the results of these two experiments, conducted at the 'Art and
---. ster \\'as served while the Senses' conference, highlight just how dramatically environmental sounds can
r :he background (this influence (or bias) people's perception of the foods they consume. Interestingly, these
:ri :.nth'on the beach), results led directly to the introduction of the 'Sound of the Sea' seafood dish on the
: - :: hrst experiment.'o tasting menu of The Fat Duck restaurant in Bray (see Fig. r1.1o). With this dish, diners
=; i¿d consumed while are presented with a plate of food that is reminiscent of a beach (with foam, sea-
: -:¿s¿nt than the oyster weed, and sand all visible on their plate). The diners are also presented with a mini
..'. :nstead (t(32)=2.34, iPod hidden inside a sea-shell, with only the earphones poking out. The diners are
"
-:. r'ro such effect was encouraged to put on the headphones (whereupon they hear the'sounds ofthe sea'
-:. Srrund had no effect soundtrack) before startingto eatthe dishplacedin front ofthem. The response ofdiners
on the tasting menu has been very positive. The diners appear to really enjoy the dish.
In fact, it has become a signature dish at the restaurant. The dish apPears to work at
several levels: by getting the diners to think about the role of hearing in eating and
by helping to illustrate the importance of sound to the appreciation of food. Second,
we believe, on the basis of the evidence reported here, that the dish is so successful
because the soundtrack serves to intensift the flavour of the dish. That is, it heightens
the flavour (see'Fat Duck dons an iPod', zooT).lt is also worth noting that wearing
the headphones has the originally unanticipated additional effect of concentrating the
diners' attention on the dish since it becomes difficult to converse with one's dinner
partners while listening to the seaside sounds (see Marks and Wheeler 1998)'
Finally, it is worth thinking about the effect of the sound of the dish's name itself. to date, the topic oi thc
As Gerard Samper, the effete Englishman and ghostwriter to the stars puts it in James not received the seritr--
Hamilton-Patterson's novel, Cooking with Fernet Bronca: 'I sometimes wonder if one anisms (both ph1'sitl-
is not more seduced by the mellifluous sound of a dish than by how it would actually explain the various rr ¿',
taste' (Hamilton-Patterson,2oo4, p. 67). While one might not necessarily agree with and behaviours. Thes¡
the claim, the latest evidence nevertheiess does suggest that certain food attributes/ arousal (e.g. Smith anr
qualities really are better matched with particular food names, or rather with particular from the modulatio::
speech sounds. In fact, this research represents an extension ofone ofthe best-known putative conditionir: s
early results from the study of experimental psychology: namely, the finding that if with the packaging i,
people are shown two shapes, one anguiar and the other curry, and are asked which Rozin and Tuorila rg'-,
of the two shapes is a 'kiki' and which is a 'bubu', the majority will say that the angular Researchers have als.. ,
shape is the 'kiki' whereas the more curvaceous shape is the 'bubu' (see Kóhler r9z9; tion of the identin' i-,r
Sapir r9z9; Ramachandran and Hubbard zoor). tainty (see the 'baco:: :
The results of the latest research from the Crossmodal Research Laboratory in appear that the tooc ., :
Oxford have extended this idea by showing that certain foods (such as a soft and :.oo6), and even th¿ . '
creamyBrie cheese) are more naturally associated with more rounded/softer-sounding seamlessly integratec :
words whereas other foods (such as sharp and acidic cranberries) are associated with clrink.
sharper-sounding words (Gallace et al. in press). The results suggest that there might That the 'atmosp:.
be a systematic link between the taste of foods and the kinds of sounds that match, ence our perceptlo:l , -
or are synaesthetically congruent with them. Such results are now beginning to be has purchased a che:p
used by the most innovative chefs and food manufacturers in order to ensure that the rnal questioning, it .''.,
sound of the names of their food dishes are matched synaesthetically to the tastes that Kingdom) have h¿i ,:
consumers are about to experience (see also Klink zooo; Bertolli zoo3; Yorkston and consumed at u aartr.-r.,
Menon zoo4). rhe sun beating do.. '
onlv to open the \ (r. - .
:r ening back ar h^::::
:.rnle. The lact th¿t ..... .
,necdotal, st,tp])urI i .
In conclusion, the evidence reviewed in this chapter illuminates just how impor- .\eap wine taste . -
tant what you hear is to your everyday experience of food and drink. We believe that ..rre some of there .,' .
the research that has been published to date now unequivocally shows that auditory re tlavouripleasir n. :.'. .
cues can (and do) influence many different aspects of our eating/dining experiences- .;1g rvine into sonlc..: ".
despite people's intuitions to the contrary. What we hear influences everl.thing from : ,ntributions ol-(.r- .
what we choose to eat, the rate at which we eat it, the perceived pleasantness of the \\ hile ir would p:
food in our mouths, and the flavour profrles of wine, beer, and presumably any other :;sJrd to certain o- - -
drink that we might have a predilection to consume (see Petit rglS; Holt-Hansen r968, -:.rture (e.9. see H, -1
'
1976; Rudmin and Cappelli tg8¡; Bronner et al. zoo8; Reid zoo8). '-llcv eflects. €.F. ... .-
The results of the research outlined in this review have been published in a variety -:,.r have been inde¡:"
of disparate research areas, including human psychophysics, food science, market- r auditory stimul; "
ing, functional neuroimaging, and animal behaviour. This may help to explain why, ,-.¿im that what pecr .
BL UMENTHAL SOUND BITES
- rhe dish's name itself. to date, the topic ofthe influence ofhearing on eating, drinking, and perception has
'-i si¿rs puts it in fames not received the serious scientific attention that it deserves. Several potential mech-
:r-till-les rvonder if one anisms (both physiological and psychological) have been put forward to try and
, :L-rir lt rvOuld aCtUally explain the various ways in which what people hear influences their food perceptions
. :.;:ssarily agree with and behaviours. These include physiological/autonomic effects on a person's level of
.:t-ril food attributes/ arousal (e.g. Smith and Curnow r966), multisensory perceptual interactions resulting
: ::rler rr-ith particular from the modulation of food-related sounds (e.g. Zampini and Spence zoo4,2oo5),
-:l: t¡ithe best-known putative conditioning effects resulting from the sounds made by our interactions
r.-'.. :he ltnding that if with the packaging of food (and possibly also the environment; see Gorn r98z;
, :rd are asked which
. Rozin and Tuorila 1993) on the pleasantness of the food or drink itself (Jacob zoo6).
,"' _,¡,, that the angular Researchers have also shown that auditory cues can help to bias people's identifica-
:r * see Kóhler r9z9; tion ofthe identity/flavour ofa food, especially under conditions ofperceptual uncer-
tainty (see the 'bacon and egg ice cream' experiment reported earlier). Thus, it would
.r:::ch Laboratory in appear that the food or drink (i.e. the product), its packaging (Spence and Zampini
:: such as a soft and zoo6), and even the environment (i.e. the place; see Pine and Gilmore 1998), are all
-::¿c sotter-sounding seamlessly integrated to give rise to our overall perception oflresponses to food and
-¡ ;13 associated with drink.
:...¡: that there might That the 'atmospherics' (see Kotlet ry74) of the environment can and does influ-
,: s':unds that match, ence our perception of food and drink should come as little surprise to anyone who
:-'-'r beginning to be has purchased a cheap bottle of plonk (wine) while on holiday. On the basis of infor-
rl:r irr ensure that the mal questioning, it would appear that most people (at least those living in the United
,::-i..'lt't the tastes that Kingdom) have had the experience of drinking wine that tastes so nice when it is
''I - i,)il Yorkston and consumed at a restaurant by the sea (preferably somewhere in the Mediterranean) , with
the sun beating down on your back, and, of course, listening to the sound of the sea,
only to open the very same wine once safely ensconced in the middle of a dark winter's
evening back at home, and to find, with disappointment, that it no longer tastes the
same. The fact that this experience is common to so many of us provides further, albeit
anecdotal, support for the claim that the atmospherics in which we consume food
and drink have a profound effect on our perception and appreciation ofthem (see also
Byrne and Clore t97o; North and Hargreaves r996a,b; Pine and Gilmore 1998). Given
the results reported in this chapter, one obvious solution to the'plonk paradox' (why
.::: just how impor-
cheap wine tastes good on holiday but terrible at home) would be to try and recap-
::-¡i. \\-e believe that ture some of these sensory impressions in one's own living room, in order to enhance
', i-t,tlrs that auditory
the flavour/pleasantness of the wine-drinking experience (and turn that horrible tast-
:-mrrq experiencg5- ing wine into something that tastes really rather nice), and to elucidate the respective
',-;.s e|en.thing from
contributions ofcontextual effects on hedonic ratings.
::;--asantness of the While it would probably be most sensible to retain a degree of scepticism with
:::s.rnably any other regard to certain of the more outlandish findings that have been reported in the lit-
r :: i-,trit-Hansen 1968,
erature (e.g. see Holt-Hansen t968,1976; especially given the possible role of expec-
tancy effects, e.g. see Holt-Hansen r968, p. 6r; Rudmin and Cappelli 1983), at least until
: .'b-:shed in a variety they have been independently verified, the weight of empirical evidence on the effects
,- -J s.ir.nce, market- of auditory stimulation on our eating experiences now unequivocally support the
-r;-p to explain why, claim that what people hear can modulate their experience of food and drink. When
23O CHARLES SPENCE, MAYA U. SHANKAR, AND HESTON BLUMENTHAL
trying to deliver the most satisfying eating and drinking experiences, the loudness,
tempo, and type of music played in the background, the particular sounds made by
food as we eat and drink it (e.g. Zampini and Spence zoo4, zoo5), the sound of the
packaging or container in which the food happens to be served or presented (see We would like t.. ,
Spence and Zampini zoo6), and even the atmospherics of the environment in which smoothly with ti- ,:
that food is consumed, should all be considered Kotler ry74; Mehrabian and Russell
(see Shankar would al.i -,
1983; Pine and Gilmore 1998; Turley and Milliman 2ooo; Spangenberg et al. zoo5)."
Of course, when thinking about the use of sound in service settings (retail environ-
ments) one must be careful not to go so far as to annoy the consumer/customer (e.g.
see Englis and Pennell 1994; Spence zooz). RnpnRnNcrs
We believe that the cognitive science-inspired approach to the design of food and
Alpert JI and Alf ¿ri l" l
drink outlined here offers a promising means of enhancing the eating experience in
and Markettttg. T.
the years to come (see 'Fat Duck dons an iPod', zooT; Reid zoo8; Spence zooS). In
Areni CS and Ki::: -
fact, the future may look very much like the Futurists predicted a century ago (see Classical \¡ersu: .:
Marinetti rg¡ohg8g). While Marinetti suggested that we could enhance the dining T6-40.
experience by implementing what we know about contextual effects, our growing -t
Bach PJ and Schae:.:
understanding of the cognitive neuroscience is already providing some important Journal o.f Sttt,:.::,
practical interventions that may make flavour more appealing. Indeed, recently, the Beeli G, Esslen \1. .:::
press has even started to refer to specific dishes, and even entire menus, as 'works Belk RW. (197.1 i, .i.. . .
ofMarketirtgR¡,... -,
of art'. Moreover, some of the world's top chefs are increasingly being described as
Bel1 R,Meiselma¡: ^.-
artists, rather than simply as cooks! For example, Alison Knowles, an American
to a restauran: ,- .-. .
experimental artist, recently prepared a giant saiad and dished it up for over 3oo people tt 11-)/
at Tate Modern in London. The coordinating of the salad was set to the music of Bellisle F and Da-r'. -
Mozart, in a'performance designed to blur the line between art and everyday activity' increased n-iear ' :
(Brooks zoo8, p. 9). The event's curator, Kathy Noble, said, 'It's a participatory event BertolliP(zoo3.C.;:
in every sense. The work of the chefs, the observation of the audience and then their Birren F (196,1 i. Cc,. - . .
chance to eat what they have seen put together' (Brooks zoo8, p. 9). Bronner K, Bruhn '.
percept¡otl o.i ,': .., . :
Ju1y. Hamburs.,-..:
Brooks R (zoo8;, l:.. .
:;i : century ago (see Areni CS and Kim D (rSs¡). The influence of background music on shopping behavior:
Classical versus top-forty music in a üne sfore. Advances in Consumer Research, zo,
-; ;nhance the dining T6-40.
r ¡rlcts, our growing Bach PI and Schaefer IM GslS). The tempo of country music and the rate of drinking in bars.
..-:rg some important Journal of Studies on Alcohol,4o, ro58-9.
. lirdeed, recently, the Beeli G, Esslen M, and Jáncke L (zoo5). When coloured sounds taste sweet. Nature, 434, 38.
r:::- menus, as'works Belk RW. (t971. An exploratory assessment of situational effects in buyer behaviour. Journal
:-', being described as of Marketing Research,ll, r5ó-óJ.
.:r,-, -r-ies. an American
Bell R, Meiselman HL, Pierson BJ, and Reeve WG (1994). Effects of adding an Italian theme
to a restaurant on the perceived ethnicity, acceptability, and selection of foods. Appetite,
: -ii ILrr o\.er 3oo people
22,77-24.
rr :.t to the music of Bellisle F and Dalix AM (zoor). Cognitive restraint can be offset by distraction, leading to
: ¡:-d even.day activity' increased mean intake in women. American Journal of Clinical Nutrition,74, tg7-2oo.
.
:arttcipatory event
-: Bertolli P (zoo3). Cookíngby Hand. Clarkson Potter, New York.
:;-.nce and then their Birren F (1963). Color and human appetite. FoodTechnology,y (May), 45-7.
Bronner K, Bruhn H, Hirt R, and Piper D (zoo8). Research on the interaction between the
perception of music and flavour. Poster presented at the 9'h Meeting of the IMRF, r5-r9
July. Hamburg, Germany.
Brooks R (zoo8). Tate tosses up super-salad as art. The Sunday Times,March 16 (News), 9.
Bruner GC (iqqo). Music, mood, and marketing. Iournal of Marketing,54, 94-to4.
Byrne D and Clore GL (rqzo). A reinforcement model of evaluative responses. Personality,
1, 103-28.
Caldwell C and Hibbert S (1999). Play that one again: The effect of music tempo on consumer
behaviour in a restauranf. European Advances in Consumer Research,4, j8-62.
Caldwell C and Hibbert SA (zooz). The influence of music tempo and musical preference on
restaurant patrons' behavior. P sychology and Marketing, l.9, 895-9t7.
Chen J, Karlsson C, and Povey M (zoo5). Acoustic envelope detector for crispness assessment
: , r,.:rs these findings
of biscuits. J o urnal of T extur e Studies, 36, g9-56.
:-'. . :cr-erse effects of
: , -:i-j.: Spence zooz). Chen f, Karlsson C, and Povey M (zoo6). Crispness assessment of roasted almonds by an inte-
r - :-- provide specific grated approach to texture description: Texture, acoustics, sensory and structure. Journal
,_:-.:..-deh Glasgow of Ch emometrics, zo, 3tr-zo
.: ::Lt,:le in this Christensen CM and Vickers ZM (rq8r). Relationships of chewing sounds to judgments of
: ,:-. :ruSic, it is food crispness. Journal of Food Science, 46, 574-8.
::.: ,Jibbert 2oo2; see Dacremont C and Colas B (rSS:).Effect of visual clues on evaluation of bite sounds of food-
::: :. :OO8). stuffs. Sciences des Aliments, 13, 6o3-ro.
232 CHARLES SPENCE, MAYA U. SHANKAR' AND HESTON BLUMENTHAL
Day S (zoo5). Some demographic and socio-cultural aspects of synesthesia. In LC Robertson Hall RL (1958). FLai'i,:
and N Sagiv, eds. Synesthesia: Perspectites from cognitive neuroscience' PP. tt-33. Oxford Inc, ed. Flayor r¿::.;
University Press, New York. research, compileti :. .
Delwiche I (zoo+). The impact of perceptual interactions on perceived flavor. Food Quality Reinhold, Nery Ic:..
and Preference, s, 737-46. Hamilton-Patersor-r I :
Dematté ML, San.bria D and Spence C (zoo9). Olfactory discrimination: When does vision Harper SJ and McD:i-...
matter? Chemical Senses, 34, to3-9. influence on desc¡r-:,
'Does it make sense?' (2007). Contact Royal Mail's magazine for Marketers. Redwood, Hashimoto Y, Nagar': ).
London. ofthe sensation o: .::
Drake B (1965). Food crushing sounds: Comparisons ofobjective and subjective data. Journal Computer Socien'. -
of Food Science,3tl, 55ó-9. Herrington J and C;:- -
Drake B (r97o). Relationships of sounds and other vibrations to food acceptability. Proceedings of Sert ices Markc::.::. !
of the jrd International Congress of Food Science and Technology, pp. 437-45.9-r4 August, Herrington fD and C.:.
Washington DC. Journal o/Serliii-. -'. l.'
Drews DR, Vaughn DB, and Anfiteatro A (rqqz). Beer consumption as a function of music Hill AJ, Magson LD ..'
and the presence of others. Journal of the Pennsylvanía Academy of Science, 65' t34-6. subjective experie:r --
DuBose CN, Cardello AV, and Maller O (r98o). Effects of colorants and flavorants on preferred food. -{::,
identification, perceived flavor intensity, and hedonic quality of fruit-flavored beverages Holbrook M and .\r.:--
and cake. Journal ofFood Science,45,t3g3-9. perceptual and aü;-
Duncker K (rS:S).The influence ofpast experience upon perceptual properties. American Holt-Hansen K (196: -
Iournal of Psychology,5z, 255-65. Holt-HansenK(r9.c.:
Edwards JSA, Meiseiman HL, Edwards A, and Lesher L (zoo3). The influence of eating and Motor S,ki1l,,. +,i :
location on the acceptability of identicaliy prepared foods. Food Quality and Preference, t4, Tacob C (zoo6). Sn'i.. "
Frolov YP (r9z4lty). Fish who answer the telephone, and other studies in experimental ñliuk RR {2ooo ,. C -
- . .- _
Goldberg RL (rgZg). Performance art: From Futurism to the present. Thames & Hudson, Psvcholog¡' o/ ):
\, .
London. ñotler P (r974). \rrr .
Green BG (zooz). Studying taste as a cutaneous sense. Food Quality and Preference,14,99-to9. Linsen MA ir975;. Lr..
Green DM and Butts IS (tS+S). Factors affecting acceptability of meals served in the air. Journal ...rud music in the 1'u '
of the American Dietetic Association,2t, 415-19. \l¿ller O, DuBo:e l^\ . '
Guéguen N, Jacob C, Le Guellec H, Morineau T, and Lourel M (zoo8). Sound level of envi- loodservice. Iot,r, .;.
ronmental music and drinking behavior: A fie1d experiment with beer drinkers. Alcoholism: '.1¿rinetti FT lr93o ro.
Clinical and Experimental Research,32, \-4. Arts, San Franci,i., ..
Guéguen N, Le Guellec H, and Jacob C (zoo4). Sound level of background music and alcohol \l¿rks LE, and \A he. -- .
consumption: An empirical evaluation. Perceptual and Motor Skilk, gg, l+-8. Cltemical Serrses. 23. . - -_
}. BLUMENTHAI, souND BrTES 233
-":::1.1. In LC Robertson Hall RL (1958). Fiavor study approaches at McCormick and Company, Inc. In AD Little
:,":.r. PP. 11-33. oxfo¡d Inc, ed. Flavor research and food acceptance: A survey of the scope of Jlavor and associated
research, compiled from papers presented in a series of symposia given in t956-t957, pp. 224-40.
-: :-¿\'or. Food Quality Reinhold, New York.
Hamilton-Paterson f (zoo+). Cooking with Fernet Branca. Faber and Faber, London.
.: ::-: \\-hen does vision Harper SJ and McDaniel MR (1993). Carbonated water lexicon: Temperature and CO. level
influence on descriptive ratings. Journal ofFood Science,58, 893-8.
" l.!-:,rk¿ters. Redwood, Hashimoto Y, Nagaya N, Kojima M, et al. (zooZ). Straw-like user interface: Virtual experience
of the sensation of drinking using a sftaw. Proceedings World Haptícs zoo7, pp. 557-8. IEEE
.r 't'ectile dafa. Journal Computer Society, Los Alamitos.
Herrington J and Capella L QgS4) .Practical applications of music in service settings. Journal
: :.: t¿bilin'. Proceedings of Services Marketing, S, 50-6.
: :_r---+i. 9-t4 August, Herrington JD and Capella LM (rqq6). Effect of music in service environments: A fie1d study.
Journal of Seruices Marketing ro, 26-4r.
., , runction of music Hill AJ, Magson LD, and Biundell lE (rq8+). Hunger and palatability: Tracking ratings of
subjective experience before, during and after the consumption of preferred and less
.:.:' ¿nd flatorants on preferred food. App etit e, 5, 367-7 7.
:: -r:- Ii¿\-ored beverages Holbrook M and Anand P (rggo). Effects of tempo and situational arousal on the listener's
perceptual and affective responses to music. Psychology of Music,18, 150-62.
1, ¡¡-rperties. American Holt-Hansen K (rq6s). Taste and pitch. Perceptual and Motor Skills,z7,59-68.
Holt-Hansen K GgZ6). Extraordinary experiences during cross-modal perception. Perceptual
-:: i:r:i¿ence of eating and Motor Skills, 43, roz3-7.
...-..:.',,¡';,1 Pre.ference, t4, Jacob C (zoo6). Styles of background music and consumption in a bar: An empirical evalua-
tion. International Journal of Hospitality Management, 25, 716-zo.
:,.::. ¡i the commercial Kellaris J and Kent R (rqqz). The influence of music on consumers' temporal perceptions:
Does time fly when you're having fin? Journal of Consumer Psychology, r, 365-76.
Kellaris l, Mantel S, and Altsech MB (rqq6). Decibels, disposition, and duration: A note on the
---.'. : :a:ings and prefer- impact of musical loudness and internal states on time perception. In Advances in Consumer
Research,23, 498-503. Association for Consumer Research, Provo.
, -- tr:s in experimental KlinkRR(zooo).Creatingbrandnameswithmeaning:Theuseofsoundsl'rnbolism. Marketing
Letters, rl) 5-2o.
_,: lir': -\rl exploration Kóhler W (rqzq). Gestalt psychology.Líveright, New York.
''...: . : \¿uroscience. Konedni VJ (zoo8). Does music induce emotion? A theoretical and methodological analysis.
;. -::¿nes & Hudson; Psychology of Aesthetics, Creativity, and the Arts,2,715-29.
Kotler P (1974). Atmospherics as a marketing tool. Journal of Retailing, 49, 48-64.
-.*-:i -\ and Belloch V Kupfermann I (lq6+). Eating behaviour induced by sounds. Nature,2ot,324.
--,':";ri.32, 906-12. Lee L, Frederick S, and Ariely D (zoo6). Try it, you'd like it: The influence of expectation,
: -r c.¿ssical condition- consumption, and revelation on preferences for beer. Psychological Science,17,to54-8.
Levitan C, Zampini M, Li R, and Spence C (zoo8). Assessing the role of color cues and people's
- -;.,r"s itt Food Science beliefs about color-flavor associations on the discrimination of the flavor of sugar-coated
chocolates. Chemical Senses, 33, 415-23.
.:. : -:.í¿rence, L4, 99-109, Linsen MA (rqzl). Like our music today, ms. shopper? Progressive Grocer, October,756.
in the aiir. Journal 'Loud music in the pub "makes you drink more"' (zoo8). Daily Mail,luly ry,26.
'-:-,'ed.
Maller O, DuBose CN, and Cardello AV (rq8o). Consumer opinions of hospital food and
!
. ¡ound level ofenvi- foodservice. Journal of the American Dietetic Association, T6, T6-42.
-¿. lrinkers. Alcoholism: S Brili, trans. Bedford
Marinetti FT (r93olr989). [La Cucina Futurista] The Futurist Cookbook,
Arts, San Francisco, CA.
: -::d music and alcohol Marks LE, and Wheeler ME (rqq8). Attention and the detectability of weak taste stimuli.
:. 99 -i-1-8. Chemical Senses, 23, t9-29.
234 CHARLES SPENCE, MAYA U. SHANKAR' AND HESTON BLUMENTHAL
Martindale C and Moore K (r98S). Priming, prototypicality' and Prefereflce. ]ournal of Ragneskog H, Brane G. !.
Exp erim ent al P sy ch o l o gy : H u m an
P er cep ti n and P erfo rm an c e, 14, 6 6t-7 o.
o intake and sympton:
McCarron A and Tierney KJ (rg8q). The effect of auditory stimulation on the consumption of 17-17.
soft drinks. App etite, 13, 155-9. Ramachandran V and I.
McElrea H and Standing L (1992). Fast music causes fast drinking. Perceptual and Motor Skills, and language. Jout'':.;.
75,362. Reid M (zoo8). Hort G,"
McFadden D, Barr AE, and Young RE (r97r). Audio analgesia: Lack of a cross-masking effect Roballey TC, McGreer-"
on taste. Perception and Psychophysics, to, t75-9. Bulletin of the Pst ; : .-'
Mehrabian AR and Russell JA (ry24. An Approach to Environmental Psychology. MIT Press, Rogers PJ and HiIl Ai .:
Cambridge, MA. stimuli: Interrelatit¡: :
Meiselman HL (zooS). Dimensions of the meal. Journal of Foodserttice,tg,]13-27. Behaviors, r4, 38;-9-
Meiselman HL and Bell R (r99riz). The effects of name and recipe on the perceived ethnicity Rolls BT, Rowe EA, an¿ l
and acceptability of selected Italian foods by British subjects. Food Quality and Preference, behaviour. P/11'-sl¡, ¡ -. .
: :relerence. Journal of Ragneskog H, Brane G, Karlsson I, and Kihlgren M (1996). Influence of dinner music on food
: f :-- rl.
intake and symptoms common in dementia. Scandinavian Journal oJ Caring Science, ro,
-r .- r the consumption of 1t-17.
Ramachandran V and Hubbard E (zoor). Synaesthesia - a window into perception, thought
'-i?.'.t.úl tuñ Motor Skills,
and language. Journal of Consciousness Studies, S, ¡-¡+.
Reid M (zoo8). How Guns and Roses can change your tune on wine. The Times, May r3.
- - .: ::oss-masking effect Roballey TC, McGreer,y C, Rongo RR, et al. (1985). The effect of music on eating behavior.
Bulletin of the Psychonomic Society,23, 227*2.
;:", :i;,.log),. MIT press, Rogers Pf and Hill AJ (rq8q). Breakdown of dietary restraint following mere exposure to food
stimuli: Interrelationships between restraint, hunger, salivation, and food intake. Addictiye
Behavíors, L4, 387-97.
:erceir.ed ethnicity Rolls BT, Rowe EA, and Rolls ET (rq8z). How sensoryproperties of foods affect human feeding
; :i:,, ttnd Preference,
behaviour. Phy siolo gy an d B eh avio r, 29, 40 9-r7.
Rozin P (r98z). 'Taste-smell confusions' and the duality of the olfactory sense. Perception and
'ri situational fac- P sy chophy sics, 3t, 397 - 4ot.
. Ftotl acceptability, Rozin P and Tuorila H (rSS¡). Simultaneous and temporal contextual influences on food
acceptance. F ood Quality and Preference, 4, 7t-2o.
- ' -:er¡tarket shoppers. Rudmin F and Cappelli M (1983). Tone-taste synesthesia: A replication. Perceptual (t Motor
Sftills,56, n8.
:.::., ior of restaurant Sánchez Romera M (zoor). La cocina de los sentidos [The kitchen of the senses]: Editorial
Planeta zoo3, Barcelona.
- is:¡is. Journal ofthe Sakai N, Imada S, Saito S, Kobayakawa T, and Deguchi Y (zoo5). The effect of visual images on
--) perception of odors. Chemi cal Senses, 3o ( Suppl r), iz44-i245.
: ' .;:.: ,¡.td Language,79, Sapir E (1929). A study in phonetic symbolism. Journal of Experimental Psychology, rz,
225-39.
:', ',",
-rsite: http://news. Shankar MU, Levitan CA, Prescott L and Spence C (zoo9). The influence of color and label on
the perceptions of flavor. Chemosensory Perception, 2, 53-8.
::::. .i,¡tLrtutl of Applied Shankar MU, Levitan CA, and Spence C (zoro). Grape expectations: The role of cognitive
influences in color-flavor interactions. Consciousness (t Cognition, r9,38o-9o.
. : crring area. Journal Smith PC and Curnow R (1966). 'Arousal hlpothesis' and the effects of music on purchasing
behavior. Journal of Applied Psychology, 5o, 255-6.
=:: "rd purchase inten- Smith R (zoo8). The louder the music, the faster men drink beer. The Daily Telegraph,lulyry
:
(News), rz.
': :.,. : --. Ortbrd University Spangenberg ER, Grohmann B, and Sprott DE (zoo5). It's beginning to sme1l (and sound) a lot
like Christmas: The interactive effects of ambient scent and music in a retail setting. Journal
.:::;ts product choice. of Business Research, 58, 1583-9.
Spence C (zooz). The ICI report on the secret of the senses. The Communication Group,
:,-!-ore music on wine London.
Spence C (zoos). Multisensory perception. In H. Blumenthal, The Big Fat Duck Cookbook,
, - . -: on restaurant cus- London: Bloomsbury.
Spence C, Sanabria D, and Soto-Faraco S (zoo7). Intersensory Gestalten and crossmodal scene
a: ::. ,1ctn,tty of the cere- perception. In K Noguchi, ed. Psychology of Beauty and Kansei: New Horizons of Gestalt
P erception, pp. 5rg-7 g. Fuzanbo International, Tokyo.
-:,: -n flavor preference Spence C and Zampini M (zoo6). Auditory contributions to multisensory product perception.
Acta Acustica united with Acustica, gz, roog-25.
'.,::,... Haryard Business
Srinivasan M (rSif). Has the ear a role in registering flavour? Bulletín of the Central Food
Technology Research Institute Mysore (India), q,r¡.6.
: ..: f ¿n the eye fool the Stevenson RJ, and Oaten M (zoo8). The effect of appropriate and inappropriate stimulus color
on odor discrimination. Perception 6 Psychophysics, 70, 640-6.
46 CHARLES SPENCE, MAYA U. SHANKAR, AND HESTON BLUMENTHAL
'The frsh with a memory who knows when it's time to eat' (zoo8). Downloaded from Zampim M and Sper;-
MailOnline at: http://www.dailymail.co.uk/sciencetech/article-5rr7o6lThe-fish-memory- beverage using auc--
knorvs-time-eat.htm1 on z5 ]une, zoo8. Zampini M, Sanabri¿ I
Treasure I éooZ). Sound Business. Management Books zooo Ltd, Cirencester. flavor: Assessir-rg tt.
Turley LW and Milliman RE (zooo). Atmospheric effects on shopping behavior: a review of Quality andPrefer¿t.-
the experimen ral evidence. lournal of Business Research, 49,193-211.
Vatakis A and Spence C (zooz). Crossmodal binding: Evaluating the 'unity assumption' using
audiovisual speech stimuli. Perception and Psychophyslcs, 69, 744-56.
Vatakis A and Spence C (zooS). Evaluating the influence of the 'unity assumption' on the
temporal perception of realistic audiovisual stimuli. Acta Psychologica, t.27, 72-23.
Verhagen ]V and Engelen L (zoo6). The neurocognitive bases of human multimodal food
perception: Sensory integration. Neuroscience and Biobehavioral Reviews,3o, 613-5o.
Vickers ZM (r98r). Relationships of chewing sounds to judgments of crispness, crunchiness
and harclness. Iournal of Food Science, 47, tz7-4.
VickersZM(rqs¡).Pleasantnessoffoodsounds. JournalofFoodScience,4S,TS3-6.
Vickers Z (tggt). Sound perception and food quality. lournal of Foorl Quality, l.4, 87-96.
Vickers Z and Bourne MC (rqZ6). A psychoacoustical theory of crispness. Journal of Food
Science,4r, 158-64.
Vickers ZM and Wasserman SS (1979). Sensory qualities of food sounds based on individual
perceptions. lournal of Texture Studies, to, 31,9-32.
von Békésy G (r963). Interaction ofpaired sensory stimuli and conduction in peripheral nerves.
Journal of Applied Physiology, l.8, rz76-84.
Wansink B (r992). Listen to the music: Its impact on affect, perceived time passage and applause.
In I Sherry and B Sternthal, eds. Advances in Consumer Research,tg,Tt5-78. Association for
Consumer Research, Provo.
Wansink B (zooz). Changing eating habits on the home front: lost lessons from World War II
research. Journal of Public Policy and Marketing, zr (Spring), 9o-9.
Wansink B, Park SB, Sonka S, and Morganosky M (zooo). Horv soy labeling influences prefer-
ence and laste. International Food and Agribusiness Management Review,3,85-94.
Wansink B, van Ittersum K, and Painter IE (zooS). How descriptive food names bias sensory
perceptions in restaurants . Food Quality and Preference, t6,393-4oo.
Wheatley I (1973). Putting colour into marketing. Marketing, October 24-29, 67.
Wilson S (zoo3). The effect of music on perceived atmosphere andpurchase intentions in a
restaurant. Psychology of Music,3t, 93-712.
Yalch R and Spangenberg E (r99o). Effects of store music on shopping behavior. lournal of
Services Marketing, 4, 3r-9.
Yalch R and Spangenberg E (1993). Using store music for retail zone: a field experiment.
Adyances in Consumer Research, 2c, 672-6.
Yalch RF and Spangenberg ER (zooo). The effects of music in a retail setting on real and per-
ceived shopping times. Journal of Business Research, 49, 139-47.
Yeomans MR, Chambers L, Blumenthal H, and Blake A (zoo8). The role of expectancy in
sensory and hedonic evaluation: the case of smoked salmon ice-cream. Food Quality and
Preference, 19, 565-73.
Yorkston EA and Menon G (zoo4). A sound idea: phonetic effects of brand names on con-
sumer judgem ents. Journal of Consumer Research, 3r, 43-51.
Zampini M and Spence C (zoo4). The role of auditory cues in modulating the perceived crisp-
ness and staleness of potato chips. Journal of Sensory Science, ry, 347*q.
\ BLUTIENTHAL SOUND BITES 237
. Dorr-nloaded from Zamprni M and Spence C (zoo5). Modifying the multisensory perception of a carbonated
: The-fish-memory- beverage using auditory clres. Food Quality and Preference, l.6, 632-4t.
Zampini M, Sanabria D, Phillips N, and Spence C (zoo7). The multisensory perception of
flavor: Assessing the influence of color cues on flavor discrimination responses. Food
--:t¿r'ior: a review of Quality and Preference, l.8, 975-84.
', assunption'using
:ssumption' on the
:, : -r :eripheral nerves.
:= :lfluences prefer-
1,. _3. Ej-9,+.
: ::¿nes bias sensory
-29.ó,
'* ::r:se lntentrons ln a
¡eh¿r.ior. Journal of
; neld experiment.
-. :-,-: of expectancy in
::.:::-. Food Quality and
THINKING
MULTISENSORY
CULTUREYT
Vrsuar and cultural studies were founded with the intention to correct the apparent
elitism and disciplinary narrowness of art history and related disciplines. However,
the turn toward visual culture has left in place the sensory hierarchy that subtends
Western philosophy, in which only the distance senses are vehicles of knowledge,
ánd Western aesthetics, in which only vision and hearing can be vehicles of beauty.
It seems that the democratization of the object of aesthetic study to include high
and low or popular arts has not really extended to non-visual objects, except for the
audiovisual arts such as cinema.
The neglect oftouch, smell, and taste (and, to some extent, hearing) in visual culture
descends particularly, of course, from'art history, and generally, from the tendency
to dismiss the proximal senses as inferior that underpins Western thought.
To include sense experience in our cultural analysis, we need to revisit the sensory
hierarchy-while trying to retain the capacity for aesthetic judgement, knowledge,
and ethics associated with the 'higher' senses. And we have to do it belore thé new mar-
kéli"'g olall sense experience does it for us. Recent questions of the affective dimen-
sion of sensuous experience permit new dimensions of epistemology and ethics that
are immanent, grounded in the particularity of experience. Sense experience operates
at a membrane between the sensible and the thinkable, and, ab facques Ranciére
lrgue¡l-
art'exacerbatesthis relationShlp:'the pláce of irt ii thÉ placeof añ adequation between-
; s"ñii6t¿ diffáiént'frb;=itsé*lf ánd ulho,rght different from itself (Ranciére zooo).
tThis chapter was originally published in zooS in Paragraph and is reprinted here with
permission of Edinburgh University Press.
240 I,AURA U. MARKS
ferm aesthesis was introduced into philosophy relatively late, by Baumgarten in the 5-alpha-androstenc,-.
eighteenth century, anditwas Lessingwho definedaesthetics asthe connectionbetween
sexual availabili¡ r i' ,
This three-part ::rÉ
the arts and the senses-but only the higher senses of vision and hearing.
semantic windort i :
In transcendental philosophy, knowledge (in Baumgarten's term, noetq, epísle-
spectrum. When r^,. i.-
mology) must be emancipated from the senses, even though it may arrive through
perception. The object of a sense perception (Baumgarten's term aestheta,perception) heimlich: it ceases :,- ;
can be called beautiful, as long as it is a distance sense-vision or hearing. Kant and
rve have in commLrrr ,!
naturalness of sme-], ,.
Hegel both developed a hierarchical relation between the two remaining senses deemed
the material to the :', :-
capable of aesthetic experience, with hearing, the inner, temporal, and more spiritual
sense being above vision, the outer, spatial, and more physical sense (Rée zooo). For
similarly, are olten ..,
themselves, Yet are rr:
Kant, the bodily senses were capable of agreeable sensations, but not of beauty. Hegel
argued that aesthetics is the transcendent rise from a sensuous particular to a universal
and refines our an::,-r,
truth, and was only possible through the distance senses: uncannybecause r: :.-
If sense experier:c.
Smell, taste, and touch have to do with matter as such and its immediately sensuous qualities ask, to what degr:r ,
. . . For this reason these senses cannot have to do with artistic objects, which are meant to
cial? I argue that ¿,- s.
maintain themselves in their real independence and allow of no purely sensuous relationship. -
What is agreeable for these senses is not the beauty of art.
eminently teachab,-..
(Cited in Korsmeyer 1999)
rrell as in the lite c,:.
perfumery, or anL¡t::a:
The place left for sensuous refinement is pleasure-the proximal senses are hedonic. level of neural plas:,c,
I propose that the proximal senses-touch, taste, and smell-are not only hedonic that they can be m¿.:,
but may also be senses of knowledge (epistemology), vehicles of beauty (aesthetics), There is an aspe .r -,.
and even media of ethics. In this essay my examples focus around smell-some- :hose olfacton- eve:.!
what artificially, since most sensory events combine the experience of several senses of general socializ¿i,,-
simultaneously, but for purposes of argument. Why smell in particular? Because smell ¿ssociations that are ¡
often brings with it a freight of personal affect: it seems to be the least translatable rhe importance o¡ s.'
trrr fundamental cc::
' r) Truffle oi1. z) Iasmine oil. neuroscience sup:.:r
THINKING MULTISENSORY CULTURE 247
i :-rnderstand any sense and most personal of all the senses. Smells can be semantically coded, but less easily
¿.ll-,' infi nite:'outward' than other sense perceptions for this reason. If smell can be a medium of shared
a;id Guattari 1994). knowledge, then I'd argue any sense can. I also note that smell may be the sense that
, ¿ reproduced.
Ideally, best shows the mutual limits of psychoanalysis and non-psychoanal¡ic affect theory,
."!' ¡¡sument rests, you in that the former pays little attention to the singularity of sense experience, while
-',,'iIat they are. the latter does not have a concept of repression, both of which are important in the
Ask a
I:'ornote,l but do not experience of smell.
:e 5ut avoid touching What you have been smelling is oil of truffles, which many people find has a
l:sht have with it. For provocatively, even unbearably gamey smell. Note how it's reassuring to have a name
i ;p','our nose without associated with what you smelled. In fact, you may be able to smell it better now:
studies show that an odour smells stronger when the smeller knows what it is. When
you smelled what turned out to be truffles, you may well have thought you smelled
ihilosophy:Aristotle human body odour, or some other organic smell that many people find noxious.
r: th. 'noble' senses. Though semantically identifying the source of the smell probably makes it less noxious,
;rrses providing reli- you were in a sense right. Truffles have a chemical fingerprint that's close to human
Dhanani 1994). The body odour. This is why female pigs are used to hunt truffles. Truffles secrete a steroid,
:, Baumgarten in the 5-alpha-androstenol, which is the same steroid that male pigs secrete to advertise their
la:Lrnnectionbetween sexual availability; it's also in human male perspiration and female urine.
:e.:rhq. This three-part smell story-smelling, associating, knowing-shows that the ideal
j ..rrn. rtoeta, episte- semantic window for perceiving smells may be in the middle of their signi$ring
: :r:¿.,- arrive through spectrum. When we know too little, or too much, about a smell, it moves beyond the
: .;¿-;¡iicirl, perception) heimlich: it ceases to communicate humanly. The smell of truffle reminds us of what
:,: :earing. Kant and we have in common with other desiring creatures. It does not reveal the fundamental
r!-rLnq senses deemed naturalness of smell; it shows that cultivated odours operate across a membrane from
,,1. :nd more spiritual the material to the syrnbolic, the asocial to the communal. The base notes of perfumes,
s¿nse Rée zooo). For similarly, are often sexual or animal odours that we have learned to find noxious in
: n,,11 of beauty. Hegel themselves, yet are seductive when masked. A fine meal or elegant perfume both recalls
l:':¡ular to a universal and refines our animal and vegetable nature. Smell, the chemical communication, is
uncannybecause it reminds us what we have in common with pigs-and with fungi'
If sense experience is to be analysed culturally, it must be communicable; we can
.:::-r' sensuous qualitiei
ask, to what degree is sense experience culturally coded, to what degree is it aso-
::-.. :rhich are meant to
i :.:rsJous relationship. cial? I argue that all sense perceptions have aspects of both. The proximal senses are
eminently teachable. Witness the cultivation of sense knowledges across cultures, as
:..: r Korsmeyer 1999) well as in the life of an individual learning archery, auto mechanics, Thai cooking,
perfumery, or another multisensory skill. This educability of the sense extends to the
r¿. senses are hedonic. level of neural plasticity. The educability of the proximal (indeed all) senses indicates
-¿:e ¡tot only hedonic that they can be means of communication, and thus of knowledge and aesthetics.
r : beautr. (aesthetics), There is an aspect of smell experience that is learnedbut not communicable: namely,
:.-und smell those olfactory events that are important in an individual's life but repressed as part
:nce of several senses of general socialization. Smell populates the imaginary, for it has intense personal
ir¿ular? Because smell associations that are difficult to communicate. Freud himself seems to have repressed
-;l:¿ least translatable
the importance of smell, harshly criticizing his friend Wilhelm Fleiss who argued
for fundamental connections between olfaction and sexuality. Recent findings in
neuroscience support the psychoanal¡ic argument for links between olfaction and
242 LAURA U. MARKS
intense emotional experience, including repressed experience. The power of smell to (d. 866)writings on ::
elicit memory has partly to do with the unique 'wiring' of olfaction directly to neural to notes and rhrtl:rt
centres for emotion and memory (the amygdala and the hippocampus), before it con- music, colours, and :
nects to cognition (the cortex) (Stopfer and Laurent 1999). Emotionally intense expe- created a place for .,,
riences (not all experiences) are likely to cement an association between the emotion Optics, described pe:;
and the odour associated with the event (hence the olfactory imaginary), even if the compared, as in the ¡l
event itself is forgotten. We may think of such events as lodged in a sensuous uncon- bank listening to m::.
sclous, which may be brought to consciousness, and its stories with it-as in the stories olfactory knorvledsc :
that tumbled from Proust's fragrant encounter with a madeleine dunked in a tisane of maintaining an in:¿:'
tilleul. each other, and the c';-
Thus the phylogenetic leaving-behind of smell that Freud posited in human his- And of course, Arat .
tory-where, in standing up, humans distanced themselves from each other's bodily classes of the trfidcr¡
smells and the knowledge that they provided-is recapitulated in individual human myrrh spurred exri,-, : ;
experience. Such a repression has occurred in recent histories of civilization as well: Recent scholars:,:
for example, Alain Corbin (1986) describes the eradication of 'unpleasant' smells (nowledge. Curre::'
as part of the construction of bourgeois experience in eighteenth- and nineteenth- ing from Marshall l,.i:
century France-smells of sewage, rot, corruption, body odour, in short the smells Classen and anthr--:
connected with death and impermanence were censored from bourgeois life and history can pursue ::.
replaced by sweet, 'clean' smells. In the resulting deodorized environment, smell ceased currents that inlor:t
to be useful as an epistemological sense and became simply a sense for pleasure-and others, and reiatirr:. .
a very limited palette of pleasures at that. Smell ceased to be a means of knowledge,
especially of knowledge to do with death and danger.
Smell is thus triply repressed. Smell is so strongly associated with excrement,
sexuality, filth, poverty, and other repressed contents of both individual and cul-
tural history, that even innocent smells have a taboo, or at least asocial dimension.
We have more smell experience than we are able to talk about, because smells are
Oor
inextricable from bodily events that are repressed. Smelling in public, or even talk-
ing about smell, seems to violate a private realm, the olfactory imaginary. Finally, The body and its s¡,,
there is an aspect of sense experience that is neither educable nor communicable, variety of laudable :r
not associated with repressed experience, and indeed has little association at all. The already twisted th: :;:
'wild', uncoded dimension of smell shows that the senses can be vehicles of intensity the inclusion of oi-:;:
that remains free, but that cannot communicate. Across the membrane between com- from critiques oi¿u:,r
municability and incommunicability, the proximal senses build an intensity that is cultural theories pr:-
not social but is the meaningful lining of experience. traditional aesthetic,
The sensory hierarchy is not only a Western phenomenon: most cuitures main- as we've 'seen' the-,
tain some version of sense hierarchy, usually with vision or hearing at the top; but because of their ab--,:
what is interesting is the way in which certain constellations of sense knowledge are the close senses of tc,;
cultivated. Anthropologists are perhaps most aware of the variability of sensuous material world, inc:"
knowledges in different cultures, as in David Howes's (zoo3) account of the complex sense experience a: .:
auditory and olfactory epistemologies of Melanesian people. In a relatively Western the perceived mat¿:,.
genealogy of the senses, classical Islamic civilization cultivated a multisensory aesthet- all the senses, ack;:¡'','.
ics, the latter especially in the luxurious courts of the 'Abbasid caliphate in Baghdad. groundedness of k¡¡,',
Arabic philosophers tended to adopt the Aristotelian conception of the body as inte- the material world ;r
gral to human happiness, and thus valued bodily pleasures in moderation. Al-Kindi's uponmortallr¡. Thus.
THINKING MULTISENSORY CULTURE 243
, :re porver of smell to (d. 866) writings on music adopted the Greek doctrine relating elements and humours
-ron directly to neural to notes and rh¡hms. He developed a kind of multisensory therapy combining
¿rl:us r, before it con- music, colours, and perfumes (Behrens-Abouseif 1998). Islamic aesthetics generally
,,t:t¡nallr- intense expe- created a place for all the senses, as when Ibn Al-Ha¡ham (d. ro39), author of the
'-.e¡r-een
r the emotion Optics, described perception to consist of a compound of sensations that are mentally
::::qi¡an'), even if the compared, as in the sight, sound, and smell experienced by a man sitting on a river-
: :it a -ieÍ-s¿lous uncon- bank listening to music and admiring lovelywomen (Nasir rg6q). Arab cultivation of
:lh lt-as in the stories olfactory knowledge and pleasure remains in customs like offering perfumes to guests,
. ¿urked in a tisane of maintaining an interpersonal distance close enough to allow both parties to smell
each other, and the cultivation of fragrant gardens as terrestrial reminders of paradise.
rL-,sited in human his- And of course, Arab and Indian olfactory pleasure was adapted by European wealthy
::r ¿ach other's bodily classes of the Middle Ages, whose eagerness for pepper, cinnamon, cardamom, and
i .r indir-idual human myrrh spurred explorers to discover continents.
,:,: ¡iliiization as well: Recent scholarship is beginning to develop a multisensory Archaeology of
- -' unpleasant' smells Knowledge. Currently this is the domain mainly of sensory anthropology descend-
:::.:- and nineteenth- ing from Marshall Mcluhan and Walter Ong, as in the work of historian Constance
::. ir short the smells Classen and anthropologist David Howes. Scholars of 'visual culture' and cultural
::1 trlurgeois life and history can pursue the questions: what are the cultural, epistemological, economic
f L-,r,-atr,, smell ceased currents that inform the development of some sense knowledges, the repression of
:ls¿ tbr pleasure-and others, and relations among them?
:r.ans oiknowledge,
lr:¿ excrement,
"rith
: :rdir-idual and cul-
:-.r ;social dimension.
-because
smells are
Opoun REVALUED, AT A cosr
-i.
. :u¡lic, or even talk-
:-- rmaeinary. Finally, The body and its senses have been embraced in scholarship and art practice, for a
: :lrr CommuniCable, variety of laudable motivations. Non-dualist philosophies in the nineteenth century
'ss¡ciation at all. The already twisted the sense hierarchy, as in Nietzsche's valuing of 'flair' or instinct and
¡ .,-ehicles of intensity the inclusion of olfactory experience in C.S. Peirce's (1955) semiotic theory. Emerging
nb:ane between com- from critiques of dualism and idealism, contemporary materialist, feminist, and inter-
.
j an intensity that is cultural theories practically reverse the values ascribed to the sensory hierarchy by
traditional aesthetics. While vision and hearing can be experienced as bodily senses,
::c'st cultures main- as we've 'seen' they are strongly associated with abstraction and transcendence
::rirs at the top; but because of their ability to seem independent of the body. The embodied nature of
: sense knowledge are the close senses of touch, taste, and smell is more evident, and thus they link us to the
ir:e'brlin. of sensuous material world, indeed bringing it close to or into our bodies. An understanding of
;¡,unt of the complex sense experience as embodied resists transcendentalism, for it links perception and
r a relatively Western the perceived material world. Embodied perception, including the experience of
nultisensory aesthet- all the senses, acknowledges the inextricability of perceiver from perceived and the
::,iphate in Baghdad. groundedness of knowledge in local experience. The close senses, which index both
: ofthe body as inte- the material world and the materiality of the body that perceives with them, insist
u' deration. Al-Kindi's upon mortality.Thus,a materialist aesthetics can find value in the close senses precisely
244 I,AIIRA U. MARKS
for their grounded, provisional, and ephemeral character. The immanence and mate- There is a cur:--l
riality of the proximal senses can thus be the ground of aesthetics, knowledge, and seems to be especi:
indeed ethics. from feminist anú -:
However, the contemporary 'bodiiy turn' in scholarship raises new problems. ity, but has devo,"'.;
An inversion of the sensory hierarchy with the bodily senses at the top is not neces- how widespread ,::-.
sarily any more conducive to knowledge or justice than the old hierarchy. Carolyn thinkl' (like'No r,, c'::
Korsmeyer writes, 'One cannot simply add taste and the other bodily senses to phi- is considered a prrs-:-
losophy as it has evolved and correct theories accordingly to be more comprehensive Fashionable ¿r.r-:
in their treatment of sensory worlds' (Korsmeyer ry99, p.37). Instead we must ask, ence: the senses ¿:a :
what are the particular pleasures and knowledges we can get from different kinds of Consumer capit:,-.:--
sensory experience? utopia is here, tho .'::
Second, since the knowledge and pleasure gained by the distance senses are asso- are both educatec ::-.
ciated with transcendence, critical theory seems to have thrown aesthetics (and reasons. That is. c-:
its presumption of a subject capable of transcendent experience) out with the bath an overwhelmrn:',,.
water of dualism. Contemporary critical theory seems to have a lot of unease and bad tion of them. \ I¡': ,
faith about whether we can still speak of aesthetic objects. There is a certain theoreti- iiom that ben_it :r:.
cal squeamishness around the senses' ability to function epistemologically, as well as tiom the globai ci'. -,
aesthetically. For contemporary visual studies, Hegel's dictum could almost be ofsense experienc.
reversed: artistic objects are not meant to maintain themselves in their real indepen- We, the rr'ell-:c -: .
'¡ubble associat¡i -, ,
dence from the material world and can only allow of a purely sensuous relationship.
Bourdieu's semiotic revaluation of taste translates aesthetic pleasure into economic :n impoverishei ..'.
and social terms alone. Contemporary theory can speak of pleasure (considered sub- erce of rotting l: . .
jective), and of politics, but not of beauty. Just when the proximal senses have been : danger. Sr.r.reL ,.¡- -
redeemed, there seems little for them to do-except go shopping (of which more in a .,- car trouble-¡ -
moment). So it is still necessary to argue that the senses are means of knowledge and t,-, rr'ith odLrl-: j ,
lir e
ethics as well as pleasure. 1",:.r.rri1edge. Bti: : :.,,
The contemporary popular tendency to reverse the sense hierarchy and reject the .. sicaL poten'i¡i
transcendental is not the answer. We still need to be able to ask, what knowledge -\ner the oll=:. .
am I gaining from sensuous experience? How can it make life worth living? . *r¡LirS t-loodec :-.. :
Charles Fourier, writing during the Industrial Revolution (r85r), believed a society .:i other boc:l' ,.-
could be judged by the degree to which it gratified and developed the senses of its citi- -:: i¿steless p:-,a:::
zens. The current stinking state of the world was a far cry from the multisensory order : ; hale qi,,-.:-- ', .'
that Fourier imagined for the cosmos, where each planet corresponds to a musical note : ', tr- inpL-r:is. r a -':
and to a fragrance: Earth emanates the odour of violet, |upiter, jonquil, Mercury, rose :'::lei \Iar,,t¡-, -
(Classen r998). Fourier's utopia is idealist, in its eradication of unpleasant odours, but it : -ees. a n:¿:-,=
- : i -
is decidedly corporeal in its appeal to embodied pleasure and knowledge. Marx adapted r :'": i¿ñ -
from Fourier the ideal of liberation of the senses that are abused and impoverished in ::: l9-,-'S {l:', , ,:.
r-
'¡e-,',:,c::::-
the capitalist system of labour, such as the factory workers suffer from noise, heat, : :::-i itt ¡
smell, and immiseration: 'the worker loses all notion of sensory refinement' and 'no --- I -l ¡\a
--1
longer knows any need . . . but the need to eat (cited in Howes 2oo3, p. zo6).'As Howes
notes, Marx had little interest in the sensuous dimension of life, but his call for humans - 1: f3f,S,-r1 l-: ,:
to be able to cultivate our sensory capacities gave a political immediacy to Fourier's : :-\ .r i - r:
fragrant idealism. Sensuous experience can make life worth living by liberating . . .l L L
capacitiesthat are not contained by instrumental purposes. But where, and how?
THINKING MULTISENSORY CULTURE 245
n]ll¿nence and mate- There is a current popular tendency to embrace feeling over thinking, which
¡t:cs. k¡rowledge, and seems to be especially embraced on the West Coast (of North America): it descends
from feminist and other critiques of mind-body dualism and instrumental rational-
:¿-ses ne\\. problems. ity, but has devolved into an anti-rationalism that is positively lazy. I am not sure
.ijle top is not neces- how widespread this tendency is, but in my west-coast city of Vancouver 'Feel, don't
.J h;erarchy. Carolyn think!' (like 'No worries'-when surely there are things we should be worrying about)
¡¡cilr'senses to phi- is considered a positive slogan.
r-1rre comprehensive Fashionable anti-rationalism plays into the commercial capture of sense experi-
,rstead rve must ask, ence: the senses are being sold back to us as means not of knowledge but of pleasure.
t i:.i d,ifterent kinds of Consumer capitalism is conquering the bodily senses from nose to toes. Fourier's
utopia is here, though not in the way he anticipated. The well-to-do people of the world
l:lae senses are asso- are both educated and gratified in their senses-but for hedonic, not epistemological
-,,'*',:r aesthetics (and
reasons. That is, contemporary post-industrial society affords its consuming classes
:c trut rvith the bath an overwhelming wealth of sense experience and the means to refine our apprecia-
- --: oi unease and bad tion of them. Meanwhile the world's poor live in sensuous poverty not so different
. -' ¿ certain theoreti- from that bemoaned by Fourier and Marx. Among the many inequities resulting
:l; logicallr', as well as from the global division of classes into the very rich and the very poor is an inequity
,:rl could almost be ofsense experience.
-r :ireir real indepen- We, the well-to-do of the Western and Westernized world, still live in the deodorized
::r s uous relationship. bubble associated with the rise of the bourgeoisie, both European and global. Smell is
::slre into economic an impoverished sense. We still use smell for survival reasons-to check for the pres-
;;:: considered sub- ence of rotting meat, mould, infection, sewage, gas leaks, and other things that pose
:r:l senses have been a danger. Smell knowledge is arguably most useful for distinguishing different kinds
; '--irrhich more in a of car trouble-oil burning, brake friction, coolant leaks. Poor people are more likely
i:. ,:,f knorvledge and to live with odours that index real events, including danger-and thus are objects of
knowledge. But these are the few opportunities that smell has to exercise its epistemo-
:::;nr- and reject the logical potential.
:¡i. ir-hat knowledge After the olfactory whitewashing of Westernized middle-class life, pleasing new
:,:r 1lr ing? odours flooded the blank canvas. In the past couple of decades the use of smell, and
.. . belier-ed a society" the other bodily senses, for pleasurable and refined consumption has accelerated.
i the senses of its citi- The tasteless processed foods that signified modernity in the r95os and r96os (Belasco
.e :rultisensory order zoo5) have given way to a great variety of fresh produce, often locally grown, as well as
:i.s ro a musical note exotic imports, because of industrial diversification and faster shipping methods. The
r:rqur1. Ilercury, rose canned Maxwell House of zo years ago has given way to a dazzling array of specialty
,,¡,'sant odours, but it coffees, a marvel of niche marketing-North Americans drink less coffee than they
' -:cge. \Iarxadapted did in the r96os but they spend more on it. Wine, which those of us who were alive
:.r:d impoverished in in the r.97os knew in the categories red, white, and rosé, is now produced and mar-
:;: lrom noise, heat, keted in bewildering variety, requiring middle-class consumers to become, or act like,
' :;inement' and 'no connoisseurs. Dish-soap comes with aromatherapeutic promises, soothing or stimu-
_¡. :. :o6).'As Howes lating, etcetera.
:; his call for humans The reason for this explosion of sensuously pleasing products is, of course, the
r¿diao'to Fourier's exhaustion of markets and the need of corporations to induce consumers to continue
-r-, ine by liberating to spend. Multisensory products promise to enhance individuality-consumers
'h¿:e. and how?
are encouraged to design signature scents, express themselves through gourmet
LAURA U. MARKS
cooking, etc.-confer cultural capital-connoisseurship of wine, coffee, chocolate, smelling jasmine. i,, -
'artisanal'bread, and other luxury goods and the concomitant opportunity for snob- This time I noticec ::
bism-and enhance sensuous pleasure. This they probably do: it is a great thing to markets with camt-,.,-,
savour good wine, choose among many varieties of international tasty food, inhale 'noses' capable oi d-s:
stimulating fragrances in the hurried morning shower, and bury one's nose in Dove at the fragrance sou\
cleansing pads for a brief respite from screaming children, holiday in-laws, and airport Hugo Boss, and othe:
delays (scenarios provided by Dove advertisements). my research dream i,-
However, as these descriptions of pleasure begin to indicate, multisensory prod- given the Bush adn.r:r
ucts are conceived to defray the pressures of contemporary life, where most people, the last Gulf war, thi.
even middle-class people, are working too hard and have too little time. Many aroma- the little vials with a .-,
therapy products and bath products emphasize the time of consumption, 'time for drops at customers. -,.,'
me'; they sell a product in the name of creating free time, and their principal target atmosphere of sharel
seems to be harried, hard-working women.2 Sensuous pleasure is sold to us funda- This time I hapr.:r
mentally to ensure that we get back to work-relaxed, refreshed, and ready. Thus one the precious oil of t..:
of the most cynical promises of the new sensuous commodities is that they will give you are perhaps sn-l." I
people more time. replacedbysynthet.; I
Sensuous refinement now serves mainly hedonic ends; the aesthetic, epistemo- of real smells, the l
logical, and ethical dimensions of sensuous experience are far from the commodity of jasmine, appro\rr::
landscape. The 'emancipation of the senses' envisioned by Fourier is only a further a strong base note c,-
enslavement if our senses are marshalled only for the purposes of consumption. The base note of excren:::
proximal senses, in their intimacy, relation to memory (especially smell), and affec- Such economicallr-;:-
tive intensity, are the verF senses most resistant to mass communication. Sensory be worried if all the :,
commodities that invade and encode the private space of the proximal senses are future; with what -.,,'rl'
Please turn to your olfactory illustrations and smell substance number two. philosophical treat:r --
Two years ago I returned to the Souk al-Hamadiya in Damascus. This souk, where involves the natu¡e ,- :
people have plied their trades in carpets, silks, blown glass, hammered brass, and of being directed ..u:-
other handcrafted goods, has expanded its focus with the changed direction ofglobal bodily and cultural .L::
trade, as well as the vast expansion of the Syrian factory economy. You can get every- operate at the lrtere- ¡
thing there-cheek by jowl with the metalworkers' souk are now the plastic kitchen communication th¡.''
equipment souk, the imported biscuit souk, and the outrageous ladies' underwear trust, and sociali¡-.
souk. But I was confidently in search of essential oils: the attar of rose, jasmine, Recall that inco¡::::
amber, (a tiny flower with overpowering scent), and other locally extracted scents
fl as comprising an ci-:
that perfumers have imported from the orient for hundreds of years. You are now dimension, adaptin: :
a passion (Spinoza : :
' I thank Suzanne Lindgren, a student in Art and Culture Studies at Simon Fraser University, for narrative (Massunti :,
pointing out this temporal-olfactory paradox. of gathering force. -,,,:'
THINKING MULTISENSORY CULTURE 247
:re. cottee, chocolate, smelling jasmine, which just possibly might come from the Souk al-Hamadiya.
r¡pLrrtunity for snob- This time I noticed that the expert Syrian knockoff artists, who have flooded Arab
: r: is a great thing to markets with camouflage Nike and misspelled Calvin Kline garments, now include
::: tastv food, inhale 'noses' capable of discerning the elements of a composite perfume-for it is possible
-.
',rne 5 nOSe in DOve at the fragrance souk to buy reasonable facsimiles of Coco Chanel, Tommy Hilfiger,
i- :n-iarvs, and airport Hugo Boss, and other commercial fragrances for a tinyfraction of the price. Indeed it's
my research dream to visit a Syrian perfume factory and meet the great 'nose'-but
. nuitisensory prod- given the Bush administration's practice of bombing Iraqi perfume factories during
. i., he re most people, the last Gulf war, this is probably not likely any time soon. The perfume monger fills
. rime. IIany aroma- the little vials with a syringe of precious oil, then in a jolly manner sprays the remaining
:su:nption, 'time for drops at customers, who spread out their arms to be engulfed by fragrance. It's a social
:::eir principai target atmosphere of shared smelling.
r: sr-rid to us funda- This time I happened to look at the vat from which the vendor syringe-extracted
¿:d readr.. Thus one the precious oil of jasmine that is so tipical of this part of the world. It-this jasmine
:s rhat they will give you are perhaps smelling-was a German synthetic. Complex original smells are being
replaced by synthetic smells at what ought to be the centre of original smells, the library
:tsietic, epistemo- of real smells, the Damascus perfume souk. Synthetic jasmine is like a photocopy
::m the commodity of jasmine, approximating its complexity with a compound of a few chemicals and
r¿: is onlv a further a strong base note of petroleum. (By contrast, it seems to me that real jasmine has a
, : c,-rnsr-rmption. The base note of excrement, which is perhaps why jasmine is planted outside lavatories.)
-i' snlell), and affec- Such economically driven smell replacements worry me the way a film archivist would
:u:lication. Sensory be worried if all the 35mm reels were replaced by MPEGs. What will we study in the
::c,r,i¡ral senses are future; with what will we enrich and educate our senses?
Yet the socialness of the perfume souk at Hamadiya involves another dimension of
smell experience that is perhaps more durable than smell itself.
I have mentioned a few times the ethical dimension of multisensory experience,
and it is high time to explain this seeming oxymoron. It has to do üth the position
of the proximal senses on the membrane between shared and private, codified and
uncodable experience-on the middle bandwidth of sense experience. Korsmeyer,
a philosopher, argues (against Bourdieu) that the philosophical baby can indeed be
retained when we toss out the bath water of purist aesthetics: the issue is what kinds of
:r-r nfo. philosophicaltreatment do the close senses invite? (Korsmeyer r999,p.66). Her answer
us. This souk, where involves the nature of the proximal senses as both objective and subjective: capable
i:rrnered brass, and of being directed outward (epistemological), and inward (hedonic) within a set of
i J.irection of global bodily and cultural limitations (Korsmeyer ry9g,pp.94-8). Thus, the proximal senses
,'.fou can get every- operate at the literal border between the intimate and the communal. Knowledge and
,' :he plastic kitchen communication that makes use of them maylose in'objectivity'but theygain in depth,
:s -¿dies' underwear trust, and sociality.
ar of rose, jasmine, Recall that incommunicable dimension of sensuous experience, which I qualified
¿ll"' ertracted scents as comprising an olfactory unconscious. We might refer to this incommunicable
lears. You are now dimension, adapting Spinoza's ethics, as affect. Between passive and active, affect is
a passion (Spinoza): an intensity, an 'excess,' a suspension of the linear progress of
i :.:. :r University, for narrative (Massumi zoo2,p. z6). As in the affection-image of Deleuze, it is a moment
of gathering force, which may or may never be acted upon. This is a volatile moment.
248 LAURA U. MARKS
It's when a person feels the great pressure and potential of the virtual-of the community, even ii ¡,:
broad realm of possibilities, one of which can be summoned into being. It is not yet may be actualized. S--,
communicable. a social sense as the ..'-r
To what degree can affect be experienced in common, socially? The concept by the smeller. \\-h¿. :
of excess from film theory, for example, shows that affect is marshalled around a to which more is goi::.
common image, as when audience members all cry at a movie like the classic'weepie' group. Thethickness ,-- i
Stella Dallas, salivate emotively during Chocolat, or feel the urge to slay Persians while not even be identifiec :
watching 3oo. However, I would argue that affection-images allow people to respond that is its virtue. Per',:,
as they will-not necessarily all in the same way. What the participants in an affec- and complexified br =
tive situation have in common is that some affection-image stimulated their affec- experience. People ;:-'
tive response-this intensity, this passion, not yet bound as emotion, and not yet virtual to actual is er¡_
communicqble-and they had the response together. So they would attribute to the (freon, melted plas:-;
affection-image, be it Stella Dallas, Chocolat,3oo> or a sudden rush of jasmine, the the love he gave me h:
ability to arouse affect in them. This is Brian Massumi's reasoning when he argues missed appointmelt.',
that the television addresses of former President Reagan unleash affect, which makes In its autonomr'. "-,
people feel good, and vote for the man, even without knowing why (Massumi, 2oo2, omy ofaffect is its p::..
pp.39-42). as emotion,'Someti:,::
Affect arises from a break in the continuity of experience> a shock in Walter any particular, func:r . :
Benjamin's sense. It may thus have the critical ability to disrupt the clichéd narrative to the degree to rrh,.:
of daily life. For example, I'm working at my crummy telemarketing job and some- In certain \r?r'S th:
one passes wearing the perfume my mother wore when she used to kiss me goodnight Its result is not. tor .:'.
years ago. I feel the giddy vertigo oftravel, between actual and virtual, then a resur- social or political et.;
gence of the original emotion (love? safety? excitement? jealousy?) beforethe memory virtual powers rr ith ', ''-
resolves itself. For this is how smell memory works-first you experience the disrup- ized in the social, i::.',
tion, then you feel the emotion, then you identif' the source. Such an experience Let us conclude -,'
could well make me burst into tears in my nasty little cubicle andremindme of the vast subtends. I've men:--
powers of the virtual hovering below the surface of my crummy actual life. it sometimes brins. :,
The closer the sense, the stronger the effect: smell seems to be the perception capa- private, between ¡¡,:,:
ble of the most powerful affection-image, though I note that music seems to have an rience and uncodi'.::
effect exactly analogous to smell-a heard fragment of an old song arouses piercingly ity, not between nt::::
specific rnemories in just the same way as a whiff of a scent from long ago may. Physical one of the war.s oi tr.',
gestures, tastes, and colours can also sometimes call up intensely embodied emotional the onlyperceptrb,: - :
memories, though less often than smell and music. be the mascot of th:: -
In an essay titled 'The Logic of Smell' I argued that smell cannot be a basis of com-
munication (Marks zooz).I still believe that smell's asocial nature is a great virtue of
this sense. The charged individual responses lhaf may gather around particular smells
protect olfaction from ever being completely culturally coded. This is a value in an
economy that is forever finding new ways to encode meaning in sensory experience in
order to eke out more profit.
However, smell is also, like the other senses, a social sense. As Korsmeyer points
out, taste is a social sense because one brings one's memories associated with par- I thank two grouis '
ticular foods to the present encounter. Smell functions similarly. Being in the pres- for the StudyofVis --:
ence of smells familiar from the past sometimes has the quality of socializing with a of Modern and trle c:.
THINKING MULTISENSORY CULTURE 249
¡f lhe r-irtual-of the community, even if others are absent. Such associations are a virtual presence that
rti{-r being. It is not yet may be actualized. So too, smell, though it is even less codifiable than taste, can be
a social sense as the virtual experiences it evokes are aclualized, and communicated,
s,,ci¿11r.? The concept by the smeller. What's rich about these experiences is lheir virtual depth: the degree
, na¡shalled around a to which more is going on, more is potentially present, to each individual and to the
,:e the classic'weepie' group. The thickness of communication includes what is /rotcommunicated, what may
- :rr sia\-Persians while not even be identified as more than a potentiality. Affect is incommunicable per se, and
-,-r.,r people to respond that is its virtue. People may respond in common, but that response will be enriched
l:-r:ipants in an affec- and complexified by a core of absolutely individual and relatively incommunicable
;ti:rulated their affec- experience. People cry at the movies, even happy movies, because the passage from
Irtt-tion, and not yet virtual to actual is exquisite-it is painful. I cried once when an unanticipated smell
;';'-,uld attribute to the (freon, melted plastic and cigarette smoke) reminded me of my grandfather, because
: :ush of jasmine, the the love he gave me has slipped forever back into the virtual. Affect reminds people of
- r::lg rvhen he argues missed appointments with love.
i: aiilct, rvhich makes In its autonomy, affect is the foundation of an ethics. Massumi writes, 'The auton-
i., :r'!- -\ Iassum i, zoo2, omy ofaffectis its participation inthevirtual' (zooz, p.35). Evenwhen affect is expressed
as emotion, 'something remains unactualized, inseparable from but unassimilable to
:. ¿ shock in Walter any particular, functionallyanchoredperspective Ioriginal italics] ' (zooz, p.3l). Affect,
i the clichéd narrative to the degree to which it remains free, is the force underlying emotion and action.
:-ii11-i{ iob and Some- In certain ways the protection of the freedom of affect is a conservative goal.
i-.': iss me goodnight Its result is not, for example, social action, but a retaining of the quality of life. Any
' -fual, then a resur- social or political effect comes later and not necessarily. If people could believe that the
,': i.:.iore the memory virtual powers with which they came into contact through affect could reallybe actual-
-';; erience the disrup- ized in the social, they would not just weep, they would proceed to fight.
:u¡h an experience Let us conclude with another look at that sensory hierarchy and the dualism it
:.:rind me of the vast subtends. I've mentioned that we may think of sense experience, and the affect that
";:ual life. it sometimes brings forth, as a membrane between several pairs: between social and
: :r. Derception capa- private, between communicability and incommunicability, between codified expe-
;):a seems to have an rience and uncodifiable experience. In a certain way this membrane reinstates a dual-
:i arouses piercingly, ity, not between matter and spirit but between actual and virtual. Sense experience is
::rs aqo may. Physical one of the ways of traversing this duality. Smell (which, as the ancients remarked, was
;:rbodied emotional the only perceptible object that actually comes into contact with the brain itself) could
be the mascot of that creative traversal.
r rt be a basis of com-
:¡e :s a great virtue of
*rd particular smells
Tris is a value in an
::rson'experience in
AcrNowLED GEMENT s
r,s ñorsmeyer points
¿ssociated with par- I thank two groups of listeners for their helpful responses to this paper, at Project
'' " iieing in the pres- for the Study ofVisual Culture of the University of Southern California, and the Faculty
¡,: socializing with a of Modern and Medieval Languages at Cambridge University.
250 LAURA U, MARKS
REpnRBNcEs
disciplinarity louL: :
synonyms. In iact. :,,,
My work continues :.
propose the follort,:-.. .
. Interdisciplinari:,,
more) things Iniri;
identity is transi.-, :::
. Cross-disciplin;:i.,
acter of one thr:.1.
often conflictu¿,.
. Multidisciplinar:r,
non-dominant .-;
predominates.
'. :::n-eton, NJ.
,.-:.:.. In \Vatson JI and
-_::. Biacl¡veli, Oxford.
CHAPTER 13
:- : s :i't¿t ic Imagination.
'','.,;.
:
.
¡,. t,. tt! i o n,
::.1:. In CV Boundas,
M Kochan,
SIGHTING SOUND:
: ::; :. Tomlinson H and LISTENING WITH
. :,; :': Bttsrian Mu'tazili
':
EYES OPEN
:.;l Ti., t t¡ ry,. University
.'..
-.:. tutd Mubisensory SIMON SHAW-MILLER
,,.;.: )'[¿1 s¡ns¡t, Affect,
The prefix'inter' means 'between' or'within" and I am especially interested in the One way of telling ii
relationship between this 'between' and 'within' in art and music. In other words, and the more stylisticalir' :-:
I am well aware that words are different from the material I analyse, my work addresses away from utilitr'. Th¿r
the identities and interstices between the sounding of music and the vision of art' I am from an accompanic.r.r
not principally interested in the categories of vision and audition' Rather, I address of itself as organizeC s,-
art and music as cultural practices with specific histories. While vision and audition ever possible; indeed. :
undoubtedly have histories, they are not equivalent to art and music. Art and music question is primaril-, ,,,
are composed as part of a cultural continuum; it is at the'outer'historical
and cultural to escape strict soci¿i '.
edge of music and art that they contact each other, indeed, in places overlap. its own form a,s tor::r.
We can think of music as the cultural enactment of framing sound, an enactment formal concerns. -\ .,.
that is visual, tactile, and sonoric. The experience of music is complex; listening is not Wagner: 'absolute r'.
and
passive consumption, for our encounter with music is emotional, intellectual, first used it). It is rr-1.-'.:
physical, as well as social and historical. To listen and to see is also to think' and our absence of 'progranr::r
ilrlr-rt ing structures our listening and seeing. Music occurs as sight and
sound, and gist Carl Dahlhaus ;::.
its meaning, in enactment, is a result of mediations between eye and ear, as much
in terms, as music lrit¡ - -
p.rfor-un.., listening, and composition. Music is always more than sound, it is a which also becone. -::
point of focus for a multisensory experience that centres on sound, but is not solely Wagner and the cLrnar:
iefined by it. Analysis, interpretation, performance, and composition are in intricate Through this prtrc=,
dialogue with each other and with the sensorium. Music is, in short, an activity-an I would orgu€, d ni-,r' :
embodied, social, and historical event' helpful terms), an ld-:
The 'faculties' of the university separate the visual arts from the literary and separate from the r,:
sounding arts, as well as they do the humanities and the sciences. sophisticated meth- of relationships, to:::r :
odologies and historiographies have developed to provide tools for visual and aural a five-part choral r,'.:,
analysis. But we are only now beginning to forge ways of thinking in the
gaps between maintain its form¿, :;
disciplines and describing what we find there. Hybridity is still often thought of as r ariation. Notatior :, :
undesirable, it carries with it the ideas of contamination, weakness' even the potential sizing that notatiorl r) :
for chaos and disarray. code, is not also sig:--
There is a long history of music being defined as opposite to this, as abstract, sible to conceive ¿ :r -
pure sound. It is, in part, this perception that allowed music to act as the art to which r-isual code. Certair-. -.
;n arts might aspire, to paraphrase the words of Walter Pater: music as an art of are organized or-e: L::.
formal abstraction and theoretical sophistication, not contamination and chaos' My rdea or gesture cLrnr;:
contention is, however, that precisely because musical sound is abstract, the visual rotational con\-en:r,- -=.
experience of its production is crucial in locating it within history, society' and culture, musical object.
*h.n technology liberates the mind's eye. Music-making, for performers and Around 18oo, ¿ .:.:
",r.r, -iom centre to per,:-;
listeners, is social; it involves communication (in manydirections) between composer,
'a\tra musical', te'r -.,:.
performer, and listener. Sight, together with sound, locates this activity and gives it
meaning (see also Leppert zoo4). :his shift, music rr¿s :.
As an aside, we don't have an equivalent of music anaiysis in art history. Indeed, nL)\\: aspired to a r.}::
it is fairly unique in the humanities to have a practice that claims such a detailed and ,.bsolute practice t:r:
complex set of methodologies to analyse the essential material (the 'music' in music :ess, music emers:ú :
analysis), and that is relatively separate from questions ofhistory' :::d ma1leab1e in irs
'-
To return to the notion of pure music: I have traced this history elsewhere, so here it ;¡ss, 'Such ideas ai-^-:;.
will be enough to briefly summatíze (Miller t993). .:aped by the tun.i-,-
SIGHTING SOUND: LISTENING WITH EYES OPEN L)J
:--::Lli' interested in the One way of telling the story would be to say that art music (I prefer this term to
:;. ln other words, and the more stylistically circumscribed'classical music') developed through a movement
, .:. :r| rvork addresses away from utility. There was a gradual and slow modulation of its terms of address,
: -- "¡ r'ision of art. I am from an accompaniment to life and tasks, to a consideration, analysis, and perception
:: i. Rather, I address of itself as organized sounds. This is not a simple process, for no such total escape is
-,
-; 151e¡ and audition ever possible; indeed, music has meaning because it serves a cultural purpose. So the
::rusic. Art and music question is primarily one of aspiration, a relative achievement, for as music aspired
:,-srt',¡ical and cultural to escape strict social function, it came to be admired in terms of the development of
:aía rr\-erlap. its own form as form, rather than as fit to a purpose outside the appreciation of
: .,: lnd, an enactment formal concerns. A key term coined for this process has its origins with Richard
::--:lerr listening is not Wagner: 'absolute music', a term easier to characlerize negatively (as Wagner indeed
:::-. intellectual, and first used it). It is most often used, for example, to mean music without words, or the
:,s.r, to think, and our absence of 'programmatic' content; it is harder to define positively. The musicolo-
si:-rt and sound, and gist Carl Dahlhaus has, not unproblematically, described absolute music in Kantian
,'- :rd ear, as much in
terms, as music without a'concept, object, and purpose' (Dahlhaus 1989). It is a term
:¡ :han sound, it is a which also becomes embroiled in its opposite, in the work of the man who coined it:
';:l,j. but is not solely
Wagner and the concept of the Gesamtkunstwerk.
!i:-Lrn are in intricate Through this process of abstraction, music changes its nature, but it also becomes,
:lr-,ri. d1 activity-an I would argue, a new kind of object. It becomes a 'work-concept' (in Lydia Goehr's
helpful terms), an identitythat exists, rather like the idea of the soul, as something
:. n the literary and separate from the material world (Goehr ry92). A work functions as a fixed system
¡ ¡..phisticated meth- of relationships, formal identity being sustained beneath the surface of transcription;
,, -r,r r isual and aural a five-part choral work, for example, reconceived for the six strings of a lute, must
:. -n the gaps between maintain its formal identity in order to be recognized as a transcription and not a
'' --'t.en thought of as variation. Notation is central to the development of the work, and it is worth empha-
:s:. elen the potential sizing that notation is a visual code. (This is not to say that Braille notation, as a touch
code, is not also significant.) Notation is so important that it makes it almost impos-
- :¡, this, as abstract, sible to conceive a musical 'idea' independently from its realization in notation, in
.;, :s the art to which visual code. Certainlythis is the case in the work-concept, where musical relationships
::: ¡:-lusic as an art of are organized over large spans of time and in intricate, inter-related ways. The musical
:rltitrn and chaos. My idea or gesture comes to the imagination already contaminated with the rhetoric of
:s :bstract, the visual notational conventions and innovations, presenting itself to the eye, as it becomes a
', . culture,
>:rciel.r', and musical object.
.. :..r performers and Around 18oo, a shift occurred in what was to be understood as musical meaning,
. r iretrreen composer, from centre to periphery. There was a move, from what was thereafter described as
1s rcii\-1fi., and gives it 'extra musical', to what was definitive as 'purely musical': absolute music. Prior to
this shift, music was defined and regulated as much by context as it was by text. Music
: :t history. Indeed, now aspired to a form of emancipation from context, to become an autonomous,
-s s.¡ch a detailed and absolute practice that ultimately depended on nothing but itself. Through this pro-
.:re 'music'in music cess, music emerged as a signifier free from fixed signification, and thus mutable
and malleable in its subsequent rejoining to subjects. As Goehr describes this pro-
:"'elselvhere. so here it cess, 'Such ideas affected eveqthing musical . . . Usually [ideas about music] were
shaped by the functions music served in powerful institutions like the church and
254 SrMON SHAW-MTLLER
court . . . after rSoo fthe extra-musical] was marginahzed by and incorporated into a
new understanding' (Goehr 1992, p . nz) .
Dahlhaus has shown that this shift to absolute music coincided with the absorption
of religious sensibilities into the musical realm, and the development of sensibilities
that allowed the aesthetic contemplation of absolute music, and the ascension ofword-
less music to a position above words. He quotes Wackenroder (through the character
ofJoseph Berglinger):
When foseph was at a big concert, he would sit down in a corner and listen with just the
same devotion as though he were in church . . . In happy and delightful syrnphonies in full
voice, which he loved above ail, it seemed to him as though he saw a frolicking chorus of
youths and maidens dancing in a pleasant meadow . . . All these meaningful sentiments thus
always brought forth corresponding images and new thoughts in his soul: a wonderful gift
of music. . .
(Dahlhaus 1989, p. 8r)
:j tncorporated into a
¡ r¿hgious experience.
, -ild tlorv forth to take
r Eineral backdrop of
::e arts had even cut
r:s, 'The essence of
,,:;i¿ted u'ith. . . [the]
:-r ris 13ü Discourse, Fig. 13.1 An example of an eighteenth-century opera houser the auditorium of
:¡ ¿nd hear every day Teatro La Fenice, Venice
(Photograph : Andreas Praefcke.)
.-..rolds ryú, p. zo7).
lr \\-¿¡l¡"¡¡gder, who
l.: ¿ means to an end,
'-\r: enchanted realm.' The bourgeoisie presented a different agenda, for in the nineteenth century the
':ldane rvorld is taken instrumental concert hall flattened out the audience's address to the stage, with more
:;i ¡emoval of sound people directly facing the performers (Fig. r3.z). Social display, of course, did not
!L:r:rd to become the disappear with this democratization, although it was reduced to peripheral spaces,
:;;ption; such as the interval bar and entrance foyer. The most significant development in
concert hall design was to be found, however, in the attention given to acoustic preci-
::. phenomenal world,
::. ',.,'ire no rvorld at all, sion and control, to enhancing frequency range and controlling the decay ofsound.
These changes went hand-in-hand with the development of the Romantic symphony
,i i'.rl. r, bk.3, sec.3B). and tone poem as a vehicle of expression. Here the spectacle of massed choirs and
hundreds of instrumentalists was clearly coordinated in space and time.
ei its means of con- The disposition of the players in this space also changed during the eighteenth
¡i,ities migrated from and nineteenth centuries. From an intimate arrangement around a keyboard, or con-
- :ractice developed. cealed in a pit, the sartorially coordinated orchestra now emerged on to a raised stage,
shape ofeighteenth- with massed strings-the most visually coordinated section-in the foreground.
t chamber for sound Originally, first and second violins faced each other on opposite sides to aid contra-
:llence to displayand puntal separation, but now they were more commonly neighbours, with violas to
'Ltra'lcast (with the aid
the left, cellos opposite the first violins, and double basses behind the cellos. Bowing
became synchronized. In the middle ground came first the woodwind and then the
256 SrMON SHAW-MTLLER
brass, with the percussion raised along the back. Any choir would have been tiered up
at the back ofthe orchestra.
This new arrangement was anchored around the conductor. The introduction of which al1 the u,orks :1,.:
the baton (usually credited to Louise Spohr in the r8zos), to coordinate and control performed even'dai' : :
the expressive emoting of hundreds of different musicians, ushered in a new language and published at the-¡ ¿:'::
Liszt wrote:
\¡ision is here reEar:
In the name of all musicians, of art, and of social progress, we require: . . . the foundation tbr Hanslick undesi¡¡
of an assembly to be held every five years for religious, dramatic and symphonic music, by .omposers, the distii:
SIGHTING SOUND: LISTENING WITH EYES OPEN 257
re seating
) tfn e rs, Fig. 13.3 Verdi's Requiem being emphatically directed in an outdoor performance
at the Polo Grounds, New York City (1916).
[Photograph: Library of Congress.)
T:¿ rntroduction of which all the works that are considered best in these three categories shall be ceremonialiy
'-:,ünate and control performed every day for a whole month in the Louvre, being purchased by the government,
:: j il a nerv language and published at their expense . . . we require the foundation of a musical Museum.
(quoted in Goehr r99z,p. zo5)
- .r:ailer, controlled
l-.. r-r._i i. At first, the" This is both a plea for the fixing of musical works as absolute objects and for their
=ii:r ¿nd Franz Liszl subsequent display. In this way, value would be socially inscribed and the work's iden-
:.,-nq the audience's
-,r.' tity reinforced. But there is a potential problem with the idea of Liszt championing
:. Fig. r3.4) came
the notion of absolute music and requiring the autonomy of the aesthetic space of the
:¡-is to the audience,
museum to set it apart from the social space outside. Liszt was in part driven to this
.:"i the\. became the
position because he regarded music programme music as equal in rank to poetry and
'. ::d body-language
painting, so deserving of the status of 'fine art'. But programme music was regarded
l:e audience in the
by many as antithetical to music's absolute status, because of its relationship to paint-
;. rnaginatively pro-
ing and poetry. Eduard Hanslick, a key fi.gure in the ideology of absolute music,
famously dismissed Liszt's symphonic poems as denying 'music more completely than
::u,--so and star con-
ever before its independent sphere, and fdosing] the listener with a kind of vision-
::rej earlier. In 1835
promoting medicine' (Hanslickt8gl, pp. 5-6).
Vision is here regarded as linked both to religiosity and imaginative sight; both are
:.: . .thefoundation for Hanslick undesirable in keeping music alone. But for Liszt, and many other
- .'"-r-.phonic music, by composers, the distinction between programme and absolute music was not one of
258 srMoN sHAw-MTLLER
,{¡* o. r*¡
STrRAus$
¡¡ü".t? é
Fig. 13.4 Poster of the farewell tour of Austrian star-composer Eduard Strauss
(t a¡s-1916), lmperial court musical director, and his famous orchestra from
Vienna.
opposites, but rather one of common vision: they were competing concepts for the
same ground: spiritualiy meaningful expression.
Liszt's position as a virtuoso, perhaps rhe virtuoso, also placed him in an ambiva- Fig. 13.5 Caricature s.
lent situation. He was, above all, a visually arresting performer, as Robert Schumann
remarked: 'If Liszt played behind the scenes, a great deal of poetry would be lost'
(schumann ú4oh965, p. 158). Liszt in many ways defines the virtuoso and sets like his supernatu::- .:
the scene and syntax for all subsequent'star'performers (Fig. r3.5). But as well hands and sometinr.:
as representing transcendental expressiveness, he could also be perceived as flashy virtuoso performanc=,
and brash. of enactment and t:. ,
Liszt demonstrated his mastery of his instrument by dominating it visibly as well this when he sald o: -
as musically, as one of his French critics remarked: 'Liszt's attitude at the piano, like iust as the sight oi ar,'.
that of a p¡honess, has been remarked again and again. constantly tossing back his immediate sight of rr.
long hair, his lips quivering, his nostrils palpitating, he swept the auditorium with making il obey' rscl-.u--
the glance of a smiling master' (Ernest Legouvé, quoted in Huneker 1911, p. zg5). He Another element ¡
controlled the piano by encompassing the instrument (and the py.thoness simile is o\ert, sometime: h, '.:
interestingly feminine). His hair, a signifier of visual excess, is constantly in motion, dominance. The sc.',-.
SIGHTING SOUND: LISTENING WITH EYES OPEN
-
i:: :duard Strauss
s :'chestra from
l**¡. ¿ tqti. t, ti&i¡[q*t :!nr¡i! ¡ ¡sM¿ i5] I L*, jttst" ¡_r$¡ '* etbt. & ¡!r&¡ t! lq*t'
Ljpr¡*. l&3 i,;d{ie i r{k{ ,il¿h e!tur ; $1dJ. Iilsl I sdsa*#t h¡¡¡¿. M¡ 6¡{ir {4q4¿ h}' *u¿ e}s
the goal for absolute music, for it reminds us of the physical, effortful body, the grimac- allows us to define his
'
ing face of the performer, and the erotic potential of music, noted since Plato. and its history. Cage :
Theatricality is at the heart of Liszt's performance-a notion decried by all sup- aesthetic of 'letting sc, -:
porters of absolute music, as of modernist art, which itself used absolute music as a is divorced from rr-cr.
paradigm in the development of abstraction. I am thinking here of Michael Fried, sounds in themseh'es. :
who sees theatrrcality as the opposite of art (Fried 1967). In acknowledging the site the music of 4'.33" is ::'
ofreception (and the presence oflisteners/viewers), theatre, in this sense, surrenders So far, I have disc*
absolute and independent cultural authority. Absolute music, on the other hand, would Cage I want to du ei- --
prefer not to be passed through the hands of sweaty musicians, and the eyes as well and its reunion.
as the ears of the audience: theatre debases music by associating it with the visual and Recent reprodu.t.i-,:
the physical. Absolute music would prefer to sound alone. Above and apart from the from site (and sigh: .
particulars of reception and the crowd, it hopes for the ideal universal.' In other words, tion and performanc..
and there is, I believe, a dialectical relationship at work, Liszt is guilty of both theatrical production potenti:---"
music and theatrical performance, which are antithetical to the emerging paradigm dematerialize muslc. .
of absolute music. music cannot escap: .:
But even without specific recourse to a programme, music can still provoke visions. The radio embo.-l:.,
As the poet Heinrich Heine expressed it: lute, even hear.enlr' :.'
into' through techr:'-' .
I avow . . . how very fond I am of Liszt, but his music does not affect my mind agreeably,
itself) invisible, m.:..
the more so that . . . I see the ghosts that other people only hear, so that every tone his hand
strikes from the piano raises a corresponding sound-image in my spirit, in short, the music nature and experte:l;"
becomes visible in my inner eye. two works by Iohn r..
(Quoted in Bernstein rygB, p.n) (nos. r and 4).
"Imaginary Lanc.:.
This is, as we have seen, just how Wackenroder described Joseph Berglinger's tables, frequeno- re:,-,:
experience ol Iistenin g. Cornish School's r¿--
Although the ideology of absolute music tried to expunge the visual, it continues as a recording or t'r,-- .:
to seep out, emerging in the site and sights of enactment: notation, performance, is fairly convention:,
and reception. iour large sections. ;.
Another way of describing this normative shift would be to regard it as a move increasing progres! ', ;.
from an understanding of the concept of music as a discourse (in Foucault's terms: svmmetrical rhr-thr,-¡
see Foucault ryzz) in the context of non-absolute music, to a formalist understand- 'srveeping the bass s:::
ing best suited to pure, absolute music. The discursive conception of music allows tir-e figures (main1',' ::
issues of meaning to be located more widely, both ideologically and historically. After ib-dflat). The rest oi..,
ali, all composers of music, absolute or otherwise, are bound up in a web of tradition the work is not ultr::l,
and influence, in which they consciously situate themselves. This is largely what much of Cage's mus:--
differentiates music from sound. Cage used test-ton. r:
The composer lohn Cage was part of this history. But he is of particular interest turntables, changir¡,:
here, because he aspired to an art of non-translation or non-mediation. His art These recorded sor-r .;
attempted contact with the other, through techniques that aim to have little effect on ¡roadcast equipnei-: :
musical material beyond the act of reframing. But it is precisely this reframing that lre constant. Cage c..:
;re aiso hard to loca.¡ ,
' This point is related to the notion of unplayable works, as it separates off the notion of the music
from its performance. Schoenberg once renarked ofhis violin concerto,'l am delighted to add
another unplayable work to the repertoire'. ' "Imaginary Lands¡.:.
SIGHTING SOUND: LISTENING WITH EYES OPEN 26]'
:t:;l
bodr', the grimac- allows us to define his activitiesas music; his activities become part of music's identity
;,: stnce Plato. and its history. Cage is a good example because, through the pursuit of his radical
:: ,l¡cried by all sup- aesthetic of 'letting sounds be themselves', his work challenged the view that art music
;,: :bsolute music as a is divorced from worldly associations and its physical environment. In pursuing
:.:: oi \lichael Fried, sounds in themselves, he opened up the concept of music to its physical environment:
,;.'::t rt ledging the site the music of 4'.33" is its physical environment, its sights and site.
r ::r.s sense, surrenders So far, I have discussed the divorce of music from the 'extra-musical', but with
: -i.¡ trther hand, would Cage I want to dwell briefly on the technological issues subsequent upon this divide
.¡. ¡nd the eyes as well and its reunion.
- -t r,, tth the visual and Recent reproduction technologies have aspired to divorce music more literally
¡ ¡nd apart from the
,', from site (and sight), allowing the dislocation of music from its origins in produc-
-.
.r.al.' in other words, tion and performance. Digital technologies have made music's origin and its site of
r*:-¡ .rf both theatrical production potentially virtual. In this history, radio was one of the first devices to
--: .r::erging paradigm dematerialize music, and radio was an abiding interest of John Cage. But even here,
music cannot escape the visual.
.'' :ll1 provoke visions. The radio embodies the notion, outlined earlier, that music represents an abso-
lute, even heavenly realm. The conceit is that this celestial realm can now be 'tuned
into' through technology. As a medium, radio has tended to remain (like the music
----: ::li' mind agreeably,
-----:: -\'en'tone his hand itself) invisible, merely an agent of dissemination, not perceived as affecting the
- -:1:, n short, the music nature and experience of the music itself. I shall conclude this paper by reference to
two works by iohn Cage that employ radios from his series, "Imaginary Landscapes"'
. -.-- i;rnstein ry98,p.n) (nos. r and 4).
"Imaginary Landscape" No. r (1939) was composed for two variable-speed turn-
,: l,rseph Berglinger's tables, frequency recordings, muted piano, and cymbal. It was first performed at the
Cornish School's radio studio. The score also states that the work should be performed
,r. ., rsual, it continues as a recording or broadcast, not as, or in, a concert. It is thus sited as radio. The score
-- i-r:irrn. performance, is fairly conventiona\, rn 614, with four staves for the four players, and is organized in
four large sections, each one lasting 15 bars, with the final phrase of each one
-' :eEard it as a move increasing progressively from one to four bars in length, manifesting a deliberately
, ::.l Foucault's terms:
symmetrical rh¡hmic organization. Pitch is very limited. The piano is played by
:. :malist understand- 'sweeping the bass strings with a gong beater', and more conventionally in repeti-
::.-,n of music allows tive figures (mainly triplets and semi-quavers), within a compass of a diminished 3'd
.::¿ historically. After (b-d flat). The rest of the pitch content is limited to simple prerecorded tones. However,
:n e rreb oftradition the work is not ultimately about auditory relationships, let alone diatonic ones; like
T:is is largely what much of Cage's music it is about the relationships between sounds and their sources.
Cage used test-tone records (Victor frequency and Constant note records) on two
'-,: partrcular interest turntabies, changing speeds between 33 rl3 and 78rpm, and thus varying the pitch.
r-::rediation. His art These recorded sounds are conventionally used by sound engineers to standardize
: - har-e little effect on
broadcast equipment and on-air signals. They work in this way because the sounds
.-, tris reframing that
are constant. Cage destabilizes them by changing speed to affect pitch. As sound, they
are also hard to locate spatially or to assign to a point of origin. This is because they are
::¡ notion of the music
' :eLighted to add
' "ImaginaryLandscape"nos. 1,3, +,SOgZg,794\195\1952;no.zwaswithdrawn).
SIMON SHAW.MILLER
not sonic objects locatable in space, as the piano is (which is separately microphoned,
like the cymbal), but their sound is directly reproduced and emanates from the radio
itself. In this way they are identified totally with the medium. Hence, not only is the
recording studio perceived as a musical instrument, but so, too, is the radio; through
this process it becomes visible in its objecthood.
In r95r, with ' Imaginary Landscape" No. 4, Cage developed further this notion of
the radio as an instrument or sound source in its own right. "Imaginary Landscape"
No. 4 is scored for twelve radios, to be operated, or performed,by z4 people, two for
each radio, one to control the tuner and one to control the volume and tone controls.
The score is, like no. z, relatively conventional in its notation, with detailed indication
of duration, station-tuning, and dynamics, shown as numbers ranging from 3 to r5.
All these performance parameters were determined by chance operations expressed
in charts, with direction to be given by a conductor. Here, what is broadcast at the
time of performance becomes the sonic material of the music, and it might be any-
thing, including the white noise between stations. At the first performance it was
marked by subtle sounds rather than a cacophony, which is what you might expect Fig. 1 3.e r
from rz radios. This is due, in the main, to the low dynamics and the frequent use of
silence. Silence is central to Cage's aesthetic and is a device for emphasizing the visual
in performance. At the premiére, some thought the quietude an accident of the late
hour of the performance, and therefore the paucity of broadcast radio sources, and physical activitv of ::..
a consequent misjudgement of volume levels on Cage's part. But as James Pritchett sight. It is not an ¿;..;
has pointed out, all these characteristics are entirely consistent with other works meaning is embodl:; -
Cage was composing at this time (Pritchett rq8¡). an art of objects, pe::,. :
"Imaginary Landscape" No. 4 not only brings the site of other performances, vla
the radio, into the concert space, recontexualizing them. It is also visually determined
by, and embodied in, the design of radios and the actions of performers. In the case of
the first performance, the radios were probably the gloriously named RCA Victor REpBRnNcEs
Golden Throat radio (Fig. 13.6). Like other musical instruments, the radio is a designed
object, as deliberated over in terms of its looks as its sound. Like musical instruments, Arnheim R (1936,'r9-: .
the radio is therefore a nexus ofsight and sound, oftechnology and history. Barthes R (r982). 1t,,,..'.-.
The imaginary landscapes produced by these works are no less visual than any Bernstein S (1998,,':-'
Heine, Liszt nt¡¡l 5.,: ..,
other enactment of music. The psychologist Rudolf Arnheim was one of the first to
DahlhausC(1989.,::,
recognize this in his celebrated 1936 essay on radio, a medium, he felt, that had the
and London.
possibility to change perceptions. This was to be achieved, in part, through its utiliza- ',
Foucault M (t97z t, :¡ "..
tion of a wide variety of sounds, and in so doing, radio had the potential to suggest Fried M (t967). -\rr ...
images 'more directly, objectively and concretely than on the printed page' (Arnheim lgoo-i99o: Att -\t¡: ....
ry6ltgTr). The fact that radio, as the ideal vehicle for absolute or disembodied music, Goehr L (t992). Tltt i... ,:
should be so suggestive of images is less ironic when we consider the origin of the word Clarendon Press. r- ,..
radio itself. It comes from the Latin redius, a ray; radians, beaming, filled with light; HanslickE(r89r/r95- -'
IN.
irradiatus, to illuminate, to enlighten intellectually. The word is, therefore, indicative
ofthe fluid space that exists, as I said at the outset ofthis paper> on the'outer'historical
Huneker l (rqu). Fr,,..:-,
Leppert R (zoo4l. T--.
and cultural edges of music and art, at the site where they converge. YYZ Books, Toror.,.
Because, as I have argued, music is abstract, the visual is critical to us as produc- \Iiller S, ed. (r99_r , -
ers and consumers in locating and communicating with it. The tension between the Manchester.
SIGHTING SOUND: LISTENING WITH EYES OPEN 263
rrr¡ieh- microphoned,
1:rares from the radio
i¿rce, not only is the
. rs rhe radio; through
l;ss r-isual than any Bernstein S (t998). Virtuosity of the Nineteenth century: Performing Music and Language in
Heine, Liszt and Baudelaire. Stanford University Press, Stanford CA.
i.,'-ls Lrne of the first to
-. r.e t-elt, that had the Dahihaus C (r9S9). The Idea of Absolute Music. The University of Chicago Press, Chicago, IL
and London.
.:.. :hrough its utiliza- Foucault M Q97z). The Archaeology of Knowledge. Routledge, London.
--: ptriential to suggest
Fried M j967). Art and objecthood. In C Harrison and P Wood, eds. Ar¡ in Theory,
:-:red page' (Arnheim igoo-1990: An Anthology of Changíng ldeas, pp. 8zz-34. Oxford University Press, Oxford.
.: rjsembodied music, Goehr L i99z). The Imaginary Museum of Musical Works: An Essay on the Philosophy of Music.
: :re o¡isin of the word Clarendon Press, Oxford.
::rng. hlled with light; Hanslick E O,SgtltgSZ). The Beautful in Music, G Cohen trans. Bobbs-Merrill Co, Indianapolis,
.. theretore, indicative IN.
,:- :he 'outer' historical
Huneker ) (rqrt). Franz Liszt. Scribners, New York.
LePPert R (zoo4). The social discipline of listening. In J Drobnick, ed. Aural Cuhures.
:j. YYZ Books, Toronto.
-:-;¿i to us as produc- Miller S, ed. (lgg¡). The Last Post: Music after Modernism. Manchester University Press,
: tr:rsion between the Manchester.
264 srMoN sHAw-MILLER
Pritchett I (rs8¡). The Music of John Cage. Cambridge University Press, Cambridge.
Reynolds J (iq6r). Discourses on Art.Yale University Press, London.
Schopenhauer A (r8r8ir969). The world as wíIl and representation, EF Payne, trans. Dover
Publications, New York.
Schumann R (134o/1965). Franz Liszt: Concerts in Dresden and Leipzig. In Schumann on music:
a selection frorc the writings, p.r58, H Pleasants, trans. Dover Publications, New York.
Schumann R (rq+6). On Music and Musicians, p.15o, K Wolff ed., P Rosenfield trans.
WW Norton, New York.
Wackenroder WH (rZqq). Phantafien über die Kunst, Für Freunde des Kunst, L Tieck ed.,
Friedrich Ferthes, Hamburg.
TF
sor
IN
IN
D_r
N,lA:
IN this essay, we examine the interaction between visual and auditory information:
in the real world, in the laboratory and in the arts. We review both classic and more
recent studies on audiovisual integration and describe research that is beginning
to uncover how the human brain is 'wired for sound' and, intriguingly, predisposed
for visual-auditory interactions. The final section examines some examples of
contamination between visual and auditory stimuli in the arts. In particular, we
re-examine the building blocks of music in terms of natural and social events which are
t¡pically multisensory.
The overall goal is to break down the artificial barriers between sound and vision,
acknowledging the interplay between these two senses. The theory of 'pure vision' or
'pure sound' turns out to be problematic, even in traditionally unisensory art forms
such as music or painting.
DAVID MELCHER AND MASSIMILIANO ZAMPINI
)NS Another example of the influence of the visual information on auditory perception
is the ventriloquism effect (Howard and Templeton 1966). When we watch a ventrilo-
quist's performance, we have the illusory perception that the voice we heor is coming
from the mouth of the puppet we see moving, because it looks as if the dummy was
speaking. This art of tricking people by making one's voice emanating from elsewhere
ijei provide us with
was used in Greek and Roman times by oracles (Connor zooo). In the early eighteenth
:r' r¡biect or event. For
century, it was suggested that performers manipulated the physical properties of the
i ,-': speech is reliably sound, making it appear to come from their puppets. However, it is not necessary
rS :itrrlS. and gestures),
to attend a ventriloquist's show to experience this audiovisual illusion: we regularly
,-,urside the observer's
experience it in everyday situations. Whenever we watch a movie at the cinema, the
.i':rnation are some-
spoken soundtrack seems to emanate from the lips of the actors seen on the screen,
:r---,ie the movements
despite the fact that the sounds are actually presented from loudspeakers positioned
;nds. The comedic
.¡,
elsewhere. Of course, the ventriloquism effect is not limited to speech, but happens
:::r¿ied by the series
for sound effects or when hearing music coming from the instrument being played
r h:s first film What's
by the musician rather than the loudspeakers.
::: L.
not just a cognitive bias: the illusory doubie flash is perceptually similar to a physical overlap is not necess¿:
of asynchrony. We are
double flash. Importantly, the reverse is not true; the illusorybeeping of a single sound
cannot be induced simply by coupling it with a single flashing light.
auditory and visual ii¡
correlates of speech h:-
One explanation for the relative dominance of audition in temporal events is that
for the as1'nchronv bet-
audition has better temporal acuity than vision. However, Saldaña and Rosenblum
(1993) showed the influence of vision on audition in rating whether a heard note was over, the McGurk i1lusr
to z4o milliseconds' ,-.:
bowed or plucked on a cello. In their study, a single note (the note G; -98H2) was
of when a stimulus o:;
played by bowing or by plucking on the same cello string, and participants could both
tive attention. A sensc,:
see (i.e. on a computer screen) and hear the played note. The audiovisual pairings
to another stimulus .:
could be congruent (in both the note was plucked or bowed) or incongruent (in one
the note was plucked and in the other was bowed). The visual stimulus significantly
ofprior entry propos-.
onstrated that attenci, r
affected the identification of the auditory stimulus, suggesting that sensory dominance
relative perception o: :
in audiovisual interactions depends at least partly on the structure of the stimuli.
modality (Zampini et
between an auditon, .
eity. It is interestine t-- :
Theories of cross-modal integration limited to highlv rela:t:
and non-related sti:r;
These examples of audiovisual conflicts show that vision and audition might show
for audiovisual n-rul :: ¡t
dominance over each other under different conditions. One possible account sug-
is possible to induc: : .
gests that our perception relies more upon the sense that tlpically provides the more
exposing the partic:p;
accurate information for the task at hand (Welch and Warren r98o; Ernst and Banks
(such as speech or ¿ ::.
zooz). Hence, vision has the best spatial acuity, and so dominates audition when
ticular, the tempor;1 ", -
spatial judgements are requested (e.g. stimuli localization). By contrast, when we
of monitoring the as-,-..
are involved in temporal judgements tasks (e.g. rh¡hm evaluation), we trust more
can be found in bo::: ,:
our auditory modality because it has much better temporal resolution than the
visual modality.
However, in many everyday situations, we have to integrate coherent (i.e. not
conflicting) sensory signals reaching our senses at one time. A considerable body of The 'cost' of audi
empirical research has investigated the principles governing multisensory integration
The discussion of :*:,
and cross-modal interactions (see the chapters in Calvert et al. 2004, for recent reviews).
Spatial colocation and temporal coincidence are two of the most important cues that
ering perception rtl-..-
Iose the sense of ', r..
are used by the brain to determine whether or not sensory stimuli should be bound
together. However, the detection of sensory synchrony is no simple matter for the auditory informati¡,:r
(see Róder and Rósl.:
nervous system to solve. Auditory and visual information are propagated through air
at dramatically different rates. Light travels at a much faster rate (3oo ooo ooo metres
able to localize sou::
per second) than sound (3oo metres per second). Therefore, visual information from not rely upon an\ 1rs-
the same event will arrive to our eyes before auditory information reaches our ears. extended to other .,-
In the brain, however, auditory information arrives to the brain from the ear more fact, blind indir-idu¡-.
rapidly (i.e. t_z milliseconds) than visual information (zo-3o milliseconds from the sighted counterpans.
retina to the brain). Thus, the brain has to combine auditory and visual signals from One interesting er;:-
the same location and same temporal event despite these basic differences at the level of 'absolute pitch' b. . ,
of physics and physiology. pitch, is the abilin'rc, :-
It seems that our brain has a relatively wide window for the temporal integration of ence tone. This skill .
multisensory stimuli. Experimental evidence supports the claim that a strict temporal musical training mir*:
THE SIGHT AND SOUND OF MUSIC 269
::'i si¡jlu. to a physical overlap is not necessary, as the perceptual system can accommodate some degree
,.:tring of a single sound of asynchrony. We are relatively insensitive to small differences in the arrival time of
il.:r,. auditory and visual inputs to the brain. For instance, it has been shown that visual
r<r:poral events is that correlates of speech have to precede auditory speech by more than z5o milliseconds
'.-:¿ña and Rosenblum for the asynchrony between the two different sensory signals to be perceived. More-
::in.r a heard note was over, the McGurk illusion can also be perceived when the auditory syllable lags (by up
jr. :rote G; -98H2) was to z4o milliseconds) or leads (by.rp to 6o milliseconds) the visual syllable. Perception
: ::ticipants could both of when a stimulus occurred in time may also depend upon the allocation of selec-
:¡ :udior.isual pairings tive attention. A sensory signal will be perceived as occurring earlier in time (relative
,,: -ncongruent (in one to another stimulus) if we are paying attention toward the former stimulus-the law
:r .t:mulus significantly of prior entry proposed by Titchener (r9o8). Zampini and colleagues recently dem-
:::: sensorydominance onstrated that attending to a sensory modality (audition or vision) can speed up the
: i:- rrf the stimuli. relative perception of stimuli in that modality, compared to a stimulus in another
modality (Zampini et al. zoo5). The allocation of attention influenced the interval
between an auditory and visual stimulus that would allow for perceived simultan-
eity. It is interesting to note that this temporal window of synchrony perception is not
limited to highly related, complex audiovisual stimuli: it also includes relatively simple
: :udition might show and non-related stimuli like a tone and light pair. Moreover, the temporal window
:¡ssible account sug- for audiovisual multisensory integration appears to be quite flexible. For example, it
'
provides the more
:-l---.- is possible to induce a general temporal recalibration between audition and vision by
r r;¡ri Ernst and Banks exposing the participants to an asynchronous stream of complex audiovisual stimuli
:r,rltes audition when (such as speech or a recording ofa hand playrng a piano: Navarra et al. zoo5). In par-
E '' contrast, when we ticular, the temporal window for audiovisual integration can become wider as a result
-::lLrt l. u,e trust more of monitoring the asynchronous speech or music stream. A similar pattern of results
=, :¡solution than the can be found in both speech and music stimuli.
this ability include: Ludwig van Beethoven, Wolfgang Amadeus Mozart, Charlie Iobe, which is the firs,
Parker, Miles Davis, and Ray Charles. Studies on absolute pitch in blind musicians have and localization. Sur:i
shown that the presence of absolute pitch is more frequent among blind than sighted in Ar respond to vrsu
musicians. This is surprising, considering that, on average, musical training with blind zoo3; Brosch et al. : -
musicians with absolute pitch occurs later in time than in age-matched sighted musi- imagingwith humans :
cians. This difference might be explained by the results of studies investigating the input accompan,ving :-
neural correlates of this ability in blind and sighted participants (e.g. Hamilton et al. cortices do not seem :
zoo4). Brain regions that are normally involved in processing visual information in tant questions: (r') r'h:.
sighted people can be 'recruited' by the blind to aid their auditory performance. By promiscuiry to occur?:
contrast, sighted people cannot rely on additional brain areas but are dependent only the senses?
on auditory areas. In other words, ifone sense is absent, another sense can take over In general, there ¿:.
and involve the neurons in those areas associated with the non-functioning sen- a primary senson- a::l
sory modality. This cortical plasticity may help to explain why blind people have mixing could occu: ¡.
increased auditory (but also tactile) abilities when compared to sighted people. cortical area Ar. -{ s.¡
It is tempting to speculate on whether the special sensitivity of the blind towards relevant visual or sr.:::.
music might partially explain the long tradition, across many cultures, of blind musi- uli. Finally, the mu-tr.'
cians (and in Western Europe, blind piano tuners). It is interesting to note the large information from m:,:
number of blind musicians who have played an important role in the development of pass through the ser..,
jazz, country and blues music. Examples include: Ray Charles, Blind Lemon Jefferson, support of all three h'':
Blind Willie Johnson, Rahsaan Roland Kirk, Blind Willie McTell, Ronnie Milsap, to understand rr'hetl-,¿:
George Shearing, Art Tatum, Sonny Terry, Lennie Tristano, Doc Watson, and Stevie non-auditory intirr:r.
Wonder. Indeed,'Blind'was such a common nickname in American blues music that review, see: Brosch ¿: :
the sighted musician Eddie Lang used the moniker 'Blind Willie Dunn' when record- sounds, decoding s:.¿
ing blues records with Lonnie fohnson. While cultural factors have indeed played Thus, these difteren. :
a role, including limits on career choices that were available to the blind, it might of audiovisual inteEr :: :
also be the case that the absence of audiovisual integration in those with visual depriva- In addition to thr
tion has provided extra resources for musical ability. In sum, audiovisual integration stimuli in priman. .:.,
has a cost for sighted musicians, in addition to a benefit: it may slow down (and per- which are invoh'ei :-
haps, make less sensitive) our auditory perception of musical pitch. cept. An area in th¡ s*:
appears to plal-an 1::::
order to understanc , -
biological motion s;,,
serves as a multisen., :
AuorovrsuAl NEURoNS: How rHE BRArN sensory integratio n ¿r;
As reviewed ear-:.:
PROCESSES MEANINGFUL SOUNDS fit together based '::
visual associations. s -::
:;eus \Iozart, Charlie lobe, which is the first stage in the cortex which processes sounds for identification
-:: b-rnd musicians have and localization. Surprisingly, a number of studies have demonstrated that neurons
:.-,nE blind than sighted in Ar respond to visual and somatosensory stimuli as well (werner-Reiss et al.
'r::., trarning with blind zoo3; Brosch et al. zoo5; Ghazanfar et al. zoo5). Functional magnetic resonance
:r:.:¡hed sighted musi- imaging with humans has shown that the responses in Ar are modulated by the visual
:i:es investigating the input accompanying an auditory signal (Calvert et aL.ry9). Thus, the primary sensory
r: ..q. Hamilton et al. cortices do not seem to be as unisensory as we once thought. This raises two impor-
: ','lsual information in tant questions: (r) what are the anatomical connections that allow this early sensorial
i,:,:,r,' performance. By promiscuity to occur?; (z) how does our anatomy constrain the interactions between
r r: ¿¡e dependent only the senses?
,1-r :ense can take over In general, there are three main ways (which are not mutually exclusive) in which
:0r-functioning sen- a primary sensory area like Ar could receive inputs from the other senses. First, the
, r', blind people have mi-xing could occur early at a more basic, sub-cortical' level, prior to the arrival in
,.:;htedpeople. cortical area Ar. A second idea is that neurons in area Ar might directly integrate
:. i-,: the blind towards relevant visual or somatosensory information into their processing of auditory stim-
:;,iLres, of blind musi- uli. Finally, the multisensoriality of Ar might result from a feedback loop, in which
;):ing to note the large information from multisensory integration regions is passed back down (in a second
: -r ite development of pass through the sensory system) to modulate Ar activity. So far, there is evidence in
ts-rnd Lemon Iefferson, support of all three hypotheses, at least in animals. However, further study is required
l:T¿ii, Ronnie Milsap, to understand whether these three pathways, which differ in the latency at which the
:: \\-atson, and Stevie non-auditory information would reach Ar, carry different t)?es of information (for
-:L¡an blues music that review, see: Brosch et al. zoo5). Audition is used for a variety of purposes: localizing
:: )unn'r,vhen record- sounds, decoding speech, detecting a danger, hearing our own footsteps and so on.
r. have indeed played Thus, these different mechanisms might, in theory, be involved in different aspects
:,:, the blind, it might of audiovisual integration.
,, .: rt-ith visual depriva- in
addition to the multisensory influences on the initial cortical processing of
:'dror-isual integration stimuli in primary sensory cortices, there are alarge number of multisensory areas
:1- )-Lrw down (and per- which are involved in the integration of multiple sensory inputs into a complex per-
: a:1. cept. An area in the superior temporal lobe known as STS (superior temporal sulcus)
appears to play an important role in integrating auditory and visual information in
order to understand some physical events. In particular, STS is reliably activated by
biological motion such as walking or face movements, suggesting that this region
serves as a multisensory social event processor rather than a general-purpose multi-
.H
E B RAIN sensory integration area.
As reviewed earlier, there is evidence that certain visual and auditory stimuli
)UNDS fit together based upon shared spatial and temporal information. Other audio-
visual associations, such as in synaesthesia (see van Campen, Chapter 25, and Ward,
ssing?
' Both audition and vision involve not only cortical pathways, but also subcorcical pathways. Thus,
i> not surprising that patients without a visual cortex may sti11 have 'blind sight' based on information routed through the
: srEht-sound interac- superior colliculus, and damage to auditory cortex does not eliminate the startle response to a
sudden loud noise. Neurons in the superior colliculus respond to auditory, somatosensory, and
:rss sensoryinputs have
visual stimuli. In addition, there is evidence for direct links between the thalamus and areas of the
:i the brain's temporal limbic system.
DAVID MELCHER AND MASSIMILIANO ZAMPINI
Chapter 24, this volume) or more 'metaphorical' similarities, are less straightforward
to explain. One example is the mapping between visual shapes and the sound of cer-
tain words first demonstrated by Wolfgang Kóhler (tgzgltg+l). Kóhler showed that
people presented with a curvaceous or a pointy shape and a pair of words ('takete'
and 'maluma') consistently paired maluma with the curved shape and takete with
the pointy one. Even young children (2.5 years of age) show this tendency (Maurer
et al. zoo6) and it does not depend on exposure to the English language (Davis rg6r).
One speculation is that the shape representation is generalized between mouth
movements (open versus narrow) and visual shape, but further research is required
to test this hlpothesis (Ramachandran and Hubbard zoor). Many artists, such as
Wassily Kandinsky, have argued for a correspondence between particular visual
features-such as colour and shape-and sounds (Fig' ta.t).
Although these links can be quite personal for specific artists (see Ward, Chapler 24,
this volume), the link between visual and auditory features is made explicit in the
performances of the artist Mark Rowan-Hull. Figure r4.z shows a painting by the artist
produced during a performance of Stravinsky's 'Concertino' performed live by the
Coull Quartet.
AuorroR
that relay a preoccupation with the process involved in composition, both visual finely tuned.
and musical. Rowan-Hull's concerns are not so much about the final product but Pitch is the perce:." :
the conceptual and performed efforts entailed in pushing painting and image exists only in our ir-.,
making out of its frame. Rowan-Hull's installations and performances reiterate his rrithin a musical s;r-.
approach towards this transitory and ephemeral process of making art and music. a particular pitch.s. -:
ons fire in areas of.*
sound wave itseli. C :
visual syrnbols with the corresponding sounds, based on experience and training' It is the contour e., :
The finding that PT is involved in reading words and also musical scores suggests the birthday song ,i,-,:
that this brain region is involved when there is a communicative goal (e.g. music and In both speech anci r-'
speech). Evidence for the importance of PT in meaningful audiovisual interactions see Krumhansl ani .
comes from a recent study by Hasegawa and colleagues (zoo4). In their study, piano erample, when ask:::
players of varying levels of training (untrained naive participants, relative beginners, or context of music. a :::'
players with at least eight years of training) were shown a video showing hands playing Timbre is the har;-
music pieces on a piano in the absence of any piano-related sounds. Participants had ,,'oice singing micid-.
THE SIGHT AND SOUND OF MUSIC
to identify the piano pieces by watching the key-touching movements of the hands.
Well-trained pianists were better in recognizing familiar pieces than the remaining
two groups of participants. Although unfamiliar music and random sequences did
not show the same pattern of results, all the conditions (i.e. familiar music, unfamiliar
music, and random sequences) showed the same activation of the PT in well-trained
piano players. Naive and less-trained piano players did not show such activation.
Hasegawa and colleagues suggested that PT is engaged in transforming the visual
stimulus of hand movements into a sequence of sounds, at a stage occurring prior to
identifi cation of music.
:¡:¿: is influenced by
. rnplitude sounds-
:si ii-rat such a sudden
:i-erberation describes
J.¡-ar-s' as each sound
.. :;ho to estimate the
ijirerent'feeling' for
used high and low pitched sirens and a 'lion's roar' membranophone (a friction shattering of glass. l:
drum) in his composition 'Ionization' (r93r). Moreover, composers and musicians the mqterials invoir-ec
have explored the role of chance, in which meaningful patterns of sound (for expert information in the ',':.
listeners) emerge out of unscripted interactions between sounds (see Iyer zoo6). other auditory cues l::
Such exercises might be similar to Leonardo's recommendation to painters to seek of a sound-which de:
inspiration from random patterns in nature, as well as to the techniques used by sci- larly important fo¡ t:,
entists to uncover basic features in visual perception by looking at patterns of human apart the same ton3 ;.
responses to noise ('reverse correlation', see for example: Iones and Palmer 1987; learn to tell apart o:-:
Neri et al. iggg). In sum, looking at how the mind seeks regularities in noise, whether discriminate, base¿ F.
in the laboratory or the concert hall, provides insight into the musical brain. another. Timbre is p¡:.
particular instrumen:.
Our sensitivitr- t.. :
physical objects hap:
Music and the perception of social and emotional events ":
of the series of or er --:
Music has been famously defined by Edgard Varése as 'organized sound', reflecting perfect fourth. It is :::,
the modernist fixation on 'pure sound' or 'pure vision'(see Shaw-Miller, Chapter r3. harmonic and meic'i-.
this volume). However, music's divorce from accompanying visual information is related to the overtr i
historically recent. Before the advent ofpopular recording techniques, developed in Although it is cert¿::r--.
t877 with Thomas Edison's phonograph, music was never physically removed from ancient Greeks des.:,:
the event that produced it. In the real world, patterns of sound have meanings in Recent studies of ch-:
terms of physical and social events, with many auditory stimuli related to other sen- vals (for reviels, se:: .
sory stimuli involved in the same event. As described earlier, our interpretation of et aI. zooz). While ,,' ¿ -
sounds occurs within a multisensory context in which visual information plays an vals in Western mu.:--
important role. understand. Indeec. ,.
Theorists often distinguish between the sound ('sensation') and the recognition 'feeling' (describec i
('reference') of a particular object or event (see, for example, G. Frege, On Sense and intervals (Krantz e: :-
Reference, úgzltgsz). As Heidegger (ryzzltggí) noted, it is difficult to perceive pure processing occurs
': :
noise. Instead, we hear a motorcycle coming down the street or a door slamming erences of other an-::
shut. Musical sounds, therefore, have both a subjective experiential quality and a McDermott zoo_¡ .
property of recognition that calls to mind the source of the sound. The experiential When combined ',,;-
aspect of sound is certainly important to music (see Noé zooo), and music, like other information about ::-
forms of art, can make us more aware of our own perceptual processes (see Muniz are critical for und.:,
and Melcher, Chapter r8, this volume). However, it is useful to consider auditory a special use of cLr:r: '
stimulation within the context of the primary mission of sensation, which is to yield of proto-music itb: :.
information about the sources of events. Indeed, such an analysis might go some way used to guide the iri...
in helping to explain our emotional and embodied reactions to music, as well as fussy baby or helpir= -
to guide our understanding of how visual stimuli can interact with certain aspects provides a stronq.';; :,
of music. a question and a .¡-
The first two separable components of music, pitch and timbre, tell us something to emotional erpres.r
about the nature of the object making the sound. When combined, these two feat- say it ('prosod1" ) tha: .
ures can reveal, for example, whether the speaker is a male or female or even a par- interact in commur:;.
ticular person that you know. In the case of physical events, these features of sound ation from both sou:;
inform us whether a solid object has banged against a wall or, instead, if there was a zoo6). Thus, music ;::
THE SIGHT AND SOUND OF MUSIC 279
:¿nophone (a friction shattering of glass. In a basic sense, timbre reflects a perceptual hlpothesis about
rru-¡Sers and musicians the materials involved in a particular auditory event, similar to texture and colour
rs ,rf sound (for expert information in the visual modality. Of course, timbre is tlpically combined with
::.is (see Iyer zoo6). other auditory cues in refining our perception. In particular, the attack and sustain
.-.r ro painters to seek of a sound-which depend on the time course of loudness and timbre-are particu-
::h:iques used by sci- Iarly important for the identification of sound sources. While most of us can tell
. ¿t patterns of human apart the same tone played by different instruments, with enough practice experts
:;s and Palmer 1987; learn to tell apart one make or model of instrument from another and can even
:rr:es in noise, whether discriminate, based purely on timbre, the sound of one musician playing from
::'-cal brain. another. Timbre is part of a musician's'voice' and forms the auditory fingerprint of a
particular instrument.
Our sensitivity to pitch and timbre reflect 'natural harmonics': the way that the
)nal events physical objects happen to vibrate on our planet. For example, the first three intervals
of the series of overtones are the 'perfect' intervals: the octave, the perfect fifth, and
-:eJ sound', reflecting perfect fourth. It is interesting to note how the development of Western music, with
.',";-\f lller, Chapter r3. harmonic and melodic systems based on the octave and the perfect fifth, is directly
'.:.ual information is related to the overtones series and, thus, reflects a physical property of the world.
lri,ques, developed in Although it is certainly not the onlyway to construct music, it is far from arbitrary (the
,'.-.;1ll removed from ancient Greeks described intervals as'perfect'because theywere created by nature).
::; har-e meanings in Recent studies of child development show an innate sensitivity to these classic inter-
.- :el¿ted to other sen- vals (for reviews, see: Trehub zoo3, zoo6) and a preference for consonance (Trainor
:r *r interpretation of ef aI. zooz). While we can be trained to be sensitive to other sound patterns, the inter-
:rl¡rmation plays an vals in Western music may take advantage of what is easiest for our musical brains to
understand. Indeed, there is a general agreement, among listeners, about the general
::d the recognition 'feeling' (described by words or by body movements) communicated by particular
r ^ :ege, On Sense and intervals (Krantz et a7. zoo4, zoo6). Such differences may reflect the way that tonal
t-:'-ili to perceive
pure processing occurs in the brain, based on evidence from neural responses and pref-
,,,¡ a door slamming erences of other animals exposed to musical sounds (for review' see Hauser and
:l.ntial quality and a McDermott zoo3).
*ra. The experiential When combined with pitch and timbre, the musical contour provides important
r:id music, like other information about physical and social events. In particular, these aspects of sound
rit¡cesS€S (see Muniz are critical for understanding the social meaning of a stimulus. Caregivers make
--- consider auditory a special use of contour, known as 'motherese' when speaking to infants, in a type
:-,-,i. rr-hich is to yield of proto-music (for review, see: Trevarthen zooS). In particular, this proto-music is
,s :right go some way used to guide the infant's attention or to change emotional state, such as by calming a
rnusic, as well as
:\--- fussy baby or helping her to fall asleep. Combined with timbre, the contour of pitches
r ,,,-ith certain aspects provides a strong cue to communicative intent-differentiating, for example, between
a question and a command. The contour and timbre provide an important clue
r::. tell us something to emotional expression. In other words, it is not jusf what you say, but how you
r-red, these two feat- say it ('prosody') that caries meaning. Verbal prosody and dynamic facial expressions
::nale or even a par- interact in communicating emotions, and our perceptions tend to combine inform-
-¡; iiatures of sound ation from both sources (see for example: Walker and Grolnick rg8:; Ethofer et al.
::.tead, if there was a zoo6). Thus, music can use the same clues to emotion that are involved in emotional
DAVID MEI,CHER AND MASSIMILIANO ZAMPINI
speech, without speech getting in the way (for a review of music and emotion, see: increases in size and :
Juslin and Sloboda zoor). As the composer Felix Mendelssohn argued 'It's not that auditory and visual ...
music is too imprecise for words, but too precise . . . ' (cited in Kruse zooT). and Ichihara zoor r. T-
Music, iike prosody in speech, may tap into neural mechanisms that help us to are important in phr':-,
attend to the important aspects of sound and to keep that information active in work- voice is an importan: .
ing memory (Patel et al., 1998). Interestingly, the mnemonic power of melodic con- the extreme, loud s.. *-
tour can even be used to help treat aphasic stroke victims who have lost the ability to the music of some :c,c.
speak. 'Melodic Intonation Therapy'pairs useful phrases (such as the patient's name some musicians and -:.
or 'example') with song-iike patterns (combining both meiody and rh¡hm) and has Reverberation pr.', .
been successful in bringing back speech, perhaps by re-activating language areas via the produced. The minc -.
route of musicality (Belin et al. 1996). Many music therapy approaches are based on ties of the environnt¿ :.
the idea that musicality is a deep, fundamental and prelinguistic capacity that supports the materials of th¡ ',,,
communication and the development (or rehabilitation) of language. manipulate the rer.:.-
Contour also provides an important cue about physical events: a pattern of rising like a stomp bor. sr.:,
notes provides a different interpretation from a 'bouncing' or 'falling' contour of sound engineers a:,:..
notes. Nlumerous studies have shown that music that 'goes up' in pitch is understood hall or a padded rc;::.
in spatial terms (see Ward, Chapter 24, and van Campen, Chapter 25, this volume on duce natural reverb¡:.
syraesthesia). Independent contours can be segregated in polyphonic music, allowing beration that is n¡', :
different 'objects' to interact. This allows the different melodic streams to be segre- In practice, rer-ert.:.:
gated (either by instrument or across instruments by following the melody). Examples of auditory space ei., ,-.
include the 'call and response' in gospel and jazz music or the way that the soloists in claustrophobic) sp:.:
aMozartpiano concerto seem to 'converse', or at least react and interact, with the rest Loudness and :¡'.::
ofthe orchestra. who wish to mor-e :i. ,
u¡ic and emotion, see: increases in size and/or moving closer to the observer, and, in fact, our perception of
r ¿:gued 'It's not that auditory and visual cues about size and distance interact with each other (Kitagawa
(r"rse:oo7). and Ichihara zoot). Thus, it is not just loudness per se,but also changes in volume that
ilsms that help us to are important in physical event and social perception. In many cultures, raising one's
::1¿lion active in work- voice is an important cue in communication. Loudness is generally arousing but, at
t.rirrr of melodic con- the extreme, loud sounds can physically hurt us. Indeed, loudness for its own sake in
:-:r'e lost the ability to the music of some rock bands has lead to permanent damage to the hearing system in
-- ¿s the patient's name some musicians and listeners.
, ¡nd rhr.thm) and has Reverberation provides a vital clue about the environment in which the sound is
i i;rguage areas via the produced. The mind interprets the pattern of echoes in terms of the physical proper-
;:rreches are based on ties of the environment. Reverberation can give us cues about the size of the space and
.c:pacin' that supports the materials of the walls (such as carpet versus wooden floors). Many musicians
,-
_
.l L-¡ manipulate the reverberation of their instrument on stage using various devices
:r..,-u purr...r of rising like a stomp box, signal processor, or reverberation knob on the amplifier. Modern
i: 'talling' contour of sound engineers artificially manipulate sounds to recreate an enormous concert
:: r'itch is understood hall or a padded room for listeners who use earphones (that of course' cannot pro-
r::: ri, this volume on duce natural reverberation themselves). In fact, engineers can now produce rever-
rh,¡nic music, allowing beration that is not actually consistent with any physically possible environment.
:. streams to be segre- In practice, reverberation and loudness work together in creating the perception
:::: melody). Examples of auditoryspace giving subjective reports of 'open'space or more closed'intimate' (or
i,'¡r that the soloists in claustrophobic) space.
i :nieract, with the rest Loudness and reverb, when combined, provide a powerful tool for musicians
who wish to move the sound in audiovisual space. Examples include the'far off sound
::le iointperformance of a trumpet call in a symphony or the sound of instruments separated by stereo
:-tu:es. The importance sound (particularly with headphones). One dramatic example, in club music, is the
: ,"ladison, Chapter r7, perception of the 'approach' of a sound, which dramatically builds tension in the
,rden, Chapter t5, this anticipation of its arrival. Then, when the sound finally arrives, this moment of
,:, ra1 synchronization. climax is emphasized by the entrance of a deep, pounding bass beat (Clark zoor).
:iün are just beginning This provides another example of how music, often considered 'pure sound' or
.:ns show greater syn- 'abstract', taps into our mind's search for meaningful patterns which reflect social
-.:oo9). and physical events.
Embodied music
1'¡'¡¡f e¡¿1ien-provide The last two components-rh¡hm and tempo-inform us about the precise timing
:rt. Loudness depends of events. As described earlier, sound provides a precise source of information
:rsitr', frequency, and about collisions between objects that is more precise than the visual correlates of
:i:lÍI the event (Zahorik the same events. In the case of music, however, it is important to note that percus-
riilect either a larger sive sounds have been, up until relatively recently, human-made. More generally,
:.¡ our distance to that patterns in music happen over different timescales, which can be compared to the
rh¡hms and tempo of the body (Iyer zoo$. Over a timescale of seconds, there are
.¿tion with'collision', dynamic swells that correlate with bodily activities such as 'breathing, arm gestures
r;reased loudness are or body sway' (Iyer 2oo4, p.:q6). The musical correlates of these dynamic swells are
282 DAVID MELCHER AND MASSIMILIANO ZAMPINI
'phrase, meter, harmonic rh¡hm and vocal utterances'. In a timespan of a third of a expeflence ln e\¡en ú:-
second to one second (6o-18o beats per minute), we find'heartbeat, walking and run- problem of synchror:--
ning, sexual intercourse, head bobbing and toe tapping', which have music correlates
'groove' app€drs tcr t;
in 'pulse, walking baselines and dance rh¡hms' (Iyer 2oo4p.396). At the shortest tim- music in both \\'est¡:,-
escale, hundreds of milliseconds (¡-ro notes per second) corresponding to the musical Specifically, groor e :
features of 'fast rh¡hmic activity and bebop melodies', the motor actions include synchronybetween :t *
speech articulation and rapid body movements such as finger motion. These links standing of the curr.:--
between musical timing and human timing become clear when listening to musicians with groove (for er.r.'--
performing: for example, in the phrase lengths of wind instruments (which are con- rh¡hm across dift¡:=-
strained fundamentally by breathing in most cases), even artificial percussion calls to musicians appareni : -
mind the body rh¡hms that it hopes to provoke by compelling the listener to dance haps through their .¡',',:
or at least to 'feel' the beat. Thus, Shove and Repp describe music as 'audible human musicians perfo rnt i:-- :
motion' due to this fundamental link between the timing and range of sounds and the formers and the au¡,.
body movements that (for most of human history) generate those sounds (lSSi). Small's terminoioE:'
Human respon:a
-:
The subjective impression of the 'beats' in music has been studied by scientists using
(for an extensir.e tr;::::.
a variety of paradigms, including the seemingly simple task of tapping along to music
(for review, see Repp zoo5) One key finding is that our perception of the timing of sound and motir-¿':: - :-
music (the combination of tempo and rh¡hm) involves neural regions involved in tion) may plav a p;.: , ,:
generating movements. Overt manifestations of this embodied perception of 'the rewarding in the ¡..:,
beat', such as tapping one's fingers, reflect a covert motor response going on in our condition a resPons. r
brains when perceiving rh¡hm and tempo. Temporal entrainment to auditory beats ies suggest that the::
is a well-known phenomenon (whether in military marching or in a tribal dance). experience-of oli:;:
Interestingly, entrainment to tempo (the phenomenon in which people spontaneously of the odorant mrr,.; *
nod their head or tap their toes to music) has also been shown with vocal learning birds of the odorant lKl:.¡:-, :
(Schachner et al. zoo8; Patel et al. zoog). Parrots, which imitate speech and other sounds tifu it, and the sanri r'--
are, so far, unique examples in the animal kingdom of entrainment to auditory beats The influence ..: :
(entertaining videos of the cockatoo 'snowball', which have been viewed by millions product ofher or h.. .
of people, can be found on YouTube and at researcher Adena Schachner's website: patterns of sounrfs. '- ¡
http://www.wjh.harvard.edu/-amschach/videos.html). The world's first bird dance ponents of the sm¡i- .-:
contest was recently held at BirdChannel.com. At the same time, there is undoubtedly ofthe ear itselfch¿r=.
an influence of personal experience on synchronization to musical tempo. Drake and itivity, or even da¡:r.:,
Ben El Heni (zoo3), for example, compared finger tapping in Tunisian and French lis- processing based '.:F - :
teners and found that temporal synchronization occurred over a wider range of tempos sense, learn to he¿: :-,
and at a higher hierarchical level (rh¡hmic complexity) when participants listened to Thus, the comnr'*:'.
music from their own culture. Thus, it seems that acculturation and expertise strongly tern of sound-¿< ',,, ¡-
influence the function of embodied mechanisms for perception of rh¡hm and tempo can manipulate the ,,- .
(see Iyert998, zooz, zoo+b). history of the listene :
Taken together, all of the evidence from physics, physiology, and psychological of sounds might i:,::
research demonstrates that, despite the fact that the various sensory cues emanating dance club or con..:.
from an object may actually occur at the same time, it is quite unlikely that the auditory listeners would tenc : -
and visual information reach our brain exactly at the same time. Hence, our brains in a cathedral, to rl- .
must resynchronize, or adapt to these various sensory asynchronies in order for us to can only suggest a :.'-:
interact adaptivelywith our environments, and deliver the rich multimodal percepts we interpretation oftn¿:
THE SIGHT AND SOUND OF MUSIC 283
t-nespan of a third of a experience in In other words, our brain must continually solve the
everyday life.
"rL.ear.rvalking and run- problem of synchronicity (Madison 2oo3, 2006). The attribute of music known as
:: h¿r-e music correlates 'groove' appears to be particularly important for collaborative, improvisational
i: . ,\t the shortest tim- music in both Western and non-Western music (Iyer 1998, zooz; McGuiness zoo5).
i:..nding to the musical Specifically, groove plays a role in improvisation, which involves an embodied
:r'¡ior actions include synchronybetween musicians, such as in rock or jazz music, based on a shared under-
i: xotion. These links standing of the current state of the music and its future possibilities. Often, music
: Lstening to musicians with groove (for example the music of James Brown) explicitly shifts the pattern of
--r:enis [vhich are con- rh¡hm across different instruments, making the temporal synchronization of the
l;:o. percussion calls to musicians apparent to the listener as well. Thus, the listener might experience, per-
:s :re listener to dance haps through their own sensorimotor system, the entrainment occurring among the
*.-c as 'audibie human musicians performing. In the case of a live concert, this relationship between the per-
¿rse oi sounds and the formers and the audience becomes more interactive and develops-in Christopher
-.. st-'unds (ryg). Small's terminology-into the shared activity of 'musicking' GgBl).
;-..i br- scientists using Human response to sounds, and particularly to music, is a complex phenomenon
:::pinq along to music (for an extensive treatment ofthe topic, see: Juslin and Sloboda zoor). The linkbetween
:¡i¡n of the timing of sound and motivation (rewarding behaviours that lead to survival and reproduc-
::- regions involved in tion) may play a part in its emotional power. Pleasant visual stimuli, for example, are
itJ perception of 'the rewarding in the classical sense of the word, since faces can be used to classically
:,-nse eoing on in our condition a response (Bray and O' Doherty zooT). In the case of olfaction, recent stud-
:i:rt ro auditory beats ies suggest that there is an innate component-which can, of course, be shaped by
,r 1n a tribal dance). experience-of olfactory pleasantness, which is directly related to the discriminability
::rrDl€ spontaneously of the odorant molecules at a physiochemical level: the molecular weight and extent
.::r ., ocal learning birds oftheodorant(Khan efal.zooT).Itiseasiertosaywhetherwelikeasmellthantoiden-
,¡e;h and other sounds
ti{y it, and the same might be true of music.
:.;tt to auditory beats The influence of a certain pattern of sounds on a particular individual will be a
:n '' ierr.ed by millions product of her or his entire past. Musical training influences the sensitivity to certain
i Schachner's website: patterns of sounds, just as an expert wine taster learns to discriminate different com-
:,r.i's first bird dance ponents ofthe smell and taste ofwine and to give these a name. Moreover, the response
:. rh.re is undoubtedly of the ear itself changes over time, due to development, aging-related changes in sens-
::;.1 tempo. Drake and itivity, or even damage caused by exposure to loud events. Thus, our brain retunes its
"risian and French lis- processing based upon the statistical regularities of our auditoryworld. We do, in a real
range of tempos sense, learn to hear differently.
",,-:Cer
:r.i cipants listened to Thus, the communication of ideas and feelings involves the stimulus itself-the pat-
.:d erpertise strongly tern of sound-as well as the context and the individual's personal experience. Artists
-.i¡hrthm and tempo can manipulate the sounds and the context, and try to address the personal and cultural
history of the listener. In the context of a religious ceremony, for example, a certain set
r'. and psychological of sounds might inspire spiritual feelings, while quite similar patterns of sounds in a
i-Stl11- CLIeS emanating dance club or concert hall would be interpreted in non-religious ways. Today, many
L"lelr that the auditory listeners would tend to attribute the beauty of f .S. Bach's religious music, even if played
::. I{ence, our brains in a cathedral, to the genius of the artist rather than to the power of God. Thus, music
:ies in order for us to can only suggest a particular idea or feeling. Context and appraisal play a role in the
.':imodal percepts we
interpretation of the meaning of what is suggested by the musical sounds themselves.
284 DAYID MELCHER AND MASSIMILIANO ZAMPINI
heights as 'pure sound' to its origins as part of a multisensory, embodied and situated known visual artls:¡ .:
-,
activity (see Iyer zooz, zoo4b). Mati Klarwien. aii:
David Bltne's Playing the Buildinginstailation at the Battery Maritime Building in been numeroLls reaa:-:
New York (zoo8) provides an example of how visual information can heighten the album covers.
interpretation of each sound in terms of a physical event. Byrne attached sound-
making devices to the building structure itself, to create three tlpes of sounds: wind,
vibration, and striking. Playing a keyboard caused parts of the building (beams and
pillars, heating and water pipes) to resonate, vibrate, or oscillate, thus producing a
Exit music (for a
to inanimate obiect. .
Visualizing sound artist's projected l:..
quist effect gives ar:,. '
One reason for the power of visual images in history is the ability to create relatively There are at leasr ::.
long-lasting representations through sculpture, etching, drawing, painting and other ing a film or video. :,
techniques. Certainly, one can tell an art history based on visual representation of event in the film ::
visual phenomena, but there is an equally interesting story of the use of visual repres- events can be imp:.;
entation of the other senses (see Melcher and Cavanagh, Chapter r9, this volume). audiovisual additr,.,r
If auditory and visual information tend to interact in particular ways, then one might cue which helps us :
expect certain aspects of auditory events to be translated into visual cues. One example erample, plaful n',:sr
is the way that sound waves radiate outwards from the source (like ripples on the protagonist, lvhile,jr.
THE SIGHT AND SOUND OF MUSIC
unfortunate character. The 'Donube waltz'played as spaceships twirl through space to grant the request..i
emphasizes the dance-like movement ofthe spaceship in Stanley Kubrick's zoor (rg68). adaptation of The L,,,
Finally, music is a powerful tool to increase the general arousal, such as by creating score created tensio:r
tension which is, itself, ambiguous. Even a simple 'drum roll' could, in a film, herald BBFC felt that the u:',
the arrival of a happy event or a scary monster. In the film zoot, for example, the use for very young childr.
of 'Thus Spake Zarathustra' effectively builds tension to inform us of the monumental violence: 'The BBFC ,
nature of the events unfolding on screen. The ostinato was used effectively in Psycho didn't just see the i,¡i
(196o) and Jaws (ry2) to heighten tension, showing the strength of fear conditioning. only after makins th-
It is interesting to note that studies with animals suggest that fear conditioning to a intact-that the fi1n: ','
sound can bypass the auditory cortex and make a direct, subconscious link between While cinema ani
stimulus and fear response, which is later modified by the slower, cortical input. sual forms of art. ¿:r,,
One powerful illustration of the power of both the musical score and the sound both seen and hea¡c i
effects in film perception is the refusal by the British Board of Film Classification allow for a trulr' :r:
designed to hide t:¿ r
such as Fortunatrr I
art objects, essenir'.
artistic performan.cl
Figure 14.4 shoi','s .
bination of sounds ::-.
series of machines. -'..:
industrial sounds ¿, :
L,\ i}er, -€' *'
, ¡S4f,4 The sound sculp:-.:
c#;****
nature of audior isu¿, :
material prope¡ties ::
a breeze or human : -
the multisenson' -n'.':
metal rods or scul:: --:
initiates the souni. :',
reverberation of t:. .
audiovisual percepi,.
the integration oi¡*:
and physical evenis.
i,r,g
lr?
:,i
.:i*
In sum, interacti¡:-r.
and understand p:', .
arewired into our'¡::
Fig. 14.4 Fortunato Depero, sketch for a Plastic-Motor-Noise Apparatus with
-specialized multise, ;
Coloured Lights and Spray Mechanism, "t915. MART Museum, Rovereto. areas. Our perceptr
THE SIGHT AND SOUND OF MUSIC 287
:s ¡,rirl through space to grant the requested izA (for children 12 years or older) rating for lhe zooT movie
, (ubrick's zo ot (t968). adaptation of The Da Vinci Code.The film board complained that the film's musical
:1. -.rch as by creating score created tension and anxiety: 'Everyone was full of praise for the score but the
: - ;,d. in a film, herald BBFC felt that the way it was being used to build up the tension was simply too much
:. -.,¡ example, the use for very young children.' Also, the sound effects enhanced the visceral nature of the
r: {tithe monumental violence: 'The BBFC also thought that the film had a very high 'crunch factor'. You
:: rtlecti\.ely ín Psycho didn't just see the fight scenes, you heard the bones break' (Hastings zoo6). It was
--,-,: :ear conditioning. only after making the changes to the audio content-while leaving the violent images
-¡:¡ conditioning to a intact-that the film was granted the desired rzA rating.
. :scious link between While cinema and video are obvious exemplars of truly multimodal, audiovi-
: ;,-rrtrcal input. sual forms of art, another case in point comes from objects that are designed to be
, .:,¡ie and the sound both seen and heard. In this situation, the materials and mechanics of the object may
- ¡r1m Classification allow for a truly multisensory experience (or, alternatively, the object may be
designed to hide the relationship between visual and auditory event). Futurist artists
such as Fortunato Depero (Fig. t+.+) and Luigi Russolo designed audiovisual
art objects, essentially machines, which were used for creating multisensory
artistic performances.
Figure r4.4 shows a design sketch by Depero for a machine that would make a com-
*¡'. -1'r bination of sounds and colours (coloured lights, smoke, and water). Russolo built a
r\{ 81-l (" series of machines, which he called'intonarumori' (noise players), to sample different
industrial sounds as part of a performance.
,19t5-r978),
The sound sculptures of Harry Bertoia created in the l97os, make the
nature ofaudiovisual interactions transparent, as the pattern ofsound reflects both the
material properties ofthe objects and also the event that acts upon that object, such as
abreeze or human touch (Fig. r4.5). Bertoia used the term 'sonambient' to describe
the multisensory environmental effects of his sculptures, which included patterns of
metal rods or sculptures shaped as gongs that were meant to be struck. The event that
initiates the sound, by displacing in space one or more rods, as well as the continued
reverberation of the rods, is a true audiovisual stimulus, bringing this discussion of
audiovisual perception and the arts full circle, back to the main theme of this review:
l the integration of audio and visual information is critical for the perception of social
gtcd- .-1 and physical events.
?uu.r "t¡* e r*
é d--¡
CoNcrus r oNS
' i',]-..1 In sum, interactions between audition and vision increase our ability to localize
and understand physical and social events. The neural substrates of these interactions
are wired into our brains, but also, importantly, are modified by experience. There are
Apparatus with specialized multisensory regions as well as direct connections between unisensory
F ove reto. areas. Our perceptions reflect a fundamentally muitisensory understanding of the
DAVID MELCHER AND MASSIMILIANO ZAMPINI
REpERnNcrs
Oxford.
Davis R (196r). The , . .=
of Psychc: ':
Journal
Drake C and Ben E- ;-.:
oftheNYAc¡tl¿".. '
Ernst MO and Bank. l. -:
cally optimal f1sh.,. :-.
Ethofer T, Pourtois -.
emotional signals --. '
Fig. 1a.5 Harry Bertoia, Untitled (Sonambient), 1970s. Sound sculpture, Frege G. (r892/r95: ,-'-
beryllium-copper rods with brass top and base, Sculpture Garden of the from the Philo sop ;'.: :., .
world in terms of what and where (physical events) and in terms of communication
(social events): the audiovisual percept is more than the sum of its sensory parts.
These audiovisual interactions can influence our reaction to the arts. Music, for
example, takes advantage of the natural properties of the sounds that occur in physi-
cal events and in human speech. Our sensory receptors and brains have developed to
pick up the most useful information for our survival, and these sensory experi-
ences tend to motivate us to behave in particular ways. Although sound stimuli, in
particular musical sounds, are often studied in isolation, audition is deeply linked
to the other senses including vision and touch (the body senses). These audiovisual
interactions, honed in everyday life, are transported into the museum or theatre as
part of our experience of the arts. Artists can take advantage of these interactions
ihto.rgh multisensory congruency or, instead, pit our senses against each other for
dramatic effect.
RnrnnnNcns
Belin P, Van Eeckhout P, Ziibovicius M, et al. (rqg6). Recovery from nonfluent aphasia after
melodic intonation therapy: a PET study. Neurology,47,1'5o4-tt.
Bray S and O'Doherty | (zoo). Neurai coding of reward-prediction error signals during
classical conditioningwith attractive faces. Journalof Neuroplrysiology,97,3q6-45'
Brosch M, Selezneva E, and Scheich H (zool). Nonauditory events ofa behavioral procedure
activateauditorycortexofhighlytrainedmonkeys. JournalofNeuroscience'25,6797-806.
(1999)'
Calvert GA, Brammer ML Bullmore ET, Campbell R, Iversen SD, and David AS
Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport,
ro, z619-23.
Calvert GA, Bullmore ET, Brammer MJ, et al. (t9g) ' Lctlvation of auditory cortex during
silent lipreading. S cience, 27 6, 593-6.
Caivert GA, Spence C, and Stein BE, eds. (zoo+). The Handbook of Multisensory Processes'
MIT Press, Cambridge, MA.
Ciark E (zoor). Meaning and the specification of motion in music. Musicae Scientiae, 5,
213-34.
Connor S (zooo). Dumbstruck: A Cultural History of Ventriloqui.srr. Oxford University Press,
Oxford.
Davis R (196r). The fitness of names to drawings: a cross-cultural study in Tanganyika ' British
Journal of PsychologY, 52, 259-68.
Drake C and Ben El Heni J (zoo¡) Slmchronizing with music: intercultural differen ces. Annals
of the N Y Academy of Sciences, 999, 429-37.
Ernst MO and Banks MS (zooz). Humans integrate visual and haptic information in a statisti-
cally optimal fashion. Nature,4r5, 429-33.
Ethofer T, pourtois G, and Wildgruber D (zoo6). Investigating audiovisual integration of
emotional signals in the human brain. Progress in Brain Research, tS6' 345-6t.
scu I ptu re, Frege G. OtgzitgSz) On sense and reference. In PT Geach and M Black, eds. Translations
Blackwell' Oxford.
ien of the from the Philosophical Writings of Gottlob Frege, pp.56-78.
Gethardt JW and Mowbray GH (rgig). On discriminating the rate of visual flicker and audi-
:- ::-:.¿oh:
tory flutter. American J ournal of P sychology' 72' 527-9.
Van Swearinqen.)
290 DAVID MELCHER AND MASSIMILIANO ZAMPINI
Kenez p (zoot) . Cinema and Soyiet society from the revolution to the death of Stalin. LB. Tauris,
Róder B and Ró.lcr i -
Calvert, C Spen-. .- -
London and New York.
MIT Press. Canrt - ---
Khan RM, Luk CH, Flinker A, et ai. (zoo7) Predicting odor pleasantness from odorant structure:
Saldaña HM and R' -:- -
Pleasantnessasareflectionofthephysicalworld. JournalofNeuroscience'27,too75-23.
ments. Percepti,'" -' ,
Kitagawa N and Ichihara S (zoor). Hearing visual motion in depth. Nature, 4L6, t72-4'
Schachrter A, Bradr T: j
Kohier W Ggzg¡g+). Gestalt Psychology, z"d edn. Liveright Publishing, New York'
auditory rhithnl- ...
Krantz G, Merker B, and Madison G (zoo4). Subjective reactions to musical intervals assessed
Music lll, Montr:..
by rating scales p. zz. In Proceedings of the Eighth International Conference on Music
Sekuler R, Sekuler \F .
Perception and Cognition North Western University, Evanston. IL'
.
Krantz G, Madison G, and Merker B (zoo6). Melodic intervals as reflected in body movement. 308.
In M Baroni, AR Addessi, R Caterina and M Costa, eds. Proceedings of the 9th International Shams L, Kamil¿ni\' -- .
Conference on Music Perception andCognition, pp. 265-8. Alma Mater Studiorum, University
\.;
visual flashing. .
Sh¿ms L, Kamitan' \. . -
ofBologna, Italy.
Kruse F (zoo7) Is music a pure icon? Transactions of the Charles S. Peirce Society: A Quarteily
In GA Calver,. a :i.
pp. 27-3j. MIT Pr..
J o urnal in Am erican Philo s o phy, 43, 62635.
THE SIGHT AND SOUND OF MUSIC 29r
1,. I -r,tisensory integration Lindenberger U, Li SC, Gruber W, and Muller V (zoo9). Brains swinging in concert: cortical
," ., -,'. -r.i o.f
N eur o s cience, 25, phase synchronization while playing gútar. BMC Neuroscience Lo' 22.
MacSweeney M, Caivert GA, Campbell R, et al. (zooz). Speech-reading circuits in people born
. -:r: speech information deaf . N eur op sy cholo gia, 40, Bo7-7.
'.-- r oices in the McGurk Madison G (zoo3). Perception of jazz and other groove-based music as a function of tempo.
In R Kopiez, AC Lehmann, I Wolther, and C Woll eds. Proceedings of the 5th triennial
:-:,--r in blind musicians. ESCOM conference, pp. 365-7.School of music and drama, Hannover, Germany.
Madison G (zoo6). Experiencing groove induced by music: consistency and phenomenology.
--: -,,s-modal associations Music Perception, 24, zor-8.
-:. :l'lRI study, Cognitit,e Manuei SY, Repp BH, Liberman AM, and Studdert-Kennedy M (rq8¡). 'Exploring the
'McGurk Effect'. Paper presented at the z4th Annual Meeting of the Psychonomic Society,
.=:. :¡r children, say cen- San Diego, CA.
Massaro DW (rq8Z). Speech perception by ear and eye: A paradígm for psychological inc1uiry.
: *l:-,: a comparative per- Lawrence Erlbaum, Hilisdale, Nl.
Maurer D, Thanujeni P, and Mondloch CI (zoo6). The shape of boubas: sound-shaPe corre-
--,:.. T Stambaugh trans. spondences in toddlers and adults. Developmental Science,9,3t6-22.
McGuiness A (zoo5). 'Microtiming deviations in groove.' Master's thesis, Centre for
'. .:;..:
. ,\IGA lournal of New Media Arts, The Australian National University.
McGurk H and MacDonald J (rgZ6). Hearing lips and seeing voices. Nature,264' 746-8.
-=r. \err.York. Nakada T, Fujii Y, Suzuki K, and Kwee IL (1998). 'Musical brain' revealed by high-field
::::odied cognition in (3 Tesla) functional MRI. Neureport,9,3853-6
:iesis. University of Navarra J, Vatakis A, Zampini, M, Soto-Faraco S, Humphreys W, and Spence C (zoo5).
Exposure to asynchronous audiovisual speech increases the temporal window for audio-
:1.rr¡riiming in African- visual integration of non-speech stimuli. Cognitive Brain Research, 25, 499-507-
Neri P, Parker AJ, and Blakemore C (1999). Probing the human stereoscopic system with
l,l:,:-h'. B Edwards and reverse correlation. Nature, 4ot, 695-8.
:: ,-riversirv Press, New Noé A (zooo). Experience and experiment in art. Journal of Consciousness Studies. T, t4-36.
Patel AD, Iversen JR, Bregman MR, and Schulz I (zooq). Experimental evidence for slmchro-
- :, : 1 j úl o.f Consciousness nization to a musical beat in nonhuman animal. Current Biology, 19,827-3o.
Patel AD, Peretz I, Tramo M, and Labrecque R (rgg8). Processing prosodic and musical pat-
-:. . Critical Studies in terns: a neuropsychological investig ation. Brain and Language, 61, t23-44.
Ramachandran VS and Hubbard EM (zoor). Synaesthesia *a window into perception, thought
-:. cf simple receptive and language. Journal of Consciousness Studies, S, 3-z+.
Repp BH (zoo5). Sensorimotor synchronization: a review of the tapping literature. Psychonomic
.i.:t' Press, Oxford. Bulletin 6 Review, 12, 969-92.
.::: :.-' Srdin. I.B. Tauris, Róder B and Rósler F (zoo4). Compensatory plasticity as a consequence of sensory loss. In GA
Calvert, C Spence, and BE Stein, eds. The Handbook of Multisensory Processes, pp.7]19-48.
,: m odorantstructure: MIT Press, Cambridge, MA.
."..";..27,1OO15-23. saldaña HM and Rosenblum LD (rqq¡). visual influences on auditory pluck and bow judg-
..:,.-:. 4t6, t7z-4. ments. P er c ep ti o n d. P sy ch o p hy s i c s, 5 4, 4o 6 -t6.
. \:rr
York. Schachner A, Brady TF, Pepperberg I, and Hauser M (zooS). 'Spontaneous entrainment to
*'r¡al intervais assessed auditory rh¡hms in vocal-learning bird species.' Poster present ed at The Neurosciences and
'.
.-..'tterence on Mus¡c Music III, Montreal, Canada.
Sekuler R, Sekuler AB, and Lau R (1997). Sound alters visual motion Perception. Nature,385,
=: in bodymovement. 308.
:; .::¿ 9th International Shams L, Kamitani Y, and Shimojo S (zooo). What you see is what you hear: sound induced
S -udiorum, University visual flashing. Nature,4o8, 7BB.
Shams L, Kamitani Y, and Shimojo S (zoo+). Modulations of visual perception by sound.
-. Society: A Quarterly In GA Calvert, C Spence, and BE Stein, eds. The Handbook of Multisensory Processes,
pp.27-33. MIT Press, Cambridge, MA.
292 DAYID MELCHER AND MASSIMILIANO ZAMPINI
Werner-Reiss U, Kelly KA, Trause AS, Underhill AM, and Groh JM (zoo3). Eye position affects ¡rtistic process uu: . i
activity in primary auditory cortex of primates. Current Biology,13,554-62. ¡tion and the conr.' ,
Zahorik P and Wightman FL (zoor). Loudness constancy with varying sound source distance.
popularand the nl, . -'
Nature Neurosci ence, 4, 7 8-U.
rrith complex coc.r'. . -
Zampini M, Shore DI, and Spence C (zoo5). Audiovisual prior entry. Neuroscience Letters,38t,
2t7-22. ;onsidered one ul .---. .
ir has greatly intlu.' - -
Cooke 1999; Kirch::.: :
:tional music rellc,.: .
, , .'"1; ¡.¡,1¡r,5teinweiss.
IN TIME: THE ART
. .; :. ti! io n. Macmillan, OF IAZZ
:,:nce in z-month-old
MusrcAL coMMUNrcATroN
David Melcher (D.M.): Each instrument can play a different range of notes. Nto and
tenor saxophones, for example, have a range that overlaps-but not entirely-with
the human voice. As a bandleader and composer, working with different instruments, Fig. 1:.r
including singers, how do you use the expressive power of each instrument?
that you can elicit an array of responses from people with the deliberate and artful ctte because thcn ' -.
juxtaposition of instruments, the 'coupling' of instruments as we say, and people .ounds, for eranrp.. :
respond to ranges and frequencies and different types of combinations in a variety r ion offers a calnl i::. - -
lcknowledge ol rrr.r .'
ol'rny career. Hort --
but with a focu) r'r : -
D.V.: How is it : -
,Jrums, which-in .... -
i. :n addition to playing
hers and writers. Greg
I educator. Among his
':Slious Blue Note label.
¡ic' rr-hich continues his
e1\' artists. Skip Hadden
i r¡clude World Fusion
,i"¿-: Drumming.Hehas
:s plaring drums on the
l"ort.
ON
¿t:on-has shown me of ways. A controlled acknowledgement of these responses gives you a broader pal-
le ,ieliberate and artful ette because then you can 'push buttons' in the listener. The combination of these
as r\-e say, and people sounds, for example, might make people anxious or angry, while another combina-
r:binations in a variety tion offers a calming compound. This is an aspect that many musicians do not readily
acknowledge or may not be interested in, but it has fascinated me since the beginning
of my career. How can I draw people in, more deliberately, not only with compositions
but with a focus on the means within those compositions?
Skip Hadden (S.H.): Well, it is hard. I learned by studying expression in general, not
just in music. So, for example, i bought a book on photography to see what techniques
were used: colour, shape, patterns, and subdivisions of lines and patterns. There are
interesting parallels with drumming.
I teach a class on basic time and pulse, which is about playing colourings and feel-
ings on the drum set. It is experiential. I start with a piece of American contemporary
classical music, by Charles Ives, played by the Boston Syrnphony Orchestra. The stu-
dents listen. Then their assignment is to make the drum set their orchestra, to focus
on what they felt and what they got out of listening to the music. It is a great exercise to
help them to move inside the music.
296 DAVID MELCHER, GREG OSBY AND SKIP HADDEN
All of the aspects of drumming-the tempo, rhlthm, tone, dynamics-can be G.O.: It is hard not ¡. -
used. The idea is to have them focus on the emotion in the music and to learn to com- ized-everyone \ran:s
municate their intent. I ask them: what is your intent? They look up the meaning of me at the time. Th,.
'intent', to make their ideas concrete, which is not just important for drummers but for pounds that infornt :i:
all musicians. those experiences a::
With drums, the vocabulary of sounds is enormous, because each drummer sets up equations, or some t:
the instruments differently: the tuning of the drums, the drum sizes, the sticks they different social attituc,
use (the tip sizes and shapes make the sound different), and the energy with which
they play. All of these elements have a big influence on the final sound effect. Musical D.M.: Some inten¿,¡
sounds modulate over time: attack, decay, sustain, release. Because drumming music. Why is some ::l
involves striking something, it is all about the attack (how quickly the sound reaches
full volume). The sustain (the level period of sound after the attack and decay) is sec- G.O.: Well, the tri-:-:
ondary, because it can get in the way of the next note. You are playing lots of notes, everything can be i¡lt
and you must articulate each note or it becomes a blur of sound. However, cymbals thing, such as a pa:.,.
can be used to create a sonic atmosphere, a ringing, based on the sustain, if this trying on your sens.s
atmosphere is compatible with what the rest of the ensemble is doing. Drums, just like the music. The un¡::.
any other instrument, can be out of tune with the rest of the group' beings tend to be irr,, ,
ablyto somethins, ir
D.M.: How is it possible to communicate colours and feelings with a saxophone?
D.M.: Some music ,¡.
G.O.: The science behind the arrangement of music involves waves' sound energy' What is it about cer::-:
moving air mass-and then these create physical effects and reactions in people.
Melody is the modification of one note at a time; the intervals that make up a melody G.O.: Groove is in:. ¡
of course make people respond in certain ways. But when you are dealing with chords' feeling through mu.---
harmonies, proglessions, and different t)?es of logic and you put all of that together rather than orsant.. l,
in a controlled frame, a composition, then you can enter into this composition process staid and regime r.:;
with a whimsical attitude or you can try to be constructive, thinking that 'I want people while another pie;; ,- : ,
to emerge from this performance feeling like this', or, 'I am trying to evoke this'. the first cake is'ertr,-''.',1
D.M.: Historically, some musical forms have developed from imitation of the natural S.H.: Everyone get: -1
world and the rhythms of the body (like walking, breathing, heartbeat). Where does development of t}:¿ -",
ri. d\xamics-can be G.O.: It is hard not to use the term'organic'here-it has been over-used and bastard-
sl; and to learn to com- ized-everyone wants to be organic, but I do seek connections with what interests
,:, ik up the meaning
of me at the time. This could be experiences or studies, and they serve as the com-
rr :or drummers but for pounds that inform the work. I try to have my work be a vivid and direct reflection of
those experiences and those trials. So if I happen to be studying some mathematical
- ¿:ch drummer sets up equations, or some properties of organic chemistry, or the behaviour of children or
:il sizes, the sticks they different social attitudes, then maybe the music will mirror those interests.
:lü energy with which
' s,-und effect. Musical D.M.: Some intervals are consonant, and the tri-tone has been called the devil's
e Because drumming music. Why is some music'harder'for people to like?
-¡il rhe sound reaches
:r¡¡k and decay) is sec- G.O.: Well, the tri-tone sounds beautiful to me. I think that it is the experience:
'; :-aring lots of notes, ever¡thing can be tolerated through association. If you are not familiar with some-
-ni. l{orvever, cy'rnbals thing, such as a particular tl?e of folk music or indigenous music, it is going to be
,- 1 the sustain, if this
trying on your senses, since you are not familiar with the array of constructs within
::r.ng. Drums, justlike the music. The unfamiliarity is usually a generator for rejection, because human
--:f .
beings tend to be intolerant by nature unless they are conditioned to respond favour-
ably to something. People do need to be more open, but that is easier said than done.
,\ -:n ¿ Sa-\ophone?
D.M.: Some music seems to lock you into a groove, to make people dance or move.
! ir:1"es, sound energy, What is it about certain rh¡hms that swing or groove?
i :.¿ctions in people.
::-at make up a melody G.O.: Groove is impossible to define. It is an innate ability to feel and to project that
,:. :eahng with chords, feeling through music. You know when it is not there-things are stiff or robotic,
: *r ¿ll of that together rather than organic. Music without groove lacks that humanity, it is technical and
.rs.LrmPosition process staid and regimented. It is the same with food. You say that 'this cake tastes good',
:¡:r{ that'I want people while another piece of cake that has the same ingredients just does not taste as good. So
:¡:o er-oke this'. the first cake is 'grooving'while, for the second cake, it is just not happening (laughs).
::t:;¿tion of the natural S.H.: Everyone gets in synchronicity, like a form of entrainment. It goes back to the
l.lrtbeat). Where does development of the word in the seventeenth century by a Dutch scientist, which
put a name to it.' In the agrarian cycles, this goes way back with synchronizing with
the seasons and the stars in planting. During the year, farmers go to the fields and
:-1. mav be one reason come back together in synchrony. In Korea, there is a folkloric tradition, linked to the
:rlrduce music. There farmers, of drum playrng. They accompany the farmers to the fields, at different
: :tample, there is the times of the year, with specific rh¡hms. It is part of the military tradition, in marching.
.:. I Lile near the ocean,
'unped up and started
:i,i-thms of the natural ' The term 'entrainment' was coined in 1666 by the Dutch physicist Christian Huygens, who
:¡: ritches and rh¡hms described how pendulum clocks stabilized into the same rh¡hm, but in antiphase. This
phenomenon, which may be related to resonance in physics, has also been applied to sociology,
;¡iLrurs in the music; it
neuroscience, and other fields ofstudy. For example, biological cycles, such as the sleep-wake cycle
- balancing ofthese or menstruation, tend to become s1'nchronized in groups. Musical entrainment describes the
'he
tendency to sgichronize based on rh¡hm or melody.
298 DAVID MELCHER, GREG OSBY AND SKIP HADDEN
In the folkioric sense, it is still going on: drumming to bring on the sun and the rain, ..ie as.plavins t¿r-,. --: .
or to tell people when to go to the fields. :l:ies. soulfulness. -, . :
D.M.: The speed and loudness of the music seems to affect people-it can make then D.\f .: Can ntusic p *: :,
eat and drink more when they listen to iouder and faster music (see Spence et al., erample oi ir-hich -. -:
Chapter u, this volume). How do you use these differences in performance? :re aimed at trairs:,- ri..
G.O.: When we perform in clubs and other venues where you can see the audience-as S.H.: \\'e11, if the ,. :
opposed to concert halls or other places where you are far removed-then we use a :¡-tusic, thev sho,.,' ::r.:
process called'reading the audience'. Before starting, we programme the set with 'hot' compiicated in si:,-:. --
and 'cold', fast and slow pieces, shadings and opposites that embrace different parts davs, not hqu¡5-',, 1¡¡
of the spectrum. But then, if you feel that you are not getting a rise out of people, if rormal frame oi ir,::. .
they are not being attentive-you are losing them, they are bored or even reject what about before and ' .:: .,
you are doing-then you have to make a sudden alteration in your programme.
The tempos, the level of dynamics and volume in these songs are designed to elicit G.O.: This n¡-rsi¡ -, .
emotions, even to jump start or shock these emotions, because otherwise you can lull people into a ¡¡:--:.-
feel people slipping. Only the most self-indulgent musicians say'they are not elevated attected. Because :- -. *
to my level, they are not meeting me at my telms so I am not changing a note'. are r'r'earing arntcr -::. '" l
an aural message. -: := -,
D.M.: What is the role of expectations and chalienging expectations? Can a certain
rh¡hm be too boring, or, conversely, too challenging for the listener? D.N{.: It has been .'--:.
-,
belrond the indir ic,r.-
S.H.: It depends on the artist and the audience. Even for me, when I hear certain
beats such as 'bum TA, bum bum TA'-over and over-I would say'oh no, not that G.O.: Fundament:-,'. :
again', because there was a time when all the drum machines were playing this. But it democracy. The ,¡::-.:
depends. Even the well-known jazz drummer 'Ndugu' Chancler played that rh¡hm the president-thrr: ,,
on Michael lackson's song 'Billie fean' (1983) and that worked great for that pop the music u,ho .-.re ¡.- :
song, even though there is nothing to it. That rh¡hm is going to make you move. rvi1l add some ins::i .
And that is something that heightened his career, this ability to take things right to
their essence-to play time like that. There are other styles of music, but the key is to S.H.: With musi¡. ::.:-
reach a place where it appeals to and touches everyone. It may not be a z/4 beat, but 1oo ooo people i: . ..
there is enough motion in there that they get emotions from it. Muyb" they connect at rvould get more rr'r:a- ,..
the roots. One purpose c,- :'.-
record it. You rr'¡r',i-:
G.O.: Peopie often take the easy route. There is an overwhelming laziness that really good plar-er. . ,-:
abounds in every strata of music: classrcal,lazz, pop music. They allow some daredevil est. The music n¿-,''-,.
or maverick to take the lumps for being first, and then they see if that person sinks 'The Melody at \-¡:.. ',,
or swims. If that person prevails, then the others adopt those techniques, jump on an illness, which h¿ : -
the bandwagon. I favour the trailblazers and think that everyone has that potential. enough to plar', an.i. .
Of course we cannot have bastions of people with all-new ideas because then there said: 'this is not jus: - :
would be no coherence. There must be some threads that hold it all together. You can't it is inspirational. Il l. .
have a wall of all bricks with no mortar. There must be something to hold everlthing thing you are hittrni: r
together. And you also need a strong foundation. Some people embark on these new greater than the 1int,t:,.
IMPROVISATION IN TIME: THE ART OF IAZZ
'r i:le sun and the rain, ideas, playing far-out conceptual compositions but they don't have the basics-the
blues, soulfulness, the feelings. It is all limited, intellectual, and left-brained.
i-¡-it can make then D.M.: Can music put people into a trance? Repetition is a key feature of prayer, a good
:.:c see spence et al., example of which is Buddhist chants or reciting the rosary beads. These practices
::.,¡rmance? are aimed at transporting the listener to another dimension.
1se3 ihe audience-as S.H.: Well, if the field recordings of African music are true to the original ancient
r,', ¡d-then we use a music, they show that often there was the same rh¡hm-whether it was simple or
:rrle the set with'hot' complicated in structure-repeated over and over and over again, over a period of
:rr¡.e different parts days, not hours-what that did was to move people to ecstasy, move them out of their
. :ise out of people, if normal frame of thinking. Maybe it is similar to that trance-like state we were talking
iú r-rr €\-€rt reject what about before and that is where it may fit into religious music.
:n i-our programme.
::e designed to elicit G.O.: This music is a tool. It is a functional music, some of which is designed to
"¡¡ trrhenr-ise you can lull people into a semi-trance state, a state of passivity or release so that they can be
' 'the..-are not
elevated affected. Because if you are 'uptight' then you are resistant, you will not respond, you
i:::irq a note'. are wearing armour. Meditative and ritualistic music used repetition to release, like
an aural message. It gets past the barrier, although some people are impenetrable.
:::r.rns-l Can a certain
.¡ner'í D.M.: It has been suggested that music can play a role in creating group cohesion,
beyond the indiüdual. Which aspects of music are social and which ones are individual?
. I hear certain
',,,'nen
.
j
.ar' 'oh no, not that G.O.: Fundamentally, any kind of ensemble or combo is the best example of a mini
-:; plaving this. But it democracy. The leader, whose composition you should be playing-maybe he is like
¡lar-ed that rh¡hm the president-there is the whole cabinet, the whole cast working with the rules of
':
:i sreat for that pop the music who are called upon to be free and to input, as specialists, something that
i trr make you move. will add some instability, which is an important propellant. We need that.
- :ike things right to
rL:sic. but the key is to S.H.: With music, there is also the social aspect of seeing music live. In a stadium with
l:.rt be a z/4 beat, but 1oo ooo people it is not just your personal relationship with the music, which you
'1.:',-be they connect at would get more with stereo and headphones.
One purpose of music is to be inclusive; otherwise you would not bother to
record it. You would just be in your basement, playing. There are probably some
r¿-mins laziness that really good players out there, just like that, for whom reaching out is not their inter-
-
¿ilt'r¡.r' some daredevil est. The music may just be about your own self, For example, there is a great recording
'; ii that person sinks 'The Melody at Night with You'by Keith Jarrett (rggs), after he had recovered from
-;;hniques, jump on an illness, which he did in his home studio as a gift for his wife. He finally felt well
:¿ has that potential. enough to play, and, it is amazing, he just did melodies. When his wife heard it, she
:s because then there said: 'this is not just for me, you have to share this with everyone'. And it is magical,
a., :oeether. You can't it is inspirational. It is a challenge for a drummer-how can you do 'love'with some-
:E lo hold ever).thing thing you are hitting? You have to transcend the instrument. Your message must be
er.rbark on these new greater than the limitations of the instrument. Not that I can do that, but that is a goal
DAVID MELCHER, GREG OSBY AND SKIP HADDEN
worth pursuing. If nothing else, people should hear that I love playing the instrument. There are so n-l.r-ir- :
By offering the music up-it is outside of me-I am offering it up to a higher place to a place where th... .
beyond where I am sitting and playing. Other musicians, then, can join into that why they are doins ,, .-, -
together as a group, and the listeners are more involved by the music. Mark Orfalr..' ¿ '.:'
'I used to pla,v the c:*:
I just read about i::-..:
from their home .c) *:-,
zoo8). They \\rere ¿:::-
them zo years ago. ::- J
P¡.n¡.rrELS AND TNTERACTToNS BETwEEN texture were unmist -t...
that it was their
IAZZ AND OTHER FORMS OF ART co .-s:::-
same recipe, the s¿..n. --
their personal hist¡ :",
D.M.: Are you inspired by other art forms, such as dance or painting? smelis and taste s. c¿:r ::
G.O.: Definitely. The aesthetics of other art forms play into my music. I collaborate
with a variety of artists, including visual artists, spoken word artists, choreographers,
and others, because these collaborations are the purest, rawest form of artistic expres-
sion, People meet and find the commonalities which serve as generators for creativity.
So I seek that out, because it is motivating and inspiring.
S.H.: Many years ago, one of the main inspirations for my playing was Max Ernst. D.M.: How do r-ou ¡ ::--
I had seen his artwork and I thought: I can do this with my music. I can create a frame-
work, to be the frame of the canvas in which to put the colours, and be the backdrop G.O.: With conlprr )-:-
to those lines and colours. I went to a retrospective of his work at the Guggenheim before, maybe beti<:. l,l
in the 7os, and I was totally blown away when I saw them live, it was a most intense germ. The germ o^ ::..
experience ('Max Ernst: a Retrospective', Guggenheim Museum, NewYork, February- then your songs rr:,- - ..
April 1975). same instrument. i:trr:
start with a ditter¿:-: .:
D.M.: Do you see any synaesthetic connections between the arts? be dramaticallr' di:i := :
divinely inspired. -,,, -..:
G.O.: A lot of musicians see colours when they compose. Musicians try to fash- vou, maybe in a dr..:-.
ion a 'movement' in the compositions. Just like a painter will determine a palette, rvhile it is still fresh L: ..
in terms of shades and gradations, before we begin to compose we make decisions I want the musi; : .
about which musicians, instruments and environment will be used-what the whole current work to b¡ :--
'landscape'will be. the interesting peo-.,. ---
the music to be . ...
S.H.: When I am comfortable within the music, I am painting with sound. I actually too anall.tical, too ir::.,
'see' the themes and try to play those. I don't tell people what the theme is, but the
other players understand. One time I was playing with a piano trio, Kenny Werner
and Ratzo Harris, in New York and I was the ocean, I was trying to create waves with
rising and falling dynamically and different colours. At the end, the piano player, ' Marc Orfaly rlas :i,..
interview, he was chei ¡-
Kenny, said to me 'I am all wet' . . . not because he was sweating, but because he got cuisine began when, in :
the intent. cook.
IMPROVISATION IN TIME: THE ART OE TAZZ
:, a''ing the instrument. There are so many parallels, for dancers, chefs, painters, who are all trying to get
:i rp to a higher place to a place where they can express their intention. If artisans can get in touch with
,;r. ;an join into that why they are doing what they are doing, then they can fulfil that function.
*-..:-
- J \1L. Mark Orfaly,' a famous chef here in Boston, was a student of mine. He says,
'I used to play the drums, now I play music with tastes'. There is an interesting story
I just read about diners at a London restaurant who had ordered a favourite dish
from their home country of Pakistan, which they had left many years earlier (Aslam
zoo8). They were amazed that it tasted just like the way their aunt had cooked for
them zo years ago, and the aunt had been dead for ten years. The spices, aromas, the
BE TWEEN texture were unmistakable. They peeked into the kitchen and found, to their surprise,
ART that it was their cousin-the daughter of their aunt-who was the chef. In tasting the
same recipe, the same message, they got the essence of that direct communication of
their personal history and their culture. In the same way, certain sounds, like the
smells and tastes, can carry the information of the culture in which they originated.
:', -rg \ras Max Ernst. D.M.: How do you compose music?
¡ , ¡¿n create a frame-
,. :nj
be the backdrop G.O.: With composition, I don't do anlthing that other musicians have not done
:., ¿t the Guggenheim before, maybe better. Many composers can get stuck in a rut by starting with the same
. :: ,,r-aS
a mOSt intense germ. The germ of the idea is the catapult, so if you always start at the same point
. l{¿',i York, February- then your songs wili begin to implode, to sound the same. If you always start with the
same instrument, piano or saxophone, then this happens. So, first, it is important to
start with a different initiate. Second, the components for the composition have to
.l be dramatically different and full of charge. It must be not just to meet a deadline but
divinely inspired, where you jump out of bed at 3 a.m. because something comes to
l'.icians try to fash- you, maybe in a dream. I don't know where it comes from, but I have to document it
, :.etermine a palette, while it is still fresh in my mind.
.- .|,.'e make decisions I want the music to be informed by my most recent experiences. I want my most
*.:c-ir.hat the whole current work to be reflective of ever¡hing I went through since my last recording:
the interesting people that I meet, the books that I read, and the research. I don't want
the music to be a technical manual on CD-I have been criticized for being
,r 1i¡l 5ound. I actually too analltical, too intellectual. But I don't know any musician who just woke up as a
,r: theme is, but the
: :rio, Kenny Werner
i :Lr create waves with
r,j. the piano player, ' Marc Orfaly was listed tn Food 4" Wine Magazln¿'s ro Best New Chefs in zoo4. At the time of this
interview, he was chefico-owner of the Pigalle and Marco's restaurants in Boston. His career in
-rs. but because he got cuisine began when, in order to earn money to pay for his drum kit, he took a job as a short-order
cook.
DAVID MELCHER, GREG OSBY AND SKIP HADDEN
virtuoso, it is born of sacrifice-you must sequester yourself to do an¡hing interest- closely simulate or in:
ing as an artist, and this is true not just ofartists. tically with the int, :.
My compositions have to make a point. Each composition has a point-rh¡hmic, with other support e ::
harmonic, or structural-that I'm trying to make. But I am always trying to mould Other elements n-r¿i
these theorems into something musical, because otherwise it sounds solely technical, Each of the instrur::...
left brain, and anal¡ical. story, in terms of ne-'-,
I am working on a set of structures, of intervals, on a particular concept, study, all of those things. tti -
or research. Or some innocent children that I have met, I am trying to elicit the feel- erate: but there is sti -
ing I had talking to these people who have no inhibitions, or talking to someone
in another country through an interpreter. When I travel, I tell the promoters that
I want to talk to certain people, because they have some information that is useful and
I can transcribe it. Composition is deeper than just: 'these are some cool tunes, it's
grooving, it's swinging, it feels good, it sounds great and makes me want to tap my
feet'. That is ali good, but at the base level there is a reason behind why 7 wanf fo
tap my feet.
Not everl.thing has to be exploited, broken down into individual atoms, because D.M.: How do pe..:-. ,
that can take the fun out of things. Some things are completeiy risk taking, a role of understanding the i::..
the dice. I will juxtapose one element with something completely different that rh¡hm in your boc', :
I learned a year ago: maybe Bulgarian folk music with the drum chants of Ghana,
.'
and combine that with communicative drumming from the Caribbean. I might com- G.O.: We are plarir-.
bine music from the African Diaspora with five-tone scales from lapan that are tuned has a certain nun.ib.:
differently. It is like a soup and you try to expand the ingredients. Sometimes it is There are directicri¡. ¡ "
a dismal failure, sometimes you only learn only that these ingredients should not ness. I think that it '. .,
be combined. The proportion of those ingredients is important. Some people because that is the ::: ,
may use all the ingredients, but use them differently. Some people might be heary- a composition-\ c *.:
handed with the salt, others might not use that much salt, or might use only some of the tightrope. Some :.
the ingredients. recall': they have r::e:r
Some compositions have a primary means of focus, for example a dominant recall that. For m¿:', :
rh¡hm that is recurrent. It is not looping or repetitious, but it is referred to liberally the rest is true spo: -::--:
and deliberately at structural points within a composition. It is the thread that holds
it together, the underlying structure or 'centre'. That can be intervallic, or it can be S.H.: Everyone k¡c-"'.
a melody. In classical music, it is called a recapitulation. There is a melodic theme, players. This mar' ¿i, - ,
followed by a restatement of the theme with variations built upon it. At key points, what is being plar e; :
the theme is modulated and taken into other areas; it is almost obscured but it is the cymbal and the l .
still there. And then the recapitulation involves a restating at the end. There is a thread you can use the oth:: ..
that is weaved through the music. it is moving along. ::::
Structurally, you can set up the composition almost like an architectural blue- themselves, the ent.:. :
print. For example: 'for eight bars I am going to do this and then for sixteen bars the right hand, anc ,:,¡
do something else'. You are effectively 'building' a composition, using instru- the snare on the drur:, s
ments as building blocks and constructions such as a crescendo, then a spread- within the music.
ing out of the instruments or a solo flight with one instrument. There is support: a You do not neei t,-
second instrument can come in, and then there are two voices crossing each other. the same phrase max.
There could be an improvisational section. There are other structures that more same style, not chai:::-
TMPROVISATION IN TIME: THE ART OF IAZZ 3O3
j,--, anrthing interest- closely simulate or imply a geometry. There are different directions: playing ver-
tically with the information ascending and different elements cascading, along
... ..¿i¡¡-rhythmic, with other support elements under it so that it is not out there alone.
-'",,:-,'s tn'ing to mould Other elements may be stark and unsuspected, like a narrative: a play or an opera.
-:Js solely technical, Each of the instruments in the combination is role-playing. They each fit into the
story, in terms of melodic structures, moving the tonal centres, chord progressions,
: ;..,.:r concept, study, all of those things. Of course, it can be more intuitive. Some people are not as delib-
-,',:{ to elicit the feel- erate: but there is still the structure, even if they are not aware of it.
- :.JJng to someone
.-- ire promoters that
::.i,;: that is useful and
: :-,utc cool tunes, it's
.> :ti. \rant to tap my
t::t-rd rr'itl | 1/¿¡1 ¡a
lvrpRovrsATroN
',,:-i¿l atoms, because D.M.: How do people stay in sync during improvisation? Is improvisation based on
', :-sk taking, a roie of
understanding the musical structure, papng attention to visual cues or feeling the
:: :iel''' different that rh¡hm in your body?
""-:: ;hants of Ghana,
:-¡'¡e an. I might com- G.O.: We are playing within forms. Each song has a certain form and structure; it
-: lFan that are tuned has a certain number of guidelines, although you have choices that you can make.
-::r:s. Sometimes it is There are directions, but you are only limited by your imagination and your fearless-
--.::Jients should not ness. I think that it is essential that musicians develop their improvisational prowess,
-::¿nt. Some people because that is the most honest and genuine detailing of who they are. You can play
:,:-e might be healy- a composition-yours or someone else's-but when you improvise you are walking
r.:i: use only some of the tightrope. Some people are very adept at improvisation, but for others it is 'total
recall': they have memorized a host of variables and then when they get on stage they
a dominant
=,'.;.n.rple recall that. For many, there is a proportion of about 5oo/o to 6o7o memorization and
:. retlrred to liberally the rest is true spontaneous inspiration.
:r-:hread that holds
::.:','allic, or it can be S.H.: Everyone knows where they are, and where everyone else is, if they are good
: -s ¿ melodic theme, players. This may allow more freedom for the drummer to 'comment' musically on
: : r lt. At key points, what is being played, as opposed to just keeping time. In jazz,you keep time with
- .' .,bscured but it is the cymbal and the high-hat, with the repeated ostinato (a repeated pattern). Then
=rc. There is athread you can use the other hand and your feet to comment on the musical conversation as
it is moving along, and you can help move it along. The drummer can become, by
,- ::;hitectural blue- themselves, the entire rh¡hm section: piano, bass, and drums. I can keep time with
,;:¡:l tor sixteen bars the right hand, and then play commentary on the chords with the left hand around
,-:ron, using instru- the snare on the drum set, and the base line with the base drum, playing all those parts
=..1,r. then a spread- within the music.
.:. T:ere is support: a You do not need to directly answer every comment, just as repeating back exactly
. ;:.rssing each other. the same phrase makes no sense in a conversation. However, it needs to be in the
:,:latllres that more same sty1e, not changing the topic of the conversation. It is the same thing in music.
DAVID MELCHER, GREG OSBY AND SKIP HADDEN
If the music is totally improvised, you may not play tunes. Playing with Wayne Shorter, You should be able tt- :,
for example, you must be open to whatever the conversation might be. And if you are without it being pairr :.
talking to him, he has favourite subjects that he likes to respond to, like talking about would be a caricature. -
music, books, or food, so that he is thinking of something outside ofjust melody. Rather than a coprisr. .
More often, in jazz, there is a melody or form that people are playing over, that a personal reflection ,': '.
you can return to in the music. Rh¡hm, harmony, and form can be the foundations as an artist.
for improvisation, but you can also open it up to other forms or sounds. Starting
with 6os avant-garde music, for example, the tonal centre of the music could D.M.: What makes tc:, .
move within the piece. Some artists, such as john Coltrane, began to use a second
drummer, bringing in Rashid Ali, in addition to Elvin fones, to have the energy of S.H.: I try to get stui¡::
two drummers. the melody on thc Jr-.
trol over the dynanr-;.
D.M.: A recent neuroimaging study (Limb and Braun zoo8) suggests that impro- the music side, then .'
visation differs in many ways from playing a set composition, particularly in terms reaches people, riher '.
of changes in attention and awareness. During improvisation, there also seemed of time, drums har-e -- -
to be a deactivation of areas involved in monitoring and censoring behaviour. The The primary in'trun:.:
study suggests similarities between improvisation and forms of altered awareness like accompanied drums.
meditation or REM sleep. What is the relationship between thinking and intuition in
improvisation? Do you think when you improvise? D.M.: In every tl?e r¡. :;
For improvisation. ., - --
S.H.: For the most part, you can't think while you improvise playrng music. Other-
wise, the best that you can do is to play the instrument. What you need to do, instead, S.H.: I think that rh :.-
is to play the music. The drums, in this case, are like your pen or the computer the m usic. With pl': . . :-. -
keyboard, something that is used only to articulate the larger ideas and feelings. It is so that you are al) r
a language that you are using. The combination of sounds is used, rather than words, moving, in a supp, r ....
to communicate your intention. ual within lhe s,rrr*: :-:
leader, can be the l.-:
G.O.: Intuition and instinct are critical. You are responsible for more, personally, always being out lion:
because when you work with memorized material the audience is familiar with and
accepts that material. When you are improvising, as they say in hip hop 'coming off C.O.: It
is a gooJ :.-:.-.
of the dome', that responsibility leads to a freeing up of the inhibitions and impasses responding. the m*: - -
that come naturally when you try to satisfy people. It is liberating and honest. There mentary tO the en:c:-.. -:
is also no true way to grade it: there is some bad improvisation, when it is done reck- version of the piec..
lessly without any concern for or adherence to the parameters that have been set up: at the conclusion .r: .:. ,
otherwise, there is total cacophony. There is plenty of that (laughs). may barely resemhl¡ . - .
when recording. \o -.
D.M.: So what makes for a good saxophone solo? The venue and i.:..
'vibing with the audier,--
G.O.: First, having a good tone, a tone that is as distinctive and individual and pro- cannot even see thei::. :.=.,
voking as someone's voice. When you hear someone speak, the patterns make it menl.s are too lar ap::.
clear with one word who that is. For me, that is the bottom line. I don't care how obscure every4hine.
many notes they play, if it is technical and virtuosic, with all this dexterity and jumping A great performar.-.
around, if they sound human than they can't go wrong. ment, you are able to r -
Also important is an adherence and acknowledgment to the precedent that has it is complementan i: -
been established, in terms of the lineage of the music-the great saxophone players. volleying of intent. \r :.:
IMPROVISATION IN TIME: THE ART OF JAZZ
:i rrith Wayne Shorter, You should be able to hear the contribution that these great players have made, but
:rght be. And if you are without it being patronizing or directly mimicking what they did, because then it
rc to. like talking about would be a caricature. There is a fine line between acknowledgment and copying.
:l¡ of iust melody. Rather than a copyist, I would rather hear someone who channels this history into
: '::e plaving over, that a personal reflection of who they are based on their own experiences and aspirations
;¿: be the foundations as an artist.
:is Dr soutds. Starting
. ..f the music could D.M.: What makes for a good drum solo?
r-ian to use a second
ir¡ have the energy of S.H.: I try to get students to learn the melodies and then embellish those. You play
the melody on the drum kit as if you were a trumpeter or saxophonist, with con-
trol over the dynamics. And then embellish that melody. But by coming, first, from
igqests that impro-
: the music side, then you end up with 'music', not just drums. And that is where it
r. l¡rticularly in terms reaches people, when it is not just limited to rh¡hm. However, from the beginning
: i. :here also seemed of time, drums have been used as a motivational force, in order to move people.
.::ine behaviour. The The primary instrument was voice, with the melodic content that preceded and
i .-:ered awareness like accompanied drums.
::king and intuition in
D.M.: In every type of performance, there are going to be good nights and bad nights.
For improvisation, what makes it all go right or go wrong?
:,:',ing music. Other-
:u reed to do, instead, S.H.: I think that things go wrong when everybody, collectively, gets in the way of
:3:r or the computer the music. With playing, you need to get out of your way, out of everyone else's way,
i':¿¡s and feeiings. It is so that you are all moving towards the same direction. It is only when everybody is
moving, in a supporting role, that anyone can step out successfully as an individ-
'el. rather than words,
ual within the group. Because when you are not thinking, then everyone can be the
leader, can be the lead voice. Your function is to find grace in that supporting role, not
:.¡i more, perSOnally, always being out front.
;¿ :s tamiliar with and
--r :ip hop 'coming off G.O.: It is a good night when-to use a cliché-ever1'thing is in sync. People are
r:b:rions and impasses responding, the musicians are making wise choices, artful choices that are comple-
:1i:q and honest. There mentary to the ensemble and elevate the music beyond the familiar. The most familiar
. -,,,-hen
it is done reck- version of the piece will likely be the recorded one-this is the model. But hopefully
.hat have been set up: at the conclusion of those performances, those compositions will have regressed and
may barely resemble the recorded version. People are more tentative and conservative
when recording. No one wants to make a mistake and blow the take.
The venue and context are important, too. In a live performance, when you are
'vibing with the audience', then you can see that they are with you. Sometimes you
:C indir idual and
pro- cannot even see them, the venue is set up that way, or you can't hear because the instru-
tne patterns make it ments are too far apart. There might be too much reverberation, so that the drums
''ne
. I don't care how obscure everything.
le r-te ritv and jumping A great performance is when everi.thing that you wanted to say with your instru-
ment, you are able to execute with ease. If you hear something and you can play it and
::: :recedent that has it is complementary and it inspires someone else to respond in turn, then there is a
::. sa\ophone players. volleying of intent. When you speak with someone, you want to say something new.
30-6 DAYID MELCHER, GREG OSBY AND SKIP HADDEN
It is more interesting, for example, if inan interview, you do not have to come up per second, articulat¡ :-,
with newways to answer the same old question. This interview is a good change, because trumpet or saxoph.r' - -
I do not know the next question, I do not know what you are going to ask. those instruments rr::
G.O.: Jazz in and , ': -.
enous music. it i. . :..
differently in difi'e:.'-
HrsroRY AND cuLTURE example. But the s¡'ic.
partmentalize [he n:..-. -
Musicians don't nc..,, ,
D.M.: Art is often a response to the cultural context and changes in society. In the early
twentieth century, the Futurist Luigi Russolo (ú85-1947) wanted to incorporate the D.M.: How ha. co;:t.:. .
sounds of industry, to 'harmonize noises' and play the city as a giant i¡5¡¡u¡¡s¡¡- sational music is he¿:.
using the noise produced by trains, factories, and automobiles as sounds. john Cage,
in his composition 4'33", makes us heart the ambient noise. Similarly, there was a S.H.: One big irt.r. r :.
d
recent installation by David Byrne, Playing the building (zoo8), using a big industrial the world is heari:r- -
space in which the elements of the building itself made all of the sounds. There is a music. The drum .. :
category called 'urban music', but so much of music in the twentieth century might be and feet. ln Carn¡: - -
considered urban, as the city plays such a role in people's iives. Does jazz music reflect the two hands t,ec,..,. '
this urban influence? The recent Johr, \1. -
of lndian drumnl.:,
G.O.: Urban areas, as opposed to more remote areas, progress more rapidly. There is drum kit, traditio::, '-
a flux of ins and outs, transitory elements, with people moving in who affect the situ- of course, the recor:.:'-
ation with their character, habits, and traditions, and then people moye on. Since it
evolves rapidly, I would like to use that as a pool of inspiration, rather than a place G.O.: With the in .:.'..
that never changes. That is why you have more contemporary and progressive also opens the tl,',':-. . :
artistic movements happening in these so-called urban areas, as opposed to outlying rveb. You need .r n---, -.,
areas which are more favourable to folk musical forms which change more slowiy. the gems, and r' * : -
rvho have storre:t. . -
D.M.: Music is categorized in certain styles. Some of this is labelling or pattern Was there €\ r - -r' :
e
recognition. Even a music novice can tell if a piece sounds like a waltz or sounds between t959 and ,. "
'Latin'-is there something essential about the music reflected in that label or is it one ideology. The .,. ,
entirely a cognitive association? ol dual concert> \1,.
and sage-and rr lr-. ,
S.H.: Well, Latin sounds are the product of a particular culture, so people would lrom The Villaec C, :
relate to the sounds from their background. In Brazil, for example, the sounds that one upstairs ¿nd ' -. -
they are piaying stems from that produced by the birds, animals, or insects-so all his group with orhc- -:-
those elements are influencing what's happening with the music. we would not neces- rvhoever, four ditr.. -' '
sarily know about that from here, but we would get that laid-back feeling-regardless had arrived with ¡ .:-
of how fast or slow the music was-because it lays back on the beat more. when you night.I asked lack jc . .-
come further north, even to the Caribbean, then the music has much more staccato,
is much more 'on top'. The Brazilian stuff is 'down the middle', or even behind the
Carn¿tic I-Tru: r i: ..-.:
beat. As you come up to NewYork City, it is even quicker and faster. Everybodyis play-
lhi'music as based o- -.. l
ing more notes. And the technology plays a role, so that people can play more notes on conrpositions blr '-.
\ IMPROVISATION IN TIME: THE ART OF IAZZ 3O7
.. nrrt have to come up per second, articulate more, and still have it be heard. Even older instruments, like the
s ¡ gurod change, because trumpet or saxophone, need to keep up, so that technology is influencing the way even
:,lng to ask. those instruments are played.
were lines down the street'. I had an overwhelming feeling of being born late, because have a level of fear-.s,
that is when j azz really flourished. the environment. T::_.
Things happen in cycles, it is inevitable. There will be another 'lazz Age', another to transitions. Ther . ,
Renaissance: history has shown it. After WWII, Vietnam, Korea, after these wars or a instinctive muslci.:,
political malady, once the situation stabilizes, the musicians who were forced out of 'elephant ears': r\.'. -_
work-who had ro get the day job-have been in the lab tooling for something that happen and lher rr: '. :
will be their marker once the dust settles. Right now, as we speak, there are a great they pick up on rhi:r-.
number of musicians on the threshold of new discoveries. The venues will open up, Music must bc .:.- _
the economy will flourish, and people will have disposable income to pay to see these j:. - -
thing, what is thc
musicians, and then it happens again. I am looking forward to that. accomplished and r-i: ::
parable in the an ,.. -..
anizing their life ar - - -
menl involves ult.;::.
practice, ali the .¡;:--:.
ExpERTISE AND LEARNING slate, with the inn,,,, - -,
the glue that binJ,
paradox. I hare ru..
D.M.: How is listening to music different, as a professional musician? order nol to be b.,: - -
S.H.: First, I try to get past the first level, of what the drummer and the other musicians S.H.: I found rh:r _ -
are doing, to listen to the music itsell the total experience, the energy of that music. successful later o:-. '
But if something is out of tune, out of time, it jumps out to me, of course. I heard them isapotenti.. :' . -.
someone play recently, someone whose playing I really enjoy, but the cymbals were 10 present nrusi¡:_ - '
out of tune with the rest of the music, and I almost had to ieave the hall. I had to move to do that. Eren :'
beyond that. areas ofyour life.
D.M.: What can and cannot be taught? There is a debate in cognitive science New York.
about whether talent exists, or, instead, if some people just pay more attention and
practise more.
G.O.: With enough practice, then people who have marginal talent can become pro-
ficient. But they may not become the most creative artists. The innovative musicians
IMPROVISATION IN TIME: THE ART OF IAZZ 3O9
.'i'rng born late, because have a level of fearless and a keen intuition-an awareness and a quick response to
the environment. There are a lot of proficient musicians who cannot respond quickly
¿r'Iazz Age', another
't
to transitions. They have to be prepared to know what is going to happen. The most
.r. alter these wars or a instinctive musicians only need a blueprint and an idea. They have what we call
,. trr l\-ere forced out of 'elephant ears': they can hear around corners, they can anticipate what is going to
r-:rs ior something that happen and they are ready for it. These people become the prodigies, with less training
':.:k. there are a great they pick up on things and their response time is shorter.
ri l'enues rvill open up, Music must be 'artful', otherwise there is chaos. If everyone is doing their own
-,::re io pay to see these thing, what is the delineation between people who have learned, studied, and become
-i- :-
accomplished and other people who have been playing for two weeks? There is a Zen
parable in the art of the Bushido swordsman. After learning all the techniques, org-
anizing their life around their sword, their 'instrument', they say that ultimate achieve-
ment involves 'unlearning what you have learned'. You must first go through all the
practice, all the sacrifice: only then can you enter into something creative with a blank
\G slate, with the innocence of a child-that is when you are the most pure. The rules are
the glue that binds it together, but you are not bound by the rules. It is almost like a
paradox. I have rules but no rules. I am thinking non-thinking. You have to learn in
: !: l.
order not to be bound by your learning.
:ie rtther musicians S.H.: I found that I cannot tell who, among my students, are going to be the most
::rqv of that music. successful later on, who you are going to read about in the magazines. Every one of
Lri course. I heard them is a potential master. There are limits. What teachers can do-what I try to do-is
tL. .he cymbals were to present musical postcards. Do you want to go here? Or go here? I can help you
ral1. I had to move to do that. Even if you never play drums again, you can use the process in other
areas ofyour life.
: ,:r¿1ism in order to
.¡ ilat I am ivith by sit-
---
': ¿ lot of mistakes or RnrnnnNcns
-:. I need to listen as.a
Aslam N (zoo8). The homesick restaurant. The New York Times, November 28.
, :; s,rpportive, to deal Cooke M (199il. Jazz. Thames and Hudson, London.
.rair-se, break things Hadden S (rSS:). Woild Fusion Drumming. Alfred Music Publishing Los Angeles. CA.
Hadden S (rSSf ). The Beat, The Body, 6 The Brain /L Who's Counting? Gloucester, MA.
..'.:rple, and they are
Hadden S (zoo5). Prof.Ies in Jazz Drummlng. Who's Counting? Gloucester' MA.
¿::d at Thanksgiving,
Kirchner E (uoo5). The Oxford Companion to Jazz. Ortord University Press, New York.
:-¡.t makes people lose Limb Cl and Braun AR (zoo8). Neural substrates of spontaneous musical performance:
An fN4RI strdy of jazz improvisation. PloS ONE 3, eú79.
Shipton A (zooil. A New History of lazz: Revised and Updated Edition. Continuum,
, ,:r cognitive science New York.
r-,'nore attention and
Plate 2 Ano'
Plate 1 Jane wildgoose, vanitas still Life at the wildgoose Memoriat Library,2oos.
(See Fig. 1)
(photo: courtesy of the artist)
3))
'.' :
".' JSeurn, Rovereto)
Plate 8 Brain regions that increased selectively at cuts that were identified as
event boundaries (blue) or points in time that were discontinuous in space or
time but were not identified as event boundaries (red). Data from (Magliano et al.
2oo7). (See Fig.21.a)
C nema
;r- movies of
':56). Data from
; 'c icate the
AI-]TTNTTN A
I P. 0.0005 (Corr.)
B
Posterior C¿lc¿' -. -
p.
!*vX
I o.os (corr.)
Plate 9 Neuroimaging data of subject EA for drawing and scribbling objects. Plate 10 Neurcim¿: -:
A) Example of behavioural data collected from a scanning run. Subject E.AJs versus tactilely ex: -- -
sketches (above) are shown compared to the object tactilely explored or in objects is presente: :-
response to the control scribble condition (below). B) Activation patterns and medial views). '.
associated with the contrast of drawing versus scribbling presented on a full occipital areas, Ac: .
inflated cortical reconstruction of E.Als brain (lateral and medial views). Most calcarine sulcus. Sr. , .
striking is the activation seen in the occipital areas localized around the calcarine from this area, Tl"r : -:
sulcus (identified by cyan arrows). The calcarine sulcus anatomically corresponds objects (red), menra
to early visual areas (V1 and V2) in normally sighted individuals. ln EA, these control (orange), ¿'. - .
areas are active for drawing. (See Fig. 23.2)
Adapted from Amedi et al. (2008).
A Draw > Touch
B
Poster¡or Calcar¡ne LH Anter¡or Calcarine / PO Sulcus LH Lateral-Occip¡tal LH
t.
P
2
l
1
t'
,
l-__
,'\,
I
,l__
,,"' \
:' :: ing objects. Plate 10 Neuroimaging data of subject E.A. for a direct comparison of drawing
--. Srbject E.Als versus tactilely exploring objects. A) The contrast of drawing versus touching
:'¡ ¡red or in objects is presented on a full inflated cortical reconstruction of E.A.'s brain (lateral
': -' patternS and medial views). Note again the robust activation in several posterior ventral
:.:':ed on a full occipital areas. Activation for drawing was strong and selective even in the
.:: views). Most calcarine sulcus. See also (B) for the time courses and magnitude of activation
::'l,nd the calcarine from this area. The time courses are depicted by the following colour index:tactile
:-:al y corresponds objects (red), mental imagery (brown), drawing (blue), scribbling (cyan), motor
:-... ln EA, these control (orange), and naming/verbal memory (green). (See Fig. 23.3)
Adapted from Amedi et al. (2008).
Piano
Sine
String
e¿
--F -J-
--- Plate 14 Merce C'
figure of a dance'.
Plate 12 An example of a synaesthetic participant's choice of colours for
10 different musical notes that were either pure tones (sine wave), or from piano
or strings. The synaesthete chose each colour on two different occasions and the
Figure shows that similar colours tend to be chosen over time. (See Fig.2a.2)
F'
Animation Machine'
Plate 13 St¡ll taken from an animation created by the Music
Sebastian Bach's organ work
representing the first ten bars of Johann
tn Dutci Jubilo. (See Fig. 2s.7)
(with permission from Steven l\4a inowski' www musanim com)
b1-
,bL
CARO
MUSICAL TENSION
INrnopucrroN
THn arts offer a rich and largely untapped resource for the study of human behav-
iour. This collection of essays points to the potential of studying the various art forms
from a variety of disciplinary perspectives. The present essay describes a project
involving a music theorist-composer and a cognitive scientist who conducts empir-
ical research on music.
From a psychological point of view, the arts can extend our understanding of the
physical, sensory, and perceptual processes that enable us to apprehend the world
around us. Art works are special cases that, at a somewhat higher level, can yield
insights into processes such as memory and attention, and show how knowledge
shapes our interactions with, and interpretations of, artistic objects. At a still higher
level, they raise questions about emotional, social, and cultural factors at work in the
aesthetic realm. Finally, perspectives from neuroscience, genetics, and evolutionary
theory illuminate questions about the underpinnings of artistic creativity.
Over the last 30 years, the cognitive science of music has become a mature field
with different branches. One important impulse has come from the intersection
of music theory and experimental psychology. Empirical studies have demon-
strated that music theory provides a description not only of structural regularities in
music, but also of the way music is perceived, organized, and remembered by listen-
ers. Descriptions of music employ a set of terms such as scale, chord (harmony), key
(tonality), melody, and melodic contour. These terms are defined in the glossary at
the end of this chapter (Thompson 2oo9; see also Melcher andZampini, Chapter r4,
this volume).
372 CAROL L. KRUMHANSL AND FRED LERDAHL
With the development of music cognition, these questions have moved from well- music produce emo:i : :-.,
established theoretical concepts such as harmony and tonality (see, for example, attention to the ir a\ '-- - .
Krumhansl ry9o, CognitiveFoundations of MusicalPitch;abbreviated CFMP) to newer it asks what pa{tenr. - -
music-theoretic proposals (Narmour r99o; Lerdahl zoor: Tonol Pitch Space, abbrevi- becomes of intere.: .-
ated TPS). For these, empiricai research acts not only to test a theory but also to sug- the music, rather th:: -.
gest modifications that more accurately describe how the music is perceived. Before that are engendereJ b :'-
we present one such theory and one of its tests, we provide some background on how The essential ,l¡--:.
this project relates to more general concerns. the listener actitelr' :: .:
tions for what is ro
he does not claim th::
styles, sometimes c:,,.
The¡e are patterns rr-.:
One line of argument, from psychoacoustics, is that music4l- gou-nds are inher-
upon its relationsh.'¡. .,
ently pleasing because of the intervallic relationships th¡ po{uce consonant sounds.
p. 54). Empiricai .tu:,., :
Another position is that music has an emotional effect because it has been paired pre-
that appear with h,¡:: :
viously with significant events in an individual's life. Yet another is ihat music has
et al. 1984; Kruml¡:.: .r
characteristics such as tempo, amplitude, and contour that are also found in emo-
probabilities fun;. -
iional qpeech. Still anothei ihread qpeculates about how music might have evolu-
Temperley zoo; .
' Gestalt principles of perception describe our tendency to group elements so that they provide
'good form', including principles such as grouping by similarity or proximity. Many of these
principles apply in both visual and auditory modalities and may interact in audiovisual perception.
' Information theory, developed by Claude Shannon (A Mathematical Theory of Communication,
r9+8), is a set oftheories and principles developed to describe the transmission ofa message. These
ideas, which came out of work in Bel1 Labs during studies of the transmission of signais over
telegraph or telephone wires, have been expanded upon and used in a variety ofapplications, ranging
from psychology to data compression (such as MP:s). Fig. 16.1 The first pl.ras
MUSICAL TENSION 3I3
:.r', : :tlLrved from well- music produce emotions?', to 'How does music produce emotions?' This step draws
:, see, tbr example, attention to the way music is constructed. The question thus becomes more focusedl
,..,-; CFIÍP) to newer it asks what pa,tterns in music pr-oduce emotion. Also, the perceptual response that
-..
,':.:ii Spcce, abbrevi- becomes of interest is the moment-to-moment experience of the ongoing flow of
-...:,--r but also to sug- the music, rather than generalized 'real-life' emotions such as happy, sad, and tender
.¡ -, :erceived. Before that are engendered by passages or entire pieces of music.
:. ::;kqround on how The essential claim of Meyer's theory is that music produces emotions because
the listener actively responds to music (mostly unconsciously) by generating expecta'
tions for what is to follow. Meyer emphasizes three sources of expectation, although
he does not claim that they are exhaustive. The first is the listener's knowledge of the
styles, sometimes called extra-opus knowledge (Meyer ry56, t989; Narmour r99o).
The¡e are p.4-tterns lhat are characteristic of a musical sryle, as the first phrase of the
TATION familiar song'Over the !.ainbow', shown in Fig. 16.r, will illustrate.
Expectation as extra-opus knowledge is exemplified in bars 7-8 by the V-I (or V'/-I)
cadence in the tonic key of C major. The pattern L . . V-l appears in a large percent-
age of phrases in tonal music. A further extra-opus feature of this phrase is its internal
gro.Jp¡lg structure, shown by the brackets beneath the music: two bars of statement
',, .'"' ¿nd that is most
and two bars of counterstatement, followed by a four-bar continuation and cadence.
'.:: lhls is a question This so-called'sentence form' is common in a variefy of tonal styles.
'. -,,:ic¿1 science, evo-
-., For knowledge of this sort, Meyer draws on information theory and its use of
::.:arch concerned
probabilities. 'We have stated that styles in music are basically complex systems of
probability relationships in which the meaning of any term or series of terms depends
--.- s.runds are inher-
upon its relationships with all other terms possible within the style system' (Meyer r956,
-- : :,-r:lsonant sounds.
p. 54). Empirical studies demonstrate that listeners can use information about patterns
.. -,:. been paired pre-
it.rut upp.á, with high probability to orient themselves to a novel style (e.g. Castellino
:,.:: 1s that music has
": .i.o tound in emo- et al. 1984; Krumhansl et al. t999,2ooo), and recent music theory has discussed how
probabilities function in learning and recognizing musical patterns (Huron zoo6;
,,- :right have evolu-
Temperley zooT).
'-: :lhance courtship The second source of expectation comes from the experience of a particular piece
. l, l:dison, Chapter r7,
of music, called intra-opus knowledge. These are expectations that depend on the
^:.s :11 these elements
characteristics of that work. Even on first listening, a listener might expect that if a par-
ticular motive or theme appears early in the piece, the same motive or theme will recur
:.1.',:i s Errtotion and
'.-.^-::¡d considerable
:. -';: important theo-
- a.;,'. : : ttt i o n' and infor
: :-L.-ition) 'Why does
"(lvcr thc Rainboú''
¿h
: i : -r¿i ther. provide
.,1:::r ofthese
: -:- r lsual perception.
. - :- Conlfirllnication,
counterstatement
- " message. These
.-.:als over
.:-: p-ications, ranging
Fig. 16.1 The first phrase of 'Over The Rainbow'.
3T4 CAROL L. KRUMHANSL AND FRED LERDAHL
later in the piece, or reappear in variation. Figure 16.r illustrates intra-opus knowledge
of 'Over the Rainbow'by its motivic structure, shown above the music by the labelled
brackets. The succession'a' to 'b' in bars l-2 creates the expectation that this pattern
will follow later in the phrase. After the varied repetition of 'a' in bar 3, however, the
whole note G in bar 4, marked'c,'belies this expectation. Bars 5-Z compensate for this
move by stating 'a' again and following it by two 'b's' before the music comes to rest in
bar 8 on'c'.
Meyer's third source is the role that Gestalt principles of organization play in gen-
erating expectations. These principles are assumed to function in all individuals, not
based on specialized training, and only rarely influenced by prior experience. Some
consequences of these principles are that a simple well-organized melody will be
remembered better than an irregular one, that a pattern will be expected to continue
in a manner similar to how it began, and that a large pitch jump will lead to an expecta-
tion that the next pitch will fill in the gap. In this tradition, Narmour (r99o) proposed
five principles for melodic expectations that have been tested using a fairlywide variety
of musical styles and listeners in different cultures (e.g. Krumhansl 1995; Thompson
and Stainton 1998; Krumhansl et aL.1999, zooo).
'Over the Rainbow' is a classic example of a gap-fill melody. Figure 16.r shows this
by the small notes in the top staff, which strips away relatively ornamental pitches to
reveal the phrase's melodic skeleton. After the opening upward leap from C to C an
octave higher, the melody slowly descends down the C major scale until the original
C is reached.
According to Meyer, these various kinds of expectation lead to waves of tension
caused by the relationship between the expectations and what actually occurs in the
music. 'Thus in a very general way expectation is always ahead of the music, creating
a background of diffuse tension against which particular delays articulate the affective
curve and create meaning' (Meyer 1956,p.59). If the expectation is fulfilled, no strong
response occurs. However, when the expectations are not fulfilled, this produces a
feeling of tension. Or, in the words of the then contemporarybehavioural psychology,
'emotion and affect is aroused when a tendency to respond is arrested or inhibited'
Tr
(pU). The feeling of tension is not necessarilynegative, nor is the feeling of resolution
necessarily positive. Rather, the response depends on the particular way expectations
are fulfilled, perhaps in an especially artful way or through an unexpected delay.
:
Trln TENSToN MoDEL -t E-
Complementary to Meyer's approach is another tradition, that of pitch attractions,
articulated in psychological terms by Bharucha Q984, t996) and developed in music-
theoretic models by Larson (zoo+) and Lerdahl (t996, zoot). According to this tradi-
re
tion, which in music theory can be traced back to the early nineteenth century, some
pitches or chords seem to want to go to other pitches or chords, such as the leading tone
MUSICAL TENSION 3r5
.:ir¿-opus knowledge to the tonic pitch. This tonal urge elicits an expectation: a dominant seventh chord
. ::rusic b1.the labelled is expected to resolve to the tonic chord, for example, and motion to another chord
::rrrn that this pattern is felt as a denial of that expectation. A strong attraction-expectation can also be
-tar
::. _i, however, the expressed as a kind oftension.
-- . r'rlltP€Its&te for this Thus each event in 'Over the Rainbow' generates some degree of expectation felt
:- ...rc COmeS tO reSt in as a kind of attractive tension. The strongest attraction in Fig. 16.r occurs inbar 7 atY7,
which 'wants' to resolve to the tonic I. Specifically, the pitches D and B are strongly
,-- ¿:rion play in gen- attracted to C, F is strongly attracted to E; and G is strongly attracted to C, expressing
- ' ¡1. rndrviduals, not the urge for V7 to resolve to I. These resolutions indeed take place (with the B surrep-
:-,,: e\perience. some titiously displaced to G).
: 'e¿ melody will be Figure 16.2 illustrates another kind of tension based on tonal stability and instabil-
: ..,::eated to continue ity. The tonic is stable and relaxed, whereas a pitch or chord distant from the tonic is
,., - j ad to an expecta- experienced as unstable and tense. Suppose, for the latter case, that 'Over the Rainbow'
^e
: r: r99o) proposed resolved away from the tonic, creating tension through instability. In Fig. 16.2, the
:: .: - ¿rrlr. rvide variety melody moves to C, as before, but C is supported by an unstable harmony, V'/lV (that
:,:r.i 1995; Thompson is, the D-F#-A-C form a dominant-seventh chord in thekeyofthe dominant, G major).
Such a'deceptive' cadence away from the tonic would create tension and the expecta-
: :--ire 16.r shows this tion for a further continuation that would bring the expected tonic close.
::--::¡re ntal pitches to Intuitions of tension based on attraction and stabilityboth depend on the structural
- ;;r trom C to C an schemas of tonality uncovered in CFMP and related empirical studies. TPS provides
:.-: until the original a music-theoretic account of both. The schemas address a specific, learned level of
cognitive organization. At present, it is an open question as to whether these structural
: :, ,,ra\-es of tension schemas can be learned through extra- and intra-opus probabilities as proposed by
.--: ::.Lh' occurs in the Meyer.
: :::: music, creating
-.:'- ¡ :1¿rte the affective
: .. :uiñlled, no strong
----1. this produces a
. .:', -r¡ürdl psychology,
TrrnoRY AxTD ExPERTMENT
.i:;sied or inhibitedl
. :¡:,ing ofresolution
:-:: -'r-ar-
expectatiOnS We have collaborated in an empirical study (Lerdahl and Krumhansl zooT) to test
:.,---.ected delav. this double account of tonal tension. It is important, in this connection, that the
: :rtch attractions,
.,-
: ::','eloped in music-
-: -:line to this tradi- Y7 V|VT
::.::th century, some
i... ,rs the leading tone Fig. 16.2 A deceptive cadence for the 'Over The Rainbow' phrase.
3T6 CAROL L. KRUMHANSL AND FRED LERDAHL
TPS model makes quantitative predictions, so that these can be compared with CFMP summarizes e.'.:.
experimental observations, which most often take quantitative form. Because of the chords,andkers.l-..
variability of human behaviour, the quantitative data need to be processed by statistical a range of music¿l - -
techniques to uncover the underlying regularities and assess whether these regular- instructions.
ities conForm to predictions. One empirical r.. -.'
In general terms, the stability-tension theory creates a tree-structure that describes files of Krumhan.- :--.
the hierarchical structure of the music-it is somewhat like diagramming a sentence experiments loun.l , -
in terms of nouns, verbs and adjectives. It builds, first, on what is called the prolong- and third degree. .' -
ational clmponent in Lerdahl and lackendoff s Generetive Theory of Tonal Music scale tones. TPS p: - -
(1983; abbreviated GTTM) and, second, on the theoretical and empirical measures the basic space. sh, . -
of distances between musical elements (tones, chords, keys) described in CFMP. the levels of pitch. --
These two factors combine (together with two other theoretical components briefly fied formalism.
described later) to produce precise predictions about the relative tension and relax- These two colr'.r -:'
ation of each event in a musical passage. This is a complex theory, and only a few of operations on th¡ r - ,
its essential elements are presented here; but it is important to note that musical com- alional tree for ; -:-i'
plexity necessitates theoretical complexity and that empirical methods are available principles or rule.
to test such subtle and precise theoretical predictions. are computed. T-. :
GTTM considers prolongational organizalion as a psychological phenomenon superordinate to:. -
that consists of nested patterns of tension and relaxation. These nested patterns are computed fronl F :' - -
depicted in an inverted tree notation, illustrated abstractly in Fig. 16.3. Right branches distance is the sl .-
stand for a tensing motion (or departure from stability), left branches for a relaxing For instance, Er r:. -
motion (or return to stability). At a local level, Event 1 tenses into Event 2, Event 3 the distance lion- : '
-
relaxes into Event 4, and Event 5 relaxes into Event 6. At a higher level, Event I tenses th e tree-th irt h -: :. . - .
into Event 4, Event 6 relaxes into Event 7, and Event r relaxes into Event 7. associated with it,
Added to the tree structure is a method to compute distances between any pitch These distanc.': ::..
or chord in one key and any pitch or chord in the same or any other key. The hypoth- one is a treatm(:rr , -
esis is that greater distances between events contribute to greater experienced tension. acoustic: the int::-. -.
increases disso-u::'-- .
component in tl-: - -
Listenersexpericr.-: . :
especially nlorc ).:: .
!a
t;
It
1.4
Fig, I €
____L______>
----+ --r-...-.> C ma :'
___--L_-__+
Theoreticallr toti....-'
Fig. 16.3 Hypothetical tree structure showing tension (t) and relaxation (r). le.g. Krumhan'l and h. .. ,'
MUSICAL TENSION 3r7
1r: te compared with CFMP summarizes experiments investigating the perceived distances between tones,
: i'-rrm. Because of the chords, and keys3. These results have been replicated in several laboratories, using
: :o cessed by statistical a range of musical contexts, participants with varied training, and different task
,\ :.ljler these regular- instructions.
One empirical result of interest in connection with the TPS model is the tone pro-
¡:r.:ure that describes files of Krumhansl and Shepard (tglg) and Krumhansl and Kessler (1982). Those
::::mming a sentence experiments found a hierarchy of tones headed by the tonic, followed by the fifth
: -. called the prolong- and third degrees of the scale, then the remaining scale tones, and finally the non-
:'.:::'" t'¡.f Tonel Music scale tones. TPS proposes a similar, theoretically motivated, tone hierarchy, called
.i :ntpirical measures the basic space, shown in Fig. 16.4. The basic space and its transformations represent
::s.ribed in CFMP. the levels of pitch, chord, and key and their distances within a unified and quanti-
., c..:nponents briefly fied formalism.
,-, :ension and relax-
¿ These two components of the TPS model, the prolongational tree structure and
¿nd only a few of
, :-,-. operations on the basic pitch space, are combined in the following way. A prolong-
.. :¡ that musical com- ational tree for a segment of music is constructed based on a number of precise
::thods are available principles or rules (GTTM and TPS). This tree structure determines which distances
are computed. The distance of any given event is computed from the event that is
, -, :icai phenomenon superordinate to it in the tree. For example, in Fig. 16.3 the distance for Event r is
:r; :eSted patterns are computed from Event 7. When an event has two or more superordinate events, its
: :1._r. fught branches distance is the sum of the distances along the paths leading to the root of the tree.
:l:r.jles for a relaúng For instance, Event z's distance is first computed from Event r, and then it inherits
l:-:Ll f1'g¡¡ 2, Event 3 the distance from Event 1 to Event 7. Generally, an event that is deeply embedded in
.: -:', ei. Er,ent r tenses the tree-that has many superordinate events-will have a large tension value
-:..ant,. associated with it.
:.¡ .-¡lrr-een any pitch These distances are combined with two other components of the theory. Briefly,
: :¿¡ -<er-. The hypoth- one is a treatment of surface or sensory dissonance. This measure is largely psycho-
: .r,-:e rienced tension. acoustic: the interval of a seventh is more dissonant than a sixth; a non-scale tone
increases dissonance, as does a chord not in root position; and so on. The other
component in the model is the tension caused by attraction, as mentioned earlier.
Listeners experience the relative pull of pitches toward other pitches in a tonal context,
especially more stable pitches that are nearby. The factors of proximity and relative
stability together generate a quantitative prediction ofthe degree oftension caused by
----------->
r Theoretically consistent patterns have been found on these three levels of musical structure
re axat¡on (r). (e.g. Krumhansl and Kessler 1982, for tones and keys; Bharucha and Krumhansl 1983, for chords).
318 CAROL L. KRUMHANSL AND FRED LERDAHL
attraction for each event in the passage analysed. These are then compared with the
judgements of tension that listeners make in the experiments.
To study the rise and fall of tension, real-time measures have been developed in
.<'
,/// //:
which listeners move a device to indicate the amount of tension they experience Events: 1 I :
throughout the course of a piece (e.g. Nielsen t983; Fredrickson 1995; Krumhansl
r996). In some preliminary results (Krumhansl 1996) these judgements were found to
be quite consistent across listeners, independently of musical training and of degrees
of familiaritywith a piece.
We undertook a series of experiments to provide a quite comprehensive test of the
model's predictions over a range of musical styles. The participants made tension e
responses for Wagner's Grail theme from Parsifal in both its diatonic and chromatic Et
versions, a Bach chorale, Christus, der ist mein Leben, Chopin's E Major Prelude, and
the opening of the fifth movement of Messiaen's Quartet for the End of Time. (The Fig. 16.5 Analysis :" :
beginning of the first movement of the Mozart Piano Sonata, K. z8z was also anal-
ysed similarly; see Krumhansl 1996; Lerdahl 1996.) We present here only the exper-
iment using the Chopin'Prelude;'the other experiments are described in Lerdahl and
Krumhansl (zooz).
Chopin's E major Prelude was analysed in Chapter 3 of TPS. It divides into
three phrases corresponding to Figs. t6.5-t6.7.It is a highly chromatic nineteenth-
century piece; the degree of chromaticism is reflected in the number of sharp, flat, and
natural signs in the score, especially in the second and third phrases. As such, it pres-
ents analltic and perceptual complexities that make it an interesting test case. One
complexity that we decided to avoid in the experiment is the textural ornamentation
in the melody. Thus, the prelude was presented in block chords as shown. The chords
were presented at a slow rate of one chord per two seconds, giving listeners ample
time to respond to each event. ,/:
Following are some technical details ofthe TPS model applied to the Chopin Prelude.
Readers may prefer to continue reading from the 'Summarizing the findings' section. 11 ts _
: -:'.1 oiTin-re. (The Fis. 16.5 Analysis of the first phrase of Chopin's E major Prelude.
i i:
rtas also anal-
-
- -:-
. !rltl\' the exper-
, - :. -- -,i in Lerdahl and
- lj S It dir,ides into
.'.: ::::lic nineteenth-
..-
:r ,ri sharp, flat, and
t8[16]
..:::. -\s such, it pres-
::r.-rl'q test case. one
-. --::i ornamentation
:: ::Lr\\'11. The ChOfdS
: -:r. listeirers
".
ample
:.. :ChopinPrelude. /
. . ir.rdings'section. 9 j2
'.F
P LIED
E ¡1# r-ii7 ivó_4 viil
Ab: ii6-1 viiT
. ::r..-tion. Events are Fig. 1 6.6 Analysis of the second phrase of Chopin's E major Pre I ude.
- ::-,.. rr'hich refers to
CAROL L. KRUMHANSL AND FRED LERDAHL
prolongational cont: .
one indicated bv ¿ :,
with the TPS mod-l -
respect to the preci- :-
between the aitern:t: , ,
rz; that Event z8 is --:.
tension); and that .'...
tonicprogression . -..
Díss.:l 0 I l Iqtl 10tt 020 formulation of hc.',,.. ,.--,
lht¿t:l ) I / l9 23 19 24 29 34 36 6(1 020
Attr.: 1.4 4.85 2.'72 2.97 t.4 4.85 t.'12 0.93 2.4 4.85 t.24 4.85 0 -------------
The sum of these distances is 7+ro+r8+o = 35, to which a dissonance value of r is added
giving a total of Tn = 36.
",
Before considering how well the predictions fit the tension judgements made by
listeners, one aspect of the predictions should be mentioned. At three places in the
prolongational component (Events 73,28, and 44) two alternatives are indicated (with
one indicated by a dashed line in the tree). These alternatives are both consistent
with the TPS model. In these cases, the question is whether the event is heard with
respect to the preceding event or the following event. The data can be used to select
between the alternatives. The data suggest that Event r3 is heard with respect to Event
rz; that Event z8 is heard with respect to Event z9 (giving a sharp peak in predicted
tension); and that Event 44 is heard with respect to Event +S (the familiar dominant-
tonic progression). In this way, theory and data interact resulting in a more accurate
formulation of how the TPS analysis corresponds to the perceptual data.
i----------
SuvrvrnRrzrNc THE FTNDTNGS
Figure 16.8 plots the predicted tension (dashed line) and the judged tension (solid
line). The disparity at the beginning of the piece reflects the fact that the slider listeners
use to make the tension judgements is initially set to o, so it took a few events before
r -r ihe hierarchical the listeners' judgements reached the level of the predictions. It should also be noted
::.: plus dissonance,
:,r: ro the successive
-. ;hromaticism, the
- :: rEressions refer to Chopin prelude
: -il.o, as can be seen, 125
! . rLrm E major to C
100
-:- :::e b¡anches of
the
.l: r attaches to Event
! -:, E\-ent r7. All these
::rem and zero sums
- r. rs added the disso-
- , rhe chord is not on
:r-¡ rrith the presence
'
..:rsion for the begin-
1 3 5 7 9 '1
1 13 15 17 19 21 232527 2931 33 35 37 39 41 43 45 47
r: ihe end of the final Event
*:.! to Event 41 (with ---.Predicted
-, -r- attaches to Event
: :_. rr'ith distance o). for the TPS analysis of the Chopin Prelude.
Fig. 16.8 Tension graph -Judged
322 CAROL L. KRUMHANSL AND FRED LERDAHL
that the scale of the predicted values in this graph is adjusted so that it matches the
tension judgements as well as possible. Nonetheless, it can be seen that the predictions
Drsc
for the second phrase are generally below the judgements and the predictions for the
third phrase are generally above the judgements. The degree to I'hic¡ :..
This discrepancy suggests that the model is missing an important factor. Observe itprovides a lars¡ . .
that in the first two phrases the judged tension rises and falls in waves that correspond sounded events ar_ü '.
to the rise and fall of the melody (its contour). In this case the melody consists of tension. ln part h'1...
the highest tones. Although the harmony here is complex, the shape of the melody continually van'in:- :.
is simple. To add the factor of melodic contour to the analysis, we quantified it and TPS model that nl¿, . ,
used it as an additional predictor to the statistical analysis. The results are shown in prolongational colr:
Fig. 16.9, with a clearly better match between prediction and data (which is substanti- The tree rrprcs;-'-:-
ated by statistical tests that evaluate how well the data match the predictions).a events that are nol l: .
This example illustrates the process through which listeners' experience of ten- each event not on . .- .
sion and release of tension can be modelled precisely in terms of a specific music events at a distan;r ..
theoretic model. The other examples we tested each raised a number of interesting encies and are hcLl :
issues that arose in the interaction between theory and data. Various alternative hier- (Chomsky r95-: H.. ..
archical analyses were tested, leading to a better understanding ofwhich choices, within results are neutr.rl ' -.-
the constraints of the TPS model, more accurately describe listeners' responses. unique to hum¡n. -.-.:
ical organization.
Iust as peoplc .:. -
Chopin prelude
though they imp. . .
it is likely that lrir. _ -.
125
feel the tension: ur-._ : _
In an exploratory study with real-time measures, judgements of tension in the may be different or ::,-rl
study of Krumhansl Q997) correlated with changes in emotion physiology over time. the musical experienc;.
In addition, the tension judgements correlated with real-time judgements of fear,
but judgements of huppy and sad also made contributions. Changes in emotion
physiology showed a similar pattern. This suggests that tension is a multivalent qual-
ity with contribuiions of different reported emotions. Supporting this, another study
(Krumhansl and Schenck 1997) showed tension judgements were virtually identical
with judgements of the total amount of emotion. Krumhansl (zooz) summarizes
additional studies relating real-time measures to emotion responses. Thus, tension
does not map uniquely to a single emotion (see essays by Melcher and Zarrrpini, Chord: in a music¿r :
Chapter r4, and by Madison, Chapter 17, this volume). taneously or nea: s,::l
A growing number of studies using measures of neural activity5 have shown major and mino: ;:.,
brain responses when musical expectancies are violated. For example, a functional Chord (or harn.-:,:;
magnetic resonance imaging (fl\4RI) study by Koelsch et al. (zool) found responses the key. The mos: .:;
to unexpected chords. In another fI\4RI study, areas ofsecondary auditory cortex were chord, I), foiloi' e: :'
active when listeners heard violations of expectations for pitch and rh¡hm (Krumhansl V), and then the .-.. :
zoo5), evenwhen the listenerswere notperforming a task, suggestingthatthe responses are also usualL,,' t:.
to expectations are automatic. Studies using electroencephalography (EEG) (Besson sequences of chrr::j :
and Faita 1995; Janata 1995) have found early event-related correlates of the degree Key: a concept d.i::-
to which expectations are violated, a result replicated in other studies (e.g. Koelsch determined br- t}:¿ ..
et al. zooo). Melody: the me-,-,:,
Some studies also suggest that music can activate the same brain structures that ized accordinE ,. :
are involved in emotional responses in other situations. In the Koelsch et al. (zool) remembered.
study cited earlier, the unexpected chords elicited orbital frontolateral cortex Pitch: the psvch,.-- .-
activation, an area known to support emotional processing (see Calvo-Merino and (pitch height . I: ::..
Haggard, Chapter 27, this volume, for the role of this brain region in 'rewarding' it is called a /1¡;r.
behaviours such as listening to music, looking at preferred visual artworks or even Scale: the set oi t, .,
eating chocolate). Blood and Zatorre's (zoor) PET study showed brain responses or all tones in t1--. ::.
at specific listener-identified moments of strong emotions, as well as other physiolog- the major (ser e r-: ::
ical changes. And, finally, a recent study (Steinbeis et al. zoo6), found tension, Tonality: the qu:,:,,
subjective emotionality of the music, early negativity event-related potential (ERP) involves sensiti-, -., .
response, and electrodermal activity (EDA) increased with harmonic unexpectedness, ships betrveen i. ir:i
thus demonstrating connections between all four kinds of responses. Tonal centre I or : :
In summary, empirical evidence, using a variety of behavioural and neurocog- and minimum te:r:.
nitive measures, strongly supports the idea that listeners develop constantly chang- name to the scale . . ¡
ing expectations while listening to music, and these give rise to waves of tension and Tonal hierarchr': ::-. -
relaxation. Here we described one music-theoretic model that is able to identify stable tone in a ::.'-
musical patterns that give rise to tension. At present, however, it appears that these fifth tone of the s..-,
moment-to-moment responses do not map in a simple way on to the traditional tones that are noi -rl
emotional states studied within psychology. Although the emotions evoked by music
t Techniques to indirectly measure the activity of groups of neurons in the brain include EEG
(including event-related changes in EEG activity called ERPs), positron emission tomography (PET), " The following der,r.: -
and f\4RI. zoog) which can be .¡r. - -,
MUSICAL TENSION 325
lrrrs of tension in the may be different or more complex than 'real-life' emotions, they are fundamental to
:hr-siology over time. the musical experience.
e iudsements of fear,
fhanges in emotion
Ls ¿ multivalent qual-
.- :oo:.) summarizes
GrossARY6
lL-,nses. Thus, tension
1¿lcher and Zampini, Chord: in a musical context) a chord is a collection of tones that are played simul-
taneously or near simultaneously. The most common chords in Western music are
¿;ririn-¡ have shown major and minor chords (called triadsbecause they consist of three tones).
: ta:nple, a functional Chord (or harmonic) hierarchy: the hierarchical stability of chords established by
.: -,5 tbund responses the key. The most stable chord is the chord built on the first scale tone (the tonic
'-- :.'.Lditon' cortex were chord, I), followed by the chord built on the fifth scale tone (the dominant chord,
I r}:rthm (Krumhansl V), and then the chord built on the fourth scale tone (the subdominant, IV). These
i-rg rhat the responses are also usually the most frequent chords in a musical composition. Common
¡r;ph1. rEEG) (Besson sequences ofchords that mark the end ofphrases are called cadences.
,::.,¿tes of the degree Key: a concept describing the organization of a passage of music. A sense of key is
: .:udies (e.g. Koelsch determined by the establishment of two features: a scale and a tonal centre.
Melody: the melody is the change in pitch in a passage of music. If it is organ-
: ¡r¿in structures that ized according to musical conventions and perceptual principles, it is well
: lioelsch et al. (zoo5) remembered.
;ir¡¡rtolateral cortex Pitch: the psychological quality of periodic sounds that extends from low to high
¡¿ Calr-o-Merino and (pitch height). In the context of music it is called a toneandwhen it is written down
:¡Eion in 'rewarding' it is called a note.
su¿l ¿rtrr'orks or even Scale: the set of tones that represent a musical composition, and from which most
:,',.,.eC brain responses or all tones in the music are drawn. In Western music, the most common scale is
e-- ¿s other physiolog- the major (seven-tone) diatonic scale.
: ,-,ó , tbund tension, Tonality: the quality of music that involves the use of a key or keys. Its perception
:-::ed potential (ERP) involves sensitivity to a complex system of relationships, including the relation-
ri rrc unexpectedness, ships between tones, chords, and keys and the tonal and harmonic hierarchies.
I1:eS. Tonal centre (or tonic): the tone that functions as a point of maximum stability
r¡,ural and neurocog- and minimum tension. It often appears most frequently in the music and gives the
-¡,: constantly chang- name to the scale, e.g. C is the tonal centre for the key of C major.
'f,'lr es of tension and Tonal hierarchy: the hierarchy of stability of tones established by the key. The most
r¿: is able to identiSz stable tone in a major key is the first tone of the scale (the tonic), followed by the
". it appears that these fifth tone of the scale (dominant), and the third tone of the scale (mediant); the
,--n to the traditional tones that are not in the scale are the least stable.
-rr
r'ns evoked by music
Bharucha II (rSS6). Melodic anchoring. Music P erception, t3, 383-4oo. within the di¿to-.... -
Bharucha JJ and Krumhansl CL (rg8:). The hierarchical representation of harmonic structure Performance.5. i---- -
in music. Cognition, l_3, 63-toz. Krumhansl CL, Toi'..-,
Blood AJ andZaforre R] (zoot). Intensely pleasurabie responses to music correlate with activ- Cross-cultural n*,._
ity in brain regions implicated in reward and emotion. Proceedings of the National Academy Cognition,76, r_i-: l
of Sciences, 98, uBrB*23.
Larson S lzoo-1 . -\ l-. .
Fredrickson WE (rqqf). A comparison of perceived musical tension and aesthetic response. Narmour E rr99,
P sy cholo gy of M u sic, 23, 87-7. realizat ion rr;,;i. - .-
Hauser MC, Chomsky N, and Fitch WT (zooz). The faculty of language: What is it, who has it, Nielsen FV r rq¡-r .
Huron D (zoo6). Sweet anticipation: music and the psychology of expectation MIT Press, SchenkerHtrg..i -l- :
ing: 'Non-musicians' are musical. Journal of Cognitive Neuroscience, L2, 52o-4r. Cognit ive ,\'r ¡, r '.. . .
Krumhansl CL (t99o). Cognitive foundations of musical pitch. Oxfoñ University Press, Temperley D r:o, -
New York. Thompson \VF I i.
Krumhansl C.L (rqqS) Music psychology and music theory: Problems and prospects. Music OxfordUniver.- --.
Theory Spectrum, 17, 53-Bo. Thompson \t'¿nd i:- '
Krumhansl CL (rqq6). A perceptual analysis of Mozart's Piano Sonata, K. z8z: Segmentation, ol implicative ¡ri.. - :. .
:a L-niversity Press, Temperley D (zoo7). Music and probability. MIT Press, Cambridge, MA.
'-j,
Thompson WF (zoo9). Music, thought, and feeling: understanding the psychology of music.
:. ]:.i prospects. Music Oxford University Press, New York and Oxford.
Thompson W and Stainton M (rqqs). Expectancy in Bohemian folk song melodies: evaluation
' r. :E:: Segmentation, of implicative principals for implicative and closural intervals. Music Perception, 15, z3r-52.
.rJ lsr-chophysiology.
: .l:,rrett Directions in
CHAPTER r7
CAUSE AND
AFFECT:
A FUNCTIONAL
PERSPECTIVE
ON MUSIC AND
EMOTION
'SeealsospecialistjournalslikelournalofNewMusicResearch,Leonardo,MusicPerception,
Psychology of Music, and Psychomusicology. Other sources are occasional papers in Annals of the
330 GUY MADISON
music and emotion (e.g. Rigg rg6+; Budd rqal; Ahlberg rgg4; Scruton 1997; Juslin and effort put into crea:--.
Sloboda zoor; London zoor; Gabrielsson and Juslin zooz; Kir,y zooz; Scherer zoo4). the musical genius.
Although many different ideas about the relationship between emotions and
music have been proposed in the last z5oo years, there is yet no inkling of consen-
sus. Three related reasons in particular should be mentioned. Firstly, music is not one
thing, but a multifaceted phenomenon that may serve more than one function (Fitch
zoo6). A number of different levels of analysis may therefore be appropriate' such EuorroN
as the individual, cultural, and evolutionary. However, a literature that cuts across
different functions that music is assumed to serve, across mundane and peak experi-
ences, and across explanatory perspectives ranging from the humanities to the sci- These are terms i:.. .
ences, will necessarily be heterogeneous and unlikely to reveal any consistent patterns' consensus among :.:::
Secondly, the function of emotions is primarily to guide action. From this it would experientiai catesL-- i. ; :
seem to follow, as will be elaborated in the next section, that a plausible link between An emotion .. -: :.
emotion and music might involve adaptive benefits. However, the adaptive value involves ph\ sio.(.: -. :
of music is itself a difficult, controversial, and as yet unresolved issue. Since these biologically enrh: - -: :
issues are so intimately connected, researchers have likelybeen wary to commit them- which in turn l:l: ..
selves theoretically on one issue until the other has become more elucidated. Thirdly, modification hr ,.-- '
both music and emotions are prevalent in everyday life, and their close connection ment implies a lir-.: .:
with our individual lives may make it difficult to assume a perspective above and Plutchik 199-1;. ll :.
beyond culture, geography, and history. emergent sirua:i,. -.
Rather than review the literature on music and emotion' which has been done For example. a -,'- - -
repeatedly and competentlybefore (see references listed earlier), I will adopt a func- and increased h-,-. --
tional perspective on the relation between emotion and music, focusing on such the sound is asso.:.:.:
functions that may be considered general or important enough to account for the which takes con.iJ.. .:
ubiquity and universal nature of music. A main thrust of the functional perspective the sound. Thu, .-.. .
is to create patterns of predictions, and I will suggest that predictions concerning the to a potentialh' ¿::-,..:
type and content of affective responses differ with respect to the different functions completed. ThcIr .. .-: -
attributed to music. This can be helpful in assessing the plausibility of causal relations anger, fear, and ;-,:-'
between these functions, and for formulating empirical hlpotheses. The structure emotions(Plut¡:. --
of the chapter is as follows. I will first introduce the affective categories emotion, feel- be induced br s, ir..-
ing, mood, and ffict, and then review a few biological ond uniyersol aspects of music, one yesterdar' .lJ:-=,-,
thus setting the stage for discussin g the adaptive volue and function of music. Having (another likelr 1..-.- .:
considered function in relation to both emotion and music, I proceed to discuss demanding task r:r. .--
cause and ffict inthe context of music, and find a number of theoretical objections can be defined ii -r. :r
to the idea that emotions are the point of music. That affective responses emerge motor activin . ¡n, .::
from the motivational logic for engaging in music seems, in contrast, to provide the cf. KJeinginna ar:J K .
best agreement with empirical data and everyday experience. I conclude with some Feelings inciuic :--:
reflections on music experience and practice about the perceived meaning of music, the wide range oi evaLu..,',
It is conceivable r:.,: '-
selves as feelings. e i:'lt:
New York Academy of Sciences, Bulletin of Music Education, Cognition, Cognition and Emotion, unavailable or bc¡¡-.,
Cognitire Psychology, Computer Music Journal, Contemporary Music Review, Empirical Studies of the
them. One mar. li.: .:..
Arts, Interface, Journal of the Acoustical Society of America, Journal of Cognitite Neuroscience, Journal
of Experimental Psychology: Human Perception and Pert'ormance, Memory and Cognition, and
early, without beinE :
P erce pt io n 6 P sych o ph ysi cs. broad concept that :'.
CAUSE AND AFFECT 33r
,:rton 1992; Juslin and effort put into creating still new and more sophisticated music) and the rise and fall of
:- rr:: Scherer 2OO4). the musical genius.
:lt\ een emotions and
n,,. inkling of consen-
-:>th'. music is not one
ar: one tunction (Fitch
: be appropriate, such
:f:;re that cuts across
EvrorroN, FEELING, MooD, AND AFFECT
l¿ne and peak experi-
:'in¿nities to the sci- These are terms that tend to overlap somewhat in everydayusage. There is some
:r',. . Lrnsistent patterns. consensus among researchers, however, that they constitute distinct functional and
,::. From this it would experiential categories (e.g. Oatley and lenkins r996).
:l¿usible link between An emotion is relatively fast, brief, and transient, has an apparent cause, and
-:. the adaptive value involves physiological reactions and often also expressive behaviours. Emotions are
.r e.1 issue. Since these biologically embedded in terms of being, to some extent, built into the neural system,
rr'"1F,' tO COmmit them- which in turn implies that they have a direct adaptive value and are less subject to
:e elucidated. Thirdly, modification by cognitive processes. That they are part of our basic biological equip-
rh;:r close connection ment implies a limited set of so-called basic or fundamental emotions (Ekman r99z;
r.rsPecti\-e above and Plutchik 1994). Emotions work by invoking action or action readiness, especially in
emergent situations where prior knowledge or a rational solution is not available.
i,,'}::ch has been done For example, a loud, unexpected sound causes a start, a dilation of the sweat glands,
:.1-'rilladoptafunc- and increased heart rate, regardless of the source of the sound. Evaluating whether
..ir.. tircusing on such the sound is associated with danger or not requires cognitive evaluation of the event,
.:¡ to account for the which takes considerably more time than to just react in a preset, stereotypical way to
- ;¡r¡tional perspective the sound. Thus the organism is preparing for immediate fight or flight in response
--iLr:ts concerning the to a potentially dangerous event, even before an evaluation ofthat danger has been
-re .lilterent functions completed. There is reasonable consensus among researchers that happiness, sadness,
--:,-
'
oi causal relations anger, fear, and disgust constitute biologically distinct emotions, often called basic
-,i-1eses. The structure emotions (Plutchik 1994; Power and Dalgleish rygZ).Of course, emotions may also
.:¡gories emotion, feel- be induced by cognitive processes rather than by physical events. Remembering that
'.:.:úl aspects of music, one yesterday addressed a friend with someone else's name might induce shame
'.:::i)ri o.i t|1uslc. Having (another likely basic emotion), and realizing that one has successfully completed a
. - proceed to discuss demanding task might induce a sense of joy or happiness. In any case, an emotion
:h eoretical objections can be defined as an episode exhibiting coordinated changes in physiological arousal,
:i-,-e responses emerge motor activity, and feeling, according to the component process model (Scherer zoo4;
rrtrast, to provide the cf. Kleinginna and Kleinginna r98r).
- ctrnclude with some Feelings include the cognitive and conscious recognition of emotions, but also a
r:e aning of music, the wide range of evaluative and motivational sensations that are not emotions proper.
It is conceivable that many evaluations and other cognitive processes manifest them-
selves as feelings, either because the underlying processes or information is consciously
:::'. :t1 L1.nd Emotion)
unavailable or because one experiences a relatively high level of uncertainty about
: :':ttrical Studies of the
'-. : ){turoscience, Journal them. One may, for example, feel that something is good, bad, unnecessary, or too
':.r C,r.girlrlon, and early, without being able to account for why this is. This means that feeling is a very
broad concept that may cover the range from biologically based sensations, such as
):t z GUY MADISON
excited, tired, andbored, to more cognitive ones such as confident, solemn, suspicious,
primates and mammals
triumphant, ugly, or even Machiavellian. lack the ability or the n -
Affect is commonly attributed the sensation of valence, in other words an evalua- All tonal systems fearu
tion on a good-bad or positive-negative continuum. There are numerous studies indi- octave (.f = .fzt'') are ct
cating that valence is the most common and general dimension applied by humans. generally, to be more sLr
This makes sense from an adaptive perspective, since virtually ever¡hing may effec- the fact that we also De :
(Wright et al. zooo ¿:-.
tively be valued as good, bad, or neutral with respect to survival and reproduction
(cf. Batson et al. ry92;Oatley et aI. ry96). scales in different cuLt:r
Moods are diffuse affective states. In contrast to emotions, moods are slow, per- the order of relatir el-.' .=
(Burns 1999). These :r: -.
sistent, and have often no apparent cause. One could argue that moods may reflect
based on information-:
endocrine levels, or be the outcome of a cognitive appraisal of persistent situational
(e.g. Balzano r98:.1 T. - -
factors, such as unemployment, good or poor social relations, and so forth. Mood can
probably be induced by anything that induces emotions, feelings, and affects, although within the octave, rrh:;:
some factors more than others are more likely to play a role. Functionally, one can
An even more fun;¡:
those of human IanE-: :.=
say that moods influence the way we think, while emotions, in contrast, influence
called particulate prir: ;- :
action (Davidson r9g+).
Emotions, feelings, affects, and moods are, in practice, often difficult to distin- and durations mar'-¡.
guish from each other, for example because all information is not available in a real- configuration u'ithou: -:
life or laboratory situation. We may not be aware if the feeling reported by a listener unique combination..,r.
sets ofa foreign cultu:-
was in fact preceded by a physiological response, in which case it should have been
categorized as an emotion. I will therefore use the overall concept ffictiue responses tion being limrted c::,,
(Grewe et al. zooT) whenever the precise affective category is unknown or irrelevant. ducing and recombir-..
Since feelings involve labelling of the content or quality of both emotions and other
that learning a forer:r s:
affective responses, I will refer to feelings as the category of quality, such as happiness discus. -.
(cf. the later
and anger, good or bad, and so forth. These are but ¿ -.-. .
appear to be culrl:. .
sounds in synchrony with someone else, or to remember a melody from one day to across individual: nr .-. .
the next (cf . Peretz zoo6). This circuitry may to some extent be dedicated to musical of musical experienc- .,.
processing and reside in specific brain areas (Zatorre and Halpern rgg¡; Falk zooo). scope (but see Krumr.-
Temporal entrainment is the key feature underlying both the ability to synchronize however, that all hum,-..
and the temporal structure of music in terms of the isochronous beaf or tactus ficient to make mu=i¡ .
(Merker et al. zoog). Only a few species are known to synchronize, fot example fire- defi ning design learur=,
flies (Pteroptlx) and crickets (Mecopoda), and this behaviour is predominantly used whereas cultural dit*rr
by males to attract females (Greenfield r99a). With the notable exception of humans, parameters' rr¡i1h ¡sspr -
-
CAUSE AND AFFECT 3J3
::. solemn, suspicious, primates and mammals in general do not s1'nchronize, which means that they either
lack the ability or the motivation to do it.
-:i-r \\'ords an evalua- All tonal systems feature octave generalization, which means that tones spaced one
: -lirerous studies indi- octave (f = f zt") are considered to have the same pitch identity (chroma), or, more
r ::plied by humans. generally, to be more similar than tones with any other spacing. This corresponds to
;'".n1hing may effec- the fact that we also perceive tones this way, a feature shared with Rhesus monkeys
.'":- and reproduction (Wright et al. zooo) and hence probably with most mammals. Moreover, musical
scales in different cultures share striking similarities in gross structure, for example,
r,rods are slow, per- the order of relatively large and small intervals with respect to consonant intervals
::l noods may reflect (Burns 1999). These structural properties are well accounted for by theoretical models
: :¡rsistent situational based on information-processing efficiency with respect to audition and cognition
::i so torth. Mood can (e.g. Balzano r98z; Tymoczko zoo6). This also puts a limit to the number of scale tones
ar:d altects, although within the octave, which is Qpicallyless than eight.
'.
i:nctronally, one can An even more fundamental characteristic of music is that its constituents-like
:i ;ontrast, influence those ofhuman language-are discrete and non-blending, and thus conform to the so-
called particulate principle (Merker zooz). This means that although the sets of pitches
:¿: lirfrcult to distin- and durations may be, and typically are, small, they can be combined in almost any
r : t ¿r ailable in a real- configuration without information loss, yielding a virtually infinite number of possible
: :::orted by a listener unique combinations. As a consequence, any individual can quickly learn the specific
sets of a foreign culture and partake in its musical behaviours, the level of sophistica-
'. it should have been tion being limited only by his/her cognitive skills in terms of remembering, repro-
;: : t,t.tf e ctiv e r esp ons es
.:rrJ Lr \\TI Or irrelevant. ducing and recombining. Even if all members of the sets were novel, it is conceivable
---,¿inotions and other that learning a foreign spoken language would be several orders of magnitude slower
il::''. such as happiness (cf. the later discussion ofcostly signalling and pattern elaboration in songbirds).
These are but a few examples of features of music that on the surface level may
appear to be culturally determined, but which in their principal characteristics are
biologically determined. Although samba and reggae utilize different rh¡hmical pat-
terns, both build their patterns on integer multiples and subdivisions of a continu-
ous isochronous beat. Examples of music that seem to overstep these neural settings
can always be found, but they are rare cases when cultural evolution has for some
SAL reason taken the narrow path away from nature. When Arnold Schoenberg decided to
make all twelve tones equal (instead of subordinate to each other, as in the Western
tonal system), he meant to create a more refined and artistically free music. The idea,
however, was purely intellectual. In disregarding the neural substrate of the listener,
,r lt. no human could the result was deemed obscure and unattractive by all but a few devotees.
.:-. lroduce a series of More complex and higher-level processing of music also exhibits high consistency
eL'',dr- tiom one day to across individuals from different cultures, across infants and adults, and across levels
- dedicated to musical of musical experience. A closer description of musical universals is outside the present
"::rn 1993; Falk zooo). scope (but see Krumhansl and Lerdahl, Chapter ró, this volume)-the point being,
:bilin'to synchronize however, that all humans share a genetic disposition that is necessary and may be suf-
.:Lrnous beat ot tqctus ficient to make music a human universal. It should also be noted that these central,
,:lüe. ior example fire- defining design features of music are those that are common to the world's musics,
:: predominantly used whereas cultural differences are found for features that can be considered 'free
; . \-aeption of humans, parameters' with respect to the processing of musical patterns by the neural system.
334 GUY MADISON
rna
ryu+t'. E.---'-.--
a 1!!---j '
That music is biologically based and universal suggests that it could have been zation rr'c.u,: .:. .
selected for at some point in our evolutionary history. There are other possibilities, favoured b, r. . -:.
of course, for example, that music is a cultural invention that opportunistically draws to the srrc:r :-. .
on capabilities that were selected for some entirely different function (cf. Brown et al. of music¿l i-r',. - :
zooo; Fitch zoo6\Perefzzoo6). This exemplifies the uneven interface between func- tion beniee: :-. --.
tion, traits, and adaptive value, in contrast to the apparent uniformity that obtains Merker's:a::::.
for behaviours with direct biological value, such as eating. These are difficult issues' in er,oluticr,. ..'. *
constantly a matter for debate, as mentioned, and this is no place to enter into this blorvn mus,:.
controversy. Instead, I will briefly delineate what I see as four main themes in the l) The /argr;,;-¡i : ,
literature, which all allude to evolutionary mechanisms but differ in their evolu- been coopei:.-,.
(Brorvn lci-, r 1,.
tionary logic:
neCessan' :I:---r-.-
1) The claim that seems to be implied by many theorists is that induction and then, or ir :.-,.,
communication of emotion constitutes a function of music and a reason why both cases n--.., ,
we are attracted to it; in other words, no less than an explanation for the very (Gould anc -.',,
existence of music. Emotions and their expressive behaviours are innate, which at a later siai: ::
provides a general, on-board, ready-to-go detector of their associated expres- purpose is s-,,,*-,: .
iive features. That emotions themselves have an obvious adaptive value could
further buttress and quali$' this claim by indirectly endowing the musical I will nort dis;,-,,
experience with a connection to innate biological mechanisms. The evolutionary hlpotheses.
significance of this emotion hypothesis is, however, that musical capacities them-
,.1u., -"r. not directly selected for. One could argue that once they were in place
for other reasons, the ensuing interpersonal emotional communication or self-
regulation might have assumed adaptive value, for example in terms of preverbal
communication between infants and their caregivers, along the lines suggested Cr
by Hodges (r9s9) and Roederer (rs8+)'
2) Charles Darwin (r87r) was the first to propose that human music evolved as a
sexually selected courtship display. A widely accepted account for such selection
is the handicap principle, exemplified by the peacock's tail. The main idea is that The function oi e:r :.
the display itseif is costly and therefore cannot be afforded by an individual of each different teel-:l:
poor quality. The learning and production of complex patterns in animal song negative feelings c-r- - .
is costly in terms of resources, and serves therefore as costly signalling of mate and positive f'eclir -. :
quality in birds (Marler zooo; Slater zooo) and humpback whales (Payne zooo). Of course, the cor::: = .
Songbirds invest energy to grow and maintain neural tissue dedicated to singing, outcomes of ou r ; -',,-.'
in addition to the time spent learning and practising song patterns, resources that Outside the nar:-,'',.
might have been spent on nutrition or some other direct adaptive value (Catch- and ideas have bqc:t .-'
pole and Slater 1995). In addition, the sound they produce reveals their posi- lutist, cognitir isr. e ,.::,
tion to predators and increases their risk of becoming prey. The fact of surviving r985; Davies zoor; L :.
despite such handicaps is a mark of quality. Miller (zooo) provides many argu- affective responses i--
ments why sexual selection should be considered the evolutionary null hlpoth- what it is about ¡nu.:;.-- .
esis, and notes, for example, that music making is predominantly an activity of edgecontributes. ani .
young males.
CAUSE AND AFFECT 335
TON OF MUSIC 3) The social bonding hypothesis asserts that music as a cooperative activity may
have an important bonding effect, and may facilitate social organization (Roederer
1984; Freeman 2ooo; Ujhelyi zooo). Inasmuch as such changes to social organi-
t-s that it could have been zation would entail increased adaptive value for the individual, they would be
'he¡e
are other possibilities, favoured by natural selection. Freeman gives particular significance in this regard
hat opportunistically draws to the synchronization of bodily movement by means of pulse, the foundation
nt tunction (cf. Brown et al. of musical rh¡hm, a trait that may have occurred during the period of specia-
,-en interface between func-
tion between chimpanzees and early hominids (Merke r 1999, zooo). Incidentally,
:nt uniformity that obtains Merker's scenario is limited to temporal entrainment and to a critical period
¡. These are difficult issues, in evolution, and is therefore unlikely to bear on affective responses to full-
no place to enter into this blown music.
.-< lbur main themes in
the 4) The language hypothesis refers to the idea that human language and music have
hut differ in their evolu- been cooperative during some period of their common evolutionary history
(Brown zooo; Molino 2ooo; Richman zooo; Mithen zoo5). Music was either a
:ists is that induction and necessary transitional stage towards language that has remained with us since
: music and a reason why then, or it relies on the same capabilities that were selected for language. In
r e-rplanation for the very both cases music would be a side effect of a true adaptation, a so-called spandrel
har-iours are innate, which (Gould and Lewontin ry79), or, if music has assumed some other adaptive value
,,1 thei¡ associated expres- at a later stage, an exaptation, meaning that a trait originally selected for one
rrrus adaptive value could purpose is switched to another (Gould and Vrba r98z).
-i' endowing the musical I will now discuss functions of affective responses in relation to each of these
ranisms. The evolutionary hlpotheses.
r musical capacities them-
r¿t once they were in place
rl communication or self-
nple in terms of preverbal
along the lines suggested
C¡.USg AND AFFECT IN THE
-r:nan music evolved as a CONTEXT OF MUSIC
r-.-ount for such selection
tail. The main idea is that
::ded by an individual of The function of emotions and affective responses in general is to guide action, and
:rpatterns in animal song each different feeling signifies different action tendencies (Frijda 1986). Basically,
::ostly signalling of mate negative feelings direct us away from that which may ultimately threaten well-being,
,¿ck rvhales (payne
zooo).
and positive feelings direct us to that which may ultimately facilitate well-being.
ssue dedicated to singing, Of course, the complexity of modern society entails many examples of non-adaptive
:q patterns, resources that outcomes of our affective processes.
ur adaptive value (Catch- Outside the narrow confines of adaptive hlpotheses a wide variety of theories
¡duce reveals their posi- and ideas have been proposed and subsumed under different labels, such as abso-
'rer. The fact of suwiving lutist, cognitivist, expressionist, formalist, and referentialist (for reviews, see Budd
lo: provides many argu_ 1985; Davies zoot; London zoor). These theories encompass issues such as whether
'''olutionary null hlpoth- affective responses to music are different than those evoked by everyday events,
lominantly an activity of what itis about musical expression thatwe reactto andwhy, howformal musicalknowl-
edge contributes, and so forth. These distinctions tend in some cases to be conceptually
336 GUY MADISON
fine-grained to the extent that evaluation in practice becomes impossible. More Table 17.1 Suggester
importantly, theories that cannot distinguish between innate and acquired relations respect to music accc
lack the power to explain the universal attractiveness of music, and will therefore not means that no affrct
be further considered. response may occur ;
Affective responses act on innately determined stimuli related to, for example, responses. See text fc
food and sex, but also on associations acquired through personal life experience,
and on continuous cognitive processing. A lifelong fear of boats may follow upon a T'¿pt c
near-drowning accident, or curiosity upon receiving a letter. Associative and cogni- ^--__
tive factors must figure prominently in emotional responsiveness to music, since the
social and cultural uses of music are laden with value and meaning. Compared with
innate grounds, however, they are much less likely to play a universal functional role, Emotion
and therefore less likely to be associated with a motivational machinery in terms of, hypothesis
for example, emotions. They are also inherently unsystematic, even though cultural
transmission may instil a modicum of order. I will therefore focus on innately deter-
mined associations, although this should not be taken to mean that acquired associa-
signalling :- :: :
Costly
tions may not be profound and extremely important for the individual.
One central point for the study of emotion and music has been that emotions are
,,
hvpothesis ¡--
:: -.-
li-_'
associated with specific expressive behaviours in terms of facial, vocal, and bodily
¡:---
movement displays. These displays are highly general in terms of similarity across
humans and higher non-human animals, and features of vocal and bodily displays bonding --,, ,
Social
can be reflected in an acoustic signal (Juslin and Laukka zoo3). The timing profile of hypothesis =t: -;
a caressing hand may be duplicated in a series of pitch changes, or the harshness of
I t_--
an angry voice may be imitated by a strained, distorted timbre on a saxophone. For
example, timing patterns in music performance could be described by a non-linear - :-
statistic that correlated with perceived emotional expression (Madison zooo). A happy Language l-:: :
person is inclined to walk steadily and briskly, which maybe inferred from music with hypothesis =t: -;
a fast and steady tempo, and so forth. An extensive list of such homologous features is l*':::
found in Juslin and Laukka (zoo3).
This apparent similarity in form is likely to have inspired the notion that music Notes. Elated is a qi-:.. ::.
possesses a unique power to express and communicate emotion, and that this might and loneliness ,ho, r ,. -.-,
constitute its prime attraction (e.g. Cooke ry59;Langer r95r; Langer 1953; Meyer 1956). from the establis"r: ::: - i-
bThese
are ot r t*o :' ; :.-:
The influential Affektenlehre was also based on these ideas, with composer johann ecstasy, euphor'a . .t:. -: :'
Mattheson as its most prominent exponent (Der vollkommene Capellmeister 1739 / reported for strong ilt::. :-::
Buelow r983).
Table r7.r attempts to summarize the affective responses that may be predicted on
the basis of the foregoing sections. There are four blocks, one for each musical func- perlormance br' .,¡.. .;
tion, and four rows within each of these, one for each type of affective response. most coveted mu:----,
We know from everyday experience that a musical event can range from the unde- and staggering. dfl- : .
sirable exposure to music one dislikes, or a failure while attempting to play something accordingly rea:on : _
on an instrument, to an eagerly awaited concert with a favourite artist. While unpleas- riences, a distin.ti,.r. -__
ant musical situations are very rare and in most cases can be dealt with by simple avoid- motivates us to s(cñ :'
ance, by far the most common musical event can be described as humdrum. (This iour, e.g. Timberl¿i. .:
is, by the way, no doubt a result of ubiquitous exposure to music made possible by some aflective re)F, ..:
audio reproduction technology: in the days when the only source of music was live which the column en.-.-
CAUSE AND AFFECT 337
::::r be predicted on
: :¿ch musical func- performance by oneself or others, interest was conceivably higher, on average.) The
,,-::'''e reSpOnse. most coveted musical experiences are, according to some people, intense, joyous,
:..:-ge trom the unde- and staggering, and also relatively rare (Maslow 1976; Gabrielsson zoota). There is
-: r :o play something accordingly reason to distinguish between humdrum and good or peak musical expe-
.:: - st. \\¡hile unpleas- riences, a distinction represented by the two rightmost columns of the figure. Hunger
.,. -:h br-simple avoid- motivates us to seek food, not only to eat when food is available (cf. appetitive behav-
: .. humdrum. (This iour, e.g. Timberlake and Silva ry9).By analogy, appetency for music might entail
some affective response that motivates us to actively seek musical experiences, for
-,-c made possible by
.r-: r,-if music was live which the column entitled'seeking musical event' is intended.
338 GUY MADISON
According to the emotion hlpothesis, number (r) in the earlier list, the function feelings, and positr,,':
of music is to induce or evoke emotions. Emotional communication is the strong other specific feelir¡.
version of the hypothesis, namely that composers and performers intentionally convey Apart from the s::
emotions that listeners actually experience (Meyer 1956; Cooke ry59; luslin ry97; cal experience, thei- :
zooo; 2oor; juslin et al. zoo3). A weaker version of the hlpothesis is that the expression not evident from ¡1.
of emotion in music somehow modulates the listener's affective responses in desirable functional hwoth¿sr.
ways (Langen ry5:l,r953;Kir,y r99o). Both versions predict avariety of emotions accom- First, althoueh s.-.
modated by the phrase 'Any specific basic emotion' in Table r7.r, which leaves open argument whr- m;. ;
whether the emotion is constant or varies throughout a musical event, or if it is posi- bodily, gestural. r'i, : :-
tive or negative. Indeed, a substantial proportion of the world's musics can be char- equally efficient.
acterized as melancholic, serious, or sad, according to the correspondence between Second, there i¡ : -
their expressive features and human emotionally expressive behaviour. The function acteristic feature ,-,- -":
should further be limited to basic emotions because these are the ones that have an perceived or appr:-,..
innate and universal form of expression. Since feelings are the appraisal of emotion, not a sentient be.¡: .-
the feeling should be identical to the emotion. Also the mood should have the same (tgSZ). While he ¡;.;.
valence as the emotion because emotions are intense and pervasive. However, if being avails itself of the::r. ,
emotionally aroused in general is positively appreciated, a positive affect might be the attempt to e\r:.::
expected, as indicated within parentheses. Events with desirable outcomes are some- ety of factors. en.i: ',,
times associated with emotions that direct us to seek these events, but among ten listener emotiona-..
such emotion-goal pairs there is no goal even remotely similar to taking an emotional Darwins's (t8;: s;::-
roller-coaster ride, so dashes in this column means that no such prediction is made theorylends som.;:.
(e.g. Plutchikry94, p.56). Finally, it is conceivable that a humdrum musical event For comparison. :-:.
induces weak emotions, while a good event may be characterized by strong emotions, former by seekirs :,. -
and increased intensity of the responses is indicated by plus signs (+) in Table r7.r. in point, chara.l¡n..:
Music experience does not correspond very well with these predictions. Rather, suited for acous:.; :
positive emotional feelings are predominant and negative ones are very rare. Also, intrinsic relation l ¡. ,,¡.
feelings associated with basic emotions seem to play a lesser role than other experi- but rather that nt l-,:; ,
ential dimensions, such as intensity, valence, and grooviness (Madison and Merker because musical e-¡:-. - :
zoo3; Madison zoo6). These and indeed most studies have asked for the perceived pattern, motii a:: -:t:-:
expression ('How well is this music described by the word happy?'), which may of evoke similar e\p3r:.-
course obscure possible affective responses ('How h*ppy do you feel'). Nevertheless, n'hl LrL¡ ¡:
is no reason
one would expect some amount of transfer from felt emotions to perceived emotional A third problei:: :,
expression, since it is natural to attribute what we feel to the music. A large number emotional displa,,. I: ,
of adjective rating studies (partly summarized in Gabrielsson zoorb) have shown experience anl em':,:r l
that emotional experiential dimensions generally account for a very small part of the to me, I might feel ¿:.
variance. This does not necessarily mean that emotion words receive low ratings, but I might feel afraid '¡ ¡;:
it does show that ratings do not vary much across the included musical selections, feel amused bec¿u..
contrary to what is predicted by the emotion hlpothesis. Instead, ratings are predomi- nisms related to a:1..i
nantly positive (Juslin and Laukka zoo4), even when the music is characterized as sad music are more lii¿l-,'
(Schellenberg et al. zoo8). Even when musicians try to express different emotions, rat- might be portraved :l
ings are markedlybiased in the positive direction across all expressive intentions (Iuslin diate and long-tern ;
and Madison 1999). One of the few studies that allowed for a wide range of different upon personal anci sli-
affective responses found that in order of frequencythe most common experiences for and highly conte{-i:
classical art music were moist eyes or tears (related to 'being moved'), other specific and unpredictable :¿,¡
CAUSE AND AFFECT 339
Fourthly, affective responses to the affective displays of other organisms may have sexual selection bv i'
adaptive value, since it may signal their propensity to help or harm. But it would not species-specifi c dir:li
make sense to react to, say, movements of branches that resemble the movements of displays for difie :,
of a happy or angry person. That emotions are innate does not imply that their expres- rivalry situations. T:t
sive characteristics evoke affective responses indiscriminately. They are, of course, song patterns stor-l .
contingent upon appraisal of the context; otherwise our emotional lives would be is dedicated br- th¿ ::
chaotic. Phobia is a case in point, which exemplifies automatic emotional responses of his achievemen: l
to stimuli with strong significance for survival, such as snakes and spiders (Dimberg ciple are pattern hd:L
et al. 1998). new songs or soul]c.:
Finally, if communication of emotions were the purpose of music, would we not ertoire, some disasse:
long ago have devised optimal means for this? It should not be that difficult, given ways, and some in.,'.
that we already have the expressive characteristics of emotions built into our limbic bird song as pred.icr,
system. If music were designed for optimally communicating emotions, it would con- highly predictable. si
ceivably focus on these expressive characteristics, and in effect sound quite different. depending on the s:!
There would be no need for a finite alphabet and hierarchical structure, and certainly and spend manv h,-- -
no explanation for their ubiquitous presence in music. We should also note that the eventual payoffis :.:
emotion hlpothesis focuses on the experience of the perceiver, for which the sender imitates long an.l ;-,:
of the musical signal is merely a means. Once optimal music for communicating withhim.
or inducing this or that emotion or this or that pattern of emotions has been made, What might be su;
would there really be such a relentless drive to create still new music and still new the costly signali:n: .
interpretations and renditions? Indeed, it leaves open the question why such a large imitation with a h..::
proportion of the Western population engages in musical productive behaviour, given little is known abo;: .
that they have all the music of the world available z4 hours a day,365 days a year, at a birds to these ven, .:.
negligible cost. If anlthing, this proportion has increased during the industrial revo- seems best furthe:¡: i
lution. This points to a motivation for producing music as well, not only receiving, action tendencr- -,.;¡ :,
flourishing as wealth and leisure time makes it possible to buy music equipment and appropriate phvsic,- -,,
spend thousands of hours practising. related to, or appr::..
The point of costly signalling (see number (z) in the earlier list) is to provide an infatuation, lust. ,,--., ;
honest measure of mate qualitybywhich, typically, the female can optimize her invest- strong or peak mLS:.;
ment in terms of the quality of her offspring. An ideal signal for serving this function The descriptio:r ::
conveys properties that are directly related to the factual cost for obtaining them. One same distinct ser :,-,-¡
could surmise that a number of features may further contribute to the signal's utility signalling hlpothes,s
as a basis for mate choice, for example that it is relatively unaffected by extraneous musical behar-iours .
factors and that its significant property bears a continuous, monotonical, and unam- response, both tb: i:t.
biguous relationship with mate quality. Sometimes this signal is merely an elaborate to learn and par- atr,
bodily ornamentation, such as the parrot's coloured plumage, but often behaviour is emotions for peak .'.
involved. In the appropriate situation the peacock stretches out and vibrates his tail, indicated by a dasl ::
and the songbird sings. Where the handicap is in the form of a behavioural display, it tion or other afte ci:', ,
presumably has motivational underpinnings with positive valence. Thus, if a function partaking in music¿-
of music is to be a vehicle for costly signalling, then it should reasonably be associated column. Although .:,
with positive affect and elated mood, as indicated in Table t7.r. tive responses tha: -
Birdsong may be a good model by which to consider music, because it, too, uses corresponds u,ell',,,'1.
acoustic patterns, and it is also the clearest and most extensively studied example of this chapter.
CAUSE AND AFFECT 34r
Lrrganisms may have sexual selection by way of costly signalling (Catchpole et al. 1995). There are many
:n. But it would not species-specific differences in details, such as the size ofrepertoire and differentiation
rrle the movements of displays for different functions, like mate attraction, territory defence, and male
r-r'that their expres- rivalry situations. The main principle is, however, high fidelity duplication, often of
trL:er- are, of course, song patterns stored in memory since hatching. Weeks or months of solitary practice
,¡nal ]ives would be is dedicated by the male to perfect his song, who during this time is the sole judge
:motional responses of his achievement (Merker zoo5). The quality measures that ensue from this prin-
rd spiders (Dimberg ciple are pattern fidelity and how much was copied (repertoire size). Some species add
new songs or sounds learned from conspecifics or arbitrary models into their rep-
rusic, rvould we not ertoire, some disassemble patterns into fragments and reassemble them in different
ihat difñcult, given ways, and some invent novel patterns. Therefore, the specific structural content of
:uilt into our limbic bird song as predicted from one subpopulation to another will range from being
r,trons, it would con- highly predictable, subject only to unintentional errors, to entirely unpredictable,
,.rnd quite different. depending on the species (Merker, in press). Once the males have attained adult song
:.ture, and certainly and spend many hours a day singing for weeks, the females are the judges, and the
io also note that the eventual payoff is early pair formation. But the basic principle is that the male closely
t r rt'hich the sender imitates long and complex signals, the success of which inclines the female to mate
ror communicating withhim.
Lrns has been made, What might be suggested by the analogy between bird song and music in light of
nusic and still new the costly signalling hlpothesis? For sake of argument we can equate unsuccessful
[rr \rhy such a large imitation with a humdrum event, and successful imitation with a peak event. Very
l','e behaviour, given little is known about emotions or other psychophysiological processes that may direct
_:6i davs ayear,ata birds to these very specific behaviours. For humans, however, the first action tendency
:he industrial revo- seems best furthered by feelings of interest, excitement, and curiosity, and the second
, not only receiving, action tendency would seem to be furthered by strong positive emotions that entail
i-sic equipment and appropriate physiological effects, as suggested in Table r7.r. These effects should be
related to, or appraised as, feelings such as adoration, compassion, desire, enthralment,
.sr is to provide an infatuation, lust, love, and passion, words that are very common in the description of
;,trimize her invest- strong or peak musical experiences (Gabrielsson zoora).
.ening this function The description of qpical bird behaviours should not be taken to suggest that the
b,raining them. One same distinct sex roles would obtain for humans, if there is some reality in the costly
:r¡ the signal's utility signalling hlpothesis, but both sexes may be sensitive to mate quality indicated by
rcted by extraneous musical behaviours. Interest would nevertheless be a suitable motivating affective
rtonical, and unam- response, both for the sender to learn and to produce the patterns, and for the receiver
nerely an elaborate to learn and pay attention to the produced patterns. Apart from strong positive
i¡ olten behaviour is emotions for peak events, no further emotions seem necessary for this scenario, as
:rd r-ibrates his tail, indicated by a dash in the emotion row in Table r7.r. Nor would any further emo-
rar-ioural display, it tion or other affective response seem to be connected with an action tendency for
:. Thus, if a function partaking in musical behaviour, as shown by a dash in the 'seeking a musical event'
: nablv be associated column. Although space does not permit any systematic review, the pattern of affec-
tive responses that I have suggested on account of the costly signalling hypothesis
teaause it, too, uses corresponds well with empirical data, some of which are mentioned throughout
s;udied example of this chapter.
342 GUY MADISON
The social bonding (number (¡) in the earlier list) hypothesis would seem to imply Rrrrrc
a similar pattern as for costly signalling with respect to affect and mood, given that
musical behaviours were selected for the function of putting neighbours on friendly
terms with one another. Gregariousness is enjoyrnent of the company of others. If
musical behaviours promote socially valuable relations, then peak experiences should
Lthough th;:::¡:.:
reasonably indicate particularly favourable conditions for bonding and cooperation, ,-.---t
-lú)f'Lr-i:(: -L'L, -..-
'\ -- -..
-
and should therefore induce positive emotions like joy or happiness. In contrast to :rgumenta:i,l r-:: a, :r -
the positive affective responses in the costly signalling scenario, gregariousness has an j .--.-:,
ine hrpothe s:
',
opposite in terms of loneliness (Plutchik \994,p.56). I therefore suggest the possibil-
mind ithen L-'na . - *:-:
ity (within parentheses) that loneliness, negative affect, and depressed mood in this Horr-do pei:-. ---..:
scenario may act as motivators for musical behaviours.
pertorming L¡r b;: . :
The language hypothesis (number (+) in the earlier list) tries to explain how the --.- ^-: ^-
ú,\F'(1.i(il!1. :! !-:- : : -
capability for music could emerge, although in itself it has never been selected for. see Persson ::,
-- ..
-
There are different varieties on this theme that I need not delve into in order to draw : ,.
is mood res;i.::.
the simple conclusion that this renders innate affective responses highly unlikely, as andO'\eiL: -:..
indicated by the plethora of dashes in Table t7r The prevalence of musical behav- one, but 3lllrrr: -t:..---
iour must therefore either reflect some other adaptive function, or some non-adaptive piness and r','e-i-:..:.
function that has pleasant outcomes. For the latter one could, for example, submit music-npi.a-l', .' .
the purely functional part of the emotion hlpothesis. Although musical behaviour t Sloboda et a.. : :
appears quite irrelevant for adult language, one could argue that it might nevertheless
interesting a:; *:, -.,
play a role for language acquisition in phylogeny (Hodges 1989). The motivational
through perso:::- -...
mechanisms related to the latter scenario are unclear, however, and further work is
sure to specin; ..,.;:
required to address it properly.
It strikes me th.:.:
Apart from these four hlpotheses, temporal entrainment to music should be men-
longer histon,'.
tioned as another possible source ofaffective responses. The isochronous pulse serves It is a rriderp:.':
as a s1'nchronization device, without which ensemble performance of music or dance
attempt to aa;,-.*:-: :.
would not be possible. On a more general level, the pulse creates expectations, as
successful. sin;e :::= '
amply demonstrated by electroencephalogram responses (so-called event related i. r , :
the informatitrl
potentials) at the point in time where an omitted sound event in an isochronous train
1998). The ide¿ --:,:. :
should have occurred (Jongsma et al. zoo5). This indicates a fundamental neural
useful end is ais¡ -.:-.
mechanism that may have the function of focusing attention (Large and Palmer zooz)
is not surprisi:ri '.-..:
within a relatively narrow range of variability (approximately 8o/o of the interval), in their u.orkr lq'-: j::
which probably corresponds with a limit for useful precision in temporal prediction
any.thing to do rti::- .-:
(Madison and Merker zooz). On the other hand, it also adapts to gradual tempo change,
problems, accorl,::¡ :
even to fast rates of change (Madison and Merker zoo5), making it useful as a means
meaning that musr¡ .
for synchronizing with variable models, such as other humans. The utility of such a
ever, is, rather. a :ria:-i
mechanism suggests that it may be associated with some motivating affective response
cance is no less th::l :¿
(cf. Damasio zoo4), and the strong positive valence associated with the experience of
of this communi;::: -
groove is a likely candidate (Madison zoo6). However, temporal entrainment will not
late that this inna:: ---
be considered an alternative to any of the other hlpotheses since it represents merely
music with the ch:::.
one component of rh¡hmic music and also yields overlapping predictions (positive
(Laiho 2oo4). -\ m¿r :
affect and mood).
thing to them, and r ,:
CAUSE AND AFFECT 343
eEariousness has an argumentation available is no ground for committing to it. To be sure, the social bond-
ing hypothesis yields similar predictions. However, a number of observations come to
"Lsqest the possibil-
mind when one considers music in this light, a few of which are mentioned next.
:essed mood in this
How do people themselves account for their involvement with music? Reasons for
:r-1 e-\Plain how the performing or becoming a performer are primarily the quest for positive emotional
: been selected for. experience, social status and recognition, and the sense of achievement (for a review,
see Persson zoor). The most common reason that people report for listening to music
il{l in order to draw
i highlv unlikely, as is mood regulation, according to seyeral surveys (Iuslin el aI. zoo4, p. z3r; Sloboda
r 'ri musical behav-
and O'Neill zoor; Saarikallio zooT). The desired mood is mainly a more positive
s,rme non-adaptive
one, but a more negative mood could also be sought with the goal of increasing hap-
piness and well-being in the long term (Saarikallio zooT). Exposure to a particular
-: erample, submit
:rusical behaviour music-typically even a particular performance-is critical for the intended effect
(Sloboda et al. zoor), which indicates that music in this capacity mainly serves as an
miqht nevertheless
. The motivational interesting and motivating stimulus for recalling memories and associations acquired
n,l lurther work is through personal life experience. Mood induction relies on swift and effortless expo-
sure to specific selections, and hence in practice on audio reproduction technology.
:-.ic should be men-
It strikes me therefore as a recent invention, subordinate to other functions with a
:{-}nOuS PulSe SerVeS
longer history.
. i¡f music or dance It is a widespread notion that music communicates something significant. No
:;r erpectations, attempt to account for this from a referential or semiotic perspective has been very
as
successful, since they assume, in analogy with language, that the significance lies in
iled er-ent related
the information conveyed (for discussions, see Davies 1994; Clarke 1995,2oor; Sloboda
l isochronous train
1998). The idea that communication of emotions would provide either a feasible or
rndamental neural
e and Palmer zooz)
useful end is also faced with many objections, as I have argued earlier. In this light it
r- of the interval), is not surprising that composers and performers rarely seem to consider emotions
::r:poral prediction in their work; Igor Straünsky is one who has publicly disavowed that his music has
iual tempo change, anlthing to do with emotions. However, the costly signalling hlpothesis resolves these
t u-seful as a means problems, according to which the specific musical structure would be irrelevant. The
re utility of such a meaning that music always carries, regardless of melody, complexity, style or what-
ever, is, rather, a measure of information processing ability, as it were, whose signifi-
: eltectir.e response
cance is no less than reproductive success for the individual. As such, the relevant level
h .he experience of
r:rainment will not of this communication is from one individual to another. One can therefore specu-
: represents merely late that this innate biological significance might somehow endow our experience of
'edictions (positive music with the character of a personal message or a narrative, as is often described
(Laiho zoo4). A majority of listeners indicated that music often communicated some-
thing to them, and that this something was emotions (47o/o), emotions, opinions, and
344 GUY MADISON
messages (34o/o), and messages (zo/o) (Juslin et al. zoo4). Another kind of evidence soon break the ctri.
comes from a task in which listeners had to choose one of two opposite words, such mation reduction. SL
as male-female and bright-dull, in response to a number of short music excerpts. cultures, all chara:i ¿ l
It turned out that the agreement among participants was significantly higher for words variation, but so¡n: i
that might apply to a person (e.g. good-evil, old-young, joful-sad) than for words on rh¡hm (e.e. \t-esi
unlikely to apply to a person (e.g. narrow-wide, day-night, dry-moist), whereas no Similarly, this s:;¡.
such difference was evident for words chosen in response to foodstuffs (Watt and tinue to create nei,.' :
Ash 1998). music we alreadr h::
But would not the principle of imitation that renders birdsong its utility as a music that áre r€-r :,
costly signal seem to be at odds with the human creativity so highly esteemed in the competitive signa,. ::
field of music? Not at all. Firstly, song patterns do vary in nature, as mentioned, but is only limited b, ---
the point is that memory capacity and time constitute the main biological costs, not musically produ;::.,;
innovation as such. Therefore, variation on a theme or rearrangement of song elements gests an innatelr' :c,::
is just as costly as high fidelity duplication, because the same memory representation also the speed. pre:r,
is required. Secondly, we must admit that the lion's share of human music produc- reasonablr'mar-be ..
tion is indeed replication; some music is performed over and over again by different more energ¡'or ¿-f,.:'
artists, concert goers may even see the same artists repeat their songs, and we also listen and clubs is a qroi'.:::
to the same recorded performances again and again. If the primary demand charac- resources permi:. -::.
teristic for music were creatiüty or novelty, then musical practice would certainly be suspect sorne fltr"-:
quite different. Thirdly, even a novel piece of music is quite similar to previous art; of their resourc.) ,,
otherwise we would not have the concept of musical style or the difficulty in defining increasing ofl á p:: ',,
plagiarism. Each alleged new composition complies with a multitude of production tion of compete:: :-
rules implicitly set by the culture from which it stems, and by which it is judged. In onstrate greater r-1 .- :
other words, higher-order musical properties may serve as a costly signal just as well as human faculties n-...
specific pattern structure does, since acquiring such knowledge also requires resources arts such as scu,p:*:,
in terms of time and memory. This is one possible explanation for the fact that most craftsmanship, kir.::
of us prefer the musical styles we are familiar with, whereas a demand for novelty and tion for this ditl¿:.,-;
variation would direct us to unfamiliar styles. Familiarity is nothing else than implicit thanreproducti-.:.-
knowledge about higher-order properties, acquired by repeated exposure to a suffi- Earlv quanlun --:
cient sample of music. If the ultimate motivation for music is to display and gauge der.eloping duc:rL¿, -
the invested cost, then this knowledge is essential, and the more familiar the style the resources in con'..:,:
more precisely can we do it. Indeed, preference for initially unfamiliar music increased cians, no doubt :,-- --,
monotonically during one month of daily listening, and across four levels of musical the novel situatit-,: ,-,
complexity (Madison et al. submitted). tor subsistence ¡ *-
Although the particular musical structure would be arbitrary for the purpose of tion and compie:l-.
costly signalling, its general characteristics such as complexity, repetitiveness, and 15-minute piece ,:,: ::'
constituent elements can be more or less optimal according to the circumstances. It is small orchestra.,-.::.
conceivable that the discrete nature of musical elements (e.g. the particulate principle ner-ertheless qr-ri:: ::-
mentioned under biological and adaptive aspects, see Merker zooz) and the hierarchi- u'ere the onlv stru"l.
cal, repetitive structure (Krumhansl and Lerdahl, Chapter l6) both conspire to reduce 1;OO to 1gOO. th. -rl
cognitive load while increasing splendour. In addition to simply learning and repro- cumstances had r-.. r
ducing patterns as-is, then, inventing strategies for reducing this effort would increase could acquir€ dn l-rri
an individual's competitive advantage as a sender by accommodating even longer and plaved, arded. ::
patterns. However, as soon as this individual applies these devices, others will very Horr.eyer, the¡e is :
CAUSE AND AFFECT 345
rr klnd of evidence soon break the code and apply it themselves, which leads to an arms race in infor-
,¡osite rvords, such mation reduction. Such may be the path to the various musical systems in different
:,:t music excerpts. cultures, all characterized by small finite alphabets and repetition of groups with some
t r higher for words variation, but some focusing more on pitch (e.g. Western tradition) and other more
.¿l than for words on rh¡hm (e.g. West African, Balkan, and Latin America).
noist ), rvhereas no Similarly, this selection pressure for apparent splendour could explain why we con-
ri,;¡s¡1i (Watt and tinue to create new music instead of merely enjoying the apparently very gratifying
music we already have, as demonstrated by, for example, the classical works of art
¡r:q its utility as a music that are recorded and performed in concert houses over and over again. As a
r1',' esteemed in the competitive signal, music can never be good enough (Miller zooo), but its splendour
¿s mentioned, but is only limited by the available resources. As mentioned before, the proportion of
r¡loqical costs, not musically productive people seems to be much larger than the demand, which sug-
l: of song elements gests an innately founded motivation for producing music. Not only the pattern, but
:,,,r' rePresentatiOn also the speed, precision, and loudness of a musical performance are properties that
r¿r music produc- reasonably may be associated with biological cost. A louder signal does indeed require
: :Eain by different more energy or a better technique, such as the operatic voice. The volume at concerts
s. ¿:rdrve also listen and clubs is a growing concern, with hearing loss in its wake. Again, it may be that when
r'' demand charac- resources permit, there is an element of 'the more the better', and one may sometimes
',';¡uld certainly be suspect some not-so-great artists of having invested a somewhat higher proportion
-:r io previous art; of their resources into sound amplification. Likewise, musical skill is constantly
.:--¡u1n'in defining increasing on a par with athletic skill. Industrialized countries see a growing propor-
:ice of production tion of competent musicians, and the top stratum of musicians today clearly dem-
-;h it is iudged. In onstrate greater motor skill than did previous generations. In contrast, a number of
just as well as human faculties might not exhibit this progressive amelioration, for example other
'rrna1
:¡uuires resOurces arts such as sculpture, literature, and visual art (nor, for that matter, common sense,
: :he tact that most craftsmanship, kindness, mathematics, or even verbal skill). One speculative explana-
.:d tbr novelty and tion for this difference is that such competences ultimately have a more confined utility
s else than implicit than reproductive success, and therefore good enough is quite sufficient.
riposure to a suffi- Early quantum leaps in this regard is a case in point. For example, the city states and
cisplav and gauge developing duchies in Medieval and Renaissance Europe provided concentration of
::iliar the style the resources in conjunction with relative stability. Potentates were soon to hire musi-
::.r music increased cians, no doubt to increase the magnificence of their courts. This put a selected few in
;r ier-els of musical the novel situation of being able to devote all their time to music, rather than working
for subsistence in more direct ways, which brought a very fast increase in the dura-
-ü: lhe purpose of tion and complexity of musical works. One can imagine the impressive impact of a
:e:.etiti\¡eness, and r5-minute piece of music with several more or less independent voices, played by a
:rrcumstances, It is small orchestra. Although there was certainly some repetition, early art music was
rrt;.u1ate principle nevertheless quite different from the common ballad, in which a refrain and a stanza
¡nd the hierarchi- were the only structurally unique units. A point worth making is that, from about
.L-jnspire to reduce lzoo to r9oo, the unequal distribution of wealth and other social and practical cir-
:arnilg and repro- cumstances had the effect that an extremely small proportion of the entire population
;'rt rvould increase could acquire an extreme superiorityin the musical sophistication ofworks composed
i;:ing even longer and played, aided, of course, by proficiency in reading and writing musical notation.
:s. others will very However, there is no reason why these skills would imply the general superiority of
346 cUY MADTSON
a genius if musical ability was not considered something very important in the first
place. Although this would seem unlikely given the small practical use music was to
the common man, the costly signalling hlpothesis could account for why one had
to assume divine influence and innate talent. The theory of talent prevails still today,
although systematic research has determined that most, if not all, variability in musi-
cal ability is accounted for by the quality and quantity of practice (e.9. Sloboda rggo;
Ericsson et al. 1993; Ericsson t997).The musical genius has now faded considerably
since its heyday, as would be expected by the widespread exposure to music and distri-
bution of musical skill in the general population.
Finally, I would like to emphasize that the present focus on function, in particular,
function related to adaptive value, is a strategy for highlighting large general patterns
in a web obscured by the conceptual and empirical difficulties hinted at in the intro-
duction. If the present attempt has any merit, it is certainly not to explain or downplay
the utility or experiential value of music or its related affective responses. It may be
helpful in furthering our understanding of these complex phenomena, however, by
suggesting where to look and what to look for, primarily in terms of ultimate motiva-
tional factors leading to the role that music plays in human cultures and in our lives.
AcrNowLED GEMENT s
RnprnENcns
Osio.
Ahlberg LO (rqg+). Susanne Langer on representation and emotion in music. British Journal Ericsson K\. Li:-::. .
of Aesthetics, 34, ó9-8o. accluisition tr: ¡:':: ¡ ::
Aristotle Qyz.). Poetics. (Vol. 23.) William Heinemann, London. FalkD (:.ooo . H..:'..:,.
Avanzini G, Faienza C, Minciacci D, Lopez L, and Majno M, eds. (zoo3). The neurosciences and S Brorrn. ec,. -l
and music. Annals of the N ew York Academy of Sciences, 999, 1-532. Fitch \\rT (:ooó . Th. :
Avanzini G, Lopez L, Koelsch S, and Majno M, eds. (zoo5). Neurosciences and music II: From roo, u3-115.
perception to performance. Annals of the New York Academy of Sciences, ro6o, r-487. Freeman \\IJ ro - .l
Balzano GJ (rq8z). The pitch set as a level of description for studying musical pitch perception. Blvlerker,¿n.iSf:'
In M Clynes, ed. Music, mind, and brain, pp.3zr-5r. Plenum Press, New York. Frijda NH (1986 . r ;:. .
Batson C, Shaw LL, and Oleson KC (rqqz). Differentiating affect, mood, and emotion: Toward Gabrielsson -\ rr, - -:
functionally based conceptual distinctions. In MS Clark, ed. Review of personality and social Sloboda, eds. -\I:,,.:- .
psychology: emotion, pp. 294126. Sage, Newbury Park, CA. Nerv York.
Brown SW (zooo). The 'musilanguage'model of music evolution. In NL Wallin, B Merker, Gabrielsson A r:c':::
and S Brown, eds. The origins of music, pp. z7t-3oo. MIT Press, Cambridge, MA. S cientiae, 5peci,r, . -.,...
CAUSE AND AFFECT 347
:,rrtant in the first Brown SW, Merker B, and Wailin NL (zooo). An introduction to evolutionary musicology.
r '¡se music was to In NL Wallin, B Merker, and S Brown, eds. The Origins of music, pp. 3-24.MIT Press,
:: Cambridge, MA.
'or rvhy one had Budd M Q98). Music and the emotions: the philosophical theories. Routledge, London.
irer-ails still today,
Buelow GI (rs8¡). lohann Mattheson and the invention of the Affektenlehre. In GJ Buelow
r¿:iabiliry in musi-
and HJ Marx, eds. New Mattheson studies, pp. 393-407. Cambridge University Press,
t.E. Sloboda r99o;
Cambridge.
:¡,jed considerably Burns EM (1999). Intervals, scales, and tuning. In D Deutsch, ed. The psychology of music, znd
:-- music and distri- edn., pp. zt5-64, Academic Press, San Diego, CA.
Catchpole CK and Slater PJB Q99). Bird song biological themes and variatíons. Cambridge
:tl¡n, rn particular, University Press, Cambridge.
:. Eeneral patterns Clarke EF (r995). Expression in performance: generativity, perception, and semiosis. In I Rink,
ed. The practice of performance, pp. zr-54. Cambridge University Press, Cambridge.
;¿,1 at in the intro-
Clarke EF (zoor). Meaning and the specification of motion in music. Musicae Scientiae, 5,
,-:.ain or downplay
2t3-34.
sf L-rnses. It may be Cooke D (1959). The language of music. Oxford University Press, Oxford.
:rena, however, by Damasio AR (zoo4). Emotions and feelings: a neurobiological perspective. In A Fischer,
,f Jtimate motiva- ASR Manstead, and NH Frijda, eds. Feelings and emotions: the amsterdam symposium,
. :rd in our lives. pp. 49-57. Cambridge University Press, New York.
Dar-win C (rS¡). The descent of man and selection in relation ro sex. (Vois. r-z) John Murray,
London.
Darwin C (tslz). The expression of the emotions in man and animals. John Murray, London.
Davidson RI (rSS+). On emotion, mood, and related affective constructs. In P Ekman and
RJ Davidson, eds. The nature of emotion: fundamental questions, pp. 51-5. Oxford University
Press, New York.
Davies S (ry94). Musical meaning and expression. Cornell University Press, Ithaca, NY.
Davies S (zoor). Philosophical perspectives on music's expressiveness. In PN fuslin and
:entS on prevrous JA Sloboda, eds. Music and emotion, pp. 23-44. Oxford University Press, New York.
n:lents and for his Deutsch D (rsss). The Psychology of Music. Academic Press, San Diego, CA.
Dimberg U, Hansson G, and Thunberg M (1998). Fear of snakes and faciai reactions: A case of
rapid respond ing. S candinavi an J o urn al of P sy ch o Io gy, 39, 7 5-8 o.
Ekman P (1992). An argument for basic emotions. Cognition and Emotion, 6,769-200.
Ericsson KA (rssz). Deliberate practice and the acquisition of expert performance: An over-
view. In H Jórgensen and AC Lehmann, eds. Does practice make perfect? Current theory and
research on instrumental music practice, pp. 5_-57.The Norwegian State Academy of Music,
Oslo.
"::''c. British Journal Ericsson KA, Krampe RT, and Tesch-Rómer C (lsS:).The role of deliberate practice in the
acquisition of expert performance. Psychological Review, too, 3@-+o6.
Falk D (zooo). Hominid brain evolution and the origins of music. In NL Wallin, B Merker,
Ihe neurosciences and S Brown, eds. The Origins of music, pp. t97-2r6. MIT Press, Cambridge, MA.
Fitch WT (zoo6). The biology and evolution of music: A comparative perspective. Cognition,
r
":ld music II: From too, t73-215.
:,.. ro6o, r-487. Freeman Wf (zooo). A neurobiological role of music in social bonding. In NL Wallin,
;.. ¡itch perception. B Merker, and S Brown, eds. The Origins of music,pp.4rt-24. MIT Press, Cambridge, MA.
i' 'frrrk. Frijda NH j986). The emotions. Cambridge University Press, New York.
:; enotion: Toward Gabrielsson A (zoora). Emotions in strong experiences with music. In PN Juslin and JA
: :':i lnlity and social Sloboda, eds. Music and emotion: theory and research, pp. 43r-49. Oxford University Press,
New York.
i.1,-¿1lin,
- B Merker, Gabrielsson A (uootb). Perceived emotion and felt emotion: Same or different? Musicae
:.¡, \lA. Scientiae, Special Issue zooT-2o02, 123-47.
348 GUY MADISON
RJ Davidson, and KR Scherer, eds. Handbook of ffictive sciences, pp. 503-34. Oxford
University Press, New York.
Gould SJ and Lewontin RC (rqZq). The spandrels of San Marco and the Panglossion paradigm:
a critique of the adaptationist programrne. Proceedings of the Royal Society of London, zo5,
58r-98.
Gould SJ and Vrba E (1982). Exaptation - a missing term in the science of form. Paleobiology,
8, 4-r5.
GreenfieldMD (1994). Cooperation and conflict in the evolution of signal interactions. Annual
Review of Ecological Systems, 25,97-126.
Grewe O, Nagel F, Kopiez R and Altenmüller E (zoo7). Emotions over time: s1'nchronicity and
development of subjective, physiological, and facial affective reactions to music. Emotion,
7,774-88.
Hanslick E GgSl). The beautiful in music / Vom Musikalisch-Schónen. Bobbs-Merrill,
Indianapolis, IN.
Hodges D (rq8g). Why are we musical? Speculations on the evolutionary plausibility of musi-
cal bahavior. Bulletin of the Council for Research in Music Education,99,7-22.
|ongsma M, Eichele T, Jenks KM, Desain P and Honing H (zoo5). Expectancy effects on omis-
sion evoked potentials in musicians and non-musicians. Psychophysiology, 42,191-2ot.
Juslin PN (t997). Emotional communication in music performance: A functionalist perspec-
tive and some data. Music Perception, l.4,383-4t8. S Bror.,'r. ¿:. -
fusiin PN (zooo). Cue utilization in communication of emotion in music performance: ]'l¡r¡erB : : I'
Reiating performance to perception. lournal of Experimental Psychology: Human Perception \1¿:kerB :-' .
and P erform ance, 26, 17 97-8t 2. -.'illril-'.-, .i
Juslin PN (zoor). Communicating emotion in music performance: a review and theoreti- l',ierker B ir ¡:.,,
cal framework. In PN fuslin and lA Sloboda, eds. Music and emotion: theory and research, T- \- i.:r-:. :-
pp. 309-37 . Oxford University Press, New York. --
pr¿sS, Ltr,:,r:
Juslin PN and Laukka P (zoo3). Communication of emotions in vocal expression and music .,:lK3: -Lt -!-;!-!
performance: Different channels, same code? Psychological Bulletin, rz9,770-814. i,-J,lt -,t!iL!1-:1.
L
-- -- :,,-- :
-
Iuslin PN and Laukka P (zoo4). Expression, perception, and induction of musical emotions: I'lererLB r9:. .:
A review and a questionnaire study of everyday listening. lournal of New Music Research, I'li-lerGF :,, ", ¡
33, zr7-38. a¡j ! g¡¡.rrr.:
Juslin, PN and Madison G (ISSS). The role of timing patterns in the decoding of emotional
expressions in music performances . Music Perception, 17,7g7-zzr.
Juslin PN and Sloboda JA, eds. (zoor). Music and emotion. Oxford University Press,
New York.
Kiry P (r99o) . Music alone. Philosophical reflections on the purely musical experience. Cornell
University Press, Ithaca, NY.
Kir.y P (1999). Feeling the musical emotions. British Journal of Aesthetics, 39, 7-13.
Kir,y P (zooz). Introduction to a philosophy of music. Oxford University Press, Oxford UK.
Kleinginna PR and Kleinginna AM (r98r). A categorized list of emotion definitions, with a sug-
gestion for a consensuai definition. Motivation and Emotion,5,345-7r.
Laiho S (zoo4). The psychological functions of music in adolescence. Nordic Journal of Music
Therapy, t3, 49-65. :-,rr.hil R r99:. . I::, :
Langer SK (rqsr). Philosophy in a new key, z"d edn. New American Library, New York. :,¡-,.'er \f and D¡-:1. ,:
Langer SK (rqs¡). Feeling and form. Charles Scribner's Sons, New York. Press. Hore. UN
Large EW and Palmer C (zooz). Perceiving temporal regularity in music. Cognitite Science, F-;hn-ran B :o ,¡ . ::-'
26, t-37. etition. and n.;:,-:-:
London J (zoor). Some theories of emotion in music and their implications for research in pp. -tor-1+. \fll F:",
music psychology. Musicae Scientiae, Special Issue zoot-zooz, z3-36.
CAUSE AND AFFECT J49
: i: HH Goldsmith, Madison G (zooo). Properties of expressive variabiiity patterns in music performances. Journal
. ll. joj-34. Oxford of New Music Research,29, n5-6.
Madison G (zoo6). Experiencing groove induced by music: consistency and phenomenoiogy.
::-l¡r55i6¡ paradigm: Music Perception, 24, zor-8.
i
- :::..', o.f London,2oS, Madison G and Merker B (2oo2). On the limits of anisynchrony in pulse attribution.
P sychologícal Research, 66, 2ot-7.
. : ,¡rm. Paleobiology, Madison G and Merker B (zoo3). Consistency in listeners'ratings as a function of listening
time. In R Bresin, ed. Proceedings of the Stockholm music acoustics conference, pp. q9-42.
-,r:eractíons. Annual Royal College of Technology, Stockholm, Sweden.
Madison G and Merker B (zooS). Timing of action during and after synchronization with
:r.: s.,rchronicity and linearly changing intervals. Music Perception, zz, 44r-59.
r. :.' nusic. Emotion, Madison G, Schiolde G, and Paulin J (submitted). Repeated listening increases preference for
music regardless of its complexity: implications for music aesthetics and the mere exposure
-'.:.,:. Bobbs-Merrill, effect. Submitted for publication.
Marler P (zooo). Origins of music and speech: Insight from animals. In NL Wallin, B Merker,
:-.:usibility of musi- and S Brown, eds. The origins of music, pp. 3r-48. MIT Press, Cambridge' MA.
r9. --rl. Maslow AH Q976). The farther reaches of human nature.Pengtin Books, Princeton.
:= 1¡ elfects on omis- Merker B (rSSS). Synchronous chorusing and the origins of music. Musicae Scientiae, Special
---,.-t. 42. 1g1-2O1.
issue t999 -zooo, 59-73.
'*r :tionalist perspec- Merker B (zooo). Synchronous chorusing and human origins. In NlWallin, B Merker, and
S Brown, eds. The Origins of music, pp. 375-27. MIT Press, Cambridge' MA.
:r:--rr;c performance: Merker B (zooz). Music: the missing humboldt system. Musicae Scientiae,6,3-zt.
:, n:ttttan Perception Merker B (zoo5). The conformal motive in birdsong, music, and language: An introduction.
Annals of the New York Academy of Sciences, to6o, 17-28'
:¿"'-¡rt and theoreti- Merker B (in press). The vocal learning constellation: imitation, ritual culture, encephalization.
: :l: jar_r' and research, In N Bannan and S Mithen , eds. Music, language, and human evolution. Oxford University
Press, Oxford UK.
::.-j:r:ssion and music Merker B, Madison G, and Eckerdal P (zoo9). Human rh¡hmic entrainment: Comparative,
19. --o-8r4. functional, and developmental aspects. Cortex, 45, 4-r7.
,, . :rlusical emotions: Meyer LB (D56) . Emotion and meaning in music. University of Chicago Press, Chicago, IL.
li:rr -\fl-slc Research, Miller GF (zooo). Evolution of human music through sexual selection. In NLWallin, B Merker,
and S Brown, eds. The origins of music, pp.3z9-6o. MIT Press, Cambridge' MA.
:: d:ng of emotional Mithen S (zoo5). The singing neanderthals: the origins of music, language' mind and body.
Weidesfeld and Nicolson, London.
:: L-nir-ersity Press, Molino | (zooo). Toward an evolutionary theory of music and language. In NL Wallin,
B Merker, and S Brown, eds. The origins of music,pp.t65-76. MIT Press, Cambridge' MA'
,. :'¡ ¿riettce. Cornell
¿
Oatley K and lenkins IM (rgq6). Understanding emotions. Blackwell, Oxford.
Payne C (zooo). The progressively changing songs of humpback whales: A window on the
-19. L-1_1. creative process in a wild animal. In NL Wallin, B Merker, and S Brown, eds. The origins of
::¡s. Oxfbrd UK. music,pp.t3t.-5o. MIT Press, Cambridge, MA.
. .iitions, lvith a sug- Peretz I (zoo6). The nature ofmusic from a biological perspective. Cognition, loo, 1-32.
Persson RS (zoor). The subjective world of the performer. In PN Juslin and JA Sloboda, eds.
-.;:: lotLrnal of Music
Music and emotion: theory and research, pp. 275-89. Oxford University Press, New York.
Plutchik R (rsS+). The psychology and biology of emotion. Harper Collins, New York.
,. \:rr- York.
Power M and Dalgleish T (tgg).Cognition and emotion: from order to disorder. Psychology
Press, Hove, UK.
'-;. (,tgttitive Science,
Richman B (zooo). How music fixed'nonsense' into significant formulas: On rh¡hm, rep-
etition, and meaning. In NL Wallin, B Merker, and S Brown, eds. The origins of music,
:-Lrns tbr research in pp.301-14. MIT Press, Cambridge, MA.
350 GUY MADISON
Rigg MG (r96a). The mood effects of music: A comparison of data from earlier investigations.
Journal of Psychology, 59, 427-$8.
Roederer I (rs8+). The search for a survival value of musi c. Music perception, t, 350-6.
Saarikallio S (zoo7). The role of music in adolescent's mood regulation. Psychology of Music,
35, B8-1o9.
Schellenberg EG, Peretz I and vieillard S (zoo8). Liking for huppy- and sad-sounding music:
Effects of expo sttre. Cognition and Emotion, zz, zt8-37.
Scherer KR (zoo4). Which emotions can be induced by music? What are the underlying mech-
anisms? And how can we measure them? Journal of New Music Research, 3¡3, z3g-5t.
scherer KR, zentner MR and Schacht A (zoor). Emotional states generated by music: An
TH
exploratory study of music experts. Musicae scientiae, special Issue zooT-2o02> 149-77.
Scruton R (ISSZ). The aesthetics of music. Oxford University press, Oxford, UK.
slater, PJB (zooo). Birdsong repertoires: Their origins and use. In NL wallin, B Merker, and
REP
S Brown, eds. The origins of music,pp.49-e. MIT Press, Cambridge, MA.
sloboda, JA (rqqo). Musical excellence - how does it develop? In MJ Howe, ed. Encouraging the
development of ex.ceptional skills and talents, pp. t65-78. BpS Books, London.
AC
Sloboda, JA (rqq8). Does music mean an¡hing? Musicae Scientiae,2,27-32.
Sloboda JA and O'Neili. SA (zoor). Emotions in everyday listening to music. In PN fuslin and
JA sloboda, eds. Music and emotion: theory and research, pp. 415-29. oxford university
WI
Press, New York.
Sloboda JA, o'Neill, sA and Ivaldi A (zoor). Functions of music in everyday life: An explor-
atory study using the experience sampiing method. Musicae Scientiae,5,9-32.
Timberlake wD and Silva KM (1995). Appetitive behavior in ethology, psychology, and VIK 1\IU
behavior systems. In N Thompson, ed. Perspectittes in ethology,pp.211-53. plenum press,
New York.
Tymoczko D (zoo6). The geometry of musical chords. Science,3lg,,7z-4.
Ujhelyi M (zooo). Social organization as a factor in the origins of language and music. In NL
wallin, B Merker and s Brown, eds. The origins of music,pp.I25-34.MIT press, cambridge,
MA. D¡'ip lvlelche: -, l
wallin NL, Merker B and Brown S (zooo). The origins of music. MIT press, cambridge, MA.
in science?
Watt RI and Ash RL (tgq8). A psychological investigation of meaning in music. Musicae
Scientiae, 2, 33-54.
wright AA, Rivera Jf, Hulse sH, Shyan M and Neiworth |] (zooo). Music perception and Yik I'Iuniz (\-.\l r I
octave generalization in rhesus monkeys. Journal of Experimental Psychology: General, rz9, rr'hat ] rranted iL. .1.-*
297-307. horr' alrvays relar.a :.
ZangwiIl N (zoo4). Against emotion: Hanslick was right about music. British Journal of is the nature of re::.
Aestherics,44, 29-43. like James Gibscn. I:
Zatotre R) and Halpern AR (rqq¡). Effect of unilateral temporal-iobe excision on perceprion has nol becone :l:.
and imagery of songs. Neuropsychologia, 31, 227-32.
is the sum of nar:-, -:
Zatorre Rf and Peretz I (zoor). The biological foundations of music. Annals of the New york
Academy of Sciences,93o, z1t-99.
:ave a name back il.:
\lv rr'ork has ::.:-
5oth directir- ¿n; i:-.i
.lsc¡ interestei r¡: ¿-.:
non. betl een ¿t ¿n:
;.:d complete .¿;:. :
,rt go in anLrth3: ,'.:','
::rentists dc¡. O::.. --,
-, 'lll\ LU t.¡-. -:-
- uL1---- --.-
,::iier investigations.
:- 'r. r. _3io-6.
?:'.:i:ology of Music,
CHAPTER 18
i,:i-rounding music:
THE, MYSTERY OF
REPRESENTATION:
A CONVERSATION
,-:. Ir: PN luslin and
,tr-.Jord University
WITH VIK MUNIZ
:...'-re: An explor-
:¡ .nd music. In NL
I ?:e:s. Cambridge,
D¡vrp Melcher (D.M.): How has your work been informed by your interest
; --:.nbridge, MA. in science?
,:: nusic. Musicae
vik Muniz (v.M.): I am interested in science in part because, when I was 16' that is
::-. :erception and
:: .:
what I wanted to do-I wanted to be a psychologist. But also my interests wefe some-
. : General, tz9,
how always related to art, and I was thinking aboul wlry people saw images. what
g'.
by focusing on mind. Scientists define a tangible world with formulae and symbols - (\::Cl úL( - -: i--
while artists deal with a world made of sensations and feelings, through the making i:laal¡nr!:1 l,:1:-: ::.:
of tangible products. 'r:3'.--Si!r-1.
D.M.: Let's talk about Atalanta and Hippomenes, affer Guido Reni (zoo5-6), which is i \ f T ..-- .--
! .-\1,. I t-.t t-1. _ -
part ofyour'pictures ofjunk'series (Fig. r8.r). From a distance, it appears as a copy
tlañ io i\lsra i:--:
of Reni's large canvas dating fromt6zz-5, emphasizing the outline of the two char- -- I--t -:-L ---
acters and their shadows. Upon moving closer, however, the image decomposes into
¡s-, c}:tr,rr5. --,:--: -
the countless objects that form it as if it were a giant collection of worthless trash.
taj :epf esan:;i:,- ,=
Wittily quoting from history of art at large, from Renaissance to Impressionism
(which prescribed a certain distance from the canvas in order to recognize what its
!.\1.:Ytru -t¿:. a )*
fragmented surface represented), you managed to shift our attention from the sub-
;,-'me. fi¡n .:::
ject matter to the process of vision and of drawing. Humble bins, tyres, crushed cans,
ropes, broken motors, and tubing-in short the debris resulting from mechanical
work-give visual existence to a subject matter of the highest cultural content: a
legend from Greek mlthology, painted for a rich patron, which the intelligentsia of the
seventeenth century would pride itself in recognizing. In describing your work, you
have said:
Ir,\Í.: \I.Lr.r:ls . : :.
;1,:u.1s ir the ..:-, :--
ilLlSi áDCleIlI f .l:i,lll :
, :rr abLlin'tLr s-- :., . .
nulae and symbols I exaggerate the gimmick in my work because I want to engage the viewer with some kind of
rrough the making mechanical image that is almost inescapable, where they not only see the artist, but they feel
the vision.
(Magill zooo)
:oo5-6), which is
V.M.: I am involved in a level of art and representation that is very raw-I try to go
: ¿Fpears as a copy
back to basic things. There are thinkers like Arnheim or Gombrich who were trying
le of the two char-
to deal with these basic questions about representation from the point of view of
ie ,lecomposes into
psychology. And I also thought, in my work, that I would need to do something related
¡i rvorthless trash.
to representation.
to Impressionism
recognize what its
D.M.: You have suggested that, historically, two types of art have developed: 'Art that
:rrrn lrom the sub-
comes from embodiment, which is theatre, dance and music' and art that's a graphi-
:,res. crushed cans,
cal projection like drawing. These two arts were probably developed by primitive
I trom mechanical
shamans. They understood theywere exelcising a kind of power' (Magill 2ooo). How
;¡ltural content: a
does your work tap into these two forms of art?
ur:eiLigentsia ofthe
nS lour work, you
V.M.: Because I see these as complementary primitive forms of symbolic exchange'
Even though my work is predominantly involved with the projective kind of repre-
.: at something and sentation, I have always tried to incorporate concepts of theatre and performance
:i -r. is to be able to art in my work without having to fuse them into a seamless indiscernible format like
r¿ iime r-ou're able to
the motion picture.
i. ,ike Buster Keaton.
D.M.: Millions of people watched your work in which an aeroplane drew pictures of
clouds in the sky above NewYork City (zoor). The cloud-watching game is one of the
most ancient pastimes, explored also in your series 'Equivalents' (rSS¡).What does
our ability to see recognizable shapes in clouds tell us about how our mind works?
V.M.: Clouds have been a recurring theme in my work for zo years. In 'Equivalents"
I made 'cotton clouds' that resembled something like a man rowing a boat, or a kitty
cat. We can only read one meaning at a time. If you see the cat, you lose the cotton.
Ifyou see the cotton, you lose the cloud. Like Necker's cube, you see one interpretation
or the other. J.T. Mitchell called this 'multistable images'. This was used in the early
Renaissance, with the goal of creating images that had multiple interpretations.
In the of clouds, in the sky, it was even more interesting. I tried to make a
case
drawing of a cloud, in the sky, that looked exactly like a cloud. But even so, it did not
work, because with a small change over time in the shape of the cloud, people could
not help seeing meaning, a baseball glove or even a phallus, in something that was
supposed to be self-referential!
D.M.: At the same time, you have also noted how shamans, priests, artists, and con
men have used these'tricks' to evoke power (Magill zooo). You have said :
after Guido I don't want to amaze you with my powers to fool you. I want to make you aware of how
much you want to believe in the image to be conscious of the measure of your own belief,
354 VIK MI]NIZ AND DAVID MELCHER
rather than of my capacity to fool you. You see it, but at the same time you see how it works.
I have been called an iliusionist, but I have always considered myself a tr,visted kind of realist
(Magill zooo)
V.M.: The visual cortex is a sucker-it sees things even when they are not there. We
have this tendency to see things; we developed recognition from a very primitive
stage as a tool to see shapes for food or survival. And the visual system still fancies the
shape of the prey in everything it focuses on. We have become very good at detecting
patterns that look like something.
I have a three-year-old, who has been teaching me for two years now about vision
science because she tells me that 'this looks like this' or that, and it is amazing the
power of associations, to match patterns to recognition.
D.M.: This ability to links our current experience with our memories.
see patterns
Your work has also explored the role of emotion and memory in lived experience. Your
work Individuals (ry92) is a series of photographs of lumps of Plasticine modelling
clay, yet they seem to convey emotions. Feeling is an important theme in your work.
To quote from one ofyour essays:
I remember as a child, running through bed sheets hung out to dry, the exhilarating feeling
of moving in space with my entire body. I needed the bed sheets not only as a surface of
contact but also as a cognitive instrument that revealed systematically the successiveness of
surfaces. Like the peek-a-boo game that teaches a child the parent will always be there, rev-
elation is a tool to generate the emotions that punctuate the secret clock of our lives. We
remember the past through these moments when something was revealed, and surprised, we
became full of emotion. These two things, surprise and emotion, go hand in hand in such a way
that we sometimes don't even notice them. To be able to draw attention to the importance
of these things, we must understand the mechanics of surprise and subvert it, positioning the
audience at the same time as a victim and analyst of the experience.
(Muniz and Eccher zoo3)
V.M.: i was talking the other day with a chef and she said: 'it is no good writing
about cooking', and maybe it is no good writing about painting either. There is a
big gap between what you can feel and what you can write or explain by language
alone. It is interesting to try to create works that bridge this gap between sensation
and knowledge.
D.M.: To bridge this gap between sensation and knowledge is difficult, if vision seems
so fast and effortless. You have said that:
I try to slow down the perceptual input of the image in my photographs so that you actually
look at them . . . What I want to show you is that there is a machine, in the back of your head.
I don't want to show you exactly how it works; I want you to guess a little bit.
(Magill zooo)
:: l:t.-Ii
V.M.: The physiology of the eye and the visual cortex is fascinating. I am surprised
when people want to become artists but are not interested in the basics about how
-:l ', !) -Lf -!\,: f:
THE MYSTERY OF REPRESENTATION 355
r --'r see how it works. light behaves, how vision works. Muyb. 9o% of what we know about vision is rela-
".si:d kind ofrealist tively recent. I am fascinated about the way the eye works, because the eye is very
{Magill zooo) primitive and is not so good at making optical images. Our eyes are not as good as
',"¡¡e not there. We birds' eyes or many other animals. The fovea, with best vision, covers just a few
degrees and all the rest is peripheral, it is blurry. It is only by moving the eyes around
:i ¿ \'ery primitive
that you have the impression of a continuous whole. Partly because our eyes are so
item still fancies the
poorly designed, this creates a narrative through the series of saccades. One of the
:' Etrod at detecting
most impressive things is our ability to recognize faces'
:s :iLr\\'abOUt viSiOn
j it rs amazing D.M.: A classic image from vision science is Yarbus's (rg6o) illustration of fhe gaze
the
pattern when looking at a picture of a face. Most of the fixations alternate between
the eyes and the mouth. This reminds us of a quote about your interest in
face perception:
,tth our memories.
:j erperience. Your When I became involved r,vith portraiture, I was more fascinated by the fact that we can rec-
-¿strcine modelling ognize one face from millions of others than an1'thing else. After working for so long with
:¿ne rn r.our work. recognition, I decided to explore the subject in which this phenomenon is exercised with the
greatest skill by our brains. How do we recognize a face? The eye wanders through a face,
scanning for familiarity. It goes from point to point, making every face a narrative. Every face
. .:',:ilarating feeling is a story.
- r-i as a surface of (Muniz and Eccher zoo3)
:::¿ successiveness of
; ',1 l'!'s be there, rev- V.M.: The development of narratives, through saccades that can also be remem-
" ¡.. r¡i our lives. We bered, is one of the most important elements in the creation of consciousness in the
r'¿. :nd surprised, we human mind. We have this huge visual cortex devoted to analysing stimuli-that is
r-- L:¡d in such a way our true 'eye'. This inabilityto see ever¡hing in focus introduces a concept of narrative
:: :o the importance
and attention, which is important.
:r: :t. positioning the
D.M.: You have suggested that the human intellect has, to some extent, evolved,
:.-: and Eccher zoo3)
based on this limit of clear vision in the fovea and the need to move the eyes. Is the nar-
no good writing
s rative aspect of vision important also for the creation of art?
¡ :ither. There is a
,.pl¿in br'language V.M.: My work Pictures of magazines explores this 'saccadic narrative'. There are a
bet-'r een sensation series of dots. The viewer stands at a distance where the picture covers the entire visual
field, and this means that each dot covers about 3olo of the entire visual field.
. rlt. if r.ision seems D.M.: You have said that this idea was inspired by how people look at paintings
in museums:
.i :hat 1.ou actually they approach the picture and position themselves so that it comfortably fills up their visual
:: back ofyour head. field, but is always close enough so they can sense the texture of the paint itself. The moment
: brt. when the image dissolves back into matter is as revealing as the moment dabs of paint become
iI{agill zooo) the likeness of an angei or a fish. These are the moments that contain in their transcendence,
the very nature of representation.
rrE. I am surprised (Muniz and Eccher zoo3)
b¿-.ics about how
But in your work, there are not just dabs of paint.
356 VIK MUNIZ AND DAVID MELCHER
V.M.: The dots have an identity themselves: pictures, faces, houses, body parts, let-
ters, or words. This gives your saccades target points, where the local image makes
you pause longer; it slows down your saccades. The picture refuses a smooth read-
ing. As your eyes are sweeping the picture, your eyes have to 'stop', and so the picture
becomes problematic. This is something I try to do in much of my work: to create
little problems that make you 'feel'what is going on. It is like finding a little bone in
the fish that makes you stop andrealizethat, right now, you are eating fish.
D.M.: So this is the idea of making perception 'problematic'. You have said that
'Art is somehow like brain-science; you only get to know something works by looking
at things that have stopped working' (Galassi and Muniz zoo4).
V.M.: When I was little, electricity shortages were frequent in my native Sáo Paulo
and it was only then that we had a chance to evaluate our dependency on machines.
Only once it had stopped working. It had a different effect on us than it had on my
grandmother who lived most of her life without electricity. In a sense, her numbness
to technology and her failure to adapt made her successful in those situations; the
frustrated rest of us just ended up with lots of things to think about it.
D.M.: Having something to think about does not necessarily lead to frustration, but
there can also be a pleasure to being fooled. You have said that:
The logical blur that one experiences in front of an illusionistic picture is similar to what one
experiences after hearing a joke. Suddenly there is an enormous vacuum in your mind that
the cognitive apparatils registers as pleasurable because that's what the mind iikes to do: to fill
these places with wild and abstract thoughts.
(Muniz and Stainback 1998)
V.M.: How would you define success in a scientific experiment? There's always a
loose measure of satisfaction that, while rewarding you with some merit and sense
of direction, always leaves you with the necessary feeling of incompletion to keep on
working. I think my work becomes successful the moment when I can identift patterns
on the viewers'reactions to it that can be used in further research.
D.M.: Keeping with the theme of scientific experiments, your work also explores
the nature of scientific images themselves. For example, your series 'Principia' involved
looking at pseudo-scientific images through an apparatus. What motivated this work?
V.M.: What you see when you look through the stereoscope is a series of things that
'look like' the kind of specialized imagery that science generally gives us. The viewers
tend to take these absurd concoctions seriously because of the 'scientific' way they are
THE MYSTERY OF REPRESENTATION 357
,rs. body parts, let- displayed. Scientific knowledge is worthless if we cannot find ways to demonstrate,
-,rcal image makes sell, and disseminate it, and that usually comes with an unavoidable dose of rhetoric.
se. ¿ smooth read-
. ¡nd so the picture
:', rrork: to create
::r:. ¿ little bone in RErnnnNcns
_-:-L
--11i1¡ll.
Galassi P and Muniz V (zoo4). Natura pictrix: Vik Muniz with Peter Galassi. Natura pictrix:
interviews and essays, pp. 88-sz. Edgewood Press, New York.
:'. -- har-e said that
Magill M (zooo). Vik Muniz. Bomb Magazine,T3 (Fallzooo).
: ',"orks by looking Muniz V and Eccher D (zoo3). Vik Muniz: exhibition catalog. Museum of Contemporary Art,
Rome (September zoo3-January zoo4).
Muniz V and Stainback CA (rqqS). Vik Muniz and Charles Ashley Stainback: a dialogue.
', :::iile Sáo Paulo ln Seeing is believing (exhibition catalogue). Arena Editions, Verona.
.:¡ on machines.
.:r::: it had on my
: ia. ner numbness
-,- .: sttuations; the
':
. -: Stainbackr99B)
- ¡ ,:is a measure of
. ,rere's always a
r: nerit and sense
.:-etitrn to keep on
:r, i:entif.patterns
,,
-:-< also explores
:':-rcipia' involved
-:,-, ¿ted this work?
PICTORIAL
CUES IN ART
A\TD IN VISUAL
PERCEPTION
INrnooucrroN: wHY Do
wE LOOK AT prCrUnnS?
Anrrsrs have been looking at the world for thousands of years, and thus paintings
and drawings can be considered to form a 40 ooo-year-old corpus of experimental
psychologyofperception. Through observation andtrial-and-errortheyhave exploited
the principles of how our brains interpret the input from the retina, giving priority to
only certain regularities of the visual pattem. Thus, a study of pictorial cues can tell us
about the way that the brain recognizes objects, understands spatial depth, and uses
illumination information in natural environments. Conversely, a better understand-
ing of visual perception may help to explain the effectiveness of certain techniques
used by artists. Therefore, this essay will focus on some basic techniques in pictorial
depiction that allow blobs of paint or charcoal marks to evoke objects, depth, move-
ment, transparency, illumination, and reflection. The development of these pictorial
36o- DAVID MELCHER AND PATRICK CAVANAGH
t;;
The visua system is incredibly adept at reconstructing from a photograph the fLatness of objects such
as a photograph or even a rubber mask. However, a photograph ofa folded, crumpled or bent image no
Longer supports the percept of the veridical shape of the image. In this case, a simp e rubber mask that
is twisted and fo ded takes on a distorted, almost horrific, appearance.
PICTORIAL CUES IN ART AND IN VISUAL PERCEPTION 361
ies about the neuro- lack of three-dimensionality in pictures. A world without tolerance to flatness
would contain no paintings, posters, teleüsions, or cinema, but would instead be
i of electromagnetic filled with statues of people and sculptures of scenes, in place of pictures and
ofthe eye and then, photographs.
hring that is sent to This essay will focus on the techniques used by artists to depict recognizable objects
the pattern of light and scenes in flat representations. Of particular interest is the many ways in which
nts. but even a flat, pictures deviate from'correct' optical representations. Our visual system has a lifetime
¡itrn of objects and of experience with real scenes, but the same brain is able to make sense of pictures
rtects and scenes, nor which, in many ways, look nothing like optically produced images.
nirror. Instead, the For example, the style of film-making known as the 'Hollywood style', which has
',-rerring point of the developed over the past century (Bordwell et al. 1985) aims to shoot and edit the scenes
.. of photographs or in such a way that the lighting, placement, and use of cameras, set design, and editing is
r.9.1 ). Across various not noticed by the viewer (Reisz and Miller 1968; Bordwell et al. 1985; Cutting zoo5).
'"ura 1994), there is a This focused approach to narrative film-making does not attract attention to the pro-
: rnsensitivity to the cess itself,but rather the story. The principles that allow for seamless perception of
r- ihe foundations of such an artificial medium are critical for understanding the mind. In order to go
'l.in-to overcomethe unnoticed, film techniques must work within the constraints of the human visual
system. Thus, movies can tell us something also about how we perceive the real world,
even ifthe scene in question (such as a flying superhero) could not exist in everydaylife.
Similarly, artists working with static media such as photography, drawing, or painting
can choose to portray pictorial information-such as object shape and colour, light-
ing, depth, and reflection-rather than drawing attention to their technique itself.
It is useful to investigate the nature of pictorial cues that maximize the ability to
recognize objects and scenes in order to better appreciate the deviations from these
rules. It is this ability of art to reproduce certain aspects of natural scenes without
merely copying them that provides insight into the workings of the human perceptual
system: an understanding of the 'psychophysics' of the brain rather than the physics
ofoptics.
have all been shown to interpret line drawings as objects argues strongly against ¡.:ikr¡q eram:-== ,
children's pictures, and from an art historical point of view-considering that the lrloreoler.::::.
artists' training has traditionally included sketching from life as one of the most t:a',,, inq of a t,- ,:, .
important exercises. When an artist is able to depict the form of an object, this :l:spine lrr¡i'i:nrl
signifies an understanding (implicit or explicit) of the key contours that must be - -*--l-
:.'-1-.1frlc. at_^ _,-
LI-ú - _-- _
extracted by the visual system in the brain in order to recognize the three-dimensional -:l,,olr es 'atfi¡¡;:::,:
structure of a particular object. The process of drawing is essentially a process of ,\tl,n our rish- :.*
selection. Extracting all lines from an image would create a confusing mess, which lacks :nted torr'arús :.t: :
the economy of a child's drawing or the poetry of an artist's sketch (Fig. t9.z). The art- :¡:ron bl the :-::: l
ists of the Palaeolithic era discovered the essentials of the principles of shape-by- :: lraph and th: ,. :
contour, the rules of which are still being investigated by scientists of the twenty-first --bject. Sonte p:.,.:
century. As such, artworks can be viewed as scientific data: perhaps by studying the :rsorder k¡or".'r :,
use of lines by artists, scientists may learn the explicit rules of this implicit, secret ,: :he patient. tl..',
knowledge. a,,'aglasses, he .-: .'.
Boundaries are critically important not just for vision but also for haptics, under- :;ir. The patie:tl ;.
standing objects through touch. Lines perceived through the eyes can support the rec- .,-r use it.r
ognition of objects that have only been experienced through touch. One of the most Or-erall. rese.rr.a
::,:.1 of er-okins ¡ .,,
:¡¡t'orks enqaEe l,-
L--aude Heath .;¿ I
:¡,; discussine the ..
.,:,1ded, usinq c-,rl-,. '
p-ece of Blu-Ta;k.
.:uplores the obie;:
;¿relul to mainta-:
:andjournels. T:r
:eal-the contou: i
rrread out' in rel¡
Fi1.19.2 Photograph of the sculpture 'Saint Francis as Brother Wind' by Fiorenzo Interestinglr'.; r.:,
Bacci (San Damiano, Assisi), a version of the same modified using the 'extract \h-slobodskr'.:- _,
contour'function of Photoshop, a version ofthe same using the'sketch/chalk and -: u-ith its abrli¡':¡, ;:,
S::ri1ar1y, phenomcn,
charcoal effect' function of Photoshop. :rrironment full oi :
[photograph by Nicoia De Pisapia and modified versions by Francesca Bacci) :' beginning to rer ¡¿, .:
PICTORIAL CUES IN ART AND IN VISUAL PERCEPTION 363
:ues strongly against striking examples of this multisensory representation of objects is the case of Sidney
Bradford, who acquired vision at age 56 after a life of blindness (Gregory zoo+). The
:ritial stages of üsual vision scientist Richard Gregory interviewed him shortly after the surgery, but
:. and depth, a scien- suspected a hoax when the patient immediately read the time of the clock on the wall.
¿e to Torsten Wiesel It turned out that Bradford could üsually recognize all objects-and only those
l: intbrmation about objects-that he had previously recognized through touch (see Bacci, Chapter Z, this
¿nd uninformative. volume). The similarity between identif,ing an object by seeing its drawn contours
:r of these transitions and feeling the contours with the eyes closed has been confirmed in several studies
ir¡ivs and transitions using neuroimaging (see Amedi et al., Chapter 23, this volume). Indeed, there are many
iimensional shapes. blind artists that can create images on flat surfaces that capture their experience of
:.nt from looking at objects they have only touched and are interpretable to the sighted.
c"-,nsidering that the Moreover, there are various lines of evidence to show that viewing the outline
ai one of the most drawing of a tool can lead to activation in the brain's regions involved in planning
r: of an object, this grasping movements, even if there is no motor response required for the task (for
r:ours that must be example, Chao and Martin zooo; Devlin et al. zooz). One line of experimental evidence
le three-dimensional involves 'affordance' effects, such as the finding that we are quicker to press a button
entially a process of with our right hand when responding to a photograph of a cup whose handle is ori-
-rq mess, which lacks ented towards the right (Craighero et al. 1998; Tucker and Ellis 1998). The priming of
n Fie. r9.z). The art- action by the right hand, in that experiment, occurs even though the stimulus is a pho-
:rciples of shape-by- tograph and the observer is making a judgement about the name, not the shape, of the
,:s of the twenty-first object. Some patients with damage to a specific area of their frontal lobes develop a
raps b,v studying the disorder known as 'utilization behaviour' (Lhermitte t9s3). If you place a cup in front
this implicit, secret of the patient, they will pick it up and attempt to drink out of it. If the patient sees
eyeglasses, he or she will immediately put them on, even if akeady wearing another
;br haptics, under-
--, pair. The patient cannot resist a bowl of fruit, they feel that the object 'orders' them
.an support the rec- to use it.t
lch. One of the most Overall, research suggests that artworks, even simple line drawings, have the poten-
tial of evoking a visual-motor response. This raises the empirical question of which
artworks engage the observer's brain in this way and why. Drawings by the artist
Claude Heath (see Bacci, Chapter 7, this volume) provide an interesting starting point
for discussing the meaning of drawn contours. Heath has drawn objects while blind-
folded, using only the sense of touch. He places a small point of reference, such as a
piece ofBlu-Tack, on the object and another on the piece ofpaper. Subsequently, he
explores the object with one hand while simultaneously drawing with the other, being
careful to maintain exactly the same distance from the reference points in both of his
hand journeys. This exercise reminds us that drawings tend to use lines which are not
real-the contour simply gives the useful information about where the object ceases to
'spread out' in relation to the centre of our gaze at that moment. Movement by the
[Vind' by Fiorenzo ' Interestingly, a recent study ofobesity suggests that food also tempts our hands as soon as we see
ng the'extract it (Myslobodsky, zoo3). According to the Stoics, the perception of an object was automatically tied
up with its ability to create an impulse towards or away from that object (phantasia hormetiké).
e 'sketch/chalk and
Similarly, phenomenological philosophy has stressed the 'embodied' nature of humans üthin an
environment fu1l of objects that are 'at-hand' and suggest action. Recent work in the study ofvision
":': :'s by Francesca Bacci) is beginning to reveal these links between vision, the other senses and the body.
364 DAVID MELCHER AND PATRICK CAVANAGH
observer or the object changes the project of this contour in two dimensions. For our intormation can ¡t
visual system, this'viewpoint specific' representation of shape is useful, but our brain thev are presentei
also appears to calculate an estimate of the 'unseen', real three-dimensional shape :each conscious ¿',.
of the object. Why? Because visual contours are not interesting, per se, for actions Thus, artists :r:.'''
such as grasping, which are based on where to put the hand in relation to the three-di- cietailed pathrra-, .
mensional shape of the object. These 'touch contours' are not viewpoint-specific. does not depenJ '::
\^/hen you reach to grab a cup, or to embrace a lover, your grasp extends around into in the depiction. T
the unseen, but'perceived'limits of the body. When Heath closes his eyes and draws recognition in t¡. :
what he feels with his hands, he may be trying to access the information that we use for i:r random paftir::
action. Interestingly, the artist uses Blu-Tack to anchor his hand at a fixed point on the col::.
stars: rr.hen,
paper. Does this mean that he is using a hand-centred representation of space with their eyes. On ¡ ;-"
his drawing hand to draw what he is perceiving with his other, feeling hand? The shape this capacin- as :::
of the lines suggests a three-dimensionalitythat is difficult to translate into the con- stare at naturai:,:
tour-based medium of drawing. That may be one reason why Heath's drawings are Giovan Ba:ti.:: "
One reason that coarse information might be particularly useful is time. There are l¿ces in part,¡-,::
fundamental constraints on how biological systems can encode and extract informa- i¿ces er-en rrhe :'.,, .
tion over time. With our big, energy-consuming brains processing all of these minute, :oo5 ). In surn. i-r;
fine details, the whole process can get pretty slow. Thus, it seems that in addition to the The intluen:-:
'fine detail' pathway there is also a visual 'fast track' that can begin more quickly to ¡rtracted bv c¡::.-
influence behaviour based on the coarse pattern without waiting for the exact details to -or us the mLr:i ',\:.
arrive. Presumably, this design allows for quick responses to salient events, such as the ',trll be our sra:i.:
arrival of a speeding bus or the presence of a rattlesnake in the hiking path. -or rapid r-isua. :"
Using sophisticated laboratory techniques, it is possible to measure the processing sLlpport Gom:r,¡:
of coarse visual information. One finding from these studies is that certain patterns .hat appear de:,-:
evoke emotional responses and that it is the coarse information that plays the pre- er-olutionan' a;',':
dominant role (for review, see Vuilleumier and Pourtois zooT).In fact, this coarse despite their ini:ir
PICTORIAL CUES IN ART AND IN VISUAL PERCEPTION 365
dimensions. For our information can affect our responses even to stimuli that we do not really see, when
*seful, but our brain they are presented in the laboratory under conditions in which the stimulus does not
¿-Jimensional shape reach conscious awareness.
!. ler se, for actions Thus, artists have at their disposal at least two routes to visual recognition: the slow,
¿:itrn to the three-di- detailed pathway and the fast but coarse pathway. Face recognition, for example,
:ljeu?oint-specific. does not depend on seeing local details or even the presence of eyes, nose, and mouth
¿:ttends around into in the depiction. The right pattern of light and dark patches is sufficient to evoke face
.s hrs e1's5 and draws recognition in the brain. In fact, people are surprisingly adept at seeing faces or animals
-r¡¡ion that we use for in random patterns like clouds. At night, our ancestors played a similar game with the
:.: :1ixed point on the stars: when, connecting mentally the celestial dots, archers or bears appeared before
::aiion of space with their eyes. On a clear night, even the man on the moon shows his face. Leonardo saw
-ns hand? The shape this capacity as the basis of artistic creativity and recommended to his students to
¿:rslate into the con- stare at natural patterns and try to find people, animals, and scenes hidden inside.
leath's drawings are Giovan Battista Alberti noted that 'nature herself seems to delight in painting, for in
the cut faces of marble she often paints centaurs and faces of bearded and curly headed
kings' (Della Pittura, Book II, z8 [author's translation]). People flock to see such
miracles of nature, particularly when the subject matter is religious. Recent examples
include an appearance by the Virgin Mary on a tortilla, which drew tens of thousands
of pilgrims, or her miraculous appearance on a toasted cheese sandwich, which sold on
eBay for an incredible US $28 ooo. Likewise, the devil has appeared on the Canadian
I GNI TI ON
dollar bill and in a smoke pattern on the CNN news channel (smoke, like clouds, is a
common medium for this phenomenon).
:¿inlr'not the whole The fact that people tend to see people or animals in naturally occurring patterns is
I t-rur g?Z€, the world called pareidolia (from the Greek para (beyond, over) and eidolon (image, percept)).
l leripheral areas of The fact that pareidolia tends to involve animals and faces may be related to the way
:. usetu1 for tasks like that the visual system quickly and automatically identifies those stimuli (Melcher and
;iation gives a more Bacci zoo8). 'Rapid visual categorization' is the term given by scientists to describe the
.r i'in g. B ecause visual ability of people in a laboratory setting to discriminate whether or not an animal is
drrectly at objects of present in a picture (Thorpe etal. ry96 Rousselet et al. zooz) . This ability appears to be
sreat deal about the largely automatic, since it requires little or no focused attention and occurs even for
;rere the 'fine detail' stimuli viewed out of the corner of the eye. Rapid categorization seems to work par-
ticularlywell with animals and people. Moreover, a number of studies have shown that
:;L is time. There are faces in particular can be detected quickly: our brain appears hardwired to process
¡nd extract informa- faces even when we do not see them clearly or even consciously (for review, see Johnson
ls all of these minute, zoo5). In sum, the visual system wantsto see faces and animals.
hat in addition to the The influential art historian Ernst Gombrich suggested that the visual system is
:gin more quickly to attracted by certain patterns, since'the greater the biological relevance an object has
,-': the exact details to for us the more will we be attuned to its recognition - and the more tolerant therefore
ni e\-ents, such as the will be our standards of formal correspondence' (1963, pp. 6-7). The scientific evidence
r:g path. for rapid visual categorization, as well as the widespread phenomenon of pareidolia,
::sure the processing support Gombrich's claim of perceptual tuning and our ability to recognize objects
'.hat certain patterns that appear deformed from the real thing. As Gombrich notes, there are obvious
r: üat plays the pre- evolutionary advantages to a finely tuned sensitivity to face-like patterns. Infants,
. In tact, this coarse despite their initially poor visual acuity, are primed to recognize face-like patterns and
366 DAYID MELCHER AND PATRICK CAVANAGH
this aids their social interaction with caregivers (Johnson zoo5). Similarly, the survival
of our ancestors depended on their ability to quickly recognize the presence of an
animal in the environment. Poor 'recognizers', who failed to react to faces or preda-
tors, would not have been likely to pass along their genes. There is a fine line, of course,
between recognizing subtle patterns and seeing objects that are not there. Interestingly,
the subject matter of hallucinations tends to follow the same pattern as pareidolia.
A recent survey of the phenomenological experience of Parkinson's disease sufferers
found that a majority of hallucinations (over 8o%) involved people, disembodied
faces, or animals (Barnes and David zoor).
Artists may have taken advantage of this quick and coarse processing system in sev-
eral ways. First, artists can provide only the 'sketch' and expect the brain to fill in the
details. Cubism and Impressionism, for example, rely on the brain to reconstruct
scenes suggested by a pattern, by exploiting perceptual memory (Cavanagh zooS). The
ability olmodern artists such as Picasso, Severini, or Miró to distort images while
still evoking perception of faces demonstrates the brain's 'holistic' processing of
faces based on the gestalt-like pattern, rather than specific details (Figs. 19.3 and 19'4)'
Fis. 19.3 Gino Severini, Ritratto di Madame M.5.,1913-15. (see Colour plate 5)
(Mart lMuseum, Rovereto)
Understanding this work by Severini requires a series of complex perceptual processes in which
elements are qrouped together in order to aLlow the recognition that this is a portrait of Madame M.S.
PICTORIAL CUES IN ART AND IN VISUAL PERCEPTION 367
Recognizing the identity of patterns is, after all, one of the basic functions of our per-
ceptual system, so it makes sense that many people would enjoy pushing the system to
its limits as a form of perceptual problem solving.
Second, it has been hlpothesized that artists have used coarse information to spe-
cifically target emotional processing (Cavanagh zoo5). Coarse information, such as
Mooney patches of colour, is sufficient to trigger emotional responses in the brain even
without awareness, while more detailed information is needed for recognition of the
specific face (Vuilleumier et al. zoo3). A painting with numerous distracting or, worse,
:e Colour plate 5) inaccurate local details might hinder the rapid understanding of the subject matter by
'.' :: t,luseum, Rovereto) the slow pathway. The coarse information present in the work, on the contrary, finds
-:.r:. . lvhiCh rapid access to the emotions and areas of the brain invoived in understanding the
'-', -. :' Madame l\4.S. context and overall meaning of a scene. What happens, for example, with paintings
368 DAVID MELCHER AND PATRICK CAVANAGH
slavishly obel se .
Cuns FoR DEPTH PERcEPTToN out drawing lines ,-
'
depth in pictures have been addressed at length elsewhere (Kuboly rg86; Maffei and visual perceptit-rl.
Fiorentini 1995; Livingstone zooz). Not surprisingly, many of the techniques that matical perspecti,.
have been developed to portray depth take advantage ofoptics and the physics ofhow Leonardo \\¡as th¡r:
images are projected into our eyes. As such, they are more 'discoveries of physics'than linear perspectivc '
principles ofvisual neuroscience. There are, however, a number of phenomena related graphs, unless dt r
to depth cues in pictures that may tell us something about how the brain processes view the world r,, -:
the input from the eye. culmination oice::
First, in the majority of figurative artworks making use of perspective there are local imitative depictic'r
depth cues that do not conform with the system used in other areas of the picture tography is not d-¡;
(Figs. r9.5 and 19.6). These local inconsistencies go largely unnoticed. Even in Renais- raphy also rer-ealed
sance painting, perspective was usuallylocallylinear for particular regions ofthe paint- show that a paitrtec
ing, not consistent globally. For most artists, linear perspective was simply a tool for of the same scene :
telling the story, or a device to show their cultural refinement, rather than a rule to time how hearilr' :
PICTORIAL CUES IN ART AND IN VISUAL PERCEPTION
Local perspective cues are sufficient to give the perception of depth, even when the overall
construction of the scene contains multiple, incompatible vanishing points.
Nfovr
Ln,ike ¿ r¿1:::r:
Fig. 19.6 Giorgio De Chirico, Piazza d'ltalia (Souvenir d'ltalie), 1924-25. t.. ihe iiani";--
(Mart lVuseum, Rovereto) ili:¡tor l¡'¡:: i
The shadows in this work are oriented in varying directions, consistent with the presence of more than :j t; lh,ng.',t= ::
one lighting source, yet this confusing feature is not typically noted by observers. -- I ..-- -;- - -.,-
:--U -':.!,1!'--r.,:
:-¡:n a¡1 ,.t"¡:-: - :
;- .' --
not segregate into figure and ground (such as in works by Mondrian and Pollock). sive the brain e
PICTORIAL CUES IN ART AND IN VISUAL PERCEPTION 371
One can argue that'the trouble with optics is the trouble with photography: it's not real
enough, it's not true enough to lived experience' (Hockney zoo5). \A/hile this statement
is open to interpretation, it does point out the fact that pictorial representation, when
based on an artist's sensorimotor interaction with the world, necessarily differs from a
single snapshot. Of course, photographers know that there is no such thing as 'the
photograph' of a scene: even when a print comes from an unretouched single negative,
it still reflects a series of decisions about focus, focal length, aperture, lighting, and
composition. In a painting, artists have even greater licence to manipulate the observ-
er's neural response by, for example, changing the surface properties, adding new
colours, or enhancing the cues to movement.
Unlike a painting, in cinema the picture changes rapidly. Why are we not bothered
1924-25. by the many 'cuts' that suddenly change the picture shown during the movie? The
','¿a \¡useum, Rovereto) director ]ohn Huston claimed that:
: :':::.Ce Of mOre than
all the things we have laboriously learned to do with film, were already part of the physiological
:-:
and psychological experience of man before film was invented . . . Move your eyes, quickly,
from an object on one side ofthe room to an object on the other side. In a film you would use
a cut. . . . in moving your head from one side fto the other].
In addition to limits on minimum duration, cuts must also fulfil other requirements to
avoid calling attention to their existence (Hochberg and Brooks 1996). One necessityis
that the scenes are very different, from different angles ofdifferent objects and, when
possible, with a variation in the lighting. These conditions, of course, mimic the differ- What our eyes det,
ences in fixations that tend to occur after saccadic eye movements. Thus, one can con- of light is imporra
clude that a major factor in whether or not we notice cuts in films is the degree to which there is a basic di
those changes in image match what happens when we move our eyes in natural scenes cerned with /i*lr c
(Hochberg and Brooks 1996; Cutting zoo5). tion are most imp(
A third issue is viewpoint. This issue is most clear in photography and cinema, where schools (during *
an explicit choice is made about the location ofthe camera, but drawings and paintings painting, influen'.
may take advantage of the same techniques. A simple example is whether to film a region ofthe r.or:
mixed group of adults and children from the eye height of the child or the adult. the use of colour,
Looking down is equated with dominance, while looking up with a more submissive approach that an:
role (Messaris rgg+). sidney Lumet's t957 film Twelve Angry Men provides a useful perception oicoLr
example of how üewpoint can be manipulated for narrative effect without the audi- ofcentral ltalr- s;
ence noticing (Cutting zoo5). The first third is filmed from eye height (for a standing traying the sohd:¡
person), enabling a wider view of who is sitting at the table. Then in the second third, colour, was a sec[
the camera is placed at sitting eye height, where the discussion between the jurors is objects. In other in
taking place. Memory allows us to remember who is where in the room. In the final that do not chang,
third, the camera was lower slightly, eliminating from view the table top and, presum- rather than sub;e¡
ably, making each actor into more of an individual rather than part of a group sharing ceive objects as su
a table together (Cutting zoo5). 'perceptuai const;
The perception of depth can be strongly manipulated by the way in which an artist Artistic con\-en
(or photographer) chooses to portray the texture gradient (Gibson r95o). In general, poses. The mosr b
the texture of an object is most easily seen when it is nearby and when it is viewed at the surfaces and dark,
centre of gaze. This particular depth cue has not been used widely in art history (it was the Elder, \ras to .
often used, at least in some form, in Renaissance painting), becoming much more lighter shade of rL
important with the advent of photography and, subsequently, cinema. The lens of the this, however, rc¿,
camera can be used to exaggerate or minimize texture in a photograph, because the make the source I
image that it transmits presents, much like eyesight, an area of sharp focus at the very scene. Nou,, conr
centre and a progressive loss of acuity in the periphery. part ofthe render
cinema has even attempted to mimic the physiological accommodation of the eye. many inconsisten
One exaggerated example is provided by Vertigo,where the zoom lens intends to simu- no reason for a:ri
late the visual component in the sensation of vertigo. As mentioned earlier, painters several reasons trl
also use various tricks ofperspective to focus the attention ofthe observer. A case in better. Photograr
point is Mantegna's Dead Christ,which seems to suggest that Jesus had very small feet opment, and prlr
compared to the size of his head. That Mantegna was simply mistaken in his use of most powerñrl rr
perspective is one very unlikely possibility. Perhaps instead he chose to 'zoom' into the masking techniq-;
area of the scene that was most important for the narrative and emotional power of the
sources in the nal
work-the face of the dead Christ-by reducing the feet that were in the way. Modern or moons in the sk
filmmakers have the possibility, not available to Mantegna, to shift focal length over years the trick ofp
time and thus direct attention using these dynamic cues. print', was largeil
PICTORIAL CUES IN ART AND IN VISUAL PERCEPTION 373
::rer requlrements to
ii6 . Onenecessityis
IrruvrrNATI oN
:1r obiects and, when
:.e. mimic the differ- What our eyes detect is the reflection of light off of surfaces. To our brains, this pattern
,. Thus, one can con- of light is important mainly for allowing us to recognize and act upon objects. Thus,
.j the degree to which there is a basic distinction between the interpretation of visual perception as con-
:'!'as in natural scenes cerned with light or with objects. Artists may choose which of these aspects of percep-
tion are most important. The differences in style between Venetian and Central Italian
:' ¿ndcinema,where schools (during the Renaissance) reflect these two sides of this argument. Venetian
:,,'ings and paintings painting, influenced by the mercurial and atmospheric nature of illumination in that
:s -uihether to film a region of the world, often emphasized the play of light upon surfaces by privileging
¡ ¿lrild or the adult. the use of colours over the linear construction of the image. It is an introspective
h a more submissive approach that attempted to portray the phenomenological experience of the fleeting
'¡,r rroyides a useful
perception of colour and light as it happened with the flow of time. In contrast, the art
:;t r¡ithout the audi- of central Italy (such as Florence and Rome) was typically more concerned with por-
.:ght ifor a standing traying the solidity and depth of objects through the use of the line. Lighting, and thus
: rl the second third, colour, was a secondary factor and served to emphasize the three dimensionality of
ri¡reen the jurors is objects. In other words, these artists were mainly interested in those aspects of objects
r¿ rLrom. In the final that do not change with the variations of light-in object permanence' one could say'
:-e top and, presum- rather than subjective transience. This impressive ability of the visual system to per-
:: Lrf a group sharing ceive objects as similar across different viewpoints and lighting situations is known as
'perceptual constancy'.
:','in rvhich an artist Artistic conventions to manipulate light and shade have served a number of pur-
L:,r 1!-jo). In general, poses. The most basic manipulation, used at least since antiquity, is to lighten nearby
r¡: it is r.iewed at the surfaces and darken further away surfaces. One widespread device, described by Pliny
,r ¿rt history (it was the Elder, was to modulate the lightness of a folded garment so that the near fold was a
:,:,ning much more lighter shade of the colour (see Gombrich rg6o). One important caveat to note about
:¿:i-ra. The lens
of the this, however, was that the manipulations were entirelylocal. There was no attempt to
,-,:raph, because the make the source of the lighting consistent across the various objects portrayed in the
¿i: iocus at the very scene. Now, computer generated graphics will perform these sorts of calculations as
part of the rendering process. However, such technology may not be necessary' since
:odation of the eye. many inconsistencies in lighting are typically not noticed by the viewer. Thus, there is
.¿ns intends to simu- no reason for artists to worry about overall consistent lighting and, in fact, there are
:-te.i earlier, painters several reasons to avoid consistent lighting, since often the'impossible lighting'looks
: tbsener. A case in better. Photographers, for example, use a variety of techniques during shooting, devel-
.srad r-ery small feet opment, and printing to manipulate the local lighting of scenes. Many of Steichen's
-s:¿ken in his use of most powerful prints (shot before the Photo Secession Movement) used burning and
'se to'zoom'into the masking techniques that appear to represent, if one analyses closely, multiple lighting
.riional power of the sources in the natural world, which would only be possible by adding a few extra suns
: ii the way. Modern or moons in the sky (or manipulating the negatives in the darkroom). Duringhis young
lu¡t tbcal length over years the trick of printing from a few different negatives, a process called 'combination
print', was largely popular.
374 DAVID MELCHER AND PATRICK CAVANAGH
A similar situation is found with cast shadows. The use of shadows to portraythree- pure colour of t:e
dimensional form and depth has been intermittent in Western art (Casati zoo4; angle. Manyobiecr
Gombrich rg95). One finds examples in classical Greek and Roman works (paintings one might wárrt rt-
and mosaics), but it was not widespread until the Renaissance. Initially, investigations Theimportance oi
into painting shadows in the Early Renaissance can be described as an experiment in to paint images th:
trial and error (Casati zoo4). Eventually, knowledge about shadows was included in and water a1l mieh
treatises of painting and canonized as part of European painting techniques (Casati ability in imitarins
zoo4) In contrast, there are few examples of shadows in non-Western art before the paint look like a Cr
reciprocal influence between East and West occurred in modern times (Gombrich successfui, strate$-
1995). Some artists used shadows for practical reasons while others did so to show off closely as possib-e
their bravura or even play visual jokes (Casati zoo4). when depicting m;
Shadows are tricky objects for the scientific study of perception, since in many cases convey qualities i,i
they are not necessar/, while in others shadows are a vital cue to interpreting ambigu- Another example
'
ous situations (Cavanagh and Leclerc 1989; Kersten et al. rygZ). The brain appears to These artists rr-ere :
have a very limited understanding of physics when it comes to shadows-as shown in made paintine h:r
Fig. 19.6, we fail to notice when shadows are inconsistent with the lighting source or are the bright colou:s
even the wrong shape or colour (Casati zoo4; Cavanagh zoo5). One rule is that the could give sparlie
shadow must be darker than the background. This rule is demonstrated by scientific saturation and ht:
experiments, but also apparent from observing the way that artists have represented of representine 1,E:
shadows. In addition, shadows cannot look opaque (as if they are solid). This can be One technical c:
demonstrated experimentally by tracing a line with a thick pen around the edges of a transparent surl¿;e
cast shadow, which transforms our perception so that the shadowthen looks like paint. artists can use is :c,
Paradoxically, in order for a shadow to contribute to our understanding of shape in a the background n.
picture, the shadow must first be recognized as such (Casati zoo4). tor depictine tran.
In sum, the brain computes lighting locally, not across an entire scene. Before the ment of the X-iu::,
advent of artificial lighting, the source of lighting in real life was fairly obvious (sunlight tions from phr-si;s.
during the day, moonlight at night, and the occasional fire or candle). There is no refiaction throus:
'lighting inconsistency detector' in the brain. On the other hand, the large changes in be discovered. b,.:r
the brightness coloration of light during the day, as well as the reflectance properties depict objects suci
of objects, have led to a sophisticated-but local-set of tricks to understand the true surprisinglr'. does ,r
shape, lightness, and colour of objects in the world. Impossible lighting situations have huppy rvith depi¡r.
now become commonplace in photography, film, cinema, and theatre. In a case of eractly in the san;
life imitating art, real-life performances with live actors can now include the use of been drinkins. sr.',r:
differential local lighting strength and directionality and other features that have been out histon'. This'i
used bypictorial artists throughout history. only a subset oi th-
encv of a particur"::
\Iirrors are an,-
R.et-lection on a m:
them to introdu;r
Tn¡.NspARENcY AND REFLECTToN narrative contin i::
ing at his retlec:-,::
Indeed, the rrto pe:
Our earliest experience in colouring surfaces in pictures is to fill in the space within the ¿nd reflections ¿r:.
boundaries with a solid colour (colouring between the lines) to capture the actual or most often inclu,ie
PICTORIAL CUES IN ART AND IN VISUAL PERCEPTION 375
)'¡,'s to portray three- pure colour of the surface and ignore its variations (impurities) due to lighting or
r art (Casati zoo4; angle. Many objects, however, are shiny rather than matt. Moreover, many objects that
¿:r '"' orks (paintings one might want to portray in a picture are partially transparent rather than opaque.
liellr-, investigations The importance of gold and jewellery in various cultures has led, naturally, to the desire
:s an experiment in to paint images that look like gold and jewellery. Likewise, transparent fabrics, glass,
I,i-s h'as included in and water all might want to be included in a painting, as a testament to the artist's
l techniques (Casati ability in imitating textures. Throughout history, artists have devised ways of making
l:iern art before the paint look like a different type of surface. The most obvious, but not necessarily most
:1 times (Gombrich successful, strategy is to attempt to recreate the visual experience of a given material as
:' 'lid so to show off closely as possible by using that same material as medium. Giotto used marble dust
when depicting marble surfaces in the Cappella degli Scrovegni, presumably to try to
. i:nce in many cases convey qualities of the surface material that were difñcult to replicate only with paint.
rrerpreting ambigu- Another example is the use of glass particles and gold highlights by Venetian artists.
ie brain appears to These artists were responding to limitations of the medium (al fresco or tempera) that
I
ju',rt S-as shown in made painting highlights impossible without adding white paint, which washed out
:3i1ting source or are the bright colours and weakened their intensity. For these artists, adding glass dust
fine rule is that the could give sparkle to patches of clothing, light, or water without compromising the
rstiated by scientific saturation and hue of the colour. They were tackling the basic problem in painting
':s
har-e represented of representing light without using a specific colour to indicate light.
: strlid). This can be One technical challenge for painters has been the depiction of diaphanous or semi-
:.. rnd the edges of a transparent surfaces such as lace or see-through materials. The simplest technique that
¡en looks like paint. artists can use is to 'overlay' the transparent material so that it crosses the contours of
nding of shape in a the background material. These crossings, known as 'X-junctions', appear to be critical
for depicting transparency in a convincing way (Metelli ry74).Even a small misalign-
.:É s.ene. Before the ment of the X-junctions destroys the perception of transparency. Many other devia-
'i',' obrr-ious (sunlight tions from physics, however, are simply not noticed. One example is the depiction of
.:"ndle). There is no refraction through a medium such as glass or water. The principles of refraction had to
lhe iarge changes in be discovered, but this discovery was not at all necessary for artists to convincingly
:-rctance properties depict objects such as a flower vase. Indeed, correct refraction is not necessary nor,
..nderstand the true surprisingly, does any refraction need to be depicted at all. People are tlpically perfectly
:tine situations have huppy with depictions of vases containing a mysterious liquid in which light moves
:i¿atre. In a case of exactly in the same way as it does in the air. If one believes the paintings, people have
,' Lnclude the use of been drinking, swimming, andbeingbaptizedinthisliquid air-likesubstance, through-
:;res that have been out history. This 'found science' demonstrates that the visual brain takes into account
only a subset of the information available in an image when determining the transpar-
ency ofa particular surface (Cavanagh zoo5).
Mirrors are another area in which perception and physics appear to part ways.
Reflection on a mirrored surface has often been interesting to pictorial artists, allowing
them to introduce another viewpoint in the painting without compromising the
ION narrative continuity (Fig. lq.z). Examples include the classical story of Narcissus look-
ing at his reflection in the water, which has been found in mosaic form in Pompeii.
Indeed, the two periods during which there was the greatest interest in optical images
:te space within the and reflections are late Roman and Renaissance art, when mirrors, glass, and lenses are
:::ure the actual or most often included in painting. One fascinating question is whether this interest in
J76 DAVID MELCHER AND PATRICK CAVANAGH
in a typical wall-m,
ics students do nr
often not gettine tl
psychology, and th
noticed. Overall. r
surface (ifthe mir:
scene (not necess¿l
zoo4). The brain i
guesswork: we drr :
The Arnolfni
Pa,;,
Folie Berger Q88r_,
use of mirrors is to portray a person looking at his or her reflection in such a way that than issues ofdepl-
both the person and their reflection are shown-a physical impossibility. This basic depict motion in s:
ignorance of the physics of reflection is confirmed in experiments, which show that The list of artist:
people are clueless about the size and angle of mirror reflections, even for the most has been explorec
basic qpe of mirrors used in bathrooms and changing rooms around the world instead a basic que:
(Croucher et al. zooz; Fleming et al. zoo3). The reader can test her/his understanding
of mirrors by answering the following question: how big is the reflection of your head ' The reflection o, ¡
PICTORIAL CUES IN ART AND IN VISUAL PERCEPTION 377
MovrNc PrcruRES
The brain is highly sensitive to motion cues, with specialized brain regions involved in
detecting different patterns of motion. For the artist, there are two ways to portray
motion: imply it with a static image or cfeate a moving image. The advent of 'moving
pictures' as a form of art and entertainment is a relatively recent development in art
history. Of course, the images do not actually move: the perception of motion is all in
our head. There are psychological laws of motion perception that describe the
Ye1 Museum, Rovereto) spatiotemporal patterns that do or do not lead to motion perception. Film (and, later,
, -; :- a:ed the exact television and digital computer files) involves an economic compromise that allows
:'-:S:r S image in us to perceive smooth motion without wasting extra frames. This compromise works
.: -: v éllu dtLr well for us because of the human brain's visual 'refresh rate', but probably looks like a
flickering mess to a fly that wanders into the cinema.
With static pictures and sculptures, the problem of depicting motion, and more
general change over time, remains a fundamental challenge. One can see even in the
-e:lected images as a earliest cave paintings the way in which space and time dominate artistic choices'
rcred since antiquity Portraying an animalwill always give a sense of relative depth (the figure is seen against
a background), while the pose of the animal will make it appear either lifelike or dead.
¡hvsics? The answer Some cave paintings, even those 30 ooo years old, appear to us today to show animals
¿s no understanding in motion (Cutting and Massironi 1998). Given the importance of motion in art, it is
rrtrafed. One tlpical intriguing that the question of movement (time) seems to have received less attention
rn i¡ such a way that than issues of depth (space) in art history (Gombrich r98z). How, then, is it possible to
u'ssrbility. This basic depict motion in static media such as sculpture or drawing?
rt-s, rrüich show that The list of artistic techniques that can lead to an understanding of implied motion
.s. e\-enfor the most has been explored in more detail elsewhere (Cutting zooz). Here we will consider
s around the world instead a basic question about art and brain: do we truly'perceive' motion in a static
¡: his understanding
flection ofyour head ' The reflection of one's head is incorrectly perceived as life-sized by most people.
378 DAVID MELCHER AND PATRICK CAVANAGH
:st set ofevidence for A second issue raised by these image studies is the influence of experience on picture
:lres. These lines are perception. It is reasonable to hypothesize that the techniques used by artists to por-
::dicate motion, but tray motion build upon other, earlier methods. Once the observer learns to 'see' motion
:,;nd appear to blur, in one tlpe of painting, this increases sensitivity to those motion cues, which
,-¡ t¿ken advantage of then influences how they see other artworks. This is a process known as 'perceptual
\-ie\ying oriented
:,r',^,' learning'. One compelling example is provided by the pictorial experiments into
Ros5 2es2; and neu- the representation of motion that were central to the artistic practice of the
: :\fRI) (Krekelberg r9tos. Artists as diverse as Picasso, Duchamp, and Italian Futurists such as Bragaglia,
:sa lines into account Balla, Boccioni, and Russolo were influenced by the chronophotographic experiments
peed lines are part of of Marey and Muybridge. A recent experiment looking at how the brain reacts to
:rr:. -\s Kennedy has such artworks may shed some light on how this developed. Two groups of students,
r: i,-heel (rSS:). one naíve to such depictions and the other well-versed in art history, were shown
.; :naqe is not strictly images of Duchamp's and Balla's paintings, 'static' abstract works (which do not give
:r-processing areas
:,,:' any impression of motion), and Marey's chronophotographs (Kim and Blake zooT).
::ese studies, observ- For both groups, motion-processing areas were active for the chronophotographs and
sirorving people in inactive for the 'static' paintings. The main difference between the groups was that only
h; question was how the art students showed significant activation in motion-processing areas for the works
-\fotion processing by Balla and Duchamp. This experiment demonstrated, in agreement with other stud-
:rir-e to local motion ies ofperceptual learning, that experience can influence the wayvisual cues are used by
,-'i'er space and time. the brain.
tls ..i r-isual stimulus What is not clear from the imaging study is whether the change in the students'
i-:emporal change) brains (which was also reflected in their perceptual judgements) was based on a direct
.:n. ir'hich have been influence of cognitive processes (explicit knowledge) on these motion processing
,:rrtrn. become active regions, or whether the artistic training with rgros artworks had influenced the way
nnus. rt is now clear that the students paid attention to-and utilized-the motion cues that the artists
i.-,ns involved in per- had included in their works. Some evidence for the latter explanation comes from find-
l::ls of motion based ings with another implied motion stimulus, the rotating rectangle. The rectangle
stimulus, which gives an 'impression' of motion that continues into the future, does
:-, iie a useful way of not seem to activate the same motion-processing areas found in the earlier study.
:Ln: hnding in these . Instead, the rotating rectangle appears to activate more 'cognitive' areas of frontal
:srinq areas are most cortex implicated in expectations and prediction (Rao et al. zoo4). Thus' it is reason-
r :re brain are inter- able to hlpothesize that our experience of viewing art can influence both our expecta-
.nson'and thus may tions and-more interestingly-the waywe actually see the world.
:et',t g¡d5, aCtivatiOn
: \-r ma1'teil us 'this
,,1 we are not
-¡lnt Biological motion
:e-r'abstract3) mean-
Ourvisual system is fine-tuned to detect'biological' motion: the movement patterns of
people and animals (Fig. rg.8). One of the seminal studies of biological motion is a
::.: rom any particular studybyiohanssonandcolleagues fromrgT3inwhichactorswerefiimedundercondi-
-:::::r of'apple' that tions in which only lights attached to parts of their bodies were visible. This extension
:;:tJ. or tasted).
.. :;t experience ofthe of chronophotography (Fig. r9.9) separates motion and form information: it is impos-
-. a'f iects. sible to identifi' the object based on form alone in any given frame. Only by combining
information across
1998), can the patte
A number of e¡
'point light wallier'
experience with mt
as other animals su,
Interestingly, the p
the gender (Kozlo¡'
and activityof the a
aging studies suggr
and animal movenr
Interestinglr', r'-e
position over time
have noted in his d:
muscular contractl,
succeed in making
French sculptor Rc
ing awayto portrar
Fig. 19.8 Renato Guttuso, BOOGIE WOOGlE,1953. (See Colour plate 6) stages in the mo..-e
information across several different frames, over a period of up to 8 seconds (Neri et al.
1998), can the pattern of motion be understood.
A number of experiments have examined the nature of our perception of these
'point light walker' stimuli. The ability to understand these images does not require
experience with movies: infants only3 months of age (Fox and McDaniel r98z), as well
as other animals such as cats (Blake 1993), perceive biological motion in these displays.
Interestingly, the pattern of a few dots can yield quite detailed information, including
the gender (Kozlowski and Cutting 1977), emofional state (Brownlow and Dixon 1997),
and activity of the actor, such as running, leaning forward, or playing sports. Neuroim-
aging studies suggest that our brain has a specialized network for perceiving human
and animal movements (Oram and Perrett 1994; Grossman et al. zooo).
Interestingly, we do not actually have to see physical movement (a change in spatial
position over time) in order to perceive biological motion. As Socrates is reported to
have noted in his discussion with the sculptor Cleiton, 'by faithfully copying the various
muscular contractions of the body in obedience to the play of gesture and poise . . .you
succeed in making your statues like real beings' (Xenophon, MemorqbiliaIIIXz).The
French sculptor Rodin brought this principle to its extreme consequences, inaugurat-
ing a way to portray motion in sculpture by representing the different limbs at different
stages in the movement, so that the observer could'sense' the motion unfolding in
r plate 6)
',¡ ¿': lri useum, Rovereto)
time-thus also proving how sophisticated our sensitivity to detect human motion
' -::--:3. really is. In other words, even static art forms such as sculpture, relief, drawing, or
n the back-
painting can create a 'feeling' of movement in the depicted agents (Freyd r9A¡). Studies
of the 'mirror neuron system' (see essays by Gallese, Chapter zz, andby Calvo-Merino
and Haggard, Chapter zT,this volume) suggest that some actions, particularly goal-
directed actions, activate a'motor-like' reaction in the brain which can even influence
muscle tone. The eye is a channel by which images can influence the body and all the
senses. The exquisite sensitivity to biological motion may be a precursor of the ability
to understand, both through cognitive processes and the mirror neuron system, the
meaning of a particular action.
Just as the interpretation of biological motion in a static picture can be influenced
by previous experience, the mirror response to watching the action of another person
also depends on our own experience with such actions. This idea has fundamental
implications for our understanding of how people perceive pictures. Consider the
example of dance. Art takes advantage of the human interest in dance and the inferred
motion and pleasure in viewing it. The portrayal of dancers is a major theme in Asian
and African artworks,l a fact which has greatly influenced twentieth-century Western
art (Fig. r9.ro). It is important to remember, however, that our ability to understand
and interpret static dance positions depends on our personal experience with the real,
dynamic dance movements. A recent neuroimaging study confirms that the activity of
1 887.
',r : -: l,'ruseum, Rovereto)
a Dance is a basic human activity found across many cultures. Many dances make use of visual
: - . i.rr ogy Of the
props, such as masks constructed in ways that accentuate the impression of motion during the dance.
-. ^ti locomotion, Some African masks, for example, use reflective material for catching light, while Japanese'no' masks
change expression with change ofhead position.
382 DAVID MELCHER AND PATRICK CAVANAGH
'' ::
-:-
:
Fig. 19.10 Robert Mapplethorpe. Grace lones, 1984. @ Copyright The Robert
Mapplethorpe Foundation. Courtesy Art ft Commerce.
This is a photograph of Grace Jones who had been painted by the artist Keith Haring. Andy Warhol
introduced Haring to Jones and helped arrange for the photography session with lVapplethorpe. Jones -- : -- _
-"-----: -''
-. I - -
is wearing a huge crown and rubber jewelry that were designed by David Spada. The influence of art- . :\ \.í"
works from Africa on 2Oth-century Western art is well known. At the same time, the interpretation of
these artworks, in the West, has been influenced by generalization, ignorance and stereotypes, as
cleverly exploited ín this photograph by Robert Mapplethorpe. Here, the diverse elements of the
decoration and pose ofthe model play into expectations and stereotypes, while the static (almost
sculptural) pose and the medium of photography give the subject matter an 'objective', Western view
of the subject.
PICTORIAL CUES IN ART AND IN VISUAL PERCEPTION 383
motor cortex while perceiving dance moves is influenced by personal experience with
those dance movements (see Calvo-Merino and Haggard, Chapter 27, fhis volume).
Thus, for example, the response of a qpical Western museum-goer when looking at an
African sculpture demonstrating the dooplépose may differ fundamentally from that
of member of the Quenon people (Ivory Coast), who would recognize the forward
tilted torso, bent knees, open hands, and gaze fixed straight ahead as part ofa dance
pose (Tiérou , 1992) .'Although the capacity to learn to discriminate biological motion
from non-biological motion appears to be innate, the mechanism is flexible enough to
be influenced by experience. This suggests that the influence ofindividual experience
and culture is greater at the level of complex perceptions, such as the interpretations
of particular movements, than for simpler features such as contour lines or depth.6
Our sensitivity to biological motion is a double-edged sword for artists. On the one
hand, theycantake advantageoftheobseryer's sensitivitytobodymovement. However,
this sensitiüty to real bodily motion makes us particularly attuned to detect deviations
from natural movement. The Renaissance art theorist Alberti, among others, argued
that it was critical to understand the skeleton and the muscles in order to depict a real-
istic posture. The tricks required to create a natural pose become much more complex
with dynamic stimuli. One telling example is computer graphics, which recently has
been striving to move from 'cartoonish' depictions to hyper-realistic scenes in film
and in video games. There are several interesting techniques that have been developed
to incorporate knowledge about the structure and psychology of humans and animals
into computer graphics. The 'skeleton system' approach, which started in the late
r98os in university laboratories, and then quickly moved to commercial and artistic
applications, involves explicitly modelling the body's mechanical structure (skeleton,
muscle, tendons, and so on) in the computer program used to create the image. The
animation appears to move in a more natural way because its surfaces move in accor-
dance with an underlying'natural' structure.
A more advanced form of animation, 'artificial life', moves beyond body mechanics
to also include a model of the basic'psychology' (motivations and perceptions) of the
organism. A computer model of fish, for example, can create a cartoon in which the
ht The Robert
fish move in a strikingly naturalistic way that makes them look like animals rather
than drawings (Terzopoulos 1999). Their behaviour evolves over time, with the fish
', -: : ncy Warhol
-.' ','¿ac eihorpe. Jones 'learning' how to swim based on the constraints of its body and influenced by its goals-
, -': " :ence of art- such as seeking prey and avoiding predators.
: -': ':e'pretatiOn Of
-: :-:':3:ypes, as
:::--:":SOfthe
, .-: s:a. c (almost 5 Artists have been great obsewers of anatomy and biological motion. Leonardo da Vinci, for
- :-- .a.,,niestern view example, studied the physiognomy and the physiology of emotional expression, and used this
knowledge to detect which aspects of the face were critical in driving the facial expression. Many
artists have attempted to draw 'the motion' rather than simply showing a static representation
of a body in motion.
6 The ability to perceive lines and depth disparities does depend on experience in early life. This
flexibility is necessary because of the variability across persons in basic factors such as visual acuity
and the distance between the eyes. Given normal visual input, most people naturally develop
remarkably similar visual systems that have much in common.
DAVID MELCHER AND PATRICK CAVANAGH
Another interesting approach in computer graphics is 'motion capture', famously with very few cones
used in films such as The Lord of the Rings (zoot) and King Kong (zoo). The movement spot'where the op:r
ofa real actor's body and/or face are recorded as a set ofdiscrete points that change Since clear and ,
position over time. These points are then'grafted' on to a computer-graphic skeleton our eyes an a\.erag3
that changes in the same way. The results can be impressively realistic. The motion ments. When it cc,:r
capture concept descends from ideas developed by the photographers Muybridge and in a perceptioo-ár-.1
Marey and later extended by Edgerton and by the psychologist Johansson (1973) in his drawn by that rrh,.
point-light walker displays. In animation, the idea of motion capture was used by day, it is estimatec :
Disney studios in the ry37 frIm Snow Vrhite, in which animation stills were traced over Contraryto this iun
film footage ofreal dancers (a technique known as rotoscoping, invented in r9r5 by also capable oin¿¡
Max Fleischer). This fusion between art and science has been used extensively in dance, ouÍ gaze someth-¡r
film, and also for popular entertainment. active vision u-ork.
Thus, the most straightforward way to depict human motion is to 'record' and copy and then wait for --'
the motion of real humans. For a painter, this is not an option. Alberti suggested that decision ofrvhere t',
painters and sculptors began with an understanding of the skeleton and muscles, in eye and head ¡n¡'"'"
order to understand motion and pose, through the dissection of cadavers, when pos- Eye movements ¿ra
sible, and anatomical studies on faithful wax models. Leonardo's intense study of is possible to co,,.r
anatomy, body postures and physiognomy is well known. What is fascinating about us to scrutinize s,-.:
this combination of art and science is how it emphasizes the way in which our percep- go together.
tion is based on an understanding of what is beneath the surface. Artists, as super- The consequer;,
perceivers, have tried to understand the driving force behind the optical data. Scientific detail and colou¡.
evidence, along with the more practical evidence of the usefulness of 'motion capture' make and percei. e
techniques and'artificial life'provide, instead, support for Alberti's idea that the artist mingles t..inq ..ti,:
must understand what is beneath the surface, beyond the optical projection, to capture requires us to nra,','i
the coming-to-life ingredient ofbiological motion. our retina as rie l.-
mo\.ements leads :
One great mlth about visual perception is that the eye is like a photographic camera.
The camera obscura, and later devices that fixed an image permanently, were designed
based on the optic principles of a pinhole camera like the eye. While there is some
similarity between the eye and the mechanical camera, it is clear that in terms of the
medium for imprinting the image one would never want to imitate the human eye. In
fact, it is difficult to overstate the extraordinary difference between the image at the
back of the eye and what we perceive. A partial list of the terrible flaws in the retinal
image that is passed to our brains includes: (r) the image is only registered at high
resolution at the very small portion in the centre, with most of the peripheral vision
mainly useful only for detecting movement and no details; (z) there is a complete lack
of uniformity in the presence of each type of cone (colour receptor) across the retina,
PICTORIAL CUES IN ART AND IN VISUAL PERCEPTION 385
n capture', famously with very few cones outside the centre; (3) there is a big hole in the image at the 'blind
: L- ¡ . The movement spot'where the optic nerve leaves the eye.z
i Flrrints that change Since clear and colourful vision is only possible at the centre of gaze, we move
l:cr-graphic skeleton our eyes an average of three to five times per second in jumping'saccadic' eye move-
.¿,istic. The motion ments. When it comes to vision, the eye is physically drawn towards what it perceives,
i:;rs \lu1üridge and in a perception-action cycle: 'The lover is drawn by the thing loved, as the sense is
-r¿n:son (rgZ¡) in his drawn by that which it perceives' (Leonardo da Vinci, Notebooks, Tr. Tav. 9a.). Each
:¿::Ufe \\¡aS USed by day, it is estimated that we make more saccadic eye movements than beats of the heart.
:i--. --ere traced over Contraryto this jumpyvisual input, our perception is smooth and continuous.8 We are
. rn'''ented in r9r5 by also capable of making smooth eye movements, but only when we are following with
;.,*ensivelv in dance, our gaze something that is moving within a certain range of speeds. Metaphorically,
active vision works somewhat like sonar. We direct our gaze towards a region of space
i : --, ':ecord'and copy and then wait for the results to hit the retina. Then we move our eyes again and this
j--erti suggested that decision of where to look determines what we see. There is a perception-action cycle of
.:,-: ¿nd muscles, in eye and head movements which bookmark brief periods of relatively stable fixation.
":l¿r-ers, rvhen pos- Eye movements are tlpically accompanied by a shift in our attentional focus. While it
i: s intense study of is possible to covertly shift our attention elsewhere from our gaze (perhaps to allow
. :: :¿scinating about us to scrutinize someone without being caught), shifts in gaze and attention tend to
':r i,"'iich our percep- go together.
:--.. ,{rtists, as super- The consequence of vision with a roving eye is that we never see a scene in uniform
: --, ;¿l data. Scientific detail and colour. This has numerous implications for understanding how people
'::-ic¡tion
. --, capture' make and perceive artworks. As the art historian lames Elkins notes, 'each act of vision
:r . ri¿a that the artist mingles seeing with not seeing' Ggg6, p. zor).e Looking at details of specific objects
I : -,' ¿.tion, tO CaPture requires us to move our eyes and head and, thus, even stationary objects move across
our retina we look around. The way that our visual system interacts with our eye
as
movements leads to different timescales of perception. When viewing a painting, for
7 It is extraordinary that there are no known records indicating knowledge of the blind spot
previous to 1668. Leonardo had claimed that perception was actually best at the point where the optic
EYE nerve enters the eye (mistakenly equating it with the refina). fovea, where acuity is highest). The
physicist and priest Edme Mariotte published a small book in 1668 (Noarelle Découverte touchant la
rcüe) based on a simple perceptual experiment that is now replicated in perception classes around
the wor1d. Mariotte's demonstration was popularized at the Academie des Sciences (Mariotte was a
,: -- :iaphic camera.
founding member) and taken 'on tour' for various audiences.
,",. -,.,'ere designed
--', . 8 One of the first to notice the fact that saccadic eye movements must pose a 'problem' that our
-,',
.:',: rnere is some brain must solve was the scholar Alhazen (Ibn al-Ha1tham, 965-to4o). He proposed that the mind
: --r;: ln terms of the must have developed a 'habit' that would take into account the differences between the retinal
:-: --:- human eYe. In motion caused by eye mo\¡ements and by motion in the world.
e The important distinction between the vast visual fie1d and the small subset of the world that is
, -:., --i1. image at the
actualiy perceived by the mind has a long history. Examples include Aristotle's theory that it is
,. :,."',-: ú the retinal impossible to perceive two objects simultaneously (On Sense and the Sensible, ¡¡o BC), the Stoic
--, :.9-stered at high concept of the'phantasia kataleptiké'that compares looking to grasping sensations by the mind,
-,-. :::iphe ral r'ision Lucretius' idea ofvividness in perception (De rerum natura, So BC), Alhazen's tis distinctiva (ro4o),
Leibniz's 'perceptions petit' (La Monadologie, ry4), and, more recently, in Wundt's introspectionism
.::. -: ;. ;on]plete laCk and the beginnings ofpsychology. Freud's ideas ofthe unconscious have had a great impact on
:,: lJi.tss ¡he retina. artistic production and an unrelenting influence on art criticism.
386 DAVID MELCHER AND PATRICK CAVANAGH
example, some visual information is specific to that moment and vanishes from our
mind after eyes move on to a new position. Other visual attributes are combined
across separate fixations over periods of time measured in fractions of a second
(Melcher and Morrone zoo3). Details about the visual properties of objects and their
locations can accumulate over a period oftens of seconds and persist in mind (Melcher
zoor; Tatler et al. zoo5), allowing us to keep track of objects in common tasks such as
making a sandwich or cup of tea (Land and Hayhoe zoor). Most of these visual details
will be forgotten days or weeks later (Melcher zoor). Critically, none of these different
visual representations over different time periods is like an image in your head that
matches exactly the painting on the wall.
Separate aspects of the painting inhabit different mental places in space and time.
One can heuristically examine this problem by observing Cubist paintings or, more
recently, the photocollages byDavid Hockney and Polaroid collages by Joyce Neimanas.
These images, like cubist artworks, draw our attention to several fundamental differ-
ences between paintings and perception.
First, as described earlier when discussing depth, there is a basic discrepancybetween
An¡
the perception of depth by humans (Leonardo da Vinci's natural perspective) and the
transformations required to create pictorial representations of depth (artificial per-
spectiye). This problem becomes infinitely more complex when head and body
movements are allowed. There are a number of view point-dependent effects when
looking at perspective paintings that we tend to ignore. For example, Deregowski and
colleagues have argued that the phenomenon ofviewing converging lines as parallel-a
foundation of linear perspective-does not hold for oblique views (Deregowski and
Parker 1988; Deregowski et al. ry91. On the contrary, observers may judge converging
lines as parallel under many conditions, in agreement with pre-Renaissance perspective.
Another aspect of real-world viewing is that the position of the head and the focus of
gaze tend to shift significantly when looking around a real scene. In other words, we
often look at one side and then shift completely to look in the other direction. Thus,
converging perspective may portray more accurately what it is really like to make these
dramatic shifts in view, since it is impossible to have the same vanishing point for a
glance 45 degrees to the left ofcentre and 45degrees to the right.
Another interesting example is the fact that our eyes converge when viewing
near objects and diverge when we stare off into the distance. Leonardo portrayed the
struggle to focus the eyes (which he observed in the immature infant's visual system)
beautifully inthe Benois Maclonna (cr478). when our eyes diverge, they gaze into
'optical infinity': the parallel lines of gaze do not meet. Filmmakers, like painters,
use depth planes (in particular, the furthermost depth plane in a shot) in order to
manipulate the feeling of space and openness. At this point, there is no clear evidence
about why it 'feels' different to see the horizon or to be walled in (although estate
agents include such information in calculating property prices). Presumably, for our
ancestors, a large vista would have been useful for safety and for finding important
resources-even today people travel en mosselo 'outlook points' where they can see far
distances. Like in film, painters have exploited convergence and divergence as a cue to
depth. A picture blocking the perception of the horizon can create the impression that
PICTORIAL CUES IN ART AND IN VISUAL PERCEPTION 387
::.,'anishes from our the subject matter is closer or even closed in. One can compare David's Death of Marat
r.--rtes are combined Gzgl), where a wall behind the dead activist blocks our view, to Gerome's Death of
:¿;tions of a second Ceasar (1367) with the full perspective that continues into the distance. The former is
s ,¡f crbjects and their more closed, claustrophobic and anxious, while in the second painting the vanishing
¡:.t in mind (Melcher point in the distance underplays the supposed tragedy and makes the cadaver go
,,i:1non tasks such as unnoticed even though it is in the foreground. A similar theme from the same period,
-'f these r.isual details which combines the two methods, can be found in Manet's Execution of the Emperor
,- r¿ of these different Maximilian (ú62), where the wall blocks the horizon at eye level thus making the scene
i{. i¡ r'our head that take place almost in our personal space, but an escape for the eye is allowed at the top
of the painting above the wall.
.s :n space and time.
.t :aintings or, more
:s bl Tor-ceNeimanas.
- i-¡rdamental differ-
Ann THERE VISUAL PREFERENCES?
Jrs;:epancybetween
;. :.rsoective) and the
i:pth i nrtifcial per- Thus far, we have only considered the techniques which an artist could use to depict
-:r head and body the existence of a certain object, scene, or event in a picture, without asking the ques-
::ndent effects when tion of why observers might actually like to see some of these features in an image.
::1<. Deregowski and Inherent in this question is the intuition that our sensitivity to particular visual cues is
:l: -ines as parallel-a somehow tied to our response to an artwork that uses those cues. Gombrich argued
:','.'s Deregowski and that artistic techniques were like 'the forging of master keys for opening the mysterious
::::"- iudge converging locks of our senses to which only nature herself originally held the key' (rgg¡' p. ¡:).
: :issance perspective. Thus, for example, our sensitivity to colour allows the artist to access the ability of those
:e:i and the focus of visual processing mechanisms that make the observer think and feel. At the same time,
; ,:l other words, we scientists face an enormous explanatory gap between showing that an aesthetic object
::er direction. Thus, activates a particular visual process in the brain and understanding why fhalvrstal
rl,', liie to make these object produces an aesthetic experience. The basic problem is that a great number of
',.::rishing point for a objects activate visual areas, and most of these objects are not considered artworks (see
Hyman zoo6).
.', erse rvhen viewing Visual preferences can be studied at various levels of analysis. At the simplest level,
,::¿:do portrayed the certain techniques make images easier or more difficult to interpret. A blurry image
r::ni' r-isual system)
s will tend to cut out high spatial frequencies, for example. There is some evidence that,
,, -:ge. they gaze into in a world saturated with images, many people prefer stimuli that are slightly challeng-
:-rders. like painters, ing (see Schifferstein and Hekkert, Chapter 28, this volume, for a discussion of the
i : shot) in order to MAYA principle in design). Alternatively, certain techniques may focus our attention
-i -s no clear evidence on one particular feature, such as colour, shape, or motion. Although some have sug-
: :: although estate gested that it is this 'super-stimulus' itself, by effectively driving the neurons in the
?:e sumably, for our visual system, which leads to visual preferences (see Ramachandran and Hirstein 1999;
,-r inding important Zekiry9fi,this idea is controversial and seems to fall into the aforementioned explan-
.,,"'i¿re theY can see far alory gap (Hyman zoo6).
.'1i eroence as a cue to If preference is not based on exaggeration of bright, colourful, moving stimuli, then
:: ;re impression that how could a particular feature cause an aesthetic response? One limiting factor might
388 DAVID MELCHER AND PATRICK CAVANAGH
be that certain features give access to the brain's reward system, based on experience behaviour within t::t
and, to some extent, innate tendencies. The colour red, for example, has the potential may be critical in sh;
to stimulate the limbic system-colour is tied to the motivation system because it is can trulybe'uglr-' a:
critical to survival: red is the colour of blood or raw meat, it is the colour of ripe fruit At the same tim..
and dangerous animals. In other words, it makes sense to be aware of the colour red play a role in prete:-
and to associate this colour with the reward system in order to survive. The exact nature ancing act in rr'hi¡:r
(Fig. r9.u). As Le C:
of this association will depend on a lifetime of experience.
Another example of this complex mapping between visual features and the brain's foundlaws...ar..
reward system is shape. The visual system is highly sensitive to shape, a fundamental choice by the artrs: ,,,
visual property. This aliows artists to 'suggest' a particular visual interpretation (such not exist in isolatrc,-
as the three-dimensional shape of an object), but it is also an expressive tool to suggest
the role ofthe ceni:e.
a particular emotional state. In a series of recent studies, neuroscientist Moshe Bar and
ciples such as golce:
colleagues have shown that curved and pointed shapes influence people differently indeed in nature ¡.
and that this difference is correlated with the way that sharp corners (potential dan- begun to elucidate :r
gers) activate the amygdala (Bar and Neta zoo6, zooZ). The work of Bar and Neta sug- a composition. Fc:
gests that one of the 'master keys' used by artists might be explainable in terms of visual attention to a par:.;
neuroscience: jagged lines and pointy corners, through their greater activation ofthe lines) increases :t::
amygdala, offer expressive power to the artist. Given the importance of round versus Treue zoo6; lrlelc:,.
sharp corners for most people, this artistic tool might tend to work across cultures. In
contrast, many other of our shape associations are specific to our own experiences and
culture. Laboratory studies have shown that simply associating a particular shape with
an otherwise affective stimulus (such as a picture of a gun or a smiling baby) can lead
to a change in preference for that shape-even when the participant does not con-
sciously remember this association (for review, see Ghuman and Bar zoo6).
Thus, there are at least three key elements to visual preferences. First, visual prefer-
ences depend fundamentally on what the visual system processes: no one can like or
dislike ultraviolet light or pitches beyond our hearing range. Our sensitivity to features
such as symmetry or colour is a precondition for preference for certain values of those
features. Second, some preferences are likely based on deep-rooted links between cer-
tain visual features and the brain's 'survival system' that manages our search for food,
safety and reproduction. Third, these building blocks of pictorial (and non-pictorial)
representation have flexibility and cultural specificity. Preferences, such as for particu-
lar faces, is directly influenced by our 'diet' of images of faces (for review, see Rhodes
et al. zoo3). Overall, the sensitivity to particular visual features, and their link to the
brain's reward system, provides an expressive tool that artists can use not onlyto simply
make a picture, but to imbue that picture with other suggested resonances.
It is interesting to note the overlap, within the brain's reward system, between the
neural activation to different stimuli that people prefer, including one's favourite
music, food or touch sensations (Blood and Zatorre zool; Kawabata and Zeki zoo4;
Di Dio el al. zooT; Stice et al. zooS; see Calvo-Merino and Haggard, Chapter z7' this
volume). This suggests interesting commonalities between the various arts, and also Fig. 19.11 Alberto !
'--a-.ed
on experience behaviour within the set of rules (Tabibnia et al. zoo8). This flexibility to social mores
r:,e. has the potential maybe critical in shaping behaviour, but it also suggests that inappropriate behaviour
1 s-!-stem because it is can truly be 'ugly' and that the link between good and beautiful is not accidental.
:: ¡olour of ripe fruit At the same time, the distribution of features within a composition also seems to
, ::: oi the colour red play a role in preference (Arnheim ry74). Great artists seem to manage a complex bal-
',:',':. The exact nature ancing act in which the elements fit together in some undefinable, yet satisfying way
(Fig. r9.rr). As Le Corbusier argued, in the context of architecture, 'the true and pro-
.:;:es and the brain's found laws . . . are established on mass, rhythm and proportion' (t985 p. 286). Every
):¡pe. a fundamental choice by the artist works within the context of the entire work individual features do
- -::erpretation (such not exist in isolation. Much work in vision science has attempted to better understand
:...t.,'e tool to Suggest the role ofthe centre, of symmetry, and of composition in artworks, investigating prin-
.:::rst \loshe Bar and ciples such as golden ratios that recur throughout art and architectural history, and
,.r people differently indeed in nature (for example, Locher 1996). In addition, recent studies have also
:::.:s potential dan- begun to elucidate mechanisms that may underlie balance in terms ofvisual features in
. . - Ba¡ and Neta sug- a composition. For example, studies on 'feature-based attention' show that paying
::l: :n terms ofvisual attention to a particular value of a feature (such as the colour red or rightward tilted
:::r: ¿cti\-ation of the lines) increases the response of neurons to other matching features (Maunsel and
:.:;e oiround versus Treue zoo6; Melcher et al. zoo5). Thus, there are long-range interactions between
:¡. fiioss cultures. In
-,-.1 :1 .\Periences and
.:.:s also seem to be Many artists balance the elements within a picture, using both relatively simp e proportions and more
.::; Lrr tnappropriate complex balancinq acts in which elements seern to fit together in intangible ways.
DAVID MELCHER AND PATRICK CAYANAGH
certain features, based on perceptual opposites (red/green, blue/yellow, up/down, left/ and bodily state of t
right and so on). In a well-balanced composition, paying attention to the red square a conversation her-,,,
on the left does indeed'call forth'the red shapes on the other side of the screen, so that Interestingl..'. rec
the firing of neurons in the brain create an expectation that may be met when matching our ability to noti;t
features are found in other areas ofthe visual field. lished a studr- sht¡r
improvement in dt
Training in art he'
school. Thus, the hr
other in recent til¡É
artistic depictions
CoNcrusroNs medical research. T
'All the arts are bas,
Our brains are extremely ready and able to interpret flat images in terms of the and also the pers..r:
three-dimensional world. This fact, by itself, is an incredible scientific discovery. It is more acute.' For ti
widely accepted, within the scientific community, that picture perception reflects physics that matt¡:
properties of how the brain normally works, rather than simply a cultural convention.
As described in this chapter, pictures embody hidden knowledge by artists about how
the brain works, and our willingness to interpret patterns of light and dark on a flat
surface in terms ofmeaningful objects in spatial relationships to each other. One impli- R EFERENCE S
cation is that the brain does not necessarily represent the world, in terms of internal
ArnheimR(r9;+.,"-"
computations, as a full three-dimensional scene. In other words, there is no theatre in
Berkeley, CA.
the head in which a recreation of the world takes place. We must continually, using our
Bar M and Neta lrl
eyes, brains, and body, interrogate the world for more details. When it comes to cues 6+s-8.
from lighting, this information is strictly local. Otherwise, if the world in our head Bar M and Neta \Í :
really was three-dimensional, then flat images should be distorted perceptually when tion. Netlrol-.r'¡':.-
we move. There could be only one seat in a movie theatre and each painting would Barnes J and Dar-id. ,:
need to be viewed through a pinhole. At the same time, people rarely confuse pictures nomenologica, .'-
Blake R (1993). Car.:
with the scene represented in the picture. Trompe I'oeil, for example, works at its best
Blood AJ and Zatr-: :
with constricted observer motion and viewing from a relatively far distance so that
ity in brain reti',,r
small head and body movements do not break the illusion.
Sciences,98,11S::-
Certain techniques in pictorial representation enable artists to turn pigments into Bordwell D, Staiger'
perceptions. Understanding these rules does not, by itself, lead to the creation of 'art', Mode of Prod:i:.::
just pictures. It is critical not to confuse recognition with aesthetic experience. On the Brownlow S and !.r
contrary, any painting can potentially provide a discovery in the alternative physics point-1ight dis:-:-,
of pictures and, therefore, the psychophysics of the visual brain. Many of the rules of Burr DC and Ross -
physics that constrain real scenes are optional in pictures. It is the choice of the Journal o/-\ cr,',,..:
Calvo-Merino B. G..
artist, based on her or his goals, whether or not to obey the rules discovered by science.
tion and acqu::.:
Such basic transgressions of standard physics, including contours, blurry images, 7243-9.
impossible reflections, or shadows, tend not to concern the observer. For the neurosci- Casati R (zoo-1 t. I ;;i
entist, previously unnoticed differences between pictures and real physics can be CavanaghP(zoo5.-i
viewed as data, as basic discoveries about the visual brain. For artists, on the contrary, Car.anagh P and L¿.-
these deviations may allow them to more effectively and economically achieve their Human Perc¿t:::.
aims. For the viewer, the picture can then be used as a stimulus that sets off a chain Chao LL and Ilanir
stream. Neurt-l'.:.;
reaction of events in their body and brain. The outcome can influence the emotional
PICTORIAL CUES IN ART AND IN VISUAL PERCEPTION 391
', ¿-Lrrr, up/down, left/ and bodily state of the observer. Pictures, rather than mere copies of the world, can be
::¡r. ro the red square aconversation between minds.
¿ ¡,f the screen, so that Interestingly, recent studies suggest that learning how to look at art can improve
ii met \\'hen matching our ability to notice fine details. The lournal of the American Medical Associationpub-
lished a study showing that training in noticing the details of paintings led to an
improvement in detecting medically relevant details in patients (Dolev et al. zoor).
Training in art has become part of the core curriculum at Yale University medical
school. Thus, the history of art and science, somewhat artificially segregated from each
other in recent times, forms a circle, with art influenced by the study of anatomy, while
artistic depictions of anatomy have then been used to foster further scientific and
medical research. The sculptor Henry Moore expressed a similar idea when he stated:
'All the arts are based on the senses. What they do for the person who practices them,
:E.s in terms of the and also the persons interested in them, is make that particular sense more active and
-:trhc discovery. It is more acute.' For the neuroscientist, looking at art can provide deep insights into the
:i t.r.eption reflects physics that matters to the visual brain.
Croucher Cf, Bertamini M, and Hecht H (zooz). Naive optics: understanding the geometry of
HrnanJ(:ooo) r-- -
mirror reflecttons. Journal of Experimental Psychology: Human Perception and Performance,
*rlrv.interdisci: -,: .
28,546-62.
-:akura S (1994). R.:
Cutting lE (zooz). Representing motion in a static image: Constraints and parallels in art, sci- dytes). Jountoi ,:.' ..:
ence, and popular culture. Perception, 3r, rt65-94. -ohansson
G {r9-,.
Cutting IE (zoo5). Perceiving scenes in film and in the world. In
lD Anderson and BF Anderson, Perception attiL !: , :l
eds. Moting Image Theory: Ecological Considerations, pp. g-lT.University of Southern -.rhnson
MH (:'oo¡ . :
Illinois Press, Carbondale, IL.
Kerr'abata H and 7¡.-
Cutting JE and Massironi M (t998). Pictures and their special status in cognitive inquiry. In
1699-7o5.
I Hochberg, ed. Perception and cognition at century's End,pp.47-68. Academic press, san r:nnedv IM (r9;: . - ;:
Diego, CA.
i,ennedy ilvl (r99-r -- -"
Deregowski ]B and Parker DM (r98s). On a changing perspective illusion within Vermeer's
iiennedl'li\{ and F., .. .,
The Music Lesson. Perception, 17, tj-zr.
1. j9r-qo6.
Deregowski )B, Parker DM, and Massironi M (rSS+). The perception of spatial structure with
r:¡sten D, Nfama.s,.'.
oblique viewing: an explanation for Byzantine perspective? Perception,23,5-r3.
in depth. Perc¡::: '
Devlin lT, Moore CJ, Mummery CL et al. (zooz) Anatomic constraints on cognitive theories
¡.,n CY and B1a¡¿ l.
of category specificity. Neuroimage, $, 675-85.
abstract painti:t:. i
Di Dio C, Macaluso E, and Rizzolatti G (zoo7). The golden beauty: brain response to ciassical
:..,-rurtzi Z and K¿r-,.-.,
and Renaissance sculptures. PIoS ONE, 2, e72o1..
implied motro:r.
-'.
Dolev JC, Friedlaender LK, and Braverman IM (zoor). Use of fine art to enhance visual diag-
l.,.zlorvski LT anc --:
nostic skills. Journal of the American Medical Association, z86, tozo_t.
light displar. F:',:-
Elkins J j996). The Object Stares Back. Simon and Schuster, New york.
!..:ekelberg B, \-a::..:, r-
Fleming RW, Dror RO, and Adelson EH (zoo3). Real-world illumination and the perception
.-ortet. /olrritir. ,' j-'..
of surface refl ectance properties. J ournal of Vision, 3, 347-68.
i.:¡on- Xl (19E6 , 1 ,
Fox R and McDaniel C (1982). The perception of biological motion by human infants. Science,
Press, Cambrid¡:
2]8,486-7.
,:rd \f F and H¿','.--. .
Freyd lJ (1983). The mental representation of movement when static stimuli are viewed. ¿ctivlties? \ l-,i..': ¡ ...
Perceprion 4, Psychophysics. 33, 575-81.
-- Corbusier i rgS¡ - .
Geisler WS (1999). Motion streaks provide a spatial code for motion direction, Nature,
4oo, -.-.nardo da |inc- -:-
65-9.
Book Compan',. ),,,.
Ghuman AS and Bar M (zoo6). The influence of non-remembered affective associations on
-:e¡mitteF irgE_: . -,-.
preference. Emotion, 6, zt 5-23.
ro6. rj7-ii.
Gibson II (rSSo). The Perception of theVisualWorld.Hotghton Mifflin, Boston, MA.
- :¡rgstone \IS : :
-.,
Gombrich EH (196o). Art and lllusion. Princeton University press, princeton, NJ.
Gombrich EH (rq6¡). Meditations on a Hobby Horse and Other Essays on the Theory of Art. - :her PJ (1996 . T;-.
rature ofpictoi..- :.
Phaidon Press, London. :,: o ciaf o tt c.if ,. :: : *;
s i
Gombrich EH (r9sz). The Image and the Eye: Further Studies in the Psychology of pictorial '.1:¡ii L and Frore:::.:.
Repr esentati o n Phaidon, London. ,l::rn L and Pear¡e ,- _l
Gomb¡ich EH (1993). Recognising the world: pissarro at the RA. Royal Academy Magazine,
untan'saccaCic e ',. :
39,33. .l.unsel IHR anc
Gombrich EH (rqqi). Shadows. National Gallery Publishers, London.
.:.
)i ¿tLrosciettce, 29.,: ---
Gregory RL (zoo+). Seeing after blindness: the blind leading the sighted. Nature,
43o, g6. .1:-¡her D (zoor P-:,-,
Grossman E, Donnelly M, Price R, et al. (zooo). Brain areas involved in perception of biologi-
.1.-:her D and Ba¡c :
motion. Journal of Cognitive Neuroscience, t2r 7tr-2o.
ca1
.:ecihc artrvo¡ks. i:.
Hochberg I and Brooks v (1996). Movies in the mind's eye. In D Bordwell and N carroll, ,i--;her D and \lor:-:.,
eds. Post-theory: Reconstructing Film Studies, pp. z68-87. University of Wisconsin press,
.accadtc e\.e nto', a,:--a
Madison, WI.
l:lcher D, Papathr':--:.
bound visual ie:r*:i:
PICTORIAL CUES IN ART AND IN VISUAL PERCEPTION 393
:r..:oi primin g. Visual Hockney D (zoo5). Interview by Lawrence Weschler. In David Hockney: Looking at landscape /
being in landscape. LA Louver, Los Angeles, CA.
.rl:ne the geometry of Hyman J (zoo6). Art and neuroscience. Art and Cognition Workshops, |anuary, 2006.http:l I
-i :'. i': Litld Performance, www.interdisciplines.org/artcognition/papersh5
Itakura S (rSS+). Recognition of line-d¡awing representations by a chimpanzee (Pan troglo-
::l iarallels in art, sci- dytes). Journal of General Psychology, rzr, tB9-97.
|ohansson G Ggz) Visual perception of biological motion and a model for its analysis.
,:- -,r ¿nd BF Anderson, Perception and Psychophysics, r.4, zor-u.
- ::r','ersin' of Southern lohnson MH (zoo5). Subcortical face processing. Nature Neuroscience Reviews,6,266-l+.
Kawabata H and Zeki S (zoo4). Neurai correlates of beauty. Journal Neurophysíology, 9r,
:- -.,giriti\-e inquiry. In t699-7o5.
'i .ic¿demic Press, San Kennedy IM Q974). The Psychology of Picture Perception.Iossey-Bass Inc., San Francisco, CA.
Kennedy IM Q9y) Drawing and the Blind. Yale University Press, New Haven, CT.
-.1 - :r r'ithin \¡ermeer's Kennedy IM and Ross AS (1975). Outline picture perception by the Songe of Papua. Perception,
4,39t-4o6.
: .:¿iral structure with Kersten D, Mamassian P, and Knili DC i,ggl). Moving cast shadows induce apparent motion
-"r.:,-1. j-13. in depth. Perception, 26, 77r-92.
:r ,-:: cognitive theories Kim CY and Blake R (zoo7). Brain activity accompanying perception of implied motion in
abstract paintings. Spatial Vision, zo, 545-60.
1-:, :3::Onse tO ClaSSical Kourtzi Z and Kanwisher N (zooo). Activation in human MT/MST for static images with
implied motion. Journal of Cognítive Neuroscience, rz, 48-55.
r .:rir¿nce r.isual diag- Kozlowski LT and Cutting lE GgZl).Recognizing the sex of a walker from a dynamic point-
light display. P erception (t P sychophysics, 2r, 57 5-8o.
Krekelberg B, Vatakis A, andKov¡ziZ (zoo5). Implied motion from form in the human visual
:--': :nd the perception cortex. I o urn aI of N eur o phy si ol o gy, 9 4, 43n-8 6.
Kubory M (rg36). The Psychology of Perspective and Renaissance Art. Cambridge University
:r *:r.rn intants. Science, Press, Cambridge.
Land MF and Hayhoe MM (zoor). In what ways do eye movements contribute to everyday
-,: ¡n¡ruii are viewed. activities? Vísion Research, 4r, 3559-65
Le Corbusier (t98). Towards a new architecture. Dover, New York.
:,:..:ion, Natttre, 4oo, Leonardo da Vinci (rqz¡). In Leonardo da Vinci's Notebooks, EMC Curdy trans. Empire State
Book Company, New York.
-,-.-i1r e associations on Lhermitte F (tg8¡). Utilization behaviour and its relation to lesions of the frontal lobes. Brain,
1L06,237*55.
-, l,- ston, \1A. Livingstone MS (zooz). Vision and Art: The Biology of Seeing' Harry N Abrams, New York.
:.:-.-1. -\J. Locher PJ (rqq6). The Contribution of eye-movement research to an understanding of the
,. '": :l:¿ Theory of Art. nature of pictorial balance perception: A review of the literature. Journal of the International
Association of Empirical Studies, r.4, t43-63.
: :,:.,-;:.riogl of Pictorial Maffei L and Fiorentini A (rqgf ). Arte e Cervello. Zanichelli, Bologna.
Matin L and Pearce DG (rq6l). Visual perception of direction for stimuii flashed during vol-
, .,.'. -:,:.;,iertt1, Magazine, untary saccadic eye movements. Science, r48, t4B5-8.
Maunsel JHR and Treue S (zoo6). Feature-based attention in visual cortex. Trends in
N eur o s cien ce, 29, 3r7 -22.
: 43o, 836.
,'','-;:i;r-c'. Melcher D (zoor) Persistence of visual memory for scenes. Nature, 412, 4or.
:. r.i;eption of biologi- Melcher D and Bacci F (zoo8). The visual system as a constraint on the survival and success of
specific artworks. Sp atial Vision, zt, 347-62.
-::r,.':Li and N Carroll, Melcher D and Morrone MC (zoo3). Spatiotopic temporal integration of visuai motion across
'-. -: \\-isconsin Press, saccadic eye movements. N atur e N eur o s cien ce, 6, 87 7-Bt.
Melcher D, Papathomas TV, and Vidnyanszky Z (zoo).Implicit attentional seiection of
bound visual features. Neuron, 46, 723-9.
394 IJAVID MELCHER AND PATRICK CAYANAGH
Messaris P (rss+). Visual Literacy: Image, Mind, r'" Reallry. Westview Press, Boulder, CO.
Metelli F (tgZ). The perception of transparency. ScientiJic American,23o, 91*8.
Myslobodsky M (zoo3). Gourmand savants and environmental determinants of obesity.
Obesity Reviews, 4, rzr-8.
Neri P, Morrone MC, and Burr DC (1998). Seeing biological motion. Nature,395,894-6.
Nougier LR (rq6q) Art Preistorique. Librarie Generale Francaise, Paris.
Oram M and Perrett D (rSS+). Response ofthe anterior superior. polysensory (STPa) neurons
to 'biological motion' stimuli. Journal of Cognitive Neuroscience, 6,99-u6'
Panofsky E (tgzlltggt), Perspective as a symbolic form. Zone Books, New York.
Ramachandran VS and Hirstein W (rSSq). The science of arI. Iournal of Consciousness Studies,
6-7, t5-5t.
Rao H, Han S, |iang Y, et al. (zoo+). Engagement of the prefrontal cortex in representational
momentum: An fMRI sttdy. N euroimage, 23, 98-103.
DI
Reisz K and Miller G (rq68). The Technique of FiIm Editing. Focal Press, London.
Rhodes G, feffery L, Watson TL, Clifford CWG, and Nakayama K (zoo3). Fitting the mind to TF
the world: face adaptation and attractiveness aftereffe cts. Psychological Science, t4, 558-66.
Rousselet G, Fabre-Thorpe M, and Thorpe Sl (zooz). Parallel processing in high level catego-
rization of natural images. Nature Neuroscience, 5, 629-30.
Savarese S, Fei-Fei L, and Perona P (zoo4). What do reflections tell us about the shape ofa
mirror? Applied perception in graphics and visualization (ACM SIGGRAPH), us-S. RL
Senior C, Barnesa l, GiampietrocY, et aL (zooo). The functional neuroanatomy of implicit
motion perception or representational momenfum. Current Biology, to, 16-zz.
Solso RL (ry9$. Cognition and the Visual Arts. MIT Press, Cambridge, MA'
Stice E, Spoor S, Bohon C, and Small DM (zooS). Relations between obesity and blunted stri-
atal response to food is moderated by the TaqlA Ar allele. Science,322' 449-52.
Tabibnia G, Satpute AB, and Lieberman MD (zoo8). The sunny side of fairness: Preference Tn
for fairness activates reward circuitry (and disregarding unfairness activates self-control
circuitry). Psychological Science, tg, 339-47.
Tatler BW, Gilchrist ID, and Land MF (zoo5). Visual memory for objects in natural scenes:
From fixations to object files . Quarterly Journal of ExperimentalPsychology Section A-Human
Experimental P sychology, 58, 93t-6o. UNpBn the entn -S.::
Terzopoulos D (rsss).Artificial life for computer graphics. Communications of the ACM,4z, five pages, hear ih r-, *
32-42. and cartographic ;:::
Thorpe S, Fize D, and Marlot C (tqq6). Speed of processing in the human visual system. 'Chopin, Frederic F:.
Nature, 38t, 520^2.
tration, ata1ll The r:
Tiérou A Gggz). Dooplé: The Eternal Law of African Dance.Rotfledge, London.
may suffice, here. -i. '
Tucker M and Ellis R (rgg8). On the relations between seen objects and components of poten-
tial actions. Journal of Experimental Psychology: Human Perception and Performance, 24, very probabl,y, lnán..'
83o-46. incipit rtmay be oi s," ::
Vuilleumier P and Pourtois G (zoo7). Distributed and interactive brain mechanisms during Treccani (a ponderc u,
emotion face perception: Evidence from functional neuroimaging. Neuropsychologia. 45, Chopin, Fryden-k F::
174-94. is illustrated rvith th: ,
Vuilleumier P, Armony L Driver J, and Dolan RJ (zoo¡). Distinct spatial frequency sensitivities The author of tjl. .
for processing faces and emotional expressions. Nature Neuroscience,6,624*3r.
University of Bonr: -
Yonas A and Arterberry ME (rqq+). Infants perceive spatial structure specified by line junc-
fions. Perception, 23, t427-35.
Brussels. Ironicalh ¡i:
Zeki S (rqqq). Inner Vision: An Exploration of Art and the Brain. Oxford University Press, of an accident. \\-h.L: I
Oxford UK. adventures are all r er-"
friendship, collabo r: :
;::. Bouider, co.
io, i1-8.
'irlli¡ants of obesity.
i";":,:;- 395,894-6.
CHAPTER 20
.-.s,:,r' STPa) neurons
"
:
':'rii.
--¡';-;;l¡ ¿1-s¡res s Stu die s,
THE MANY
--:'- :r representational DIMENSIONS OF
THE THIRD ONE
.
-
-.rdon.
: . :rtting the mind to
-:",-,i;:iriie, t 4, 558-66.
-: rr high level catego-
Sight', a ciever ., ¡.
'Out of Sight, oui :
of human behalr. ,-
the 'Stereo-Telerneter'was built in 1899 by H. de Grousilliers. This object, designed by procedure \4¡as tL, :, _
Zeiss, was able to provide the fláneur de campagne', in love with bell towers, bombs, scope was just t-':::
and explosive deposits and factories lost among lovely groves, with the exact distance instrument pro\',i;:
of these objects from his artillery. Not only that, but the clever machine was also able humans are soci..-- .
to produce a pair of accurate stereo-pictures of the landscape, facilitating the profes- percept public, i:. .
sional killing. If we would look at fhe Britannica, again, under the entry'Sight', in Italian, D'Almeida ir-r t85r. -,,
we would read: 'mirino, sistemi di puntamento, ecc' (systems of targeting) and we enjoyment of the s-.:
would be rewarcled by a beautiful illustration showing the 'Krupp Independent Line of porary glory, \\,as .:.
THE MANY DIMENSIONS OF THE THIRD ONE 397
Sight', a clever system of aiming, which was well appreciated throughout Europe.
'Out of Sight, out of Mind' one could say (or, in Italian'Lontano dagli occhi, lontano
dal cuore', where, curiously enough, the British 'mind' is transformed into its more
Mediterranean equivalent 'cuore'). 'Sight', as well, claims more than seven pages. In
comparison, 'Karl Philip Emanuel Bach', in Germany, should be content with his
three-quarters of a page.
Prof. Pulfrich was not only the most competent man in Europe in the perception
and measurement of 'stereopsis', i.e. spatial vision, but he was also a subtle observer
of human behaviour. So, he writes: 'The development on stereoscopy has in no way
been uniform: on the contrary, a long period during which practically no interest
was taken in stereoscopy or stereoscopic phenomena, was preceded, during the
middle part of the r9'h century, of universal interest . . .' (Encyclopedia Britannicatgzí).
The essential reason for such an absence of interest may have resulted from the
basic ignorance on the part ofthe potential user about the actual working ofthe instru-
ment. Obviously, Herr Pulfrich had at hand the final solution, always with the help
of the faithful Carl Zeiss. Pulfrich's observation, however, is a surprising one because
the stereoscope popular inhistimeswas actuallyaverybasic objectmade up ofaholder,
two frames, and an opaque screen in between. The'working'was very simple. The left
frame supported a small print, the left picture; the right frame did the same with the
right one. The diüding, opaque, separator allowed the right eye to see only the right
image and the left eye to see only the left one. The two pictures could be drawn
by hand or, more 'scientifically', obtained with a'two lens' camera. This special dou-
ble-lens camera had the two lens axes parallel one to another and separated by a dis-
tance of exactly 65 mm. Sixty-five mm is the average distance of the two pupils in the
adult man; for the woman, the value is 63 mm. I am afraid that the historical stereo-
scopes were gender biased. The simultaneous observation of the images induces an
irresistible, compulsive, illusion: vision in depth. It is a simple, primitive, easy-to-
control experience, nothing difficult, nothing abstract: a plain sensation more than a
true perception. It is, also, a quite 'private', almost'secret', experience. Although the
mechanical part of the illusion appears simple and quite elementary, still the emotion,
the surprises, and the cognitive puzzles are complex to solve and the psychological
management of the 'illusion' may turn out to be healy and complicated. Pulfrich was
very right at pointing at this more cultural than sensorial blindness.
The basic fact, very trivial and elementary, is that the so-called third dimension,
when used to represent objects alreadytri-dimensional, was not'historicallyhealthy'.
.,: úe 'stereoscopes', On the contrary, for millennia the most used, economical, factual, and practical
::. obiect, designedbY procedure was to scale down from three to two dimensions. The history of the stereo-
:: '¡el-l torvers, bombs, scope was just one of the many chapters of this history. As mentioned earlier, the
,rú. the exact distance instrument provides its user with a most intense, albeit, very private experience. But,
:r¡;hine lvas also able humans are social and gregarious and so they would not miss the chance to make the
a;ilitating the profes-
-
percept public, or even better, political. Two technicians, Rollman in 1853, and
e:-r''Sight', in Italian, D'Almeida in t858, devised different strategies aimed at the public and contemporary
rt targeting) and we enjoyment of the stereo effect. One of the most successful ones, which enjoyed a tem-
: Independent Line of porary glory, was the clever idea to oblige people to use the right eye to see the right
398 RUGGERO PIERANTONI
image and the left one to see the other one. The camera, or later on, the movie camera
had two lenses again, the right one screened in green, the left one in red. The two pic-
tures, or films, were projected, superimposed on the same screen. The audience was
provided with disposable spectacles, having the right eye screened in green and the
TN
left one in red. The consequence is simple, the two images are separated: right image to
the right eye, left image to the left eye. Unfortunately, the shows met with disaster from
-ET
the economical point of view. Space was conquered only much later and, surprisingly
enough, by sound: the Dolby Stereo system vicariously conquered the space aban-
t'4*:,:
doned by the defeat of three-dimensional light.
I
-
-
verse ofhis endeavour. This 'illustration' ofthe Sphinxwas shown at his one-man show
at the Museo Pecci di Prato, Italy, held from ro October to 9 fanuary zooo (Fig. zo.4).
Italo Moscati, in his opening text of the exhibition catalogue, chose the image as the
whole allegory of the exhibit. The painting, realized in a pale monochrome and very indistinct, as if n¡-.
deliberately blurred and smeared, was given a title: in the characteristic descriptive objects, their visua- i
style normally used for historic photographs: Die Sphinx von Gise (Sphinx des Kónigs gestalt. Unde¡ thes:
Chepren). Im Hintergrunde die Piramide des Kt)nigs Cheops. 4 Dyn. ( um z6oo v. Chr.). totally unable to sul
Even the title itself calls attention to the 'visual clifl between foreground and success of the stere':,r
background, lhe Hintergrunde.Bufit is precisely in those areas of the painting that the clearly defined opr:;,
artist blurs the image making the boundary between foreground and background very small distan¡c,.
THE MANY DIMENSIONS OF THE THIRD ONE 399
Kg.
:i. the movie camera
:n red. The two pic-
r. The audience was
eJ in green and the
?
.r;ted: right image to
er r.-ith disaster from
:e i and, surprisingly
:ed the space aban-
TO
S HARP
D
¡¡;urrrering volume,
.rrttr.dl position, with
:,:,rular for stereo-
:raphers Felice and
:s q'ite large prints:'
aa.¡¡. r 4 . rt, ,. r
r ;rispness and preci-
.Jarruan' I 4. l9'J2
,:rr¡al vision of the
:. :ailosophical uni- Fig.2O.2 Cover of the The Saturday Evening Post, January 'l4th '1922.
ris one-man show lArtist: Norman Rockwel l.)
r ¿:
L.l:-, :ooo (Fig. zo.a).
: s. the image as the
. -,1¡',hrome and very indistinct, as if melting into a viscous space. The geometric and optical limits of the
:. ¡:eristic descriptive objects, theirvisualboundaries, have been almost completelyeradicated from the main
:-.. -i_rlrln.t des Kónigs gestalt. Under these conditions, even a perfect stereo-pair of the Sphinx would be
."" :-tlit:6oo v. Chr.). totally unable to suggest the spatial layout of the image. A technical condition for the
ri-n loreground and success of the stereoscope is that the two pictures should be generously endowed with
- the painting that the clearly defined optical boundaries and also rich in abrupt changes in luminositywithin
:nc and background very small distances. This may be more precisely stated, in a more academic fashion,
400 RUGGERO PIERANTONI
Fig.2O.4 Gerharo
pairs of correspr-,
Fig. 20.3 The Sphinx and the Giza pyramid.
can be considere;
(Photograph: F.A. Beato.)
simplifying assu:r:
numbers: its corr:d:
i.e. the two reiin::
that if the contrast gradient is at its maximum, the stereo effect is at its best. The'ter- two eyes then ¡hrs
mini expediti delle cose' as Leonardo writes, are the basic elements facilitating the sponding points. -,'
stereo illusion. It is exactly this element, the strong discontinuities in the image, which
located into the ';,
provides the basis for the working of the hardware and software of our perceptual number of points.
system. The outcome of this computation will be served to the'higher' cognitive cen- mustfall on poirlri
tres to allow them to produce, eventually, the very idea of space. Or, more accurately, value between the .
of the reciprocal distance among objects present in a scene. the result is zero. ¡,\
In contrast, Rockwell's painting shows us, close to the shoes of the American boy, for that point. -\ll r:
other stereo-pairs which are still to be seen or already seen. Rockwell had worked computation, dtrre
hard in making them very recognizable, despite their small size in the illustration. intense, and true :¿
It is easy to recognize the 'Colossi of Memmon', the Saqqara pyramid, and a this computatio:l I
lateral vision of Abu Simbel. All images are sharp, simple, and strongly contrasted: a internal model r',f
sure hit. that the differenti¿"
The physiological mechanism which enables us to build inside us a strong sensa- tion of millions c,f
tion of the distance between two objects involves a differential computation between sharpness ofthe sre
THF, MANY DIMENSIONS OF THE THIRD ONE
pairs of corresponding points in the two retinas. In short: the two retinas
can be considered two surfaces sharing the same system of coordinates. Under these
:' ::ograph: F.A. Beato.)
simplifying assumptions, each point of the two surfaces is defined by a couple of
numbers: its coordinates. Points which occupy the same positions on the two surfaces,
i.e. the two retinas, are called corresponding points. If we fixate a point with the
two eyes then this point, belonging to the 'object space', projects on to two corre-
.: ¡t its best. The'ter--
sponding points, which belong, now, to the 'image space'. Now, we consider an object
lents t-acilitating the
located into the 'object space': it may be thought of as composed with an infinite
r. rn ihe image, which
number of points. If we fixate only one among these, the images of the all other ones
i:: trf our perceptual
must fall on points which are not corresponding ones. If we calculate the differences in
lL$er' cognitive cen-
value between the coordinates on the two retinas of the only one point under fixation,
'-,r. more accurately,
the result is zero. We call this number (zero in this unique case) the'retinal disparity'
for that point. All the other ones must have a retinal disparity different from zero. This
,:,: :he American boy,
computation, done for all of the couples of corresponding points, produces a personal,
i,-.1¡'el1 had worked
intense, and true feeling of the distance from us to the object. And, equally important,
-:. in the illustration.
:::a pr-ramid, and a this computation provides all the spatial information necessary for us to build an
st:rrnglv contrasted: a
internal model of its spatial organization. We extract the immediate consequence
that the differential computation of all the retinal disparities (an immense computa-
1¿. us a strong sensa-
tion of millions of points) is strongly based upon a clear vision of the object or the
sharpness of the stereo-pairs.
aLrn]putation between
RI]GGERO PIERANTONI
as is the reassuring feeling it should convey to obedient, tax-paying citizens. Around a \\¡e are, certainl-.
rectangular table we see the officer with some citizen in the act of depositing his :trrverful pror-ince. F
contribution. The money, apparently, is represented as thick discs of metal. To the rficer, not a centr:-
side, the officer stacks a well-ordered pile of documents: the receipts of the deposit. :¿r'eais an extrern¡1.. :
Near to the documents we see a basket full to the brim with the discs. If we look at the r;:'
: -.r and his team
marble slab from a'legitimate' point of view, at its centre and along a horizontal line :l Line with a lon.-..
ofsight, the scene and its spatial setting are explicit and 'ecologically correct'. The table
"re spatial settir.rs .:
is horizontal, rectangular, and the documents form a perfect rectangular pile. The
basket exhibits a logical cylinder-conical shape and its opening is, obviously, circular.
The suggested depth of the whole scene is about 5o to Zo cm measured along the
horizontal direction. Note that the whole scene is completely logical and absolutely
explicit. But if we measure, ruler in hand, the distance between the closest point of
the slab (to our eyes) and the farthest point, then we discover that this distance is
onlyrS mm.
In reality, the surface of the table is strongly tilted (it is almost vertical) and the
rectangular documents are slender lozenges that are not piled one on top of each
other (as coins would be) but rather arranged like the tiles of a very oblique roof. Even
more dramatic is the basket: its mouth is an elongated ellipsis and not a circle at all.
Although the scene is squeezed into a tiny slab ofspace, our perception ofthe scene is
compulsively 'spatial' in a way that is comparable to looking into a stereoscope. The
slab was sculpted by an anonymous provincial marmororius around the second
: le¿r-e the lab and century AD. It is a small provincial work that was likely commissioned by a central
r:¡¡ under which a administrator or a local ruler. This bas-relief is not unique. Another similar one has
-; prLrcesses which, been found in France, at Saintes, in the 'Gallo-Roman' provinces of the empire
: sn¿l1 museum in (Fig. zo.6). In this work, the picture is executed at a strong angle in relation to the plane
of the bas-relief, probably due to issues of space. The advantage of such limitation for
:: is taking place in us is that we may look at the two works side by side. The surface of the table appears
:.irnonition is overt, almost vertical, as in Trier.
: ;rtizens. Around a We are, certainly, not at the far limits of the empire; Gallia was an important and
;t t-'i depositing his powerful province. But we are not in Rome, so the functionary must have been a local
¡;: trf metal. To the officer, not a central figure. Yet despite this, the sophistication ofthe representation
-r¡rs oi the deposit. reveals an extremely high competence in the illusive treatment of the space. The sculp-
.¡.¡. If r''e look at the tor and his team must have been exposed to highly sophisticated works of arts, directly
i: ¿ horizontal line in line with a long-standing Hellenistic tradition and technique in the rendering of
-'*. c --'rrect'. The table the spatial setting of a complex scene. A comparison with an almost contemporary
.¡¡nzular pile. The
. br iousil', circular.
::r:¿sured along the
:-;¡- and absolutely
:r¡ closest point of
:::rI this distance is
:!'res
Fig. 20.6 Tax Cottection, bas-relief. Saintes (France), Musée Archéologique.
RUGGERO PIERANTONI
work found in Rome may be of some interest (Fig. zo.7). The Hateri were a rich and
influential family of builders. Under Domitian they commissioned a sculptor to
create a detailed slab representing their family shrine which would communicate their
professional skill, technological achievements and private power. The detail we are
looking at is the roof of the temple-shrine. The roof appears to be oriented in the
same way as real buildings of this type. The line of the tiles follows the slope of the
pediment as it should appear in the real space. But here the surprise is similar, albeit,
N,{enc Hr
- C:r¿rbonna::-,- -:
-.:i,r-\i,1i ---: i -.
Ia-1:, _ :
. .a'ihar].ar] tn .j,::.. :. ,
:.:ple of F.,::::. -
:
__ __
.--_LtLrl uf'Lr-1,\_ _!-.
,-- ^r:\.L---, ,--
-:-> t)l-1LIlú-l: -: -- :-
::Sr¡:ute periec:l -..
.: .-r:jr¿ir¡-.. ffi::,::' . .,,
l-,.,-.r short qui-r:::- : !
::.ri \\'ere a rich and opposite to the solution of the two'taxation'slabs. The objective orientation of the
-,-,red a sculptor to sculpted roof is perfectly vertical, and this can be easily deduced in the picture by the
i ¡':'mmunicate their shadow cast by the roof on the surface of the slab. The lines of the tiles are strongly
:. The detail we are oblique, because they have to be parallel to the pediment's sides. Their inclination is
:¡ oriented in the logical more than suggestive. But the effect is again achieved with mastery.
,, ',',
: the slope of the At Treviri the horizontality of a table has been obtained by cutting the marble along
::.c t. similar, albeit, a very slanted surface; here a slanted surface, the roof, has been obtained with a series
ofoblique lines, the tiles cutting obliquely across a vertical surface. In both cases the
impression of the orientation of a surface has been very successfully obtained. The
examples show how easily, under a certain point of view, our stereo-vision can
be fooled.
We have considered three cases showing how a tri-dimensional representation
may suggest a convincing and coherent spatial setting, even despite 'optically wrong
s1'ntax', or ignoring (or perhaps being ignorant of) the canonical laws of stereo-vision.
It is clear that the tri-dimensional representations of tri-dimensional objects have
met both with unexpected success and equally bitter disappointments. In the third
dimension there is more than meets the eye(s).
In developing the image of the procession of the Athenians around the Parthenon's sekos, the
sculptor instead has matter and space at his disposal, and his/her work suggests with ease the
thickness of reality and the living diversity of the city.
(Loraux r98r, pp. 284-5)
Beyond this well-known 'train illusion', Stillwell also considers the way that the frieze
would guide the behaviour of the observer. For example, the west frieze runs high,
above the opistodomós, where the offerings from the islands that were part of the Delya
League were deposited. The insistent invitation to proceed, to walk further, would
'discourage'the observer to attempt entry:'The motive of starting off the procession
might be thought of as discouraging any arrangement that would incline the visitor to
stop with the idea of entering the west door. . .'. The opposite effect takes place on
the east frieze where an accurate symmetrical setting of the figures and their spatial
relationships strongly suggests the end of the procession and guides the observer to
enter and make an offering to the Goddess, waiting inside: 'Since the east frieze
represents the culmination of the Panathenaic procession there is every reason to sup-
pose that the intent was to allow the greatest interrelationship between the sculpture
and the architecture. . .' (Stillwell 1969).
This leads us to consider the role of the spatial structure in guiding behaviour.
The three-dimensional orchestration of the representation ignites a chain reaction of
perception, motion, visuomotor interaction, thought, and even personal and shared
memories. However, a technical detail should be taken into consideration: the relief is
much higher than the one adopted for the Ara Pacis. This higher relief was necessary
to introduce into the volume of the sculpted space the necessary'optical corrections'
intended to suggest complex spatial relationships between the bodies: mainly the
self-occlusion of a single body and the reciprocal ones between two or more bodies.
THE MANY DIMENSIONS OF THE THIRD ONE 407
But, ahigher relief suggests a stronger and more eloquent playwith the shadow, which
l' ::thenon's sekos, the
would be expected to be intense under the Acropolis's sun. Both of these points will
! iiq3sts rvith ease the be considered now-and the second one will bring us a disconcerting'surprise'.
It was not by chance that Richard Stillwell would express his gratitude to his
.-,::.x rg8r, pp. 284-5) teacher, Rhys Carpenter, who studied the lost statues of the east pediment in great
detail (Carpenter 1933). Carpenter was, eventually, able to reconstruct the dimensions
:i ün its readability and the general outline of the statues by examining the geometry of the pediment,
r. rriLoraux, but add the depth of the recesses, and also from an acute awareness of the viewpoint of the
:.sentation (Stillwell observer (the conditions of visibility and of invisibility, the social station of the
:r it reiies heavily on observer, and his/her potential movement in relation to the gods).
1;::.ations. The opti- We have observed a link between the local use of space in individual episodes of the
:-L:i to the perimeter frieze and the overall space surrounding the temple. This consideration brings us to
1. F¡om each one of introduce the unusual theory of the architectural space in Greece proposed by the
i1 :ne1e of45 degrees. young Greek architect C.A. Doxiadis in his controversial, but fascinating, study on the
::1' Lrther point the distribution of the structures in ceremonial and religious centres in classical Greece.
::;ept of an observer It is not by chance that the title of one of the most interesting histories of architecture
written by a Greek architectural historian, Spyro Kostoff, is Rituals and Settings.
':''; -ng, and observes Doxiadis's analysis is based upon very careful surveys of many complexes: Acropolis
,-..i11-)n opposite to his III, Sanctuary of Aphaya in Egina, sanctuary of Atena in Pergama, and many others.
':.:¿nt landscape seen In all the case studies, the strategies of the complexes show a careful avoidance of
:: :::idlv backward. symmetry so that the main structures would have had to be seen from the main entrance
, Stillrvell 1969) points, always from the side rather than from directly in front of the building. There is
the elimination, or, at the least, a deliberate ignoring of any system based on external
r. ',1-a\-that the frieze
coordinates such as the classical grid. According to Doxiadis, the centre of the coordi-
ist rieze runs high,
nate system-in polar coordinates rather than Cartesian ones-is found at a point
'.:e part of the Delya midway between the two eyes of the observer. In other words, each object in the
'..- ¿-i further, would
space is defined in relationship to the observer and his visual system. An external,
r¡ t¡i1 the procession
eternal, autonomous 'space' is cancelled and replaced instead with an 'observer-
:ncline the visitor to
-:racji takes place on centred' estimation of distances and depth. These observations of the architect will
appear even more pregnant later in this essay, when we consider that Donatello's bas-
¡;s and their spatial
reliefs would also evoke an observer-centredralher than object-centredpercept.
*ies the observer to
As mentioned earlier, the observations of the Parthenon lead to a disconcerting
::r;e the east frieze
'surprise'. The frieze runs veryhigh on the wall of the cella, immediatelybelowthe ceil-
-','en- reason to sup-
ing at about 14.3 m of height. Thus, direct light never reaches the sculptures and the
::!\'een the sculpture
only illumination comes from reflections off of the nearby surfaces: the lateral portion
of the adjacent columns, the pavement, and the external platform of white rock upon
;-uding behaviour.
which the Parthenon rests. The greater contribution of illumination comes from the
:. : chain reaction of
wall of the cella itself. We have to consider the orientation of the building with its long
:;:iona1 and shared
axes oriented along the east-west. In this case the northern side will never receive direct
'er¿¡ion: the reliefis
light from the sun (see later for other geographical data), and this implies that the
: :¡-jeirr.as necessary
':p:ical corrections' cella's wall is lighted only by the sky and a portion of the platform far from the building
'::dies: mainly the perimeter. At the south side, the cella is illuminated only at the end of the day when
the rays of the sun shine almost horizontally from the horizon. The consequence is,
¡\ Ll Lrr more bodies.
very simply, that the sculpture is practically invisible. Further information comes from
408 RUGGERO PIERANTONI
the Parthenon built in Nashville, Tennessee (Fig. zo.8). The cladding, as far as the
r,\'ith her:"1-:.
absorbance of the light is concerned, is quite similar to the more noble marble from .-t:rral 'hanc.-¡ .
Paros. All of the dimensions and architectural detailing are quoted verbatim from the
original. The Nashville Parthenon does not have the frieze, but if you walk along the
narrow path between the columns and the cella and you look high, the frieze
could well be there: but you would never know for sure, one way or the other. Athens's
latitude is about 38 degrees north and Nashville is 36 degrees north. Sun illumination
is, then, more propitious in Nashville than in Athens.
BnuxELLl
STEREO-'
idd.ng, as far as the With her tlpical clarity of statement and argumentation Gisela Richter, in her
r noble marble from seminal'handbook' on greek art, writes these lines about the frieze:
c i-erbatim from ihe The 'scorcio' is, now, perfectly understood. The farther parts of the bodies appear reduced
f i'ou rvall< along the in size according to a gradual recession of the planes. Moreover, although a frontal, uniform,
"c,k high, the frieze plane is always perceivable, the variations of the depth of the relief where the different forms
,r rhe other. Athens's intersect each other give the strong impression of standing or moüng figures, one near the
ü. Sun illumination other ones. The archaic bi-dimensional idea has been substituted with a tri-dimensional one
where the sensation of depth has been perfectly obtained.
(Richter r959)
Unfortunately, in our case, the tri-dimensional 'effect' has not been obtained because
the light, i.e. the shadows, are absent. It is, for certain, very difficult, at least for us, to
solve the problem posed by an immense, mar-vellous and surely very expensive art
work, plunged, deliberately we may say, into an almost perfect darkness. The now
christened Elgin Marbles have been made perfectly visible, but they are located in the
absolute wrong position and under historically inaccurate lighting. The third dimen-
sion also needs, apparently, shadows in order to be perceived. Retinal disparity it is not
sufficient, at least so it seems.
Fig. 20.9 Donatello, St. George and the Dragon, bas-relief, 1417-142O.
Florence, Orsanmichele.
This exposure to the so-called costruzione legittima, here at its beginnings, will
find even greater realization in the altarpiece for the St. Anthony church in Padua
j445-t45o). The sculpture, in bronze this time, explores the intersection between
the rigorous graphical implementation of classical perspective and the techniques of
three-dimensional rendering. In a few words, we find, astutely woven together, a real
stereo-stimulus, which works on weak but pervasive retinal disparities, and a 'cogni-
tive-perceptual' structure which analyses all of the geometrical cues. It is not difficult
to recognize here some of the basic procedures that were seen earlier in the 'tax collec-
tion' reliefs of the marmorariiin Gallia and Germany. The inheritance, in Donatello's
work, of Hellenistic plastic solutions is beyond discussion. But a very dramatic
novelty has been introduced: the stage simulation. Susanne Lang, in her study titled
'Brunelleschi's Panels', analyses the intentions of Brunelleschi to devise and produce
the two famous panels. Lang rules out the idea that the purpose of the'pannelli'was
Fig.20.10 Dc'al
merely to produce some didactic, or magic 'perspective exercises'. The main purpose,
instead, was a theatrical proposal for suggestive backdrops, which were to be used on
the stage (Lang r98o, pp.6Z-Zz). destruction ol :..:
It is of some interest to look now at Donatello's Herod's Banc1uet created in ilerod ,\grip¡,; i:, .
1423, with its clear-cut division between the actors on the stage and the backdrop is a terracotte sti.*:
behind them (Fig. zo.ro). The Ara Pacis, fhe lonic Frieze, even the two modest 'tax col- The statue rgas t:i:-
lection' slabs mentioned at the beginning, did not show a foreground that stands out rrith the Padua .. :,:
against a fully separate background. Here, the physical, objective distance between the doubt that the p;,:-t.
heads of the actors and the 'scenario' is dimensionally significant when considered sional procedlLre! ::. i
in relation to the linear dimensions of the whole bas-relief. The retinal disparity, tr-escoes.The st¿r,,-. ..
then, must play a role in building our internal 'sense of space' when looking at this as to whether thc :.::
bas-relief. In contrast, in a flat painting, the only cues to distance are based on line on a painting.
and colour. The two vu-orks ::.-
The professional life of Mantegna was spent at the intersection between two statue obvioush' l---.¡
and three dimensions, and he made frequent and deep excursions across this dimen- different points tr- ',,
sional border. A classical case in point is the centurion saved from the quasi-total relation to a pain:-:,.
THE MANY DIMENSIONS OF THE THIRD ONE 411
{"
4T2 RUGGERO PIERANTONI
Fi9.20.12 1,,1
Mantova, lvl'll
same chapter Agosti himself writes: 'We have not to forget that, in this cultural envi-
, Destroyed fresco,
ronment, it was common to frnd the painter-sculptor and the sculptor-painter'
and this was the case because the passage from one technique to the other one was
;.
common in the 'botteghe' of a few years before' (Agosti 2oo5, p' 15). However' this
lateral shift between two and three dimensions, or the reverse, is not syrnmetrical. The
soulce and the final text are connected in a complex way and, something always' gets
:ling the unity of the
lost in translation, or, even, invented. But the langue is similar and in both texts the
.ependently from fhe
same mistakes and the same errors are possible.
.:j an anamorphosis).
,,l.-ts and r.iews rather
r:eise of painting vs.
j,liogue between the
rr r sophisticatedpas-
i ;u-,m the fecundation Sr. IevrES MARTYRED AND
ne follower of
, Sa,xon
LOST IN TRANSLATION
i trtrc¡k Su Mantegna I
n r¿se oi Ulisse Aleotti,
ii 1-{J8, by the young The damage to the Eremitani fresco has almost completely cancelled the figure of the
:' . Further down the saint, who stands against a pillar to the extreme left of the panel (Fig. zo4). A good
414 RUGGERO PIERANTONI
: a,omputation.-::.:
.: ::pth beLrr'een ::= '
:.rnot be obtaine; ,,".
Fig. 20.13 Copy from Mantegna, 5f. James, tempera on wood. , ,:-i;.ctical rules c': : =
copy of the fresco, translated in a quite different pictorial technique, allows us to recon-
struct with some confidence the original appearance of the fresco: the copy, which is
almost contemporary with the original, is on display at the Musée |acquemart-André
in Paris. The saint is a giant; his stature is twice that of the other persons around him.
At first glance, the size of the saint appears as a very rough 'perspective error', also
because another 'normal sized' figure shares the same marble slab and stands a few
centimetres away from the giant. But the next panel seems to deny the 'error'. In fact
the body of the saint, which lies on the ground, is being carried away, with great efforts,
by his tormentors. Here, the physical contact between an average-sized person and the
saint is made explicit and compelling because the immense leg is directly handled. t\
Actually, the perspective treatment of the enormous leg and of the body is almost
perfectly rendered, which shows, as if this was necessary, the Mantegna's full control in
,i
!--
I
reserved only for the martyrdom, seems to be ruled out. Mantegna's work on the panel
was confined to between 453 and 1457. Similarly, Donatello worked on the Padua
altarpiece until r45o-in particular on one of its panels, the Mirecle of the new-born
child,whtchwas produced between 1446 andr45o (Fig. 20.14).In manyways, the scene
is typical of the Donatello of those years: it shows a violent and instantaneous
commotion of agitated people. But the key element which we seek is a 'gigantic'
figure of a young man, standing against a pillar, which opens the narrative on the left
side. The man has climbed a pedestal to watch the scene from above. But, apart from
this different placement in the scene, he also appears out of scale in relation to the
other people.
Could this be another 'perspective error'? By such an expert as Donatellol Of course
not: the young man is not a giant. He is only located higher than the other actors. But
we are not allowed to see the pavement. The vanishing point is kept so low that the
the eye level is practically coincident with the base line. Donatello, very astutely, repre-
sents the lateral sides of the pedestals in such a way as to have their receding
surfaces slightly trapezoidal in shape, obtaining in this way the total cancellation of the
base plane. In absence of any explicit indication of distance we 'must' deduce that the
other persons are acting quite far away from the 'limelight'. Our active and very sensi-
ble computation of retinal disparities is able to record the weak, but intense, difference
in depth between the young man and the others. This sensory-perceptual aid to depth
cannot be obtained within the fresco language for two reasons, one technical the other
linguistic. First, the fresco lies on a single, flat, vertical plane and so the retinal disparity
algorithm does not work under this limiting condition. Second, the grammatical or
r" OOd, syntactical rules of paintings, in Mantegna's time, did not allow for the elimination
of the horizontal plane (upon which the human figure rests, or runs or stands).
rlattamelata-rr hicl
-rbvious. From d¡a,ri
:'rom two to thre¡ *
,diom. The translar¿5
ioost-Gaugier l9 ).
.¡'[e has predisposeJ
the third dimension. -,
'.he Gattamelarn.
T¡=
onwards, a small
1439
ro the history ot-rhc ,
:hree dimensions r p¿:
It is difiicult ro rh:-_
.rhelmed by its as.u::
Jimension in the par_r
¡ervasive. This e.t¿:: _
3 Dimer.,_
*
scale. But his reliei ..
much higher series .,- :
glance at lhese \\(,:]i
Fig.2O.15 Mantegna, Martyrdom of St James, c.1455. Destroyed fresco, formerly rr ith great evidenc.. :
Ovetari chapel, church of the Eremitani, Padua. Detail. ttew-born child. Bw., ,-.
rative because \\'e nt _:
prominenl wings rfi;- -
Has Mantegna committed a perspective crime? Of course not. But he may have have the aspect otlo... -
attempted a backward translation from Donatello's 2.5 dimensions to his full z.o ¿t the canonical poi:.-
dimensions. But the two syntaxes were different-while Donatello could easily do away
with the horizontal plane at will, Mantegna could not.
shaped. When seen :. .
'deformation'. This ri _
A short passing example may clari$. this asymmetry in translation. But, this time, DelPozzo ceiling rn S,_
the first (source) 'text'in our translation is a tenuous, delicate drawing, while the final
technique used for ¿ ..
(destination) text is an enormous sculpture, indeed a very bulky one. The notebook of
sional work. But urir. .
Jacopo Bellini is a collection of fabulous, almost immaterial and delicate drawings; rich surface pat tern, Tl_,
some of them represent fable-like architectures and settings.
orthogonal direction.. .
One of the most famous drawings from this collection is the Cavaliere. Drawn pattern in the pavem. _ .
in c.r4zo, it represents a horseman, or a perhaps instead a statue of a horseman. The face pattern helps enr.:i
similarity between this drawing and Donatello's monument to the war lord polished and homos.:-..
THE MANY DIMENSIONS OF THE THIRD ONE 4r7
Never has the classical mantra of the Gestaltists been more appropriate: 'The
whole is more than the sum of its parts'. The evaluation of the reciprocal distance
among physical objects is a final internal statement, which is the result of a flowing,
continuous, and multilevelled process. The retinal disparitymaybe the first step, but it
is immediately 'followed' by computation of luminance levels, shape recognition,
mental rotation of the perceived object, and more. When Donatello re-represents' on
a much larger scale, a volume very similar to a real horse, he plays on a lower symbolic
scale. But his relief sculptures of the San Lorenzo Pulpit in Florence sets in motion a
much higher series of perceptual-cognitive sequences (Figs. zot6 andzo.rT). A cursory
glance at these works may be useful. In the panels, the pavement is often displayed
led fresco, formerly with great evidence, following a strategy opposite to lhe St George or the Miracle of the
new-born child.But,inlhe Pentecosf, the vision of the pavement is necessary to the nar-
rative because we must see crosses on the ground. Again, the stage is confined between
prominent wings that protrude from the background. These wings, in Italian 'alette',
;,:. But he may have have the aspect of low buildings covered by a tiled roof. When seen with the eye located
:ri-L'r:rs to his full z.o at the canonical point, according to the rules of perspective, they appear correctly
. r .truld easily do away shaped. When seen away from the 'legitimate' point of view, then they show all their
'deformation'. This procedure is identical to the one followed in large frescoes, as in the
,-:rion. But, this time, DelPozzo ceiling in Sant'Ignazio's church, in Rome. We have, here, a representational
::-,,irlg, tvhile the final technique used for a two-dimensional (flat) representation applied to a three-dimen-
The notebook of
¡r::¿. sional work. But using a more basic trick, Donatello has given the walls of the wings a
,ri delicate drawings; rich surface pattern. The suggestion is that we are looking at a surface woven along two
orthogonal directions, such as a wall made with reeds. We may recognize the same
..:te Cst'aliere. Drawn pattern in the pavement of the fresco by Mantegna. But there is no doubt that the sur-
:,i trf i1 horseman, The face pattern helps enormously to give retinal disparity. Had the walls being perfectly
.ni to the war lord polished and homogeneous, the perceived depth would have to be computed based
4T8 RUGGERO PIERANTONI
Fig. 20.16 Donatello, Pulpit of the Resurrecfron (The Pentecost), bronze, c.1 465.
Church of San Lorenzo, Florence.
on the luminance only. Here the explicit and intense pattern ignites a process, dis-
covered (or at least explored) by Bela Julesz, which we will discuss later.
The disconcerting observation of the 'invisibility' of the Parthenon frieze deserves :ri¡ror and its su::¡--.
a completely different cultural analysis: after all the St George and the Dragon together l: is difhcult to be :-
THE MANY DIMENSIONS OF THE THIRD ONE 419
with other numerous examples of sculpture are perfectly visible. The difficulties in
seeing the relief have nothing to do with our discussion about the dialogue between
'dimensions'. But the absence of light, and thus of shadows is, instead' of paramount
importance. A different approach may be of some usefulness. In the decade between
196o andt97o, a sudden acceleration occurred in the field of anatomy, physiology, and
biophysics ofvision. One of the leading factors was the decision to 'concede'the power
:sil cfonze, c.1465. of vision to machines. A quasi-perfect handbook on this subject may be suggested for
the inter'ested reader: Stephen E. Palmer's Vision Science: Photons to Phenomenology
(rSSS). Practically all the lines of inquiry, from anatomy to robotics, from physiology
to information processing, have been excellently treated. Here we deal only with
.ia::a:: IIOCeSS, diS- one subject: form from shading. It has been demonstrated that a non-planar surface
aI i_::.
illuminated with a single source, or, in more complex cases, an extended one,
gives information about its curvature. The perception of shape may be perfect even if
the surface is absolutely homogeneous under all points of view. A surface perfectly
homogeneous and diffusive is called lambertian, from the name of the French scientist
who theorized such optical objects. A lambertian surface provides only information
HE SUN about luminance, but on these bases only our perceptual system is able to extract
spatial information. Such surfaces, by definition, lack of any form of reflection. But
this theoretical condition is seldom obtained. Common, every day surfaces show a
mixture of reflectivity and specularity. A very smooth surface tends to behave as a
l_
J-¡11 . c: mirror and its surface is constellated with intensely luminous spots: the reflections.
iL-rga:jla: it is difficult to be fooled by these luminous areas and to take them for 'objects'.
ri{,
42O RUGGERO PIERANTONI
Fig. 20.1 8
Donatello. David a.k.a. Mercury, bronze,
c.1440. Florence, Museo Nazionale del Bargello.
Let's consider this beautiful picture of Donatello's David. The statue is made with
extremely smooth bronze. In particular, the photograph shows the body of the boy
spotted with 'white' irregular areas. Our tri-dimensional reading is not hampered at all
by this black-white puzzle. The form, the spatial distribution of the volumes, stands
out, immediately.
Even if we do not know that these 'white spots' are the virtual images, deformed,
shifted, enhanced, obliterated, of the source/s of light, we are able, nonetheless, to
read the surface with ail its evolutions, convolutions, and curvatures. It took a
thousand years to connect the geometry of the surfaces to the reflections and then to
the light sources. But we have performed, quite successfully, the job for the same
thousand years.
Let's move, now, to a quite different sculpture, a different materiai and, above all,
a very different photograph. Giovanni Becatti, for years the head of the Italian
archaeological mission in Athens, was given the unique opportunity to photograph
THE MANY DIMENSIONS OF THE THIRD ONE 427
the Ephebe by Critias (Fig. zo.r9). He decided to have the statue brought to the open
and, under the azimuthal sun of the Acropolis, Ephebe produced a series of unique
images. Due to the surface and internal micro-structure of the marble from Paros,
we have no reflections, no bright spots, but a continuous variation of the luminance
across the body. The boy is lambertian from this point of view. But, exactly as for
Donatellot figure, we build our own internal rendering of the surface, without too
much effort. The process is, then, independent and based only on computation of
Iuminance value on a very local region, but at the same time extended to the whole
image. The simple fact that the intense perception of tri-dimensionality comes from a
bi-dimensional image rules out, automatically, retinal disparity algorithms. The ways
of the 'third dimension' appear, and perhaps, are infinite. The 'shape from shading'
algorithms are well treated in a large amount of literature, including, of course, Robol
vision (Klaus and Horn r986; Churchland and Sejnowski 1989).
A popular paper on shadows and their role in shape recognition appearedin Scientific
Americanín 1988. One of the most quoted illustrations was the one where some objects
appeared as concave cavities. It was enough to re-orient the paper and to look at the
same illustration, now upside down, and, 1o!, the previous shallow plates were trans-
formed into convex bumps. The image was computer generated, monochrome, and
without any other detail. The mere substitution of a black pixel with a white one, and
this coherently through all the greyscale, was sufficient to revert the spatial perception
(Ramachandran r988; Johnston and Passmore r994). Shadowswere not enough to gen-
erate, or to suggest, shapes. The scientists' brilliant obserwations had been dutifully
anticipated by Leonardo, of course. In the "Precetto 38: Dell'obbligo che ha Ia scultura Ar
col lume, e non la pittura" introduced into the Parte Prima del Trattato della Pittura, \\.IT
Leonardo writes: 'If the sculpture is lit from below, it will appear to be a strange and
monstrous thing; this does not happen in painting, which brings all of its parts with it'.
In another precetto, number 33, Leonardo takes into consideration the illumination
conditions under which the sculpture was originally conceived and built:
(. . .) a mortal enemy of the scuiptor in the high and 1ow relief of his works, which will not be
worth anything unless they have the lighting set in a way similar to how it was when they were
being sculpted. The reason is that, if they are lit from below, the sculptures will appear verv
Imonstrous] and most of all the bas-relief, which almost becomes impossible to recognize and
origin to opposing readings Iofits appearance]
gives
Leonardo connects the conditions under which the sculptor had worked and recom-
mends they must, or should, be present again at the moment of the observation of the
completed work. A rather mysterious connection, indeed. The lighting conditions
under which Donatello had worked on his St George and the Dragon, or Celiini
on his Perseus, will have no strict counterpart of those 'ecologically' common in Via
Orsanmichele, or under the Portico della Signoria. But this 'limitation', or rather
'peculiarity', of the work of the sculptor is in perfect agreement with the low opinion
Leonardo pretends to have towards his work, 'La scultura non é scienza ma arte
meccanicissima' and, consequently, the sculptor is at mercy of his environment. He
cannot 'escape light', the natural, ecological, local light. The painter makes himself
THE MANY DIMENSIONS OF THE THIRD ONE
independent from the immediate environment and proceeds, boldly, to follow his
veryprivate'maggiore discorso mentale'. The artworkis object centred, space centred,
and not observer centred. The tri-dimensional image is built according to a system of
external reference. Leonardo cannot be more explicit than here in the Excusazione dello
scultore, j7:
The sculptor says that he cannot make a figure, without making infinite figures due to the
DI SOTTO coninuous and thus infinite characteristics fof a sculpture]; one can answer that the infinite
characteristics ofsuch a figure can be reduced to two half-figures, i.e. one halfback figure and
l¡ one halffront figure. . . and these two half-figures by having all oftheir proper relieves in all
their parts, will represent, without needing an¡hing else, the figures that the sculpture claims
to have made.
The sculptor, 'claims to have made all ofthe infinite figures'but he did not create them.
::,¡eared ín Scientific It is the observer, an intelligent painter, maybe, who actually created the forms. But
:. ',",'heresome objects Leonardo in his extreme disdain is not alone. A form of self-sufficient image technol-
r.r and to 100k at the ogy had, independently from him, mechanized the process. It appears that walter
-,'".,' Plates lvere trans- Benjamin's 'age of mechanical reproduction'had already appeared.
:. nonochrome, and
-,f,:ih a \\'hite one, and
:l-.¿:: :.,
Fig.20.2O The definitor, etching from L.B. Alberti, De Statua,1464. r -l-.: -
such as Dürer, Raffael, Michelangelo, Leonardo, etc. did not need such contrivances
in order to draw a naked woman, or a lute or a vase. They had only to certify the level
of investigazione ed invenzione sottilissima degli studi matematici': exactly those
competences that Leonardo, so brutally, denies to the sculptor. The extension to
the third dimension of the 'quantitative procedure' by Alberti, and not by him alone,
confirms the mechanical procedure and lower conceptual level of the act of sculpting.
The use of an external reference system confirms the object-centred philosophy, or
better technology, of the sculptor. But these 'machines', real or only dreamt about,
have not sprung to life by themselves. Behind them we may see, and clearly, the com-
pass, the cartographic representation of the whole planet, Ptolemy, and the scientific
THE MANY DIMENSIONS OF THE THIRD ONE 425
The Raphaelesque solution follows the creation of a base line, similar to Pulfrich's ste-
reoscope. The building's elements, corners, columns, etc. are aimed at from two differ-
ent directions. The applied 'trigonometry' was very straightforward. Apparently, a
different approach was followed by Leonardo in his celebrated Plan of Imola. rn
this case it seems that he followed a system of polar coordinates. Rome had been sur-
veyed according to the Doxiadis procedure, Imola according to Alberti's cylindrical,
self-centred hero. Although Christopher Columbus was moving into an external refer-
ence system, he was always at the centre of his space. He was sculpting his road among
the waves.
While Alberti's 'cylindrical hero' may be well known, let us now turn our attention
to a much less celebrated image, which shows the atelier, museum, and studio of
Bartolomeo Cavaceppi in Rome (Fig. zo.zr). Bartolomeo Cavaceppi was an intelligent
artist, modest and erudite, admired for his dedication, competence, and virtuosity
by all over Europe and, in particular by Winckelmann (see Pinelli 1986). His publi-
cation Princeps, which followed the commercial trend inaugurated by G. B. Piranesi,
appeared in Rome between ry68 and ry72. Our illustration taken from this book
shows, at the professional level, the complex interactions between sculpture and the third
1 464. dimension. The museum-studio-atelier in Via del Babuino shows, in a very un-
Piranesian way, many statues under restoration. Some of them are located under large,
rectangular frames, probably iron, which have been fastened, horizontally, to the walls.
From the side of the frames plumb lines graze significant points of the statue:
:ei such contrivances an index finger, a prominent hip, or the fore leg of the faithful greyhound of the god-
:,-. rt-, certify the level dess Diana. At first glance, we may think about assonance with the polar coordinate
:::lici': exactly those system of L. B. Alberti. Nothing could be more misleading. At the studio in Via del
":. The extension to Babuino, the statues arrived already made and consumed by history. They were
u-ri not by him alone, historical subjects under close study and tender care. Alberti's illustration suggested,
- f the act of sculpting. instead, a creative or, atleast, productive process. The basic act ofsculpting involves
philosophy, or carving away the stone from a block where the creature may be thought of as still sleep-
'n::ed
r ':,nlr- dreamt about, ing and waiting for the Pygmalion caress. In contrast, in Cavaceppi's studio, the work
:rd cleariy, the com- took place in the external, public, historicized space. For Cavaceppi, the sculpture was,
*-.. and the scientific in substance, only surface. For Alberti it was still a volume.
426 RUGGERO PTERANTONT
Fi1.20.22 ia.
used to en a"tr
in expansion or contraction, works more under the rhetorical laws of analogy than
according to the mathematical operations of diüsion or multiplication.
Raccolta di antiche
reo Cavaceppi
illuminated upside down are revealing: 'e massime il basso rilievo, che quasi cancella
negli opposti giudizi la sua cognizione'. It is sufficient, apparently, to substitute a black
dot with a white one in order to spatially invert an image. And to lose its 'knowledge': a
typical, seemingly human capacity. But before entering into this new world, our own>
where machines have conquered the rights to see, recognize, classifr, memorize, and
manipulate physical objects, it may be wise to visit again the tax collector's office in
Trier. The marble slab presents to us a basket with elongated ellipses as its opening,
rhomboidal documents, a table at a nearlyvertical plane, and so on. But we 'recognize'
the objects immediately, without hesitation. And this happens because we know, we
already have 'knowledge' of it. But machines do not pay taxes. The 'objects' appearing
in the slab are not 'objects', they do not 'appear', the 'basket' is nothing but a swift tran-
sit o[electrons into a computer circuit.
The great enterprise to which, in the rgzos, the vision scientist Bela Julesz set to
work was the absolute destruction of shape. And the creation of some luminance pat-
terns which, despite their almost perfect absence of meaning, could generate a strong
stereoscopic effect' even in a machine (Fig. zo.4). So he invented the Random 1ñ
Dot Stereogram (RDS). This 'stimulus' has been perfectly defined by Patricia S.
t0ú1¡
Churchland and Terrence J. Sejnowski in their book on the 'computational brain'
Here, the definition of the RDS is reported, verbatim:
[A] Pair of images made up of random patterns of black and white dots. The two images are
identical except that in one image [a] set of dots has been displaced to the right or the
Ieft by a few dots widths. When fused by the two eyes the displaced dots appear to be at a
different depth plane from the background.
(Churchland and Sejnowski r99z)
It can be easily seen that the ancient stereoscope has had a very long shadow. After
all, a RDS is a stereo-pair that transfers to the observer the computation, point
to point, of the retinal disparity. The significant difference is that each of the two
'stereo-pairs' has no meaning. No Sphinx, no approaching train: just ten thousand
black and white dots scattered at random on the visual plane. To feel as a machine may
do, it may suffice to look at an 'informal' sculpture. Even before falling into some
aesthetic trap, or cultural suggestion, our brain is immediately set to work. Is this 'thing'
convex or concave, transparent or solid, shiny or matt, cutting or smooth? An impres-
sive number of experiments with 'meaningless' objects had defined an interesting
boundary between solid, bulky, massive 'things' and spidery, light, filamentous ones.
Let's consider, just as a pure suggestion, a sculpture by Henry Moore and one by
Barbara Hepworth. A scientist interested in the phenomenon called'shape constancy',
and to whom the two names are unknown, would concentrate on the 'self occlusion'
factor (Palmer 7999, pp. 327-i.A Hepworth sculpture will show a much reduced _ :--:-( -: ,: ,
self-occlusion in comparison with one by Moore. Shape constancy is the ability we . :, .-,:: rr ith ¿ ::.-
-- -..1- ! . +- _
have to recognize, for example, a square or a disc under all possible orientations, !r ..11 LLr ---.,,i
:,i _, ' _ __t
even the most extreme ones. Leonardo, again, must have the floor: 'Dice lo scultore :l-:e¡n ther h;,,¡:
che non puó fare una figura, che non ne faccia infinite per gl'infiniti termini che ¡ ,.i-e SSlr e Stud', ,- : -,-.
hanno le quantitá continue'. It has been demonstrated that the correct perception :=:ns rer-ealed the -l:-.
THE MANY DIMENSIONS OF THE THIRD ONE 429
.- lela
Julesz set to Left eye view Righl eye view
,-
-:: luminance pat-
*-- ienerate a strong
'" -::::d the Random
I 0 t 0 t 0 0 1 o 1 t 0 0 1 o o I 0 1
.-::l bv Patricia S.
!
I t' 0 1 0 1 0 0 0 1 0 0 1 0 t 0 I o 0
:r:utational brain'.
G 0 I t 0 1 I 0 1 0 0 0 I 1 0 1 I 0 1 0
0 1 o A B B o 1 0 1 0 B B x o I
., -ie nvo images are
I x
:- .. :he right or the 1 1 B 0 1 1 1 1 B A B Y o 1
* .) :ppear to be at a 0 0 1 x I 0 0 o I A s Y 1 o
¡ I 1 s I 0 't 1 f B I B X 0 I
.- ":l Sejnorvski r99z) t 0 o 1 0 1 1 0 1 1 0 0 1 1 0 1 I t,
I t ,l
:. l- nq shadow. After 1 0 0 1 1 0 1 1 0 0 I 1 0 1 1 1
. -;::putation, point o 1 0 0 o 1 I t I 0 o 1 0 0 o t 7 I o
:. :. ¡ach of the two D¡splaced cenler squar€
,:,r ;*st ten thousand
=:- :s a machine may
:. ::Lhng into some
, ',.,'¡rk. Is this'thing'
I An impres-
:-:1 -roth?
,- :l:l an interesting c
. -: ..1;nentous ones.
- l,l,-,.¡re and one by Fig.20.23 Bela Julesz. Random-dots stereogram (RDS).
.,: : :aape COnStanCy',
:-.,-e'seii occlusion'
'.,, :
much reduced of shape is very low with objects with a low 'self-occlusion factor' and yields good
:: ---',. rs
the ability we results with a high 'self-occlusion'. Bela lulesz intended, at the beginning of his
-,.:-.le orientations, inquiry, to provide machines with information about space, even if the luminance
:: lice lo scultore pattern they have to rely upon was deliberately built as meaningless, at least for us. A
successive study on the more 'private' statistics of the already thought-of random pat-
- ,:,-niti termini che
i.: , -- lf 3Ct perception terns revealed the unexpected ability of our visual system to 'make sense' of the deep
RUGGERO PIERANTONI
statistical structure of the RDS. It is as if the study of space-started for and held on
behalf of the machines-had given place to a more deep inquiry into the visible world
for us (still) humans.
marble turning to life . . . the effect is marvellous when we observe the group, in the dark,
under the torch lights.
Evidently, in the experience described in the Propylren, the statue (we know it was a
copy seen in daylight and the alternation between light and dark was obtained with the
artifice of closir.g and opening the eyes. But the'real' experience Goethe had intenselr'
felt comes from his time in Rome when he saw the statue in its niche in the Vatican
Gardens: 'the Laocoon, in his niche, can be seen appropriately only by torch-light'
(Goethe 1989, p. zr8).
For us, what had taken place in the eyes, or better, in the brain of Goethe, is decep-
tively simple. The 'pixels' of the 'Laocoon one' change state every fraction of a
second, giving rise to lhe'Laocoontwo'. Most of the background did not change, so the
retinal disparities set in motion by the 'movement' of the pixel provided the necessarl'
motion in space: Laocoon movesl In this case only the light had moved, not the
eyes and, fortunately, not the serpents. The motion of the light had created the
third dimension. Retinal disparity can either be interpreted in terms of depth
(space) or movement (time). In fact, for perception, space and time are intimately and
inextricably linked.
We conclude this discussion of the third dimension in art with another kind of
motion. On z4 March 1787 Goethe arrived in front of the temples in Paestum. He
is disconcerted, he does not understand them, they appear alien, almost beyond com-
prehension: 'at first sight they excited nothing but stupefaction . . . these crowded
masses of stumpy conical columns appear offensive and even terri8ring . . . It is only bv
walking through them and round them the one can attune one's life to theirs'. These
immense stereometric volumes may be understood only by walking around them,
moving your eyes in relation to the columns, to their shadows. The wanderer evoked
by Stillwell has to move around the Parthenon friezes in order to 'attune its life to his
own'. A surprising conclusion arises from these excursions to the temples: 'Repro-
ductions give a false impression; architectural designs made them look more elegant
and drawing in perspective more ponderous than they really are'.
This imaginary conversation with Bela ]ulesz should have taken place while having
a drink at the buvette in the central hall of the Frankfurt Airport under the lofty, yet
benign, eyes of an intensely tri-dimensional Goethe, clad in plaster-white garments.
It is a sculptural translation of the famous two-dimensional portrait of the writer
by Iohann Heinrich Wilhelm Tischbein (Frankfurt Stadhische Galerie). Walking
around the statue, one wonders whether giving Goethe this third dimension added,
or took away from our perception of the profundity of the writer.
RnpnnnNcEs
:l:: group, in the dark, Carpenter R (rS¡¡).The Lost Statues of the East Pediment of the Parthenon. Hesperia, Journal
of The American School of Classical Studies at Athens,2) 1-110.
Churchland PS and Sejnowski Tl (1989). The computationalbrain. MIT Press, Cambridge, MA.
Je \\ e know it was a Finch C (r9zz). Norman Rockwell's America. Harry N Abrams, Inc., NewYork.
/,r-i obtained with the Goethe JW (rq8g). Italian journey, WH Auden and E Mayer, trans. Penguin Books, London.
r-¡ethe had intensely fohnston A and Passmore PJ Ogg+) .Shape from shading. I: Surface curvature and orientation.
Perception,4, 169*89.
niche in the Vatican
' ¡';rlr- by torch-light' foost-Gaugier CL (r98o). Some considerations regarding the Tuscanization of Iacopo Bellini.
In MD Emiliani. ed. La prospettiva rinascimentale codifcazioni e trasgressioni,pp.t65-76.
Centro Di, Florence.
: ¡f Goethe, is decep- Kellum BA (rgq+). What we see and what we don't see. Narrative structure and the Ara Pacis
-l'en. fraction of a Augustae. Art History, t7,26-45.
¿r,1. not change, so the Klaus B and Horn P (rq86). Robot vísion. MIT Press, Cambridge, MA.
¡,-'-,ided the necessary Lang S (r98o). Brunelleschi's Panels. In MD Emiiiani, ed. La prospettiva rinascimentale codifi-
:rad mor.ed, not the cazioni e trasgressioni, pp .q*72. Centro Di, Florence.
Loraux N (t9Br). L'inyention d'Athénes. Histoire de I'oraison funébre dans Ia'cité classiqud.
rJ.:t had created the
Mouton, Paris, pp. zB4-5.
1 in terms of depth
Pacciani R (r98o). Ipotesi di omoiogie fra impianto fruitivo e struttura spaziale di alcune opere
:i3 ¿re intimately and del primo rinascimento fiorentino. In MD Emilianí, ed. La prospettiya rinascimentale. codi-
ficazioni e trasgressioní, pp .73-95. Centro Di, Florence.
'.'rl:r another kind of Palmer SE (1999). Vision science. photons to phenomenology,pp.z4z-6.MlT Press, Cambridge,
::les in Paestum. He MA.
. ¡L¡ost beyond com- Parronchi A (rq8o). Donatello e íl potere. Cappelli, Florence.
Pinelli OR (1986). Chirurgia della memoria: scultura antica e restauri storici. In Storia dell'
*...thesecrowded
Italiana, volj: Dallatradizione all'archeologia. Einaudi, Turin, pp. r83-2o5.
:rrrng...Itisonlyby Ramachandran VS (1988). Perceiving shape-from-shading. Scientific American,25g,76-83.
. *ii: to theirs'. These Richter RMA (rqsq). A handbook of Greek art. Phatdon, London.
:-li:ng around them, Stillwell R (r969). The Panathenaic Frieze. Optical Relations. Hesperia, Journal of The American
The',randerer evoked School of Classical Studies at Athens,38,4r-4l
- '¿t:une its life to his Uttal WR, Davis NS, and Weike C (rss+). Stereoscopic perception with brief exposures.
-1e temples: 'Repro- Perception ú Psychophysics, 56, 599*6o4.
.::l look more elegant
l' ['l¿n.
CHAPTER 21
FILM, NARRATIVE,
AND COGNITIVE
NEUROSCIEI{CE
lNrnooucrroN
CoupnBu¡NDING a film is an amazingfeafof neural and cognitive processing. A series
of still pictures are projected quickly on a screen, accompanied by a stream of sound-
and a viewer has an experience that can be as engaging, emotionally affecting, and
memorable as many experiences in real life. Comprehending film is all the more
amazing when one considers how films are constructed. Films tlpically are comprised
of hundreds and into the thousands of individual camera shots, which are continuous
runs of the camera. Shots are edited together in such a way to create scenes, which
are sequences ofgoal-directed actions or unintentional events that take place in a par-
ticular location (or one or two locations when events are depicted as occurring con-
currently). Shots on average last only a few seconds and are edited together in such a
way that the vast majority of them go unnoticed by the viewer (Magliano et al. zoor;
Bordwell and Thompson 2oo3; Smith and Henderson zooS).
Consider Fig. zr.r, which depicts about r5 seconds of film from the James Bond film,
Moonraker (Gilbert 1979 ) . This shot sequence takes place after a fi.ght in mid air between
a villain, Jaws, and Bond, in which Bond escapes. Shot r depicts the villain pulling the
ripcord for his parachute, which breaks. Shot z is a high-angle long shot of the charac-
ter reacting to the fact that the ripcord broke. Shot 3 depicts a reverse, high-angle long
shot and the character starts to flap his arms. Circus music, whose source is not yet
$6 IEFFREY M. ZACKS AND IOSEPH P. MAGLIANO
apparent, starts during this shot and continues throughout the rest ofthe scene. Shot +
is where things get particularly interesting from our perspective. This shot depicts a
\ .:1,
seemingly incongruent view of the outside of a circus tent; Shot 5 returns to a medium
shot of the character who continues to flap his arms. Shot 6 depicts the inside of a circus
tent. At this point, many viewers generate the inference that faws will land on the
circus tent (Magliano et al. 1996). It is impressive that despite the fact that the visual
flow of information across shots in film (including those depicted in Fig. zr.r) bears
little resemblance to the perceptual flow of information as we interact in the real world
(Cutting zoo5), viewers can process the juxtaposed and seemly disparate camera
shots and comprehend the story events depicted in them with ease. Viewers perceive
spatial and temporal continuity across the shots in Fig. zr.r (Magliano et al. zoor),
allowing them to infer that the villain will fall into the circus tent. This example is
particularly impressive because viewers must infer that the villain is spatially over the
circus tent and that the events depicted in Shot 6 are actually taking place inside the
location depicted in Shot 4.
Sequences such as this vividly illustrate that viewers construct representations of
events in film that go far beyond the physical stimulus. How do the mind and brain
transform a stream of flickering lights and oscillating speakers into a coherent world?
Some parts of how people understand film can be explained by examining how the
components of film are processed at perceptual, cognitive, and neurophysiological
levels: Much ofwhat is said about music and sound in this volume is important for film
comprehension. But somehow audition, vision, and their interaction aren't the whole
story. This is because films do not simply present audiovisual signals. Rather, in most
cases films are vicariously experienced events (Tan rggt; Tan ry96; Zillmann 1995;
Copeland et al. zoo6; Magliano etal. zooT). Events have their own psychology above
and beyond the psychology of the sensory modalities. Thus, to understand how film
works one needs to understand how experiences ofevents are constructed from audi-
tory and visual signals. To do so, we will take a cognitive neuroscience approach, which
4
means we will be interested in phenomena in terms of their information-processing
properties and in terms oftheir neurobiological properties. We will discuss both behav-
ioral and neurological data that shed light into how filmed events are processed and
how comprehension emerges from those processes.
Fi7.21.1 A story u:
:: r.rfesentations of
:. . :irind and brain
, r .oherent world?
- '.:rlrning how the
- :-.rroph,vsiological
.: -:llltlrtáflt for film
Shot 2 Shot 5
.: r ;ren't the whole i-...\.*..,
i: ...
';r{,';¡,1..',."
""....-..r"r\,it*.
- .-s. Rather, in most ... ;}";
'
-::: Zillmann 1995;
',. :.i-chology above
-: :.¡stand how film
,-:::lcted from audi-
:r- --: lf prosch, which
. ::::.1;ion-processing
-.: s:ussbothbehav-
Shot 6
:: fl3 PrOCeSSed and
Fig.2"l.1 A story board depicting a scene from the James Bond film, Moonraker.
16 frames per second to z4,flicker rates tripling to 7z frames per second, visual contrast
increasing, colour being added, sound being added, and going from one channel, to
two, to five or six. This increased rate in the presentation of images does not change
the essential task of a viewer: úewers must construct a mental representation of the
depicted events. This situation is complicated in film given the fact that camera
shots are often íilmed at different times and places and that individual shots that
contain minimal feature overlap can be edited together in such a way that viewers
perceive continuity of action in spite of the lack of feature overlap (Bordwell 1985;
Bordwell and Thompson zoo3). (Shots 3 through 4 in Fig. zr.r illustrate this aspect
of films.)
In order to tell a story in film, film-makers rely on formal devices that separate it
from activities 'in the wild' (Cuttin gzoo). Film-makers adopt narrative devices, such
as the cut, framing content in a shot, the placement of objects in the scene, directions
to actors, music, sound, and dialogue (Bordwell and Thompson zoo3). Perhaps the
most significant of these is the cut, which is a term used to refer to the juncture between
shots. At a cut, every point in the image changes discontinuously. How do human
perceptual systems cope with such a jarring transition? According to classical film
theory (Bordwell 1985; Bordwell and Thompson zoo3), cuts can serve two distinct
purposes. The majority of cuts are continuity edits,whtch serve to bridge incidental
breaks in the physical features of activity as smoothly as possible in order to maintain
continuity of action across spatiotemporal discontinuities. Techniques to do this
include preserving the direction of motion across cuts, maintaining ongoing sounds
in the soundtrack, and overlapping the objects and characters visible before and after
the cut. Approximatelyg5% of all cuts are continuity edits (Cutting zoo5). The remain-
ing cuts are scene breaks, at which one action ends and another begins.
Continuity edits are valuable clues to how the brain processes events. This is particu-
larly true for films made in the dominant'Hollywood style'. Hollywood style attempts
to be invisible, so that the viewer is absorbed by the events of the story and does not
much notice cuts, camera motion, camera angles, lighting, and so forth. When a
film-maker uses a technique (say, a cut) and we as viewers do not notice, this is a hint
that whatever is changed by the technique is not very salient to our perceptual and
cognitive systems (Levin and Simons zooo; Cutting zoo5).
Some authors have argued that cuts in general 'work' because they correspond to
visual interruptions that occur naturally due to the movements of the eyes, in particu-
lar blinks and saccades (Murch zoou Cutting zoo5). Blinks, of course, are the brief
closures of the eye that tlpically occur several times a minute. Saccades are rapid bal-
listic movements of the eye that occur when we shift from looking at one thing to look-
ing at another. Because both result in transient insensitivity to visual input, it is
tempting to think they correspond to cuts. The film editor and director Walter Murch
(zoor, pp. 6z-3) proposed that'a shot presents us with an idea, or a sequence ofideas,
and the cut is a'blink'that separates and punctuates those ideas'. He went on to argue
that if you were to watch the eyes of an audience viewing a well-edited film you would
see them all blinking together at the cuts. The viewthat cuts correspond to the transient
blindnesses caused byblinks or saccades is intuitive, but a quick investigation suggests
FILM, NARRATIVE, AND COGNITIYE NEUROSCIENCE
: l:1d. \'isual contrast it can't be quite right. Try this: watch a film or TV programme from a distance of at least
:,-:r one channel, to 6 feet, so the image of the screen on your eyes is not too large. Fix your eyes on the
.e. does not change middle of the screen and do not move them. Don't blink for 3o seconds or so. Did the
:l¡esentation ofthe cuts suddenly appear strange and jarring? we suspect not-and the limited experi-
r¿ :act that camera mental data available support this informal observation (Smith and Henderson zooS).
:-¡-''-idual shots that So, although in natural viewing some cuts may be effectively hidden by saccades or
I : '!ia\- that viewers blinks, this can't be the whole story. Even without transient blindness, a üsual change
:-¡: Bordwell 1985; may be effectively camouflaged by another visual change that occurs near it in time.
rll-istrate this aspect Called masking, this effect is pervasive and can lead to pronounced failures to detect
changes (Rensink et d.. ry97; o'Regan et al. 1999). Many continuity edits may build in
;':;3s that separate it masking by placing the cut within or before a period of fast motion, as when a film
.::a:ive devices, such edilor matches on action. Match-on-action cuts have been found to be less detectable
::3 s.ene, directions than cuts that aren't followed by fast motion (Smith and Henderson zooS). So, cuts
r :rrr1l,l l. Perhaps the may be unobtrusive either because they correspond with brief blindnesses or because
:*; iuncture between they are masked by other visual changes. we propose that these two mechanisms
s-'"'. Horv do human account for the unobtrusiveness of most continuity edits. But this still leaves a bunch
r:i.g to classical film of cuts left over-those that we seem not to notice despite large unmasked visual
: !er.\'e nvo distinct changes that don't co-occur with blinks or saccades.
:-. 5ridge incidental To account for how cuts function more broadly, we make a perhaps counterintui-
::t L-rrder to maintain tive proposal: some cuts may'work' (depending on how they are executed), because
::riques to do this human perceptual systems are already segmenting ongoing activity into discrete events
:::: ongoing sounds all the time. If a cut is placed where the observer would naturally segment the activity,
then the cut will be experienced as natural even if it is readily detectable. Such cuts
'-¡,: betore and after
::-rr:,iI.Theremain- needn't be hidden by an eye movement or visual masking-an1'thing goes. To make
:,:1 S. this argument, we will have to explain a little bit about the psychology and neurosci-
, ;:rts. This is particu- ence of event segmentation.
i. d stvle attempts
"-,."',--,
Fi1.21.2 Example of event boundaries. These frames show the six coarse-grained
event boundaries selected most frequently by a group of younger and older adults
(Zacks et al. 2006b, exp. 2). These boundaries marked the ends of events that
could be described as: 1) Put down the tent.2) Spread it out.3) lnsert the front
tent pole. 4) Stake out the ends of the tent. 5) Stake out the sides. 6) Attach the
rain fly. Reproduced with permission from Zacks and Swallow (2OO7).
F,, c,. t r,
L t V Ji
: six coarse-grained
Jei' and older adults
; of events that
3J lnsert the front
s des. 6) Attach the
[2007).
Fveryday ñvents French Cinema
Fig.2"l.3 (See Colour plate 7) Evoked responses at event boundaries in
¡ - grained events, par-
laboratory-made movies of everyday events, and in a narrative French
.', ents appear to be
art film (Lamorisse 1956). Data from (Zacks et a|.2001, left) and (Zacks et al.
¿: al. :oor). In short,
2010 right). Arrows indicate the approximate location of area MT+.
i:. suggesting that the
"r data to suggest that Finally, researchers have used pupil diameter as a measure of cognitive load during
;¿:tion. The relevant film viewing (Swallow andZacks zoo4). In studies of memory and problem-solving,
rr the event segmenta- pupil diameter has been found to increase as the cognitive requirements of a task
:.;n. Then, the viewers increase (Beatty and Lucero-Wagoner zooo). In this study, transient increases in pupil
r:-.. The event bound- diameter were observed around those times that viewers would later identify as event
:r.rEate the previouslY boundaries. Thus, these neurophysiological data converge with the behavioural data to
.,," }:.at u-as changing in suggest that viewers spontaneously segment ongoing activity into events as a normal
-ie rierr'er would later concomitant of perception.
:¡,d is that any neural What determines when observers perceive event boundaries to occur? One recent
t i i¿]ie eYent segmenta- theory (Zacks et al. zooT) proposes that, as part of ongoing understanding, observers
. :iat they will later be make predictions about what will happen next in an activity. A particular individual
:as been applied using doesn't make predictions about everi'thing that might happen in the activity, every-
r rcluding movies of thing that might change. Rather, viewers monitor a set of features of activity that are
i ¿n ñlm (Zacks et al. salient and relevant to their goals. When predictions about these features are violated,
:¡e ñfRI data showed viewers perceive the onset of a new event.
ed netn'ork (Fig. zr.3). According to this theory, event boundaries should tend to occur when features in an
:,eraphl (EEG), which activity change, because in general changes are less predictable than stasis (though
airr Sharp etal.zooT).
L there are some exceptions).
442 iEFFREY M. ZACKS AND JOSEPH P. MAGLIANO
What sorts of changes matter? Viewers tend to segment activity at physical changes- trn the scene but did :
changes inthe movementofactors andobjects, inspatiallocation, andintime (Newtson ilm theory, a scene :
el al. ry77; Magliano et al. zoor; Zacks zoo4; Hard et aL. zoo6). For example, event .o-occur with spati;,
boundaries tend to occur when objects or body parts are accelerating relative to each Viewers segnenti!
other and moving fast (Zacks zoo4;Zacks et al. zooga). If movement is important for and qpes of cuts tc ¡
event segmentation, one would expect that activity in brain areas specialized for brain activity durii:i
movement processing would be related to event segmentation. This appears to be the rlm theory's accolr:.
case. In particular, area MT+ is an area in the lateral posterior cortex that responds cuts that introducec
selectively to motion (Tootell ef al. ry95; Chawla et al. 1998). As can be seen in Fig. 2r.3, boundaries. Cuts th"
MT+ is strongly activated during event boundaries (Speer et al. zoo3) such that there is ;ontinuity of actron. i
a three-way relationship between movement, event segmentation, and activity in ;oarse-grained bour;
MT+: when things move quickly, activity in MT* is greater and people tend to perceive :nore on the processi:
event boundaries (Zacks et al. zoo6). nore sensitive to .,--:
In addition to physical changes, viewers tend to segment activity at conceptual associated with er er:
changes, such as character goals and causal relationships. These have also been found ',irthin a scene, it h¿c
to predict where event boundaries will occur (Magliano et al. zoo5;Zacks et al. zoogb). sn-rall effect on fine-.:
The neural processes that relate conceptual changes to event segmentation are less ¡ries in both the t-rn¿ .
well understood than those relating movement to event segmentation. However, we do rn action. These d;i:
know that changes to both physical and conceptual features predict activity in most rechniques are suaa::
ofthe areas that increase at event boundaries (Zacks et al. zoro; Speer et al. zooT; Speer .ies and that scene'¡,:
et al. zoog). This supports the idea that we perceive event boundaries because we How does the br¿-:
process changes in the features to which we are attending. some hints. \\re an¿i',
rtas perceived as al ¡
.hange. One set oib::
eventboundaries si-
had previouslr'be ¿i: :
Cu:rS AND CONTINUITY iateral temporal al¡ :
activity at those c'--i:
spatial or temporal .:
With this account of event segmentation ready to hand we are in a position to return cingulate gyrus, a i.r,
to the question of how cuts function. We propose that continuity edits and cuts at remporal lobes. \r ¿ -
scene breaks follow different rules. For continuity edits the cut must be hidden by a spatiotemporal dis;,- :
blink, saccade, visual masking, or something else in order to 'work'. For scene breaks, results do suggest :'-
however, it is not necessary to visually hide the cut if it happens at an event boundary. rvith a network th:: -
This leads to a clear prediction: in a well-edited movie, cuts that classical film theory to srvitches from r;,i.
would identify as scene breaks should be identified as event boundaries, whereas con- continuiry gaps ii.t r---:
tinuity edits should not. trol. This fits rtrti :
We recently set out to test this using behavioural and fMRI data collected while blinks, saccades. ai:
viewers watched The Red Balloon (Lamorisse ry56), a French art film about a boy who This hi,pothesis cc'*- j
befriends the balloon of the title. The Red Balloon is 33 minutes long and contains zr4 of attentional cont:--
cuts. Each cut was categorized based on whether it introduces a change in spatial loca- The studies jus: ü.:
tion (e.g. moving from indoors to outdoors or from a trolley to the street), a change in late the locations i'r
time (usuallyjumping forward to elide an unimportantpart of the activity), or a change Erperimental stuci¡:
in the action being performed. The remainder of cuts produced changes in viewpoint eventboundaries,,r:
FILM, NARRATIVE, AND COGNITIVE NEUROSCIENCE 443
,: rnr-sical changes- on the scene but did not change the temporal or spatial location. According to classical
..-1:n trme (Newtson film theory, a scene break occurs when a new action is performed; scene breaks often
F".r erample, event co-occur with spatial or temporal changes, but this is not necessary.
:::ng relative to each Viewers segmented the film into coarse and fine events. We used the locations
:t is important for
-.e and types of cuts to ask how editing was related to behavioural segmentation and to
::r:s specialized for brain activity during film viewing. The event segmentation data supported classical
:rs appears to be the film theory's account of continuity editing. For both coarse and fine segmentation,
:,:':.er that responds cuts that introduced action discontinuities were quite likely to be perceived as event
r :e seen in Fig. 2r.3, boundaries. Cuts that introduced spatial or temporal discontinuities, but preserved
- _r such that there is continuity of action, were associated with fine-grained event boundaries but not with
-:,-,r. and activity in coarse-grained boundaries. One possibility is that fine-grained segmentation depends
:.'ple tend to perceive more on the processing of physical changes, whereas coarse-grained segmentation is
more sensitive to conceptual changes. Notably, cuts in and of themselves were not
---'.i-:tr- at conceptual associated with event boundaries-if a cut merely changed the camera viewpoint
:¡-, e also been found within a scene, it had no discernable effect on coarse-grained segmentation and only a
.: Z¿cks et al. zoogb). small effect on fine-grained segmentation. In contrast, viewers perceived event bound-
.imentation are less aries in both the fine and coarse segmentation task if the cuts coincided with a change
-:i--,t. Ho\vever, we do in action. These data suggest that-at least for The Red Balloon-continuity editing
:i,.r acti\.ity in most techniques are successful in perceptually smoothing over full-field visual discontinui-
r.3r et al. zooT; Speer ties and that scene boundaries require a break in action.
.r,c¿ries because we How does the brain achieve this perceptual smoothing over? The fl\4RI data provide
some hints. We analysed the ÍMRI response to cuts as a function of whether the cut
was perceived as an event boundary and whether it introduced a spatial or temporal
change. One set of brain regions selectively increased in activity at those cuts that were
event boundaries (shown in blue in Fig. zlfi. These regions overlapped with those that
had previously been found to respond at event boundaries, including regions in the
lateral temporal and parietal cortex. A different set of regions selectively increased in
activity at those cuts that were not judged to be event boundaries but that did have
spatial or temporal changes (shown red in Fig. zr.4). These regions included the mid-
: .r posltron to return cingulate gyrus, a region in the lateral inferior parietal lobule, and the lateral anterior
u:l'' ¡dits and cuts at temporal lobes. We cannot be sure exactly how these regions participate in bridging
:'n;st be hidden by a spatiotemporal discontinuities in these types of cuts based just on these data, but these
:.. . For scene breaks, results do suggest further lines of experimentation. The activated regions overlap
:: ar event boundary. with a network that has been identified with attentional control tasks, in distinction
.: :ies.ical film theory to switches from task to task (Dosenbach ef aL zooT). One possibility is that bridging
ri¿:ies, u,hereas con- continuity gaps in movies requires the same processes as within-task attentional con-
trol. This fits with the proposal that continuity editing works by hiding cuts using
I ;:ra collected while blinks, saccades, and masking-all three are associated with attentional reorienting.
'.,n about aboywho This hlpothesis could be tested using task batteries designed to assess different aspects
,-,¡:s and contains 214 ofattentional control directly (e.g. Fan et aI. zooz).
::l:nqe in spatial loca- The studies just described were correlational-we did not experimentally manipu-
::3 street), a change in late the locations or t)?es of cuts, but took them as they came from the director.
: :crilifi.), or a change Experimental studies also suggest that cuts are less detectible when they correspond to
. ch¿nses in viewpoint event boundaries. In a study of infant perception (Baldwin et al. zoor), infants watched
JEFFREY M, ZACKS AND P. MAGLIANO
'OSEPH
,,--.
';;ron had begu:t
:-:,n ¡esults. ,{s,-, i - l
:J:c: on reco€ilrti :
:::s as conststen: i"1--
¡'. In a llnal sr;i',
'\-:it cuts place d -,::
¡¿sed on pfer.tL-,;r'
,:r:dies, cuts ii :r¡
::r,ndaries. Culs ¿-..
:,-,: details at the l-,::.
: ::rts in the hl::r s.
Bnyoxo c
Fi1.21.4 (See also Colour plate 8J Brain regions that increased selectively at :r¿¡sitions as r''¡-1. - :
cuts that were identified as event boundaries (blue) or points in time that were .:¡lude pauses ir r,,:
discontinuous in space or time but were not identified as event boundaries (red). ). -1n. Segments Lri :--:,
Data from (Magliano and Zacks (under review)). ,:::nselr'es don': .-.;
:,,-,rs do? A sm¿L r-..
;,--rducted br-Bt t.t -
a short film of a woman cleaning a kitchen until they become bored and looked away.
:u:ted bv zero. i:::,
They then were tested with versions of the film that had been modified by introducing
aaur at natural ¡i.:
brief pauses, placed either to correspond with event boundaries or slightly before or
*¿pendent measL¡a!
after. The infants looked longer at the versions with the pauses at non-boundaries,
Boltz 1995i ani ir:'
suggesting that placing pauses before or after the event boundary made the films more
,-Ltntmelcial b¡e¿- :
different to them than placing the pauses at event boundaries. Another experiment
::rore commercials :,
asked adults to detect cuts in movies presented either intact or with the order of
: ,.undaries hurt p ei: .
shots scrambled (d'Ydewalle and Vanderbeeken r99o). Scrambling shot order disrupts
Schrr an and cr-r'-¿;:
the structure of events, and thus we would expect it to make cuts more detectable.
l:r- er-ents ( Schri:n ;'
This was what was found. In a study of memory for events depicted in film, viewers
ri e\-ent boundar::s , ,
watched brief film clips that contained zero or one cuts, and consisted of one or two
replaced rtit.: :, :
actions (Carroll and Bever t976). After each clip they were shown a six-frame excerpt ",'ere
¡r¡ recall as much ¿¡ ;
from the sequence they had just seen or from a similar clip, and asked to distinguish ,,r recall. Horrever. ;
whether the excerpt came from the clip they had watched. The excerpts that did come
t¡r the remaining =¡
from the just-watched clip were selected from either the first or second half. Excerpts
:,imemon'.
from the first half were recognized more slowly than those from the second half,
These studies suSi:
particularlywhen the second half included a new action. This suggests that when a new
:rentation and men - :
FILM, NARRATIVE, AND COGNITIVE NEUROSCIENCE 445
action had begun viewers perceived an event boundary, consistent with the segmenta-
tion results. Also consistent with the segmentation results, cuts themselves had little
effect on recognition. (For reasons that are not clear, however, the authors interpret
this as consistent with the idea that cuts themselves produced psychological boundar-
ies.) In a final study (Schwan et al. zooo), viewers watched movies of everyday events
with cuts placed either at event boundaries or at points in between the boundaries
(based on previously collected normative data). Consistent with the correlational
studies, cuts in and of themselves had little effect on where viewers identified event
boundaries. Cuts also had relativelylittle effect on memor¡ they accentuated memory
for details at the locations of the cuts themselves, but did not affect memory for other
points in the films.
editing techniques such as fades and wipes. Commercial breaks are more extended, Baggett t979 t. Fc
and do not come into play in current popular or art cinema. Pauses (blank screens) are sequence ma\' aoii:_
closer to commonly used editing techniques, but themselves are rarely used. In the .ord), (flap: arms.
future it would be valuable to directly investigate the effects of intrusive editing tech- :epresentation c¿n l
niques as they are actually used in cinema on event segmentation and memory. Fig. zr.r, continLiir
In sum, some cuts in film may 'work' by being camouflaged in blinks, saccades, or I{or.r.ever, deep n:..
other visual transients, but other cuts may work because they co-opt an ongoing pro- described bv the r:¡. -
cess of segmenting continuous activity into discrete events. As an artist, the director or et al. r999). This rs *.
editor has leverage to work with the natural tendencies of the perceptual systern or ilere u,e rvill use ::-
againstthose tendencies. Workingwith naturaltendencieswill tendto produce smooth, lhat u.e mean an a',;
naturalistic continuity editing. Working against them may introduce jarring percep- tron-r real life.
tual effects, comprehension gaps, and memory encoding difficulties. Poorly used, these An event mod¿, :,
may frustrate the viewer, but when used to deliberate effect they may afford a richer ¡f diinensions. sl;l-
and stranger cinematic experience. For example, a jump cutis a continuity edit that is ,ronaiitr'(Zrraan e:.
not well masked and in which an object or person appears in two nearby screen loca- tt al. zoo9b). Ftri : ,
tions on either side of the cut (Anderson 1996').Itproduces the sensation that some- , iervers to gener¿:: ::
thing has 'jumped.' In Dancer in the Derk (von Trier), Lars von Trier uses a hand-held ;:iute and the er e r-,,
camera to film the just-jailed protagonist exploring her cell, and several times drops -nference that ]ar,, s '.,.
a few frames to produce a jump cut. These jumps disorient the viewer slightly and b eine part of the ¿-, ¡:
deepen the sense that time passes very slowly in the cell. t,t reflect the erpi:c-:.
:¡r of episodes th::
:L al. 1989), an.i.,..
-:rplicit plor stru;: *:
-.'-e\\.e
rs must inle: -.:
i ¿:e more extended, (Baggett 1979). For example, a representation of the events depicted in this event
.. t¡ianli screens) are sequence may consist of several propositions, such as (pull: ripcord, Jaws), (fall: rip-
'. rarelv used. In the cord), (flap: arms, ]aws). Relationships between propositions in this aspect of the
rt:usi\-e editing tech- representation can be established through co-reference. So, in the sequence shown in
r-ld memory. Fig. zr.r, continuity can be established because Jaws is present in shots r, 2,3, and 5.
:r blinks, saccades, or However, deep meaning emerges with the construction of a model of the situation
--r:r an ongoing pro- described by the text (van Dijk and Kintsch 1983; Zwaan and Radvansky 1998; Magliano
:rt rst, the director or et al. 1999). This is usually referred to as a situation model(sornetimes 'mental model').
:.:ceptual system or Here we will use the more general term eyent model (Zacks et al. zooT) to emphasize
¿ trr produce smooth, that we mean an event representation that could be derived from text, from a film, or
,i -ice jarring percep- from real life.
Pooriyused, these An event model provides an index of how story events are related along a number
"..
"' ::at afford a richer of dimensions, such as agents and objects, temporality, spatiality, causality, and inten-
::,r:huitv edit that is tionality (Zwaanet al. 1995a,b; Zwaan and Radvansky 1998; Magliano etal.ry99;Zacks
-,r¡¿rbv screen loca- et al. zoogb). For example, comprehending the events depicted in Fig. zr.r requires
i.ns¿tion that some- viewers to generate spatial temporal relationships between Jaws falling without a para-
::i: uses a hand-held chute and the events depicted in the circus tent. These inferences afford a predictive
j .er-eral times drops inference that Jaws will land on the circus tent. These inferences are conceptualized as
: .,-ierr-er slightly and being part of the event model because they go beyond the propositions that are derived
to reflect the explicitly seen events. Finally, narrative plots typically consist of a hierar-
chy of episodes that are casually related to one another (Thorndyke 1977; Trabasso
et al. r989), and viewers must be able to construct representations that reflect the
implicit plot structure. Just as is the case with narrative texts (e.g. Trabasso et al. 1989),
viewers must infer and represent the causal relationships between events within a scene
and across scenes> the latter of which provide a basis for understanding the plot struc-
ture. For example, in a stereotlpical action-adventure such as a James Bond film, the
character has a primary goal to stop the villain, which can only be accomplished by
-rlto discrete events, achieving a series of subordinate goals. If viewers cannot infer that Bond initiallyüsits
,-'i narrative compre- the villain in order to find out information regarding his involvement in the mystery
.iins and listening as- he is to solve, then theywill not be able to comprehend the film.
:-r'.sch 1983; Kintsch Neuroimaging studies indicate that narrative comprehension during reading relies
.r:r clearly differs in on a distributed text-processing network, and also on neural systems that are selec-
::r; rtorkingassump- tively activated when processing texts that allow one to construct a coherent event
:r representations
:1ar model (Ferstl zooT; Ferstl et al. zooS). One important consequence of processing
I'l¿sliano et al. zoor, meaningful coherent text is that brain changes throughout the language network
increase in strength (Yarkoni et al. zoo8). Hor¡r'ever, there also is evidence that weaving
l¡1.r¡-i¡vglves con- a set ofsentences into a coherent discourse selectively activates an area in the prefrontal
F.:aders construct a cortex, specifically on the medial surface near the front of the anterior portion (Ferstl
:r:
particular words et al. zooS; Yarkoni et al. zoo8). This area is usually referred to as dorsomedial pre-
*-J be analogous to a frontal cortex, or dmPFC. One possibility is that dmPFC is selectively involved in the
s ,- : characters' cloth- inference processes that allow one to fill out an event model based on a propositional
::.".:t .Llntent, which is textbase. Data consistent with this hlpothesis come from a study that measured the
','
-:. oi propositions response to changes in agents, objects, space, time, and goals during narrative reading
*:: the erplicit ideas (Speer et al. zoog). The dmPFC was selectively activated when these situational dimen-
:-:ir.:ttation as well sions changed. Interestingly, a portion of this region increased in activity as more
448 JEFFREY M. ZACKS AND JOSEPH P. MAGLIANO
dimensions changed, but did not show a strong increase for any particular change. >¡matosenson'anc. L--
Other regions of the brain responded selectively when particular dimensions of the :heir tongues, harcs.
situation changed; we will return to these in the following section. -(tncepts are repres::
In sum, event models for narratives reflect the events that are explicit and implicit :'ro7). Similar resio:rs
parts of the narrative, who are involved in them, what are their goals and what to do to :reir names, and ¡¡
achieve those goals, and the outcome of those activities within the spatial temporal ,,rtth activiqv in dti--¡:
framework of the story world (Zwaan and Radvansky 1998; Magliano et al. r99g). :ations are interrel¿::
Constructing these representations may depend on the dmPFC as well as other brain nore strongh. assoc:¿
regions. This sort of conceptual information is relatively abstract-it seems on its face :.tivated when thin^:
removed from the perceptual experience of watching or reading and the motor experi- ¡i the existing dat" ;
ence of acting in the world. How do the contents of event models relate to perceptual :hrase, or picture, .:
and motor experiences? We turn next to this question. :ecent studies har e i,,
:uring ongoing st.r:-,.
According to thr. ',
'.r through a book. ,-,-
;eptual and motcrr ¡
-,r'hen features of th .¡
ErvrsoDrED EvENT REPRESENTATToNS e
:egmentation'), the :: I
," iervers do. In par:ic-
Traditionally in artificial intelligence and cognitive psychology, event models were rroposes that reade::
thought to be something like a description in a formal logic or a computer program' ;hanges in the dim¡:r:
specifring parts of the event and features in an abstract language (e.g. Minsky r97z). :nd objects, tempor.
More recently, however, cognitive scientists and cognitive neuroscientists have come j upport this propt-. s
"-
to view event representations as simulations of the situations they represent, which rng, and whether it -. :
preserve some of the perceptual and motor details of the activity in a form that is closer :ime, people appe¿: ,
to perception than to logic (Barsalou et al. zoo3). This view is referred to as the embod- Ztraan et a1. 1998: )i l,
ied cognition or perceptual symbolview. In this section we will briefly review evidence :oo5; Zacks et a1. : :
for the perceptual and motor properties of event models and discuss how these are ¿re observed at tir3s:
important for understanding film. lierving (Zacks er :1. -
There have been a number of demonstrations that perceptual and motor features In short, r,r'hen :-,
are activated automatically during reading. For example, in one study (Zwaan et al. that represent rrh¿: L,
zooz), participants read sentences such as 'The ranger saw an eagle in the sky' or tuai and motor con:¿l
'The ranger saw an eagle in its nest', and then verified whether a line drawing matched rt-el1 asdirectlr' err'¡r,
one of the words in the sentence. After reading the sentence about the eagle in the sky, activity, a findine ir.:
participants were relatively faster to verifr a picture of an eagle if it were depicted with ries of event segmen::
its wings outstretched. However, after reading the sentence about the eagle in the nest,
participants were relatively faster to verifir a picture of an eagle shown sitting with
its wings folded. Such data suggest that during reading participants constructed repre-
sentations that included perceptual information about the spatial configuration of
the scene.
Neurophysiological data also support the construction of perceptual-motor con-
tent during reading. Some action words such as 'lick', 'pick', and'kick' are strongly
associated with movements of particular parts of the body-in this case the tongue, \'Vhat does all of rh:.
hand, and leg, respectively. Reading such words selectively activates those parts of directors and edit,-.¡.
FILM, NARRATIVE, AND COGNITIVE NEUROSCIENCE 449
','rarticular change. somatosensory and motor cortex that are activated when participants actually move
l: drmensions of the their tongues, hands, and legs (Hauk et al. zoo4). Object concepts as well as verb
concepts are represented in terms of their perceptual and motor properties (Martin
ti:phcrt and rmphcrt zooT). Similar regions in the temporal cortex are activated bypictures of objects and by
¡s ¿nd rvhat to do to their names, and (as noted previously) different categories of objects are associated
-:re spatial temporal with actiüty in different regions in the temporal cortex. concepts of objects and
lr3-iano et al. 1999). actions are interrelated, such that the regions activated when thinking about tools are
is ',u'eil as other brain more strongly associated with regions activated during tool use than are the regions
seems on its face activated when thinking about non-manipulable objects (Johnson-Frey zoo4). Most
-it
:ld:he motor experi- of the existing data come from studies in which participants view a simple word,
s :elate to perceptual phrase, or picture, and make an explicit judgement about the stimulus. However,
recent studies have found evidence that perceptual and motor contents are activated
during ongoing story reading (Zacks et al. zoogb) and film viewing (Zacks et al. zoro).
According to this view, whether one experiences an activity in real life, in a fiIm,
or through a book, one creates a set of conceptual representations that include per-
ceptual and motor content. If this is true, and if people segment activity into events
[TIONS when features ofthe activityare changing unpredictably (aswe argued above; see'Event
segmentation'), then readers should break up stories into events at changes just as film
viewers do. In particular, the account of event models given in the previous section
. rl,'ent models were proposes that readers and viewers both segment activity into events when there are
I -omputer program, changes in the dimensions of the situation represented in their event models-agents
:e -.s. \Iinsky r97z). and objects, temporality, spatiality, causality, and intentionality. The data seem to
-,s;lentists have come support this proposal: whether segmentation is studied in story reading or film view-
h:., represent, which ing, and whether it is measured directly or through indirect measures such as reading
r ¿ lorm that is closer time, people appear to segment activity when features in the situation are changing
::r:,1 to asthe embod- (Zwaan et al. 1998; Magliano et al. zoor, zoo5; Rinck and Weber zoo3; Speer and Zacks
--rilr rer.iew evidence zoo5;Zacks et al. zoogb). In neuroimaging studies, selective increases in brain activity
l:.:cuss how these are are observed at these points, both for reading (Speer ef al. zoo7, zoog) and for film
viewing (Zacks et aL zoo6a).
,. :rd motor features" In short, when people understand activity they appear to construct event models
¿ -.iudr' (Zwaan ef al. that represent what is happening at any given moment. Event models include percep-
r ¿asle in the sky' or tual and motor content, which may be constructed from language or by inference, as
-re drarving matched well as directly experienced. Event models are updated at changes in features in the
:: :he eagle in the sky, activity, a finding that brings together theories of narrative comprehension with theo-
'it ru-ere depicted with ries of event segmentation.
.: the eagle in the nest,
,- shortn sitting with
r:s constructed repre-
::lai configuration of
::ceptual-motor con-
CoNcrusroNS
ri kick' are strongly
r:ris case the tongue, What does all of this say for understanding the film viewing experience? First, film
:ii ates those parts of directors and editors in the Holllwood style exploit properties of perception and
IEFFREY M. ZACKS AND JOSEPH P. MAGLIANO
attention to create unobtrusive cuts. They can do this, we propose, in two distinct ways. Hochberg and Br,-
One way is to hide a cut using visual masking or by diverting attention. Another way is auts, temporai re..:;
to make a cut coincide with a perceptual event boundary. rom perfect au,lt.'.
A second conclusion is that perceptual event boundaries mark the major units of :reuroscience of ii:r
narrative comprehension, in film as well as in other media. The nervous system appears
to devote a lot ofeffort to assembling representations ofcoherent events. Event repre-
sentations are individuated based on features on multiple dimensions. Preüous research
has focused on agents, objects, space, time, causes, and goals. We believe-though we
can present no evidence for it at this point-that these dimensions are instances of a
broader principle: event representations are individuated based on whatever features
are important to the viewer's task. Agents, objects, and so forth are often found to be
associated with comprehension because these dimensions are often important for
?reparation of t1:::
¡rom the Nationet --
understanding. When structural features of film align with the situational features
that define events, understanding is easier and memory is better. We believe that think-
ing about the alignment between structural features in film and situational features in
events may allow new creative insights for film artists. By aligning structural eiements
RrpEnE¡¡cts
with situational features one can create a narrative that is comprehensible and flows
smoothly. By misaligning the two, one may deliberately disorient the viewer. In this -lrdersonJ(rgg6.-:,
vein, it is interesting to think about commercials in broadcast television such as situa- Southern Illinc.i. -':
tion comedies and serial dramas. Our hunch is that commercials often are placed just Beggett P (.t9¡g ), Str'.;.
after situational changes that would tlpically be perceived as event boundaries. This on recall. Jourr.,;. . t
may encourage viewers to remain in their seats through commercial breaks, but also Baidrvin DA, Baird .-.
Developntent,72. - ,
may reduce comprehension.
E¿rsalou L\\r, Krle S-
Although the psychological and neurophysiological findings we have reviewed
knorvledge in n¡:.
here can greatly inform our understanding of the experience of film, they are also Beattl'I and Lucer¡ -', ,
greatly incomplete. For one thing, we have said nothing about viewers' affective and GG Bernts..:- .
response to film-though some would argue this is what film (and all forms of narra- Press, Nerv York.
tion for that matter) is all about (Brewer and Lichtenstein r98z; Oatley t995; Tant995;' 3oitz N{ (t992 ) . Te r:a .
Z1llmann 1995; Tan rgq6). We have done this reluctantly to restrain the scope of the o.f Experintertt't: -:: :
chapter-there is much that could be said. Movies have been widely used to study the 3oltz MG (1995 . E..= ---.
Psychophysics.5; '
emotions in perception, cognition, and memory, because they are effective induces of
Sordrvell D (1985 . -1,., -'
emotional responses (e.g. Tan 1996). One important bridge between the perceptual-
Bordrvell D and Th,-'-::
cognitive aspects of frlm understanding we have discussed here and emotion is the Branigan E (t99: . -\.,-"
response of surprise, which has both cognitive and affective components (Brewer and Brener WF al-rd Ll¡:.-
Lichtenstein r98z; Brewer and Ohtsuka 1988). For a thoughtful exploration of the stories. Journai :'.'
cognitive neuroscience of emotion as it applies to film, we recommend a recent volume Brerver WF and Oh:-. '-...
by Greg Smith (zoo¡). short stories. In C l, i.
The results reviewed here are also incomplete because our knowledge is incomplete. pp. 133-58. Hein,. _
Carroll JM (r98o . , .-
As those data reveal, the cognitive neuroscience of perception and cognition has much
.
New York.
to tell us about the film viewing experience because our experience of film shares Carroli fM and Bei:: -
much with our experience of real life. However, some aspects of film are unique, and Chawla D, et al. ,r9::
about these aspects our psychological and neurophysiological knowledge is much less Neurolmage,7. !r,:
well fleshed out. Here is a partial list of technical features of film that are simplybegging Copeland DE, Ilaeir.r
for greater psychological and neurophysiological understanding: rate of cutting memory and aur:-.
FILM, NARRATIVE, AND COGNITIVE NEUROSCIENCE 45r
. -:r ¡r'o distinctways. (Hochberg and Brooks zoo6), viewing angle (Cuttingry87), camera motion, jump
:,rrrn. A¡other way is cuts, temporal reordering, relations between music and narrative action, deviations
from perfect audiovisual synchrony, dubbing, and subtitles. clearly, the cognitive
:., the major units of neuroscience of film is just in its infancy.
:-, ,-rus sYstem aPPearS
: ;, 3:tts. Et ent repre-
:.. Previous research
, ::,ier e-though we
f) lre instances of a AcTNowLEDGEMENT
r',,.'natever features
,:. ¡lten found to be
n important for Preparation of this chapter was made possible by Grant number #1R01AG031150
'- -e
: .-.;ational features from the National Institute on Aging.
', - '¡eiieve that think-
,: -¿rional features in
: .:::ctural elements RrpnnnNcEs
:.hensible and flows
-. the vie\ver. In this Anderson I Ogs6). The Reality of Illusion: An Ecological Approach to Cognitive Film Theory.
;',-1:-L)n such as situa- Southern IIIinois University Press, Carbondale, IL.
' - ri.r are placed just Baggett P (tglg). Structurally equivalent stories in movie and text and the effect of the medium
.:: boundaries. This on recall. Journal of Verbal Learning 6 Verbal Behavior, rB, 3¡3*56.
:;-:, breaks, but also Baldwin DA, Baird lA, Saylor MM, and Clark MA (zoor). Infants parse dynamic action. Child
Development, 72, 7 o8-r7.
Barsalou LW, Kyie Simmons W, Barbey AK, and Wilson CD (zoo3). Grounding conceptual
ir ',". e har-e reviewed
knowiedge in modality-specific systems. Trends in Cognitive Science, T, B4-9r.
- - :in, they are also Beatty I and Lucero-Wagoner B (zooo). The pupillary system. In JT Cacioppo, LG Tassinary,
-:'''rerr'ers' affective and GG Berntson, eds. Handbook of Psychophysiology, pp. 14z-62. Cambridge University
r: ¿-l lorms of narra- Press, New York.
'::-er-199;; Tan 1995; Boltz M (1992). Temporal accent structure and the remembering of filmed narratives. lournal
.:¿l the scope of the of Experimental Psychology: Human Perception (z Performance, 18, 9o-1o5.
::,i' used to study the" Boltz MG (1995). Effects of event structure on retrospective duration judgments. Perception ú
-¡ ¿ile ctive induces of P sy chophysics, 52, roBo*96.
Bordwell D (rq8S). Narration in the Fiction Film.llniversíty of Wisconsin Press, Madison, WI.
i"..en the perceptual-
Bordwell D and Thompson K (zoo3). Film Art: An Introduction. McGraw-Hill, New York.
: ::rd emotion is the Branigan E (rqgz). Narratiye Comprehension and Film. Routledge, London.
;l-r:lents (Brewer and Brewer WF and Lichtenstein EH (rq8z). Stories are to entertain: A structural-affect theory of
" :,tploration of the stories. Journal of Pragmatics, 6, 437-86.
n.n.i a ¡ecentvolume Brewer WF and Ohtsuka K (r9BB). Story structure and reader affect in American and Hungarian
short stories. In C Martindale, ed. Psychological Approaches to the Study of Literary Narratit,es,
,"".-¡dge is incomplete. pp. 133-58. Helmut Buske Verlag, Hamburg.
Carroll fM (r98o). Toward a Structural Psychology of Cinema. Mouton Publishers, The Hague,
I ...snition has much
New York.
:-:r.e of film shares Carroll JM and Bever TG (1976). Segmentation in cinema perception. Science, rgr, 1053-5.
--lm are unique, and Chawla D, et a1. (r99S). Speed-dependent motion-sensitive responses in V5: An fMRI study.
: --,','.'ledge is much less Neurolmage, Z, 86-g6.
r.: ::e simplybegging Copeland DE, Magliano JP, and Radvansky GA (zoo6). Situation models in comprehension,
ir:E: rate of cutting memory and augmented cognition. In C Forsl.the, ML Bernard and TE Goldsmith, eds.
4'2 JEFFREY M. ZACKS AND JOSEPH P. MAGLIANO
Cognitfue Systems: Human Cognitive Models in Systems Design, pp. 37-66. Erlbaum, =1.¡no JP, Rad-,: ,.
Mahwah, Nl. ..,rr: Situation r., :,
Cutting IE (1987). Rigidity in cinema seen from the front row, side aisle. lournal of Experimental , Goldsmith. ec: -::
Psychology: Human Perception ú Performance, rr3,323-34.
.
-,-r!)_\!-\.1,f,._._
-
Cutting IE (zoo5). Perceiving scenes in film and in the rvorld. In ID Anderson and BF Anderson, =;.,,:no JP and Z.::...
eds. Moving Image Theory: Ecologícal Considerations, pp. 9-27. Southern Illinois University. ,'. . ri segtt1ei1.;,1.:: .'..
stable task control in humans. Proceedings of the Natíonal Academy of Science of the United - -.s ,\nqeles, C r-
States, to4, rro73-8. :..:LlnD 19-. t---
Fan L McCandliss BD, Sommer T, Raz A, and Posner MI (zooz). Testing the efficiency and . .:. :.. t., n l ii.r. ¡,; L¡.
independence of attentional networks. Journal of Cognitfue Neuroscience, 4, 340-7. : ! .s!r:r D anc l:.-: -
Ferstl EC (zoo7). The functional neuroanatomy of text comprehension: What's the story so
' - i.:':r ¿r:ti
t'i:,; : :
far? In F Schmalhofer and CA Perfetti, eds. Higher Level Language Processes in the Brain: 1 ,,
-¡¡rl D. Ens.':,,
Inference and Comprehension Processes, pp. 53-102. Erlbaum, Mahwah, NJ. ¡'--_r.rr;¡i l¡-, ,;,1,; _i_
Ferstl EC, Neumann l, Bogler C, and Yves von Cramon D (zoos). The extended language .--,.',ñ r99;.i:-
netwo¡k: a meta-analysis of neuroimaging studies on text comprehension. Human Brain .:. iatirrnal n,:::"i
Mapping, z9,5Br-93. ...:;r iÑ. Rer,-.--.
Gilbert L (Director) Q979). Moonraker lMofion picturel. Pinewood Studios, United Kingdom. .',,"1¿-c-.398.,r:.
Hard BM, Tversky B, and Lang D (zoo6). Making sense of abstract events: Building event ,.-..-:¡. R\. L) R¿:.
schemas. Memory 6 Cognition, 34, 722t-35. :.:¡er-,-e Ch¿ng.,.
Hauk O, lohnsrude I, and Pulvermuller F (zoo4). Somatotopic representation of action words ..-.:..llrnd\'r-¡¡¿-
in human motor and premotor cortex. Neurofi,4t,3ot-7.
_, : - .,/-¡._ -
Hochberg I and Brooks V (zoo6). Film cutting and visual momentum. In MA Peterson, - ,..:r S. G¿ts,--i..
B Gillam, and HA Sedgwick, eds. In the Mínd's Eye pp. zo6-z8. Oxford University Press, . ',. .rrg,trtz"tt,_:_
New York. -,:: tr-tl. Le; I. :-t:
|ohnson-Frey SH (zoo4). The neural bases of complex tool use in humans. Trends in Cognitive .' ,:; hlrir dc,:, -:
Sciences, S, '.-. ;,..,_;;; ¡,. ;:-
7r-8. _i.- _-1.
Representations during Reading, pp. 2i,g-45. Lawrence Erlbaum Associates, Mahwah, N]. . :S r99; , Fi;::- :.-
-- - li :¡.ln :'
Magliano JP, Miller l, andZwaan RA (uoor). Indexing space and time in film understanding. -
fp ,r/-66. Erlbaum, Magliano ]P, Radvansky GA, and Copeland DE (zoo7). Beyond language comprehen-
sion: Situation models as a form of autobiographicai memory. In F Schmalfolfer and
-
: :,. r: il o,f Exp erimental T Goldsmith, eds. Higher level language processes in the brain: Inference and comprehension
pro cesses, pp. 37-66, Erlbam, Malwah Nf .
;:sr:,n and BF Anderson, Magliano lP and Zacks JM (under review). The impact of continuity editing in narratite film on
--=¡:: Illinois University event segmentation.
Martin A (zoo7). The representation of object concepts in the brain. Annual Review of
: ::rr;essing of editing Psychology, 58, 25-45.
',-: :: )vlittd: Information Minsky M Q.972). Aframework for representing knowledge. In PH Wilson, ed. The Psychology
---r:1.¡erdam. of Computer Vision, pp. ztr-77. McGraw-Hill, New York.
:' ::-.ls ibr adaptive and Murch W (zoor). In the Blink of an Eye: A Perspective on FiIm Editing. Silman-]ames Press,
j,i;i¡irce Los Angeles, CA.
-- of the United
Newtson D Ggz). Attribution and the unit of perception of ongoing behavior. Journal of
.:r::E rhe efficiency and Personality and Social Psychology,28, 2B-38.
:,.::.. 11,34O-7. NewtsonDandEngquistG(1976).Theperceptualorganizationofongoingbehavior. Journal
-r\ of Experimental Social Psychology, rz, 436-5o.
, :,:
hat's the story so
; ,r,-:_'¿,<-ss,s in the Brain: Newtson D, Engquist G, and Bois J (lgZZ) The objective basis of behavior units. Journal of
¡-. \[. P er s o n ality an d S o ci al P sy cholo gy, 35, B 47
-62.
1..: ertended ianguage Oatley K (1995). A taxonomy of the emotions of literary response and a theory of identification
--.r-sron. Human Brain in fictional narrative. Poetics, 23, 53-74.
O'Regan lK, Rensink RA, and Clark JJ (1999). Change-blindness as a result of 'mudsplashes.'.
- :r.''. United Kingdom. Nature,398,34.
Rensink RA. O'Regan JK, and Clark JJ OggZ).To see or not to see: The need for attention to
=r'.:::s: Building event
perceive changes in scenes. Psychological Science, 8, 368-7.
::ltion of action words Rinck M and Weber U (zoo3). Who when where: an experimental test of the event-indexing
model. Memory and Cognition, jr, tz84-gz.
.-::. In I'lA Peterson, Schwan S, Garsoffky B, and Hesse FW (zooo). Do film cuts facilitate the perceptual and cogni-
:':, rd University Press, tive organization of activify sequences? Memory (z Cognition, 28, 214-23.
Sharp RM, Lee L and Donaldson DI (zoo7). Electrophysiological correlates of event segmenta-
.:.'. Tr¿nds in Cognitive tion: how does the human mind process ongoing activity? Annual Meeting of the Cognitit,e
N eur o science So ciety, 235.
:rt..n: a construction- Smith GM (zoo3). Film Structure and the Emotion System. Cambridge University Press,
Cambridge.
-:.ion and production. Smith T| and Henderson JM (zoo8). Edit blindness: The relationship between attention and
global change in dynamic scenes. Journal of Eye Movement Research, z, r-t7
. i.r !r 11' of evenfs. Tr ends SpeerNK andZackslM (zoo5). Temporaichanges as eventboundaries: Processingandmemory
consequences of narrative time shifts. Journal of Memory and Language, 53, 125_40.
.:"re_. Lopert Pictures Speer NK, Swallow KM, and Zacks lM (zoo3). Activation of human motion processing areas
during event perception. Cognitive, Affective (z Behavioral Neuroscience,3,335-45.
'i-,1: Combining shots Speer NK, Reynolds JR, and Zacks fM (zoo7). Human brain activity time-locked to narrative
"
:. :
:', :iiology, z, 357-Bo. event boundaries. Psychologícal Science, L8, 449-55.
:' -: moYies. Discourse Speer NK, Reynolds JR, Swallow KM, and Zacks JM (zoo9). Reading stories activates neural
representationsofperceptualandmotorexperiences. PsychologicalScience,zo,g9g_-99.
,,:l ;ontinuity in narra- Swailow KM and Zacks JM (zoo4). Hierarchical grouping of events revealed by eye move-
., ,:t::truction of Mental menfs. Abstracts of the Psychonomic Society, rr, 8r.
L .i.r:-s. Ilahwah, NI. Tan ES (1995). Film-induced affect as a witness emotion. Poetics,23,7-32.
. -: ilm understanding. Tan ES Q996). Emotion and the Structure of Narrative Film. Lawrence Erlbaum Associates
Hillsdale, NJ.
""1:::itr¡ring the goals of Thorndyke PW Q977). Cognitive structures in comprehension and memory of narrative dis-
course. Cognitiue Psychology, 9, 77-7to.
454 JEFFREY M. ZACKS AND IOSEPH P. MAGLIANO
Tootell RB, Reppas fB, Kwong KK, et al. (1995). Functional analysis of human MT and related
visual cortical areas using magnetic resonance imaging. Journal of Neuroscience, 15, 32r5-3o.
Trabasso T, Van den Broek P, and Suh SY (rqsq). Logical necessity and transitivity ofcausal
relations in stories. Discourse Processes, lr2, t-25.
van Dijk TA and Kintsch W (rgs¡). Strategies of discourse comprehension Academic Press,
New York.
von Trier L (Writeri Dir ector) . Dancer in the Dark. [Motion picture] . Fine Line.
Yarkoni T, Speer NK, and Zacks lM (zoo8). Neural substrates of nar¡ative comprehension and
memory. Neuroimage, 4r, t4o}-25.
Zacks JM (zoo4). Using movement and intentions to understand simple events . Cognitive MI]
Science,28, 979*roo8.
Zacks JM and Swallow KM (zoo7). Event segmentalton. Current Directions in Psychological
Science, t6,8o-+(f).
Zacks |M, Tversky B, and Iyer G (zoora). Perceiving, remembering, and communicating struc-
ture in events . Journal of Experimental Psychology: General, ryo, z9-58.
Zacks JM, Braver TS, sheridan MA, et al. (zoorb). Human brain activity time-locked to per-
ceptual event bounda ries. N ature N eur o s cíen ce, 4, 6 5t-5.
Zacks fM, Speer NK, vettel lM, and Jacoby LL (zoo6). visual movement and the neural cor-
relates ofevent perception. Brain Research, ro76, 15o-62.
Zacks fM, Speer NK, Swallow KM, Braver TS, and Reynolds fR (zoo7). Event perception:
A mind/brain perspective. Psychological Bulletin, r3.3, 273-93.
Zacks ]M, Kumar S, and Abrams RA (zoo9a). Using movement and intentions to understand
human activity. Cognition, u2, 2ot-16,
Zacks lM, Speer NK, and Reynolds JR (zoogb) Situation changes predict the perception of TH
event boundaries, reading time, and perceived predictability in narrative comprehension.
Journal of Experimental Psychology: General, r3B, 3o7-27. -,.:l:f
Zacks JM, speer NK, swallow KM, and Maley cl (zoro). The brain's cutting-room floor: seg-
.:.. ire past lt
mentation of narrative cinema. Frontiers in Human Neuroscience, 4, r-15. :. ' sroltrgl anú --- '-
Zillmann D (rSS;). Mechanisms of emotional involvement with drama. poerics,23, il-51. , ;,,¡r-e that th¡ ::: ,.
Zwaan RA, Langston MC, and Graesser AC (rggfa). The construction of situation models in :::l3lll \l\S l -
narrative comprehension: an event-indexing model . Psychological science,6, z9z-7. : thls disc.¡r'e:-,' - .
Zwaan RA, Magliano lP, and G¡aesser AC (r995b). Dimensions of situation model construc- :- ; 5asic tac¡s, ..,: - - :
tion in narrative comprehen sion. Journal of Experimental psychology: Learning, Memory, d* :-r ,,nke\- b¡aii . ¡-.,,
Cognition, 2t,386-97.
;,.-aUteS gOa-- rt,.,. :
Zwaan RA and Radvansky GA (rqg8). Situation models in language comprehension and
memory. P sy cholo gi cal B ulletin, n3, 762-85. -h-e: indiridu;... ::-_ .
Zwaan RA, et al. (1998). Constructing multidimensional situation models during reading. .: .-. 1996 r. \\'e c. :: .
: :::rrocallt cLrlra - :a
. . *:oph\-siLrlos:::- :
::,ie bl other lni._.. :
,,, :an aati\-eh' er:.,-L::
:.:i:tiLrns to understand
' This essay originates from an interview of Prof. Gallese by the editors.
456 VITToRIo GALLESE
to act, and at the input side, to directly understand the actions of others. What is ¡ l.-3Sf 1O Of , : O r-r :
remarkable is that this matching system has also been demonstrated in humans (see .:dior-isual nlr¡r: :,-
Gallese et al. zoo4; Rizzolatti and Craighero zoo4). Furthermore, new empirical srantiates proF\ -:-_.
:i'-
evidence suggests that the same neural structures that are involved in processing felt :::istrc of human r' -
sensations and emotions are also active when the same sensations and emotions are )ll:te ConceptU¿-:-.
to be detected in others (Gallese zoo6) .It appears therefore that a whole range of dif- , rrsuming it. sC,',-:-.:
ferent 'mirror matching mechanisms' may be present in the human brain. ¡l
br-audior isu..-- -..
When the discovery of mirror neurons occurred, theories regarding embodied '"
.r¡ 'to break'is -1,.:
cognition and multisensory perception were being explored. Although the discovery , : meone breaki::: . ;
seemed to fit in this research context, one should keep in mind that we did not look - re predicate . si::t,_.:
for mirror neurons. In other words, the discovery was not guided by a preconceived ::pending on th- ,- .
thesis about social cognition. However, we were ready to see mirror neurons, because -, ::t.ring the acti,,-:r r_
we were looking for visual properties within the motor system, something that back
'his is panic,*-.: ,
then was extremely unorthodox. The mainstream view in cognitive science was, and : ::llrrical appir-r i : :.
to a certain extent even today is, that action, perception, and cognition are to be seen ¡-a', eshorr'n tha: :: .
as separate domains. The discovery of the MNS challenges this view as it shows that ::::aning actrr ¿te: ::.
such domains are intimately intertwined. : r:rte nt. Silent r¡::.,-
The sharp distinction, classically drawn between the first- and third-person experi- :;i1tenCeS e\pra)¡-:.
ence of acting and experiencing emotions and sensations, appears to be much more , -:h produce a.i_', ::,
blurred at the level of the neural mechanisms mapping it. The gap between the two .::;ctOr uSed i:: ::-. .:
perspectives is bridged by the way intentional relations are functionally mapped at the ::::ts. These acl ,:,-,:
neural-body level. The mirror matching systems in our brains map the different inten- :,:aautloniobSei-, :. ..
tional relations in a fashion that is neutral about the specific quality or identity of the :" l\S : ..,
is tnr-olr'ed
agentive/subjective parameter. By means of a shared functional state realized in two :-l¡ acoustrca-l-. _: .
different bodies that nevertheless obey to the same functional rules, the 'objectual _ ,r-. :oo8 t.
other' becomes 'another self . Our seemingly effortless capacity to conceive of the Such evidence s_..
acting bodies inhabiting our social world as goal-oriented selvesllke us depends on the :-"' neural erpltrt:.,- :
constitution of a shared meaningful we-centric space. I propose that this shared mani- :., L-r-1'gd tbr sensc,:-, -::--
fold space can be characterized at the functional level as embodied simulation, a basic
-tctional archi,:¡:_:
functional mechanism by means of which our brain/body system models its inter- :. ,;ts as rlell see ,.,_,
actions with the world. Embodied simulation constitutes a crucial functional mecha- .-;hitecture th¿t c:.. _.
nism in social cognition, and it can be neurobiologically characterized. The mirror ¡ ,rperation, the ;::: *
neuron matching systems so far discovered in the human brain represent subpersonal ,rd rnaginatiLrn. ,,. .-
instantiations of embodied simulation. - r,ical areas. \\-h¡: -
While one could think that the input to the MNS needs to be visual, we actually .::ivated, leadinE :.,:
showed that a particular class of F5 mirror neurons, the'audiovisual mirror neurons', .;:-on is obsenec , : _
discharge not only when the monkey executes or observes a particular tlpe of noisy ':-ir-ork is actii at.a :.
action (e.g. breaking a peanut), but also when it just listens to the sound produced by ---r:€nsin-), but ac:t,--:. ,,
the same action (Kohler et al. zooz). These 'audiovisual mirror neurons' not only in the second ;:.1 -:.
respond to the sound of actions, but also discriminate between the sounds of different :i,:cution/percep:r,_- r
actions. The actions, whose sounds ma-ximally trigger the neurons' discharge when :::tson'-notorpaf ,¡ " -
heard, are those also producing the strongest response when observed or executed. :-¡r's a pivotal role, ',., :
Among their several possible functions, it has been hlpothesized that mirror ,- rnal architectur. .,: :
neurons may be involved in the processing of meaning (Gallese and Lakoff zoo5; : erarchical structL:: ,
MTRROR NEURONS AND ART 457
: r¡; others. What is Gallese zoo7, zoo8). The multimodally driven simulation of action goals instantiated by
at¿d in humans (see 'audiovisual mirror neurons', situated in the ventral premotor cortex of the monkey,
rt're. fl€w empirical instantiates properties that are strikingly similar to the symbolic properties so charac-
':d 'rn processing felt teristic of human thought. The similarity with conceptual content is quite appealing: the
is ¿nd emotions are same conceptual content ('the goal of action A') results from a multiplicity of states
¡ ,.,'hole range of dif- subsuming it, (sounds, observed and executed actions). The action simulationembod-
¿r b¡ain. ied by audiovisual mirror neurons is indeed reminiscent of the use of predicates. The
relarding embodied verb 'to break' is used to convey a meaning that can be used in different contexts: 'seeing
.-r-¡ugh the discovery someone breaking a peanut', 'Hearing someone breaking a peanut', 'Breaking a peanut'.
*.:t rr e did not look The predicate, similarlyto the responses in audiovisual mirror neurons, does not change
ri b-,-a preconceived depending on the context to which it applies, nor depending on the subject/agent per-
iL-,r 1۟f ofrs, because forming the action. All that changes is the context the predicate refers to.
i,- rething that back This is particularly interesting also because these results paved the road to a new
::'" e science was, and empirical approach to the study of human language. several brain imaging studies
:::i:on are to be seen have shown that processing action-related linguistic material in order to retrieve its
i.r.h- as it shows that meaning activates regions of the motor system congruent with the processed semantic
content. Silent reading of words referring to face, arm, or leg actions, or listening to
:lrrd-person experi- sentences expressing actions performed with the mouth, the hand, and the foot,
.:s ro be much more both produce activation of different sectors of the premotor cortex, depending on the
::p ben\'een the two effector used in the action-related linguistic expression read or listened to by partici-
-- r:lilmapped at the pants. These activated premotor sectors coarsely correspond to those active during the
": rhe different inten- execution/observation of hand, mouth, and foot actions. Thus, it appears that the
*i:, or identity of the MNS is involved not only in understanding visually presented actions, but also in map-
stare realized in two ping acoustically or visually presented action-related linguistic expressions (Gallese
:iles. the'objectual zoo7, zooS).
r. tLr conceive of the Such evidence suggests that key aspects of human social cognition are produced
,,e us depends on the by neural exploitation, that is, by the exaptation of neural mechanisms originally
rrt this shared mani- evolved for sensory-motor integration, later also employed to contribute to the neuro-
;"i. :iinulation, a basic functional architecture of thought and language, while retaining their original func-
¿n models its inter- tions as well (see Gallese zoo8). Within the premotor system there is a'structuring'
:.1 É¡nctional mecha- architecture that can function according to two modes of operation. In the first mode
,:terized. The mirror of operation, the circuit structures action execution and action perception, imitation,
.:resent subpersonal and imagination, with neural connections to motor effectors andlor other sensory
cortical areas. When the action is executed or imitated, the corticospinal pathway is
:e r-isual, we actually activated, leading to the excitation of muscles and the ensuing movements. When the
s;¿l mirror neurons', action is observed or imagined, its actual execution is inhibited. The cortical motor
:.rcular qpe of noisy network is activated (though not in all of its components and likely not with the same
. sound produced by intensity), but action is not produced, it is only simulated.
-: neurons' not only In the second mode of operation, the same system is decoupled from its action
r¡ sounds ofdifferent execution/perception functions and can offer its structuring output to non-
:c,rs' discharge when sensory-motor parts of the brain, among which the dorsal prefrontal cortex most likely
;:,, ed or executed. plays a pivotal role. When engaged in the second mode of operation, the neurofunc-
:ncsized that mirror tional architecture of the premotor system might contribute to the mastering of the
¡s; and Lakoff zoo5; hierarchical structure of language and thought.
458 vTTTORTO GALLESE
ulative. While the evidence we deal with allows for modulation by contextual factors ,,tith respect to thc.-
of a wide variety (e.g. temporal, social, cultural, and even personal factors), David )ense, One Can Sar,' :::
Freedberg and I are concerned with the basic mechanisms brought to light by recent :n open question .
research on mirror neurons, empathy and embodiment. While most of the research in a\pertise and,'or :-:.
these areas has concentrated on responses to living actors, we proposed its relevance ::rodulating these s::::
not just for realistic representation but also for the understanding of the effectiveness In the case ofr::::,
of images and works of art. soal oriented, ratr:::
Given the variefy of forms and techniques employed in contemporary art (video t\lRI paper bv Ga¡; -
and performance art, digital, painting, and photography), it is interesting to address :ooZ). The result' s:,
the difference between representation and object or, more specifically, between watch- -.rrth human ani r,-.:
ing a video of an action, watching the action being performed for real, and watching a .ir-o agents. This se-::-_
static image portraying the same action. The very few experiments that specifically ,-.i a tvider tanee r,: :
addressed the issue of the relationship between the degree of activation of the MNS lieht be more r::r:,
and the real or virtual nature ofthe agent being observed showed that real action per- :s performed.
formed in front of the observer elicits stronger responses with respect to videos por- The relationsh-: --.
traying those same actions. That said, it might be argued that since the activation ofthe -\s David Freedbe r¡ .
MNS is typically induced by the observation of ongoing actions, the relevance of -,''as originallr'
in::,, :_
such mechanism for the aesthetic experience while contemplating static art works rse in psvcholo¡ . :
could be negligible. However, recent research carried out on the MNS in humans with :esponses genera::: :
a variety of techniques has shown that even the observation of static images of actions particular forms .: *
lead to action simulation in the brain of the observer, through the activation of the nityto the desisn :_:
same brain regions normally activated by execution of the observed actions. The obser- iimbs and to our'¡¡ c .
vation of pictures of a hand reaching to grasp an object, or firmly holding it, activates speculated on rh. .,., r-
the motor representation of grasping in the observer's brain. On the basis of these beholder's bodil-, :.,
results it is highly plausible to hlpothesize that a similar motor simulation process \\-arburg, adaptii: -,':
can be induced by the observation of still images of actions carried out by other effec- P athosformel, rlh¡:.1:'
tors. In any still image, the 'decisive moment' of the action portrayed, to use an expres- , e.g. the hair or tlc,-,,'--
sion of Cartier-Bresson, is captured by an embodied simulation of the contemplator. tions of the figure : : _-
There are situations that lead to greater activation than others, which support the about the relation::--:
view that MNS response can be increased through learning. Recent neurophysiological enjoyment on the t-- i: ;
studies carried out on macaque monkeys have shown that a particular class of ventral ical involvement in . :
premotor mirror neurons starts to respond to the observation of unfamiliar actions sense of imitatir-rs ,.-. :
MIRROR NEURONS AND ART 459
T after extensive visual exposure to them, or after motor training. Both results seem to
suggest that when an action performed by others becomes familiar, independently
from the perceptual or motor source of its familiarization, it is nevertheless always
:alding of responses mapped on to a motor representation belonging to the observing individual. Even
:útrred ivith the art more strikingly, several brain-imaging studies carried out on humans have shown
in our responses
::'l ]l that the intensity of the MNS activation during action observation depends on the
:-ndamental element similarity between the observed actions and the participants' action repertoire. In par-
:,- embodied mecha- ticular, one functional magnetic resonance imaging (ÍMRI) study by Calvo-Merino
a- t!eling sensations; and co-workers focused on the distinction between the relative contribution of üsual
;ears theorists of art and motor experience in processing an observed action (Calvo-Merino et al. zoo5,
ii i\ith rvorks of art, zoo6; see Chapter zTby Calvo-Merino and Haggard, this volume). The results revealed
i<d r"'r entirelyspec- greater activation of the MNS when the observed actions were frequently performed
'., .0ntextual factors with respect to those that were only perceptually familiar but never practised. In this
-rlel lactors), David sense, one can say that the MNS actually learns through experience. It is, nonetheless,
:: to light by recent an open question to be empirically investigated, to which extent the professional
¡,st ¡fthe research in expertise and/or personal and cultural background ofthe beholder do play a role in
,-.rosed its relevance modulating these simulations when contemplating art works.
. ¡: the effectiveness In the case of mirror neurons, what seems to matter most is that the action must be
goal oriented, rather than performed by a human, or at least biological, actor. A recent
::tliroran'art (video fA4RI paper by Gazzola and co-workers specifically addressed this issue (Gazzola et aI.
riÉresting to address zooT). The results showed that the mirror system was activated strongly by the sight of
.'',,'. benveenwatch- both human and robotic hand actions, with no significant differences between these
:eel. and watching a two agents. This seems to suggest that the MNS could contribute to the understanding
:n:i that specifically of a wider range of actions than previously assumed, and that the goal of an action
i-i.riion of the MNS might be more important for mirror activations than the way in which the action
:l¿t real action per- is performed.
rs:ect to Videos por- The relationship between art appreciation and empathy is certainly not a new idea.
:lhe activation of the As David Freedberg and I recently pointed out, the notion of empathy (Einfiihlung)
ri. the relevance of was originally introduced in aesthetics by Robert Vischer in t873, thus well before its
:LnE static art works use in psychology. By Einfühlung literally 'feeling-in', Vischer meant the physical
'Í\S in humanswith responses generated by the observation of forms within paintings. He described how
i:i; images of actions particular forms aroused particular responsive feelings, depending on their confor-
:r- actiYation of the mity to the design and function of the muscles of the body, from those of the eyes to our
C ¡ctions. The obser- limbs and to our bodily posture as a whole. Developing Vischer's ideas, Wólfflin (1886)
, it'rlding it, activates speculated on the ways in which observation of specific architectural forms engage the
lr thre basis of these beholder's bodily responses. Iust before the end of the nineteenth century, Aby
r .imulation process Warburg, adapting Vischer's notions of Einfühlung developed his own theory of the
:: out bv other effec- Pathosformel, whereby the outward forms of movement in a work of Renaissance art
t'¿d. to use an expres- (e.g. the hair or flowing draperies of a painting bv Botticelli) revealed the inner emo-
rf t]:e contemplator. tions of the figure concerned. Very shortly afterwards, Theodor Lipps wrote at length
.. ,,rhich support the about the relationship between space and geometry on the one hand, and aesthetic
r: :europhysiological enjoyrnent on the other (1897, r9o3). All these scholars believed that the feeling ofphys-
:;'l¿r class of ventral ical involvement in a piece of painting, sculpture, or architecture, not only provoked a
,l ,-nlamiliar actions sense of imitating the motion or action seen or implied in the work, but also enhanced
460 VITTORIO GALLESE
our emotional responses to such a work. The relationship between embodiment and -:-fetched. Creali.,--
aesthetic experience was further highlighted byphenomenologists like Merleau-Ponty. .:iaid can hardl., :.
Merleau-Ponty not only suggested the possibilities of felt bodily imitation of what is :teurons, mirror ::e *:
seen in a painting, but also of the implied actions of the producer of the work itself, as On the other h¿:r
in the case of the paintings of Cézanne. :educed to the m¿:¿ r
Thus, what is new in our approach? Perhaps the revitalization of this tradition of It fe ::it
rleau-Pontl- ,'
thought in light of what the empirical research in neuroscience has recently taught us -ommenting on :it.
about the enormous importance of the body in a variety of aspects of the human Ponf\ argues th¿::!_
condition, aesthetic experience included. David Freedberg and I are proposing a theory i;ard this line of t:l,:,:
of empathic responses to works of art that is not purely introspective, intuitive, or :.. being able to re::r
metaphysical, but that has a precise and definable material basis in the brain. Our shouldn't impi,,'::::
theory is articulated in two complementary aspects: r) the relationship between -{ctuall1', I think ::.:
embodied simulation-driven empathic feelings in the observer and the content of beautifullv empha.. ¡,
artworks, in terms of the actions, intentions, objects, emotions, and sensations por- :pitomizes horr t:r¡ .
trayed in a given painting or sculpture. This aspect can be viewed as the 'what' of aes- torces buried insi¡. :
thetic experience; z) the relationship between embodied simulation-driven empathic Ilany schola¡s ::.
feelings in the observer and the quality of the artwork in terms of the visible traces of :opular level in ¿r -_"
the artist's creative gestures, like brushwork, chisel marks, and signs of the movement :ave used it in c¡;:.
of the hand more generally. We can refer to this component as the 'how' of aesthetic scientific resea¡ch ::-
experience. Of course, both aspects require now an empirical validation by means of :heir empirical rtr,r.:
specifically designed experiments, which we will be busy designing and carrying out in dence suggests to::::
the coming years. eries, is totailr- be-,-,-,:;
Regarding the question of whether the MNS can be activated by abstract art, accord- ]f lve ask ourse.','-,
ing to the earlier-mentioned second aspect of our theory, the answer would be lerv light upor Lrr i:.r
affirmative. Our proposal posits that even the artist's gestures producing the art work :ion and mimetrc :.--
can induce an empathic engagement of the observer, by activating the simulation of ¡elatedness to ae:i:-.:
the corresponding motor programme. The marks on paintings and sculptures are the ¡he relevance ofthe l, i
visible traces of goal-directed movements, hence in principle capable of activating ¡iom a specific enc.'
the somato-topically relevant motor areas in the observer's brain, as suggested by neurons', so to sFar;
mirror neuron research. While at present there are no published experiments tackling specific input-outp *:
this issue, there is empirical evidence indirectly suggesting that this could be indeed the brain.
the case. Several studies show that motor simulation can be induced in the observer's The MNS and rr: -
brain also when what is observed is not someone else's action, but the static graphic lation, are inlolr-e: ..
artefact produced by the action, such as a letter or a graphic stroke (Longcamp et al. the multiple and :.:
2oo5, z006, zooS). This shows that our brain is able to reconstruct actions a posteriori underpins a fund¿:.r.
by merely observing the static graphic outcome of the agent's past action. This ttess and reciproci:,, .:.'.
reconstruction process during observation is an embodied simulation mechanism, its phylogenetic arl
relying on the activation of the same motor centres required to produce the observed of action and altlc:. -
graphic sign. We predict that similar results could be obtained using as stimuli simulation and the 1,1.
artworks characterized by peculiar gestural signs of the artist, as in the case of sophisticated soci¿, ;,
Fontana's or Lucis Iackson Pollock's works. during the last ti \-e :: j
As far as a supposed link between the MNS and creativity, often suggested by the the sensory-rflotor S.,-:
media, it is my opinion that such a relationship at the present stage of the research is too cognition, languaue ,-
MIRROR NEURONS AND ART 46r
;n .rnbodiment and far-fetched. Creativity is a distinguished feature of the human condition that I am
-xe \lerleau-Ponty. afraid can hardly be reduced to the functional properties of specific populations of
rnit¿tion of what is neurons, mirror neurons included.
--,i the rvork itself, as On the other hand, though, I would like to say that creativity cannot either be
reduced to the mere product of a disembodied cognitive apparatus. In L'(Eil et l'esprit
-r ..¡r this tradition of Merleau-Ponty writes that we could not possibly imagine how a mind could paint.
:: ¡ecentlvtaught us Commenting on the statement by Cézanne that 'nature is on the inside', Merleau-
::;ts of the human Ponty argues that things arouse in us a carnal formula of their presence. Pushing for-
:e E:oPosing atheory ward this line ofthought, I'd suggest that the artist expresses her/his creativityprecisely
::.ii\-e. intuitive, or by being able to retrieve such carnal formula and making it available to others. This
:¡ ir the brain. Our shouldn't imply that such expression is the outcome of a deliberate conscious effort.
:-.¡lt-rnship between Actually, I think that most of the time it is totally unconscious. As Gilles Deleuze has
: :nd the content of beautifullyemphasized (zoo5), Bacon is a perfect exemplification ofthat. Bacon's work
¡rJ sensations por- epitomizes how the artist's creativity can be so effective in making visible the invisible
I .: the 'rvhat' of aes- forces buried inside material objects as well as inside the human psyche.
-- r-irrr-en empathic Many scholars are worried that the idea of mirror neurons has filtered down to a
- :r:r-isible traces of popular level in an inaccurate way, thus often being pushed too far. Some news reports
a:¡ the movement
Llf have used it in contexts where it does not apply. Naturally, one should distinguish
i¡ rorr-'of aesthetic scientific research from its trivialization by the media. Scientists are responsible for
ir¿¿ricln b,v means of their empirical work and the theories they put forward on the basis of what the evi-
l :nd carrying out in dence suggests to them. How the media communicate, or even distort scientific discov-
eries, is totallybeyond our control.
:bs:ract art, accord- If we ask ourselves what are the aspects of human social cognition we can shed
r- ¿ns\\'er would be new light upon on the basis of our discovery of the MNS, the answer would be imita-
-,cucing the art work tion and mimetic behaviours, action and intention understanding, empathy and its
:r: the simulation of relatedness to aesthetic experience, and language. I'd like to emphasize, though, that
ri sculptures are the the relevance of the MNS in so many different aspects of social cognition does not stem
,-::able of activating from a specific endowment of these neural cells, as if mirror neurons were 'magic
":li:. as suggested by neurons', so to speak. Mirror neurons derive their functional properties from the
¡:o-:eriments tackling specific input-output connections they entertain with other populations of neurons in
:::is could be indeed the brain.
:¡ed in the observer's The MNS and the functional mechanism describing their activity, embodied simu-
r:t the static graphic lation, are involved in so many aspects of social cognition because the activation of
--.,e Longcamp et al. the multiple and parallel cortico-cortical circuits instantiating mirror properties
;r :ciions a posteriori underpins a fundamental aspect of social cognition, that is, the multi-level connected-
: s rast action. This ness and reciprocity among individuals within a social group. Such connectedness finds
:u-ation mechanism, its phylogenetic and ontogenetic roots in the social sharing of situated experiences
::.-,duce the observed of action and affect. The MNS provides the neural basis of such sharing. Embodied
r:i using as stimuli simulation and the MNS certainly cannot provide a full and thorough account of our
sophisticated social cognitive skills. However, I believe that the evidence accumulated
':. as in the case of during the last r5 years indicates that embodied mechanisms involving the activation of
*jn suggested by the the sensory-motor system, of which the MNS is part, do play a major role in social
- ¡,itheresearchistoo cognition, language included. A second merit of our discovery consists in the fact that
462 VITTORIO GALLESE
it enables the grounding of social cognition into the experiential domain of existence,
something that classic cognitivism has totally neglected.
RnpEnENcEs
Calvo-Merino B, Glaser DE, Grezes i, Passingham RE, and Haggard P (zoo5). Action observa-
tion and acquired motor skills: an FMRI study with expert dancers. Cerebral Cortex, t5,
t243*9.
Calvo-Merino B, Grézes J, Glaser DE, Passingham RE, and Haggard P (zoo6). Seeing or doing?
Influence of visual and motor familiarity in action observation. Current Biology, t6,
19o5-1o.
Deieuze G (zoo5). Francis Bacon: The Logic of Sensation.University of Minnesota Press, Ann
Arbor, MI.
Freedberg D and GalleseY (zoo7). Motion, emotion and empathy in esthetic experience.
Trends in Cognitive Sciences, r r. 197-2oj.
Gallese V (zoo6). Intentional attunement: A neurophysiological perspective on social cogni-
tion and its disruption in autism. Experimental Brain Research Cognitive Brain Research,
1079, t5-24.
Gallese Y (zoo7). Before and below theory of mind: embodied simulation and the neural
correlates of social cognition. Philosophical Transactions of the RoyaI Society of London, 362,
659*69.
Gallese V (zoo8). Mirror neurons and the social nature of language: The neural exploitation
hypothesis. Soci al N e u roscie nce, 3, 317-33.
Gallese V and Lakoff G (zoo5). The brain's concepts: the role of the sensory-motor system in
reason and language. Cognitive N europ sychology, 22, 455-7 g.
Gallese V, Keysers C, and Rizzolatti G (zoo4). A unif ing view of the basis of social cognition.
Trends in Cognitive Sciences, 8, 396-403.
Gallese V, Fadiga L, Fogassi L, and Rizzolatti G (1996). Action recognition in the premotor
cortex. Brain, ttg, 593-609.
Gallese V, Fadiga L, Fogassi L, and Rizzolatti G (zooz). Action representation and the inferior
parietal lobule. In W Prinz and B Hommel, eds. Common Mechanisms in Perception and
Action: Attention and Performance, Vol. NX., pp. 247-66. Oxford University Press,
Oxford.
Gazzola V, Rizzolatti G, Wicker B, and Keysers C (zoo7). The anthropomorphic brain: the
mirror neuron system responds to human and robotic actions. Neuroimage,35, t674-84.
Kohler E, Keysers C, Umiltá MA, Fogassi L, Gallese V, and Rizzolatti G (zooz). Hearing sounds,
understanding actions: Action representation in mirror neurons. Science,297,846-8.
Lipps T (:897). Raumtisthetik und geometrisch-optische Triuschungen. lA Barth, Lerpzig.
Lipps T (r9o3). Einfühlung, innere Nachahmung, und Organempfindungen. Archiy für die
gesammte Psychologie, t, t}5-2o4.
Longcamp M, Anton lL, Roth M, and Velay IL (zoo5). Premotor activations in response to
visually presented single letters depend on the hand used to write: a study on left-handers.
N eur op sy cholo gia, 43, tB ot-9.
Longcamp M, Tanskanen T, and Hari R (zoo6). The imprint of action: motor cortex involve-
ment in visual perception of handwritten letters. Neuroimage,33, 68r-8.
MTRROR NEURONS AND ART 463
t
'-':rarn of existence, Longcamp M, Boucard c, Gilhodes JC, et al. (zoo8). Learning through hand or tlpewriting
influences visual recognition of new graphic shapes: behavioral and functional imaging
evidence. Journal of Cognitive Neuroscience, zo, 8oz-t5.
Merleau-PontyM Og6+). L'(EiI et l'esprit. Gallimard, Paris.
Rizzolatti G and Craighero L (zoo4). The mirror neuron system. Annual Reyiew Neuroscience,
27, t69-92.
Rizzolatti G, Fadiga L, Gallese V, and Fogassi L (rqq6) Premotor cortex and the recognition of
motor actions. Cognitive Brain Research,3, 131-47.
---¡.Lctionobserva-
Vischer R (r8z¡). Über das optische Formgefi)hl: Ein Beitriig zur Asthetik. Credner, Lerpzig.
. J:.¿l.ral Cortex, t5,
Wóltflin H (1836). Prolegomena zu einer Psychologie der Architektur.Berlin.
: . Seeing or doing?
,.,':¿itf Biology, :.6,
: -.:hetic experience.
-¿ rtural exploitation
:- :. -:rlotor system in
r rn the premotor
,.,.:,ls in response to
:: .' :-,' Lrn left -hande¡s.
A\{IR" -i.
lt\
P:i:. :-.: , .
t'ti:.¿-.:,
I rri¡l: I...-, :
percea t-
The ar'¡:,.--.
than the .'. - ,.
cHAPTER 23
PICTORIAL ART
BEYOND SIGHT:
REVEALING THE
MIND OF A BLIND
PAINTER
Claude Monet
The average blind person knows more about what it means to be sighted
than the average sighted person knows about what it means to be blind
-- -. 1i¡r r¡r-
INrnoo ucr r oN ]..*.,'.:.1-'.-
- .-,f . J\L ;,-L\ L
:
- --.ir3 of a h:cl- :
Anrrsrrc expression is a complex behaviour that engages many aspects of perception, ,r_r. 19ó-5i Zeki :--:
:-- -L - _ -
cognition, and ernotion (Zeki zoor). Painting and, in particular, drawing, represent a :--lÉL(\ 111 tl.ú l -- -
form of visual communication associated with the ability to construct, manipulate, --.:r-.- \'isuat Crrtti '-
and ultimatelytranslate the contents of one's own mental representations (Livingstone .,.¿,. Thes¿ i'-: ..., .:
r988). Inherent to this process is the engagement of mental imagery-the role of which : :r:r in the t,-:.. :
has been well established in visuospatial reasoning and creative thought (Kosslyn et al. . : .iese are¿s ir: .:
zoor). Current evidence suggests that in sighted individuals, mental representations , . -:r¡ -1t sight? S:u: ,--
are pictorial in nature rather than symbolic or verbal (Kosslyn et al. zoor; Kosslyn zoo5; : i¡maqe tL-r ::- :'. :
Slotnick et al. zoo5), and that mental imagery draws on much of the same neural .: :¿r¡ícuiar. irr Lr-. ,
machinery as visual perception (Mitchison r996; Kosslyn et al. zoor). .:": behar-iour¿i -- . : '
Conventional wisdom assumes that visual mental imagery is largely dependent on llind indirrc,r.,, :
recollections of prior visual experience. If so, any impairment in the development, ,, : r-d that re L¡s
'.:.',
or function, of the visual system should critically modifr one's ability to create and .-- n about the-r ,--::
manipulate visual mental representations. Thus, the lack of any prior visual experience *:al tasks inr c,-', .-
would likely impede upon a person's ability to create the visual mental constructs .::-r'rduals pe rl-- i:--
required for artistic expression. Is it possible to generate mental pictures via alternate i.<.Lel rgEE: R¿us;:.
sensory modalities in the absence of any prior visual experience? If so, what are the : - ,'o: -\medi e: ¿i. :
neural substrates that support such a process? This is what we discuss here in the ::¿nges and ¿d::::.
framework of understanding vision, brain reorganization in blindness, and the inter- ::,¿se con-]pensa:, :,
actions between brain research and art. -:3se neuropl¿sl; :
l rurc€55 of r-tsi0:: ::,.
::e occipital ., isu:- .
into precise patterns and connections. Initially, these networks are laid out during :nemoñ'(Roder c: .-
development and controiled by a variety of genetic factors. Neurons, given their Given that the :.:..
sophisticated and complex degree of specialization, are likely highly stable and resis- cessing, to proces: .
tant to change. However, what is now becoming increasingly clear is that neural net- 'r'isua1' brain of ::;
.
works and the connections between neurons are highly dynamic and constantly cortex alrvavs proc:::
changing in response to environmental input and experience. The nervous system's blind and the sier''.:
abilityto change, or'neuroplasticity,' is an ongoing, lifelong process and is an intrinsic the sighted, o¡ do ;r*
property ofthe brain (Pascual-Leone et al. zoo5). different? What i. ::,
Alcmaeon of Croton, probably the first person to suggest that the mind is located How could one q¿ir -:
in the brain and not the heart, also suggested that the optic nerves are light-bearing so how does the la;..
paths to the brain. His revolutionary ideas, formulated about z5oo years ago, were discuss here, one rt:.',.
PICTORIAL ART BEYOND SIGHT 467
ignored by Egyptian and Greek scholars alike (most notably by Aristotle) . On the basis
of the vast amount of anatomical and electrophysiological studies, we now have a
picture of a highly diverse and hierarchically structured system (Hubel and Wiesel
:ects ofperception, t963, 1965; Zeki ry28; Felleman and Van Essen r99r). This hierarchical organization
L:a,,ring, represent a originates in the photoreceptors of the eye, to the optic nerve via the thalamus, to pri-
-slruct, manipulate, mary visual cortex (visual area Vr), and beyond to an array of higher-order visual
:¡rions (Livingstone areas. These visual areas, located in the occipital lobe (one of the four lobes of the
, role ofwhich cortex) in the back of the brain, cover a large portion of the brain. But what is the fate
.
-the
urht (Kosslyn et al. of these areas in an individual who becomes blind, or even, in an individual born
:-tai representations without sight? Studying people who are peripherally blind (for example, from disease
:rou Kosslyn zoo5; or damage to the eye) provides a unique opportunity to investigate this question and,
:': the same neural in particular, to uncover the neuroplastic changes that follow blindness and its associ-
1 ated behavioural consequences.
:,:eh'dependent on Blind individuals have to make striking adjustments in order to adapt within a
: ihe development, world that relies heavily on vision. Remarkably, they learn to extract relevant informa-
:b-lin to create and tion about their surroundings using their remaining senses. When assessing behav-
-- r i-isual experience ioural tasks involving touch, hearing, and memory, there is ample evidence that blind
- ¡rental constructs individuals perform on par, or in some cases better, than those with sight (Hollins and
,;tures via alternate Kelley 1988; Rauschecker 1995; Lessard et al. t998; Roder ef aI. t999; Van Boven et al.
: -f so. n'hat are the zooo; Amedi ef al. zoo3; Gougoux et al. zoo4; Collignon et al. zoo8;). This suggests that
i:.cuss here in the changes and adaptations take place in the brain and, in turn, may be responsible for
1''.ss. and the inter- these compensatory behaviours. Accumulating evidence now points to the notion that
these neuroplastic changes occur within the part of the brain once dedicated to the
process of vision itself. Over the last decade, key studies have shown that, in the blind,
the occipital visual cortex is engaged in processing non-visual sensory information
such as Braille reading and tactile object recognition (Sadato et al. t996, t998, zoo4;
Buchel et al. 1998; Burton ef al. zooz, zoo6; Amedi et al. zoo3), while deactivating the
visual cortex of early blind by transcranial magnetic stimulation (TMS) abolishes
IN Braille reading capacity (Cohen et aL. ry97;Kupers et al. zooT), andTMS of the occipital
cortex induces tactile sensations in blind Braille readers (Kupers et al. zoo6; Ptito
et al. zoo8). The occipital cortex is also engaged in auditory information processing,
including, particularly, sound source localization (Kujala et al. t995, zoo5; Gougoux
.r:icateh' arranged efal. zoo4, zoo5) and even cognitive linguistic tasks such as verb generation and verbal
¿re laid out during memory (Roder etal. zooz;Amedi et al. zoo3, zoo4; Ofan andZohary zooT).
ilrrrns, given their Given that the blind use parts of the occipital cortex that sighted use for visual pro-
h,', stable and resis- cessing, to process other sensory modalities, many interesting questions arise: is the
:r is that neural net- 'visual' brain of the blind the same as that of the sighted? Can the occipital 'visual'
::; and constantly cortex always process other sensory modalities, or is it fundamentally different in the
:,i nen-ous system's blind and the sighted? Do the blind mentally imagine the world the same way as
ss ¿rd is an intrinsic the sighted, or do other sensations cause a form of mental imagery that is inherently
different? What is the nature of the mental representations within the blind brain?
::le mind is located How could one gain insight into this question? Our brains are fundamentally creative,
':. are light-bearing so how does the lack of prior visual experience influence this process? As we will
i _ i_r r'ۇIS ago, \Mefe discuss here, one way to explore these questions is through the expression ofart.
468 AMrR AMEDr ET AL.
Research using sensory substitution devices (Bach-y-Rita et al. 1969; Bach-y-Rita tound blindness. t':.
and Kercel zoo3) provides an opportunity to further test these questions, and shows :orneal scaring. a ;.:
that blind individuals are able to construct an image perceived by an intact modality ness. Examinatici- :
(e.g. tactile (Ptito et al. zoo5) or auditory (Arno et al. zoor; Renier et aL. zoo4, zoo5; -,-lsion and lr'as p1,,,,
-
Amedi et al. zooT)) by 'translating' information originated by the substituted modality E.-\. has descr-b¿:
(vision, via a camera instead of the eyes). Whether this process is performed by cross- he telt sociaLlv rs,- ,.:
modal changes (in other words, the recruitment of brain areas normally devoted to Jrarr ing in the s¡r; :
processing information from one sensory modality to process information coming -r:e of sir, he er:E::.
from another modality) or mental imagery is a source of a wider debate (for extensive 'r: schooling. E.,i. .:
discussion see Poirier et al. zooT) . Evidence also suggests that congenitally blind indi- acn'lics and rr-¿te :-.
viduals are able to create mental sensory constructs and manipulate them (e.g. mental paper using a spe:i:-
rotation or mental maze navigation) at performance levels equal to that of the sighted kit i that allorrs ::::
(Heller and Ballesteros zoo6), to construct and understand tactile drawings (Kennedy :,tplore tactilel.,',
1983; D'Angiullietal. r998; KennedyandMerkaszooo; Kennedyandlgorzoo3;Kennedy The themes of :.,
and luricevic zoo6a,b), and according to Bertolo (zoo5), they are able to construct tactilelr'eramin:: .
and make a drawing of one of their dream scenes. Next, we present a summary of palette of colour. . -
recent results (Amedi et al. zoo8), of a unique individual, E.A., which serves to discuss einploved br-sign:-:
and further shed light on the questions we raised so far. :rv instruction r,-,:-,
associations ti.: ::,-.
often red). Othe: ¿,:
receir.ed from s:5:..
his ability to er¡l ,,::
E.A. Is A UNIQUE CASE oF A BLIND PAINTER \-antage points. a::.
back from sightec , :
Subject E.A. is a profoundly blind painter who lost his vision at a very young age.
Like other blind individuals, E.A. can capture the external world by touch. However,
thanks to his extraordinary drawing ability, he can reveal the nature of his internal
representation of objects captured by touch in a manner that is unequivocally under-
standable by a sighted person. Thus, by revealing his drawing to us, he is in a sense Rrs
revealing the internal representations of his brain (for a nice collection of his paintings
and drawing visit: http://www.armagan.com/paintings.asp). Furthermore, it appears \Ve designed a s:.*C.
that this representation matches that of what a sighted person would create despite his rvithsubjectE.A.'s j:
Iack of prior visual experience.
E.A. under six dr::-:
PICTORIAL ART BEYOND SIGHT 469
E BRAIN In a recent study (Amedi et al. zooS), we concentrated on E.A.'s skill of drawing
novel objects explored by touch and used functional neuroimaging (specifically,
functional magnetic neuroimaging, or fMRI) to investigate the relationship between
.¡oithe brain (Zeki neural activity within the brain and specific mental functions. fMRI is a powerful tool
:lagery-that is, the that has long been used to measure certain aspects of brain function. In short, fMRI
-; of mental imagery measures changes in the concentration ofblood oxygen levels near a neural event and,
:rical areas) as visual through a carefully designed experiment, allows one to identifi. the regions in the
-err-). As mentioned brain that are associated with a particular task or function (Logothetis and Wandell
:n;e impede upon a zoo4; Friston zoo5).
:ural level, we know At the time of study, E.A. was a 5r-year-old, right-handed male. A detailed neuro-
ophthalmologic examination, including electrodiagnostic testing, confirmed his pro-
r. r969; Bach-y-Rita found blindness. One of his eyes had never developed and the other showed massive
;.stions, and shows corneal scaring, a dense cataract, and retinal damage consistent with early-onset blind-
i' an intact modality ness. Examination of previous medical records revealed that E.A. has never had normal
::: et al. 2oo4> zoo5i vision and was profoundly blind by the age of five.
,-rrstituted modality E.A. has described himself as a self-taught artist and is Braille illiterate. Growing up,
:¡rtormed by cross- he felt socially isolated because of his blindness and would often spend hours alone
lrrrmally devoted to drawing in the sand and exploring the relief patterns ofthe figures he had drawn. At the
r:ic.rmation coming age of six, he engaged seriously in art and painting. Without any formal instruction
lebate (for extensive or schooling, E.A. started to paint using the tips of his fingers employing fast drying
::nitaily blind indi- acrylics and water-colours as well as oils. He also learned to draw with a pencil and
re üem (e.g. mental paper using a specially designed rubberized writing tablet (a Sewell raised line drawing
:¡, that of the sighted kit) that allows him to generate relief images that he can subsequently detect and
.trairings (Kennedy explore tactilely.
i i:or:oo3; Kennedy The themes of his paintings vary, and include objects that both can and cannot be
:: able to construct tactilely examined (such as fruit and clouds, respectively). His scenes utilize a vibrant
-s3nt a summary of palette of colour, often containing shadows, depth cues, and perspective akin to that
r-ch .en'es to discuss employed by sighted artists (see Fig. z3.r). His use of colour has obviously been guided
by instruction from sighted individuals who have informed him about tlpical colour
associations (for example, water is blue, trees are green, and the roofs of houses are
often red). Other aspects of his painting are likely similarly influenced by information
received from sighted observers (e.g. the painting of clouds or mountains). However'
his abilityto explore a novel object, rapidlydraw it in exquisite detail, and from various
) PAINTER vantage points, appears to be a skill he has developed without constant input or feed-
backfrom sighted observers (see Fig. z3.r).
ii ¿ \'en'young a8e.
i','touch. However,
:¡ure of his internal
iequir-ocally under-
-, us. he is in a sense
Rnsurrs FRoM NEURoTMAGTNG
.r:rr;r of his paintings
-iermore, it appears We designed a study aimed at uncovering the neural activation patterns associated
;id create despite his with subject E.A.'s drawing ability (Amedi et al. zoo8). Using fMR[, we scanned subject
E.A. under six different conditions: (i) while he explored and recognized an object
470 AMIR AMEDI ET AL.
including its tactile features through palpation; (z) while he drew the object he just
tactilely explored; (3) while he mentally imagined the same object; and (a) while he
retrieved the names of objects from a list previously memorized (verbal memory test).
As control conditions, we also scanned E.A. while he (5) scribbled a 'nonsense' figure
instead of an actual drawing and (0) moved his hands in space as if he were exploring
an object.
In order to reveal the brain activity associated with E.A.'s drawing skill, we sub-
.'-l1*--*¡.A\
'\-;:E:::- tracted the activity we found while E.A. scribbled from that measured when he actually
<¿ ? ¿. drew an object. In this way, we could better isolate the brain activation specifically
\l:
..\, associated with his drawing skill. We found clear activation in a variety of areas in the
brain including prefrontal and parietal areas (see Frg. 4.2). Of particular interest
\r, l/ was the activation seen within the occipital cortex, that is, the part of the brain that
\ .n,{!-
is normally activated when sighted individuals view a visual scene. Thus, when E.A.
draws actual objects (even subtracting the activity associated with the sensory and
motor acts associated with drawing itself), the part of his brain that is normally associ-
a, ated with seeing is activated even though he never saw (and has never seen) the actual
l. objects he draws.
As a further analysis, we also compared activation patterns for the different control
conditions. This included E.A. exploring objects tactilely, imagining the pictures
he drew, and recalling the names of the objects he drew. Consistent with previous
studies by our group and others (Ishai and Sagi 1995; Ishai et al. zoooa,b; O'Craven
and Kanwisher 2ooo; Kosslyn et al. zoor; Mechelli el al. zoo4; Amedi et al. zoo5), we
found activation in a variety of brain areas in each of these tasks. What was consistent
(and contrary to what we found when E.A. drew) was the fact that early visual cortical
areas were significantly less activated in all of these control conditions (Fig. 23.3). To
further test this directly, we also performed a direct comparison between drawing
objects and the tactile objects condition: we subtracted the activifywe foundwhile E.A.
.,ovel object tactually explored objects from that measured when he drew an object (see Fig. 23.3A).
ra y designed We again found greater and selective activation of the occipital cortex during the
:cnnique allows drawing condition. Specifically, we found the largest magnitude for drawing in the
no explore calcarine sulcus, corresponding to primary visual cortex of sighted (Fig. 23.38). This
nc ude both suggests that the visual cortex is engaged in the drawing task and that the activation is
scene and not a manifestation of basic or high-level object related tactile processing.
often contain
akin to that
oblect which E.A.
for a few minutes,
A -./,..
'L|r. ':'- i.
H[:H
@'""''oo'"'flp"*'o'n
B f] Draw and Scribble Ll Draw > Scribble
! e. o.O5 ico*
]p. o .0ú5 rCor
Wp.
np" 0.0005 (Corr.)
d
B
Posieiff :.¡,.:ilr
Ipo
Xp. o.oos (corr.) lroucx I
Fi1.23.2 (See Colour plate 9) Neuroimaging data of subject E.A. for drawing Fig.23.3 (See Cc r.
and scribbling objects. A) Example of behavioural data collected from a scanning comparison of d.a ¡,
run. Subject E.A.'s sketches (above) are shown compared to the object tactilely of drawing vers!s :.
explored or in response to the control scribble condition (below). B) Activation reconstruction of E.l
patterns associated with the contrast of drawing versus scribbling presented on a activation in se'',era
full inflated cortical reconstruction of E.A.'s brain (lateral and medial views). Most strong and selecli,-':
striking is the activation seen in the occipital areas localized around the calcarine and magnitude o'ar
sulcus (identified by cyan arrows). The calcarine sulcus anatomically corresponds following colour rt,
toearlyvisual areas(Vt andV2)in normallysightedindividuals. lnE.A.,these (blue), scribbling Ic,'
areas are active for drawing. (g reen).
Adapted from Amedi et al. (2008).
rncluded difterent i;
ability including the tactile perception of an object, verbal memory of object names, ivith planning an¿ ¿
sensory-motor control, mental imagery of objects explored by touch, and scribbling. parietal regions k::,
Thus, we were able to isolate each part of the perceptual and cognitive process involved. integration; Coib-,' ¿:,
When E.A.'s drawing activity was compared to nonsensical scribbling, strong activa- zoo5; Gobel et a,. :
tion implicating a specific network of cortical brain areas was found. This network not surprising that r:
PICTORIAL ART BEYOND SIGHT 473
I "*,*,,
r > Scribble
@r Left hemisphere
[]r
B
Posterior Calcarine LH Anter¡or Calcari¡e I PO Sulcus LH Lateral-Occ¡p¡tal LH
4". for drawing Fig.23.3 (See Colour plate 10) Neuroimaging data of subject E.A. for a direct
i from a scanning comparison of drawing versus tactilely exploring objects. A) The contrast
object tactilely of drawing versus touching objects is presented on a full inflated cortical
). B) Activation reconstruction of E.A.'s brain (lateral and medial views). Note again the robust
ng presented on a activation in several posterior ventral occipital areas. Activation for drawing was
redial views). Most strong and selective even in the calcarine sulcus. See also (B) for the time courses
:r.lnd the calcarine and magnitude of activation from this area. The time courses are depicted by the
cally corresponds following colour index: tactile objects (red), mental imagery (brown), drawing
s. ln E.A., these (blue), scribbling (cyan), motor control (orange), and naming/verbal memory
(green).
:: =':- i¡edi et al. [2008). Adapted from Amedi et al. (2008).
included different parts of the brain including frontal cortex (known to be involved
,:-.' of object names,
with planning and executive functions; Smith and Jonides rggg; Miller zooo) and
ucr. and scribbling. parietal regions (known to be involved with spatial representation and multisensory
'!'- process involved.
integration; Colby and Duhamel 1996; Grefkes etaI. zooz; Saito et al. zoo3; Amedi et al.
:L:re. strong activa- zoo5; Gobel et al. zoo6). Given the known function of these areas, it is perhaps
:und. This network not surprising that they were also active while E.A. drew. Studies in sighted subjects
474 AMIR AMEDI ET AL.
have also revealed strong activity in parietal-frontal networks during drawing early age, so, what l-:
tasks (Makuuchi et al. zoo3). In the case of both sighted and non-sighted artists, rvithin visual areas r:
the activity in frontal and parietal regions may correspond to processes that are must inherently be rr
necessary in drawing, such as transformations from perception to two-dimensional have yet to be ansrt'er
image production. The process of pr,-
What is perhaps most astonishing is the fact that while E.A. drew pictures, there component. In this .i
was also widespread activation in areas within the occipital cortex including primary cal aspects of E.A.'s c
visual areas-the part of the brain normally used for processing visual information draw an object, E. i.
in a sighted person (Figs. z3.z and 43). Given E.A.'s denial of any visual experience, rvas specificallv askec
present absolute blindness, and certain early and longstanding profound blindness, to the details as pLrs:l
it is surprising that his visual cortex area would be active in a skill that is so inherently duce each object. Tnr
visual. The specific pattern activation in E.A. implicating'visual'cortical areas during representations dS tr!
drawing is consistent with the notion of recruitment of visual areas for non-visual future studies). In h:
functions. However, this pattern of recruitment is different from other examples of nature of E.A.'s i¡::.
neuroplasticityreported in earlyblind individuals. Specifically, unlike other earlyblind unequivocaily unde:,
subjects (Amedi et al. zoo3), E.A. did not show activation ofvisual cortical areas during to comprehend the t
the verbal memory task. This might be because he has practised painting and drawing and quickly and pr:;
throughout his life and thus the resources of the 'visual' cortex were recruited for E.A. is revealins a :¿
this purpose and not for other compensatory behaviours such as verbal memory skills. internal representa.r -
Consistent with this idea are the facts that E.A. never learned to read Braille, and he The study of art:st
admittedly reports that his verbal memory skill is, in fact, quite poor. We have previ- interesting findine. :
ously shown that blind individuals showing superior verbal memory capabilities ral dementia (FTDr :
show maximal plasticity in the visual cortex for this skill while blind subjects having personality chanses ¿
average verbal memory showed modest plasticity (Amedi et al. zoo3). The resources can be accompaniei.
offeredbythe visuallydeprivedvisual cortexin earlyblind subjects are likelyinherently 1999; Drago et al. :
limited, and thus it is not surprising that extensive and selective training of one artistic abiliw and :r
skill (drawing in E.A.'s case) may come at the expense of the development of other and colleagues { rc,'-,i
capacities. A similar scenario exists in the case of proficient Braille readers. The expan- tion, which are de:¡
sion of the brain representation responsible for the preferred reading finger comes at reorganize itself. T::
the expense ofthe representation ofadjacent fingers ofthe same hand (Pascual-Leone similar form of ada::
and Torres 1993; Sterr et al. 1998). Thus, representation ofthe preferred reading index Through his tech::
finger in the sensorimotor cortex is significantly larger than normal (which is associ- of a uniquely adapre :
ated with a dramatic increase in precise motor control and tactual perceptual abilities), so much as a blind i-
while the representation of the little finger in the same hand is reduced, apparently just happens to be ¡i
encroached upon by the enlarged index finger representation. In a similar manner, neural correlates r-r: :
recruitment of the 'visual' cortex for drawing may limit the availability of the same our understanding '-,
brain region for verbal memory or auditory localization, thus accounting for E.A.'s visual art. Evidenc¿ :
poor verbal memory in the face of his extraordinary drawing abilities (or simply due to representations ca:: :
the fact that he is not as reliant on his verbal memory). (Pascual-Leone ari l
Another interesting finding is the fact that activation was seen in the occipital cortex abstract features ¡,i
while E.A. imagined the objects he drew (albeit at a reduced magnitude of activity). images. Such evii,-n,
This pattern of activation is also typically seen in normally sighted subjects during organization of the :
visual imagery tasks (Kosslyn et al. zoor, zoo5). E.A. is absolutely blind from a very reality.
PICTORIAL ART BEYOND SIGHT 47'
:".s during drawing early age, so, what kind of imagery does E.A. possess? Does the sole fact there is activity
ron-sighted artists, within visual areas in E.A. during imagery and drawing signiS, that his experience
-, process€s that are must inherently be visual, despite his lack of actual vision? These intriguing questions
¡o nr.o-dimensional have yet to be answered.
The process of producing an artistic image requires both a technical and creative
,irerr pictures, there component. In this study, it is important to note that we attempted to assess the techni-
:t rncluding primary cal aspects of E.A.'s drawing ability and not the source of his creativity. When asked to
¡',isual information draw an object, E.A. was given the instruction to reproduce the object on paper and
r',',,-rsual experience, was specifically asked not to attempt to create an 'artistic rendition', but be as faithful
I : lo üDd blindness,
L--¡
to the details as possible. During the scanning, E.A. was given only 15 seconds to pro-
.-t-,at is so inherently duce each object. Thus, our design emphasizes the more direct translation of internal
:,:'njcal areas during representations as opposed to large-scale creative elements (an interesting topic for
::e¿s tor non-visual future studies). In light of this, we think that it is more likely that we are revealing the
: ¡iher examples of nature of E.A.'s internal representation of objects as essentially visual (or at least
,oe other earlyblind unequivocally understandable by vision). This is confirmed by the fact that E.A. is able
; -:t:;al areas during to comprehend the shape of any novel, previously unexplored object by touch alone
::::inq and drawing and quickly and precisely draw it from any vantage point. However, it is possible that
i -,.,'i:e recruited for E.A. is revealing a remarkable ability to 'translate' into a visual image a non-visual
-::ai. memory skills. internal representation of objects.
:-ai Braille, and he The study of artistic development in other clinical settings has also revealed many
:,::" \\-e have previ- interesting findings and parallels. For example, studies in patients with frontotempo-
:.:::L-)n' capabilities ral dementia (FTD; a condition characterizedwith behaüoural syrnptoms including
::l .ubiects having personality changes and problems with executive function) suggest that this condition
- , _: . The resources can be accompaniedby dramatic changes in artistic abilities and expression (Gordon
¿r¿ Lriielr' inherently ry9q'Drago et al. zoo6). Studies of sighted artists have shown dramatic changes in
.1 -
training of one artistic ability and style following focal brain damage (Bogousslavsky zoo5). Annoni
', ¡-¡prnent of other and colleagues (zoo5) suggest that new artistic abilities may arise after brain dysfunc-
:-:,jers. The expan- tion, which are dependent upon the lesion site, as well as the ability of the brain to
,:::l nnger comes at reorganize itself. The occipital activity present in our study of E.A. may demonstrate a
l:¡ Pascual-Leone similar form of adaptation in a blind painter.
-:reci reading index Through his technical and creative skill, E.A. has likewise externalized the workings
r- r''hich is associ- of a uniquely adapted mind. In a sense, it is perhaps more logical to consider E.A. not
.
e ic;ptual abilities), so much as a blind individual who has learned to paint but, rather, a true artist who
r¿uced, apparently just happens to be blind. While there is still much work to be done to elucidate the
: ;. sintilar manner, neural correlates of artistic creativity, this case provides an important contribution to
-.,-lli¡'of the same our understanding of the perceptual transformation that must occur when creating
;,::rnting for E.A.'s visual art. Evidence from this case also supports the hlpothesis that internal mental
:s tr¡ simpll due to representations can be generated by sensory experiences that are universal in nature
(Pascual-Leone and Hamilton zoor). It is possible that the brain may contain more
úe r-,ccipital cortex abstract features of object form rather than just simple representations of visual
:rtude of activity). images. Such evidence contributes towards our understanding of the fundamental
::j subiects during organization of the human brain, the basis of creativity, as well as the perception of
bl,¡rd lrom a very reality.
AMIR AMEDI ET AL.
Arno P, De Volder AG, Vanlierde A, et al. (zoor). Occipital activation by pattern recognition Psychophysic:. q3. .:.
in the early blind using auditory substitution for vision. Neuroimage,4, @z-645. Hubel DH and \r--;s:-
Bach-y-Rita P, Collins CC, Saunders FA, White B, and Scadden L (1969). Vision substitution lournal of Pln':::.":
by tactile image projection. N ature, 22l^, 963-4. Hubel DH and \\-r¡,"
Bach-y-Rita P and Kercel SW (zoo3). Sensory substitution and the human-machine interface. striate visual are¡;
Trends in Cognitive Sciences, T, 54t_-6. Ishai A and Sagi D ,:
Bertoio H (zoo5). Visual imagery without visual perception? Psicologica,26, 173-88. 268, t772-1.
Bogousslavsky J boo). Artistic creativity, style and brain disorders. European Neurology,54, ishai A, Ungerleid:: L
1O3-11. of visual images. -',,,
Buchel C, Price C, Frackowiak RSi, and Friston K (rqq8). Different activation patterns in the Ishai A, Ungerleide: L -
visual cortex oflate and congenitally blind subjects. Brain, rzr, 4o9-r9. human occipitai i--r.
PICTORIAL ART BEYOND SIGHT 477
Burton H and Mclaren DG (zoo6). Visual cortex activation in late-onset, Braille naive blind
individuals: An fMRI study during semantic and phonological tasks with heard words.
N euro s cience Letter s, 392, 38- 42.
::.s of Health (Kz4 Burton H, Snyder AZ, Conturo TE, Akbudak E, Ollinger JM, and Raichle ME (zooz). Adaptive
changes in early and late blind: a ÍMRI study of Braille reading. JournalNeurophysiology,sT,
:¿, L.B.M and by the
589-6o1.
',i Foundation (GIF
Cohen LG, Celnik P, Pascual-Leone A, et al. (1997). Functional relevance of cross-modal plas-
?s','chobiology grant
ticity in blind humans. Nature,389, r8o-3.
¿nd the Eliyaho Pen ColbyCLandDuhamelJR(r996).spatialrepresentationsforactioninparietalcortex. Cognitive
:luals for the invalu- Brain Research, 5, to5-r5.
¡ress to undergo all Collignon O, Voss P, Lassonde M, and Lepore F (zooS). Cross-modal plasticity for the spatial
:¿ coordination and processing of sounds in visually deprived subjects. Experimental Brain Research, r9z,
-'rng out the neuro- 3$-58.
D'Angiulli A, Kennedy IM, and Heller MA (rqq8). Blind children recognizing tactile pictures
i-sh translation; and
respond like sighted children given guidance in exploration. Scandinavian Journal of
-::d -\IB) for estab- P sy cholo gy, 39, t87-9o.
Drago V, Foster PS, Trifiletti D, FitzGerald DB, Kluger BM, Crucian GP, et al. (zoo6). What's
inside the art? The influence of frontotemporal dementia in art production. Neurologt,67,
rz85-7.
Felleman DJ and Van Essen DC (rqqr). Distributed hierarchical processing in the primate
cerebral cortex. Cerebral Cortex, L, 7-47.
Friston, KJ (zoof). Models of brain function in neuroimaging. Annual Review of Psychology,
)*: .ortex activation j6, 57-87'
,'-; ::. ri,\c¿r¡oscience, 6, Gobel SM, Calabria M, Farne A, and Rossetti Y (zoo6). Parietal rTMS distorts the mental
number line: Simulating'spatial' neglect in healthy sub jecfs. Neuropsychologia,44' 86o-8.
:,,' ::ragnetic stimula- Gordon N (1999). Unexpected artistic creativity. Developmental Medicine and ChildNeurologlr,
',,:,: subjects. Nature 41,27r.
Gougoux F, Lepore F, Lassonde M, Voss P,Zatone RJ, and Belin P (zoo$. Neuropsychology:
: li¡¡mer MJ (zoo5). pitch discrimination in the early blind. Nature, 43o,3o9.
: -,blect recognition. Gougoux F, Zatorre R|, Lassonde M, Voss P, and Leopre F (zoo5). A functional neuroimaging
study of sound localization: visual cortex activity predicts performance in early-blind indi-
' -:,:,-auditory sensory viduals. PLoS BIOLOGY, i, 324-33.
,:--;- ro. 687-9. Grefkes C, Weiss P, Zilles K, and Fink G (zooz). Crossmodal processing of object features in
. ,:'¡relates of drawing human anterior intraparietal cortex: an fMRI study implies equivalencies between humans
and monkeys. N eur on, 35, 17 3-8 4.
'- riot). Changes in Heller MA and Ballesteros S (zoo6). Touch and Blindness: Psychology and Neuroscience.
jr i --r-ii'r' an d P sy chiatry, Lawrence Erlbaum Associates, Mahwah, NI.
Hollins M and Kelley EK (rqSS). Spatial updating in blind and sighted people. Perception and
:'" F¿ttern recognition Psychophysics, 43, 38o-8.
.I_1. '_ij-u+). Hubel DH and Wiesel TN (rq6¡). Shape and arrangement of columns in cat's striate cortex.
. \-rsion substitution Journal of Physiologlr, t6i, 559-68.
Hubel DH and Wiesel TN (rq6f). Receptive fields and functional architecture in two non-
:::-- n¿chine interface. striate üsual areas (r8 and r9) ofthe cat. Journal ofNeurophysiology,28,229-89.
Ishai A and Sagi D (rSSi). Common mechanisms of visual imagery and perception. Science,
:. :6. r-_¡-88. 268, r77z-4.
:,' -: ¿,1r1 Neurologlt, 54, Ishai A, Ungerleider LG, and Haxby |V (eoooa). Distributed neural systems for the generation
of visual images. Neuron, 28, 979-9o.
-1:i:on patterns in the Ishai A, Ungerleider LG, Martin A, and Haxby fV (zooob). The representation of objects in the
human occipital and temporal cortex. lournal of Cognitiue Neuroscience' L2,35-5:..
478 AMrR AMEDr ET AL.
Kapur N (t996). Paradoxical functional facilitation in brain-behaviour research. A critical Pascual-Leone A, -\ r--: ;,
review. Brain, t9, 7775-90. Reyiew of Neurt:::.'
Kennedy JM (tg8¡). What can we learn about pictures from the blind? American Scientist,Tr, Poirier C, De \/oide: .r
19-26. substitution. -\ ;:,-;,
Kennedy JM and Igor J (zoo3). Haptics and projection: drawings by Tracy, a blind adult. Ptito M, Moesgaarci. Sl
P erception, 32, 7rl5g-77. electrotactile stin;-
Kennedy iM and |uricevic I (zoo6a). Blind man draws using diminution in three dimensions. Ptito M, Fumal A, \f ::
Psychonomic Bulletin and Rettiew, 13,5o6_-9. of the occipital cr,:
Kennedy JM and )uricevic I (zoo6b). Foreshortening, convergence and drawings from a blind Experimental Br,t.:'.
adrlt. P er ception, 35, 8 47-5r. Rauschecker lP (rgg¡
Kennedy JM and Merkas CE (zooo). Depictions of motion devised by a blind person. Tr en ds in -Nr¿¿¿ r,..¡;¿'
P sy chono míc Bulletin ancl Review, 7, 7 oo-6. Renier L, Collignon i,
Kleege G (zoo5). Blindness and visual culture: an eyewitness account. Journal ofVisual Culture, sighted subjects u.L
4, 179-90. zz, Sr.
Kosslyn SM (zoo5). Mental images and the brain. Cognitive Neuropsychologlt,22,333-47. Renier L, Collignoe.'.-.
Kosslyn SM, Ganis G, and Thompson WL (zoor). Neural foundations of imagery. Nature depth perception u-
Rey iews N eur o science, 2, 635- 42. Roder B, Teder-S¿,¿r¿:
Kujala T, Huotilainen M, Sinkkonen J, et al. (1995). Visual cortex activation in blind humans auditory spatiai rrr-
during sound discrimination . N euro science Letters, 183, 143-6. Roder B, Stock O. Br.:
Kujala T, Palva M, Salonen O, et al. (zooi). The role of blind humans'visual cortex in auditory cortex in coneenit:-.
change detection. Neuroscience Letters, 37g,727-3t. Sadato N, Pascual-L.,-- :
Kupers R, Fumal A, Maertens de Noordhout A, Gjedde A, Schoenen f, and Ptito M (zoo6). Braille reading il :-
Transcranial magnetic stimulation of the visual cortex induces somatotopically organized Sadato N, Pascual-i¿ - :
qualia in blind subjects . Proceedings of the National Academy of Sciences, rc3,4256-6o. blind. Brain. nt. ..:-
Kupers R, Pappens M, Maertens de Noordhout A, Schoenen ], Ptito M, and Fumal A (zoo). Sadato N, Okada T. L
rTMS of the occipital cortex abolishes Braille reading and repetition priming in blind sub- visual cortex of rl:. :
j ects. N eur olo gy, 68, 69t-3. study in humans l.
Lessard N, Pare M, Lepore F, and Lassonde M (rqq8). Early-blind human subjects localize Saito DN, Okada T. t.i
sound sources better than sighted subjects. Nature,395,278-8o. shape matchins: . ,,
Livingstone MS (rq8S). Art, illusion and the visual system. Scientific American,258,78-85. Slotnick SD, Thon:, - :
Logothetis NK and wandell BA (zoo4). Interpreting the BOLD signal. Annual Review oJ pically organize; .":
Physiology,66,75-69. Smith EE and lonic:, '
Makuuchi M, Kaminaga T, and Sugishita M (zoo3). Both parietal lobes are involved in draw- t657-6t.
ing: a functional MRI study and implications for constructionai apraxia. Brain Research: Sterr A, Muller \f \Í. :.
Cognitiue Brain Research, 16, fi8*47. of changes in c'-,:: ,
Mechelli A, Price CL Friston Kf, and Ishai A (zool. Where botrom-up meets top-down: neu- J ourn al o/r\-e lr r, -.-- :.
ronal interactions during perception and imagery. Cerebral Cortex, t4, tz56-65. Van Boven R\\'. H:r:,-
Miller EK (zooo). The prefrontai cortex and cognitive control. Nature Reviews Neuroscience, t, spatial resolutio.. .-.
t9-o5. Zeki S (rqZ8). Fun;:r ::
Mitchison G (rqq6). Where is the mind's eye? Visual perception. Current Biology,6, 5o8-to. 44-8.
O'Craven KM and Kanwisher N (zooo). Mental imagery of faces and places activates corre- Zeki S (zoor). Essa..'. ,-.-.
sponding stimuius-specific brain regions. Journal of Cognitite Neuroscience, t2, ro73-23.
Ofan RH andZoharyF. (zoo7). Visual cortex activation in bilingual blind individuals during
use ofnative and second language. Cerebral Cortex,V,t249-59.
Pascual-Leone A and Hamilton R (zooi). The metamodal organization of the brain. Progress
in Brain Research, r3.4, 427-45.
Pascual-Leone A and Torres F (rSS¡). Plasticity of the sensorimotor cortex representation of
the reading finger in Braille readers. Brain, rr6(Pt r), 39-52.
PICTORIAL ART BEYOND SIGHT 479
.: research. A critical Pascual-Leone A, Amedi A, Fregni F, and Merabet LB (zoo5). The plastic human brain. Annual
Reyiew of Neuroscience, 28, 377-407.
:-,t t t' r i can S cientist, 7 t, Poirier C, De volder AG, and Scheiber C (zoo7). what neuroimaging tells us about sensory
substitution. Neuroscience and Biobehavioral Reviews, 31, to64-7o
.:an-, a blind adult. Ptito M, Moesgaard SM, Gjedde A, and Kupers R (zoo5). cross-modal plasticity revealed by
electrotactile stimulation of the tongue in the congenitally blind. Brain, :r¿8, 6o6-t4.
-i ihree dimensions. Ptito M, Fumal A, Martens de Noordhout A, Schoenen l, Gjedde A, and Kupers R (zoo8). TMS
of the occipital cortex induces tactile sensations in the fingers of blind Braille readers.
::.r';r¡g5 from a blind Experimefital Brain Research, fi4, t93_zoo.
Rauschecker lP (rSss). Compensatory plasticity and sensory substitution in the cerebral cortex.
:'.' a blind person. Trends in Neuroscience,lS, 36-43.
Renier L, Collignon O, Tranduy D, et ai. (zool. Visual cortex activation in early blind and
-,'-.;. ,:.f Yisual Culture, sighted subjects using an auditory visual substitution device to perceive depth. Neuroimage,
zz, Sr.
-.jr.:2, j33-47. Renier L, Collignon,O, Poirier C, et al. (zoo5). Cross modal activation of visual cortex during
. ,:i imagery. Nature depth perception using auditory substitution of vision. Neuroimage 26, 573-8o.
Roder B, Teder-Salejarvi W, Sterr A, Rosler F, Hillyard SA, and Neville Hi (rSSS). Improved
:- :: rr blind humans auditory spatial tuning in blind humans. Nature, 4oo, t62-6.
Roder B, Stock O, Bien S, Neville H, and Rosler F (zooz). Speech processing activates visual
::"- ;rrtex in auditory cortex in congenitally blind humans. European Journal of Neuroscience, 16, 93o-6
Sadato N, Pascual-Leone A, Grafman J, et al. (1996). Activation of the primaryvisual cortexby
.:,: Prito M (zoo6). Braille reading in blind subjects. Nature,38o, 526-8.
.,: -:icaLlv organized Sadato N, Pascual-Leone A, Grafman f, et al. (1998). Neural networks for Braille reading by the
.],.. r03. 1_1t56-6o. blind. Brain, t2L, tzl¡¡-zg.
--:: Fumal A(zoo7). Sadato N, Okada T, Kubota K, and Yonekura Y (zoo4). Tactiie discrimination activates the
,:-::r-ils in blind sub- visual cortex of the recently blind naive to Braille: a functional magnetic resonance imaging
study in humans. Neuroscience Letters,35g, 49_-52.
:.: 'ubiects localize
Saito DN, Okada T, Morita Y, Yonekura Y, and Sadato N (zoo3). Tactile-visual cross-modal
shape matching: a functional MRI study. Cognitive Brain Research, L7, :14-25.
-:-:.;':. :,58, 78-85. Slotnick SD, Thompson WL, and Kosslyn SM (zooS). Visual mental imagery induces retinoto-
- .:--.t:ual Review of pically organized activation of early visual areas. Cerebral Cortex, 15, t57o-83.
Smith EE and lonides I G99g¡ . Storage and executive processes in the frontal I obes. Science 283,
.--. -rloir-ed in draw- 1657-6r.
:;¿. Brttin Research: Sterr A, Muller MM, Elbert T, Rockstroh B, Pantev C, and Taub E (rqqS). Perceptual correlates
of changes in cortical representation of fingers in blind multifinger Braille readers. Tfte
:;::. lop-down: neu- lournal of Neuroscience, 18, 44t7-23.
--t l--1]). Van Boven RW, Hamilton RH, Kauffman T, Keenan JP, and Pascual-Leone A (zooo). Tactile
-.,",, : \ e ur os cience, \ spatial resolution in blind Braille readers. Neurology, 54,2no-6.
Zeki S (rqZ8). Functional specialisation in the visual cortex of the rhesus monkey. Nature,274,
i:-'.:gr'.6,5o8-1o. 44-8.
.:aa! acti\-ates corre- Zeki S (zoor). Essays on science and society. Artistic creativity and the brain. Science, 2g3, 5t-2.
:..._,'i. lr,. 1013-23,
: ,:: j:riduals during
:r: rep¡esentation of
VII
AR'
E]
S'
\I¡Nv forms of ¿n =
VISUAL MUSIC IN
ARTS AND MINDS:
EXPLORATIONS
WITH
SYNAESTHESIA
JAMIE WARD
M¡Nv forms of art either directly or indirectly stimulate more than one sense. This
chapter will consider one of the most common combinations-namely'visual music'
(e.g. Brougher et al. zool). Multisensory forms of art will be processed by the same
mechanisms that support multisensory perception in other contexts, such as listening
to speech whilst watching the speaker's lips move. However, the way in which multi-
sensory perception may affect our appreciation and understanding of art has received
very little empirical investigation.
In this chapter, I use the term 'visual music' in the broadest sense. At one level, it may
refer to forms of art in which both the senses of vision and hearing are directly
stimulated. For example, people will opt to go to a live concert not just because of the
acoustics, but because they like the visual experience of seeing the conductor and
violins moving in time to the music (see also Shaw-Miller, Chapter r3, this volume).
Watching ballet, going to a nightclub with a VJ (video jockey), animated movies
such as Disney's Fantasia, and 'Son et lumiéres' would be other instances of visual
music in which hearing and vision are both directly stimulated. However, there is
another sense in which the term üsual music maybe used. In some instances, only one
482 JAMIE WARD
sense may be directly stimulated but a second sense may be indirectly conjured.
For example, Kandinsky wanted people to be able to hear his paintings (Ione and Tyler
zoo3) and listening to music may evoke particular visual images.
The appeal of visual music should be explicable, at least in part, from a neuro- Sr-naesthesia has t.::
scientific understanding of multisensory perception. Multisensory perception occurs .erved; thev are e-i;
automatically whenever two or more senses are stimulated in the appropriate way ¿ted rvith that crr:ls
(e.g. at the same time or place). For example, responding to a visual stimulus can be \ lattingler. zoo6: \''-:
enhanced (fasterresponse, fewererrors) whenit is pairedwith irrelevant sounds (Miller highlv stable or e ¡ :r:
t98z). In some instances, stimulation of a secondary sense can modif,' the conscious ¡iir-avs trigger th¡ s::
experience of the attended sense. For example, in the McGurk illusion, listening to 'ba' elicit a red sphere. T:
while watching 'ga' results in an auditory percept of 'da' (McGurk and MacDonald use of the term's1r:,
1976). The use ofmultisensorypathways in thebrain maybe so common as to be evoked :erception in rr h:;:t
even when a stimulus is presented in a single sensory modality. Thus, functional share certain pro:.:
brain imaging has shown that watching silent lip-reading can activate the auditory olume). The part-:'.
"
cortex (Calvert et al. ry97), discriminating voices can activate regions dedicated to face have a difrerent ¡.:;.
processing (von Kriegstein et al. zoo5), smelling chocolate can activate the gustatory throughout liie r¡n:
cortex (Leger et al. zoo3) and so on. \\'ard and Simn¡r. :
There are a number of factors that determine whether an event will be processed a result of certain d: -
in a multisensory way (i.e. different sensory parts of the event are combined) or not 'blindness
blin¿: -
or
(i.e. different sensory parts treated independently). The timing and the spatial location Nthough on.- . r:
of the multisensory parts are crucial. However, the content of the multisensory parts one of the fir-e -{:r.r,
may also be critical. Evidence will be discussed later that shows that certain auditory Simner et al. :o,, ¡
features (e.g. pitch) tend to combine with certain visual features (e.g. lightness) in sys- etc.) arranged spa::.-
tematic ways. The matching of different multisensory attributes on the basis of content most common ii-r:r-lt:
is likely to partly reflect the perceptual history of the perceiver. For example, the smell certain verbal rn¿:e:-
of chocolate but not perfume will activate gustatory cortex because only the odour not pose a probiei:-, :
and taste of chocolate have previously been paired (Leger et al., zoo3). Some content- multisenson' per.-::
based multisensory correspondences could also reflect innate biases that exist in our r-ia semantic leve-s , :
perceptual systems that are relatively independent of learning (e.g. Mondloch and direct senson--s.:t: . l
Maurer zoo4). evidence to sugqes: :l
In this chapter, I will present evidence to sholv that certain auditory features tend iound in others. ¿i'" =
to be associated with certain visual attributes in systematic ways. Moreover, I will representations Ft..
discuss some new research that shows that different combinations of visual and audi- colours to letters
',,':;
tory features have consequences for one's hedonic judgements of multisensory stimuli Synaesthetic errc:,
(e.g. which do you prefer, or how pleasant is it?). In particular, I shall draw on evidence rare (o.zo/o of the p,- :
from one unusual variety of multisensory perception, namely synaesthesia, in which sound tend to be ; -
processing of one stimulus, attribute (e.g. its sound) automatically leads to the con- example is shorr'n :l
scious perception of an additional attribute (e.g. colour). To some extent, synaes- normally unidirec., ::
thesia can be viewed as an extension of the norrnal tendency to process unimodal sounds. Whilst it .,,..
stimuli in multisensory ways (e.g. watching silent lip-reading activates the auditory common than r-isit r-
cortex). However, the syraesthetic experiences are, by definition, consciously experi- four synaesthetes o i.
enced, whereas other forms of multisensory biases elicited from unimodal stimuli motion or the onse :
are not. whooshing. Nlorine :
VISUAL MUSIC IN ARTS AND MINDS 483
lrdirectly conjured.
:.res (lone and Tyler
Wrr¡.r rs syNAESTHESTA?
p.:rt, from a neuro- Synaesthesia has three defining characteristics: the experiences are consciously per-
r perception occurs ceived; they are elicited by processing some stimulus that is not normally associ-
::e appropriate way ated with that conscious percept; and they are elicited automatically (ward and
. "ral stimulus can be Mattingley zoo6; Ward zoo8). In addition, there is a tendency for the experiences to be
¡','ant sounds (Miller highly stable over time (Baron-Cohen et al. 1987). For example, the number <5\ may
'.:,lih'the conscious always trigger the same shade of brown or a particular note on the trumpet may always
¡n. listening to 'ba' elicit a red sphere. There is a trend, particularly in the art literature, to over-extend the
'r
::k and MacDonald use of the term'synaesthesia'to other situations. For example, normal multisensory
lrrtrrl 4S to be evoked perception in which two senses are stimulated is not synaesthesia although it may well
.'. Thus, functional share certain properties and mechanisms with it (see van Campen, Chapter 25, this
;i:I,-ate the auditory volume). The particular characteristics of synaesthesia described here are believed to
:i dedicated to face have a different causal antecedent. Developmental forms of synaesthesia that persist
---r-,'¿te the gustatory throughout life tend to run in families and are believed to be under genetic influences
(Ward and Simner, zoo5). Synaesthesia may also be acquired, typically temporarily, as
::: ',,,-Lll be processed a result of certain drugs (e.g. from LSD) or from sensory deprivation (e.g. progressive
;,rnbined) or not blindness or blindfolding). Here, I shall focus on the developmental forms.
:r; spatial location Although once considered very rare, synaesthesia involving elicited experiences in
:n lltisensory parts one of the five Aristotelian senses are found in around 4o/o of the adult population
:: certain auditory (Simner et al. zoo6). Synaesthetic experiences of sequences (days, months, numbers,
- -:qhtness) in sys- etc.) arranged spatially occur in at least po/o of the population (Sagiv et al. zoo6). The
:r;'rasis of content most common forms of synaesthesia are those in which colour or space is elicited by
-:',ample, the smell certain verbal material-particularly days, months, letters, and numbers. This need
-.¿ only the odour not pose a problem for the suggestion that synaesthesia is an adaptation of normal
r . Some content- multisensory perception. This is because normal multisensory integration can occur
:.s tnat exist in our via semantic levels of processing (i.e. based on stimulus content) rather than reflecting
q. -\londioch and direct sensory-sensory links (e.g. Pourtois and de Gelder zooz). Similarly, there is
evidence to suggest that the same types of associations found in synaesthesia can be
;:iLrn'teatures tend found in others, albeit unconsciously. Thus, processing of numbers may evoke spatial
-., l',Íoreover, I will representations (Fias and Fischer zoo5) and non-synaesthetes often associate similar
¡ :', isual and audi- colours to letters when forced to pick one (Simner et al. zoo5).
*rtrsensory stimuli
:-- Sgraesthetic experiences ofvision elicitedbysounds (e.g. music, noise) are relatively
: lrarr- on evidence rare (o.zo/o of the population, Simner et al. zoo6). The visual experiences induced by
:l:-ithesia, in which sound tend to be complex; consisting of colour, texture, shape, and movement. An
-', I¿ads to the con- example is shown in Fig. 24.t. For most people with synaesthesia, the experience is
r:le eÍent, synaes- normally unidirectional such that sounds may trigger vision but vision does not trigger
' trtlceSS unimodal sounds. Whilst it was once thought that sound-to-vision synaesthesia was far more
r:., ¡t.s the auditory common than vision-to-sound, this is now in doubt. Saenz and Koch (zoo8) reported
: , nsciously experi- four synaesthetes (out of 'several hundred' students who were asked) for whom visual
: *nimodal stimuli motion or the onset or offset of visual stimuli triggered sounds such as popping and
whooshing. Moving their eyes across static visual scenes did not trigger sounds for
484 JAMIE WARD
Svsrrrr
.: the non-srraes::;
Fi1.24.1 (See Colour plate 11) An example of a complex synaesthetic visual .re linked tosethe: :
experience (also consisting of movement) induced by a simple sound (F# on : resent particip¿:ti: ',,
the cello). ind get them tcr e:,¡-r
best match. For e,-.:--
¡reater visual lieh:nc- ¡
these s1'naesthetes, but this tlpe of synaesthesia does exist. One s1'naesthete, J.R.,
is louder than n:t,_,:-,
describes the Kandinsky painting, Composition VIII (r94), as
ent methodolog'' -' -
There is a huge splurge of sound left-hand top - booming but also a bit vulgarl Below it a .-rking them to resr,r:
rather mousy little meee sound which then translates into ohs and ahs and pops at the :nq another dime¡:s-,.
various circles. The lines are sharp and moving to the right with the sound of steel (like blades
performance on tr: :
scraping against one another). The triangle and boomerang shape are surprised and pop up
studies, moreo\-er. ..:
laughing with a whooo.
(Goller et al. zoog)
pte, judging the :::;:
:hat stimuli that .r. :
Most contemporary accounts of spraesthesia speak of anomalous links between sep- class as high/lort :r':
arate senses (e.g. extra pathways, greater feedback connections). For example, sound- colour (Marks r9r-
to-vision synaesthesia is generally construed as anomalous connectivity between our being lighter, \f ¿¡.-¡
senses of hearing and vision. These accounts all depend on the assumption that it is less round, N1a¡k_i r::
possible to define, from first principles, what our separate senses are. It also assumes paired with r-ertic¿- :
that all individuals have the same set of separate senses. Of course, there is no logical 1995). The effects a:. .
reason to limit our senses to the famous five described byAristotle (what about intero- used instead of ¡::::::
ception, proprioception> or pain?) and, to some extent, submodalities within each these links der-elop .::
sense (e.g. colour, motion) are functionally separable. An alternative way of conceptu- Turkewitz r98o r. a1::,.
alizing sound-vision synaesthesia is in terms of a failure to develop separate senses, metaphors.
rather than abnormal connectivity between two separate senses. The perceptual world Marks (1975 ) pro-, r:
of infants has been described as a 'sensual bouillabaisse. Sights have sounds, feelings literature. He notei s:
have tastes and smells can make him dizzy' (Maurer and Maurer 1988). Indeed there is sions of pitch and b:r
good empirical evidence to suggest that infants process unimodal stimuli in multi- visual size (lolver p::;
sensory ways that could be analogous to some adult varieties of synaesthesia. In this andvisual dimensro:r,
VISUAL MUSIC IN ARTS AND MINDS 485
In the non-synaesthetic population, the way in which auditory and visual dimensions
resthetic visual are linked together has been investigated in a number of ways. One method is to
sound (F# on present participants with unimodal stimuli that vary in one dimension (e.g. the pitch)
and get them to explicitly vary a second dimension (e.g. visual lightness) to choose a
best match. For example, increasing pitch and increasing loudness is associated with
greater visual lightness (Marks 1974). The same is found if words are used: thus sunlight
t: .iraesthete, J.R.,
is louder than moonlight, and a sneeze is brighter than a cough (Marks r98z). A differ-
ent methodology involves presenting participants with multisensory stimuli and
--.: -:igarl Below it a asking them to respond to one dimension (e.g. the pitch of the stimulus) whilst ignor-
.:. :nd pops at the ing another dimension (e.g. visual lightness). If the ignored dimension interacts with
.: -,: steel (like blades performance on the attended dimension then they are considered to be linked. These
.-::rsed and pop up
studies, moreover, establish that such links may be automatically accessed. For exam-
ple, judging the pitch of a sound is biased by the presence of a visual stimulus such
il¡ller et al. zoog)
that stimuli that are high pitch/light colour and low pitch/dark colour are faster to
-r -,¡ks benveen sep- class as high/low than stimuli that are high pitch/dark colour and low pitch/light
.:r erample, sound- colour (Marks 1987). The same is found for loudness paired with lightness (louder
:-.-"'in- between our being lighter, Marks 1987); pitch paired with shape (high pitch being sharper and
I jsunption that it is less round, Marks 1987); pitch paired with size (Gallace and Spence zoo6) and pitch
, ¡:;. It also assumes paired with vertical position (high pitch being higher in space, Ben-Artzi and Marks
'.. there is no logical 1995). The effects are still found when written words (e.g. dark, light, night, day) are
: i,,-:r¿t about intero- used instead of manipulating visual lightness (Martino and Marks 1999). Many of
::-iiies rvithin each these links develop early and are unlikely to be learned via language (Lewkowicz and
,', i i\'al of conCePtU- Turkewitz t98o), although they nonetheless enter into our language through certain
.,-'p separate senses, metaphors.
-.:r Marks (t975) provided a summary of historical cases of slaraesthesia reported in the
:erceptual world
,¿'.. sOunds, feelings literature. He noted structured relationships between the auditory and visual dimen-
.;i¡ .Indeedthereis sions of pitch and brightness (higher pitch being brighter) and pitch and perceived
::, .:irnuli in multi- visual size (lower pitch being larger). The fact that similar links between auditory
j-".1:isthesia. In this and visual dimensions can be found in synaesthetes as well as in the general population
486 JAMIE WARD
suggests that synaesthesia may be mediated by the same mechanisms that support mul- relationships be:-,,, c
tisensory integration more generally. the colours plotied
Although the relationship between auditory pitch and visual lightness had been around the midcle
noted many times before both in synaesthesia and the general population, there svnaesthetes and c¡,
had been no direct quantitative comparison of these two groups until very recently. dent of timbre ¿ s
Ward, Huckstep and Tsakanikos (zoo6) redressed this balance by directly comparing the same lightne ss
a group of sound-vision synaesthetes with controls. They were presented with a variety less saturated ani r:
of tones comprising different pitches, and different timbres (e.g. strings, piano, pure The studr. of \r-:.
tones). The task for the synaesthetes was to choose the nearest colour corresponding spondence benr-ee:r
to their synaesthetic experience and the task for the controls was to choose the 'best' However, the stuc-,-
colour that goes with that sound. The experiment was repeated within the same session to music consist o: ¡
and several weeks later, in order to ascertain the intrasubject consistency. As expected, position, and tert;:
the synaesthetes had far higher intrasubject consistency over time than controls. these additional a.:
An example of the colour choices from one of the synaesthetes is shown inFig.24.2. ested in extendir=
However, this study was specifically interested in intersubject consistency: would pleasantness and p:,
different participants choose colours in the same systematic way? If so, this would appreciate and e', ¿-.
provide evidence that synaesthesia recruits the same types of mechanisms used in music that ober- s',:
normal multisensory processing (in this instance, imagining a colour from a sound). because thev are tap:
Our results confirmed that the two groups did not differ quantitatively in their ten- Evidence that s-,:t
dency to associate pitch and lightness. However, the results also revealed more subtle from the lifesn le ci:
thetes are more li.-.,
a recent studt'bi'c,u:
Piano they have (\\'arti. t,
sr,naesthetes thal ¿:,
are far more 1ñ:.;
Sine svnaesthetes or n1:,a.
experiences are :i.l::-
those who percer'''. t:
String
ing too?
*
Anr
A
--]-
--- -rl- In order to capture ti
short animated clips .
Fi}.24.2 (see colour plate 12) An example of a synaesthetic participant's choice verbal descriptions. .
of colours for 10 different musical notes that were either pure tones (sine wave), sounds used to eirc::
or from piano or strings. The synaesthete chose each colour on two different played in a varie¡,'r,,
occasions and the Figure shows that similar colours tend to be chosen over time. loudness, and inte:.,-;
VISUAL MUSIC IN ARTS AND MINDS 487
ms that support mul- relationships between auditory and visual dimensions. For example, the saturation of
the colours plotted against increasing pitch showed an inverted-U function, peaking
.l lishtness had been around the middle-C point on the western scale. Again, this was found for both
:: population, there synaesthetes and controls equally. Moreover, whilst lightness was relatively indepen-
's until very recently. dent of timbre (a string, piano, and pure tone of the same pitch tends to produce
','directly comparing the same lightness) the same could not be said of saturation. The pure tones elicited
:.ented with a variety less saturated and more greyscale colours in both controls and synaesthetes.
. strings, piano, pure The study of Ward et al. (zoo6) provides evidence of an intricate system of corre-
ürrrur corresponding spondence between sound and vision that is common to synaesthetes and others.
,. :o choose the 'best' However, the study focused specifically on colour. The synaesthetic visual experiences
:h:r the same session to music consist of far more than colour. They also involve motion, shape, size, spatial
;rs:enn'. As expected, position, and texture. The studies reported later in this chapter were designed to take
:ri:re than controls. these additional aspects of their synaesthesia into account. Moreover, we were inter-
s shor'n inFig.24.2. ested in extending our research to take into account people's hedonic ratings of
;rrn5istency: would pleasantness and preference. This may lead to a better understanding of how people
¿"'': If so, this would appreciate and evaluate 'visual music' as a form of art. Combinations of vision and
:::echanisms used in music that obey synaesthetic principles may be judged to be particularly appealing
: -f ur lrom a sound). because they are tapping universal multisensory processes.
.:¿:ttelr- in their ten- Evidence that synaesthetic audiovisual experiences are aesthetically pleasing comes
r¿,,'ea1ed more subtle from the lifesty'e choices of synaesthetes themselves. It is often remarked that synaes-
thetes are more likelyto be engaged in the arts (e.g. painting, music, poetry). However,
a recent study by our group suggests that this depends on the qpe of synaesthesia that
they have (Ward et al. zoo8a). Sound-vision synaesthetes are the only subgroup of
synaesthetes that are more likely to play a musical instrument, and this subgroup
are far more likely to spend time producing paintings (or similar) than other
synaesthetes or non-slmaesthetes. Our explanation of this is that the audiovisual
experiences are intrinsically appealing and provide a strong source of motivation to
those who perceive them. But would others lacking synaesthesia judge them as appeal-
ing too?
.t
ba_
e:
Ann SYNAEsTHETIC ExPERIENcES
APPRECIATED BY OTHERS?
_-
In order to capture the movement, shapes, and textures associated with synaesthesia,
short animated clips were prepared (Ward et al. zooSb). These were based on drawings,
larticipant's choice verbal descriptions, annotations, and colour selections made by the synaesthetes. The
tones (sine wave), sounds used to elicit the synaesthetic visions were taken from cello and violin notes
t¡ro d ifferent played in a variety of pitches, manners of playing (e.g. plucking, vibrato), duration,
chosen over time. loudness, and intervals. Although the animations were not exact reproductions of
488 JAMIE WARD
their experience, the slmaesthetes confirmed that theywere adequate approximations. ::e suntmarizec t: i
This set of animations was then used as stimuli in a set of experiments conducted on i:i:dom pairines ',",,
non-synaesthetes in the general population. Our aim was to determine whether lence discussed::.
combinations of sound and vision based on synaesthesia would be rated as more :-irprisine. It is :,-- _-
pleasant and preferable to other sound-vision combinations. The rationale was that .:l left to rrsht c:::
a
the synaesthetic mappings between sound and vision would also be present in the gen- -: that it reflects tr:
eral population and so these would tend to be preferred. ,-'n rhelett of sp::. :
In order to produce a control (i.e. non-synaesthetic) set ofsound and vision, several :Fatial attentlor : .
manipulations were used. Firstly, the same set of visual experiences and sounds were ::ict arisine trtr:r :
randomly paired. This enabled us to determine the extent to which preference/liking .',raesthesia in ri. :
was related to the combination of audiovisual parts rather than the parts themselves. in the third an: ..
Secondly, the actual synaesthetic animations were distorted by changing the hue (by :nimations p¡Lrc.-. -
choosing the opposing hue on the colour wheel). Thirdly, the original synaesthetic :..-nts \\'ere instruc : -
animations were distorted by rotating them through 9o degrees whilst maintaining :.\ture in orde: i_ .
the same aspect ratio. This enabled us to assess the effect of position in space and/or -rs in the second ..,-,
direction of motion. The hue-changed stimuli and rotated stimuli were presented with .,\'.re paired to g-::,.- :
the original sound. :hoose u'hich tr:r; ,-
In the first experiment, participants were presented with single audiovisual pairs :hoose the sr-nea:,.-_.
(AV) corresponding to the original AV synaesthetic experiences or the three distorted
AV conditions. In addition theywere presented with the sounds alone (A) or the visual
animations alone (V). Theywere required to rate theirliking ofthe stimulus bymoving
a slider. Points on the slider were marked with smiley faces, ranging from very unhappy
through neutral to very huppy. The ratings were scored in terms of distance from
neutrality. The ratings for the AV pairs were compared to that expected if participants
based their rating on the mean of the A and V parts. For the random AV combinations, 20.0
the liking judgements were entirely predicted from the liking ratings of the parts.
However, for the synaesthetic AV combinations the liking judgements were signifi- c6 15 O
cantly greater than that expected from a part-based analysis. This suggests that these c-
.a
stimuli, but not the random pairings, are treated as multisensory objects in their own cd
0_ _^ ^
right. Contrary to our predictions, the distortions to the hue and spatial position were ñ^
IU-U
rated equivalently to the original synaesthetic AV stimuli. As such, this initial study
suggests that liking judgements may be governed by the temporal characteristics of
the AV combination (which was disrupted in the random pairings) but not necessarily
by the synaesthetic content of the stimuli.
In order to explore this further, a second experiment was conducted in which the
same stimuli were used but the experimental procedure was changed. There were two
visual stimuli presented with a single sound and the participants were required to
choose which animation they preferred to go with the sound. The two visual stimuli
were drawn from the three conditions in the previous experiment: random pairings,
hue change, and position change. This forced-choice manipulation was expected to be Fi1.24.3 When pla,,,
more sensitive to the different manipulations given that attention would be drawn to participants tend tc r
the different contents of the stimuli. Indeed, in this manipulation, the participants that sound than vit:
were sensitive to the synaesthetic content and, in all three conditions, the synaesthetic when a randomly cn:
animations were preferred over the control'non-synaesthetic'animations. The results played. The figure is r
VISUAL MUSIC IN ARTS AND MINDS 489
,:; .:.pproximations. are summarized in Fig. 24.3. The finding that the synaesthetic colour changes and the
:.:ls conducted on random pairings were rated less favourably is perhaps not surprising given the evi-
-l¿;:rmine whether dence discussed previously. However, the effect of rotating the image is perhaps more
: :c rated as more surprising. It is to be noted that most of the original synaesthetic animations moved
. :::ionale was that in a left to right direction. At present, the source of this bias is unclear. One possibility
: ::.sent in the gen- is that it reflects the general trend that we all have to attend more strongly to events
on the left of space that is believed to be due to the right hemispheric dominance for
: :::d r-ision, several spatial attention (e.g. Nicholls et al. 1999). A second possibility is that it is a cultural
., and sounds were effect arising from reading direction. I shall return to the possible role of culture in
: :reference/liking synaesthesia in the concluding section.
-. -:lrts themselves. in the third and final experiment, the synaesthetic AV animations were paired with
:r-¡rng the hue (by animations produced from control participants who lack synaesthesia. The partici-
:,:,n¿l svnaesthetic pants were instructed to think about the colour, movement, position, size, shape, and
rist maintaining
',,,
texture in order to make them as comparable to the synaesthetic stimuli as possible.
- r .:r space and/or As in the second experiment, the control animation and the synaesthetic animation
" ::; presentedwith were paired together with the appropriate sound and the participant was required to
choose which one they preferred. The participants were significantly more likely to
= =:drovisual pairs choose the synaesthetic animation over the control animation. This result is perhaps
: -.,r: three distorted
.; -\ orthevisual
, ,.r: -lus bymoving
: - :r lerv unhuppy Chance
:, -f distance from i
I
I
-:::j ifparticipants ¡\ I
I
I
i
'.,---'- ¡ombinations, I n
.l
I \
..-lss of the parts. I I \F'
.- iris \vere signifi- a
E ls.o I
I
I t\
,\
I
I
(ü I
: -l:qe sts that these o- I
I
:q I I a
I
:'-ais in their own i(ú
I
I
.j l,
t I
I
d1
:.:: r- position were 1o.o I
,-
I I
j .U j".. .l
I
.- :hrs initial study I
a
., ¡::¿racteristics of 5.0
/' f'
- -::rot necessarily
\.a---a--'1
:,---:rd in rvhich the
::: There were two 25-30 30-35 3540 40-45 45-50 50-55 s5-60 60-65 65-70 70-75 75-80 80*85 85-90 90-95
% Synaesthet¡c animations chosen
::',",ele required tO
: r'isual stimuli
--.", r¡ - -O - - Colour change . '.-Er.. .- Or¡entation change Random pairing
: =:dom pairings,
: ',. :s erpected to be Fig.2a3 When played a sound and two animated clips, non-synaesthetic
',
- -.1d be drawn to participants tend to prefer the animation based upon a synaesthetic experience to
:-.. :re participants that sound than when the same animation is changed for colour or orientation, or
rr ). the synaesthetic when a randomly chosen synaesthetic experience generated by another sound is
::.:,.rr-rs. The results played. The figure is adapted from data in Ward et al. (2008b).
49O IAMrE WARD
not surprising given the evidence that non-synaesthetes tend to match auditory and :re sumabh'. the::;
visual features in similar ways to synaesthetes. Whilst this maybe true, the animations i shillingr. \o:r-.'.
produced from the synaesthetes were far richer in detail than the controls and it is :onding to nu;::b¡:
likely that this could be driving the effect that we observed. l:and and larse¡ :-
In summary, there are systematic correspondences between the auditory and visual reduced in Pers:¡:, ,
domain that are present in all people but are consciously experienced by those with crossi¡: tF. :
e-,'en br-
sound-vision synaesthesia. These correspondences not only bias performance on sLrund-space ¿:t,j i
cognitive tasks (e.g. pitch discrimination in the presence of irrelevant vision) but also ¡he rr'al that tn: _-
affect hedonic judgements of multisensorystimuli (ward et al. zoo8b). It mayprovide relationship e.q. :-:
a strong source of motivation for those who experience such sensations to be engaged \euroscient:n: -
in music and visual art. :otentiallr-be ls.::
:rot ne cessarilr ; ¡,1
har-ing visual im¿:"
to heighten the:.,-::
rlorking in the r_¿-;
Wronn rMpLrcATroNS: syNAESTHESTA. methods fbr m¿n', ,
CULTURE, AND ART hoped that this ¿: _":
Having discussed the nature of the mechanisms by which sound and vision may be
related and how this affects hedonic ratings, it is worthwhile considering some of the
wider implications of this research. In particular, to what extent could neuroscientific
investigations of audiovisual integration and synaesthesia impact on visual music as an
art form, and, to what extent is synaesthesia itself a product of cultural and artistic
trends? The animations ¡:
It is to be noted that the animated reproductions of the synaesthetic experiences \Volverhampton. :t:
would be interpreted, by contemporary art historians, as abstract. However, it seems
highly unlikely that this is due to their exposure to such forms of art. The experiences
of synaesthetes in previous generations would almost certainly resemble those of pres-
ent-day synaesthetes given that the basic multisensory mechanisms of the brain would RnpnnENcr s
predate such cultural trends and are even present, to some degree, in infants. This
raises the intriguing possibility that abstraction is not a purely cultural phenomenon: Baron-Cohen S. \r-,,,.-
it could have existed in the experiences of synaesthetes before being recognized as a imental inr-es:i:":: -
distinct artistic style. Although synaesthetes in previous generations would have been Ben-Artzi E and -\ f .¡,, .
unable to interpret their experiences in terms of the concept abstraction they would stimulus ditte¡¿r --:
Brougher K, Stnc¡ . ',,
have comparable experiences to present-day synaesthetes.
Music Since 19,, . ..
This does not necessarily mean that synaesthesia is completely immune to cultural Calvert GA, Bullm,--:¿
biases. For example, it
is possible that the left-to-right direction of visual motion silent lipreadinr. _. _-:
induced by sound (Ward et al. zoo8b) reflects a cultural bias of reading direction, Dehaene S, Bossini S. .:
although there are no empirical studies of this to date. Other tlpes of synaesthesia do magnitude. Jou¡.,i,;.
show similar biases. For example, those sy'naesthetes who experience numbers spatially Fias W and Fische: _.,i.
often have deviations in their number forms at decade boundaries (ro, zo, 3o, etc.) Handbook o.f 1[L;:i.:,
Gallace A and Spence ,_.
reflecting our decimal system (Sagiv et al. z.oo6). However, synaesthetes in nineteenth-
century Britain tended to have deviations around the digit rz (Galton r88o) reflecting,
tion ofvisual size. ¡.
Galton F (1880). \'is-;._
VISUAL MUSIC IN ARTS AND MINDS 491
ratch auditory and presumably, the greater use of a duodecimal system (e.g.rzinches in a foot, rz pence in
:rue. the animations a shilling). Non-slmaesthetes in Western cultures also show a left-right bias in res-
re controls and it is ponding to numbers: they are faster at responding to smaller numbers with their left
hand and larger numbers with their right hand (Dehaene et al. 1993). The effect is
auditory and visual reduced in Persian speakers who read right-to-left, but is unaffected by handedness or
:nced by those with even by crossing the position of the hands over in space (Dehaene et al. 1993). As such,
as performance on sound-space and number-space relationships may reflect intrinsic properties of
ant r-ision) but also the way that the brain represents these dimensions, but the precise nature of the
: !¡ . It mayprovide relationship (e.g. direction) could be culturally modulated.
lirons to be engaged Neuroscientific investigations of audiovisual integration and synaesthesia could
potentially be used to create deliberate emotional responses in the perceiver. This need
not necessarily consist of the most compatible audiovisual pairings. For example,
having visual images that are 'non-synaesthetic' with respect to the music may be used
to heighten the experience during moments of musical tension. It is likely that artists
working in the field of visual music have, through trial and error, been using such
HESIA, methods for many years (see van Campen, Chapter 25, this volume). However, it is
hoped that this enterprise can be guided and illuminated by research in the sciences.
Goller AI, Otten LJ, and Ward I (2009). Seeing sounds and hearing colors: An event-related --jl r1- --
potential stucly of auditory-visual synaesthesia. Journal of Cognitive Neuroscience, 21, .'lJi_s:ta---..
--,rs: An event-related Ward L Thompson-Lake D, Ely R, and Kaminski F (2008a). Synaesthesia, creativity and art:
::. ¿ ]'ieurosc'ence, 21, What is the link? British J ournal of P sy chology, 99, I27 -41.
Ward J, Moore S, Thompson-Lake D, Salih S, and Beck B (2008b). The aesthetic appeal of
:, : : :'t r;' of N euro s cience, auditory-visual synaesthetic perceptions in people without s)maesthesia. Perception. 37,
t285-96.
-,:asal presentation of
::i to orthonasal pre-
:or. Chemical Senses,
:::;r-: auditory-visual
.:-.-:tions of sensory
, \¡rn York.
.::.',, 264,746-8.
:- ¡¿cundant signals.
: -:r:on' sr.¡resthesia.
T
S
Tr¡r.oucHour h-s:
lic's consciousne-.: .
experiments rr t:,. ,
research into sr n...
'S,vnaesthesia is :_ .
them.' I u,ill tbc:,
co-perception oi; .
M,rl
The interest in th¿
old as Greek philc.
in the middle of ti:
CHAPTER 25
VISUAL MUSIC
AND MUSICAL
PAINTINGS:
THE QUEST FOR
SYNAESTHESIA
IN THE ARTS
CRETIEN VAN CAMPEN
THnoucuour history artists have explored new multisensory ways to reach the pub-
lic's consciousness. In this chapter, I would like to give a historical review of artistic
experiments with synaesthesia and then relate them to current neuroscientific
research into synaesthesia. My working definition for this chapter is the following:
'Synaesthesia is an involuntary uncommon cooperation of the senses as we know
them.' I will focus on a prevalent q?e of synaesthesia: coloured hearing, i.e. the
co-perception of colour in hearing sounds or music.
Zeuch zooz). One of the questions raised by the classical philosophers was whether
lar::icho¡d ;. ;r ..
::--eit builde: :.. :.
the colour (chroia, what we now call timbre) of music was a physical quality that ',,'hich rose ¿b,,-,-.. '
could be quantified (Gage rqq:). P¡hagoras had already discovered that there was
a mathematical principle behind pitch, demonstrating that the pitch of a note could be
--riortunatel',. ,:-.
determined by the length of the string that produced it. Aristotle and his students tried
'," ls never hni.- -::
¡','Tohann Gor:, _ :
to take his deductions about pitch one step further to find a similar mathematical
Renorr-nei ¡'. ,:::
explanation for the relationship between pitch and colour. They assumed that the
:elelopmer:t ,-, ....
colour spectrum, ranging from dark to light, could be matched to a spectrum of pitch,
from low to high. They took just one aspect of colour-lightness-and ordered all
:;riormance,-,:¡.,
colours on a greyscale from black to white (nig' z5.r). They then linked the lowest
,,- shorr-ing the : ,
iow to high pitch, were linked to corresponding grey values, from light grey to dark
::chnical limi-:: :
grey (Gage 1993).
-
Ever since these early investigations, the quest to find the mathematical principles
underlying the correspondence between pitch and colour has been taken up by phi-
losophers, artists, and scientists (Peacock 1988; Gage ryg3,7999; Jewanski 1999; Maur
1999). Some philosophers used their imagination and speculative theory to connect
music and colour. For instance, Plato wrote that the eight celestial spheres are coloured
and are accompanied by eight tones (Gage 1993) '
The seventeenth-century physicist Isaac Newton had a more down-to-earth
approach and tried to solve the problem by assuming that musical tones and colour
tánes have frequencies in common (Peacock 1988; Gouk rggg). He attempted to link
sound oscillations to respective light waves. According to l'{ewton, the distribution of
white light in a spectrum of colours is analogous to the musical distribution of tones in
an octave. He identified seven discrete light entities (colours) to which he then matched
the seven discrete notes of an octave ((Newton úZSltl5Z; see Fig' z5'z)'
Inspired by Newton's theory of music-colour correspondences, the French
Jesuit Louis-Bertrand Castel took the theory a step further when he designed a colour
Fi1.25.2 Newton id
Fig.2S.1 An ascending chromatic scale with pitch matched to tones of increasing them to the seven ci
darkness.
VISUAL MUSIC AND MUSICAL PAINTINGS
-rphers was whether harpsichord (clavecin oculaire) (Peacock 1988; Franssen r99r). Assisted by the instru-
,¡r-sical quality that ment builder Rondet, castel designed a harpsichord with coloured strips of paper
ered that there was which rose above the cover of the harpsichord whenever a particular key was hit.
;h ofa note could be Unfortunately, the coloured strips could only be lit by candlelight and the instrument
:,1 his students tried was never finished. other versions were under development and a drawing of one
nllar mathematical by Iohann Gottlob Krüger has survived (Fig. z¡.¡).
:i- assumed that the Renowned composers like Telemann and Rameau were actively engaged in the
¿ spectrum of pitch, development of clayecins oculaire (Peacock 1988; Gage 1993), although the practical
ss-and ordered all performance of coloured music remained hampered by the technical impracticality
:r Li¡lied the lowest of showing the colours in a large concert hall. A set of paper strips illuminated by
:rs in between, from candles lacked the power to evoke a convincing visual experience for the audience. The
: Light grey to dark technical limitations made the experience very remote compared with what we can
i-natical principles
er :¿ken up by phi- P
-r..'anski 1999; Maur
¡ üeon'to connect c Itltt"x "x
á&,
¡:ieles are coloured t
Purplc
I
l¡,:: dorrn-to-earth I
:' IL-rnes and colour rl
:; ¿ttempted to link lndigo
r. :re distribution of (;:
::ibution of tones in Fa
-,:h he then matched Blerv
lr.l ¡
l;nces, the French
H
S¿l
:- cesigned a colour Grccn
I
ll
Ycllow
Xi
luti
'iÍrs" td
Rüd
I
I
I g Ss¡
{I
rr
I
Fi1.25.2 Newton identified seven discrete light entities (colours) and matched
tones of increasing them to the seven discrete notes of an octave (Newton 167511757)
498 CRETIEN VAN CAMPEN
experience nowadays with light shows in concert halls. At that time, the colours of -ieht cn-stal tlasss .
the sky at dawn or sunset in a natural setting came closer to the idea that the inventors Blshop, constrL.i.:
had in mind.
'-,t a rvhite senti-¡::
The invention of gas lighting in the nineteenth century afforded new technical ¡ series of ler-e:s ::
possibilities for the colour organ. In England in the r87os, the inventor Frederick r,lQdfIS rvoUld S..: I
Kastner developed an organ that he named a Pyrophone. He used thirteen gas jets to rrstrument attr¿a:i.
-.ne tbr his horn¡ i,
The British i::,, ; r:
,he term 'colour ,--::
\errlon's idea th¿t :
;o1our spectrun i:t
rotes. The sama r, :
Peacock 1988 .
At aimost three -
,'ke a commLri :: .
$
y'=*:
(A)
Fig. 25.3 Early drawing of a design for a colour organ by Johann Gottlob Krüger, Fi1.25.4 Colour orq
"t743. the 1870s.
VISUAL MUSIC AND MUSICAL PAINTINGS
::ne, the colours of light crystal flasks (see Fig. z5.aA). A few years later an American inventor, Bainbridge
:.: :hat the inventors Bishop, constructed a machine that could be set on top of a house organ. It consisted
of a white semi-circular screen on to which coloured light was projected through
:¿ed new technical a series of levers and shutters operated by the organ player (see Fig. 25.48).Colour
-n1 entor Frederick organs would soon become very popular and the simplicity and mobility of the
- :ni¡teen gas jets to instrument attracted the attention of the famous circus master Barnum, who bought
one for his home (Peacock 1988).
The British inventor and professor of fine arts, Alexander Rimington, documented
the term 'colour organ' for the first time in a patent application in rg93. Inspired by
Newton's idea that music and colour are both grounded in vibrations, he divided the
colour spectrum into intervals analogous to musical octaves and attributed colours to
notes. The same notes in a higher octave produced a lighter shade of the same colour
(Peacock rg88).
At almost three metres in height, Rimington's colour organ (see Fig. 25.5) looks
like a common house organ with a compartment on top of it containing four-
teen coloured lamps. The light of the lamps could be controlled by the performer in
r Gottlob Krüger, Fi}.25.4 colour organs by Frederick Kastner (A) and Bainbridge Bishop (B) from
the 1 870s.
CRETIEN VAN CAMPEN
gradients of colour tone, lightness, and saturation (Peacock 1988). Rimington's The MusicalCour
colour organ represented a giant step forward from the original harpsichord and the colour organ. th,
paper strips of Castel. However, like most colour organs of that time, Rimington's onstrated his color¡l
light machine did not produce musical sound and his colour organ had to be played same year, Rimingt.
simultaneouslywith an organ that did produce musical sound. the music of\\¿agne:
To summarize. :.
.'?& r....rr.......i..r
evoke perceptions c'.
*:.)4r.. :rrr.r.:..).{:i..::.r-ry.
aii.-,....:'." .:*qt,'.3¿i*ii organs and perforrn
t,,{; . i*
teenth century (cf. P
|&w:
SE r. 'ii
Psv
Scriabin becarne
composer Rimsl¡' I
performance of a p:.
Korsakov wh€r€uDt-rl
colour. From that :::
effects of different ti:
while listening to pl
corresponded rr-ith r,
tonalityhad no colrrl
Scriabin explahe,
When the tonali¡'' ,-
the tonality; it ma.ke
this was one lvar- ir',,
couldintensifythe in
each other simultane
he heard the change :
1988). Rimington's The Musical Courier of lune 8, 1895, reported that Sir Arthur Sullivan improvised on
al harpsichord and the colour organ, though with his eyes closed. A few days earlier, Rimington had dem-
: time, Rimington's onstrated his colour organ in London for an audience ofa thousand people. In the
,n had to be played same year, Rimington played the colour organ in four more concerts accompanying
the music of Wagner, Chopin, Bach, and Dvorak (Peacock 1988).
To summarize, the age-old quest for colour-pitch correspondences in order to
evoke perceptions of coloured music had finally resulted in the construction of colour
organs and performances of coloured music in concert halls at the end of the nine-
teenth century (cf. Peacock 1988; Gage 1993; ]ewanski and Sidler zoo6).
PsycrroLocrcAL EXPLoRATToNS
Around the beginning of the twentieth century, musical performances accompanied
by displays of coloured light were given quite regularly. Since most of the technical
problems had been overcome by this time, it was the psychological questions concern-
ing the effects of these performances that came to the fore. The Russian composer
Alexander Scriabin was particularly interested in the psychological effects on an audi-
ence experiencing sound and colour simultaneously. His theory was that when the
correct colour was perceived together with the correct sound, 'a powerful psychological
resonator for the listener' would be created (Myers r9r4; Peacock rg8S; Galeyev and
Vanechkina zoor).
Scriabin became aware of his synaesthetic perceptions while sitting next to the
composer Rimsky Korsakov in a concert hall. Scriabin describes noticing that the
performance of a piece in D-minor appeared yellow to him, and he mentioned this to
Korsakov whereupon his colleague replied that the piece appeared to him in a golden
colour. From that moment on, Scriabin started to pay special attention to the colour
effects of different tonalities. He realized that he had spontaneous colour perceptions
while listening to pieces in the keys of C, D, and F-sharp. Music in the key of C
corresponded with red, D with yellow and F-sharp with blue. A single note apart from
tonality had no colour for Scriabin (Gleich 1963; Peacock rg8S).
Scriabin explained that colour played several roles in the perception of music.
When the tonality of a piece changed, the colour changed: 'The colour underlines
the tonality; it makes the tonality more evident' (Pear zooT). According to Scriabin,
this was one way in which the addition of colour projections to musical performances
could intensify the impact ofthe music. In fact, the music and the colour could enhance
each other simultaneously. Sometimes, he even anticipated the change in colour before
he heard the change in tonality (Myers r9r4; Galeyev and Vanechkina zoor).
His most famous synaesthetic work, which is still performed today, is Prometheus,
Poem of Fire (Fig. zS.6). He used key-colour correspondences for this work (Table
z5.t), and not the tone-colour correspondences which had been commonly used by
I of nineteenth colour organ performers like Rimington. Thus, only when the tonality or key of the
piece changed would Scriabin change the colour (Gleich 1963).
CRETIEN VAN CAMPEN
Alongside the other musical parts on the score for 'Prometheus', Scriabin wrote
Table i
directions for the tastiere per luce, the colour organ, treating it almost as another musi-
Alexan
cal instrument. The colour organ's part consisted of two movements, one ofwhich was
'consonant'with the music, and the other'dissonant'. His intention was to structure Key
and intensifu the listeners' experience not simply by imitating or reflecting the music
D#
F#
CL
. Scriabin wrote
'-. Table 25.1 Scheme of colour-key correspondences by
. ¡; ¿s another musi-
Alexander Scriabin
,!. Lrne ofwhichwas
I r \\ as to structure
Key
:::lecting the music
C# violet or purple
yellow
D#
sky blue
deep red
G# lilac
green
A# 9rey
light blue
1l
¡.
J
ll -t
in colour, but by giving the colour organ an autonomous role where it could point and
g il counter point at changes in the musical movement with colour changes (Gleich 1963;
Peacock r985).
:l
-t)
Meanwhile, in the visual arts, interest in the correspondences between music and
colour was raised by the concept of the painter's palette as a keyboard, which became
commonplace in the artistic parlance of the second half of the nineteenth century.
Many visual artists considered music the highest form of art and believed that music
approached the ideal of an immaterial art form (Gage 1993; Maur 1999).
The Dutch painter Vincent van Gogh (r8l¡-r8go) recounted a personal anecdote
about the music behind his own paintings. The artist took piano lessons to become
acquainted with the subtleties of colour tone. However, his elderly music teacher sent
him away soon after he noticed that Van Gogh was constantly comparing the sounds
of the piano keys with Prussian blue, dark green, dark ochre, and cadmium yellow. The
teacher thought his pupil was a madman. In his later writings, Van Gogh compared
painting with pencils to playing the violin with a bow (Uitert 1978; Campen zooT).
He was not the only visual artist in the nineteenth century to experiment with paint-
ing and music. It was common for artists to sing or whistle songs whrle they painted to
create paintings that approached a musical expression. In the second half of the nine-
teenth century, a tradition of musical paintings began to appear that influenced, among
'cr'Prometheus: others, the Dutch symbolist painter, Janus de Winter G88z-tgS4. A characteristic
of symbolism is the frequent use of musical titles. In that respect, symbolism was a
,o4 CRETIEN VAN CAMPEN
trailblazer for the synaesthetic experiments conducted by later abstract artists in the ::inciples in his r¿-r
twentieth century. S,,.und) he espen::l,
Janus de Winter exhibited works with titles like Musical Fantasy (Wagner). In :rovement (film . ::
response to a request by the psychologist Ten Haeff to describe his synaesthetic per- r9i:). Like Scri¿bi-.
cepts, de Winter wrote to him: .:,ider to intensit,' s-.
a trumpet and a brass band sounded yellow; a viola and a warm alto voice had orange ects are not rrel1 :-, '.
colours; a tuba and drums evoked red colours; a bassoon sounded violet; a cello and applying some .L::-,
an organ produced blue; and the sound of violin evoked green. He played the cello \\¡indows N'Iedia Pr.'
himself and described the dark blue colour of the instrument as the most personal to find a form th¿: :.
spiritual colour (Kandinsky r98o; Ione and Tyler zoo3).
Both Kandinsky and the Viennese composer Arnold Schónberg were interested
in the phenomenon of synaesthetic dissonance, which also intrigued Scriabin (Hahl- '
Perhaps one sh.,- :
Koch 1985; Costa Meyer and Wasserman zoo3). After Schónberg published his for zoot: A Space O,i , :::'
ideas on atonal music and dissonant harmony, Kandinsky wanted to make use of these seventeenth-centun- t:, :
their technical oppo:: -,
rvebsite http:i /Rhlth ::. :.
' There is still debate on Kandinsky's synesthesia (cf. Jewanski and Sidler zoo6). documented introdu::.
VISUAL MUSIC AND MUSICAL PAINTINGS 505
bst:act artists in the principles in his painting and theatre. In his theatre piece Der gelbe Klang(The Yellow
Sound) he experimented with the opposition of three types of movement: visual
i,tt;tsv (Wag4er). In movement (film), musical movement, and physical movement (dance) (Kandinsky
us srlaesthetic per- rgrz). Like Scriabin, Kandinsky wanted to alternate dissonance with consonance in
order to intensify synaesthetic perceptions, so they would, in his own words, have a
'deeper inner impact' (Poling 1983; Düchtingry96).
r ¿nd i-lutes vary from
red or brown violet
r:-i
Kandinsky was not the only artist at this time with an interest in synaesthetic per-
:¿ed rr'ith silvery grey. ception. A study of the art in the first decades of the twentieth century reveals in the
;rquisite green, silver work of almost every progressive or avant-garde artist an interest in the correspon-
dences between music and visual art. Modern artists experimented with multisensory
(Uitert rgZS)
perception like the simultaneous perception of matching movements in music and
film. The German art historian Karin von Maur has compiled an elaborate catalogue of
::: artists called Der the immense production of musical paintings in the twentieth century, published as
k:rorrrr as audition The Sound of Images (Maur 1999).
:rp Lrf painters, com-
'e tocused on three
'u t :,i t u tkunstw erke) ;
::;rsion ofspiritual-
i',-¿ssilv Kandinsky's
'":ri¿al in Art Qgtt),
Drcrr¿.L ExPLoRATToNS
;,:,loured hearing as
:,:caLin'to another. In the time of Scriabin and Kandinsky, the performance of sound and colour composi-
riss oi a piano, and tions did not have the technical advantages we have today. The rapid technological
¿r,nrng next to each advances in computer technology that have occurred since the middle of the twentieth
.,,',-¡:
'un ell acquainted
century have allowed for the creation of new digital instruments, and brought sound
i' ¿t that time and and colour into the same digital domain (cf. Brougher et aI. zoo5; Riccd and Cordoba
:ris of pst.chologists zooT).
From the late r95os, electronic music and electronic visual art have coexisted in
F..r erample, in a the same medium, and the interaction between these two fields of art has increased
,-¿ned that when he
tremendously. Nowadays, students of art and music have at their disposal digital
-.:c eloked a fantastic software that uses both music and visual imagery. The opportunities afforded by the
ie sia. he later began internet for publishing and sharing digital productions have led to an avalanche of
r;T üolours for him: synaesthesia-inspired art on the internet (Collopy zooo). Unfortunately, many proj-
-:,r ,,-oice had orange ects are not well thought-out. Appln"g synaesthesia to digital art involves more than
:d riolet; a cello and appl)4ng some computer algorithms that arbitrarily translate music into colours;
He plaved the cello Windows Media Player on your PC can akeady do that trick for you. The challenge is
-< :he most personal to find a form that truly reveals some aspects of synaesthesia.'
To give a fine example, Stephen Malinowski and Lisa Turetsky from Berkeley,
california wrote a software program, entitled the Music Animation Machine (MAM)
that translates and shows music pieces in coloured measures for children (http://
www.musanim.com). Children using this tool do not have to puzzle over complicated
staves, with flat or sharp symbols and key signatures, but instead see coloured bars
that scroll across the screen in time with the music they hear. The vertical positions of
the bars on the screen represent the relative pitches, while the colour may indicate
instruments or voices, thematic material or tonality.
Figure 25.7 shows a still from the MAM representing the first ten bars of |ohann
Sebastian Bach's organ work, In Dulci Jubilo. Bach loved to employ complex mathe-
matical structures in his music which are not always easy to distinguish by ear. The
piece contains a double canon, which means the same melody is repeated with a time
Fig.25.7 (See Colo,
Music Animation l,/
shift. In the animation, one can see that the dark green pattern is followed by the light
Bach's organ work I
green pattern, while the violet-blue follows the red pattern. By synchronizing them, the
sound and the image are easily linked in our perception. Musical structures within
Bach's canons or his many-voiced compositions maythus be made more accessible by
a visual aid.:
Many software devices have been developed, not only for educational purposes
but also for use in digital art, which show visual aspects of sound, sound aspects of Covrpen
images, and enhance the perception of sound andvision in an integrated and meaning-
ful way. several programs, such as MAM, seem to serve as extensions of the senses,
The current n۟rL-rSs
affording a better awareness of the physical world in which sounds may have colours,
views discussed earl
as Aristotle and his students had already asserted.
number of points.
In summary, I have presented three episodes in the history of artistic experiments
First, one shoulc
and three respective models of synaesthetic multisensoryperception. The first episode
synaesthesia is mar
consists of mathematical exploralions of the senses in order to reveal how the human
mind is tuned to universal harmonies, which in the end results in a verypractical device: Ward in this r-olur
the colour-organ. The second episode consists of psychological explorations of the of a theatre, a mu,i€
Cordoba zooT '.
senses in order to reveal introspective regularities and deeper meanings of perceived
intermodal perceptions, resulting in a spiritual modern abstract art. And the last epi- Second, the ob'e;
nisms of multisen.;,
sode shows digital explorations of the input of the senses in order to find regularities in
is to develop instru::
the man-machine environment, resulting in digital software and systems that'extend'
the reach of the human senses. The distinctions between the episodes are, of course, very different tron I
less sharp than I have presented them here. Thus, the quest for synaesthesia, i.e. or measure srn¿esl:
coloured hearing, appears to be threefold: slmaesthetic perceptions could reveal developed to evok¿
(r) harmonies in the universe; (z) deeper spiritual meanings in the human mind; and ]ewanski 1999).
(3) extended perceptions of the human senses.
a Neuropsycholor:: -
:u;:.tional purposes
l. :ound aspects of Covrp¡.RrsoN To cuRRENT NEURoscrENcE
:::eci and meaning-
1-r;.-';-; gf the senses,
The current neuroscientific viewa on synaesthesia has some overlap with the artistic
.s :ral have colours,
views discussed earlier, but also some major differences. I shall compare them on a
number of points.
:::lstic erperiments
First, one should consider the different contexts ofinvestigation. In neuroscience,
,¡. The hrst episode
synaesthesia is mainly a brein phenomenon (cf. Ward and Mattingley zoo6 and
¡.1 :lol- the human
Ward in this volume), while in the arts, slmaesthesia is a phenomenon on the stage
::-.:racticaldevice:
of a theatre, a museum, or digital community (cf. Brougher et al. zoo5; Riccó and
¡:,:riorations of the
Cordoba zooT).
:ii:rqs of perceived
Second, the objective of neuroscience is to test theories of mind and the mecha-
r -. -r:rd the last epi-
nisms of multisensory perception (cf. Emrich et al. zooz).In the arts, the main goal
: ird regularities in
is to develop instruments and devices for public performances. These instruments are
,-r:i:nS that'extend'
very different from the instruments in scientific research that are developed to assess
:,aes are, of course,
or measure synaesthetic responses in individuals; in the arts the instruments are
: sinaesthesia, i.e.
developed to evoke synaesthetic perceptions in a large audience (cf. Peacock 1988;
--¡'1. could reveal
::rurnan mind; and Jewanski rggg).
a Neuropsychological research into spraesthesia (see reviews in Baron-Cohen and Harrison r997;
Ramachandran and Hubbard zoor; Cltowic zooz; Robertson and Sagiv zoo4; Ward and Mattingley
zoo6) is relatively young in comparison with artistic explorations of the phenomenon. Halfinray
through the r98os, scientilic research into synaesthesia experienced something of a revival when new
-- ,r¿t,i-r.ear-old brain scanning techniques and psychological consistency tests became available. Synaesthetes had not
-r, . . tiv bt' the been taken seriously for a iong time, but when physiological and psychological proofwas given for
:1 :
: ¿en drawing and synaesthesia it soon became a serious scientific and popular subject of research. Note that coloured
:r -.:;¿i pieces and are hearing and synaesthesia had been studied extensively by scientists around the end of the nineteenth
century (cf. Marks 1978; Campen ry99, zooT).
5O8 CRETIEN VAN CAMPEN
Third, the selection of the forms of synaesthesia to be studied differs. Scientific :n the arts as 'tl:3
research into synaesthesia in the last decades has focused mainly on the perception of
erperience'.'
coloured letters and numbers (graphemes) (cf. ward and Mattingley zoo6). on the To distLnguish r!
other hand, artists have focused on different forms of synaesthesia: visual music, the terms 'neu¡a- s
musical paintings, coloured music, and have mostly been concerned with relating the
reier to the sa¡¡e . b
visual arts and music (Gage 1993; von Maur 1999; Brougher et al. zoo5; Riccd and ihat scientists ani .
Cordoba zooT). ¿re related rr-ithü: :
Fourth, given these different qpes of synaesthesia that are the focus of scientific and ronr ard a hrpc-t-Jres
artistic investigations, different typical characteristics of synaesthesia have been Ilv thesis, o¡ ¡.-,
studied: more common fal
' While scientists started with a strong causal model of unidirectional synaesthesia non-s\naesthet.s :
(e.g. a sound evoking a colour image) (Baron-Cohen and Harrison 1997), artists sense) or abrlin':.-, i
started from a simultaneous model in which several sensory domains influence :ooz; Ilarks t9-S
each other at the same time (cf. Maur 1999; Collopy zooo).5 colour than a t-lut; :
' Another assumption of the scientific model was that synaesthesia is idiosyncratic, late paintings b,. l1;
strictly individual; e.g. the letter A can be consistently blue for one synaesthete But then, rt'ha: ¡
and consistently red for another synaesthete (cf. Dufft zoor). To the contrary, art- sia'? A neural s,,r:e
ists used a model of synaesthesia as a universal trait of perception (cf. Gage ry99; results in a multr.¿:
Galeyev and Vanechkina zoor). Practically, an idiosgrcratic model would not be colour in a pei.,-,:
feasible for public performance.u synchronous sti:r':
' While medical research started from the assumption that synaesthesia was a result perception. Fo: ilt
of a brain impairment (using metaphors like miswiring and leakages between brain animation on ¿ );:r
sections) (cf. Baron-Cohen et al. ry93), artists have always considered synaesthesia (cf. MAlt,f, Fig. :; -
as a talent or a gift, a state of consciousness one could become aware of (cf. Duffy Figure 2j.8 shr',.
zoor; Campen 2oo7; Dittmar zooT). Fortunatel¡ synaesthesia is considered a percep- naesthesia' in hr¡::t
tual ability by most slnaesthesia researchers nowadays, after conducting interviews exampies. Fig.:;.:;
with synaesthetes who did not feel handicapped at all by this 'impairment'.2 video clips in qen.:
understand murt; ;.
The concept of synaesthesia in neuroscience has a different background and is When a colou:::
defined slightly differently from that in the arts. In contrast to the classic neuroscien-
colours and tbrn.
tific definition of synaesthesia as 'a stimulation of one sensory domain leading to field in the smile,,'
a perception in another sensory domain' (Baron-Cohenry97), or in a recent defini-
in 'sprchronesthÉ!1"
tion as'the elicitation of perceptual experiences in the absence of the normal sensory perception (blue r::
stimulation' (ward and Mattingley zoo6),I would define the concept of synaesthesia
ceptions (= a dark ¿:
say one is produci:
5 By the way, it is interesting to observe that scientifrc modeis
of spraesthesia are being adjusted
on the basis ofreports ofbidirectional syraesthesia by synaesthetes and artists (Cohen Kadosh et a1. 8 Iconsiderages:._:
zoo5; Cohen Kadosh and Henik z,oo7).
6 AIso in this respect, scientists meaningful confisu:"::
have moved more to the artistic model, since several studies have
Merleau-Ponlr'. \\-;:-L=
been carried out to find universals in, for instance, colours ofletters (Day uoo4).
thought that can bc ;;-:
'Z In line with the reports on spraesthesia as an ability, 'creativity' has become a current theme in Marks, Cltoüc, an¡ l:
scientific research. The prevalence ofsl'naesthesia appears to be higher among persons in creative e Artistic and scie¡:r'
professions than in the general population (Rich et al. zoo5; Ward et al. zoo8). Many artists that
I have interviewed seemed to consider synaesthesia as self-evident and as another one of their
consistently matchi:rs i:
spraesthetes impliciJr :
tools in their artistic toolbox.
them (Marks 1978: h ::i
VISUAL MUSIC AND MUSICAL PAINTINGS 509
¡o drfrers. Scientific in the arts as 'the simultaneous perception of two or more stimuli as one gestalt
¡n the perception of experience'.t
:rgler- zoo6). On the
To distinguish the neuroscientific and artistic definitions of synaesthesia I will use
hesia: visual music, the terms 'neural synaesthesia' and'synchronaesthesia'. The two definitions do not
red rrrth relating the refer to the same, but rather to two closely related phenomena. Both the phenomena
a1.':oo5; Riccd and that scientists and artists study are real perceptions. I suggest investigating how they
are related within the framework of multisensory perception. As a start I would put
,.-u-i of scientific and forward a hlpothesis.
.sthesia have been My thesis, or conjecture, is that 'neural synaesthesia' is a particular form of the
more common perception of 'synchronaesthesia'. Many people, synaesthetes and
;lt-tnal synaesthesia non-synaesthetes alike, have a common sense (seísus communis in the Aristotelian
.:ison 1997), artists sense) or ability to match stimuli from different sensory domains (cf. Adler andZeuch
d..mains influence zooz; Marks 1978). One can tell that a double bass corresponds better to a darker
colour than a flute and one can relate the typical Boogie Woogie rh¡hms better to the
:s-, is idiosyncratic, late paintings by Piet Mondrian than to colour field paintings by Mark Rothko.e
::: ¡ne synaesthete But then, what's the difference between'neural synaesthesia' and'synchronaesthe-
-,-, sia'? A neural spraesthetic perception is the result of a stimulation of one sense, which
:ie contrary, art-
i-,-:r ct-. Gagery99; results in a multisensory perception. For instance, a sound evokes a coperception of
:-,¡ei rt'ould not be colour in a person. A synchronaesthetic perception is considered the result of a
synchronous stimulation of two or more senses, resulting in a multisensory gestalt
:ith.sia u'as a result perception. For instance, a simultaneous perception of a melody and a visual
i.=.es befir'een brain animation on a screen evokes the feeling of a match of music and image in a person
,;¿red sr.naesthesia (cf. MAM, Fig.z5.7).
:',r ¿r€ of (cf. Duffy Figure 25.8 shows a schematic comparison of 'neural synaesthesia' and 's1'nchro-
,-- isrdered a percep- naesthesia' in human information processing. I will explain the scheme with two
r"]'ucti¡g interviews examples. Fig. 25.8 can illustrate, as a first example, why synaesthetes do not like music
:::¡irment'.t video clips in general, and as a second example, why children (and adults too) can
understand music animations without being synaesthetic.
¡¡¡,<ground and is When a coloured hearing synaesthete hears music (the red arrow) his brain adds
-.:ssic neuroscien- colours and forms (blue arrow), so he experiences music in colour and form (purple
:-inain leading to field in the smiley). When he watches a music video clip, his senses are stimulated like
' -:: ¿ recent defini- in 'synchronesthesia' (blue and red arrows). But his brain is already producing a visual
:i< normal sensory perception (blue arrow). So on top of his musical perceptions he gets two visual per-
:e :: of srnaesthesia ceptions (= a dark and a light blue arrow). If the blue arrows interfere with each other,
say one is producing a green colour and the other a purple colour, the synaesthete
: l. :.:ng adjusted
-::¿: Kadosh et a1. 8 I consider a gestalt perception as a direct (almost pre-conscious or pre-intellectual) perception of
meaningfui configuration in the physical environment. This is in line with theories by, among others,
:.r'¿;ai studies have Merleau-Ponty, Wertheimer, Lipps, beginning with the aesthetics of Baumgarten-a tradition of
thought that can be retraced in the work of current thinkers on synesthesia like Ramachadran,
: -: :-trrent theme in Marks, C¡owic, and Emrich (Campen zooT).
:,:::a:ts in creative
i,l::.r artists that ' Artistic and scientific experiments have shown that non-synaesthetes are also capable of
consistently matching stimuli from different sensory domains, which suggests that many non-
.: :re oftheir synaesthetes implicitly prefer these sensory correspondences while synaesthetes explicitly perceive
them (Marks 1978; Ward ef al. zoo6; Galeyev zooT).
510 CRETIEN YAN CAMPEN
Synesthesia RErEnEucr s
fi
MA.
¡b.
ffiffiffie nffiffiffie, Cohen Kadosh R ¡::
Trends in Cogti:::..:
@. we"
t''v¡ Cohen Kadosh R. S.-
blue is larger th.::
Co gnitiv e t\'e rr ¡¡,.,':.
strmulus: perr:eptual LrerceFtion: CollopyF (zooo . [-,
? moelalities processíng I mod¿rlities experlenrte 33,355-60.
Collopy F. Rhrthni. _
Fig. 25.8 Schematic comparison of neural synaesthesia and synchronaesthesia in Costa Meyer E da .-:
human information processing. Scala, Nerv for^.
(with permission from Robert Hulsman). C¡owic RE (zoo: "
and vision to perceive a match. For instance, first they only hear a funny melody (reá Holt, Nerr,Yorf:.
arrow), that is going up and down in pitch. They find it hard to follow and do not Emrich HM, Sch:r
understand well what is happening. Then a video (blue arrow) is played synchronous Stuttgart.
with the music. They see coloured marbles going up and down the screen in the Evers F (1995). \{ut-..
Muziekpsychoi;, :: :.
rh¡hm of the melody. Their brains connect the melody and the visual movement -.
RnrnnnNcns
Adler H and Zeuch U (zooz) . Syntisthesie: Interferenz, Transfer, Synthese der Sirene. Kónigshausen
and Neumann, Würzburg.
Baron-Cohen S and Harrison IE, eds. Q99). Synaesthesia: Classic and contemporary readings.
Blackwell, Cambridge.
Baron-cohen s, Harrison |, Goldstein LH, and wyke M (1993). coloured speech perception:
\-/ Is synaesthesia what happens when modularity breaks down? perception, zz,
4t9-26.
Brougher B, strick l, wiseman A, and zllczer f, eds. (zoo5). visual Music: synaesthesia in Art
el\:e¡ienCe and Music since 1goo. Thames and Hudson, London.
Campen C van (1999). Artistic and psychological experiments with s1'nesthesra. Leonardo,3z,
9-r4.
Campen C van (zooz). The Hidden Sense: Synesthesia in Art and Science.MlT Press, Cambridge,
MA.
Cohen Kadosh R and HenikA (zoo). Can synaesthesia research inform cognitive science?
Trends in Cognitive Sciences, t(|, y7-84.
Cohen Kadosh R, Sagiv N, Linden DE), Robertson LC, Elinger G, and Henik A. (zoo5). When
blue is larger than red: colors influence numerical cognition in spresthesia. lournal of
Cognitite Neuroscience, ry ,lt), ry66-7.
Collopy F (zooo). Color, form, and motion. Dimensions of a musical art of light. Leonardo,
33,355-60.
Collopy F. Rh¡hmic Light (website). http://rh¡hmiclight.com/index.html
¡chronaesthesia in costa Meyer E da and wasserman F, eds. (zoo3) . schoenberg, Kandinslcy and the Blue Rider.
Scala, New York.
' ,. :- '':- Robert Hulsman). C¡owic RL (zooz). Synaesthesia: A tJnion of the Senses.MIT Press, Cambridge, MA.
Day SA (zool. Trends in Synesthetically Colored Graphemes and Phonemes - 2oo4 revision.
http://home.comcast.net/-sean.daylTrendszoo4.htm
:. s¿me time. Hewill Dittmar A (zoo7). Synásthesien. Roter Faden durchs Leben? Die blaue Eule, Essen.
Düchting H (rqq6). Farbe am Bauhaus. Synthese und Synasthesl¿. Gebr. Mann, Berlin.
-:l:i:l¿tion of hearing Duffy PL (zoor). BIue Cats and Chartreuse Kittens. How Synesthetes Color their World. iF'enry
. i;nnr- melody (red Holt, New York.
:.- :oilo\v and do not Emrich HM, Schneider U, and ZedIer M (zooz). Welche Farbe hat der Montag? Hírzel,
Stuttgart.
:-¿r ed synchronous
Evers F (t995). Muziek en de eenheid der kunsten. In F Evers, M Jansma, and B de vries, eds.
r:r the screen in the
Muziekpsychologie, pp. 313-38. Van Gorcum, Assen.
::e iisual movement Ferwerda R and Struycken P (zoor). 'Aristoteles' oyer kleuren. Damon, Budel.
r¿ibles match with Franssen M (t99r). The ocular harpsichord of Louis-Bertrand Castel: The science and
:. ir-hsn for instance aesthetics of an eighteenth-century cause célébre. Tractrix, 3, 15*77.
:r niss the match. Gage | (1993). Colour and Culture. Practice and Meaning from Anticluity to Abstraction. Thames
.,rchronaesthesia'. If and Hudson, London.
Gage J (rqqq). Making sense of colour. The synaesthetic dimension. In I Gage, ed. Colour and
-','. s.,11".rr¡.tic chil-
.; :: rr ith the aid of a meaning. Art, Science and Symbolism, pp. 261-8. Thames and Hudson, Oxford.
Galeyev BM (zoo7). The nature and functions of synesthesia in music. Leonardo, 4o(3),
:::r ¿hr avs available.
285-8.
:sla. I n'ould suggest Galeyev BM and Vanechkina IL (zoor). Was Scriabin a synesthete? Leonardo,34,357-6t.
rer.eptual phenom- Gleich CCJ von (1963). Die sinfonischen Werke von Alexander Skrjabin. Creyghton, Bilthoven.
b'u' srnaesthetes is a Gouk P Q'999). Music, science and natural magic in seyenteenth-century England.Yale University
'
: a:-Ld image. Press, New Haven, CT.
5t2 CRETIEN YAN CAMPEN
Hahl-Koch I (rSSS). Kandinsky, Schónber und der 'Blaue Reiter'. In K von Maur, ed. Vom
Klang der Bilder. Prestel, München.
Harrison I (zoor). Synaesthesia: The Strangest Thing. OxfordUniversity Press, Oxford.
Ione A (zoofi. Kandinsky and Klee: chromatic chords, pollphonic painting and synesthesia.
Journal of Consciousness Studies,11, 148-58.
Ione A and Tyler CW (zoo3). Was Kandinsky a synesthete? Journal of the History of the
N euro sciences, tz, zz3-6.
Jewanski I (ISSS). What is the color of the tone? Leonardo,32,227-8.
Jewanski I and Sidler N, eds. (zoo6). Farbe - Licht - Musik. Synaesthesie und Farblichtmusik.
Peter Lang, Bern.
Kandinsky W (r9rr). Ueber das Geistige in der Kunst. Bentelli, Bern.
Kandinsky W (r9rz). Der gelbe Klang: Eine Bühnenkomposition von Kandinsky. In
W Kandinsky and F Marc, eds. Der blaue Reiter. Piper, München.
CF]
Kandinsky W (r98o). Die gesammelten Schriften, HK Roethel and J Hahl-Koch eds. Benteli,
Bern.
Marks LE Q97s). The Unity of the Senses: Interrelationships Among the Modalities. Academic
A]
Press, New York.
Maur K von (1999). The Sound of Painting. Prestel, München.
Myers C (r9r4). A case of synaesthesia. British Journal of Psychology, 6, zz81z.
Newton I Q675h757). An hypothesis explaining the properties of light. In T Birch, ed. The
History of the Royal Society, Vol 3, pp. 247-305. London.
Peacock K (rq8S). Alexander Scriabin's color hearing. Music Perception, z, 483-5o6.
Peacock K (r988). Instruments to perform color-music: Two centuries of technological experi-
mentation, Leonar do, zr, 397- 4o 6.
Pear TH (zoo7). Remembering and Forgetting. Church Press.
Poling CV (t9g). Kandinsky - lessen aan het Bauhaus: kleurentheorie en analytisch tekenen.
YouTune , the r-iie',,
Cantecleer, De Bilt.
Ramachandran VS and Hubbard E (zoor). Synaesthesia. A window into perception, thought is one of the latest :r
and language. J ournal of Cons ciousness Studies, 8 (rz), 3-34. online laboraton'-,::
Riccó D and Cordoba M| de, eds. (zoo7). MuVi. Video and moving image on synesthesia the song'Here ít g,: e
and visual music. Edizioni Poli design, Milano. bers can be seen d¿::
Rich AN, Bradshaw fL, and Mattingley IB (zooS). A systematic, large-scale study of s1'naesthe- As a matter of tact. l--
sia: Implications for the role of early experience in lexical-coiour associations. Cognition,
in which the band n
98, f-84.
spawnednumero;i -
Robertson LC and Sagiv N, eds. (zoo4). Synesthesia. Perspectives from Neuroscience. Oxford
Press, New York.
ers to post their r.inr:
Uitert E van (1978). Beeldende kunst en muziek: de muziek van het schilderij. In C Blotkamp, The fact that a ,, -:
ed. Kunstenaren der idee: symbolistische tendenzen in Nederland. Haags Gemeentemuseum, million people hai.: .
Den Haag. it in its entiren'. St:-1.
Vanechkina IL (rgg+). Musical graphics as an instrument for musicologists and educators, with far less vierr's.
Leonardo,27, 437-9. OK Go on YouTul¡"
Ward J and Mattingley IB (zoo6). Synaesthesia: an overview of contemporary findings and
ofthe song, perha;!
controversies. C o rtex, 42, t2916.
more than 4o miilii':
Ward J, Huckstep B, and Tsakanikos E (zoo6). Sound-colour synaesthesia: To what extent
does it use cross-modai mechanisms common to us all? CorteS 42,264--gc.. not just the song !h:
ward I' Thompson-Lake D, Ely R, and Kaminski F (zoo8). synaesthesia, creativity and art: the video is basicalt','
What is the link? British lournal of Psychology, 99,727-4t. might explain the', r;
watching the dance.
li r-on Maur, ed. Vom
i':ess, Oxford.
:tirg and synesthesia.
CHAPTER 26
, :,t' the Hktory of the
; ¿ t,t¡'l F arblichtmusik.
DANCE,
r'¡n Kandinsky. In CHOREOGRAPHY,
:r-Koch
l, ! :
eds. Benteli,
i,ilities. Academic
AND THE BRAIN
f Birch, ed. The IVAR HAGENDOORN
-. _ )vv.
-
: :.;:uological experi-
human capacities such as thinking, perceiving, and feeling possible. Elsewhere I have motion graphics sr;,
argued that for this reason psychology and cognitive neuroscience can be said to have and blurred pencrl-::
a critical dimension (Hagendoornzoo4a). Like structuralism and post-structuralism,
took care ner er ::
rve _
they reveal some of the processes that underlie human behaviour and the cultural When our stick ñe,r:.
artefacts it produces. back together asarn ,::
As a philosopher I am aware of the limitations of cognitive neuroscience. Functional human motion und.:-r
neuroimaging studies only reveal that a certain brain region or network is activated in
an experimental task, but this doesn't say anything yet about the contribution of this
During the perlc::
region to execution of the task and the underlying causal relationship. The relationship
parent scrim covg¡11;
between functional magnetic resonance imaging (fMRI) signals and neural activity is
it. The visual eltect :.
also far from resolved (Heeger and Ress zooz). Neuropsychological studies may show
visual image (Fie. :¡.r
that damage to a particular brain region impairs execution of certain tasks, but the
Neuroimagine s:r:.
extent of the damage can be difficult to determine. If a certain brain region is activated
the superior tempc,::
when healthy individuals perform a task and if damage to that region impairs perfor-
logical motion. \e i:
mance at the same task, we may conclude that this region is an essential component
to brain regions lt..
of the network subserving this task, but we still don't know exactly what it contrib- -
recognizing motion .,
utes. These are but some of the methodological problems that one should be aware of
when extrapolating findings from cognitive neuroscience to other fields such as the
In another experint.,:
arts. If we piece together findings from different studies, a picture may emerge of what
goes on inside the brain, when people perform a wide-ranging task such as watching
' The production is p.:
a dance performance and of how this affects and constrains perception, emotion, and regularly performed c:,,
judgement. http://www.mkz.com.
DANCE, CHOREOGRAPHY, AND THE BRAIN 515
rs 1t is an important Some videos on YouTube, especially illegal recordings of dance performances filmed
, rt'eddings, victories, with a mobile phone from the rear of a theatre, are heavily pixelated and yet we have
ldia, Cambodia, and no difficulty discerning the dancers and may even take pleasure in their movements.
reached great refine- It is quite remarkable how little information is needed to detect a human body and
l9;) and social func- a human body in motion. From an evolutionary point of view it makes sense to be
rs to self-expression able to quickly detect a moving body and recognize conspecifics. It is therefore not
fieatre, but it wasn't surprising that recent experiments have pointed towards the existence of at least two
:ndent artform. The areas in the human brain that are selective for the visual presentation of human bodies
:iion of dance styles (Peelen and Downin g zooT).
le movements in the In the earlyrgTos, followingin the footsteps ofthe nineteenth-centuryFrench scientist
and pioneer of photography and cinematography, Étienne-fules Marey, the Swedish
lfuter screen receive psychologist Gunnar fohansson filmed an actor as he walked about in a darkened room
'ilresis that watching with lightbulbs attached to some key joints. Still frames from the resulting video show
reople glued to their nothing but a random collection of dots, but when the üdeo is played, the moving
.rle about on a stage? actor instantlypops out from the screen. fohansson's experiment has since been repro-
,ns has intrigued me duced in a wide range of settings, with the actor performing all kinds of actions, the
!\\-er to this question markers placed on other parts of the body, and with animals instead of a human actor
uroscience literature (Blake and Shiffrar zooT).
'ioring, thrilling, or rn Biped Gggg), a ballet by Merce cunningham with üsual décor by multimedia art-
,:r. This is hardly an ists Paul Kaiser and Shelley Eshkar, a similar technique is deployed to great artistic effect.
,nd cognitive neuro- At the time of the rehearsals some movement phrases were recorded using motion-
:rocesses that make capture, a technique whereby sensors track reflective markers placed on the body. Using
¡le. Elsewhere I have motion graphics software the data files were transformed into dot figures, stick figures,
e can be said to have and blurred pencil-drawn figures. As Paul Kaiser explains:
Ipost-structuralism,
we took care never to lose the underlying perception of real and plausible human movement.
rur and the cultural When our stick figure leaped, its various lines were flung upward in the air, then gathered
back together again on landing. While no human body could do this, you could still feel the
-..cience. Functional human motion underlying the abstraction.
n';ork is activated in (Kaiser zoor)
crrntribution of this
:l,ip. The relationship
During the performance the final animation sequences were projected on a trans-
parent scrim covering the entire front ofthe stage, with the dancers performing behind
:¡d neural activity is
:a-l studies may show
it. The visual effect is breathtaking, as real and virtual dancers blend together in one
visual image (Fig. z6.r).'
ertain tasks, but the
n region is activated
Neuroimaging studies have implicated one particular region, the posterior part of
the superior temporal sulcus (sTS), in the perception of what has been termed bio-
rit¡n impairs perfor-
rssential component
logical motion. Neuropsychological studies report the case of patients with damage
rvhat it contrib- to brain regions involved in motion perception, but intact STS, who have difficulty
-t1,.'
recognizing motion, yet are still able to perceive biological motion (Vaina et al. zooo).
eshould be aware of
er trelds such as the In another experiment, temporarily disrupting cortical activity in the posterior STS
rx¿r,'emerge ofwhat
isk such as watching
' The production is part of the repertory of the Merce cunningham Dance company and is
r.r:t0n, emotion, and regularly performed on tour. A video recording is avaitable on DVD from French label MKz,
http://www.mkz.com.
5r.6 IVAR HAGENDOORN
regions involved in h
response (Ramachar
neuroimaging stud:c
and that the repor-:e,
the case of biologlca
information is needo
do not present a hr:,t
already constitutes ¿
In the remai¡ri¡s:
reography and son.ie
how each may illur:
neuroscience mar- aC
choreographer creetu
various decision m,--:
cesses. For budgeta:-;
that we will create e s,:
We will starr iror:
I will not concern ::
dance there is no-.:r::
Fig. 26.1 Merce Cunningham, Biped ('1999). Projected onto a scrim pencil-drawn movements. \\-hat s i
figure of a dancer. In ry57 the Americ::
(Photograph : Stephanie Berger.) rrüich he and ano:l..
Louis Horst responi,
square inches of r.¿:
\lcCormick 20o_r. i.:
using a technique called transcranial magnetic stimulation,'resulted in impaired rec-
ognition of biological motion displays (Grossman et al. zoo5). Interestingly, another
Let us suppose ú:
fbrming at alterna::i
study demonstrated that the posterior STS is also activated when people listen to
choice. Theatres h¡','¡
footsteps (Bidet-Caulet et al. zoo5). These findings suggest that the STS plays a central,
empty canvas. -\lthL:, *
cross-modal role in the perception of human motion.3
b1.,theatres are rela:ri
Some years ago, the neuroscientist Vilayanur Ramachandran and philosopher
distract the audienc¿ I
William Hirstein advanced eight principles of aesthetic experience derived from
Most of the petrpl;
Gestalt psychology and various properties of the üsual system (Ramachandran and
have seen a flr-er or r¿
Hirstein 1999). The main principle is what they refer to as a peak shift effect, a phe-
others may har-e bee:r
nomenon known from studies of animal behaviour, according to which the response
review. All of this rrrlr
to exaggerated features is stronger than to average features. This hypothesis can be
Experimental psr cr"-,.
illustrated with reference to caricatures and representations of women in Indian art
mation in generai. a:
and popular culture. In an interview, Ramachandran speculated that motion cap-
ture displays of biological motion would serve as a hyper-optimal stimulus for brain
, I should add th¿1. :¡ .
to other goais. A cht :e _ ::
' Transcranial magnetic stimulation, often abbreviated as TMS, is a technique whereby a brief but material to this el1ec¡. ¡.::
strong electric pulse is applied through a coil placed on the head, creating a magnetic field, which and principles, l-hethe: _:
temporarily disrupts local neural activity. t The piece is remirr...
3 It should be noted that the STS has also been associated with the perception of facial expression piano without touchils :
and eye gaze (e.g. Engell and Haxby zooT). of white paintings, rr¡1¡¡,
DANCE, CHOREOGRAPHY, AND THE BRAIN 5r7
regions involved in human motion perception and, as such, would generate a stronger
response (Ramachandran 2oo1). It is interesting to observe that the stimuli used in
neuroimaging studies are usually presented in isolation in highly controlled settings
and that the reported actiüty may therefore akeady represent a peak shift effect. In
the case of biological motion studies, though, the original idea was to show how little
information is needed to still observe motion. Biological motion displays therefore
do not present a hyper-stimulus but a hypo-stimulus. As I would argue, dance in itself
already constitutes a peak-shift effect in human motion perception.a
In the remaining part of this essay, I will juxtapose some aspects of dance and cho-
reography and some insights from cognitive neuroscience and psychology. I will show
how each may illuminate the other, whereby the emphasis will be on how cognitive
neuroscience may add to our understanding of dance. I will take the perspective of a
choreographer creating a dance performance. I will deconstruct the creative process into
various decision moments and will show how they can be related to various brain pro-
cesses. For budgetary and artistic reasons, but also to keep things simple, I will assume
that we will create a solo.
We will start from scratch. At this moment an1'thing is possible. For this reason
I will not concern myself here with a definition of dance. In much contemporary
dance there is nothing perceptible to differentiate dance movements from everyday
{rrm pencil-drawn movements. What's more, dance performances need not involve any movement at all.
In ry57 the American choreographer Paul Taylor performed a piece called Duet, in
:' :.'.:" . Stephanie Berger.) which he and another dancer stood and sat still for four minutes.5 The dance critic
Louis Horst responded with a review inthe Dance Observer, which consisted of four
square inches of blank space with the initials 'L.H.' at the bottom (Reynolds and
McCormick zoo3, p. 383).
:ei ln impaired rec-
Let us suppose that our performance is taking place at a theatre. Even though per-
iterestingiy, another
-,.r people listen to forming at alternative locations may be interesting, there are good reasons for our
choice. Theatres have all the necessary facilities to stage a performance. They are like an
: !TS plays a central,
empty canvas. Although in some theatres you can hear cars or the underground passing
by, theatres are relatively closed offfrom the outside world, thus limiting whatever may
n ¿nd phiiosopher
distract the audience from watching the performance.
:¿rce derived from
Most of the people who attend the performance will have paid to see it. They may
R.¡nachandran and
have seen a flyer or have heard about it from friends. Some maybe interested in dance,
.l .hilt etTect, a phe-
, -,"'rich the response others may have been dragged along by their partner, and some may be paid to write a
review. All of this influences how people perceive and will remember the performance.
,s }:rpothesis can be
Experimental psychologists use a technique called priming to study how prior infor-
r'-,r:l-'n in Indian art
mation in general, and unconscious attitudes in particular, bias people's perception
,i rhat motion cap-
'' stmulus for brain
a I should add that, as I have argued elsewhere (Hagendoorn zoo5), this line ofreasoning extends
to other goals. A choreographer wishing to confuse the audience would have to organize his or her
: -. r, i:erebr. a briefbut materiai to this effect. Actual dance performances, however, are always a mixture of different goals
::.,:.:i¡ neld, which and principles, whether implicit or explicit.
5 The piece is reminiscent of John Cage's
4'jj" (1952), in which the performer sits behind his/her
.:: --icialexpression piano without touching a single key for the duration of the piece, and Robert Rauschenberg's series
of white paintings, which he painted in r95r.
518 IVAR HAGENDOORN
and judgement. This knowledge can be put to creative use and I would like to contend However, it is a bl
that, unconsciously, this is what many artists do. dancers we couid ca:
In a performance setting almost everything can be controlled and that means that might actuallr- be i:
almost everithing involves a decision. Before the audience take their seats, we already just let her stand l:e:
have to make a decision. We could have the performer take the stage before the doors our dancer to taik. si
open, so that, upon entering, the audience has a sense they have to hurry so as not to a way as to keep rhe
miss something. This is how the American choreographer William Forslthe begins there are no eas\- ¿1,
many of his performances. Let us assume that for our piece the lights in the audience and choreograph,,' ;,,
are on and that the stage is closed offfrom the audience by a black curtain. attempt to ansrr er ::
When the audience has entered and taken their seats, at some point the lights in ence's attention. Fie:
the audience are dimmed. This signals to the audience that they should stop their or deviates from a:l r
conversations and switch off their cell phones, because the performance is about to In the work of cr,
begin. Dimming the lights in the audience also removes some more potentially dis- at any momentther;
tracting elements in the form of the visible shape of other people in the audience inglyrandom groupr
and emphasizes the division between audience and stage, between the real and the action, which dont::
imaginary. Some choreographers like to confound the audience's expectations by The audience has r¡ ¡
keeping the lights on until well into the performanceó or by transgressing the imagi- tion (Fig z6.z). This ,
nary boundary between stage and audience, for example by having the performers show that there is ¿ :
mingle with the audience. But we will refrain from such experiments. track simultaneou.i',
With the lights in the audience dimmed we can, overtly or covertly, open the cur- Attention is th¿ ::
tains. Now we face another decision. We could switch on the stage lights to reveal the rvhile reiaying e\ er-! .
dancer, who can be either just standing there or already dancing, in which case the down and botton--
audience again would feel as if it has dropped in on a conversation, or we could keep knows that sihe is ; :
the stage empty. This is what we will do for our present piece. The audience will be On the other hanc.
waiting, curious as to what is going to happen. They will be on high alert. In neural shows that attenti¡:
terms the baseline activity of neurons in visual, auditory, and somatosensory cortices down control anci .:i
will be raised (Voisin et al. zoo6). may therefore e\¡e:
When at some point, not surprisingly but still unexpectedly, the dancer walks on dance form lvhich c,:
stage, all eyes will be focused on him or her. The moment the dancer takes the stage an important role. ¿s
the audience will notice whether the dancer is male or female. We could postpone wander or because
'-,:
this moment and put it to creative use by obscuring the dancer's body and unveiling tilting of the heaci ci
it later. Let's assume that our dancer is female and that she is simply wearing a non- Salient stimuli t¿i
obtrusive pair of jeans and a plain tank top. There is a reason the costumes in dance group of men, she ,.",..
tend to be simple. The lavish costumes in classical ballet notwithstanding, anything too if she is wearing a ."-
sumptuous may distract from the movements. The dress will stanc -
Now that the dancer is on stage and has captured the audience's attention something In many dance pe :-,-,
has to happen. We could have our dancer stand there for a few minutes and then walk the work of Germ¿i
off stage again.T We could call this short conceptual piece Breathing. Never mind that i just gave is, in iac. :
Paul Taylor has done something similar. Most people in the audience won't know. grapher and pionee:
oftenwearwhite par:
u A famous example is Steptext (t984) by William Forsl'the.
7 The German theatre director Einar Schleef (t9++-zoot) began his production of Oscar Wilde's This relates to \\'r.:
Salome (t997) with a tableau vivant of the entire cast facing the audience for about 15 mínutes in
utter silence, interrupted only by the protests ofsome people in the audience and the sounds of 8 Glass Pieces (r9.S_: :r
people coughing, laughing, and leaving. endless stream of dan;¿-.
DANCE, CHOREOGRAPHY, AND THE BRAIN 519
u ouid like to contend However, it is a bit cheap and we should be able to do better than this. With more
dancers we could continue having dancers enter the stage until the space is full, which
and that means that might actually be fun,8 but with only one dancer at our disposal, if we don't want to
:eir seats, we already just let her stand there, we have to come up with something different. Ifwe don't allow
tage before the doors our dancer to talk, she will have to do something, that is, she will have to move in such
to hurry so as not to a way as to keep the audience's attention fixed on her body. This is the hard part and
:am Forslthe begins there are no easy answers as to how to proceed, for we are now at the heart ofdance
ights in the audience and choreography as an artform. In a way, every solo performance can be seen as an
. curtain. attempt to answer the question of how a single body can capture and hold the audi-
re point the lights in ence's attention. Here, again, expectations are important, since each new piece fits in
e.,' should stop their or deviates from an existing repertoire ofprevious choreographies.
:¡rmance is about to In the work of choreographers such as William Forsythe and Merce Cunningham,
rLrre potentially dis- at any moment there is a lot of concurrent activity on stage. Dancers perform in seem-
: pie in the audience ingly random groupings and orderings at different locations on stage. There is no main
:É.n the real and the action, which dominates the stage. It is impossible to keep track of everything at once.
;;'s eryectations by The audience has to choose where to look and is constantly challenged to shift its atten-
:'gressing the imagi- tion (Fig z6.z). This observation has a parallel in various scientific experiments, which
ir-inE the performers show that there is a task-dependent upper limit to the number of objects people can
-i.Lt\. track simultaneously (Cavanagh and Alvarez zoo5;.Llvarez and Francon eri zooT).
',.:dr-. open the cur- Attention is the process ofselectively concentrating on one object, feature, or event,
¡¿ -rqhts to reveal the while relaying everything else to the background. It can be modulated both by top-
::. :n u-hich case the down and bottom-up signals. Trying to focus on one particular dancer because one
i:r. or \\-e could keep knows that s/he is a famous performer is an example of top-down attentional control.
r:¡ audience will be On the other hand, the fact that other events may automatically grab our attention
:lrjr alert. In neural shows that attention can also be driven by sensory cues. In the absence ofboth top-
:1::LrSenSory COrtiCeS down control and salient stimuli, attention drifts and the mind wanders. Some people
may therefore experience performances in which not much happens, as in butoh, a
::.l: dancer walks on dance form which originated in post-Second World War lapan in which stillness plays
':;er takes the stage an important role, as boring. They can also be a revelation, becqusefhe mind is free to
i-r-e could postpone wander or because one is invited to pay close attention to more subtle changes, a slight
; rr-'dl and unveiling tilting of the head or a tiny movement of a finger, which can make all the difference.
:::1i' rr earing a non- Salient stimuli tend to grab our attention. A female dancer will stand out among a
:a aLlslumes in dance group of men, she will stand out even more if she is dressed differently and even more
:::iing. anlthingtoo if she is wearing a colourful floral dress and the men wear plain trousers and shirts.
The dress will stand out at a perceptual level, but it will also emphasize her femininity.
:::intion something In many dance performances we can observe such a doubling of contrast. Women in
-:- ites and then walk the work of German choreographer Pina Bausch always wear dresses. The example
\er-er mind that
:,:,.-. I just gave is, in fact, taken from a scene in Masurca Fogo (rSS8). In the work of choreo-
-i:ence rvon't know. grapher and pioneer of contemporary ballet George Balanchine (r9o4-r983) women
often wear white pants and black tops, whereas men wear black pants and white t-shirts.
::, ' ,:i Oscar Wilde's This relates to what is known in cognitive neuroscience as grouping by features
:: -: i; :ninutes in
r-: --re rOundS Of I Glass Pieces (r9s3) by the American choreographer Jerome Robbins opens with a seemingly
endless stream of dancers crisscrossing the stage from all directions.
52O rVAR HAGENDOORN
the positit¡r ¡ i:
see that to k:¡:
than that. shr r',
nto\-ement oi al
A simple :e;l
rdth a randLrrl
arm, right 1e<. :r
the numbe¡ o- :
mo\-ements aua
compositions l:
tempo. \\-e c¡r -
der, or im¿gi:r¿
a cf
r-ocabular,,'
in different c',,
vourselt. \-oi-r ir-
There is e :l'
l'onder rrha: --h
nr-isting and:i:
and bodilr agil:
Fig.26.2 William Forsythe, The Second Detail(1991). Dancers: Ballet de L'Opéra
rve don't ius, :,
de Lyon.
IPhotograph : Vincent Jeannot.) Forsrthe on.. ;,
This mar-be r,:s:
In receni -,;.
in somethin:. .
and/or con:-¡,.
and 'object-based' (global) attention, in which al1 stimuli sharing a similar feature other (Srh-ia :-
are selected. of modern ¿:,. .
It is important to note that salient sensory stimuli only grab our attention when sis. Particrpa;::.
they are relevant to the task we engage in, are known to be of behavioural significance, rvas compler :::
or when we diffusely attend to a broad spatial region, as in our opening scene just stand it.' Th¡:;
before the dancer entered. Various experiments have demonstrated that, when focus- guessed ours:,., .
ing on a particular task or when attending to a particular object, people may fail to a perceptual ^:., ,
notice another salient and distinctive object or event, a phenomenon known as inat- cal ballet. Th¿., .
tentional blindness. In one experiment, people had to watch a video of a group playing eral public. lr-.'
a ball-game and to count the number of passes (Simons and Chabris 1999). At some becomes mor3 --
point a person in a gorilla costume walked through the scene. After the experiment as Merce Curn-:
about half ofthe participants reported not having seen anything unusual. Dance perfor- audience rrii. ls
mances offer many illustrations of this phenomenon. When our attention is focused on course, totaLl-,'-":
one dancer, we may fail to notice the entrance of another dancer in the periphery appraising the I.
of our visual field, a change in the lighting, or the fact that a television on stage has
started playing.
I Willian-r Fct,r---
Motion and novelty are two features that tend to guide the deployment of attention.
refers to anoth¿: * :
shtml
Both are, of course, related, since motion can be seen as a change of the position of "' These ñndr:.,
the body as a whole or as a change in the configuration of the body, that is, a change in Acceptable, prin;.:
DANCE, CHOREOGRAPHY, AND THE BRAIN 527
the position of the limbs relative to each other. Now, ifwe add motion and novelty, we
see that to keep attention focused on the dancer, she would have to move and, more
than that, she would have to diversifi, her movements. If she were to repeat the same
movement over and over again, attention would again wane.
A simple technique for bringing variation into a movement sequence is this: start
with a random limb and then keep moving a different limb or part of the body, left
arm, right leg, head, left shoulder, etc., one limb at a time. To add some more variation
the number of body parts being moved can be alternated, as can the time span between
movements and the speed with which they are performed. This is similar to musical
compositions in which a theme is repeated with different instruments and changes in
tempo. We can also imagine drawing figures with the hand, foot, head, elbow, or shoul-
der, or imagine avoiding and stepping over imaginary obstacles. We can also define
a vocabulary of ten or more movements, which we can then deconstruct and perform
in different constellations (Hagendoorn zoo3). If you practise these techniques
yourself, you will notice that what you are doing resembles dance.
There is a risk that after a few minutes the audience will lose interest. They may
wonder what the fuss is all about. All they may see is a person moving about on a stage,
twisting and turning and swinging her arms. They may admire the dancer's virtuosity
;: Ballet de L'Opéra and bodily agility, but it may leave them unmoved. To keep the audience interested,
we don't just need variation and differentiation we also need structure. As William
:+ :::,ü'¿¡h : Vincent Jeannot.) Forsythe once commented, 'if there isn't a directing mind, it looks like a can of worms'.e
This maybe fascinating to look at for a little while, but not for a full hour.
In recent years, psychologists have begun to study what makes people interested
in something. One theory proposes that interest depends on an appraisal of novelty
and/or complexity on the one hand and an appraisal of comprehensibility on the
i¡s a similar feature other (Silvia zoo5). Various experiments using both experimental artefacts, pieces
of modern art, and contemporary poems have provided evidence for this hlpothe-
L')r.rr attention when sis. Participants in the experiments tended to spend most time with a picture when it
uiou¡al significance, was complex and, in the case of modern art and poetry, when they felt able to under-
r op,ening scene just stand it.'o These results may sound obvious, since they confirm what we may have
:ed that, rvhen focus- guessed ourselves. They explain why minimal and conceptual dance can be boring at
t. people may fail to a perceptual level, but intellectually interesting, compared üth musicals and classi-
::ron knolvn as inat- cal ballet. They also explain why the opinion of experts, such as critics, and the gen-
*u.oia group playtng eral public, may differ, and why, over the course of a career, the work of many artists
ab'ris 1999). At some becomes more complex and subtle, as is evident in the work of choreographers such
trer the experiment as Merce Cunningham, William Forsythe, Sasha Waltz, and JiÍí Kylián. Whereas the
;rsual. Danceperfor- audience will usually only see a performance once or twice, the choreographer is, of
tention is focused on course, totally familiar with his or her own work. A choreographer's benchmark for
;er i¡ the periphery appraising the novelty and or complexity of his or her own work is therefore likely to
e"' ision on stage has
g William Forsl.the
interwiewed on BBC Radio 3 by Iohn Tusa, z February zoo3. The interviewer
r',r¡ent of attention. refers to another unnamed source. http://www.bbc.co.uk/radio3/johntusainterview/forslthe_transcript.
shtml
le of the position of
'" These findings proüde some experimental evidence for the MAYA, Most Advanced Yet
¡. -ñat is, a change in Acceptable, principle coined by the American industrial designer Rayrnond Loewy.
522 IVAR HAGENDOORN
increase overtime. Similarlycritics,whose job itis to watch and criticallyevaluate dance light-giving? Hor,-;
performances, are likely to have a different benchmark than the average person in the written word could
audience. as the French poe: S
Structure can be narrative, as in the work of Pina Bausch and Sasha Waltz, or purely Suppose we \\'an:
spatiotemporal, as in the work of Merce Cunningham and Lucinda Childs. The tech- perhaps Sislphus. ','
niques introduced earlier to bring variation into a movement sequence can also be used he neared the top .t
to add structure. We can also invent some new rules and techniques. We could alter- try and then tn- as:.
nate between performing at a point in space and along a line. We can divide the stage ourselves to moveit
into left and right and front and back and perform different movements in different dancer perform il s,
regions. By thus emphasizing geometrical relations and the 'spatiality' of space, the slightly tilted siie.,,.'
audience's awareness of space maybe heightened. backto the starti¡t::
With the help of these techniques we may have succeeded in bringing some struc- as if realizing that sr
ture into the dancing, but that doesn't necessarily make it interesting. What we need is bewhat, inballet. c:
judgement. Up until now I have assumed that our dancer entered the stage unprepared does not exist. h is ¡
and that she is just dancing around or 'improvising' as this is called in dance and theatre. Let's rel.iel- rr.l¿:
We can also put ourselves into the seat of the audience and rehearse the piece. We can we made more ab.::
try out different beginnings and endings and different movement sequences, which we witness countless .i
can fine-tune to the desired effect, judging every subsequent rehearsal in an ongoing how they unfold a:r
trial-and-error process. As I would argue, this is how most choreographers and danc- ourselves har-e n¡,,--r
ers work, whether the breakdancer wishing to impress his friends at their next meeting edge ofthe rvorld ¿:
or the poledancer who uploads a video of herself on YouTube to show off her ski-lls. will eventualh'p:.i'i
fournalists always like to know where an artist gets his or her inspiration from and less the same orc¿:
most likely it is something audiences like to know as well, since it is a question I am often form of templates. :
asked in one form or another. My own work is not an illustration of my research and We also instantlr :
I don't take any direct inspiration from what I've read and learned. This may sound anger, or sadness I'
odd. Of course I carry it with me, but there is so much that I carry with me. Inspiration ence to get emcrti,,-'i
may come from anlwhere and may strike at any time. When I start work on a new This is whr. \le:..
dance production I may have some ideas, but the actual work is created in the studio, the order of mo',.¡:il
not in my mind. i totally concur with the American choreographer William Forslthe, in which the pan:;.
who once said in an interview that on day one he usually tells his dancers he has figures, shorr'ed rh::
absolutely no idea what he is doing because he has never made this ballet before." objects (Heider:.r:
In a recent review of a performance by the New York City Ballet of Balanchine's watching people p.:
Agon (t957), one of the greatest ballets of the twentieth century, Clement Crisp, dance everyday life. Th.s .,
critic for the Financial Times, wrote that 'after half a century it remains an innovative great expressir-e !c,',',
marvel, questioning and discovering, time-travelling still, and still light-giving' (Crisp As you're readi::
zoo8).Inanarticleforthe NewYorkTimes,dancecriticAnnaKisselgoffoncewroteof thus far. If you drci:
Merce Cunningham, that 'his idea that any movement can follow any other symbolizes fewseconds the dai:-
the discontinuity of our time' (Kisselgoff 199z). when I read such statements I am often
line to the front o- ,:
reminded of a remark by Susan Sontag: 'interpretation is the revenge of the intellect her arms in front c,-
upon art' (Sontag 1966). Leaving aside the question whether our time is discontinu- halfofher bodr, \r :
ous, how can dance syrnbolize it? And how can dance be questioning, discovering and
resentation in r.ou: :
tation. Neuroimagi.l:
"wiliiam Forsl'the interviewed on BBC Radio 3 by ]ohn Tusa, z February 2oo3. http://ww-w.bbc. reading action r-e11¡ ¡
co. uk/radio 3/j ohntusainterview/fors¡he_transcript. shtml
So, what about rr'¿t;:
DANCE, CHOREOGRAPHY, AND THE BRAIN 523
callr. evaluate dance light-giüng? How can a dancer by'writing with her body (. . .) suggest things which the
;erage person in the
written word could express only in several paragraphs of dialogue or descriptive prose'
as the French poet Stéphane Mallarmé wrote? (Mallarmé t886/r983).
iha \\'altz, or purely Suppose we want to render the idea of futility in dance. The archetlpe of futility is
L¡ Childs. The tech- perhaps Sisyphus, who was forced to roll a rock up a mountain slope. But every time
:lce can also be used he neared the top it would roll down again. With this in mind we could have a dancer
res.\\¡e could alter- try and then try again, without ever succeeding. But try what? Since we have restricted
;en divide the stage ourselves to movements, we could choreograph a movement sequence and have the
.inents in different dancer perform it several times in a row. After each attempt she stands still, her head
i¿lin.' of space, the slightly tilted sideways, visibly breathing, as if reviewing her movements. she walks
back to the starting position. Once or twice she maybreak off the sequence in-between,
úgirg some struc- as if realizing that she did something wrong. Perfection in dance is unattainable. It may
:q. \\-hat we need is be what, in ballet, dancers strive for, but the perfect arabesque, épaulement,or attitude
r¿ stage unprepared does not exist. It is an idea.
r dance and theatre. Let's review what we have just proposed. We started with a familiar scenario, which
e ihe piece. We can we made more abstract. We then added some familiar poses. Throughout our life we
3quences, which we witness countless events; some are regular, others singular. we know of many events
.:sal in an ongoing how they unfold and we understand the emotional impact if there is one, even if we
:aphers and danc- ourselves have never experienced it. Through experience we acquire habits and knowl-
: -.hei¡ next meeting
edge of the world around us. When watching an action movie we know that the hero
:,r,.' .-,lt her skills. will eventually prevail. when we brush our teeth and get dressed, we do so in more or
spiration from and less the same order every day. All of this is laid down deep inside our memory in the
úLestion I am often form of templates, action scripts, and scenarios.
:: mr- research and We also instantly recognize emotional body postures expressing fear, joy, aggression,
i. This may sound anger, or sadness (de Gelder zoo6). One consequence is that if we don'twantthe audi-
-:h me. Inspiration ence to get emotionally involved we should take great care to avoid familiar gestures.
¿:t lnOrk On a new This is why Merce cunningham frequently uses random procedures to determine
¡¿ted in the studio, the order of movements. One of the classic experiments in experimental psychology,
' -,r,'illiam Forsythe,
in which the participants watched an animation movie featuring some geometrical
:ls dancers he has figures, showed that people even attribute intentions, desires, and fears to inanimate
b:ilet before." objects (Heider and Simmel 1944). So we can surmise that they will do so, too, when
-et of Balanchine's watching people perform more or less abstract movements outside of the context of
:=ent Crisp, dance everyday life. This is one reason why the work of Merce Cunningham can still have
1¿:¡s an innovative great expressive power.
:g-hr-eiving' (Crisp As you're reading this you may have formed an image of the scenes I have described
-sott once wrote of thus far. If you didn't, you will when I tell you to do so. Picture an empty stage. After a
:',' rrrher syrnbolizes few seconds the dancer reappears from the back end ofthe stage and walks in a straight
::::Tlents I am Often line to the front of the stage, where she stops. She stands facing the audience, crosses
irse of the intellect her arms in front of her body and slowly begins to pull up her shirt, baring the upper
:-n¡ is discontinu- half of her body. when you picture the stage or the dancer you will form a visual rep-
l:. discor.ering and resentation in your mind. When you picture a movement you form a motor represen-
tation. Neuroimaging studies have shown that mentally rehearsing a movement and
. . hi¡p:',irr-ww.bbc. reading action verbs such as walking or throwing activate motor regions in the brain.
So, what about watching movement?
524 IVAR HAGENDOORN
In an intriguing neuroimaging studyresearchers compared the brain activity ofballet listeningto mus:;.
dancers and capoeira dancers as theywatched short videos of each others' movements it somervhat ils¡,-
(Calvo-Merino et al. zoo5). They found greater activity in parts of the premotor and capture and hold t
parietal cortex when dancers watched movements in their own style. In a follow-up be inferred br' ¿i,
study the authors compared the brain activity of male and female ballet dancers as they performances .,,.-i
watched short clips of gender-specific movements (Calvo-Merino et al. zoo6). The and drarr huqe ;r
idea behind the experiment was that both male and female dancers are equally visually fo¡ hours on end:
familiar with all movements, because they take class together, but they don't perform lrom just h'atch-:r
all movements. The authors found greater activityin parts of the premotor and parietal interesting on s-r;i
cortex and the cerebellum when dancers watched their own gender-specific move- this question. bui-
ments. They concluded that 'observing an action can activate the corresponding motor and sister.
representation'. But what if there are multiple dancers in your field of view (Figz6.z)? One of the pri::;
Is there really a motor representation for each movement? I doubt it. dense or eñena-a
In recent years, so-called motor theories of cognition have become increasingly har-e been tburi :
popular in cognitive neuroscience. According to one version of this theory we under- a strong reguiar ::
stand others by simulating their actions (Blakemore and Decety zoor). With reference studies har-e siL':ir-
to Freud's book on jokes, the French neuroscientist Marc Jeannerod gives the example motor areas in :,i¡
ofwatching a clown: the lateral prem,- ,
comedy. The simulation of the action we observe does not meet the expectation. allorv the predi;:r.
(Gallagher and feannerod zooz) this account. cre::
ma1' erplain r', h.,
The discovery of mirror neurons, a class of neurons in the ventral premotor cortex "
in each domai:: :¡
of the monkey brain that respond both when the monkey performs and observes a
music mar- be t,-, :
goal-directed action such as grasping a raisin, added considerably to the popularity
It is norr- trr'rt ,ll
of simulation theory (Gallese etal. ry96'). Various neuroimaging experiments have
It's not quite r'i.¿;
since provided tentative evidence for the existence of a similar mirror system in the
a real protessi.r''
human brain (for a review, see Rizzolatti and Craighero 2oo4; Iacoboni zoo5). As yet 'b¡i
movements. t,
there is little hard evidence to support the claim that the mirror system contributes to
size a movemer-:.
our understanding of the meaning and intention of observed actions (Dinstein et al.
They knorl hl','- :
zooS). Some caution in interpreting these findings is therefore warranted (Jakob and
all of this is n¿:::
]eannerod zoo5).
An important ;,, r:
I once believed the mirror system could provide the basis for an explanation as to
American chorr,-,g
why dance can be fascinating to watch (Hagendoorn zoo2i zoo4b), but I have since
a Qpical contrai-:
become more sceptical. I wonder how the mirror system might contribute to the
it with the entir¡
perception of the concurrent movements of multiple dancers, and how it could
phers have de.,'el,--,:
account for our understanding of the simultaneous pushing and pulling in a duet.
stretching vour a:l
After another brief interval our dancer returns to the stage. This is our little trick
to connect disconnected scenes and to give the audience some time to breathe (and
cause the ?rm rt¡ i:
cough). The dancer is carrying a portable music player. Music can be a means to
amplify reality, to envelop the audience in moods and waves of feeling. Only few
dance performances are performed in silence. I've got nothing against music, I enjoy " Some dance p-"-
DANCE, CHOREOGRAPHY, AND THE BRAIN 525
:rin activityofballet listening to music, or some music sometimes, but as a choreographer I've always found
-',thers' movements it somewhat disconcerting that dance in itself isn't strong enough as a stimulus to
: rhe premotor and capture and hold the audience's attention for say 3o or 6o minutes. That this is so can
t. re. In a follow-up be inferred by adopting the line of reasoning popular in economics: if silent dance
¿-let dancers as they performances would have immense popular and critical appeal, they would abound
.¡ et al. zoo6). The and draw huge crowds." why isn't this so when people can be immersed in a book
aie equallyvisually for hours on end? I must say that I myself rarely go to concerts, because I get bored
'.i-iev don't perform from just watching musicians play their instruments. I mean, do something visually
:notor and parietal interesting on stage, like a dance performance! I'm afraid I don't have an answer to
ier-specific move- this question, but we can offer a conjecture on why music and dance seem like brother
-espondingmotor and sister.
l.¡f viov (Figz6.z)? one of the principal components of music is rh¡hm. Rh¡hm can be weak or strong,
dense or extended, single or multilayered, regular or irregular. Across cultures people
.;,rme increasingly have been found to spontaneously synchronize their body movements to music with
.s rheorv we under- a strong regular rh¡hm and a beat that is neither too slow nor too fast. Neuroimaging
,r.\\'ithreference studies have shown that the production and perception of auditory rh¡hms activate
: grr es the example motor areas in the brain (Janata and Grafton zoo3). Interestingly, one of these regions,
the lateral premotor cortex, has also been associated with a number of sequential pre-
:.-: and then falling
dictiontasks (SchubotzzooT). Earlier, we sawhowatamacro-levelcognitive structures,
i-:tion by simulat- habits, and our own episodic memory create an expectation in us of how events unfold.
: :r: ihe expectation. The picture that is beginning to emerge from studies of the premotor cortex is that,
: --r-s ts the source at a micro-level, the premotor cortex extracts the sequential structure from a series to
of
--:i: rrn. allow the prediction of its continuation. watching dance accompanied by music, on
:::3 Teannerod zooz) this account, creates expectations in both the auditory, visual, and motor domain. This
may explain why dance and music go together so well. The expectations engendered
.- :remotor cortex
in each domain are mutually reinforcing. However, setting a step to every note in the
::. and observes a music may be too predictable and as a result may be experienced as boring.
,'rt¡ the popularity
It is now two days before the premiere and we sense that something is still missing.
;:r¡e¡iments have
It's not quite what we wanted yet. Thankfully, our dancer comes to the rescue. She's
-:r,.r sr-stem in the
a real professional. Dancers understand when you tell them not to go through the
i c,ni :.oo5). As yet
movements, but to let the movements go through them. They know how to empha-
;:n contributes to
size a movement, how to add an exclamation mark here and a question mark there.
:r. Dinstein et al.
They know how to let a movement linger and how to alter its intensity. Of course
::nred (lakob and
all of this is metaphor and somehow it has to be translated into motor commands.
An important concept in Graham technique, the movement system developed by the
l ;¡.¡lanation as to
American choreographer Martha Graham (r894-t99t), is'contraction'. An example of
. but I have since
a tlpical contraction would be to imagine holding a very large ball and to then squeeze
;:,nrribute to the
it with the entire body. The opposite of contraction is release. Various choreogra-
"::i horv it could
' phers have developed their own version of release technique. An example would be
-:lg in a duet.
stretching your arm and then letting go off the muscles in the shoulder. Gravity will
.¡ rs our little trick
cause the arm to drop. These two examples may give some idea of how the quality of a
r; trr breathe (and
:n be a means to
-!eling. Only few
:rsi music, I enjoy " Some dance pieces are performed in silence, but their number is negligible.
526 IVAR HAGENDOORN
movement sequence can be modulated. The use of contractions gives movements an Hagendoorn IG r :- ,:
angular look, while release technique tends to look loose and floppy. G Klein and Ch Z_:
We have now finished choreographing our performance. Whether it will be any Hagendoorn IG :c,,.
good is for the audience to judge and depends on whatever else they may have seen inspire dance a::
Hagendoorn IG i:':.- .
and the mood they are in when they attend the performance. All we need is a title.
Hagendoorn IG r:,, -.
A title can be descriptive, Solo, or borrow from the title of the music that accompa-
dance and chorec:
nies it, like Violin Concerto. Titles can also be used to prime the audience. A good title Hagendoorn IG r ::, - .
is ambiguous, mysterious, and open to interpretation. It raises the curiosity of the Thinking in Ft'¡t¿r
audience. Let's call the piece we just created To be is to be better than is not.'3 Coming
'
Heeger DJ and Ress lr
soon, to a theatre or a computer screen near you. Neuroscience,3. t::.
Heider F and Simnei l,
of Psychologl,. 57. ::
IacoboniM (:.oo;. ).,,
632-2.
RBpnnnNcns
fakob P and Jeann::-':
C o gnitiv e Scier:--:,.. 9
Alvarez GA and Franconeri SL (zoo7). How many objects can you track? Evidence for a
]anata P and Gratt¡,'. j
resource-limited attentive tracking mechanism. I ournal of Vision, 7, 7-7o.
related to seque n.---
Bidet-Caulet A, Voisin f, Bertrand O, and Fonlupt P (zoo5). Listening to a walking human
Kaiser P (zoor). St:;.
activates the temporal biological motion area. Neuroimage,28, r3z-9.
Digitale. Nlor': -\I¡.,
Blake R and Shiffrar M (zoo7). Perception of human motion. Annual Reyiew of Psychology,
KisselgoffA(r99:.1.
58' 47-73.
15 March.
Blakemore S-f and Decety I (zoor). From the perception of action to the understanding of
MallarméS (1886 r!:.
intention. N ature Reyiews N euroscience, 2, 561-7.
Universitl- Press. --:
Brown WM, Cronk L, Grochow K, et al. (zoo5). Dance reveais sgnmetry especially in young
men. N atur e, 438, [48-5o.
McNeill WH iiqq -;,
tr,-es movements an Hagendoorn IG (zooz). Einige Hlpothesen iiber das Wesen und die Praxis des Tanzes. In
G Klein and Ch Zipprich, eds. Tanz Theorie Text, pp. 429-44. LIT Verlag, Hamburg.
.iher it will be any Hagendoorn IG (zoo3). Cognitive dance improvisation. How study of the motor system can
te.,' may have seen inspire dance (and vice versa). Leonardo,36,22r-7.
j r''e need is a title. Hagendoorn IG (zoo4a). Towards a neurocritique ofdance. BalletTanz yearbook,6z-7.
Hagendoorn IG (zoo4b). Some speculative hypotheses about the nature and perception of
usic that accompa-
dance and choreography . lournal of Consciousness Studies, tt(ll +), 79-1ro.
J.rence. A good title Hagendoorn IG (zoo5). Dance, perception and the brain. In s McKechnie and R Grove, eds.
ire curiosity of the Thinking in Four Dimenslons. Melbourne University publishing, Melbourne.
,tl is not.'3 Coming Heeger DJ and Ress D (zooz). What does fI4RI tell us about neuronal activity? Nature Reyiews
N eur o s ci en c e, 3, t42-5l'.
Heider F and Simmel M (1944). An experimental study of apparent behavior. American Journal
of Psychology, 57, 243-9.
Iacoboni M (zoo5). Neural mechanisms of imitation. Current Opinion in Neurobiology, 15,
632-7.
Jakob P and feannerod M (zoo5). The motor theory of social cognition: a critique. Trends in
C o gnitiv e S ciences, 9Q), zr-5.
'::¡? Eridence for a
-: -.
|anata P and Grafton ST (zoo3). Swinging in the brain: shared neural substrates for behaviors
related to sequencing and music. Nature Neuroscience,6,63z-7.
:- ; ,,.'alklng human Kaiser P (zoor). Steps (l'arte della collaborazione). In A Menicacci and E Quinz, eds. La Scena
:.:. ::u' Digitale. Nuovi Media Per La Danza, pp.7q-62. Marsilio Editori, Venezia.
o.f Psychology,
Kisselgoff A (1992). Merce cunningham, Explorer and Anarchist. The New york Times.
r5 March.
:-. 'rnderstanding of
Mallarmé s (r8s6/r983). Ballets. In R copeland and M cohen,eds. rMhatis dance?p.nz, oxford
University Press, Oxford.
* .s;,ecially in young
McNeill WH (rgqz). Keeping Together in Time. Dance and Drill in Human History. Harvard
University Press, Cambridge, MA.
: ! :oo5). Action Peelen MV and Downing PE (zoo7) . The neural basis of visual body perception . Nature Reyiews
:: lr¿ncers. Cerebral Neuroscience, 8, 636-48.
Ramachandran VS (zoor). Sharpening up 'The science of art'. An interview with Anthony
: :ro6). Seeing or Freeman. J ournal of Conscio usness Studies, I (r), 9-29.
_,",nett Biology, t6,
Ramachandran VS and Hirstein W (rqgq). The science of art: A neuroiogical theory of aesthetic
experience. J ournal of Consciousness Studies, 6, r5-5r.
:r :-riiocal attention.
Reynolds N and McCormick M (zoo3). No Fixed Points. Dance in the Twentieth Century.
p. 383, Yaie University Press, New Haven, CT and London.
Rizzolatti G and Craighero L (zoo4). The mirror-neuron system. AnnualReview ofNeuroscience,
".;¡. \nture Reyiews
4, t69^92.
Schubotz RI (zoo7). Prediction of external events with our motor system: towards a new
: :,:r nature. Current
framework. Trends in Cognitive Sciences, rr, zu-r8.
Silvia P| (zoo5). What is interesting? Exploring the appraisal structure of interest. Emotion,
-:: l:uman superior
5, 89-102.
Simons DJ and Chabris CF (1999). Gorillas in our midst: sustained inattentional blindness for
:::.nierr- lvith Marc
dynamic ev enfs. P er ception, 28, to 59-7 4.
Sontag S (rq66). Against lnterpretation and Other Essays. Anchor Books, New York.
. -: .n the premotor
Vaina LM, Lemay M, Bienfang DC, Choi AY, and Nakayama K (zooo). Intact "biologi-
cal motion" and "structure from motion" perception in a patient with impaired motion
:,r'¿r posterior STS
mechanisms: a case study. Vision Neuroscience,5,3fi-6g.
Voisin J, Bidet-Caulet A, Bertrand O, and Fonlupt P (zoo6). Listening in silence activates
auditory areas: a functionai magnetic resonance imaging study. Iournal of Neuroscience,
r a::1. 26,273-8.
NEI
ol
BEAT
Deucp, as a dr¡¿::-,
alongside othei cr¡;
movement eithe: u,
performance setti:r.
have evol.,'ed in ¿:*
definitions of ril:L¿i
and preferences. Ti:
folk dances) to hrr:
tlpically found in sp
martial arts (kat¿s. :
components (e.9. :
oft:r
the expression
In this chapter. ,
NEUROAESTHETICS
OF PERFORMING
ARTS
INrnooucrroN
DeNCn, as a dynamic art form, has classically been included in the performing arts,
alongside other disciplines such as music, opera, and theatre. Dance refers to human
movement either used as a form of expression or presented in a social, spiritual, or
performance setting, often associated with an aesthetic value. Many dance disciplines
have evolved in different cultures over hundreds, or thousands, ofyears. Therefore
definitions of what constitutes dance may depend on social or cultural constraints
and preferences. These might vary from relatively simple, everyday movements (e.g.
folk dances) to highly skilful techniques (e.g. ballet), to more virtuosic movements
tlpically found in sport (gymnastics, figure skating, synchronized swimming) and in
martial arts (katas, capoeira). Finally, dance often involves the integration of several
components (e.g. movements, costumes, music, scenography), each contributing to
the expression of this artform.
In this chapter, we focus on two observations that we consider important for
studying the neural underpinnings of dance. The first acknowledges that, despite
individual differences, dance can, in most cases, be decomposed into single units of
movements. It is the choreographer's challenge to design and combine sequences
of movements in order to achieve a maximum level of expression and connection
t30 BF,ATRIZ CALVO-MERINO AND PATRICK HAGGARD
between the performing dancer on the stage and the audience. This leads us to the it difficult to cre;iÉ
second matter of interest. As with any art form, the ultimate aim of a dance perfor- stimuli, the en:e:sri
mance is to evoke responses in the observer. This essaywill focus on one specific response sub.iective acclit,::
elicited in an observer while watching dance: the aesthetic experience. In particular, modern neurtrr¡¡¿:
we of the neural mechanisms associated with
will describe models to guide investigation In general, mosl ,
aesthetic experience while watching the core element of dance: movements. nisms involr-ed rn ¡.
preferred item. T:e,
ated the aestheri.::
qeometrical fizu:¿s
Goel, zoo-1; laco:.".
has been little ct:..
NnunoAESTHETrc RESEARcH AND studf ing neuroae!:-t
paradigms, anci ,:::¡
AESTHETIC PROCESSING COMPONENTS
exhaustive retlectt -- r
theless, the brai: ¿.
We use the term 'aesthetic experience' to indicate a particular psychological state related to the ¿."..t,;-.
produced by a type of sensory stimulus that is often, but not exclusively, a work of art. might be distribu:¿,
With the advent of neuroimaging techniques, neuroscience has started to investigate thetic componen:::
the relationship between the human mind and the human brain. More recently, a new Studies shor' -:. :
field called'neuroaesthetics'has emerged, focusing on the neural mechanisms under- tbr the purpose c::
lying the internal processes associated with aesthetic experience (i.e. aesthetic evalua- Ol SenSOn' COntDL- I
tion, aesthetic judgement, aesthetic perception) (Cela-Conde et al. zoo4; Kawabata Zeki:.oo4; \'arra:li::
andZeki zoo4; Vartanian and Goel zoo4). triate areas t ]actr !:r
A key aspect of aesthetic and neuroaesthetic studies is the concept of evaluation. Goel zoo4) are rep.:
We can define two tlpes of evaluation related to aesthetics. The first one focuses on opposedto dislLke.,r
the properties attributed to the evaluated object ('this is a beautiful item'), while the a functional spe;r;-
second one focuses on the subjective experience or attitude of the observer ('he likes iudged (Zeki et a,. ,:
it'). This division has led aesthetic research into two perspectives: objectivisttheo- to r.iervins othei :.:
ries and subjectivist theories. Objectivist theories postulate that the perceptual system attractiveness Pa::.
of an observer will treat beauty and other aesthetic properties of our environment like Furthermore . ',', c
any other attribute of it. This perspective emerged from early psychophysical studies processing. Posit:.,-.
that focused on identif,.ing particular stimulus properties or arrangements of attri- er-okes activin- i:-l ::.
butes that induced aesthetic experience, such as symmetry, balance, complexity, and has also been desc:-i
order of stimuli. These studies have used a range of stimuli, from geometrical figures to tbr the monitonns ;
more complex stimuli such as paintings (MacManus t98o; McManus and Weatherby :.ooo; Cela-Conde .'
1997; ]acobsen2oo4;Jacobsen and Hofel zooz; Jacobsen et al. zoo4). A common finding aesthetic judgeme:i
is that aesthetic experience depends on compositional arrangements between parts of in frontomedian ::
the stimulus, and between individual parts and the whole, and that all observers Iacobsen et al. :¡ ,
share a common perceptual mechanism for seeing these attributes (Leder et al. zoo4). rnvestigating othe: :
On the other hand, subjective theories counter the objectivist viewpoint by focusing tCunningham et a-, .
primarily on the observer, and considering each human as a special indiüdual. This Finally, a hecio:r:i
perspective assumes that preference is largely a matter of attitude, individual taste, neural activitv ass¡,;
personal experience and cultural environment (Zajonc t968), and support the region of the pre ;:,_,::
common saying'beauty is in the eye of the beholder'. While psychologists might find value of stimuli, ¡¡=
NEUROAESTHETICS OF PERFORMING ARTS 531
This leads us to the it difficult tocreate models of subjective responses that vary among indiüduals and
n of a dance perfor- stimuli, the emerging field of neuroaesthetics has taken on the challenge to investigate
'rne specific response subjective accounts by focusing on the neural correlates of aesthetic feeling, using
:ence. In particular, modern neuroimaging techniques.
.l:ms associatedwith In general, most early studies in neuroaesthetics aimed to describe the brain mecha-
r"-aments.
nisms involved in aesthetic evaluation p er se, regardless of the physical properties of the
preferred item. These studies recorded brain activity while participants explicitly evalu-
ated the aesthetic properties of static images such as paintings, objects, casual scenes or
geometrical figures (cela-conde et aL. zoo4; Kawabata andzeki, zoo4; vartanian and
Goel, zoo4; Jacobsen et al. zoo6). As often in the early stages of a new discipline, there
has been little consensus on the concepts, predictions, or methodologies required for
AND studying neuroaesthetics. Several studies addressed similar questions through different
NE NTS paradigms, and this has produced a mix of both converging and diverging results (an
exhaustive reflection of these earlier studies can be found in Nadal et al. zooS). Never-
theless, the brain activations reported in these studies suggest that neural responses
:srchological state related to the expliclf process of evaluating or judging beauty of static üsual stimuli
i1'! eh'. a rvork of art. might be distributed along different processing stages. Here, we identified three aes-
¡;ted to investigate thetic components: sensory, cognitive, and hedonic responses.
fi:e recently) a new Studies showing brain activity in visual brain areas when participants watch stimuli
:echanisms under- for the purpose ofproviding an aesthetic evaluation suggest that an initial perceptual
:. aesthetic evalua- or sensory component is part of early stages of the aesthetic process (Kawabata and
'l- :oo-li Kawabata Zeki zo o 4;Y artanian and Goel zo o4; |acobsen et al. zo o 6) . Among these re gions, extras-
triate areas (Iacobsen et al. zoo6) and the occipital and fusiform gyri (Vartanian and
;tp: of evaluation. Goel zoo4) are repeatedly reported as active when subjects see stimuli that they like, as
:¡st one focuses on opposed to dislike. Although these activations might not be related to preference but to
:l item'), while the a functional specialized processing and perception of the category of stimuli being
¡bsen-er ('he likes judged (Zeki et a7. ry94 Moutoussis andZeki zooz), some of these areas also respond
:>: ¡b.iectivist lheo- to viewing other types of stimuli such as faces, while evaluating different degrees of
p<rceptual system attractiveness (Paradiso ef al. ry99; Iidaka et al. zooz).
r en..-ironment like Furthermore, we can describe a cognitive or semantic component during aesthetic
i¡'phvsical studies processing. Positive evaluation of beauty, for example, when watching paintings,
l:cntents of attri- evokes activityintheprefrontaldorsolateralcortex (Cela-Conde etal. zoo4). This region
;. ;omplexity, and has also been described as the centre ofthe perception-action interface and is critical
'netrical figures to for the monitoring and comparison of multiple events in working memory (Petrides
;s and \Veatherby zooo; Cela-Conde et al. zoo4). Another study that compared brain responses during
1, common
finding aesthetic judgements and perceptual judgements (i.e. symmetry) also showed activity
s betrr'een parls of in frontomedian and prefrontal regions, as well as temporal-parietal brain areas
--:rat all observers
(Jacobsen et al. zoo6). Interestingly, these areas are also classically found in studies
Leder et aI. zoo4). investigating other types of human judgements, such as social and moral judgements
:i-rint by focusing (Cunningham et al. zoo3).
il indivrdual. This Finally, a hedonic or emotional component might be extracted from reports of
. :ndir-idual taste, neural activity associated with observation of preferred stimuli in the orbitofrontal
ard support the region of the prefrontal cortex. This area has been often associated with the reward
lurgisrs might find value of stimuli, regardless of their aesthetic properties (Rolls zoooa,b; Small et al.
532 BEATRIZ CAIVO-MERINO AND PATRICK HAGGARD
2oo1; O'Doherty et al.2oo3; Kawabata andZeki zoo4; Kirk et al. zoo8). This rela- control and m¿n¿E
tion between hedonic responses and reward links systems for individual preference responses. In thi. it
behaviour to basic pleasure and emotion. phr.sical proper:ie s
Taken together, one can conclude that there is a dedicated set ofregions that par-
ticipate in the explicit aesthetic evaluation of beauty. Although these neural activations
are not fully consistent across studies due to the use of different paradigms, the emerg-
lüechanisms o
ing picture is that there might be different ways of 'seeing' when looking for beauty.
These ways of seeing include early perceptual processing (similar to the visuokinetic To understand :r,., ,
view of Zeki and Lamb 1994), cognitive processes related to memories, as well as hou, the human :r.
emotional components associated with reward. recruit parts of ¡L:
It is important to remember, however, that previous studies have mainly focused \ras suggested i¡-: :
on subjective approaches and have compared different sets of stimuli according to colleagues recoi¡: i
individual preferences, allowing for a general view ofthe brain responses ofthe observer monkel' brain ¿:l j. ;
during the aesthetic evaluative process. However, they have given little attention to ties during actir,:r .:
the physical properties of the selected stimuli. This provides little information about same movemen:i i:l
which physical properties of stimuli are responsible for the aesthetic experience of See also Galle:e. r-.
a participant. mirror neurons a: :
their motor ieir:!r
¿ similar svsten r. :
i'hile obsenrng ::
a netrvork oi ar¿:.
NnunoAESTHETrcs oF DANcE risual and sens.:-l
superior tempo::- .
Craighero :.oo-r .
After evaluating the state of the art, we propose a new approach for studying neuroaes- Later studre : p: -
.,
thetics. Our approach differs from existing studies in three ways. First, and most netrr-orkbl ernp-,- r.
importantly, it complements the number of studies that focus on static stimuli such studies resoile a p.:
as paintings (Cela-Conde et a7. zoo4; Kawabata andZeki, zoo4;Yartanian and Goel, are rvatching, o: i,,
zoo4) or music (Blood andZatorre zoor), by focusing on a specific dynamic stimulus In order tLr ¿n>',\
from the performing arts never studied before: dance. Second, we move away from the u-hose motor rat:ar
classical explicit evaluation of beauty to study implicit processing embedded in the actions that are ','t.,
automatic perception of stimuli. Lastly, we use a consensus approach, which compared control for the ,:ii.
to the individual and subjective approach, allows us to identify brain responses for preciseh'rvhat - :-
!r
an average observer. This last issue is particularly important, because it allows us to ent fi?es of rn¡'' ¡:
relate the brain regions sensitive to aesthetics of our average observer to each of the :oo5). Dance ot---;:'
movements used in the experiment. In this way, we will be able to describe the physical fbr trr.o reasons: I i
properties of the dance movements that influence the neural response to the aesthetic kinematicalh- ccl:t:
experience of the observer. or an obsen'er ha. '
There are at least two important components to take into account when studying mirror neuron ara:
the aesthetic experience of dance. Neuroaesthetic research in vision has been driven dancers watched i:.
by the detailed understanding of the neural mechanism underlying visual process- lr.atching visuaLl.,' si:
ing (Zeki ef aL. r99r; Moutoussis and Zeki zooz). Therefore, a first required step is to (Calvo-Merino et ¿.
understand the neural and perceptual process involved in watching dance and its key Cross et al. zoo6 s¡
elements: movements. Moreover, in order to adopt an objectiüst approach, we must lated by the desree ,-'
NEUROAESTHETICS OF PERFORMING ARTS 533
,r. :oo8). This rela- control and manage the physical properties of the movements that elicit aesthetic
¿i-,-idual preference responses. In this way we will be able to look at the stimulus space and describe which
physical properties of the movements may trigger an aesthetic response.
¡,f reqions that par-
.e neural activations
'¿irqms, the emerg-
Mechanisms of seeing dance
-,:rriins for beauty.
:,: rhe visuokinetic To understand how humans perceive dance, we should start by briefly reviewing
inLlries, as well as how the human brain perceives movements generally. It is well established that we
recruit parts of our own motor representations when watching others' actions. This
=-,'- mainly focused was suggested for the first time by electrophysiological studies. Rizzolatti and his
-l::rli according to colleagues recorded activity of neurons in the premotor and parietal cortices of the
*¡¡s ofthe observer monkey brain and described how these neurons (well known for their motor proper-
l -::tle attention to ties during action execution) responded in similar manner during observation of the
,:t¡,rmation about same movements they code in motor terms (Di Pellegrino et al. r99z; Gallese et al. 1996.
r;:i; erperience of See also Gallese, Chapter zz, this volume). This population of neurons was called
mirror neurons as they seemed to reflect like a mirror the observed motor act on to
their motor representations. Soon, researchers investigated whether humans have
a similar system to match observed and executed actions, and measured brain activity
while observing hand actions directed towards an object. These studies identified
a network of areas that participated in action observation, including a range of
]E visual and sensorimotor regions, such as parietal and premotor cortices and the
superior temporal sulcus (Grafton et al. 1996; Grézes and Decety zoot;Rtzzolatti and
Craighero zoo4).
¡i:r.l-,ing neuroaes- Later studies provided new evidence on the role and extent ofthe action observation
':. First, and most netwofk by employing more complex stimuli, such as whole body movements. These
stimuli such studies resolve a pending issue: do humans internally simulate the precise action they
'taric
::r¿nian and Goel, are watching, or do they merely show an unspecific internal response?
¡.,:lamic stimulus In order to answer this question, one needs to compare observation of actions
:,--,
''e arrat. from the whose motor representation have aheady been acquired by the observer, with other
: ¿nbedded in the actions that are visually similar but that have never been acquired. One can thereby
: '.,-hich compared control for the observer's level of 'familiarity' with the different movements. This is
:::r responses for precisely what we did in an early study by using dancers that had been trained in differ-
i*se it allows us to ent t)?es of movements: ballet dancers and capoeira dancers (Calvo-Merino et al.
:-...r ro each of the zoo5). Dance offers a promising opportunity to experimentally address this question
:.;irbe the physical for two reasons: (r) it is formed by a well-defined vocabulary of movements that can be
:i. to the aesthetic kinematically controlled; (z) it is easy to evaluate the level of experience that a dancer
or an observer has with each movement. A critical finding was that brain actiüty in
::ri',','hen studying mirror neuron areas, including premotor and parietal cortex, was higher when these
: ra,. been driven dancers watched videos of the dance style that they were familiar with, compared to
:tE -i-isual process- watching visually similar movements of a dance style they had never performed before
:equi¡ed step is to (Calvo-Merino et al. zoo5). This and subsequent studies (Calvo-Merino et al. zoo6;
: nence and its key Cross et al. zoo6) demonstrated that brain actiüty during action observation is modu-
:p:roach, we must lated bythe degree of motor familiaritythat the observer has with the observed action.
534 BEATRIZ CALVO_MERINO AND PATRICK HAGGARD
The more knowledge we have with the observed action, the stronger the internal motor experience. The m¿
simulation in sensorimotor regions while watching the movements. These results also brain regions that p.
show that the action simulation network for watching whole body dance movements presented stimuLi. T.
recruits the action observation network bilaterally, and extends the premotor activa- itly to judge the beau
tion from the initial ventral regions, shown in hand action observation, to more dorsal is there more to aesil
parts of the human mirror neuron system (see Fig. zTtfor a schematic representation We recentlv des:
of the network). aesthetic responses :
out in tr,vo parts. ln ,
tain emotional or cognitive state in order to appreciate beauty? This specific mood for dimensions relatel :
aesthetics has been called'aesthetic attitude'(Cupchik andLászló r99z). As already esting-dull, tense-r(
stated by Kant (r79o11987), an observer needs to be in a certain state to have an aesthetic only the 'liking-d:.--
on this dimensron. I
subject and then ¿';;
participants for e¿.:t
one group containe;
other those rvith th.
of all subjects' rati;:!
a generally higü c,:
consensus scoreS.
Wefoundtrro sr¿,
regions were mo¡e :!
consensus) ther lil+'
aesthetic sensitir'¡ ¿r
and in the premLri¡r ;
for implicit positire :
the opposite cornii:
favoured rather ;i::
response automati;¿.
explain why dance 1s
:r the internal motor experience. The majority of earlier studies in neuroaesthetics aimed to localize the
s. These results also brain regions that participate in making explicit judgements about the beauty of the
r- dance movements
presented stimuli. They induced in subjects an aesthetic mood,by asking them explic-
]e premotor activa- itly to judge the beauty of visual stimuli, while recording their brain activity. However,
¡ion, to more dorsal is there more to aesthetic processing than the active search for beauty?
ratic representation We recently designed a study to investigate the neural correlates of implicit
aesthetic responses to dance (Calvo-Merino et al. zooS). The experiment was carried
out in two parts. In a first session, we measured brain responses in naive participants
with no formal dance experience while theywatched dance movements and performed
to a dummy task (to ensure they paid attention to the stimuli). It is important to note
that no explicit aesthetic question was asked during the brain scan. A range of dance
movements from different cultural backgrounds (classical ballet and capoeira) were
rrselves for intense selected with the help ofa choreographer, to evaluate general responses to dance percep-
¡tional state caused tion (irrespective of style). The selected movements were classified on the bases of four
¡rrnt€-\-t that can be kinematic properties: speed, body part used, direction of movement, and vertical and
.ot'rre like a picture horizontal displacement.
need to be in a cer- In a second session, participants rated the complete set of movements on different
s specific mood for dimensions related to aesthetic experience (liking-disliking, simple-complex, inter-
j rEg:). As already esting-dull, tense-relaxed, weak-powerful) (Berlynery74). From this questionnaire,
:o have an aesthetic only the 'liking-disliking' pair correlated with brain activity. Therefore, we will focus
on this dimension. The ratings for each movement were first normalized within each
subject and then averaged across subjects to create a consensus rating ofthe group of
participants for each movement. Finally, we divided the movements into two subsets:
one group contained the top half movements with the highest scores (more liking), the
other those with the lowest scores (more disliking). We then used this group average
of all subjects' ratings to identify brain areas sensitive to whether they were watching
a generally high or low rating in this aesthetic dimension, as determined by the
consensus scores.
We found two specific brain regions showing significant neuroaesthetic tuning. These
regions were more activated when subjects üewed movements that (on average, in the
consensus) they liked, compared to movements that, on average, they disliked. These
aesthetic sensitive areas are localized in the early yisual cortex, in the medial region,
and in the premotor cortex ofthe right hemisphere. These areas therefore maybe relevant
for implicit positive aesthetic experience of dance. No significant results were found for
the opposite comparison, i.e. when looking for brain regions sensitive to viewing less
favoured rather than preferred movements. The idea of having such a sensorimotor
response automatically triggered while seeing dance movements we implicitly like may
explain why dance is so easily appreciated in so many human cultures.
making any inferences about the stimuli that evoke this response. The consensus
A. l-ligher scores
approach thatwe have described here (for further details see Calvo-Merino et al. zoo8),
does not allow generalizing the result to the general population, as we eliminate
differences between individuals when generating the consensus average. Nevertheless,
it does allow us to identifr stimulus quantities that specifically modulate aesthetically
relevant brain areas ofthe group ofsubjects that participated in this study. In this way,
we identifr which specific dance movements were responsible for maximal and mini- B. Lower scores i
mal activation for the two aesthetically responsive brain areas. Figure z7.z shows an
example of the movements that achieve highest and lowest neural responses for
the occipital area in the left hemisphere and the equivalent stimuli for the right premo-
tor activation. Because movements were selected on the basis of four criteria, we can
now produce a physical description of those dance movements that preferentially
target these aesthetically sensitive areas. This suggested that, on average, these areas .:
Fno
sw
A- Visual cortex
sffi
B. Premotor cortex
gW
PER
Initial conr.ersaiic
There is a need tL-, .:
Once this is done. -;
a common matter
tion of scientists
r:
,_.-
¿l-,,
t
Initial conversations between artists and scientists can resemble the tower of Babel.
There is a need to embrace these different views and synthesize a common language.
Once this is done, ideas can flow and ferfllize the other approach and knowledge for
a common matter of interest: in this case, dance. From our experience, the collabora-
tion of scientists and artists can be achieved in two ways. First, a full collaboration
:s and interaction needs to take place during the entire course of a study. This helps to
address questions of common interest, leading to a common and shared output. While
e ate with the this may be a desired and ideal scenario, in reality it is often difficult to pursue, due to
scaile between the different methodological approaches of each discipline. The second approach is
rernisphere and mostly unilateral, and can work for both artists and scientists. A classic example is
3-second dance when artists benefit from science as a source of inspiration for choreographing new
r sual brain movements or whole performances (see for an example; Walme McGregor, Ataxia,
ation) include zoo4, http:l lwww.choreocog.net/ataxia.html). On the other hand, scientists might
e bottom row use dance or dancers as model participants, in order to address specific experimen-
tally driven questions (this can include dance itself) (Calvo-Merino et al. zoo5, zoo6;
l: , --',¡e-ino et al. 2008) Brown et al. zoo6; Cross et aL. zoo6).In this last type of interaction, the artist-scientist
538 BEATRIZ CALVO-MERINO AND PATRICK HAGGARD
collaboration might be necessary only at some stage of the research. We can briefly
illustrate our personal experience in the collaboration process'
In scientific research as well as in dance production, there are several stages that
might involve people with different abilities and responsibilities. A dance performance
requires the involvement of choreographers and dancers, and an audience to observe
the final product. Our research focus has mainlybeen on dance perception rather than
dance execution. In particular, we studied how neurocognitive mechanisms involved
in observing dance movements are sensitive to different factors, such as the observer's
experience with the observed movement. In order to achieve this, the collaboration
with choreographers and performers was essential. In an initial stage, it was the experi-
menters' role to define stimulus requirements (i.e. kinematic properties, length,
colours, etc.) to test the experimental hlpotheses. This information and the aim of the
study was then discussed with the choreographer, who, being familiar with our experi-
mental framework and approach, made a final selection of movements appropriate for
the experiment, and then guided the dancers while performing the steps in the video
recording session. This process occurred only once at the beginning ofthe research, but
its role in the experiment was essential. Even if the aim of the study was to focus on
the obser-ver (watching a dance), there was an inestimable contribution from the
dance execution section of the performance team, just as in an actual performance for
a live audience.
One interesting difference between science and art is their approach to the relation
between the whole and its parts. Science generally aims to understand a phenomenon
(here a dance performance) by anal¡ically breaking it down into simpler constituent
elements, and investigating each of these elements separately. It is essential to study
each independent component and its contribution to a single process. On the other
hand, artists create a piece by combining together parts or elements (e.g. a dance
performance is a combination of movements, dancers, music, scenario, costumes, etc.).
The result is a performance that represents the interaction of these elements rather
than just their sum. In our studies, we reduced dance movements to their minimal
expression, by using short video clips and minimizing variability between dancers'
R.rrRE\C
bodies and costumes. In our previous study (Calvo-Merino et al. zoo8), we showed
how different movements elicited liking responses and neural responses in different
ways. Because the stimuli differ only in their kinematic properties, one can consider
each video clip as an independent movement unit. It would be interesting to see how
choreographers collect these movement units and pull together a whole performance.
For example, can a choreographer use this material to develop a dance performance
dla carte,and thereby stimulate specific brain regions required for aesthetic experience?
One could then create a performance that intentionally and effectively stimulates differ-
ent levels of the aesthetic processing such as sensorimotor, cognitive, and hedonic
or emotional.
Finally, and crucially, dance would hardly survive without an audience. In recent
decades art has evolved incredibly fast, and the role of the obser-ver has evolved from
that of a mere passive spectator to being a key interactive ingredient of the piece (for
a fecent example see Antony Gormley'Fourth Plinth Commission', entitled One and
NEUROAESTHETICS OF PERFORMING ARTS 539
Cela-Conde CJ, Marty G, Maestu F, et aI. (zool. Activation of the prefrontal cortex in the Pa¡adiso S, Johlsc: t
human visual aesthetic perception. Proceedings of the National Academy of Sciences of the riith attribuiron ¡:,
United States of America, ror, 63zt-5. a PET studv o:::-
Cross ES, Hamilton AFDC, and Grafton ST (zoo6). Building a motor simulation de novo: Petrides Nf irooc . Ti
observation of dance by dancers. Neuroimage,3t, tz57-67. Exp e rim err: a! 3 -,,.: -.
Cunningham WA, Johnson MK, Gatenby IC, Gore JC, and Banaji MR (zoo3). Neural compo- Rizzolatti G and t::r.
nents of sociai evaluation. Journal of Personality and Social Psychology, SS, 6ZS-+g. sciertce, 27, 169-9 -
Cupchik GC andLászIó I Oggz) . Emerging Visions of the Aesthetic Process Psychology Semiology Rolls ET (zoooa . ?:.;
and Philosophy. Cambridge University Press, Cambridge. Rolls ET (zooob . Th¿
Di Pellegrino G, Fadiga L, Fogassi L, Gallese V, and Rizzoiatti G (1992). Understanding motor Small D\f, Zatorr¿ Li
events: a neurophysiological study. Experimental Brain Research, gt,176_,8c. actir-in'relatec r¡ .:
Gallese V, Fadiga L, Fogassi L, and Rizzolatti G (1996). Action recognition in the premotor \-artanian O ari G,-:
corfex. Brain, Í9, 593-609. paintings. -\t:.:..-i:
Grafton ST, Arbib MA, Fadiga L, and Rizzolatti G (tqq6). Localization of grasp representations Zaionc RB {196S . .1.
in humans by positron emission tomography. z. Observation compared with imagination. Ps1'cholog1 , 9. ',-:-
Experimental Brain Research, 112, 103-11. Zeki S and Lamb I'l ::
Grézes J and Decety I (zoor). Functional anatomy of execution, mental simulation, observa- .lD.
Zeki S, \\ratson --
tion, and verb generation of actions: a metaanalysi s. Human Brain Mapping, t2, 1-r9. demonstratitrn -: :-
Iidaka T, Okada T, Murata T, et al. (zooz). Age-related differences in the medial temporal lobe tt,64r-9.
responses to emotional faces as revealed by fMF.l. Hippocampus, 12,352-62.
lacobsen T (zoo4). Individual and group modelling of aesthetic judgment strategies. British
Journal of Psychology, g5,, 41-56.
]acobsen T and Hofel L (zooz). Aesthetic judgments of novel graphic patterns: Analyses of
individual judgments. Perceptual and Motor Skills, 95, 7 55*66.
|acobsen T, Buchta K, Kohler M, and Schroger E (zoo4). The primacy of beauty in judging the
aesthetics of objects. Psychological Reports, 94, tz53-6o.
|acobsen T, Schubotz RI, Hofel L, and Cramon DY (zoo6). Brain correlates of aesthetic judg-
ment of beauty. Neuroimage, 29, 276-85.
Kant I (r79oh987). Analy'tic of the beautifuLln Critique of ludgment,WS Pluhar trans., Hackett
Publishing Co.
Kawabata H and Zeki S (zoo4). Neural correlates of beauty. Journal of Neurophysiology, 9t,
1699-7o5.
Kirk U, Skov M, Christensen MS, and Nygaard N (zoo8). Brain correlates of aesthetic exper-
tise: a parametric fMRI sttdy. Brain and Cognition,69,306-75
Leder H, Belke B, Oeberst A, and Augustin D (zoo4). A model of aesthetic appreciation and
aesthetic judgments. British Journal of Psycholog,95, 489-5o8.
McManus IC (r98o). The aesthetics of simple figures. British Journal of Psychology, 71, 5c5-24.
McManus IC and Weatherby P OSSZ).The golden section and the aesthetics of form and
composition: a cognitive model. Empirical Studies of the Arts, SQ), zo9-32.
Moutoussis K and Zeki S (zooz). The relationship between cortical activation and perception
investigated with invisible stimuli. Proceedings of the National Academy of Sciences of the
United States of America, 99, 9527-32.
Nadal M, Munar E, Capó MA, Rosselió |, and Cela-Conde Cf (zoo8). Towards a framework
for the study of the neural correlates of aesthetic preference. Spatial Vision, zr, 379-96.
O'Doherty |, Winston J, Critchley H, Perrett D, Burt DM, and Dolan RJ (zoo3). Beauty in
a smile: the role of medial orbitofrontal cortex in facial attractiveness. Neuropsychologia,
4L,147-55.
NEUROAESTHETICS OF PERFORMING ARTS 541
:¿f.ins: Analyses of
-,',r;rds a f¡amework
:-r:--"r.21, 379-96.
L :r:ro-¡). Beauty in
;.. ).,.: u r op sy cholo gia,
CHAPTER 28
MULTISENSORY
AESTHETICS IN
PRODUCT DESIGN
The sensory aspects of the normal human being should be taken into
consideration in all forms of design. Let's take the Coca-Cola bottie, for
instance. Even when wet and cold, its twin-sphered body offers a delightful
valley for the friendly fold of one's hand, a feel that is cozy and luscious.
Loettvy t95t, p. 297
INrnoDUcrroN
Tnn quote from Loewy (r95r) provides a perfect example of sensuous delight or how
a product can gratify the senses, in this case the sense of touch. Analogously, the smell
of your favourite soup in a three-star restaurant makes your mouth water, a glimpse
of your new pair of Prada shoes makes you fall in love with them all over again, and
hearing the hear,y sound of a car door closing behind you instantly gives you the com-
fortable feeling of being safe and secure. Apparently, consumer products can evoke
feelings of intense enjoyment in multiple ways and through multiple sensory modali-
ties. In our view, all the senses can contribute to pleasant experiences (Suzuki et al.
zoo6; Fenko et al. 2010) and all such experiences are aesthetic. 'Aesthetics' can
thus be defined as the pleasure attained from sensory perception (see Hekkert zoo6;
Hekkert and Leder zoo8), a definition in line with the notion of the eighteenth-century
544 HENDRIK N.]. SCHIFFERSTEIN AND PAUL HEKKERT
philosopher Baumgarten, who used the term aesthetics in terms of 'gratification of its aesthetic evalu¿t-,
the senses' or'sensuous delight' (Goldman zoor). they appreciate htr','
The tipes of stimulation that are most powerful in gratifting the senses seem or how it tastes. In
to differ between sensory modalities. For instance, auditory stimulation is probably whole, which requl¡,
enjoyed most through listening to music. In the tactual domain, the pleasure attained The studies br I
from touching is felt strongly in a massage or in sexual activities. Visual beauty can be objects is compared
enjoyed in a van Gogh painting, but also in viewing a natural landscape outdoors. outcomes of a r-isu¿
Furthermore, olfactory stimulation is tlpically enjoyed when smelling fragrances or aesthetic evaluatiLrr:
luxury personal care products, while tasting and smelling in concert with all other authors found be¡,,-
modalities provide enjoymentwhen a culinaryspecialtyis consumed. These examples (lack of) inter-rela:r'
illustrate that some objects (e.g. paintings, music) seem to be specifically'designed'to erties of the obje¡s
evoke aesthetic pleasure through a single modality, whereas other objects (e.g. foods) different modalities.
can evoke pleasure through multiple modalities. Despite the differences between the The present char
objects that gratif' the senses, the mechanisms involved in evoking pleasure may be rules relating perce!
Iargely shared among modalities. In this chapter, we try to provide descriptions of different forms. Fr:s
mechanisms that underlie aesthetic experiences in all the separate sensory modalities. We can wonder ab.- ;
In addition, we describe mechanisms that operate when sources of sensory stimulation these rules are moda
in multiple modalities together contribute to an aesthetic experience. on which the aesú:
In the latter case, the experiences evoked in the different modalities are not necessarily governing these er'¡,t
aesthetically similar. Nonetheless, Lindauer and colleagues found that aesthetic reac- if the propertv cen:l
tions to sculpted objects were largely comparable when experienced through vision or the auditon. proi\ei_
touch (Lindauer 1986; Lindauer et al. rg86). According to these authors, a sculptor who other modalities i: ;
is working on an object does not only optimize visual input, such as colour and pat- Second, there are ru
terning, but also the unique tactile characteristics ofthe object, such as its coldness, conditions under',,;i
roughness, and weight. The authors suggested that since these two sensory modalities experience of the rt'i
share the general function to extract, pick up, and organize information about the
world, the processing of visual and tactual information and, consequently, the aes-
thetic appreciation, is interrelated, holistic, and essentially amodal. This could suggest
that aesthetic quality is mainly inherent to the physical properties of the object, and is
hardly dependent on the sensory modality through which these properties are per-
ceived. However, comparing vision and touch may be a special case, because several
objectfeatures (e.g. shape, size, surfacetexture) canbedirectlyperceivedthroughboth f-obi*t -
I F""tur" 'l
modalities and, therefore, the aesthetic judgements are also partly based on the same t_
input. Aesthetic reactions to objects may not be that closely connected when sensory
modalities are involved that use very different indicators of the same property. For
I _t*=
FearL.e
II
I
Fea:-
instance, the ripeness of an apple canbe evaluatedbased on its colour (vision), its hard-
ness (touch), and its sweetness (taste), but these three perceived properties are derived ,"..-*
from distinct product features. Feat-"
Figure z8.r tries to summarize the relationships between the properties of an object I
and the different tlpes of aesthetic evaluations that can be made of the object. The
physical object contains a number of physical features (e.g. the type and amount of
light it reflects, its three-dimensional shape, the volatile odorants it releases, its tem-
perature, its thermal conductivity) that can be perceived through the different senses. Fig. 28.1 Relationsi
These perceived properties (e.g. its colour, weight, smell, warmth) form the basis for properties, and aestl
MULTISENSORY AESTHETICS IN PRODUCT DESIGN 545
s ot 'gratification of its aesthetic evaluation. For instance> we can ask people to indicate the extent to which
they appreciate how the product iooks, how it feels, how it sounds, how it smells,
:r{ the senses seem or how it tastes. In addition, we can ask them how much they like the product as a
,-rlation is probably whole, which requires integration over different sensory modalities.
:. pleasure attained The studies by Lindauer and colleagues described earlier, in which liking for
,-r.ual beauty can be
objects is compared using different sensory modes, concern a comparison between the
:rdscape outdoors.
outcomes of a visual aesthetic, a tactual aesthetic and a multisensory visual-tactual
r; ,ing fragrances or aesthetic evaluation process. However, the differences and commonalities these
:¡ert rvith all other authors found between conditions may have different sources: They may stem from
:..1. These examples (lack of) inter-relationships between physical object features, the ways in which prop-
::;all,,.'designed' to
erties of the objects were perceived, the rules relating perception to liking for the
:':iects (e.g. foods) different modalities, or from the ways in which people reported their experiences.
::.nces between the
The present chapter aims to further explore one of the proposed options, i.e. the
::i ¡leasure may be rules relating perception to liking for the different modalities. These rules may take on
,Je descriptions of
different forms. First of all, some rules can be defined for a specific sensory modality.
,.nsorv modalities. We can wonder about the mechanisms that underlie these rules, and the extent to which
::nsorvstimulation these rules are modality-specific or applicable to all the senses. Although the properties
;:. on which the aesthetic evaluation is based may differ over modalities, the principles
:: :re not necessarily
governing these evaluations may be uniform. This can, of course, only be determined
::r¿t aesthetic reac-
if the property central to the rule is not modality-specific. For instance, an effect of
,: :hrough vision or
the auditory property, loudness, on aesthetic appreciation can only be transferred to
r-:..asculptorwho other modalities if an equivalent of loudness can be found within these modalities.
:. ;olour and pat- Second, there are rules that involve multiple sensory modalities, and that describe the
-,: as its coldness,
conditions under which inputs from multiple sensory modalities enhance the aesthetic
:.nsorJ¡ modalities experience of the whole product.
::--t¿tion about the
..qrently, the aes-
- :-rs could suggest
, - :,e object, and is
Aesthet¡cs
l:L,i:\erties are per-
Visual
... because several aesthetics
.-'. ;d through both Object
Feature 1
Tactual
--,;.ed on the same aesthetics
--..d rr-hen sensory Auditory
.r: property. For
: '.rston), its hard- Chemosensory
::::i€s a¡e derived Feature 5 aesthetics
Feature 6
Chemosensory
:::,r.s ofan object
. : ,,:e object. The Multisensory
:. :nd amount of aesthetics
: :.-:ases, its tem-
:- i.:r'erent senses. Fig. 28.1 Relationships between physical object features, perceived object
: - :::.r the basis for
properties, and aesthetic evaluations of such properties.
546 HENDRIK N.J. SCHIFFERSTEIN AND PAUL HEKKERT
SrnucruRAL PRopERTTES
The way in which people perceive and structure information may be partly linked to
the sensory modality through which it is perceived. For instance, the way in which
people react to olfactory stimulation is quite different from how they react to visual
stimulation. Some fiction writers, film makers, and fragrance advertisers like to make
the public believe that smells may be very powerful in unconsciously influencing
people's preferences and behaviour. For example, at the end of Patrick Süskind's
book Perfume, the main character makes a crowd of people lose all their sexual restric-
tions by dispersing a special perfume in the air. Although such a vehement reaction to
a smell seems highly unlikely to occur in reality, empirical evidence on the effects
offragrances does show that people react instantly to the pleasantness or the unpleas-
antness of smells by an approach or avoidance reaction. However, for smells that
MULTISENSORY AESTHETICS IN PRODUCT DESIGN 547
t). because these are are hedonically neutral to positive these effects are rather subtle. Although
:a.ts that grati$z only pleasant smells can influence behaviour in ways that are to some extent predictable
,¡-,' . Although some (Bone and Ellen t999), it seems highly unlikely that behaviour can ever be steered
"s. to be beautiful to through olfaction.
: ¡esthetically appre- As regards sexual attraction, it has been found that odours can inform people
:1106 ). However, for about a person's health, fitness, and fertility (e.g. Doty r98r). whereas some of these
¡,luce have not been smells are more or less universally attractive or unattractive, the appreciation of
user-product inter- some smells seems to be part of a very specific mechanism related to the immuno-
'
¡,,lour leyels because competence of two partners. Females might be able to use a male's odour to obtain
: the quality of these disease-resistant genes for their offspring by mating with males carrying dissimilar
r<: of studies on the genes of the major histocompatibility complex, which is part of the body's defence
-:mited. Therefore, mechanism against parasites (wedekind and Füri 1997). The existence of such a
::rstic artefacts (e.g. mechanism suggests that it might be possible to use specific odorants to increase the
l: specific aesthetic aesthetic appeal ofparticular objects, at least for certain individuals.
, equir.alent studies For the auditory modality, mathematical equations have been developed that
estimate the sensory pleasantness of a sound based on several perceived sensations,
,ed on the structural which in their turn can be estimated using algorithms employing psychoacoustical
. : ""tperties aftached measures (e.g. Aures 1985; Fastl and Zwicker 2oo7,p.245). These models showthat
:u:al stimulus prop- sensory pleasantness increases with decreasing loudness, sharpness, and roughness
: ¡LrnsequenCe, the and increasing tonalness (the relative amount of harmonic structure) of sounds (van
si--,¡ 1r'i11 be mainly Egmond zoo8). In music, consonant chords tend to be preferred to dissonant chords.
:.r",-applv to stimuli When a harmonic structure is judged to be consonant, the ratio of vibrating frequen-
::.e¡-itr'. Principles cies tends to be at a minimum (e.g. Butler and Daston 1968). The pleasantness of the
:i:c object features sound is an important characteristic for the design of alarm signals, e.g. for medical
"s ,,n'e discuss include equipment, because medical staff have been known to disable or silence alarms if they
consider them to be too irritating (e.g. Meredith et al. 1999).
Given that the examples described above are very specific for the way in which
sensory information is processed, we may assume that some of the principles that will
be described later, and that were mainly derived from research on the visual modality,
cannot be transferred directly to the other modalities. Nevertheless, other principles
may operate in equivalent ways for the other senses. Therefore, we considered it
interesting to explore if certain aesthetic (visual) principles hold across the senses,
and can be regarded as 'universal'principles.
rr parrl\¡ linked to The way in which people perceive the structure of an object depends on the ways in
. :i13 \ra\. in which which perceptual information is organized. The world is loaded with information and,
-f r'i react to visual therefore, people must be able to detect order in the chaos in order to make sense ofthe
: -rsers like to make world. As a consequence, people tend to see things that are close together or things that
r.,'dsh- influencing look, sound, or feel the same as belonging together. The principles of perceptual orga-
Pat¡ick Süskind's nization have been summarized in the gestalt laws. On the basis of gestalt laws, several
:-leir serual restric- structural stimulus properties or grouping principles (e.g. similarity, balance, har-
reirent reaction to mony, unity in variety) can be defined that have been shown to positively affect the
¡r.e on the effects aesthetic appreciation of objects (e.g. Ramachandran and Hirstein 1999; Hekkert and
3ss or the unpleas- Leder zoo8). However, in order to be able to organize a number of stimuli, people first
::. lirr smells that need to detect the different elements and determine whether they are similar or not.
548 HENDRIK N.J. SCHIFFERSTEIN AND PAUL HEKKERT
Detection will improve when elements stand out from their background and, there-
and temperature .ra
fore, the perception of both contrast and similarity are important prerequisites for
or warming). Accc,:
number of instan,e s
the organization of perceptual elements. Therefore, we will start out by discussing the
seems to play a do:r:
role of contrast and similarity in multisensory aesthetic judgements, and we continue
the melting of ice-c:,
with other structural stimulus properties that seem to be common across the senses:
texture to a soft a:;
balance, symmetry, proportion, complexity, and unity in variety'
sensitive to the t¿s:'
over the tongue. Fur
matrix. Combining:,
Contrast late is likely to inc:;.
for perceiving con:r.
A characteristic that helps in perceptual grouping is contrast' Contrast helps to detect
are critical in enio.,::
dissimilarities between features or objects that are close, but do not belong together'
func- Think, for instanc:.
Such a principle is widely applied in the design of interfaces where buttons of
of meat, the thickr,
tional similarity are clearly contrasted from buttons holding other functions by
crispiness of freshl..' r
differences in shape or colour (see Fig. z8.z). Moreover, objects that stand out
against
and Hirstein (1999) delicious: thev lose :
their backgroundwill be more easily recognized. Ramachandran
blender, which tur:s
suggest that contrast extractions may be intrinsically pleasing to the eye,
because
ksround and, there- and temperature changes) and the conditions in the oral cavity change (e.g. cooling
::rt prerequisites for or warming). According to these authors, highly palatable foods produce a large
:,-¡t 'ov discussing the number of instances at which sensory contrast can be perceived. The sense of touch
:ts. and we continue seems to play a dominant role in perceiving many of these contrasts. For instance,
- I across the senses: the melting of ice-cream in the mouth includes a transition from a hard and ice-cold
texture to a soft and creamy texture. In addition, cooling the tongue makes it less
sensitive to the taste of sucrose, producing local sweetness sensitivity differences
over the tongue. Furthermore, the melting releases tastants and odorants from their
matrix. Combining ice-cream with a crunchy waffle, nuts, caramel, or pieces of choco-
Iate is likely to increase its palatability, because it produces additional opportunities
i:rast helps to detect for perceiving contrasts. Analogously, contrasts in appearance, flavour, and texture
l.¡t belong together. are critical in enjoying the different elements of a dish or a meal (e.g. Lawless zooo).
r:e buttons of func- Think, for instance, about the differences between the juicy, elastic, fibrous texture
'¡iller functions by of meat, the thickness and lumpiness of mashed potatoes, and the firmness and
stand out against
.:.r- crispiness of freshly cooked vegetables. The contrasts between these foods make them
and Hirstein (1999) delicious: they lose most of their appeal if you mix them and miniaturize them in a
the eye, because
IL-r blender, which turns them into a single, homogeneous mass.
ruch information.
nore pleasing.
sensory domains.
rst highly palatable
Similarity
rlanipulated in the According to the gestalt principle of similarity, similar stimuli tend to be grouped
mrüng with saliva, together. Figure z8.z akeady showed that buttons related to similar functions have
the same shape and colour, thereby facilitating perception and use. This perceptual
grouping seems to occur spontaneously for stimuli perceived with the same sensory
modality. Within a single modality, people find it relatively easy to determine whether
two stimuli are similar or not (e.g. Thurstone ry27;Farellry85).
However, to what extent do people also observe similarity between stimuli that
impinge on different sensory systems? Generally speaking, people tend to connect
stimuli that perform similarly on the dimensions that underlie sensory perception in
all modalities: location, time, and intensity (e.g. Boring ry42).For instance, when you
hear a person talking and see his lips moving in the same pattern (temporal synchrony),
and you hear the voice coming from the same location as where you see the lips (loca-
tion similarity), you are likely to conclude that the voice comes from the moving
mouth.'In the latter case, the covariation in the temporal pattern is a more powerful
conjunction than the apparent location of the two sources: ifyou are watching a movie
in the cinema, the voices will come from loudspeakers at the side, but you will still per-
ceive the voice as coming from the actor's mouth. This is the so-called ventriloquist
illusion (e.g. Bertelson 1999).
Furthermore, people are likely to classifz low-intensity stimuli (e.g. a dim light
and a soft tone) and high-intensity stimuli (e.g. a bright light and a loud tone) in
ing that buttons
ler) and contrast
' In fact, the gestalt law of similarity is equivalent to the gestalt law of proximity, if 'similar' is
e rnenu). interpreted as similar in time or location, respectively.
55O HENDRIK N.J. SCHIFFERSTEIN AND PAUL HEKKERT
separate groups. This probably occurs because in real life, small natural objects (piants,
animals, rocks) are likely to be lighter and to produce less noise than larger objects.
Apparently, within a natural environment natural covariation exists on the intensit,v
dimensions (a phenomenon called cue redundancy in studies using the Brunswik lens
model, e.g. Hogarth and Karelaia zooz). This correspondence lies at the basis of psy-
chophysical scaling techniques, such as cross-modality matching (e.g. Stevens and
Marks r98o). Moreover, even for modality-specific dimensions people may find some
connections between stimuli intuitively more applicable than others, such as for the
dimensions of colour and odour quality (Schifferstein and Tanudja ja zoo4; Dematté
et aL. zooT) .
Possibly, people may also base similarity judgements partly on semantic informa-
tion. For instance, environmental sounds are generally grouped not only on the basis
ofacoustical features, but also on a priori knowledge ofthe sounds and the events that
produce them (Gygi et al. zooT;Aldrich etal. zoog;Ózcan etal. submitted). Analogouslr'.
the similarity of smells seems to be partly determined by whether they originated from
a fruit, a flower, an animal, and so on (Chastrette et al. t988; Chrea et ai. zoo4, zoo5 ).
However, the connections between elements in cross-sensory groups are bound to be
weaker than in unisensory simiiarity groups.
iatural objects (plants, Hekkert et al. ry94). At best, ratios close to the golden section seem to be preferred
than larger objects. over other ratios, but the empirical findings could easily be due to a range effect in
'e
rlists on the intensity the studies (Godkewitschryzd or to an effect of averaging (Plug r98o). Next to ratios
,:ng the Brunswik lens
in the vicinity of the golden section, the square was often found to be a preferred
::s at the basis of psy- ratio (McManus r98o). Nevertheless, the golden ratio remains a proportion of unprec-
:re (e.9. Stevens and edented appeal and many see this ratio reflected in the organization of, and responsi-
reople may find some ble for, great works of art and classic design pieces, such as the Chaise Longue by
,thers, such as for the
Le Corbusier (Fig. 28.3) and the Volkswagen Beetle car (Fig. 28.+).
udiaia zoo4; Dematté
: semantic informa-
n.trt only on the basis
is ¿nd the events that
::led ).Analogously,
l:: e.,- originated from
rs¿ et al. zoo4, zoo5).
, :ps are bound to be
Fig. 28.3 Chaise Longue designed by Le Corbusier, 1929. The proportions of the
-¿- art. Ifan original
chaise relate to the harmonic subdivisions of a golden rectangle. The width of
:r¿in elements or by
the rectangle becomes the diameter of the arc that is the frame of the chaise.
-:: r,leference ratings
The stand is in direct relationship to the square in the harmonic subdivision.
¡.iual balance when
',:,uate the composi- Reprinted with permission of Princeton Architectural Press, from Elam (2001).
-'r::hony. However,
r: j0 not necessarily
.,- i of balance in the
-,-.,r auditorystimu-
r:r Lria composition
, : ;he r-a¡ious instru-
-,
/*#
//.:
j./
, n'-.m
-.--l
l ¡l
ffi
:r-i mav be mainly
: .rrlte¡ent elements fr#,tr
:: :he elements of a
¿.-e', edthatagolden
. ,- ¡tained when the
l a -r¡ a cross, equals
- -,: lhis ratio, often Fig.28.a Volkswagen Beetle car designed by Jay Mays, Freeman Thomas, and Peter
-:r t:s mathematical Schreyer, 1 997. The body of the car fits into the upper half of a golden ellipse,
.si¡ientlr. supported inscribed in a golden rectangle construction diagram. Reprinted with permission
-: lti¡\\-hinniery87; of Princeton Architectural Press, from Elam (2001).
552 HENDRIK N,I. SCHIFFERSTEIN AND PAUL HEKKERT
Nevertheless, studying properties in isolation may tell us little about the effects of
these properties in the context of art and design objects. Hekkert (t995) found that
aesthetic preference for particular rectangular proportions highly depended on the
tlpe of object the rectangle represented, such as a window, a cabinet door, or a bath-
room tile (see Fig. 28.5). More interestingly, preference was linearly related to the rated
{
commonness of the proportion, a measure of familiarity.
What a good proportion is, is likely to be modality-specific. For instance, for the
tactual modality a product's proportions are good if the product fits well with the
human body (see Sonneveld and Schifferstein zoo8). Clothes and shoes fit well if
they are not too big, nor too small. Tools should nicely fit in the hand and provide a
steady grip. A chair should support the body in all the right places. For the auditory
modality, on the other hand, the right proportions may be determined by the effect of
object shape on the properties of the sounds that are produced. Musical instruments
are typically designed to optimize the quality of the sounds they produce.
Besides being a matter of proportion, the perceived balance in a design also depends
on the degree of symmetry. Locher, Smets and Overbeeke (1995) compared the
perception of balance in vision and touch. They performed non-metric Euclidean
multidimensional scaling analyses on balance rank orders derived from sorting tasks
and rank order tasks for 4 clay objects. These authors found several differences
between the visual and the tactual stimulus configurations: whereas visual balance
appears to be a holistic object property, which results from the synthesis of component
stimulus information, for touch balance implies a symmetric distribution of weight/
shape about the central axis (see Fig. 28.6). Hence, balance in touch is closely linked
to s).rnmetry, while balance in vision is not.
Nevertheless, symmetry can enhance aesthetic appreciation in visual perception
as well: symmetrical faces are preferred over non-symmetrical ones (Grammer and
Thornhill ry94') and symmetrical abstract patterns are often seen as more beautiful
(Jacobsen and Hófel zoo3). In the case of human faces, this preference is possibly
related to physical frtness, i.e. synmetrical faces signal health and the absence of
parasites. On the other hand, when it comes to patterns or objects symmetry may be
appreciated because it facilitates perception and ease ofprocessing (Reber et al. zoo4).
Window (1/1) Picture frame (8/5) Cabinet Book (4/5) Bathroom tile (4/3)
door (1/2i
According to Berlpre's (r97r) collative-motivation model, objects are preferred for 'Unity in variet-,-
their ability to generate arousal. Objects with low arousal potential, i.e. very orderly principle, accordins
and simple, are not stimulating and leave the observer indifferent; patterns with very of as few elements o
high arousal potential, i.e. very complex and diverse, are too difficult to process and are of problems or pher
considered unpleasant. Preferred are patterns with an arousal potential at a medium to this principle rnh
(or optimum) level,leading to the famous prediction ofan inverted U-shaped function shuffle, an MP3 pla
for hedonic tone (pleasantness) as a function of arousal potential. Designers should a USB connection tc
thus strive for a perfect balance between order and complexity or unity in variety, a tion, but no displar
result that seems universally appealing. Although the inverted U-function could not designedbythe Sui
always be demonstrated in empirical studies (e.g. Frith andNias 1974; Walkerrg8o), the a winery in the hills
general principle seems very robust: the greatest beauty is achieved by as much variety took the rocks scatt
or complexity as possible with a maximum of unity or order. Examples could be drawn for the constructior
from all design domains, but this principle is probablymost easilywitnessed in the field at various ler.els: tl
ofgraphic design (see Fig. 28.8). construction is r-enu
Complexity can be increased in any modality by using more or alarger variety of Fig. z8.ro).
elements. For instance, the complexity of music can be enhanced by adding more
instruments or by playing more notes in the same time frame. In meals, complexity can
be increased by adding more dishes or more ingredients. For these sensory domains, it
can be easily deduced that increasing complexity can have an adverse effect on con-
sumer appreciation. For instance, music can become extremely restless if too many
events occur within the same time frame. Or in a dish the specific character of the main
ingredient may be lost if too many spices or garnish elements are added.
Fig. 28.8 Set of posters developed by Fabrique for the Holland Dance Festival,
expressing'unity in variety'in many different ways. For example, the text in I
each poster is restricted to specific locations (unity), but several different font Fig. 28.9 The App
types and sizes (variety) are used for different types of information. (2008), copyright t
MULTISENSORY AESTHETICS IN PRODUCT DESIGN 555
rcts are preferred for 'Unity in variety' is a special case of the 'maximum effect for minimum means'
¡ial, i.e. very orderly principle, according to which people prefer solutions, ideas, and formulas that consist
t: patterns with very of as few elements or parameters as possible, while solving or explaining a large range
rlt to process and are of problems or phenomena (e.g. Boselie and Leeuwenberg t985). Designers often refer
rtenlial at a medium to this principle when they resort to 'minimal solutions', as exemplified by the iPod
J U-shaped function shuffle, an MP3 player in a tiny white box that only has a connector for an earplug,
an. Designers should a USB connection for battery power and uploading songs, and a click wheel for naviga-
rr unity in variety, a tion, but no display (Fig. z8.q). Another typical example is the building in Fig. z8.ro
-runction could not designed by the Swiss architects Herzog & de Meuron. \^/hen commissioned to build
1: \\'alker r98o), the a winery in the hills of California that would be one with its surroundings, they simply
I br-as much variety took the rocks scattered in the vicinity of the location and put these into steel frames
Lples could be drawn for the construction of the building. This minimal design choice had a great effect
u:tnessed in the field at various levels: the building fits naturally in the landscape and, even though the
construction is very solid, it allows a poetic rh¡hm of lights to enter the interior (see
rr a larger variety of Fig. z8.ro).
¿c br.adding more
eals. complexity can
oenson. domains, it
'"-erse effect on con-
:esLness if too many
h.¿¡acter of the main
ji_r
.r-tüCü-
Dance Festival,
:, the text in
difterent font Fig. 28.9 The Apple iPod shuffle. Reprinted with permission from Hekkert ft Leder
:rion. (2008), copyright Elsevier, Amsterdam.
HENDRIK N.]. SCHIFFERSTEIN AND PAUL HEKKERT
Depending on the e:
attributes on rvhicl
people tend to attri
in recent experime:
meanings as if the,,
et al. zoo8). Logica-
familiar does not d
resides in the cogni
the principles rve d-
may operate in a sir
Familiariry, pr
of recognition
In r968 Rober¡ Zal'
thetic appreciaticn
characters or pseli(
with increasing rep
a general princip-e
people we knors, ir-
car we own. Preter:
leads to safe choice
perceptual and coE
is intrinsicallr- plea,
but because it is a :
In order to reaL:,
which share som. :
Depending on the extent to which people have overlap in their backgrounds, the formal
attributes on which these meanings are based may be quite consistent. As a result,
people tend to attribute the meaning perceived to these characteristics. For example,
in recent experiments on the meaning of materials, participants talked about material
meanings as if they were talking about their technical/sensorial properties (Karana
et al. zooS). Logically, however, the degree to which something is perceived as novel or
familiar does not depend on the presence of an attribute as such. Because meaning
resides in the cognitive interpretation of the stimulus, there is no a priori reason why
the principles we discuss in this section should be modality-specific. Therefore, they
may operate in a similar way for all the sensory modalities.
peak shift, female birds that prefer to mate with males with bright rather than dull
plumage will show even greater preference for males with supernormal or above-
average brightness' (Martindale 7990, p.47). Analogously, designers apply the peak
shift principle in products to ampliS' the 'very essence' (Ramachandran and Hirstein
1999) or defining characteristic of the category, such as the quality of aerodynamics
in car design (see Fig. z8.u).
Familiarity and prototlpicality play an important role in the recognition of
objects, animals, and people. During the interaction with an object, the object pro-
vides sensory feedback: it provides information about itself and the physical world
surrounding it. Gathering this sensory information is important in order to recognize
the object at a future occasion.
With the information the object supplies, it can guide people in what they are trying
to achieve, but it can also mislead them. The way in which objects give tactual feedback
may be experienced as the integrity or honesty of the object (e.g. Sonneveld and
Schifferstein zooS). For example, porcelain cups provide information about the tem-
perature ofthe coffee inside, whereas polystyrene foam cups do not. Therefore, people
may feel fooled when the coffee in a polystyrene cup is much hotter than expected.
Furthermore, touch screens do not let the user feel what they are actually doing,
whereas other interfaces, such as steering wheels of cars, let you know exactly what is
going on. In architecture, the theme of honesty plays a central role in the Brutalism
style. Brutalist buildings expose the materials they use, their structure, and the services
they offer to humans in the interior or exterior of the building (Fig. z8.rz). Because
of its exposition of construction materials, the Brutalist style is often referred to as Fig. 28.12 The Aula
'the celebration ofconcrete' (e.g. Banham 1966). style.
Whether an object is perceived as honest or not maybe more important for interac-
tions involüng the proximal senses touch and taste than for distal senses, because the
former sensory modalities require direct contact with objects to enable perception. Therefore, intera¡r
injuries, as is eriden
predict that peopic
feedback througi :
products, l.here rn¡
':*1%¡i:r:,.r:,:rrrr:,,r.i.::.ri.r.ri:,' on the person's }:ri
sources ofnutrienrs
In addition. ner,
are toxic taste or sn:
carbohydrates, prLrri
this reason, the che:
However, beside. i:
adequate nutrition
vitamins and mirJ
cases specific nutrl
missing nutrient o',',
nism does not help
Fig. 28.11 The Chrysler Aviat. (Courtesy Chrysler company.) but in finding tbod-;
MULTISENSORY AESTHETICS IN PRODUCT DESIGN 559
the recognition of
'iect. the object pro-
1 the physical world
¡ order to recognize
L
rnany,J
cases specific nutritional needs may lead to a preference for food containing the
missing nutrient over the deficient food (P Rozínt972, 1976). Hence, the latter mecha-
nism does not help in identifying objects or finding foods that are edible in general,
but in finding foods that match one's specific, momentary dietary needs.
560 HENDRIK N.I. SCHIFFERSTEIN AND PAUL HEKKERT
The importance of recognition is evident also in the way people react to events in A good eramp-e
which several sensory modalities are stimulated. People seem to prefer objects and of remixes of old sc,
sensory stimuli that communicate similar-or at the least non-conflicting-messages less conversation'
through all modalities (Lindstrom zoo5). Consistency of impressions will lead to ele- that the actual suc;
vated identification accuracy (Zellner et al. r99r). The congruence of sensory messages with intermediate r
in product design is desirable from an ergonomic perspective, where coherence helps over high melod:;
to clarift what a product is about and what it can do. In addition, perceived unity in than anxiousir-prc
visual stimuli has been shown to correlate with ratings of both aesthetic appeal and proved most suc.ei
liking (Bell et al. r99r; Yeryzer and Hutchinson 1998). Therefore, if product designers In the food cc,:
would like to express an idea, to elicit a specific experience among its users, or to com- introduction of a r:
municate a personality (Govers et al. zoo4), they may benefit from adapting a multi- combination of se;
sensory design approach (Schifferstein and Desmet zoo8). By employing multiple cuisines involr-e ::
sensory channels and by transmitting a consistent message through all these channels, as the tomato-g::
the product experience is likely to be more involving and more overwhelming than if wine-ginger mir:u
only a single modality is used. as flavour principl,
help to bridge the ;
(Stallberg-\\-hite a:
Novelty
Despite people's preferences for familiar products, they are also attracted to new,
Challenges, sr"
unusual and innovative products, partly to overcome boredom and saturation effects
(Martindale r99o). It has been proposed that such a preference for novel instances is While interactin. r
an adaptive trait, in that novelty facilitates learning (e.g. Bornstein r9g9). It allows In fact, recent the.r l
attractive designs comprise a thoughtful balance between novelty and tlpicality. enjoy experiencini
MULTISENSORY AESTHETICS IN PRODUCT DESIGN 561
ptrereact to events in A good example of the operation of MAYA in the auditory domain is the popularity
:o prefer objects and of remixes of old songs, such as the contemporary beat added to Elvis Presley's 'A little
rnflicting-messages less conversation' (funkieXl zooz). Furthermore, Simonton (zoor) demonstrated
sions will lead to ele- that the actual success of classical musical compositions was highest for compositions
: of sensory messages with intermediate levels of melodic originality. Low melodic originality was preferred
here coherence helps over high melodic originality, suggesting that listeners are rather quietly entertained
n, perceived unity in than anxiously provoked, but compositions with an intermediate level of originality
aesthetic appeal and proved most successful.
if product designers In the food domain, Elisabeth and Paul Rozin (r98r) have suggested that the
Iits users, or to com- introduction of a new food staple in a culture may be facilitated by adding a familiar
,m adapting a multi- combination of seasonings. According to Elisabeth Rozin (1973), many of the world's
employing multiple cuisines involve the use of distinctive and pervasive seasoning combinations, such
ü all these channels, as the tomato-garlic-olive oil combination for Italian dishes or the soy sauce-rice
,.-enrüelming than if wine-ginger mixture for Chinese cooking. She refers to these seasoning combinations
as flavour principles. Adding a familiar flavour principle to an unfamiliar food may
help to bridge the cultural gaps, by increasing people's willingness to try the new food
(Stallberg-White and Pliner 1999).
During interactions with (objects in) the surrounding environment, people also get anticipate its res..,Lu:
to know their other perceptual and motor capabilities. For example, people can test erpectations. There
how fast they can run, how loud they can shout, or how good they are in recognizing stem trom'an e\q;
the taste of different beer brands. All this information contributes to self-knowledge in rvhich the lis¡er¡
and people can use this knowledge to develop their skills. if skills are developed further, the problems soir'¡c
people will be able to handle more difficult challenges, and they will enjoy complex 199o, p. z8o t. Ho',ro
challenges more (Csikszentmihalyi r99o). In physical interactions with objects, such a , r99o) found that li
challenge can take on the form of a power match: will you be able to open the marma- ñ?es, even though
lade jar with the tight lid, or will rhe marmalade jar win this fight? (sonneveld and This preference icr
Schifferstein zooS). Challenges are not necessarily limited to perceptual and motor stations tend to har',
skills; they may also involve cognitive skills. For instance, a challenge may be incorpo- the same popular s,:
rated in a design by evoking the curiosity of the potential user without giving all inter- and unfamiliar sLrr!
esting information away at once, such as in complex buildings or in products that make
use of translucent materials to partly conceal the internal components from the eyes,
Iike the Apple iMac (Fig. zS.r3).
In some cases, a product designer may wish to trigger an experience of surprise, for
example, to increase interest or prolong the attention value of a product. In those cases,
incorporating incongruity between sensory messages in a design could be an effective
strategy (Ludden et al. zoog). The designer's challenge in these cases seems to be to
combine familiarity and originality within the same design: even designers who want In this chapter rr e
to surprise people will generally make sure that the majority of the product-related modalities. These
information communicates the same message, while only one particular aspect is aesthetic responsej
responsible for the surprise. Designers tipically limit surprising designs to well-known pies may be share .i i
product categories that consumers can easily identify, such as furniture and other Principles related t"
interior products (Ludden et al. zooS), to enhance the familiarity in their designs. on people's inter¡:
Because music evolves over time, unexpected events could be of particular impor- similar wa1'for aL s,
tance in its appreciation (e.g. Meyer 1973). when listeners hear the start of a melody, of perceptual int'-':
they expect it to continue in a particular way. when an unstable chord is heard, they therefore, be partl-''
Elsewhere tHeri.
modal correspond;
bytheories of etc--l
evolutionan' fun.-i
pleasure from seer
paintings, er-o1uii tr r
The aesthetic sense
brain and the sersc,
Simply stated, pe.-1
features and pa:t<::
of these functions ¿
information, anil r
principles that ¿:¿
s)'rnmetry, and b:-
a particular serlsrr:-
are sensory-specii;
Fi1.28.13 The Apple iMac. still holds.
MULTISENSORY AESTHETICS IN PRODUCT DESIGN 563
ment, people also get anticipate its resolution. Composers can choose to fuIfiI, delay, or deny such musical
nple, people can test expectations. Therefore, some have suggested that the appreciation of music might
e-"'are in recognizing stem from 'an exquisite game of expectational cat and mouse with the composer,
:es to self-knowledge in which the listener enjoys the tensions and resolutions, the problems posed, and
re developed further, the problems solved, the confusions followed by comprehension' (Smith and Melara
. rndl enjoy complex 1990, p. z8o). However, in their study of harmonic progressions, Smith and Melara
rrith objects, such a (r99o) found that untrained listeners generally preferred to hear the harmonic proto-
'
to open the marma-
$,pes, even though they found atypical progressions more interesting and complex.
út? rSonneveld and This preference for the familiar may not come as a surprise, given that many radio
:r"eptual and motor stations tend to have a play'ist consisting of very familiar'golden oldies' songs, or play
nEe mav be incorpo- the same popular song several times a day. People apparently need to get used to new
:rLrutgi\.ing all inter- and unfamiliar songs, before theyappreciate them (Smith etal.ry94).
r ¡roducts that make
rents from the eyes,
AcrcNowLED GEMENT s
We would like to thank René van Egmond, Marieke Sonneveld, and our other co-workers
for their helpfi.rl comments and suggestions. In addition, we thank Kimberly Elam (Figs.
28.3 andz8.4) and Jeroen van Erp (Fig. 28.8) for proüding graphical materials.
R¿rEnsr.rcns
:rence (or any other Banham R (rq66). The new Brutalism: Ethic or Aesthetic?Karl Krámer, Stuttgart.
tique centres on the Barsalou LW (rqS¡). Ideals, central tendency, and the frequency of instantiation as deter-
ssibility to test them minants of graded structure in categories. Journal of Experimental Psychology: Learning,
duct of the function- Memory, and Cognition, u, 629-54.
Barsalou LW (rgqg). Perceptual syrnbol systems. Behavioral and Brain Sciences,22,577-66o.
ught by starting out
Bell SS, Holbrook MB, and Solomon MR (rgqr). Combining esthetic and social value to explain
rnulate specific, test-
preferences for product sryles with the incorporation of personality and ensemble effects.
e ferred (see Hekkert
]ournal of Social Behavior and Personality,6,243-74.
rerng unduly reduc- Berlyne DE (r97r). Aesthetics and Psychobiology. Appleton-Century-Crofts, New York.
eriences of artworks Bermudez ]L, Marcel A and Eilan N, eds. (ryg). The Body and the SeIf. MIT Press,
mg. srylistic consid- Cambridge, MA.
reiics in the limited Bertelson P (rSSS). Ventriloquism: a case of cross-modal perceptual grouping' In
iodc may work very G Aschersleben T Bachmann, and I Müsseler, eds. Cognitive Contributions to the Perception
of Spatial and Temporal Events, pp. 347-62. Elsevier, Amsterdam.
Bone PF and EIIen PS (rqqq). Scents in the marketplace: explaining a fraction of olfaction.
,r'e r multiple sensory
Journal of Retailing 75Q), 243-62.
:lutually consistent BoringEG G94z).SensationandPerceptionintheHistoryofExperimentalPsychology.Appleton-
¡-k about the way in Century-Crofts, New York.
entertainment: in a Bornstein RF (rqSq). Exposure and affect: overview and meta-anaiysis of research 1968-1987.
:rusic played on the Psychological Bulletin, to6, 265-89.
ie music during the Boselie F (rqqz). The golden section has no special aesthetic attractivity! Empirical Studies of
the Arts,10, r-r8.
'¡und: the degree of
'nation Boselie F and Leeuwenberg E (rq8S). Birkhoff revisited: beauty as a function of effect and
is there and
means. American Journal of Psychology, gS(r), r-39.
s-:btides. Therefore,
Buss DM, ed. (zoo5). Handbook of Evolutionary Psychology. Wiley, Hoboken, NJ.
ls ila\' mayproüde Butler JW and Daston PG (rq6S). Musical consonance as musical preference: a cross-cultural
!11;. strdy. Journal of General Psychology, Tg, :,29-42.
Charters S (zoo6). Aesthetic products and aesthetic consumption: a review. Consumption,
Markets and Culture, g,235-55.
Charters S and Pettigrew S (zoo5). Is wine consumption an aesthetic experience? Journal of
Wine Research, L6, t2t-36.
Chastrette M, Elmouaffek A, and Sauvegrain P (rqSS). A multidimensional statistical study of
similarities between 74 Íotes used in perfumery. Chemical Senses, 13, 295-305.
Chrea C, Valentin D, Sulmont-Rossé C, Ly Mai H, Hoang Nguyen D, and Abdi H (zoo$'
Culture and odor catgeorization: agreement between cultures depends upon the odors.
,-:¡ other co-workers Food Quality and Preferenca 15. 669-79.
::rberh'EIam (Figs. chrea c, valentin D, Sulmont,Rossé c, Nguyen DH, and Abdi H (zoo). semantic, tlpicaiity
l¿terials. and odor representation: a cross-cultural study. Chemical Senses, 3o, 37-49.
Csikszentmihal)'r M (t99o). FIow: The Psychology of Optimal Experience. Harper Collins,
London.
Dematté ML, Sanabria D, and Spence C (zoo). Olfactory-tactile compatibility effects demon-
strated using a variation of the Implicit Association Test. Acta Psychologica' L24' 332-43.
Doty RL (r98r). Olfactory communication in humans. Chemical Senses,6,35r-76.
ElamK(zoor).Geometryofdesign:studiesinproportionandcomposition.PrincetonArchitectural
::,r :-; si¡¡il¿¡i¡y? 1'h.
Press, New York.
:t-i¿ntal Psychology,
Farell B (r985). "Same"-"different" judgments: a review ofcurrent controversies in perceptual
comparison. P sy cholo gical B ulletin, 98, 419-56.
g ,l;rr? Acustica, 59,
Fastl H and Zwicker E (zoo7). Psycho-acoustics, 3rd edn. Springer, Berlin.
566 HENDRIK N.J. SCHIFFERSTEIN AND PAUL HEKKERT
Fenko A, Otten JJ, and Schifferstein HNI (zoro). Describing product experience in different Lakoff G and Johns.-
Ianguages: The role of sensory modalities. J ournal of pragmatics, 42, 31'4-27. Chicago, IL.
Frith C and Nias D (1974). What determines aesthetic preferen ce? Journal of General Psychology, Lawless HT (zooo . S
9r,1q-73. the MeaI: The 5t:.-,,
Gaver WW and Mandler G (rqsz). Play it again Sam: on liking music. Cognition and Emotion, MD.
r,259-82. Lindauer MS (rq86 .
Gibbs RW, Lima PLC, and Francozo E (zoo4). Metaphor is grounded in embodied experience. Psychonomic St;i-::
I ournal of Pragmatics, 36, u89-zro. Lindauer MS, Ster=i
Godkewitsch M (ry7a). The 'Golden Section': An artifact of stimulus range and measure of I. fudgments. 8r,,,:
preference. American lournal of P sychology, 87, 269-77. Lindstrom M (:oo¡ .
Goldman A (zoor). The aesthetic. In B Gaut and D Mclver Lopes, eds. The Routledge Companion and Sound. Free P:
to Aesthetics, pp. 181-92. Routledge, London. Locher P (zoo6l. Thc
Govers PCM, Hekkert P, and schoormans IPL (zoo4). Happy, cute and tough: can designers with visual art iLr ¿.
create a product personality that consumers understand? In D McDonagh, P Hekkert I van Locher P, Smets G. ar
Erp, and D Gyi, eds. Design and Emotion. The Design of Everyday Things, pp. 345-9. TayIor dimensional sol-:: :
r::erience in different Lakoff G and lohnson M (r98o). Metaphors we Live By. The university of chicago press,
Chicago, IL.
:,r- G e n er aI P sycholo gy, Lawless HT (zooo). Sensory combinations in the meal. In HL Meiselman, ed. Dimensions of
the Meal: The Science, Culture, Business and Art of Eating, pp. 9z-to6. Aspen, Gaithersburg,
: :':itior.t and Emotion, MD.
Lindauer MS (t986). Seeing and touching aesthetic objects: II. Descriptions. Bulletin of the
:::¡odied experience. P sy chono mic S o ciety, 24, tz5-6.
Lindauer MS, stergiou EA, and Penn DL (1986). seeing and touching aesrhetic objects:
::q: and measure of I. ludgments. Bulletin of the Psychonomic Society, 24, 12r-4.
Lindstrom M (zoo5). Brand Sense: Build Powerful Brands Through Touch, Taste, Smell, Sight,
..::,:icdge Companion anrl Sound. Free Press, New York.
Locher P (zoo6). The usefulness of eye movement recordings to subject an aesthetic episode
:-;Eh: can designers with visual art to empirical scrutiny. Psychology Science, 48,1c,6-:r4.
,:¿l:. P Hekkertlvan Locher P, Smets G, and Overbeeke K (1995). The contribution of stimulus attributes of three-
:,¡,.. :¡. j-15-9. Taylor
dimensional solid forms and level of discrimination to the visual and haptic perception of
baiance. P ercep tio n, 24, 6 47-q.
i.:r.. eness and sexuai Loewy R (r95r). Never Leaye WelI Enough Alone. Simon and Schuster, New York.
;.:'. ; P:ychology, rog,
Logue AW Q99r). The Psychology of Eating and Drinking: An Introduction, znd edn. Freeman
and Company, New York.
"
¡rd texture effects. Ludden GDS, Schifferstein HNI and Hekkert P (zoo8). Surprise as a design sftategy. Design
Issues, z4Q),2818.
: ,i enr-ironmental Ludden GDS, Schifferstein HNl, and Hekkert P (zoo9). Visual-tactual incongruities in prod-
ucts as sources of surprise. Empirical Studies of the Arts, 27, 6r-87 .
'' '¡:rion. lournal of Martindale C (rqgo). The Clockwork Muse. The Predictability of Artktic Change. BasicBooks,
New York.
:l:i; pret-erence for McManus IC (r98o). The aesthetics of simple figures. British lournal of Psychology, 7r,
'
-::iuology. 505-24.
.,-':..]¡-i' Science, 4g, McWhinnie H| (r9S7). A review of selected research on the golden section hlpothesis. Vlszral
Arts Re s ear ch, t3, 7 3-B 4.
P Hekkert, eds.
=i Meredith C, Edworthy J, and Rose D (tsSs). Observational studies of auditory warnings on
the intensive care unit. In NA Stanton and J Edworthy, eds. Human Factors in Auditory
::, -nstruction and Warnings, pp. 3o5-r7. Ashgate, Aldershot.
. ,;::,,ii¿-. o.f the Arts, Meyer LB (ry6). Explaining Music: Essays and Explorations. University of California Press,
Berkeley, CA.
:: :cceptable': typi- Ózcan E, van Egmond R and Jacobs J (submitted). Bases for categorization and
:::.:.1 design. Bdrrslr identification.
Pecher D and Zwaan RA, eds. (zoo4) . Grounding cognition: the role of perception and action in
::::: -\f atching rules memory, language and thinking. Cambridge University Press, Cambridge.
Pecher D, Zeelenberg R, and Barsalou LW (zoo4). Sensorimotor simulations underlie concep-
--,r lrr palatability. tual representations: modality-specific effects of prior activations. Psychonomic Bulletin ú
Review, fi,164-7.
.¡.: Behar-iorai and PlugC(r98o).Thegoldensectionhlpothesis. AmericanJournalofPsychology,93,46T-87.
¡.:::;i a. A.ffective ú Power TG (zooo). Play and Exploration in Children and Animals. Erlbaum, Mahwah NJ.
Purcell AT (r98a). The aesthetic experience and mundane reality. In WR Crozier and
:is. [hicago, IL. Al Chapman, eds. Cognitiye Processes in the Perception of Art pp. 189-210. Elsevier,
.:: :: :t:- t7, 767-77g. Amsterdam.
Ramachandran VS and Hirstein W (rqqg). The science of art: a neurological theory of aesthetic
:: r,rerduct design: experience. Journal of Consciousness Studies, 6, t5-5r.
.. :: t' Destgn, 29,
568 HENDRIK N.I. SCHIFFERSTEIN AND PAUL HEKKERT
Reber R, Schwarz N, and Winkielman P (zoo4). Processing fluency and aesthetic pleasure:
Is beauty in the perceiver's processing experience? Personality and Social Psychology Review,
8,364-82.
Repp BH (rqqz). The aesthetic quality of a quantitatively average music performance: Two
preliminary experiments. Music Perception, l^4, 4;9-44.
Rolls BJ (1986). Sensory-specific satiety. Nutrition Reuiews, 44,93-rot.
Rozin E Q97). The Flawr Principle Cookbook. Hawthorne, New York.
Rozin E and Rozin P (r98r). Culinary themes and variations. Natural History, go,6-14.
Rozin P (1972). Speciñc aversions as a component of specific hungers. In MEP Seligman and
|L Hager, eds. Biological boundaries of learning. Appieton-Century-Crofts, New York.
Rozin P (1976). The selection of foods by rats, humans, and other animals. In IS Rosenblatt,
RA Hinde, E Shaw, and C Beer, eds. Advances in the study of behavior. Academic Press,
NewYork.
Schifferstein HNJ and Desmet PMA (zoo8). Tools facilitating multisensory product design.
The D e sign J o urnal, tt( z), vz -58.
Schifferstein HNJ and Tanudjaja I (zoo+). Visualising fragrances through colours: the mediat-
ing role of emotions. Perception, 33, L249 -1266.
Simonton DK (zoor). Emotion and composition in classical music: historiometric per-
spectives. In PN Juslin and JA Sloboda, eds. Music and Emotion: Theory and Research,
pp. 205-22. Oxford University Press, Oxford.
Smith JD and Melara RI (rqqo). Aesthetic preference and syntactic prototypicality in music:
'tis the gift to be simple'. Cognition,34,279-98.
Smith JD, Kemler Nelson DG, Grohskopf LA and Appleton T (1994). What chiid is this? What
interval was that? Familiar tunes and music perception in novice listeners. Cognition, 52,
23-54
Sonneveld MH and Schifferstein HNJ (zoo8). The tactual experience of objects. In HNI
Schifferstein and P Hekkert, eds. Product experience, pp. 4r-67 . Elsevier, Amsterdam.
Stallberg-White C and Pliner P (rqqq). The effect of flavor principles on willingness to taste
novel foods. App etite, 33, 209-21.
Stevens JC and Marks LE (r98o). Cross-modality matching functions generated by magnitude
estimation. Perception ú P sychophysics, 27, 37 9-89.
Süskind P (rgBS). Perfume. Penguin Books, London.
Suzuki M, Gyoba J, Kawabata H, Yamaguchi H, and Komatsu H (zoo6). Analyses of the
sensory-relevance of adjective pairs by the modality differential method. Shinrigaku
Kenlcyu, 77, 464-70.
Thurstone LL (tgzZ). A law of comparative judgment. Psychological Review,34,2n-86.
van Egmond R (zoo8). The experience of product sounds. In HN| Schifferstein and P Hekkert,
eds. Product experience, pp. 69-89. Elsevier, Amsterdam.
van Rompay T, Hekkert P, Saakes D, and Russo B (zoo5). Grounding abstract object charac-
teristics in embodied interactions. Acta Psychologica, rrg, 3t5-5t.
Yeryzer RW and Hutchinson JW (rqg8). The influence of unity and prototlpicality on
aesthetic responses to new product designs. Journal of Consumer Research, 24, 374-94.
Voland E and Grammer K, eds. (zoq). Evolutionary aesthetics. Springer Verlag, Berlin.
Walker EL (rq8o). Psychological Complexity and Preference. Brooks/Cole, Monterey.
Wedekind C and Füri S (rSSZ). Body odour preferences in men and women: do they aim for
specific MHC combinations or simply heterozygosity? Proceedings of the Royal Society of
London Series B, 264, r47t-7g.
Whitfield TWA (r9S3). Predicting preference for familiar, everyday objects: An experimen-
tal confrontation between two theories of aesthetic behaviour. Journal of Environmental
Psychology, 3, 22t-37.
MULTISENSORY AESTHETICS IN PRODUCT DESIGN 569
e:d aesthetic pleasure: Whitfield TWA and Slatter PE (1979). The effects of categorization and prototypicality on
: :i o!. P sychology Review, aesthetic choice in a furniture selection task. British Journal of Psychology, 70, 65-75.
Zajonc RB (1968). Attitudinal effects of mere exposure. Journal of Personality anrl Social
-src performance: Two Psychology, Monograph Supplement, g, t-27.
Zellner DA, Bartoli AM, and Eckard R (r99r). Influence of color on odor identification and
Iiking ratings. American lournal of Psychology, to4, 547-67.
: r,rrprcaiity in music:
¡: objects. In HNJ
. -\msterdam.
r'iJli¡gness to taste
n::¿ted bv magnitude
- :5 . A¡alvses of the
n¿thod. Shinrigaku
J rrotorrpicality on
,i,:i,- 24, 37q-94.
i-e11ag, Berlin.
J'1¡nterer'.
r:¿n: do they aim for
'.t :;:¿ Rot'al Society of
¡e;ts: An experimen-
*."1 ¡f Environmental
AI
SrNcp it is comrnc,n,
start by paraphras':r
taken from the rr r.ri-
of vision re\'€als a irr
to ieft and right. an
squint we can mar¿.
see there is an up an
don't even maneg- t
ARCHITECTURE
AND THE BODY
ALBERTo pÉnEZ-cótvtnz
Sruc¡ it is commonplace to state that architecture is 'the art of space', I would like to
start by paraphrasing Georges Perec's meditation on the topic, adding a few remarks
taken from the work oftwo philosophers, Maurice Merleau-Ponty and Plato. Our field
of vision reveals a limited space, something vaguely circular, which ends very quickly
to left and right, and doesn't extend very far up or down (Perec 1999, p.r88). If we
squint we can manage to see the end of our nose; if we raise or lower our eyes, we can
see there is an up and down. If we turn our head in one direction, then in another, we
don't even manage to see completely ever¡hing there is around us. We have to twist
our bodies round to see properly what was behind us, and yet we take for granted the
stability of the enüroning world, never hesitating to drive or fly in a plane. O:ur gaze
travels through space and we recognize depth, aprimary dimension, unlike all others,
irreducible to breath and height. Depth is the dimension of human love and longing,
hardly analogous to 'perspective', an enigma made manifest by every human culture in
their most significant artefacts.
Our spatiality is constructed with an up and a down, a left and a right, an in front
and a behind, a near and a far. When nothing arrests ovÍ gaze, it carries a very long way.
But if it meets with nothing, it sees nothing. Space therefore arrests our gaze, it is also
the substance, the obstacle, whether bricks or a'vanishing point'. Plato understood this
very well. In his famous dialogue Timaeus, chora is characlerized as both space and
primordial substance, and conceived as the mode of reality through which the things
of the world (ideas embodied in concrete things) actually appear to our experience
(Plato 1965, p. 66).Properly understood, human space ls limits, most clearly when it
makes an angle, when it stops. There is nothing ghostly about space, it has edges.
Human space is, in fact, all that is needed for the parallel lines of the straight road to
j72 ALBERTopÉp,nz-cówpz
meelwell shorrof infinity: ilis notCartesian space (for one of the best articulations of -L-L--:-
lll trldlll.
*L
LLrl --C
this problem in modern philosophy see Merleau-P on\, ry64lr9y). :r,--,imal. bc¡jil-; ,
of course, our mobile and digitally ubiquitous civilization, avid consumer of r,¿tient rnar :h:':
seductive images, can easily accept that true space ls Cartesian space, that the photo- :.'ea. riord. c,r c:
graph depicts'real' space. It can be further swayed bythe argument that speed changes Our seniier:: l
something basic in the primary gir.,en structure. We can all agree with Paul Virilio's ,:,ithis re¿ii¡,-. :-
observation that machines, ranging from television to the automobile, transform our irrost traditilrI]:-
visual relationship to the environing world, proposing their own space/time, and as a has prir-iieeed il:
consequence have an effect on our perceptions. ihe sense car:¡-
Nevertheless, when we ask questions about the design of architecture related to this lr--^ -t-.-i
Jl\)()!ldLCU -'--l-
-i;1-.
'new' condition, we must be careful not to confuse the issues. It is easy to celebrate the hor. ever in::,-- r-
world of instantaneous communication, speed, and mobility, claiming some radical trur \\'estern s;
change in the human condition and the structure of the senses, one that may justif, an n¿r'ertheless r::
architecture reduced to seductive, seemingly innovative imagery, regardless of its that rs qivel ir:¡
medium. While it is truethat the speedswithwhichwe can move our physical or mental In thrs ligr:.
bodies transforms one's understanding of the world, it is an even more fundamental tradition tf ::.
truth that, if we were not first and foremost embodied consciousness, already engaged c.,30 BCE . t,-' ',"
through our frontal and vertical position in a gravity-bound world, we wouldn't the imponar:c
be able to make any sense of so called cyberspace, we wouldn't, in fact, be able to drive correction i. it,
at rzo kilometres an hour while paying close attention to a piece of music, or to our it comes to c,Li:
friend's words, nor would we have the certainty that we fit between the truck and the standing cri:i:
curb, for we have no scaled model of the world in our brain, nor little computer-men a predomir::r-.
inside our heads providing ultra-fast solutions to our pressing dilemmas. Of course we forms and ¡:ne
make mistakes, and we may be fooled by illusions, but generally we are not. Even when to be conve'.'¿:'
very tired we know well it is a common miracle that we arrive safe to our destination. Todar-. h : ''
Regardless of neurological explanations, it is phenomenology's contention that this called expe:ir--
ultimately has more to do with bodily knowledge overcoming the limitations of an that the imas¡ :
impaired brain (see Merleau-Ponty zooz). then inve¡t ::e
Phenomenology rejects the idea of perception as constituted by independent of disorient¡i: .
senses, simply communicating data to the brain where a synthesis of some kind may be
having t-ee.in: ,
constantly taking place. The analogy of the brain with a computer always falls short. lvhat is mLrs: s -r:
while something of this sort also contributes to our perception of meanings, phenom- place in our'i:,
enology rather privileges an obvious reality: the world is first given unfragmented to normal in --ss :
the whole sentient body. Thus it argues for the primacy of embodied perception at the of meaninq.
roots of being and understanding, grounding other modalities of intellectual articula- lv{aurice \ lr
tion. For example: we know that the earth does not move. This is a primary truth for conscience ',,,---::
our bodies, one indispensable in order eventually to construct an endless number of bymotilin-i:r
scientific or m¡hical explanations of the universe. This first truth is a precondition for ings ivithout :
all others. Bodilyperception is synaesthetic and precedes our common understanding
ble of unspe :i"
of perception occurring partes-extra-partes, fhrough the independent five senses. person or a p.:
Granting the primacy of embodied perception also leads to questioning all mechanistic guide.Itisa-s::
or causal models of human understanding that have been prevalent since the seven- a body imaqe ::
teenth century, and in
transformed fashion, are still at the root of many sophisticated
a
Imagine th::
contemporary neurological models. Meaning is not something merely constructed in stand in the ¡:::
ARCHITECTURE AND THE BODY 573
le best articulations of the brain, the result of judgement or the association of ideas, meaning is given in our
l_1 i.
normal, bodily engagement with things, things that we recognize, not as an aphasic
'¡. avid consumer of
patient may through deduction and association, but instantly as the embodiment of an
space, that the photo- idea, word, or category.
:nt that speed changes
Our sentient bodies are our means for thinking and understanding. Acutely aware
:ee'rvith Paul Virilio's
of this reality, and at odds with our modern conceptualization of cognition processes,
nobi,ie, transform our
most traditional cultures tended to identift feeling and thinking. Modern civilization
n space/time, and as a
has privileged the sense of vision, the main vehicle for empirical experimentation, and
the sense capable of discriminating clarity, associated with logic. It has also radically
itecture related to this
dissociated feeling, thinking, and imagining as independent mental processes. Vision,
s easy to celebrate the
however important it may be in the constellation of perception, and particularly in
iaiming some radical
our Western scientific tradition constructed upon the Greek notion of theoria, is
r-re that may justifiz an
nevertheless rather fragile and prone to error when the question is to assess a reality
en-, regardless of its
that is given intersubjectively.
r.lr physical or mental
In this light, and coming closer to our topic, it is interesting to recall that in the
'r more fundamental
tradition of architectural theory, treatises ranging from the Vitruvius' Ten Books
ress, already engaged
(c.3o BCE), to works dating to the second half of the seventeenth century, argued for
un-srld, we wouldn't the importance of 'optical correction' (see Pérez-G ómez and Pelletier ry97) . Optical
r ;act, be able to drive correction is nothing less than the acknowledgement of the weakness of vision when
e rrf music, orto our it comes to questions of architectural meaning. Drawing from Euclid's original under-
r.n the truck and the
standing of the discrepancy between what appears to our eyes and what actually ls (in
i:rde computer-men
a predominantly tactile, synaesthetic world), architects always expected buildings'
3nÍ1as. Of course we
'e ¿re
forms and dimensions to be corrected in order for the experience of ordered regularity
not. Even when
to be conveyed to a living inhabitant or spectator.
to our destination.
= Today, in a world where vision has become hegemonic, we need to recall the so-
;ontention that this
called experiment of inverted vision to illustrate this fragility. It is common knowledge
-¡: nimitations of an that the image projected on the back of the retina is inverted. If one wears lenses that
then invert the field of vision, it is not surprising to encounter a few unpleasant hours
::d bv independent of disorientation and even vomiting; we may touch things with the left hand, while
,¡1some kind maybe
having feeling in the right hand, for example. Simple acts become problematic. Yet,
er alrr-avs falls short.
what is most surprising is how rapidly the body recovers its bearings. Except for a small
:reanings, phenom-
place in our backs, usually inaccessible to touch, the whole body and its world return to
en unfragmented to
normal in less than two days, demonstrating how bodily orientation lays at the root
.ed perception at the
of meaning.
nteliectual articula-
Maurice Merleau-Ponty has argued for a double engagement of our embodied
a primary truth for
conscience with the world. At a primary level, our body knows, this is a body inhabited
r endless number of
by motility-the motion oflife itself. Our body recognizes its location in our surround-
ls a precondition for
ings without'payrng attention', through'motor intentionality'. This is the body capa-
r-on understanding
ble of unspeakable athletic feats when threatened, and the body that knows another
,endent five senses.
person or a place long before exchanging a word with the stranger or reading a travel
;rins all mechanistic
guide. It is also the body housed by architecture. Only upon this primary understanding,
:rt since the seven- a body image and concepts of space and time can be constructed.
lanv sophisticated Imagine that we are spending some time in the beautiful Tuscan city of Siena. We
:rel.,- constructed in
stand in the main square, thePiazza del Campo, contemplating the tower. We marvel
574 ALBERTO PÉIJ.EZ-Gól¡/rEZ
at the proportions of the whole ensemble, and then our architectural imagination time and space can l
naturally leads us to estimate the tower's height, and then how far it would fall if it ever immensely usefui. l
collapsed (this example is taken from the beautifullywritten introduction to phenom- mystery of simultar
enology by van den Berg r97o). We have substantial experience with dimensions, and whiteness, and the 1
we are quite certain of our perception. A few minutes later we wander into a bookstore without the other. I
nearbywhere a measured surveyof the square is available to us. We simplycan'tbelieve meaning and lineuis
the discrepancybetween the distance we estimated and the'real measure' of the tower's When we addres,
height. Is it possible to dismiss this phenomenon as an 'optical illusion' if it happens creation or the re.e,
consistently every time we 'turn' a horizontal dimension into the vertical? Does the (see, for
example.
scale or measuring tape stretch when we move it from a horizontal to a vertical posi, Architecture is not ,
tion? Is this why even though we carefully measure the dimensions of an elevation in a three-dimensional ir
flat, horizontal drawing, when we build the building it seems to have grown? Is there a precise working dra,
more sophisticated neurological explanation? It is perhaps better simply to recognize genic. In fact, olten
that in our primary perceptual engagement with the world, before we measure, vertical silence), the tactilin
distances are larger than horizontal distances. We must acknowledge that embodied other factors that ap
orientation is fundamental for our perception of meanings. In our primary engage- qcross the senses. Fr
ment with the world this discrepancy is not an illusion but a fact. Our vertical position wayto dwell in the
"n
is unique among animals. We are like ahlphenbetween the earth and the sky, endowed lematic to reduce tl
with a spherical head and frontal vision, in a world whose transparent atmosphere centurysense of the
made it possible to contemplate the heavens and thus discover the regularity of math- (see Pérez-Gomez. l
ematics, eventually leading to our scientific theories. This verticality is not incidental; This bears some .
to consider the living body in its own terms, different from an objectified cadaver or address the same bas
a collection of mechanical systems. Research into so-called artificial intelligence dem- ity ofcultural transc,
onstrates how difficult it is to have a robot accomplish even simple tasks, like making architecture prortd
a bed, for example. Our consciousness is embodied-the stomach knows and the hand More specificallr'. it .
thinks. Everything in our environments is first given to this synaesthetic, perceiving logical coiiection u-'f
body of ours. The distance between my home and place of work changes constantly, cance was to delighr
invariably qualified by this embodied consciousness. Il on diverse occasions, while modern conception
travelling home from work, I am tired, hungry, or depressed, the distance I experience the nineteenth cen:t
is always different, and this is a more primarytruth than the distance in kilometres read dead-end. A more ¡:
by the odometer of my car. It is primary, for the quantification of distance (the abstract political and episte:r
numbers in the odometer) is meaningless apart from a specific experience, situation, ture's'universe oii-
or context. Likewise, time is elastic; the present has a thickness, rather than being a in the early Uoos.
non-existing instant or punctuffibetween past and future. To justif' regret for wasted seemed capable a:
time, we usually say: the lecture felt like an eternity, but it only lasted 5o minutes. we modes of produc:r"-
should at least acknowledge that the lecture actually took a very long time out of my pragmatic necessi:r-
life experience, thoughinmywatch I couldseethat only5o minuteswentby. Andwhen While the threa:¡
we step outside in winter, we may notice that the colour of snow is always different, at dangers posed to ,rr
night, under the glowing sun, or sheltered by a grey sky. of course, the whiteness of environment are gei
snow is not a delusion either. And no one would deny the importance of measurement; the view that the :¿
ARCHITECTURE AND THE BODY 575
tectural imagination time and space can indeed be conceived in mathematical terms, and this has proven
i¡ u-ould fall if it ever immensely useful. Rather, phenomenology shows that in human perception the
,,luction to phenom- mystery of simultaneity is always operating: both the ideal essence of the snow, its
¡th dimensions, and whiteness, and the particular shade of colour are given at once. Neither is possible
rtler into a bookstore without the other. This is the enigmatic nature of human perception, the ground of
: simplycan'tbelieve meaning and linguistic understanding.
3a,rure' of the tower's When we address the issue of architectural meaning, thinking about either the
-usion' if it happens creation or the reception of architecture, this primary reality must not be forgotten
re lertical? Does the (see, for example, Holl et al. ry94lzoo6; Pallasma, 2oo4, 2oo5; Zumthor, zoo6).
:d to a vertical posi- Architecture is not what appears in a glossy magazine, it is not reducible to two- or
s ot an elevation in a three-dimensional images on the computer screen, or even to a comprehensive set of
rr'¡ qrorvn? Is there a precise working drawings. The most significant architecture is not necessarily photo-
sulph'to recognize genic. In fact, often the opposite is true. Its meanings include sound (and eloquent
'-1 e rrreasure, vertical silence), the tactility of materials, smell, and the sense of humidity, among infinite
*iee that embodied other factors that appear through the motility of embodied perception and are given
;: primary engage- across the senses. Furthermore, because architectural projects effectively propose a
,ru¡ r'ertical position way to dwell in the world, in the best instances both ethical and poetic, it can be prob-
ii rhe sky, endowed lematic to reduce their effect to mere aesthetic experience (in the old eighteenth-
itlrent atmosphere century sense of the term). Architecture, in this sense, is also a form of political action
::3sularity of math- (see Pérez-G omé2, zoo6).
:¡-r-; not incidental; This bears some further clarification. It is an important first step not to take for
granted that architectural phenomena across civilizations can be reduced to one easily
n¿ controlled by a definable class of objects, like oranges or houses. The reality of architecture is very
s ulderstanding of complex, both shifting with cultural changes, and, in some ways, also remaining the
:ircr¡e. rvhich tends same. Architecture responds to the human condition that demands we continually
'-;;i¡ed cadaver or address the same basic questions: to reconcile our personal mortalitywith the possibil-
.' ultelligence dem- ity of cultural transcendence opened up by language. Though the questions are similar,
; ra-.ks, like making architecture provides diverse answers appropriate to specific times and places.
n¡rrs and the hand More specifically, it is naive to identify the full tradition of architecture with a chrono-
:s-¡etic. perceiving logical collection of buildings, understood as useful 'creations', whose main signifi-
;ranges constantly, cance was to delight through more or less irrelevant aesthetic ornament. This popular
ic Ltrccasions, while modern conception was a product of the Enlightenment that only came to fruition in
:sr¡nce I experience the nineteenth century. Such a reduced and objectified understanding leads us to a
¡ -:r Li-l0metres read dead-end. A more careful appraisal of our architectural traditions, and their changing
s:ance i the abstract political and epistemological contexts, suggests a different way to understand architec-
¡r:ience, situation, ture's 'universe of discou¡se'-spsr'¿1ing in the realm of what Giambatista Vico called,
-¿:her than being
a in the early r7oos, 'imaginative universals': a discipline that over the centuries has
¡- :eg¡et for wasted seemed capable of offering humanity, through widely different incarnations and
::d ;o minutes. We modes of production, far more than superfluous pleasure or a technical solution to
rng time out of my pragmatic necessities.
ru ent br.. Aldwhen While the threats of a polluted physical environment are relatively easyto grasp, the
different, at
i-l'rn-¿1'5 dangers posed to our spiritual health (i.e. to our embodied minds) by a banal built
.e.the rvhiteness of environment are generally misunderstood. The technological world often encourages
:e of measurement; the view that the real meaning of architecture is simply to fulfil a programme and
j76 ALBERTopÉREZ-cóMEZ
provide for shelter. This is not surprising, for in our global village the only truths that a ferv examples. Tlis
seem to have universal validity are the technical achievements of applied science, ulti- one of semantic eq'r
mately issuing from a mathematical language: these are the sort of truths that seem to 'meaning' is inser'¿::
'stand' for all, regardless of local languages or cultures. Our technological world is one
tectural meaning cai:
driven almost exclusively by efficiency of means. Efficiency stands as an absolute value, of recognition is e¡:.r:
not only in economics or industrial production, but also in all orders of life, demon- stantial. These art;::
strable through mathematical argumentation. Thus the means can claim to be unaf- tbrm ofbeaufi- gf rr::
fected or unconcerned by the social consequence of its ends, which at best will be open ifi'ofVenus, rath.r ::
for subjective debate. Within this framework, good architecture would be nothing rqóo). This \ias siij
other than efficient building. Indeed, both the poetic potential ofarchitecture to create \f arsilio Ficino anl
a world of beauty and the insidious capacity of the built environment to contribute to different from 'to:i:.1
psychic malaise and a repressive politics go generally unnoticed by the public, and are beauty, like erotic -,-,''
often insufficiently understood by architects. a'poetic image'. it .
Our reason may be capable of dismissing the quality of the built environment as svnaesthetic: it is tl: ¡
central to our spiritual well-being, yet our dreams and our actions are always set ln ied soul's participarr
place, and our understanding (of others and ourselves) could simply not be without Unlike most t ui-;
significant places. As I have suggested, our bodies are indeed capable ofrecognizing fbr existential orle:,:
and understanding, despite our so-called'scientific' common sense and its Cartesian \Vhen successfui. ar¡
isotropic space, the wisdom embodied in a place, the profound and irreducible to the participan: r:
expressive qualities of a culture. Architecture is manifest in those rare places that speak of theatre archite;:*
back to us and resonate with our dreams, it incites us to real meditation, to personal the heavens and ih:
thought and imagination, opening up the 'space of desire' that allows us to be 'at home' geometry. He al.s.- ;
while remaining always 'incomplete' and open to our personal death, unveiling a important classica- :
glimpse of the sense of existence. We are first and foremost mortal, self-conscious the most import;n: .
bodies already engaged with the world through orientation and gravity. As embodied that the meanrns,--i
consciousness we are deeply intertwined in a given world, in an unarticulated, precon- particularly in the r:
ceptual 'ground' that depends greatly upon 'architecture' as the external order, one 'with open pore!'. -:
primarily responsible for making our limits present. This 'meaning' is ::::
I will finish by evoking some important qualities of an architecture 'for the body', classical Greek c-¡: R.
one that may contribute to our human self-understanding. Most crucial is not archi- suffice. Underst..¡ i
tecture's capacity to communicate a particular meaning that could be 'read' or 'deci- individual's e\pir:.
phered' (like a sign or the logo of a corporation or national state), but rather the It orients throush ¿:
possibility of rr.erely presentingmeaningas it appears in the cultural and natural worlds. rooted in mlthic ,:'r
In the best cases, architecture may allow us to recognize ourselves as complete, in order times of its histo:-"'.
to dwell poetically on earth and thus be wholly human. In the Western tradition, the Its disclosure of be¿
products we may associate with the 'architectural phenomenon' have ranged from one's life in the vi,,:;
lhe daidala of classical antiquity (objects such as ships, temples, and deceptive war Iove, it strikes a b-,-'
machines, all put together from small parts through carefully crafted joints), to the Socrates, is cruci¿-:
sun-dials, machinae, and buildings that Vitruvius named as the three manifestations oflogical reason. l:
of the discipline. Architecture, in this sense, has included the gardens and ephemeral clear logic, it is kr,-,-,
architecture of the Baroque period and the built and unbuilt'architecture of resistance' therefore opaqui ..,.
of modernity, ranging from Piranesi's cerceri etchings to Daniel Libeskind's efforts of manr- ar..:
Micromegasdrawings, and including such twentieth-centuryworks like Le Corbusier's ing' can never be i,i
La Tourette, Antoni Gaudi's Casa Batlló, and |ohn Hejduk's 'masques', to give or stylistic formul¿s
ARCHITECTURE AND THE BODY 577
e üe only truths that a few examples. This recognition of wholeness that architecture may offer is not merely
applied science, ulti- one of semantic equivalence, rather it occurs in experience and, like in a poem, its
ritruths that seem to 'meaning' is inseparable from the experience of the poem itself. In other words, archi-
ological world is one tectural meaningcannotbe reducedto concepts thatone mayparaphrase. Themoment
¿s an absolute value,
of recognition is embedded in culture, it is plaful by definition, and is always circum-
rdé¡s of life, demon-
stantial. These artefacts, which the Greeks qualified as thqumata, convey wonder, a
an claim to be unaf- form ofbeauty grounded in eros (which Vitruvius significantly called Venus-ta¡ a qual-
h at best will be open ity ofVenus, rather than using the more common Latinword pulchritudo: seeYitrivius,
e'rr.ould be nothing 196o). This was still clearly understood during the Renaissance by philosophers like
irchitecture to create Marsilio Ficino and architectural writers like Francesco Colonna; a quality altogether
rent to contribute to different from 'formal composition' in the sense of modern aesthetics. Architectural
'-the public, and are beauty, like erotic love, burns itselfinto our soul; it inspires fear and reverence through
a 'poetic image'. It affects us primarily through our üsion, and yet is fully sensuous,
ui.lt environment as synaesthetic: it is thus capable of seducing and elevating us to understand our embod-
rns are always set ln ied soul's participation in wholeness.
:ph'not be without Unlike most buildings in postindustrial cities, architecture offers societies a place
¡aL'1e of recognizing for existential orientation. Architecture is the embodiment of a radical orientation.
rse and its Cartesian When successful, architecture allows for participation in meaningful action, conveying
.:rd and irreducible to the participant an understanding of his or her place in the world. In his description
a:e places that speak of theatre architecture, Vitruvius recognized that its plan responded to the circle of
litation, to personal the heavens and that the twelve divisions of the zodiac were to be reflected in its
i';s us to be 'at
home' geometry. He also developed his analysis of musical harmony in relation to these
,ieath, unveiling a important classical buildings, for he acknowledged that besides temples, theatres were
rr¿1. self-conscious the most important institutional buildings in ancient cities. And yet, he also suggested
:a,,'i¡'. ¡t embodied that the meaning of a theatre is not'objective' or'aesthetic', insisting that it appears
¿::tculated, precon- particularly in the moment of catharsis, during a performance, when the spectator sits
e:r-ternal order, one 'with open pores', in harmony with the breath of the actors and that of the universe.
This 'meaning'is therefore not what a tourist may appreciate today during a visit to a
:;re 'tor the body',
classical Greek or Roman theatre. Beautiful details and a clever, efficient plan do not
crucial is not archi- suffice. Understood as Vitruvius suggests, architecture opens up a clearing for the
C be 'read' or 'deci-
individual's experience of purpose through participation in cultural institutions.
ite . but rather the It orients through experience, in a realm 'beyond words'. So, while its theory may be
¿¡d natural worlds. rooted in mlthic or poetic stories, philosophy, theology, or science during different
complete, in order times of its history, architecture is none of these, but an event for the liüng body.
'
:stern tradition, the Its disclosure of beaufy and meaning is ephemeral, yet it has the capacity of changing
na,,'e ranged from one's life in the vivid present-exactly like magic, or an erotic encounter. Like falling in
ard deceptive war love, it strikes a blow that reveals 'reality ¿5 ls'-¿n experience, which, according to
:ned ioints), to the Socrates, is crucial to the understanding of philosophical problems beyond the scope
ree manifestations of logical reason. Thus, architecture can be said to embody knowledge, but rather than
ens and ephemeral clear logic, it is knowledge understood in the Biblical sense: a carnal, fully sexual and
:;-:ure ofresistance' therefore opaque experience of truth. For this reason, and despite the still ongoing
laniel Libeskind's efforts of many architects claiming either rational or aesthetic aspirations, its 'mean-
Corbusier's
1-lie Le
ing' can never be objectified, reduced to functions, ideological programmes, formal
'nasques', to give
or stylistic formulas. Likewise, its technical medium is open, including artefacts from
578 ALBERTopÉyBz-c6vnz
diverse media that make possible human dwelling and which, by definition, stand 'at
the limits of language', establishing the boundaries of human cultures within which
other more properly linguistic forms of expression may take place.
RnrrnnNcns
Holl PallasmaaJ andPérez-Gómez A (ryg4lzoo6). Questions of perception: Phenomenology
S,
AI
of Architecture. A+U,JuIy ryg4, reprint zoo6.
Merleau-Ponty M Q964h993). Eye and mind. In GA Johnson, ed. Merleau-Ponty Aesthetics
Reader - Philosophy and Painting. Northwestern University Press, Evanston, IL.
Merleau-Ponty M (zooz). Phenomenology of Perception, C Smith trans. Routledge, London.
Pallasma J Qoo+) . Encounters. Rakennustietot, Helsinki.
Pallasma I (zoof ). The Eyes of the Skin. Wiley, Chichester.
Perec G (ry99). Species of Spaces and Other Pieces, I Sturrock trans. Penguin, London.
Pérez-Gómez A (zoo6). Built upon Love: Architectural Longing after Ethics and Aesthetics. S
MIT Press, Cambridge, MA.
Pérez-Gómez A and Pelletier
MIT Press, Cambridge, MA.
L j997). Architectural Representation and the Perspectite Hinge.
B(
Plato (1965). Timceus and Crítias, HDP Lee trans. Penguin, London.
van den Berg H (ry7o). Things; Four Metabletic Reflections. Duquesne University Press,
Pittsburgh, PA.
Vitruvius MVP (196o) The Ten Books on Architecture, MH Morgan trans. Dover, New York.
Zumthor P (zoo6). Atmospheres. Birkhaüser, Boston, MA.
Drspr¡
Srxcs the earlv eis:,:,
a rationalized, se;..-
the gradual displ;.;::
thinking, an idea::,:
much earlier, from '','
At the same tin:: :
realms. This co:::-r.
cHAPTER 3O
¡n. London.
E:i'.its and Aesthetics. SENSE: SPACE,
:;: ; P er sp e ctive Hinge.
BODY, AND THE
si: Unilersity Press,
SENSES
i. Dor-er, NewYork.
IUHANI PALLASMAA
SINcp the early eighteenth century, architecture has been increasingly understood as
a rationalized, secularized, and visual artform. This development has resulted in
the gradual displacement of the human body from its central role in architectural
thinking, an idea that grew stronger during the Renaissance, but that originated already
much earlier, from Vitruvius (Fig. 3o.t).'
At the same time, a growing visual bias has suppressed the role of the other sensory
realms. This condition reflects the general course of Western culture, and has
' In book III of his treatise De Architectura (written around r5 BC), the Roman architect correlated
the ideal human proportions with geometry. This idea was later illustrated and popularized, with some
variations, by Leonardo Da Vinci in a famous drawing (ct+gz), now preserved at the Gallerie
del1'Accademia, in Venice.
58O JUHANI PALLASMAA
r-:::ll:ealure 'f;:! a- - .
__ ).-L:-_l--
: .'l¿1,\rlL i'í-.1 (:--..-: - -
. : ihrs media:in::.:i.
,,:.-;h sousht ic :¡.-:
--.i,^.,t-,.- --.---
.-- =\1:Uú,L(;----
,. :
iefe:ring lLr Fi.--,
l. -.,i.-....:-.- _.,-: L-. - --
* -l-'i-:-- L)' 1111;--:
-
.-1a -"
er,- sar:¿ ;
"th,;:
l:e architec:-r:.-
, ,:-,',-ed the langu:.:
::-:ire shoiiLC'r¿ .:.
:.:¿rded erchlte:: -:
.:,-- i-tr n¿ture. ]l :¡ r:-
:. i:.r.1¿n bodi' arc :: ;
The belief in the correspondence of microcosm and macrocosm, in the harmonic structure
of the universe, in the comprehension of God through the mathematical symbols of centre, Fig. 30.2 The use ,i:
circle and sphere - a1l those closely related ideas which had their roots in antiquity and attempt to combirc
belonged to the undisputed tenets ofmedieval philosophy and theology, acquired new life in Blomstedt, Pythag c r
the Renaissance. . .
the measures of tnr
(Wittkower rq88, p. :q)
ARCHITECTURE AND THE EXISTENTIAL SENSE 58r
Como,1521.
ll'úo's version is
'¡,een the world,
::eraiiilg forms of
Ever since the Greek thinkers, the supremacy of vision has been supported by philo-
sophical thought, as vision was regarded as the most precious and reliable ofthe senses,
and even the very symbol of understanding and truth. Already Heraclitus claimed that
'the eyes are more exact witnesses than the ears' (quoted in Levin 1993, p. r), but Plato
regarded vision and philosophy as humanity's greatest gifts, and insisted that 'ethical
universals mustbe accessible to the mind's eye' (Levin 7gg3,p.287)' Aristotle, likewise,
considered sight as the most noble of the senses'because it approximates the intellect
most closely by virtue of the relative immateriality of its knowing' (Levin r9g3,p. 274).
Numerous examples in the more recent history of philosophy lend support to the
observation that the association of the clarity of thought with clear vision has been
a central metaphor in philosophical writings (see Franzini, Chapter 6, this volume).
The impact of vision on philosophy is summed up by Peter Sloterdijk 'The eyes are
the organic prototlpe of philosophy t. . .1. This gives them a prominence among the
=
body's cognitive organs. A good part of philosophical thinking is actually only eye
refl ex, eye dialectic, seeing-oneself- see' ( quote d in J ay t99 4, p. z7) . :ig.30.3 The or;9 r
It is here worth noticing that, regardless of the early philosophical argumentation s'/stem of arch,tect-
for the supremacy of vision, the actual shift to one-sided visuality seems to have taken architectural orde's
place rather slowly. The primordial dominance of hearing was only gradually replaced ::,
by vision. In Lucien Febvre's view:
The sixteenth century did not see first; it heard and smelled, it sniffed the air and caught
sounds. It was only later that it seriously and actively became engaged in geometry, focusing
ARCHITECTURE AND THE EXISTENTIAL SENSE 583
-.-er's classification of
I
I j-..
t..l 1.,1 1r
o¡1""'lr-
a
--- r-- -- ^ - -'=
.;rpiine was reducible -:l ^ 31'
l I
'H .m
: superstition' (Vidler t& -ui
:l ru
:iaature as an autono-
r:ins in today's fully *;;j;¡¿'F
l'f¡ -l
*td--1 J:'
r- and technological lll tr r -L¡iij -rli: - i,ri,i-
É& g
:ld trom the dimen-
r:. reign by the aban-
::fmeasuresderiving :lii
: began to develop its
- :itat the numerous
M #
__L
':Jrnto gainground.
r¿--r' turned into an
*iL
lT-tiI'l'l
m
¡,r'flrj.s ¡,1:s ur(t¡nf,s r ! ur:\i h.\l ';'
\r¡o1Lr tg.r
!r i----f
il
ltse& i '.)
,-a i
=__:::;
,' -r].
1 l,/
j
v--.t = l- -l
: *rported by philo- t
.-i
: :.rclitus claimed that
-l
-r
-_-.1
Lt9-r. p. r), but Plato
i-,--7 r*l
:::sisted that'ethical ...: ---
i--' l---+-. -_''
- . -i,ristotle, iikewise,
rL¡-lates the intellect
4......t' 77
i' 'r. \
=:it
fI
l
!"lg
===l
af
I -r''-
-f -----------f
1 --( t- l,/
-;rd support to the ,,-...:J_ ---1 -
:t:r
,.;r r-ision has been
6. this volume).
,::.1
-r.-
i .....1
..,r:
t\ '.
LL -1
--=
: tl
l
7
.-:diik: 'The eyes are t\
i:inence among the
t-.
i .,t __ - 1,'-!'-' * \ ,.'
i
lr
r. actually only eye
Fig. 30.3 The origins of the rationalization of architecture 1819. J.N.L. Durand's
*i;al argumentation
system of architectural elements. A) Roof shape combinations. B) Details of
!e en-ls to have taken
architectural orders.
.''' sradually replaced
(Source: J.H.L. Durand, legons d'archítecture, Partie Graphique des cours d'architecture,1819.)
attention on the world of forms [. . .] It was then that vision was unleashed in the worid of
science as it was in the world of physical sensations, and the world of beauty as well.
(Quoted in I ay t 99 4, p. 34)
During the tr,ventie*
Robert Mandrou makes a similar argument:
ment, estrangemeni
The hierarchy [of the senses] was not the same [as in the trventieth century] because the eye, embodiment and lec
which rules today, found itself in third place, behind hearing and touch, and far after them. credos, 'I am and I r,
The eye that organizes, classifies and orders was not the favoured organ ofa time that preferred
in Grosset 1987, p. :
hearing.
play of masses brou
(Quoted in |ay t 994, pp. 34-5)
abstracted and pure
Anyone walking through the intimacy and tactility of matter and echo in the narrow however, that regal,
paved streets of an old town, experiencing the density of sounds in the market halls powerfully plastic ar
and fish markets, or entering the multisensory microcosm of a Romanesque church of systemic propori
or Gothic cathedral, can easily grasp an experiential world that is not dominated by body through his I.¿
the sense ofvision (Fig.3o.a). dimensions of the h
reconnects archite.,
measures of the hul
the proportional pd:
proportional princ:;
has been challengei ;
relations of simple r
During the twentieth century, the modernist artistic strategies of distancing, detach-
ment, estrangement, and fragmentation completed the gradual rejection of human
::5., because the eye, embodiment and led to the full acceptance of the hegemony of vision. Le Corbusier's
:: -d far after them.
--::re credos, 'I am and I remain an impenitent visual - ever¡thing is in the visual' (quoted
:r ¡ that preferred
in Grosset 1987, p.u5), and, 'Architecture is the masterly, correct and magnificent
i¿r' 1991, pp. play of masses brought together in light' (Le Corbusie r 7959, p . 164), epitomize the
i --: 34-5)
abstracted and purely visual understanding of architecture. We need to remember,
d *t:rc¡ i¡ the narrow however, that regardless of his polemical emphasis of vision, Le corbusier was a
i -: rhe market halls powerfully plastic and tactile artist and designer. He also sought to revive the tradition
t,:::ra-nesque church of systemic proportional harmony and architecture's connection with the human
:-" :'.,1 dominated by body throughhis Le Modulor systernof measures and proportions (1948) based on the
dimensions of the human figure and the Golden Section. Le Corbusier's Le Modulor
reconnects architecture with its age-old idea of proportionality deriving from the
measures of the human body and aspiring for cosmic harmony, the unification of
the proportional principles of the physical and biological worlds. Golden Section, the
proportional principle of Le Modulor, which is not based on exact numerical values,
has been challenged and questioned by supporters of P¡hagorean harmonics based on
relations of simple full numbers, which is also (in principle) the harmonic base of
Western musical harmony.'
However, in opposition to its general orientation, the history of Modernism also
includes architects, such as Alvar Aalto and Erik Gunnar Asplund, or the Nordic
modernity at large that aspired for tactile and multisensory architecture.
The 'other tradition of modern architecture', as presented by Colin St fohn Wilson,
is concerned with the sensory reality of architecture as well as the significance of tradi-
tion (Wilson 1995). Asplund's own summerhouse at Stennás in Sweden Qy7) and
Aalto's Villa Mairea in Noormarkku, Finland (rS¡8-S) exemplif' this multi-thematic,
multisensory and comforting architecture that addresses the senses of movement,
touch, hearing and even smell through their natural materials, rich textures and
rh¡hms, tactile and ergonomic detailing, and potent imagery (Figs. 3o.5 and 3o.6).
Already during the period of high modernity Asplund wished explicitly to expand
architectural expression beyond vision:
The idea that only design, which is comprehended visually, can be art is a narrow conception.
No, everything grasped by our other senses through our whole human consciousness and
which has the capacity to communicate desire, pleasure, or emotions, can also be art.
(Asplund 1936)
' The most notable scholar of P¡hagorean harmonics in the modern age was Dr. Hans Kayser,
r E".regel the Elder, the founder of the Institute of Harmonics at the Music Academy of Vienna. See Hans Kayser (r95o).
Lehrbuch der Harmonik. Occident Verla g, Zurich. See also, fuhani Pallasmaa (zoo5).Man, Measure
and Proportion. In Peter MacKeith, ed. Juhani Pallasmaa: Encounters, pp. 47-48. Rakennustieto,
- -:: ,', kimedia Commons.)
Helsinki.
586 JUHANI PALLASMAA
Fig. 30.5 Sensuous and intimate multisensory space. Alvar Aalto, Villa Mairea,
Noormarkku, 1938-9. Entrance hall on a lowered level, living room and main
staircase leading to private rooms above.
(Photograph: Rauno Tráskelin.)
Needless to say, today architecture continues to be theorized, taught, practised, Fig. 30.6 Site-spec f
and assessed as a visual phenomenon. The harmonic principles and cosmic relations site. Juhani Pallasna
ofarchitecture, which used to be its very essence, are hardly touched upon in today's
education and professional practice. In the architecture of the industrialized world the
aesthetic intention has taken over the existential, and instrumentality has replaced In today's medi¿ ¡-
metaphysical concerns. and instant retin¡- .
over sketching b., i:
brought about r'.: =
sensory contact.
The consumer ¡:,
Trrn AGE oF AESTHETTcTZATToN and human embüi:i
in the world thrc',.::
the established r.-'.\ !
In our materialist and digital age, buildings are regarded as aestheticized objects and 'naive realism'th;t .
judged primarily by their visual characteristics. In fact, the dominance of vision has is also directh' re:l.::
never been stronger than in the current era ofthe technologically expanded eye and the other hand, ou:
industrially mass-produced visual imagery-the 'unending rainfall of images', as Italo cult of the bodr-. .:
Calvino appropriately described the present cultural condition (Calvino 1988, p. 57). senses. The bodr' ;., :,
ARCHITECTURE AND THE EXISTENTIAL SENSE 587
t¡ Villa Mairea,
'i3rn and ma¡n
:. 'au9ht, practised, Fig. 30.6 Site-specific and tactile architecture built of materials collected from the
::ri cosmic relations site. Juhani Pallasmaa, Summer atelier of a painter, Vánó lslands,1g7O.
:-:3d upon in today's (Source: Juhani Paliasmaa Architects, Helsinki.)
an instrument of social and sexual appeal. Current consumer capitalism has even devel- The'pathologies' oi
oped a 'new technocracy of sensuality' and shrewd strategies of 'multi-sensory market- of the biased vierr-s r
ing' for the purposes of sensory seduction and product differentiation. This commercial architecture todar- s
manipulation of the senses aims at creating a state of 'hlperaesthesia' in the consumer objects that evoke a¡
(see Howes zoo5). Artificial scents are added to allkinds ofproducts andspaces, whereas our very life world ¿
'muzak' conditions our mood in shops, vehicles, and increasingly also in public spaces. encounter with the ,,,
We have undoubtedly entered an era of manipulated and branded sensations. As we The evident erper
enter one of today's shopping malls, department stores, or boutiques, we are usually standard buildings F
secretly manipulated and seduced by colour, sound, smell, and tactile materials as well sensoryrealm. The u
as directing and sequencing of movement, attention, and desire. Signature architecture, Montagu: 'We in th,
aimed at creating eye-catching, recognizable, and memorable visual images, or archi- This growing a\iare:
tecturaltrademarks, is another example of sensoryexploitation, the attempt'to colonize painful deprivation '-
by canalising the "mind space" of the consumer' (Howes zoo5, p. 288). (Montagu r98ó l. Se'
We are living in an age of aestheticization without being hardly aware of it. Every- architecture at largc
thing is aestheticized today; consumer products, personality and behaviour, politics, well as the negJectei
and ultimately even war. Formal and aesthetic qualities in architecture have replaced Salter zoo6), touc:r
functional, cultural, and existential criteria. An aestheticized surface appeal has dis- experiences of space
placed meaning and social significance. The social idealism and compassion that Alongwith the ph
gave modernism its sense of optimism and empathy have frequently been replaced by body, a renerved inlr
formalist retinal rhetoric. practising architects
shiftfrom abstract üL
place in schools of ¿:
ential and existent:¿
phenomenologicar p
today's internatitr;l¿
RnvoruTroN THRoucH THE sENSES Tadao Ando, Pete:
john and Patricia F.
The long neglect of the human sensory essence, and the disregard of the embodied the sensory qualitie,
processes in our existential experiences and cognition are, however, giving rise to an simple in order t.', s
expanding literature on the senses and the various aspects of human embodiment colour, and light. T:
(Ackerman r99r; Howes zoo5; Pallasmaa z006, zoog). The biased retinality and the lated or sentiment¿
displacement of the body in western cultural awareness have been convincingly countries as uell ¿s r
revealed by a number of philosophers (e.g. Levin 1993; Iay ryg4. The recent studies of qualities of archite¡
the role of embodied processes extend all the wayto the acts of memorizing and think- the world.
ing (e.g. Casey zooo; Johnson 1987; Lakoff and Iohnson 1999). These writers emphasize
the significance ofthe senses in the entire human mode ofbeing, as well as in the eman-
cipation of the individual. Michel Serres states explicitly: 'If a revolt is to come, it will
have to come from the five senses' Ggg5,p.7r). David Michael Levin, a critic of biased
retinality, makes an equally concerned argument:
Rr
I think it is appropriate to challenge the hegemony of vision - the ocular-centrism of our cul-
ture. And I think we need to examine very critically the character of üsion that predominates
today in our world. We urgently need a diagnosis of the psychosocial pathology of everyday It is thought-pror t'k
seeing - and a critical understanding of ourselves as visionary beings. The psychoiogist Ia:
(Levin 1993, p. zo5) dent and isolated ¡::
ARCHITECTURE AND THE EXISTENTIAL SENSE t89
* ir", :, ¡n devel- The 'pathologies' of today's architecture can likewise be revealed through an analysis
': - " :-,'narket- of the biased views of the human condition that prevail in the standard practices of
hll'," - t i -,nmercial
'':'
architecture today (see Pallasmaa zoo5). Instead of aspiring to create impressive visual
Mlil'¿ .- --:-; a0nsUmer
.. objects that evoke awe and admiration, profound architecture aims at rooting us in
: l l-:!. \\'hefeas our very life world and emancipating our senses for an autonomous and authentic
,ii.,,, -- : -.blic spaces. encounter with the world.
!rc{ .:::,::-ünS. AS We The evident experiential impoverishment and mental meaninglessness of today's
li; *:: '.\'a a¡e USUally standard buildings have given rise to a deepening concern of the suppression of the
.-:r..,.: :t::arials aS well sensory realm. The urgency of this concern is exemplified by the anthropologist Ashley
c: ::' :: ¿rchitecture, Montagu: 'We in the Western world are beginning to discover our neglected senses.
i-i.i -::1lies, or archi- This growing awareness represents something of an overdue insurgency against the
. r:-;::lrl t'to colonize painful deprivation of sensory experience we have suffered in our technologized world'
:t'i (Montagu 1986). Several recent books analyse the embodied and sensory essence of
.' :'"\ ;lie of it. Every- architecture at large (e.g. Bloomer and Moore ry77;Malnar and vodvarka zoo4), as
.: :::'riour, politics, well as the neglected potentials of the repressed senses of hearing (e.g. Blesser and
:::..f. ha\-e replaced salter zoo6), touch (e.g. classen zoo5), and smell (e.g. Barbara and perliss zoo6) in
::.;; appeal has dis- experiences of space, place, and architecture.
:-: ..rmpassion that Along with the philosophers' critique of biased visuality and the displacement of the
:-', !3.t replacedby body, a renewed interest in the multisensory essence of architecture has arisen among
practising architects, architectural writers, and educators. Today we can see a decisive
shift from abstract conceptuality and visual formalism towards a sensory realism taking
place in schools of architecture around the world. The renewed interest in the experi-
ential and existential dimensions of architecture has also resulted in the popularity of
phenomenological philosophy as a theoretical approach to the art of building. Many of
ENSES today's internationally recognized architects ofdiffering stylistic persuasions, such as
Tadao Ando, Peter Zumthor, steven Holl, Rick loy, Tod williams and Billie Tsien,
John and Patricia Patkau, and Kengo Kuma, have expressed their explicit interests in
:,1 rrf the embodied the sensory qualities of architecture. Their buildings are often formally deliberately
,-.r. si\-ing rise to an simple in order to strengthen and focus sensory experiences of materiality, gravity,
;inan embodiment colout, and light. Their architectures give rise to open poetic images instead of calcu-
; :etinality and the lated or sentimental effects. countless architects in Spain, Portugal, and the Nordic
:een convincingly countries as well as many South American countries explore the sensory and sensual
-:1.
recent studies of qualities of architecture more than architects of the celebrated metropolitan centres of
:-i.r:izing and think- the world.
"e irriters emphasize
''';ell as in the eman-
rli is to come, it will
-1. a critic of biased
the basic orienting system, and the haptic system. Already medieval philosophy identi- to speak of the place ''
fied a sixth unif ing sense, the sense of selfhood, through which the self acknowledges mingle [. . .] The sll: ..
itself as a distinct entity (Connor zoo5). Regardless of the fact that the concept of the
singular self, 'the I', is central in western consciousness, the origins of selfhood in our
embodied experiences and the sensory world continue to be neglected in contempo- In addition to rend¡r
grates our experienco
rary culture.
To make the notion and categorization ofthe senses even more ambiguous, we need
The experience .-,:
to be reminded that the philosophy of Rudolf Steiner argues that we actually utilize no
entity and recollec:iu
less than twelve senses. The twelve senses identified by Steinerian anthropology and
skin and the faculn-
express anlthine ot
spiritual psychology are: touch, life sense, self-movement sense, balance, smell, taste,
poetically (Kearnel r
vision, temperature sense, hearing, language sense, conceptual sense, and ego sense
(Soesman 1998). The Steinerian ego sense is obviously related with the sixth sense of ofarchitecture.
Even visual perce
medieval thought.
the self; my bodr- re
It is evident that experiences and qualities ofarchitecture cannot be fully described
moment of rvaking'.
by our five sense modalities. In addition to the five senses, experiences of architecture
Lost Timereconstru.
involve, for example, the sensations of gravity and lightness, the counterpoint of hori-
zontality and verticality, movement and balance, sense of centre and focus, tension and fhis body] ribs, its l.r
ease, time and duration, not to speak of the role of memory and projection of intentional
It is evident th¿:
experiential -'r'or1d ;,
meanings. These sensations call for skeletal and muscular imageries, as well as the engage-
together by a haplic ,
ment of the body scheme, memory, and imagination. Profound architecture evokes and
sensualbody is truli-
addresses our comprehensive existential sense. The phenomena of architecture do not
perspective, but as :h
take place in an abstracted and detached world; architecture occupies our life world, and
it is encountered and measured primarily by our very sense of being.
Our contact with the world takes place at the boundary of the self, through specialized
parts of the enveloping membrane and its extensions and projections. The senses are
truly amazing extensions of the physical body; as Martin fay notes, 'through vision we
touch the sun and the stars' (quoted in Levin 1993, p. r3o).
The senses are specializations of skin tissue, and all sensory experiences are modes
of touching, and thus fundamentally related with tactility. We also touch the world by
the eyes, ears' nose) and tongue. In fact, Aristotle already described taste and seeing as
forms of touching (Stewart zoo5, p. 6r).
The skin and the sense of touch are essential in the 'philosophy of mingled bodies'
(Connor 2oo5,p.3zz) of Michel Serres:
Fig. 30.7 Design iFo
In the skin, through the skin, the world and the body touch, defining their common border.
on the grip of three
Contingency means mutual touching: world and body meet and caress the skin. I do not like
ARCHITECTURE AND THE EXISTENTIAL SENSE 59r
identi- to speak of the place where my body exists as a milieu, preferring rather to say that things
f ',: -:t'¡rvledg€s
irtrrn mingle [. . .] The skin intervenes in the things of the world and brings about their mingling.
lLi'lr : :;eDt of the (Connor zoo5,p.3zz)
''
fou .r-:::t-rd in
, our
In addition to rendering a diffuse and dynamic boundary, the haptic body sense inte-
$tln:::: ,- aontempo- grates our experiences of the world and ourselves (Fig. 3o.7).
The experience of the self is primarily a haptic continuum rather than a cerebral
L I.iert15, rVe need
I llLll-
entity and recollection. Also Serres associates the capacity of self-awareness with the
M: -!--:uJh-utilize no
skin and the faculty of touch (Connor zoo5,p.3z3). 'Howwould the painter or poet
w "=ropology and
express anything other than his encounter with the world', asks Merleau-Ponty
:iL::1.e, smell, taste,
poetically (Kearney g94,p.82), and this remark surely also applies to the expressions
*:r¿. and ego Sense
-re slrlh ofarchitecture.
:--:r sense of
Even visual perceptions are fused and integrated into the haptic continuum of
the self; my body remembers who I am and how I am located in the world. At the
::: :e fullr'described
moment of waking up from his sleep, the protagonist of Marcel Proust's In Search of
.::;es oiarchitecture
Lost Timereconstructs his identity and location through 'the composite memory of its
, rnterpoint ofhori-
:i rocus, tension and
[his body] ribs, its knees, its shoulder-blades' (Proust rgr,lrggz,pp. 4-).
,e¡ion of intentional It is evident that the miraculous consistency, continuity, and permanence of the
experiential world cannot arise from fragmentary visual percepts; the world is held
. ¿-i rr-ell as the engage-
together by a haptic awareness and the continuum of the sense of self. My sensing and
:hite.-ture evokes and
sensual body is truly the navel of my world, not merely as the viewing point of a central
- ¿rchitecture do not
perspective, but as the sole locus of integration, reference, memory, and imagination.
::s rrur Li[e world, and
l
. --lrt¡ u gh specialized
irt¡ns. The senses are
s. 'rhrough vision we
r- of mingled bodies'
Fig. 30.7 Design for the touch of hand. Juhani Pallasmaa, door pull studies based
-l;i: ;ommon border. on the gr¡p of three fingers, bronze, 1 991.
i --:. ¡¡in. I do not like
Photograph: Rauno Tráskelin. Source: Juhani Pallasmaa Architects, Helsinki.)
592 IUHANI PALLASMAA
SENsonY UNDERSTANDING
The sensory systems are not only passive receptors of stimuli; they constantly stretch
out, seek, investigate, and shape the entity of the world and ourselves. My senses
and bodily being are centres of tacit knowledge, and they structure my entire being-
in-the-world. All the senses as well as my very being 'think', in the sense of processing,
structuring, and grasping utterly complex existential situations. Phenomena of art and
architecture are essentially engaged with our sense of existential understanding; works
of art are encountered and lived rather than understood intellectually. As |ean-Paul
Sartre states: 'Understanding is not a quality coming to human reality from the
outside; it is its characteristic way of existing' (Sartre ry%,p.9).
We need to distinguish a silent existential or lived knowledge that is distinct from
conceptual knowledge attached to intelligence, concepts, and language. Yet, the sig-
nificance of embodied and existential understanding is grossly underestimated in
contemporary educational philosophies.
Scientists associate the evolution of the human brain with the development of the
hand as a precision instrument, and Martin Heidegger, among others, extends the
sensory processes ofthinking to the hand:
[T]he hand's essence can never be determined, or explained, by its being an organ which can
grasp [. . .] every motion of the hand in every one of its works carries itself through the element
of thinking, every bearing of the hand bears itself in the element. All the work of the hand is
rooted in thinking.
(Heidegger ry77, p. 357)
During the past couple of decades, numerous writers have pointed out the biased
dominance of the eye in the Western culture, 'the hegemony of vision' (Levin 1993),
and especially in today's reality of mass-produced visual imagery. However, a charac-
teristic of vision, that has been studied much less, is the implicit tendency of vision to
interact and integrate with the other sense modalities (see essays by Spence, Chapter 4,
and by Melcher andZampini, Chapter r4, this volume). We learn about the optics of
vision and the mechanics of perspectival representation, but not about the thickness
and impurity of vision, the essential merging of other sensory modalities with vision.
Even the current interest in the significance of the senses continues to regard them Fig.30.8 The Mls
as independent and detached sensory realms instead of understanding our sensory with charms to i"r
relation with the world as a fully integrated existential complex of interaction. 'The London.
hands want to see, the eyes want to caress', as Goethe already provocatively argued
(quoted in Hodge 1998, p. r3o).
ARCHITECTURE AND THE EXISTENTIAL SENSE 593
SENSES
thing,a unique way of being, which speaks to all my senses at once' (Merleau-Ponty
r-r.;nne even Claimed ry64b, p. 48). Bachelard calls this sensory interaction 'the pollphony of the senses'
.::nent of his colours (ry7t,p.6). 'The translate each other without any need of an interpreter, and are
senses
: :i t-hings and will not naturally comprehensible without the intervention of any idea', Merleau-Ponty claims
-i¡ is for us the (t962,p.235). Altogether, in normal situations oflife the senses do not mediate specific,
'ihich
detached, and categorized percepts. All sensations are integrated in the full experience
:-:;u-Ponty r964, VII) of the world and of ourselves, our full existential experience, and sense of life.
Research in the neurosciences provides swiftlyincreasing information on the extraor-
. is not only what it
dinary interconnectedness and interactions ofthe various sensory areas ofthe brain
.rocosmic and con-
(Sachs 2oo5,p.33). The flexibility of the human sensory complex has also become evi-
:1e human condition
dent in studies of the sensory capabilities of the blind. 'The world of the blind, of the
:erate an indivisible
blinded, it seems, can be especially rich in such in-between states - the intersensory, the
,iect the authority of
metamodal - states for which we have no common language', Oliver Sachs argues,
1'rn orld and becomes
continuing: 'And all of these [. . .] blend into a single fundamental sense, a deep atten-
tiveness, a Slow, almost prehensible attention, a sensuous, intimate being at one with
:r Derr,ey points out
the world which sight, with its quick, flickering, facile quality, continually distracts us
id integration:
from' (Sachs zoo5, p. 36).
s. h¿r'e aesthetic qual-
-ns. not as simple and
:fstLr colouls, sounds
-::rS¡; which the total
-:Jiries are as such, or
-aisson'or associated
SENSNS IN EXPERIENCING ARCHITECTURE
t.-¡i'means of the eyes
rr-!ks, the bareness of
Spaces, places, and buildings are undoubtedly encountered as multisensory lived
.,.nspicuous and con-
c;alities do not stand experiences. Instead of registering architecture merely as visual images, we scan our
r. : . .] Any sensuous settings by the ears, skin, nose, and tongue. The suggestions that the sense of taste
wouldhave a role in the appreciation ofarchitecture maysoundpreposterous. However'
l:itev r98o, pp.rzz-4) polished and coloured stone as well as colours in general, and finely crafted wood
details, for instance, often evoke an awareness of mouth and taste' Carlo Scarpa's
3ar. and experience
architectural details frequently evoke sensations of taste. Adrian Stokes, the British
::,ns. Besides, all our
artist and writer writes appropriately of the 'oral invitation ofVeronese marble' (Stokes
:1Lrries, dreams and
r978, p.3ró).
lnd qualities of the Everypiece of architecture has its auditive, haptic, olfactory, and gustatoryqualities,
.he rvorld'..r
and those qualities even give the visual percept its sense of fullness and life, in the same
i:stant integration way that a painting of Claude Monet, Pierre Bonnard, or Henri Matisse evokes a full
. lactile and audible
sense oflived reality. Profound architectural experiences also evoke the sensations of
:que structure of the
gravity and balance, near and far, in front and behind. In addition to the physical and
material qualities, architecture also conditions our sense of duration and time, histo-
.-:: - The Chiasm'in
- Ei anston, IL., fourth ricity, and silence.
::.'bodi. is shared by Structuring and articulating time-slowing down, halting, speeding up, reversing-
:-::e that returns to is as important in architectufe as the manipulation of space. And more importantly,
. jralectical principle of
we do not live in physical settings like being on a stage; the space creates a continuum
; i-esh' as the ultimate
::..l: meaning is both with our mental space and our very sense of self. As I occupy a space, the space occupies
:r t994). me forming a chiasmatic singularity. Rainer Maria Rilke has introduced the poetic
596 IUHANI PALLASMAA
RnrnnnNcrs
Ackermann D {r99:
,rr,':,::..r::ll,:'irll:lllllrlil:ll:i:1a:1!:S1,:::tl:rt:l:i::l iuu: Asplund EG (19,16 , i,
BachelardG(rs6q. -
Fig. 30.9 An architecture for all the senses including the kinaesthetic and BachelardG(r9.r. -;
olfactory senses. The design creates countless places for intimate or group Barbara A and Per-r.,
encounters. Lawrence Halprin, Auditorium Forecourt (now lra Keller Fountain), of Smell. Skr¡a. i1-
1956' Portland' oregon' Blesser B and Salter -.
(photograph:courtesy of portrand water Bureau.) The MIT Press. t:'
Bloomer KC and l''Í:
New Ha"'en, CT.
notion of Weltinnenreum,lhe inner space of the world (Enwald ry97,p.8). An instant
Calvino I (1988 '. -{;,': -',
unconscious exchange takes place; I project my body, memories and values on the
Casey ES (zooo . -l
space and the space becomes part of me and my consciousness. 'I am what is around Bloomington. ,)i.
me', Wallace Stevens, the poet, argues (r99o, p. 86). 'I am the space where I am', estab- ClassenC(zoo5.Ir:i
lishes Noel Arnaud, another poet (quoted in Bachelard D69, p.l4.7), and finally, 'I am ConnorS(zoo5.l'1-:
myworld', Ludwig Wittgenstein, the philosopher, concludes (1972, p. 68). Berg Publishers. 'l'
A particularly important aspect of architectural experience is the unconscious tactile Dewey J (r98o). .r,-. ...
perception concealed in vision; as John Dewey and Merleau-Ponty argue, we experi- Enwald L, ed. rii-
-,
Letters tgoo-t9:c .
ence the weight, texture, temperature, and moisture of materials and surfaces, and
Grosset PA (r98- . E.,
these hidden qualities make the scene inviting or repulsive, protective or threatening.
HeideggerM(rq.. '
The sense of alienation, and even necrophilia, often encountered in buildings of today pp.341-58. H¿¡:¡:
arises, at least partly, from the tactile poverty, abstraction, and unreality of the uncon- Hodge B, ed. r99l
scious haptic experience mediated by vision. The routine curtain wall office buildings Watercolours. H,:'
in any city today exemplifi' this feeling of outsideness and alienation, or 'existential Howes D (zooj . F-r.
outsideness', to use the powerful notion of Edward Relph.a The immateriality and Empire of the S:.:,,
Jay M (tgq+). D t'¡',,,.". : ";
a 'Existential outsideness involves a selfconscious and reflective uninvolvement, an aiienation University of C¿-,:
from people and places, homeiessness, a sense of the unreality of the world, and of not belonging.' JohnsonM (r98; . I:
Edward Relph (1976). Place and Placelessness, p. 5r. Pion Limited, London. The Universi¡ c,¡ L
ARCHITECTURE AND THE EXISTENTIAL SENSE 597
RnrsnnNcns
Ackermann D (r99r). A Natural History of the Senses.Yíntage Books, New York.
Asplund EG (rq:6). Konst och TekniklArt and Technologyl. Byggmástaren, Stockholm.
Bachelard G (rg6q). The Poetics of Space. Beacon Press, Boston, MA.
sthetic and Bachelard G GgZt). The Poetics of Reverie. Beacon Press, Boston, MA.
¡e or group Barbara A and Perliss A (zoo6). Invisible Architecture: Experiencing Places through the Sense
eiler Fountain), of Smell. Skira, Milan.
Blesser B and Salter LR ( zoo6). Spaces Speak, AreyouListening?: ExperiencingAuralArchitecture.
. := tcnrand Water Bureau.) The MIT Press, Cambridge, MA.
Bloomer KC and Moore CW (tgZZ). Body, Memory and Architecture.Yale University Press,
¡9-.p. 8).An instant New Haven, CT.
s and values on the Calvino I (rqS8). Six Memos for the Next Millennium. Vintage Books, New York.
Casey ES (zooo). Remembering: A Phenomenological Study. IndiaÍa University Press,
i ¿m rvhat is around
Bloomington, IN.
: r,n here I am', estab-
Classen C (zoo5). The Book of Touch. Berg Publishers, Oxford.
- . and finally,'I am Connor S (zoo5). Michel Serres' five senses. In D Howes, ed. Empire of the Senses, pp. 318-34.
r p. 68). Berg Publishers, Oxford.
ru¡conscious tactile Dewey I 098o). Art As Experience. The Berkley Publishing Group, New York.
f,'argue, we experi- Enwald L, ed. Gggz). Lukijalle. In Rainer Maria Rilke: The Silent Innermost Core of Art
.s and surfaces, and Letters tgoo-t926. TAI-teos, Helsinki.
Grosset PA (rg8Z). Eyes which see. Casabella,Lt5,53;-2.
;:ir-e or threatening.
Heidegger M Ggll). What calls for thinking. In D Krell, ed. Martin Heidegger Basic Writings,
n buildings oftoday
pp. 341-58. Harper & Row Publishers, New York.
'eaiin. of the uncon-
Hodge B, ed. (rgq8) Not Architecture But Evidence That ItExists -
Lauretta Vinciarelli:
¡,'all of6ce buildings Watercolours. Harvard University Graduate School of Design, Harvard, MA.
i:ion, or'existential Howes D (zoo5). Hyperesthesia, or The Sensual Logic of Late Capitalism. In D Howes, ed.
: immateriality and Empire of the Senses, pp. z8r-3o3. Berg Publishers, Oxford.
IayM (tgg+). Downcast Eyes -The Denigration of Vision in Twentieth-Century French Thought.
:.rt. an aiienation University of California Press, Berkeley, CA.
.¿ uri not belonging.' Johnson M (987) . The Body in the Mind: The Bodily Basis of Meaning Imagination, and Reason.
The University of Chicago Press, Chicago, IL.
598 JUHANI PALLASMAA
wittkower R (rq8s). Architectural Principles in the Age of Humanism. Academy Editions, representation. a rr,,
London/St. Martin's Press, New York. 'as if ' he was real. r.-.
Do we beliele r:r
believes in the re¿Lr,
what we perceile ts
world, or as if hcti,-,:
It is a fact th¿t '..
fictional characteri :
is real?
:;iropeat't
:: : ::ss. \linneapolis,
irttír¿
and T Kilmartin
REALITY
o.f the Senses,
. ,.:e Enlightenment. Cscru¡ is sitting in a movie theatre and watching The Purple Rose of Cairo Q985)
once again. Suddenly, Tom Baxter, the main character, comes out from the screen
'':: ..::e,7 Projecf. Wiley and invites Cecilia for a romance. Tom, a character, has become alive and now walks
with her in the real world, while cecilia, the real girl, can have a walk in the shining,
: ::t:kielma, proposi-
fictional world of rhe Purple Rose of cairo (Fig. 3r.r). To cecilia, Tom is no longer a
: representation, a moving image with sound, which excites her imagination to behave
--cademy Editions,
'as if ' he was real. Cecilia can now believe in the reality of Tom.
Do we believe in the reality of fictional characters, places, and worlds, as Cecilia
believes in the reality of Tom? Can we forget the presence of the medium or the fact that
what we perceive is not 'filmed' reality? Can we feel as if being there, in the fictional
world, or as if fictional characters were here in our real world?
It is a fact that we are strongly captivated by the story, that we are moved by
fictional characters and worlds. But are we fooled as to believe that what we perceive
is real?
ELENA PASQUINELLI
|ohnston r98+): the illusion that fictional characters, including animated characters,
are real and alive.
Philosophers have long discussed the nature of fiction and of the public's engage-
ment in fiction. An exemplary debate was raised inry75 by a paper from philosopher
Colin Radford, who introduced the 'paradox of fiction' (Radford 1975). In short,
Radford asked: how can we be moved by the fate of Anna Karenina? In fact we can
only be moved if we believe in the existence of Anna Karenina (if we hold existence
beliefs). But we know thatAnna Karenina is not real (we do not hold existence beliefs).
And in spite of this, we are moved, irrationally moved. The subsequent debate shows
a substantial accord on the fact that existence beliefs are not sufficient and not neces-
sary in order to explain the public's engagement and emotions produced by fiction.
It is a fact that reactions to fiction are similar but not identical to reactions to reality and
beliefs (Carroll r99o; Currie 1995; Walton zoor). First of all, behavioural reactions are
mostly inhibited. Spectators might close their eyes, startle, cry, shout, but they do not
go further as if they did reallybelieve in the reality of what they see. They do not call the
police or flee away from horror movies. And if we accepted the myth that Lumiére's
spectators did leave their seats because they did believe in the existence of the filmed
train, we should nevertheless acknowledge that this phenomenon has not often been
reproduced. Second, emotions and beliefs generated by cinema and fiction in general
ed by
have a short duration: they disappear more or less with the end of the movie. In short,
it is true that we are moved by the fate of Anna Karenina, thrilled by the menace of
green slime and horrified byblood-sucking entities, but we do not come to believe that
our world is populated by monsters or grieve for long. This difference can be explained
ifwe assume that adults who endorse fictional experiences justpretendlobelieve in the
existence of the green monster, and for consequence pretend to feel (or really feel)
afraid (Walton7978, zoor). Like mud and sand in children's games of make-believe'
images, sounds, and words are the props that excite, sustain, and constrain a mental
activity that does not consist in forming beliefs, but imaginings. Imagination and
o; the image of terri- belief must nevertheless be related if we want to account for the similarity-and not
rtriections of Arrival just for the difference-between responses to the real world and responses to fictional
mrth aptly describes ones: because during the green monster movie we are afraid or quasi-afraid. This is
ill,Lrsionistic' of arts. possible ifwe consider-as Currie does (Currie r99)-thatlhe activityof imagination
ur contrast with the which responds to fiction is a form of simulation in which beliefs are run offline. This
r-ntbrce this realism. means that while beliefs are cut off from their normal causes (the existence of some
e-tion of reality, and entity or event) and from their behavioural consequences (actions and full emotions),
.-'f cinema history, the connections that normallyholdbetweenbeliefs (a monster is a dangerous creature)
io: presenting their and between beliefs and emotions (a monster is frightening) are preserved. Fictional
nsters. And creative worlds that are capable of inducing this belief-like mental state of imagination can
d nuining the capac- be defined as believable woilds. Since theories based on existence belief can explain
Clair 1985). In r8r7, only the similarity between responses to real and fictional worlds, theories based
lisbelief ' (Coleridge on imagination have a greater explanatory power, because they can account also for the
eahnr (Thomas and differences.
6oz ELENA PASQUINELLI
environment of the Pompeii ruins (for this reason called 'mixed reality system'). The
application consists ofthe simulation of a day in ancient Pompeii life. Five realistic virtual
humans interact with each other and manipulate objects around a tavern following a
Other experiencrs
script. Users wear a head-mounted display and witness the representation while moüng
through real ruins. Hair and cloth animation, skin rendering, behavioural animation,
with virtual agents i
facial emotion expression, intentionality, desires, beliefs, and actions are accurately rep-
the original script o:
resented (Papagiannakis et al. zooz, zoo5). Virtual characters of the ürtual world do not
Animated puppets n
have, however, any'awareness' ofthe real users: they do not react to their presence, but
by four ball-like crea
continue to play their script. Additionally, real users simultaneously exploring the scene
cannot interact between each other. It is as if Cecilia could jump into the world of
the Purple Rose of Cairovnthout being noticed by Tom Baxter and the other characters.
Fi$.31.2 Reconstruction of a travem in Pompeii, with virtual animated characters. Fig. 31.4 The WoEE
Project Life Plus, MlRALab, University of Geneva. animated art piece).
(Source: http://lifeplus.miralab.unige.ch/.)
MULTIMoDAL, INTERACTIVE MEDIA 6o3
NTEDIA
''¡e
d br- the develop-
. realin' applications
li:rtive forms of cre-
:rgutruslv refers to a
:;santations that are
:;ed bv the notion of
: nultimodalstimu-
::a >creens that cover
- :,,r,¡r'ersial whether
: :. appiied to more
r.-=i :ooo, Lombard
:.t-r :r-t iliustrate my
Fi9.31.3 lnteracting with virtual animated characters in pompeii ruins. project
:,: j".: up in the real
Life Plus, MlRALab, University of Geneva.
::':-l:'- >r'Stem'). The
(Source: http://lifeplus.miralab.unige.ch/.)
:. :: ,¿ realistic virtual
i .:i.trn tbllowing a
.- --ln rr'hile moüng Other experiences of virtual drama have been conceived to allow users to interact
.¿',t - -:ral animation,
with virtual agents embedded into the virtual environment, and to actively modify
ri .:; tccuratelyrep- the original script of the play. This is the case in Edge of Intention (Reilly ry96) and
,"-:::J rr'orid do not Animated puppets (Doyle 2oo2, Hayes-Roth zoor): the fictional world is inhabited
; --::::resence, but by four ball-like creatures called Woogles (Fig. 3r.4). One of them is controlled by the
, ::;-.'n¡g the scene
r -_.:¡ the world of
:i: -
--uer characters.
It is not dithc:lt:c
not just to see the it:ll
This is the sctr:e
".f
Force feedback r-',r h::
that material obiecis:
the touching aseni.
located in the nu';,,
are identified br use:
shape' can then ¡e r
and by applring rt--
of the user. Hapt:c d
Fig.31.5 An avatar in the virtual world of Second Life. interactive nature cra
(Source: http://community.flexiblelearning.net.au/.)
ception of the re¿l "',
ments (other de"ic¡,
user, while the others are controlled by the computer. There is no set story, but the provide tactile. supei
Woogles have personalities, emotions, and engage in social behaviours. Spectators are currentlY in us¿ :
become actors in a not-completely predefined world. The world is, nevertheless, struc- visualize three-dinel
tured by the definition of the environment, personality, and possible reactions of the and also explore Lrei:
computer-guided characters. There's no interactivity between human users. et al. zoo5). In a cr¿"
Interactivity between users is, on the contrary, an essential attribute of Second Life, exploration could ¡t
a slmthetic online world where agents are digital representations of users (called (Figs. 3r.7 and,¡r.S .
'avatars'). In addition to human-to-human social interaction, Second Lifeusers are It is a fact, then. --¡
enabled to create virtual objects ofart, pieces offurniture, and architectural buildings. and creative e\Teni:
This is done by combining and modifying elementary forms (a sort of virtual and But the question is: ;:
flexible Lego building set). Complex objects are subsequently associated with colours realm oftradition¿- ¿
and textures, but also sounds in order to create multimodal objects, including art Does virtual reaLit-'' r:
objects (Figs. 3r.5 and 3r.6). users by inducing e:t
An optimistic ar:s-,
tative shift.
Optimism abtrrt
encounters none:re.
has not yet receir-ed ;
tative shift torvard' ;
ture about virtu¿ r¡
'illusion of realir'": i
what follows I rril' .
sponds to specihc c c'
different possibiJi:r;
sures of achievemen'
Fig. 31.6 Virtual environment with avatars in Second Life. with virtual reaLi¡- s
[Sou rce: http://www.jmir.org/.) pursuing an undersl
MULTIMODAL, INTERACTIVE MEDIA 6O5
It is not difñcult to imagine a further evolution in which users will be able to feel and
notjust to see the texture ofthe virtual objects ofart (Frisoli et a|. zoo6)'
This is the scope of so-called haptic devices, a crucial feature ofvirtual reality systems'
Force feedback or haptic deüces exploit one specific component of active touch: the fact
that material objects resist (oppose a contrary force to) the force which is impressed by
the touching agent. This force is perceived through deep touch receptors that are
located in the muscles and joints rather than in the skin. It conveys sensations that
are identified by users as specifi.c textures, materials, or as specific shapes. A
'haptic
shape' can then be represented by producing a characteristic pattern of resistance
urrd by upplying it-thanks to a material, suitable device-to the fingers, hand' or
arm
of th" .rret. gaptic deüces are exemplary of the multimodal possibilities and of the
interactive nature of virtual reality systems. Perceptual experience depends' as in per-
ie, :,: ::- .¡.ret.au/.J ception of the real world, on the specific perceiver's actions and exploratory move-
ments (other devices exist that passively stimulate the skin through vibrations and
rl Str-rr\-, but the provide tactile, superficial sensation, or even temperature perception). Haptic deüces
:,-:rs. Spectators ur. cor.ently in use for producing virtual museums of Pure Form in which users can
(sculptures),
\-Értheless, struc- visualize three-dimensional graphic reproductions of distantworks of art
fromthe contact (Carfozzino
: :e¿fiions of the andalso explorethemwiththeirhands andfeeltheir shape
et al. zoo5). In a creative perspective, virtual sculptures could also be modified and
1Li_iers.
ie ot Second Life, exploration could become active creation of new works of haptic or multimodal art
ti users (called (Figs. 3r.7 and 3r.8).
,:oi Litb users are 11 is a fact, then, that thanks to virtual reality systems,
new forms of aesthetic, social,
:.-tura1 buildings. and creative experiences are within reach that enlarge the field of human possibilities'
But the question is: are virtual reality systems responsible for a qualitative shift from
the
¡t of r-irtual and
realm of traditional engagement through imagination to the realm of illusion of reality?
rted rrith colours
Does virtual reality make a real, qualitative difference in the perception and
cognition of
rs. including art
users by inducing existence beliefs?
An optimistic answer largely shared in the domain ofvirtual reality (Lombard and
is
Ditton 1997; Witmer and Singer 1998; Dinh et al' 1999; Slater and Steed zooo)'
Applications like second Life, Lifütus, or the Museum of Pure Form would then be
betier candidates than traditional cinema to produce a compelling illusion of reality'
Multimodality and interactivity are identified as the responsible factors for this quali-
tative shift.
Optimism about the capacity of new media for producing full illusions of reality
encounters nonetheless a major conceptual difficulty: the notion of illusion of
reality
rc has not yet received an uncontroversial definition. How, then, can we measure
tative shift towards an objective that we are not able to precisely identifu? In
a quali-
the litera-
ture about virtual realiry we encounter' in fact, at least three different definitions of
'illusion of reality': illusion of non-mediation, fidelity, and illusion of being there' In
what follows I will show that each of these uses of the term 'illusion of reality' corre-
ru sponds to specific cognitive and perceptual attitudes. Furthermore, each ofthem entails
different possibilities of success, different enhancing conditions' and different mea-
sures of achievement. My claim is thus that designers of creative and artistic experiences
with virtual reality systems should rather carefully choose their objective rather than
r .tc://www.jmir.orq/.) pursuing an underspecified idea of illusion of reality'
6o6 ELENA PASQUINELLI
the real world . . .' (Dennett 199r, p. 7). The illusion of non-mediation thus reminds us
of the famous mental experiment of the brain in a vat (Putnam l98z). A mad scientist
has removed the brain from the body of a person' The brain is now kept alive
in a vat,
thank to a special liquid. Wires connect the brain to a computer which stimulates the
brain so as io prodoce perceptual sensations. These sensations include sensations of
movement, and all the sensations that a human being can receive in response to their
actions in a real world. Only, the world is a synthetic, digital simulation produced
by
the computer. The mental experiment of the brain in a vat has been used in order
to
illustrate the sceptical argument that we cannot know whether there is a world outside
or if we are jusi brains in vats or . . . lost in a virtual reality environment' Science
fiction experts canrecognizethebasicplotof TheMqtrixGggg),the directbrain record-
ings of Strange Days (t995), the basic features of cyberpunk literature. As people
immersed in the matrix or brains immersed in a vat, deluded users of virtual reality
ln.the Baptist' and to
systems are supposed to fail perceiving the medium (perceptual component)
r, a project of forget their kno*t.ag. of the medium (cognitive component)' The illusion of non-me-
diation is double: perceptual and cognitive.
:-:;:!
This definition attracts an objection on the side of motor theories of perceptual
Antonio Frisoli.)
functioning (O'Regan and Noé zoor; Stoffregen and Bardy zoor)' According to these
theories, perception is not the simple recording of states of the world: the
content of
perception varies both along factual conditions (how the world is) and along other
:non-mediation. co.rditions, such the way the perceiving organism moves. Let us discuss an example.
as
ence ofthe tech- A visual representation, for instance a statue of the Museum of Pure Forrz, is transmit-
ted through a head-mounted display, while an exoskeleton mediates haptic
represen-
ertificial world:
tations of the contours of the same statue. The user perceives the presence of
both
,i rrr misconstrue
media through their tactile system. The medium is then perceptually acknowledged
as
r.sence Research
part ofthe factual content ofperception. When thevisual stimulus is presented
through
,u,d produce the
lurg. ,..."n, that cover the virtual reality room by transforming it into what is called a
rsirrnal object in
'cave', the visual medium is no longer experienced in this direct fashion' Nonetheless'
the presence of the medium could still affect the perceiver's relation to
how things
flow in
ure' ,"eing a baseball game on television, for instance, structures the optical
movements of
a way tha;is different iro- ,."ing the same baseball
game live' and the
the perceiver have a different effect on the perceptual result in the two situations
(Stoffregen 1997); immersion in a flight simulator will leave behind the vestibular
components that are typical of the same multimodal expelience in the real
world
(stoffregen et al. zoo3). The presence ofa medium can thus have an effect upon perspec-
would not
tival or motor contents of perception, even in the case in which the medium
mediated and
affect the factual contents. These effects make clear the difference between
nature
non-mediated conditions, and perceptually inform the user about the mediated
ofits experience.
In addition to non-interpretative, natural capacities, other interpretative' learnt
capacities are used in order to make sense of experiences with virtual and
fictional
make-believe games paves the way to adults'
:nsive Virtual environments. A long apprenticeship with
though particular
capacity of dealing *ith representations that are not displayed a
perceptual medium. Can knowledge of wearing material devices (even if they are
not
,*=:::. Antonio Frisoli.)
6o8 ELENA PASQUINELL]
perceived) and awareness that the experience is of an artificial world be evicted from formally invited tc-r ¿',
the mind ofthe user? then, US television b:.
Weaker versions of this definition exist, however, that do not entail similar objec- audiences of the fic:r,
tions. It has been proposed, for instance, to consider the'illusion of non-mediation' as media, such as r-ir¡u¿.1
an attentional state (Witmer and Singer 1998; Slater and Steed zooo). Attention can The form ofhdeLr
alternatively be focused on the virtual world (like the world represented in Second of a special r¡pe: it c"-,i
Life),the mixed world (the real Pompeii ruins with virtual agents moving around), or ence and is called -.1':
on the real room and devices. Shifts towards the second condition can be provoked tively experienced s:
by sounds (produced by other people in the room, by the functioning of the devices), between the simulatc
by uncomfortable conditions (heary, constraining haptic or visual devices, nausea), or Thus it is groundec r
other perceptual agents of distraction (odours from the kitchen announcing lunch- The user of a ¡lis:
time). These shifts can be qualitatively and quantitatively measured; they will quantifr very similar to sens¿t
the compelling power of the virtual world in engaging the user's attention to focus acknowledge the :::
on a selected group of stimuli. However, shifting the focus of attention cannot be con- experience. Again. :rl
sidered as an illusion, because there is no error. The weaker version of the illusion of conditions for und::i
non-mediation is, then, not an illusion at all. It is now an empirical matter whether this However, othei f':
compelling focalisation of attention is enhanced by new media in comparison with Subjective fideli¡ :s.
traditional ones, and which media characteristics are more effective. Multisensoriality the features of the .'u,-
and immersion in the virtual environment seem to be rather effective (Witmer and cannot reproduce "1.
Singer 1998; Dinh et al. 1999; Slater and Steed zooo). In fact, theyboth isolate users from produced bv the n¡
the stimuli that come from the real world and require their attention in order to cope representation c-tf ¿ '
with multiple stimuli and to correctly act. from the compler:i"
with no medium. Il:
these systems can b<
Illusion of reality as fidelity visual stimuli. If '.r-i:r
produce fideli¡ thr -
The second definition of the term 'illusion of reality' is: illusion of fidelity to reality. classic visuo-haptrc L
Users are aware that their experience is the effect of a medium, but believe that the be exploited br- mu-::
medium reproduces or presents reality. Now, this is not true, since the medium repre- subject moves his ]::
sents events that do not occur in reality. As for the illusion of non-mediation, the
causes the surtace :,-'
fact that the user is mistaken is a crucial condition for ranging the experience between duces the use oiti¡ :
illusions. cannibal Holocaust (r98o), The Blqir witch Project Gggg), and fake snuff sight; on soñ]€ rrir --ai
movies all try to establish this form of illusion or hoax: that a fiction film is, in fact,
that they see anot::r
a documentary that represents actual reality. Let us consider the most famous case of
continue to har-e r:e
such a hoax. In 1938, a series of newsflashes interrupted a music programme of CBS in
they had lost cont: - ,
the united States: it was orson welles announcing a Martian landing close to a us proprioceptir-e ini': r:
farm. Some audiences took the fake newsflashes for real, and panicked. Audiences does not correspc':,J
knew that the sounds of destruction came from a radio and were recorded in a distant
Other co gnitir-e ct-,r:
place from them, but some of them also thought they were listening at the sound pro-
subjective fideli¡,. -;
duced by real events. The smuggling of fictional contents in place of real ones is the
ments with nr o alr
object of strict rules in several ethical codes of communication targeting hoax and other by touch; t:e s
fraud (Steele and Black1999). Atthe time ofWelles'broadcast of theWar of theWorlds,
same surface (the e:r
CBS escaped punishment because the audience was reminded of the fictional nature
match the percer';eJ
of the performance several times during the broadcast. Nonetheless, cBS was touch is more intlu¿:
MULTIMODAL' INTERACTIVE MEDIA 6o9
formally invited to avoid the 'we interrupt this programme' device for fiction' Since
to inform
then, US television broadcasts featuring realistic news bulletins post messages
I'"i Lr.- -,¡lec- audiences ofthe fictional nature ofthe spectacle. It is reasonable to require that new
media, such as virtual realify, do not escape this rule of ethical communication'
The form of fidelity to reality which is invoked in the domain of virtual reality
r li :*-- 11 C?D
is
,, : .:, i¿cond of a special tlpe: it concerns the perceptual and eventually motor aspects of the experi-
- ,..: ']ld ¡. of .n.. urrd is called subjective fidelity (Ftrccro 1995). It consists in the extent of subjec-
|ll.iiüi-: l": :l-r\-Oked tively experienced similarity between simulator and the real-life situation, hence
between the simulator and the simulated (Stoffregen et al. zoo3; Morice
M --..:er-ices), et al. zooS).
F'rrrlr ... :: : l;Sea ), Of Thus it is grounded in a comparison between reality and simulation.
E ;.- -.-rE lunch- The user of a flight simulator who finds sensations produced by the simulation
: to
'['f -,, qUantifz very similar to sensations experienced during natural experiences thus continues
nature of the
.f: :: - , :l tO fOCUS acknowledge the perceptual medium (the simulator), or the artificial
I -:t-ilr¡t be cOn- experience- Again, mystification and misperception of the medium are not necessary
: -- : ilusion of .o.rdition, for undergoing the experience of subjective fidelity'
, :- : : ,;-r ethef thiS However, other forms of illusion can be useful for producing subjective
fidelity'
- *:ri-s.rn $.ith Subjective fidelity is, in fact, significantly different from realism as the simulation of
r * -..:sorialitr- for instance, force-feedback devices
the features of the world. As we have seen before,
-",' that is
ll' : :i¡re¡ and cannot reproduce all the variety of tactile, kinesthetic, thermal stimulation
virtual
produced by the manipulation of real objects. The sensation of touching
:) tr.: lSeIs ffom a
r : j.r to cope ..pr"r"rrtution of a statue through a state-of-the-art haptic device will' then, be far
by the touching of a real statue
frám the complexity of natural sensations generated
obtained through
with no medium. However, the fidelity of the perceptual experience
associations with
these systems can be enhanced or suitably modified by appropriate
visual stimuli. Multimodal devices can thus exploit multimodal illusions
in order to
Let us examine some examples of
produce fidelity through 'unrealistic' stimulations.
and they can
' :i¿hX'. ilassic visuo-haptic ilhisions, as they have been described in literature,
as
-'¡uirr.."
li l: ::--'.fithe be exploited by multimodal displays. Gibson (1933) describes
the following illusion: a
: *.: - - l1 lepre- subject moves his hand along á straight surface while looking
through a prism that
well. Nielsen (1963) intro-
--:--.aiation, the .u,rr", the surface to look curved; he feels it to be curved as
:.:iÉnce between duces the use of the mirror. The subjects follow a straight
line with their hand in full
. ¿nd take snuff sight; on some of the trials, a mirror is introduced, unbeknownst
to the subiects' so
: ilm is, in fact, th"at they see another person's hand (which they believe to
be their own): the subjects
si :¿mous case of continue to have the sensation that the seen hand is their own, but they also feel as if
:lnme of CBS in they had lost control over its movement. in both cases discrepant
visual and haptic/
which
Ln: close to a US proprioceptive information produces a vivid haptic/proprioceptive experience
stimulation, but to the visual one'
l;.1¿d. Audiences io", .rot correspond to the haptic/proprioceptive
- :l¿d in a distant components can play a role in the aspect of the final percept' hence on
other cognitive
of their experi-
.: ihe sound pro- subjective fidelity. Lederman et al. (r98r) have presented the subjects
r: :eal ones is the ments with two different abrasive surfaces, one to be examined by
vision and the
:{rting hoax and that they were exploring one and the
other by touch; the subjects were led to believe
when asked to
same surface (the experimenters induced an assumption of unity).
,,
theWoilds,
.¡.. t'¡.f
surface, while vision is more influential when the subject is requested to evaluate the 'resilience to knor,, -e¡
spatial distribution of the dots for the same surface: the kind of solution adopted thus knows that what he ,:'
varies with the verbal instruction assigned to the subjects. This fact indicates that Let us go back ttr ..:
narratives, stories and even simple verbal instructions can influence what the user will having created an ¿"..
feel and thus have an effect on subjective fidelity. their avatar th-ine. ;
Perceptual fidelity can be evaluated through subjective measures, like free reports in objects, manipula::n,
which the user evaluates the degree of similaritybetween sensations in the artificial set then equivalent tLr::r
and sensations experienced during comparable natural situations (Stoffregen et al. illusion that, to sc,r::
zoo3). Rather than asking'how real does this seem?', a better question might be 'how the avatar are manl:.
closely does this resemble the real situation?' Stoffregen et al. (zoo3) propose the con- the'illusion of beinE
cept of action fidelity as a measure of the perception of fidelity for interactive systems tic) bias upon pro:r,
(like flight simulators): when actions in the virtual world are comparable to actions in tificationwith son.r: ¿
the real one, then the system shows a great fidelity. by vision (a prop:i,-'';
This measure makes sense onlyin the case of perceptual experiences that, moreover, sion of movement. l-
aim at reproducing real experiences. This can be the case for the Pure Formmuseum. the virtual r.vorlci tr¿
The user will, in fact, be able to compare how the simulated experience resembles and, for sure, the-" ¡::
previous experiences in real museums. But this measure will be unsuitable for virtual As an example r':'.
theatre with caricatured, purely frctional characters such as the Woogles. In this case, tive illusion induc;¿
the concept of fidelity will, rather, be applied to the complexity of the character's per- was their orvn i\i.ls,
sonality, the reactions ofcomputer-guided characters to the user-guided character's perceptual modaLt:.
actions, and their relative coherence with natural human beings (a form of social rather Mon-Williams e.i a,":
than a perceptual or motor fidelity). In the same way, multimodalitywill lose much of body experience r¿i
its power as an enhancing factor of fidelity, while the nature of the social interactions with discrepant visu.
will take the most importance. dently confirmed ll:
ception when the t-,'.,
parts of it like tr:
Illusion of reality as illusion of transportation Proprioceptive ilus:,-
ogies such as phari
A third possible definition of the term 'illusion of reality' is: illusion of being there in with pathologies c'ti::
the virtual environment (Riva et al. zoo3; slater and Steed zooo). can users experi- et al. 1995; Ntscrru-.
ence an illusion of being there withouf being mistaken about its fictional nature and been considered:i¡ i
without losing the awareness of mediation? In the literature about virtual reality, the mation presenteJ :..
illusion of non-mediation and the illusion of being there are confounded into the auditory, proprirr.e:
definition of presence: 'The International Society for Presence Research (ISPR) sup- been contested. en¿
ports academic research related to the concept of (tele)presence, commonly referred ently to the aspecl r :
to as a sense of 'being there' in a virtual environment and more broadly defined as tion biases the le.s :¿.
an illusion of nonmediation in which users of any technology overlook or misconstrue consider that nsi¡:
the technology's role in their experience' (ISPR zooo). This is not necessarily so. I will vision is usuallr- m¡:
consider that the illusion of being there is the label assigned to a group of illusions of helpful in broader:.::
position and movement (proprioceptive illusions) in the artificial environment, and In addition, the : i'
of identification of parts of the body with artificial representations. Now, the experi- seems to be possrb,e
ence of localperceptual illusions like these (rather than of a massive illusion of reality, a subject can feei ¿ r'-
or cognitive illusion) is compatible with the knowledge of the mediated nature of is known under th:
the experience. Perceptual illusions present, in fact, a special feature described as 1937; Botvinickan,j i
MULTIMODAL, INTERACTIVE MEDIA 6u
.i ':te the
'j'i 'resilience to knowledge': they persist in spite of the fact that the subject of perception
r1;,, I .-rl thus knows that what he or she is experiencing is an illusion.
l1 :,;¡tis thal Let us go backto one of our examples. SecondLifeusers'enter'the virtual world after
ldü¿r :-- . ¡ser rvill having created an avatar of themselves, a virtual representation of their self. They see
their avatar flying, assisting virtual expositions of art and performances, creating
[iu :l-,:::ports in objects, manipulating them. The illusion of being there in the ürtual environment is
n ff : .: -:icial set then equivalent to the illusion of moving when the avatar is seen moving, and to the
¡: . f:.E¡n et al. illusion that, to some extent, the hands of the user are there where the hands of
rl¡r
*,::i: be'how the avatar are manipulating virtual objects. We have' then, three main components in
rr' :', ie the con- the 'illusion ofbeing there': the first component is a general visual (or haptic, or acous-
niC:a -- --"'a SYStemS tic) bias upon proprioception. The second component is an experience of spatial iden-
ürl .: -,rr ¿CtiOnS in tificationwithsome artificialentity, associatedwith an out-of-bodyexperience induced
by vision (a proprioceptive illusion biased by vision). The third component is an illu-
li :,--::. mOfeOVeI, sion of movement. In fact, users are often not allowed to move around while exploring
t i:-",¡ museum. the virtual world (they can do it in mixed reality like LifePlus or in immersive systems),
r...;: resembles and, for sure, they cannot fly. This component includes specific illusions of movement.
"*::1,. tor virtual As an example of visual bias, in the previous section I have illustrated a propriocep-
,+.s ln this case, tive illusion induced by vision, in which the subject feels as if the hand of someone else
: :,:::acter's per- was their own (Nielsen 196¡). There are many other examples of visual bias over other
-.-i¡i character's perceptual modalities (Hay et al. Rock and Vi ctor 1964; Welch and Warren r98r;
196¡,;
:, -rsocialrather Mon-William s el al. ry97;Holmes zoo4) . More recently, some experiments on out-of-
r*-- lose much of body experience have used virtual reality in order to present experimental subjects
,:::- interactions with discrepant visual and proprioceptive information. These studies have indepen-
dently confi.rmed the possibility, given certain conditions, of visual bias upon proprio-
ception when the two are discrepant even for the whole body, and not only for limited
puits of it like the hand (Ehrsson 2oo7; Lenggenhager et al. zooT; }./rtllet zooT).
proprioceptive illusions so produced are currently exploited for the treatment of pathol-
ogies such as phantom limbs, stroke, hemiplegy, and spatial neglect-the success
,:': i,eittg there in with pathologies other than phantom limbs is, however, still uncertain (Ramachandran
.¿:r users experi- et al. 1995; Altschuler et al. t999; Sathian et al. zooo; Holmes zoo4). Vision has long
iLrl¿1 nature and been considered the dominant perceptual modality in case of mismatch between infor-
Lnuai reality, the mation presented to the visual perceptual modality and information presented to the
.r"rnded into the auditory, proprioceptive, or haptic modalities. The model of visual dominance has
rrch iISPR) sup- been contested, and evidence shows that all the sensory modalities contribute differ-
rmon11'referred ently to the aspect of the final percept; however, the more precise, less variant informa-
t¡adh- defined as tion biases the less reliant one (Ernst and Buelthoff zoo4). Nevertheless, it is justified to
i or misconstrue consider that vision normally dominates other modalities in spatial tasks, because
:essarily so. I will vision is usually more precise and reliable for spatial location. So, vision can be very
up of illusions of helpful in broadening the limits of non-immersive virtual reality systems.
lnironment, and In addition, the identification of a part ofthe body or selfwith artefacts (like avatars)
\orr., the experi- seems to be possible, in certain conditions. In experimental conditions, for instance'
lusion ofreality, a subject can feel a rubber hand placed in face of them as their oum hand. The illusion
:diated nature of is known under the name of 'fake hand illusion' or 'rubber hand illusion' (Tastevin
ure described 1937; Botvinickand Cohen t998; Pavani et al. zooo; Austen et al. zoor): a subjectwatches
as
6IZ ELENA PASQUINELLJ
a rubber hand; the hand is stroked slmchronously with their own hidden hand highlight the fact th¿t
(positioned congruously with the rubber hand). Subsequently, subjects undergo an visual position of the i
illusion of identification and an illusion of dislocation: they identify their hands with of lowered speed and
the rubber hand and feel their hands to be 'there', where the rubber hand is. It is the target to be reache
then possible to quantifz the illusion by measuring how close the felt position is to This fact confirms
the real position of the rubber hand. It is worth noting that this illusion depends on successful behar-iou:
both visual and tactile stimulation of the real hand. A variance of the rubber hand virtual reality conter
illusion can, in fact, be obtained without vision-with tactile and proprioceptive positive effects, and ''n
modalities (Ehrsson et al. zoo5). In order to produce the same phenomenon with a For this reason, the :l
digital rather than a rubber hand it would then be necessary to dispose of a multimodal considered as a unile:
virtual reality set that includes tactile or haptic devices. Another important aspect of
the illusion is the level of 'realism' or fidelity that is required in order to generate the
illusion. Can we undergo this kind of illusion with avatars that look like animals
(as many Second Life avatars) or like Woogles? It is controversial whether the 'sense of
ownership'and the displacement of the sensed position of the hand can be obtained B¡rr r
with objects that do not resemble the human body-as suggested by Armel and
Ramachandran (zoo3) but invalidatedbyTsakiris and Haggard (zoo5). Evidence from
the incorporation of tools seems to confirm that realism is not a necessary condition We have seen that ú
for the extension of the peripersonal space to hand-held tools (Holmes et al. zoo4). local illusions oi m¡i
Experiments in mediated conditions are required in order to validate the hlpothesis museums, or artistla -
that intermodal bias upon proprioception occurs during perceptual experiences with phenomena are fa¡::'
virtual agents and objects, and that users locate their body or parts of it in the place as illusion of non-mo
where their avatar is. For the time being, IJsselsteijn et al. (zoo6) have reproduced ated) with focalizec ¡
rubber-hand experiments in virtual (projected rubber hand and stimulus) and mixed to the perceptuai ei;
reality (projected rubber hand and real stimulus) conditions, showing the presence behaviour of inh¿br:,
of a weaker illusion in the two sprthetic conditions than in the non-mediated one. fidelity to a certain r,
Different factors can explain the difference. In all the 'natural' cases of visual bias if the second can b¿
upon proprioception, for instance, active movement and the activation of another attention, and subt¿;'
perceptual modality (touch or audition) seem to be necessary conditions for the illu- experience of r-irc;.-'
sion to take place. The induction of proprioceptive illusions could hence depend on measured and enn¿i
multimodalify and interactivity. of virtual forms .-.f ¡:.
The role of multimodality in the enhancement of proprioceptive illusions is con- Subjective hdel-::, :
firmed bythe third component of the illusion of 'being there': illusions of self-motion training applicati r'::.
or vection. Vection illusions are illusions of movement that can be induced by vision, surgery> but alst¡ -:r ;
but also by audition and are enhanced by the combination of visual and auditory tions of existinq t,¡,¡;
cues and of visual cues with vibration. Top-down processes seem to enhance the illu- also multisensori¿L:;
sion as well: when the source of the auditory stimulation can be consistently inter- seem to be r.en- er-e;
preted with self-movement, chances of undergoing illusions of vection produced are We have said ih¿:
increased (Riecke et al. zoo5). The role of multimodality in the production of illusions awaken and const::-:
of position and movement confirms the idea that ürtual reality systems are the best ination can be i,ien:,1
candidates for producing effects of presence and movement in the synthetic world are capable of ex;:r:::
(Lombard and Ditton 1997; witmer and Singer 1998). virtual reality systems are, in can be inhabite,l :'" 3
fact, normally multimodal and haptic, and tactile and vibratory devices are increas- thus not the same .¡
ingly added to visual and auditory displays. on the other hand, Holmes et al. (zoo4) fictional world res¿:
MULTIMODAL, INTERACTIVE MEDIA 6t3
highlight the fact that the presence of discrepancies between proprioception and the
iil: Lr:i:
-* ¡n visual position of the hand have disruptive effects on reaching tasks (measured in terms
',iii-l I rlih of lowered speed and of accuracy, that is, of the distance between the finger tip and
the target to be reached).
This fact confirms a general consideration about the importance of coherence for
successful behaviour (Stein and Meredith ry93), and is pragmatically significant for
virtual reality contexts: the bringing about of illusions does necessarily not have
lÍ1llill' I' i:::--1'e positive effects, and while it enhances the sense of presence it can disrupt performance.
For this reason, the production of illusions of presence and transportation cannot be
considered as a universal desideratum for virtual reality applications.
' .:li.t Ot
Purrrn'1,'
U" 1 j: -. ::::: lhe
üÍhiril .l': ::inals
'.,i.: Sinse Oi
llllltr,:*
in exciting and constraining imagination towards the imaginings that are appropriate coins, sounds of cru¡
to that particular representation. Believability can thus be defined as the condition that appear believable rn-ho
enables users, audiences, and readers to imagine the specific world which is repre- crunched apples and
sented by the work of fiction, and of running the relevant beliefs offline. For these a world of pebbles ih
reasons, even abstract representations can be considered as believable (Walton zoor). pebbles that crunch t-r
If the representation is supposed to guide audiences in imagining colours, shapes, do not correspond:..
sounds, and bodily sensations in a certain way-and audiences do this-then the rep- The examples Lri :
resentation is believable as the representation of a perceptual world where colours, Woogles, of virtual m
shapes, sounds, and bodily sensations occur in a certain way. invent in Second Li.f¿-
But it is possible that a work of art fails in exciting the relevant beliefs and imagin- derful tool for eralor
ings that belong to the world that the work of art intended to represent. The work of art perceptual modalities
is not believable, then. What are the conditions that enhance or threaten believability? of unbelievabilin. ¿n
Believability seems to depend heavily on expectations. When expectations raised by alive in them throtiú
the experience are unfulfilled or coherence between the elements of the experience is
violated, one is not disposed to form beliefs, and judges the experience as unbelievable.
This is shown in particular by experiences of conflict, like perceptual conflicts between
perceptual modalities: prosthetic hands that look very realistic but feel hard and cold Rnrr,nnNcEs
can raise a sense of uncanniness that immediately lowers believability (Mori r97o). In
Altschuier EL, \\-isd¡,::
this case, believability concerns the belief that the hand is a real one. In the domain of
with a mirror. L,¡':-:¿
fiction, spectators and users can be asked to imagine that a certain fictional world is Armei KC and Ra¡n¿.':
inhabited by agents that fully look like real human beings, for example, as in recent ani- from the skin cori:
mation moües such as The Polar Express, or Final Fantasy. To achieve this, motion cap- z7o, 1499-5o6.
ture systems are employed to provide agents with realistic biological motor behaviours Austen EL, Soto-Far.,:
(obtained through captors that are placed at the level of the principal joints of real human influencing risuar-::
beings and that serve to represent the movement ofthe artificial agents). However, facial Bates J (1994). The f.-.
expressions are not as realistic as bodily movements. The result is again an uncanny School of Compu:e:
Bates L Loyall AB. a:
effect of unbelievability. Realism can, then, represent a risk for the believability of vir-
Symposium o¡i r'"::¿a'
tual agents, since it exposes the public to conflicts and to the frustration ofexpectations
Botvinick M anci Cc,it
(Garau et al. zoo3). If this is proved, a recipe for obtaining believable agents could be: Neuroscience,3gt. -;
lower the user's expectations, display odd animals and quirky aliens rather than simu- Carroll N (r99o . Ti..¿ :
lations of human beings (follow the example of the successful virtual pets called Carcozzino lvl, Fris.-'-i
Tamagotchi),and eventuallyprovide broad, if shallow, characters with a specific person- Museum of Pu¡e : ¡r
ality, tics, social and goal-directed behaviours, and reactiüty to the environment (Bates Clair R (1985). The :n .
pp. 92-6. Columbr¡
1994 Reillyr996;Papagiannakis et al. zoo5).
Coleridge W\\- trEr- , :
In any case, an important consequence of the dependence of believability on the
Currie G (r99 j ). 1¡,r,;¡: .
fulfilment of expectations is that believability is enhanced rather than bothered by the Press, Cambriciue.
awareness of mediation and virtuality. In fact, the awareness that virtual agents and Currie G (zoo5 r. In:;:''
objects are not real lowers the users' expectations and makes their fulfilment easier, comp anio n ro -{¿-.:i:.-
thus reducing the risk of a drop in believability. Dennett DC (rgqr . -"''
Naturally, artistic applications can choose to play with the frustration of the expec- Dinh HQ, \\¡alker \. i
tations they raise. The Pebble Box is a multimodal mixed object composed of a real box tance of multi-ser,.:,
-,-:'
Proceedings IEEE
filled with pebbles and of a library of sounds that are activated when the users manipu-
Doyle P (zooz). Beiier a:
late the pebbles and stop when haptic exploration ends (Essl et al. zoo5). The sound
gent characters. P*,-;
Iibrary contains synthetic sounds of pebbles, sounds of pebbles in water, sounds of Systems (Ad\11-; . :
MULTIMODAL, INTERACTIVE MEDIA 615
,{ are appropriate coins, sounds of crunching apples, and sounds of birds. While the first three sounds
he condition that appear believable when associated with the haptic sensation produced by the pebbles,
I irhich is repre- crunched apples and birds appear weird and the proposed world is unbelievable as
rttline. For these a world of pebbles that crunch or chirp. Users do not imagine themselves touching
.e L\\talton zoor). pebbles that crunch or chirp, but rather as touching pebbles and hearing sounds that
colours, shapes, do not correspond to how pebble s should sowd.
is-then the rep- The examples of the Pebble Box and LifePlus, of virtual dramas featuring the
,j. rr'here colours, Woogles, of virtual museums of Pure Form, and of online fictions like those one can
invent in Second Life, show that very different virtual reality devices represent a won-
lieti and imagin- derful tool for exploring how far artists can go in manipulating stimuli for different
:. Theworkofart perceptual modalities, for measuring the limits ofbelievability and exploring the effects
.:en believability? of unbelievability, and for creating worlds never experienced to make imagination
iations raised by alive in them through the senses.
ilre experience is
g ¿s unbelievable.
;ontlicts between
¡el hard and cold RnrnnnNcrs
.,-If ori r97o). In
Ntschuler EL, Wisdom SB, Stone L, et al. (rggq). Rehabilitation of hemiparesis after stroke
nn the domain of
with a mirror, Lancet, 353, 2035-2036.
hctional world is
Armel KC and Ramachandran VS (zoo3). Projecting sensations to external objects: evidence
:. as in recent ani- from the skin conductance response. Proceedings ofthe Royal Society ofLondon: Biological,
this, motion cap- 27o, r499-jo6.
xotor behaüours Austen EL, Soto-Faraco S, Pinel JPJ, and Kingstone AF (zoor). Virtual body effect Factors
¡rts of reaihuman influencing visual-tactile integration. Abstracts of the Psychonomic Society, 6, z.
. Horvever, facial Bates I (rqq+). The Role of Emotion in Believable Agents, Cmu-Cs-94-136, Pittsburgh, PA,
School of Computer Science, Carnegie Mellon University.
gain an uncanny
Bates J, Loyall AB, and Reilly WS (rgqr). Broad Agents. Proceedings of the AAAI Spring
:lier-ability of vir-
Symp o sium on lntegr ated Intelligent Ar chitectures, Stanford University'
,n of expectations
BoWinick M and Cohen J (rqq8). Rubber hands 'feel' touch that eyes see. Nature Reviews
r agents could be: Neuroscience, 3gr, 7 56.
:ather than simu- Carroll N (rgqo). Thephitosophy of horror. Or, paradoxes of theheart. Routledge, NewYork.
Lrnral pets called carrozzlno M, Frisoli A, Rossi F, Tecchia F, Ruffaldi E, and Bergamasco M (zoo5). The
a specific person- Museum of Pure Form. Proceedings of MIDECH Workshop. Milano, Italy.
rironment (Bates Clair R (1985). The art of sound. In E Weis and I Belton, eds. FiIm sound. Theory and practice,
pp.92-6. Columbia University Press, New York.
Coleridge \¡WV (t8lZ). Biographia Literaria. Rest Fenner, London'
liei'ability on the
Currie G 999).Image and mind: FiIm, philosophy, and cognitive science. Cambridge University
:L bothered by the Press, Cambridge.
1-rrual agents and Currie G (zoo5). Imagination and make-believe. In B Gaut and DM Lopes, eds. The Routledge
rulfilment easier, companion to Aesthetics,pp.335-46. Routledge, New York.
Dennett DC (lqqr). Consciousness explained. Little, Brown & Company, Boston' MA.
:ion of the expec- Dinh HQ, Walker N, Song C, Kobayashi A, and Hodges LF (rqqq). Evaluating the impor-
osed ofa real box tance of multi-sensory input on memory and the sense of presence in virtual environments.
Proceedings IEEE Virtual Reality, zzz-8.
he users manipu-
Doyle P (zooz). Believability through context: Using 'knowledge in the world' to create intelii-
loo;). The sound gent character s. Proceedings of the lnternational Joint Conference on Agents and Multi-Agent
lrater, sounds of Systems (AAMAS), pp. 342-9.Bologna, Italy, Iuly zooz.
6t6 ELENA PASQUINELLI
s.Science,3t7,to48. O'Regan K and Noé A (zoor). A sensorimotor account of vision and visual consciousness.
and: feeling ofbody Behavioral and Brain Sciences, 24, 939-rort.
na,l of Neuroscience, Papagiannakis G, Ponder M, Molet M, Kshirsagar S, et al. (zooz). LIFEPLUS: Revival of life in
ancient Pompeii. 8,h International Conference on Virtual Systems and Multimedia' VSMM
;rercept. Trends in 2oo2. Gyeongju, Korea, September zooz.
Papagiannakis G, Schertenleib S, O'Kennedy B, et al. (zoo5). Mixing virtual and real scenes in
ron of performance, the site of ancient Pompeii. Computer Animation and Virtual Worlds, r, tt-24.
':6,
:. :nd International pavani F, Spence C, and Driver | (zooo). Visual capture oftouch: out-of-the-body experiences
roo5. with rubber gloves. Psychological Science, r,353-9.
1. haptic toolkit for Putnam H (rq8z). Reason, truth and history. Cambridge University Press, Cambridge.
t C\[ Symposium on Radford C Ogl). How can we be moved by the fate of Anna Karenina? Proceedings of the
:'-. Crprus, zoo6. Aristotelian Society, 49, 67-80.
l:l perceived quality Ramachandran VS, Rogers-Ramachandran D, and Cobb S (lggl). Touching the Phantom
:ented at CHi zoo3: límb. N atur e Review s N e ur o s ci en ce, 377, 489-9 o.
1g s\'stems. Reilly WS (1996). Belieuable social and emotional agents. (CMU-CS-96-138) School of Computer
:,n of curved lines. Science, Carnegie Mellon University, Pittsburgh' PA.
Riccio GE (1995). Coordination of postural control and vehicular control: Implications for
:-acles. Psychonomic multimodal perception and simulation of self-motion. In JM Hancock, PA Flach, J Caird'
and K Vicente , eds. Local applications of the ecological approach to human-machine systems,
L{PH, Los Angeles, pp. rzz-8r. Lawrence Erlbaum Associates, Hillsdale' NJ.
Riecke BE, Schulte-Pelkum L Caniard F, and Bulthoff HH (zoof). Spatialized auditory cues
;re' ofarm position enhance the visually-induced self-motion illusion (circular vection) in Virtual Reality. IEEE
:jtience, 4,]l93-2oo. Virtual Reality,4t-8.
nee before me? The Riva G, Davide F, and Ijsselsteijn WA, eds zoo3. Being there: Concepts, fficts and measurements
t: Teleoperators and of user present in synthetic environments.Ios Press, Amsterdam'
Ro.i I u.rá Victor J (1964). Vision and touch; an experimentaliy created conflict between the
: presence: explica- senses. Sci¿nce, r43, 594-6.
Sathian K, Greenspan AI, and Wolf SL (zooo). Doing it with mirrors: A case study of a novel
:rsensory organisa- approach to neurorehabilitation. Neurorehabilitation (y Neural Repair, t4, n-6.
:hlsics. Journal of Schaefer M (zoo6). Fooling your feelings: artificially induced referred sensations are linked
to
a modulation of the primary somatosensory corlex. Neuroimage,29,67-73'
s-rm: manipulating Slater M and Steed A (zooo). A virtual presence counter. Presence: Teleoperators and Virtual
Env ir onm ents, 9, 473-3 4.
a's tbunding myth. Steele R and Black (rSSS). Media ethics codes and beyond. American Society of Newspapers
I
editors. http://usinfo.stategov/journ alsI itlcl o 4orl tjge/gjog.htm
Stein BE and Meredith ME (rqq¡). The merging of the senses. MIT Press, Cambridge' MA.
¡resence.
Journal of
Stoffregen TA (1997). Filming the world. An essay review of Anderson's The Reality of Illusion.
--CS-97-u3) School Ecological Psychology, g' 761-7.
llellon University, Stoffregen TA and Birdy B (zoot). On specification and the senses. B¿havioral and Brain
Sciences,24, l'95-26r.
. tllto-1. Stoffregen TA, Bardy B, Smart LJ, and Pagulayan RJ (zoo¡). On the nature and evaluation
:esia in the normal of fiáehty in virtual environments. In LJ Hettinger and MW Haas, eds- Virtual Ú adap-
tiye environments: Applications, implications, and human performance issues, pp. ttt-28.
Lawrence Erlbaum Associates, Mahwah' NI.
ri¡rual ball bounc- Tastevin J OSI). En partant de l'expérience d'Aristote. Encéphale,32,57-84'
,i;l o.f Neuroscience Thomas F and Johnston T (1984). Disney: The illusion of life. Abbevrlle, New York.
Tsakiris M and Haggard P (zoo5). The rubber hand illusion revisited: visuotactile integration
'.^mal of Psychology, and self-attrib ution. lournal of Experimental Psychology: Human Perception and Performance,
3r, 80-91.
618 ELENA PASQUINELLI
Worringer ::,.
Academy of Bactír i,;':.;:':,
Veneziano -¡
accessibiliqv of art. c..,-:-: -.
14 15, 136-;
Acconci, Vitoro
acetone, smell on b:t..--. -
relationship to lc.-: .-
aesthetic sensibili¡' :: :
aesthetic touch io--: " :
7f¿sentational arts.
srsensory discord-
INo Bx
ulents: a Presence
+'t5," John Cage z6t, 276, 306, 577 bodily aspects rrr-rz
zoor, Stanley Kubrick 286 as an emotional trigger n3
engagement of the imagination r r4
Ar area of brain, multisensory integration 27o*l openness to individual interpretation tto
Aalto, Alvar, Villa Mairea 585,586 palpability of space 712*73
Abramovic, Marina ro unconscious aspects 113-14
absolute music 253-4, z56,z6o see also totch
absolute pitch, prevaience, differences between affect 247-8,332
blind and sighted musicians 269-70 autonomy of 249
abstract art as a social experience 248
activation ofmirror neuron system 460 affective responses 332
relationship to s)¡naesthesia 49o adaptivevalue 34o
abstract expressionism rz7 functions ¡35
abstraction tz7, 169 to music a35-7
Ab str acti o n an d emP athy, W rlhelm predictions for costly signalling
Worringer rz3 hlpothesis 337,34c'*7
Academy of Baccio Bandinelli, Tfu¿, Agostino predictions for emotion hlpothesis 3j7,
Yeneziano 76 338-4o
accessibility of art, conflict with preservation predictions for language hlpothesis
74-75,86-7 337' 342
Acconci, Vito ro predictions for social bonding
acetone, smell on breath r99 hlpothesis 332,342
action fidelity 6ro temporal entrainment as source 342
Action Painting rz7 affordance effects 363
action possibilities t4o affordances, touchability r4o-r
action potentials, early studies 3z afterimages, Wells' study 45-6
adaptive value of music a34-5 Aftertrace, Georgia Chatzivasileiadi r5-ió
Adoriction Clock, Gregory Cowley zzz Agon, George Balanchine 5zz
Adriá, Ferran zo8 Agosti, Giovannr 4tz-t3
Adrian, Edgar Douglas 3z Alberti, Leon Battista 365, 383, 384
,lEgineta, Paulus, on vertigo z5 attitude to sculPture 63
aesthetic attitude 534-5 De Statuta 44-4
aesthetic experience 53o on primacy of vision 6r
multisensory 544-5 on Plthagoras 58r
sensory, cognitive and hedonic on the task of a painter u7
components 531-2 album/ cover art 285
anatomical studies ofthe senses 28-31 'Art and the Senses'workshop, Oxford balanced compositio;.r .¡,
Anaxagoras zr (zoo6) v,t,t77 proportion 55o-r
'animal electricity', Galvani's studies 3r conference dinner zo8 symmetry 552-_1
animals, recognition in naturaily occurring food perception, effect of environmental Balanchine, George,i-y
patterns Z6S-6,268 sounds zzz-6 Agon 5zz
Animated Puppets 603 Art as Experienc¿, John Dewey 594 Bandinelli, Baccio -6--
animation, portrayal of motion 383-4 artificial life (animation form) 383 Bárány, Robert 43
anosmia 78s, zo7-2 artists, interaction with scientists 537-8 perceptual porrra:i ::
artist's atelier, The, Louis Daguerre r37 studies of vestibul¿:':-
specific r98
art music, development 253-7 'barbaric art', Willhei.r',1'
anti-rationalism 245
Antoninus, on primacy of vision 6r Asplund, Barker, ]ennifer If .. i:; - .
585
Atalanta and Hippomenes, after Guido Reni, Bar, Moshe et a1. ¡6.i
Antonioni, Michelangelo, Blow-Up 4t
Vik Muniz .¡.lz Barthes, Roland :;r
aphasia, Melodic Intonation Therapy z8o
AtaXia, W ayne McGregor basic emotions 3,1
Apollo and Daphne, Gian Lorenzo Bernini 8o 537
attention basic pitch space -11-
appetite, relationship to smell r92,2o2 519-21
Apple iMac 562 and illusion of non-mediation 6o8 bas-relief 4oz-5
Aquinas, St Thomas psychologicalstudies 5zr Donatello 4o9-;;
on sensus communis t35 attraction baton, use by condu::--:.
317
Tonal Pitch Space (TPS) model 3r5,3r7 Baudelaire, Charles ?r:¡::
on touch 62
Ara Pacis Augustae f elief, Rome 4o5 in Chopin's E major Prelude jtg-zo ofthe senses 15,
architecturai language 581-2, 58i audiences for dance performances 512 Baumgarten, Alerni.¡ , ,
audio-grapheme synaesthesia r66 Bausch, Pina 519
architecture 575-ó
age ofaestheticization 586-8 audio-o1factory synaesthesia r66-7 Baxandall, Michael -:
audiovisual congruence 266 Bavard, Hippolite. ¡,:r.-,:i-
as art ofspace 57r-z
.
The,Byatt, A.S. vi, vii-üii see also dance coloured hearing :;.-1
camera obscura 384 chronophotogr aphy 37 9, j 8o Alexander Sc:::'r--- . :
colour harpsichord (clat,ecin oculaire) 496-8 illusion of non-mediation 6o7 common themes :-,.
Castiglione, Baldasarre, Book ofthe Courtier 76 incongruent dubbing 266 communicabilitr Lr: r:: ::
Cattrell, Annie,'Seeing' and'Hearing' ix, x music 285-7 sme11 u48-9
Cavaceppi, Bartolomeo, atelier of 425-6 narrative devices 438-9 communication. r¡l¿ ' .:
rNDEX 623
crispness ratings of food portrayal in artworks, perception of De Statuta, Leon Bai:i':: '
relationship to auditory feedback zú, zt9 motion 38r-3 Dewey, lohn, Art a: E::,*
reiationship to sounds offood relevance ofmirror neurons 524 Diderot, D. u9
packaging zzo-r salient stimuli 5r9-zo digitalization, s€nSo.r':-r: l
Criüque of Judgement,Imrnanuel Kant 168 sources of inspiration 5zz diseases, smell of rt"!-t
cross-disciplinaúty z5t structure 521,522 disintegration of arnt¡:*
cross-modal piasticiry r63-4 title526
design r37
cross-modal transfer r35, 363 Dancer in the Dark (vonTrier), jump cuts 446 distance perception rr ,*
cruel body, Gilles Deleuze rz6-9 dancers, neuroimaging studies 524 elasticity 574
Crum Brown, Alexander 5o Darwin, Charles, on evolution of music a34 distraction, music as :'-.
on after effects of body r otafion 44, 47 Darwin, Erasmus 36 DITHERNOISE. Sino;-. L
perceptuai portrait 44 perceptual portrait 37 Dogon, audio--olfa¡¡:' ¡
on rotation sense in deaf-mutes 48 Darwin, Robert 45 dogs, detection of c¿r¡::
Cubi,DavidSmith n3 Dayid,Michelangelo dominance effect. n rl::,.¿
79
Cubism, pattem recognition 366 David, Jacques-Lotis, Death of Marat 387 s¿e r¡lso senson' ic ::-:::
cue redundancy 55o DaVinci Code, The, audio content 286-7 Dominus winen', \¿::.-
cultural differences Dead Chrk6 Andrea Mantegna 372 Meuron, 55;-5
in appreciation of smell t9z deaf-mutes, postrotational vertigo 48 Donaldson, Henn -.i
in attitudes towards preservation of De Architectura, Vitruvius 579 Donatello
artworks r5 Death of Caesar, |ean-Léon Gerome 387 Gattamelata +t-
in food and drink zo9 Death of Marat, Jacques-Louis David 387 Herod's Banqtie: :--.. -
:eption of De Statuta, Leon Battista Alb ertr 423-4 Early Modern period 59
Dewey, Iohn, Art as Experience 594 Easy Rider (ñ1m), single frame cuts 37r
1t t!4 Diderot, D. u9 Eau Claire, Clara Ursitti r93
digitalization, sensory impoverishment 12 Edgeoflntention 6o3
diseases, smell of 198-9 effrciency 576
disintegration of artworks, incorporation into 'Egyptian' way (Riegl) ru7-8
design r37 Einfühlungnotion, Robert Vischer 459
' . Jump cuts 446 distanceperception 4oo-r electricity, physiological studies 3r-z
:5 e\asti'city 574 Eleglz5 (sculpture) ur
51,+
: u.f music a34 distraction, music as zr5 Elgin Marbles +og
DITHERNOISE, Simon Longo 277 Elkins, James 385
Dogon, audio-olfactory synaesthesia 166-7 embodied event representation, cinema 448-9
dogs, detection ofcancer r99 embodied music 281-3
dominance effect, multisensory perception 163 embodied perception rz4, 243-4, 572-3
t ),Iarat see also sensory dominance double engagement with the world szl-+
387
Ir:ent 286-Z Dominus winery, Nappa Valley, Herzog, and de elasticity 574-5
¿ Meuron, 555-6 embodied simulation
-1:2
:ngo 48 Donaldson, Henry 38 in responses to art 458
) Donatello role ofmirror neuron system 456
rrome 382 Gattamelata 477 embodiment 3
; David Herod's Banquet 41o, 411 emergence effect, multisensory perception r63
387
Mercurio-Dattid 4zo emotional responses, triggering by coarse visual
Í--alio (Souyenir Miracle of the new-born child,bronze 4t5 information 367-8
sculptures of the Altare di San Lorenzo 4t7, Emotion and Meaning in Music, Leonard
,tr 418,419 Meyer 3rz,3r3
St. George and the Dragon, bas-relief 4o9-ro emotion hlpothesis, music a34
Doni, Anton Francesco 6Z-8 predicted affective responses fi7, T8-4o
Donne, John vi, vii-viii emotions 33t
Doritos project, Value Meal: Design and (oter) communication by music a43-4
eatingexhibition zzz responses to music zr5, 294-6, 312-73, 323-4'
¡:d riii dorsomedial prefrontal cortex (dmPFC), role in 329-30
rt Carnie vili narrative comprehension 447-8 triggering by touch rr3
Doxiadis, C.A. +oZ see also affective responses
Drake, C. and Ben El Heni, J. z8z empathic responses 459
\oise Apparatus drawing role of mirror neuron system 460
and Spray capture of tactility t +s-z Empedocles zz
case study ofE.A. (blind painter) q68-z6,4zo Encyclop aedia Britannica, on stereoscope 395-7
lcsrnia zoz Driscoll, Rosalyn r4z-4 endorphin release ix
Limen to\ energy, activation ofsense otgans 27
drumming entrainment, temporal 282, 297, 332-3, 525
communication of emotions 295-6 as source ofaffective resPonses 342
elements of a good solo 3o5 environment, effect on food and drink
improvisation 3o3-4 perception zz9-3o
influence of technology 3o7 environmental sounds, use in music 276, 278,306
s'"iesia 174-6 dubbing ofmovies, incongruent 266 epileps¡ seizure prevention by conditioned
Duchamp, Ma¡cel rr smells zoo-r
chocolate grinder zo9 equivalence signals 16z-3
Duet,PaulTaylor 5ry 'Equivalents', Vik Muniz 353
Dunn, Douglas 5,zo Ernsl, Max, as inspiration for jazz joo
tó1-_1
Lrn i57-óo Durand, J.N.C., architectural language 581-2, 583 Ernst, M.O. and Banks, M.S. 9t-2,9j,45-6
Dying Slave,Michelangelo 79 Esguard de Sant Andreu de Llavaneres, L',
restaurant 2o9
Early Modern ltaiy, tactility 59-6t, q-9 Eshkar, Shelley 5r5
Michelangelo 78-8o ethical dimension, multisensory experience 247-8
paragonedebate 69-73 'Euro-whiff r84
social status ofsculptors 74-7 evaluation of aesthetics 53o-r
as theme of sculpture 8o event boundaries ¿4o
626 TNDF,X
event models 47,8, 44g film see cinema frontal cortex, activa:ic:: :
event segmentation 439-42, 45o film music 285-7 473-4
and cinema cuts 442-5 Finn, David, photography ofsculpture 137-8 frontal lobe damage. u'-:,l:,
evolution fish, lood expectancy behaviour z:r frontotemporal demen-,r ¿.
aesthetics as a by-product ofadaptation 563-4 Fishinger, Oskar 5o5 abilities 475
selection of music a34-5 Fishkin, S.M. et al. 88 Fulcrum, Richard Serr ,;,
Execution of the Emperor M*ximilian, five senses 2o,589 functional magnetic re) -,:,
Manet, 382 historical accounts 2r-6 E.A. (b1ind painter 1:
existential outsideness 596 flat pictures, visual perception 36o-1 limitations 5r.1
expectation see clso pictorial cues responses to elenl bL:tr:
in music ar3-r4 flavour r9z responses to cinen: ;:'
ftr4Rl study of expectancy üolation 324 flavour perception 'Seeing'and'Hearir: :
relationship to physiological reactions 323 effect of music z16 Cattrell r-r. .¡
role in believability 614 influence of environmental sounds zz.z-6 study of musical ei'r.:--
experience perceptual gestalts zo8 fungiform papillae rS9
influence on picture perception 379,383 flavour principles 56r Fur Covered Crip, \f e:et -
influence on the senses 3 flight simulators, illusion of reality oo9 Futurism 169
exlra-opus knowledge, music a13 Floating Point, The, John Mclaughlin 3o7 Futurist Cookbook, F.I \i.
extrapersonalspace 134 Flourens,, study ofvestibular system 47
eye Focillon, Henri rzz, rz3 Galen
central and peripheral vision 364 Foeckler, Philip zzz study ofthe senses -:
convergence and divergence 386 foliate papillae r89 on vertigo 39r.r.c,
physiology 354-5, 360, 384-5 Fontana, Lucio r4 Galileo Galilei
perception ofdistance 4oo-r food and drink on paragone debz:e - -
eye movements, viewing paintings 385-Z auditory influences 228-3o pefcePtual portr¿r. :-:
on consumption rate 274-t7 on senses 22-+
Fabrique, Holland Dance Festival Posters 554 environmental sounds zzz-8 galvanic stimulation s:L.l
face perception 355, 365 food as source ofsound zr8-zo production of ver::: :
in natural objects 365-6 music, influence on selection zrz-r4 temperature sen>3 _::-
slrnmetry 552 cultural variation zo9 Galvani, Luigi -l
fake hand illusion 6u-12 familiarity 559 pefceptual port:ai: ---:
famiiiarity of obj ects 557-60 flavour principles 5ói Ganly, Helen ro. r-,"
familiarity with actions, relationship to mirror multisensoryperception 208-1o gap-fill melodv _:ri
neuron activation 458*9, 533-4 palatabiliry, role of dy,namic contrast 548-9 Gaston de Foix^ Gia:: G:¡
familiarity with music a44 selection, contextual effects zrt-r.z Gattamelata, Dona:eL¡ :
Fat Duck, f¿e, restaurant 2o9 sensory-specihcsatiety 56o Gauricus, PomponLu. : i
bacon and egg ice cream z.z3-4 temptation by 363 Gazzola,Y. et al. 4<:
'Sounds ofthe Sea'seafood dish zz7 food names zz8 Gebhart-Sayer, -¡.. i-,-:
the FEAR of smell-the smell o/FEAR, Sissel food packaging, sound of zzo-z gelbe Klang, der, \\':'::-" i
Tolaas t96-7 food preferences, relationship to nutritional Generative Theon c: - ::
fear, smell of 196 needs 559 Lerdahl, F. anc l::.
feature-based attention 389, 389-9o form Gerome, Jean-Leor. l,:-; :
Febvre, Lucien 582,584 musical 3q,3o4 gestalt 1aw of simira:::. ..
feelings s1'rnbolic idea of n8, n8-r9 gestalt perceptron < :
contemporary value over thinking 245 Forsy-the, William n8, 5t9, 5zz gestalt principles ¡r:
distinction from emotions 33r-z Fourier, Charles 244 in generation o¡ e:.-:. - .
in Vik Muniz's work 354 'Fracture Flickers' series zóó Ghiberti, Lorenzo ¡,.-: :
Fe1d, Steve 164 frame, role in definition of art r5 Ghos¿, Rachel \\-hit¿¡¿::
female sculptures, male handling of 77 Francis Bacon: The Logic of Sensation, G1lles Giambologna
Ferber, C. and Cabanac, M. z16 Deletze n6-7 Rape of a Sabin¿ ¡.-:
Ficino, Marsilio, on primacy of vision 6r Freedberg, David 458-6o tactility of statuei.¡. :
liction, iliusion ofreality óoo-r Freeman, W.I. ::S Gibson, James J. 8-. r: -
fidelity to reality, illusion of 6o8-ro frescoes, perspective 4t3-r8 Giorgione, Sf Geor¡r .:.:
fields ofvision 57r Freud, Sigmund, on sense of smeil z4t Giotto,, depiction o: ;:-:,-r
fight or flight response 331 Fried, Michael z6o glass dust, use br'\-¿:r.:::
ligure, Deleuze on D6-7 Frolov, Y.P. zzr Glass Pieces, Jerome i.-- r:
INDEX 627
frontal cortex, activation during drawing goal of actions, reiationship to mirror neuron
473-4 activation 459
r-ulpture 137-8 frontal lobe damage, utilization behaüour 363 Goehr,L. 253-4
frontotemporai dementia, effect on artistic Goethe
abilities 475 on interaction ofthe senses 592
Fulcrum, Richard Serra g9-4o on Lessing's Laocoon 43t-z
functional magnetic resonance imaging (ÍMRI) sculpture, Frankfurt arrport 43o, 432
E.A. (blind painfer) 469-7t, 477, 472-3 go1d, use byVenetian artists 375
r _ró{Fl limitations 5r4 golden section ratio 55o-l
responses lo event boundarjes 44o,441 Le Corbusier's use 585
responses to cinema cufs 443,444 Goldscheider, Alfred 38
'Seeing' and'Hearing' project, Annie Golgi, Camillo 3o
Cattrell ix, x perceptual portrait 3i
..¡unds zzz-6 study of musical expectancy violation 324 Gombrich, Emst 365,387
fungiform papillae r89 Gonzales-Torre s, Fel:rr', Portrait of
Fur Covered Cup,Meret Oppenheim zo8 Ross in L.A. 97
irn 609 Futurism 169 Graham, Martha, contraction 525-6
aueirlin 3o7 Futurist Cookbook, F.T. Marinetti zo8 Grammer, Karl et al. t96
iritem .+Z Grant, A.C. et aL. 97
Galen graphemes 5o8
study ofthe senses 22 Great Sphinx of Gis¿, Gerhard Richter
on vertigo 39-4o 398-4oo,4o1
Galileo Galilei Gregory, Richard r35,363
onparagonedebate 7o groove, musical z$,297
perceptual portrait z3 Guéguen, N. et al. zr4
on senses 22-4 Guest, S. etal. 89,92,94
---¡ galvanic stimulation studies 3r-z Guttuso, Renato, BOOGIEWOOGIE j8o
:r8-:.o production of vertigo 4z
it¡r 212-14 temperature sense 36-7 Hadden, Sktp 294,295
Galvani, Luigi 3r Halprin, Lawrence, Auditorium Forecourt
perceptual portrait 3z (now Ira Keller Fountain) 596
Ganly, Helen ro, r79 Hamer, Andrew xiii
]-10 gap-fill meiody 3r4 Hamilton-Patterson, James, Cooking with Fernet
.ontrast 548-9 Gaston de Foiv, Gian Girolamo Savoldo 7l Branca zz8
Gattamelata, Donatello 4r7 Handbook of Multisensory Processes, The, Calvett,
Parthenon, Jonic Frieze 405-6, 407-9 movements 274-5 Portrait ofPierantc":::, : ";--
invisibility 4r8-r9 Piazza d'Italia (Souvenir d'Italie), Giorgio De Pudica Stn¡r,;:¡. .--.::
Parthenon, Nashville 4o8 Chtico j7o Portrait of Ross in L..:. - F : :
particulate principle, music a33 Picasso, Pablo 14 positron emission t¡ir: : -::
Party Number Oíq Michael Petry 44 pictorial cues 359, 36r, 39o-r emotional res:\ -.->.i :
Pathosformel theory, Aby Warbúrg 459 depth perception 368-7t Posner, M.l. Nissen. lul.' -:
pattern recognition 354, 366-7 illumination 373-4 postcentral g).rus It
pauses in cinema 445-6 motion 377-9 pregnancy, chanses r¡- .:-''.
Pavlov, Ivan zzr biological 379-84 pre-Modern era, ani :,: _:-'
peak-shift effect 576-17, 557-8 object contours 36r-4 premotor systent. n]ü-:l
Pebble Box 6t4-ts objectrecognition 364-8 preservation of artrr..¡.¡. ;
Peers, Heather ro, r79 reflection 375-7 accessibilin- 1+-r: -,1
Peirce, C.S. 243 transparency 374-5 priming of audiences .-.---
perception, rationaiization of xi picture perception, influence of experience Prinz, Jon zr9
perception-action cycle 385 379,383 prior entry, larv of :ó;
perceptual consistency 373 Pictures of Magazines, Vik Muniz 355-6 problem of shape, Ilr;. -¡,:: .
Ritter, Johann Wilhelm ¡6-z Scriabin, Alexander 'sentence form', mu.r; -.',;
on vertigo 4z coloured hearing 5ot-3 Sentinels, Carole -\n,i¡.'" ¡
ritual, in classical concerts 254 colour-key correspondences 5o3 separable componenr. ,:: :
Rizzolatti, G. and Craighero, L. 533 Prometheus, Poem of Fire 5oz Sepulchre of the Hatr:: : :,
Robbins, Jerome, Glass Pieces 5r.9 sculptor-painters 4r3
404-5
rNDEX 6Zl
tension graphs, Chopin, E major Prelude 3zt, jzz classical accounts 21 Untitled, Clara U:sr:rr : :
tension , musical s¿¿ musical tension consequences ofblindness 97, 98 urban music 3oó
test-tone records, use bylohn Cage z6t c¡eative and exploratory r4t-7 Ursitti, Clara r9-r
theatre architecture, Vitruvius 577 division into separate qualities 36 Untitled zo:-_;
theatricaliry, in musical performance 2.58*óo dominance by vision 86-8, 9o, too utilization behar-iou:,:.
Theophrastus, on vertigo z5 dominance over vision 25, 89-9o, too
three-dimensional effects 397-8 and Eariy Mode¡n Italian art 5g-6t,63-9 valence 332
Ara Pacis Augustae relief and Jonic Frieze Galileo's account 22-3 van Gogh, \¡incent. :n::::,
405-9 involvement of movement ro9 Vanitas: Flesh dre :: .t':- :".::
bas-relief 4oz*5 link with sculpture rzo-r Sterbak:o9
distance perception 4oo-r metaphors for mental processes 6z Vanitas Still Liie ú- '. i
'.\".
.