MATRIX

REDONE
SYNTHETIC FOR THE
ENVIRONMENTS MODIFICATION OF
THE BODILY SELF-
CONSCIOUSNESS
AND IMPLICIT
MOTOR TRAINING

Proposal by:
Prof. Giuseppe Riva, PhD

http://www.giusepperiva.com
 MATRIX REDONE
CONCEPT SUMMARY
In the science-fiction thriller The Matrix, the heroes "plugged in" to a new generation virtual world for
training. While their bodies rested in reclining chairs, they learned how to fight martial-arts battles,
drive motorcycles and helicopters in a virtual world.

Up to now different
methods and
technologies are used
to simulate the
"external" world: from Click here to see the original Matrix scene (http://www.youtube.com/watch?v=6AOpomu9V6Q)
mental images to
Differently from typical virtual environments, that simulates reality, the Matrix environment simulates
virtual reality (VR). bodily self consciousness: a subject can acquire a complex behavioral skill – mastering martial arts
or driving a plane – by experiencing through a somatic simulation (passive interaction) the behavioral
This proposal aims at a processes related to the skill.
radical shift to this
approach: Main objective of this proposal is to replicate this approach without the need of “jacking in” the brain:
its goal is the design its goal is - through the understanding of the cognitive processes related to it - the design and
and development of development of a tool able to modify and simulate our most "internal" experience: bodily self
consciousness.
a method and/or a
tool able to modify To reach this goal the project will integrate:
and simulate our most
• robotic and haptic devices (to provide sensorimotor information);
"internal" experience:
bodily self • virtual and augmented realities (to provide visual information);
consciousness. with the outcomes of a recent branch of cognitive science: embodied cognition and its analysis of the
link between representations, action and perception.

The project aims, in the short/medium term, to understand the cognitive processes involved in the
emergence of bodily self consciousness (1-3 years). The data and the experience gained could allow,
in the long term, both the simulation of the experience of a given body (3-5 years) - e.g., an obese
subject could experience a thin body to increase his/her motivation to change – and the development
of Matrix-like motor training tools (5-10 years). Specifically, the goals of the project are:

• Investigating both the characteristics of somatic awareness and the features of actual tool and
experiences that can influence these characteristics.

• Developing an embodiment procedure or device able to modify the actual body experience of the
subject. Simulation and haptic tools will be used to produce a controlled sensory rearrangement
able to simulate, even a distance, a different passive and active body.

• Exploring the possible use of the embodiment procedure/device for real world applications. In
particular three possible areas will be investigated: health care, training, and entertainment.

• Evaluating the efficacy of the embodiment device in user trials.

 RATIONALE OF THE
PROPOSAL
THE EXPERIENCE OF THE BODY
Even if the body is usually considered as the most personal part of our perceptive experience,
our perception of the body is not direct. In fact, evidence from studies of brain-damaged
individuals suggests our bodily experience is mediated by a cognitive representation of the body,
not used for representing other objects, that is spatially organized, supramodal, and used for
representing other bodies as well as one's own.

Evidence that this body representation can exist independently of sensory input is found with the
phantom limb phenomenon: it is the subjective sensation, not arising from an external stimulus,
that an amputated limb is still present. After amputation, many subjects still experience
sensations of movement, position, and pain, as if the limb still existed. Because there are no
sensory receptors, any kinesthetic sensations that an amputee may associate with voluntary
attempts to flex or extend a phantom joint must arise internally from some form of bodily
representation.

SENSORY REARRANGEMENT: WE CAN USE TECHNOLOGIES TO MODIFY THE

EXPERIENCE OF THE BODY
The adaptive capabilities of the human experience of the body are well known. As underlined by
Lackner & DiZio (2005), the control and adaptation of body orientation and motion are the result
of multiple sensory and motor mechanisms ranging from relatively simple, peripheral
mechanisms to complex ones involving the highest levels of cognitive function and sensory-
motor integration.

For instance, if you watch an experimenter stroking a shoe or a table surface while he
simultaneously, in perfect synchrony, strikes and taps your knee hidden from you under the table,
you will experience the touch sensations as emerging from the shoe or table. If the experimenter
then hits the shoe or table with a hammer, you will register a strong galvanic skin response, as
though the object was now part of your body. But if the shoe (or table) and your hand are
stimulated out of synchrony, no illusion occurs and no galvanic skin response is seen (so the
response is not just due to startle).

It is well known that in immersive virtual reality (VR) systems the human operator's normal
sensorimotor loops are altered by the presence of distortions, time delays and noise. Such
alterations that are introduced unintentionally and usually degrade performance, affect body
perceptions, too. The somesthetic system has a proprioceptive subsystem that senses the
body's internal state, such the position of limbs and joints and the tension of the muscles and
tendons. Mismatches between the signals from the proprioceptive system and the external
signals of a virtual environment alter body perceptions and can cause discomfort or simulator
sickness.

A BASIC PROTOTYPE: SCIENCE ARTICLES (24 August 2007)

As underlined by different papers published in the past (Biocca and Rolland, 1998; Castiello et al.,
2004; Riva, 1998; Riva et al., 2001), it is possible to use virtual reality to induce a sensory
rearrangement able to modify the bodily experience.

Two teams of cognitive neuroscientists independently reported on Science Journal (24 august 2007)
methods for inducing elements of an out-of-body experience in healthy volunteers.

• The Experimental Induction of Out-of-Body Experiences, H. Henrik Ehrsson (24 August 2007),
Science 317 (5841), 1048. [DOI: 10.1126/science.1142175];

• Video Ergo Sum: Manipulating Bodily Self-Consciousness, Bigna Lenggenhager, Tej Tadi, Thomas
Metzinger, and Olaf Blanke (24 August 2007), Science 317 (5841), 1096. [DOI: 10.1126/science.
1143439].

Both groups used head-mounted video displays to give people a different perspective on their own
bodies. Each team also drew upon the sense of touch to enhance the illusion.
It is also well known that key biases can distort perception of the location and orientation of objects and surfaces in virtual environments. While
virtual environment interfaces may be argued to be "natural" in principle, there are many features that can disrupt or distort the natural coupling of
actual reaching and walking, to create problems of stability and disorientation, lessons that have been well learned in the flight community.

To take account of these capabilities, the concept of sensory conflict has been extended into a theory of sensory rearrangement (Reason and
Brand, 1975) which states that: "all situations which provoke motion sickness are characterized by a condition of sensory rearrangement in which
the motion signals transmitted by the eyes, the vestibular system and the non-vestibular proprioceptors are at variance either with one another or
with what is expected from previous experience". In general, sensory rearrangement is a change in the normal relationship between body
movements and the resulting inflow of sensory stimuli to the central nervous system. It can also result from discoordination of one sensory inflow
pattern with that of another sense (e.g., a mismatch between vision and touch).

BEING PRESENT IN ANOTHER BODY OR IN AN EXTENDED BODY

The two experiments discussed in the previous page box show that humans systematically experienced a virtual body as if it were their own
when visually presented in their anterior extra-personal space and stroked synchronously (see also the book “The Ego Tunnel” by Thomas
Metzinger, 2009):

In sum, the experience of being localized within the physical body was determined by the visual perspective in conjunction with correlated
multisensory information from the body.

As indicated by the author of the proposal (“Virtual Reality and Telepresence”, Science 318: 1240 [DOI: 10.1126/science.318.5854.1240d]) these
experiments suggest that subjects were no longer present in their real body but were instead “present” in a synthetic body produced by VR.

More recently, new experiments showed that:

• A first person perspective of a life-sized virtual human female body that appears to substitute the male subjects' own bodies was sufficient to
generate a body transfer illusion (Slater et al., 2010);

• Synchrony between visual and proprioceptive information along with motor activity is able to induce an illusion of ownership over a virtual arm
(Sanchez-Vivez et al., 2010);

• Galvanic vestibular stimulation (GVS) interferes with the mechanisms underlying ownership, touch, and the localization of one’s own hand in
healthy participants by using the ‘‘rubber hand illusion” paradigm: left anodal GVS increases illusory ownership of the fake hand and illusory
location of touch (Lopez et al., 2010).

• The subjective illusion of ownership of a virtual hand can also be induced by the imagination of a motor act followed by movement of a virtual
hand (Perez- Marcos et al., 2009).

In short, the experiments showed that is possible to use virtual reality and eventually touch technology (haptic tools) to induce an
experience of “telepresence”, the feeling of “being there” (presence) by means of a communication medium: We want to exploit and further
expand this concept with this FET FLAGSHIP initiative.

VIRTUAL REALITY VIRTUAL EMBODIMENT

AN ARTIFICIAL REALITY THAT AN ARTIFICIAL REALITY THAT

PROJECTS THE USER INTO A PROJECTS THE USER INTO A
3D SPACE GENERATED BY BODILY SELF-CONSCIOUSNESS
THE COMPUTER. GENERATED BY THE COMPUTER.
 The medium term goal of this proposal is the understanding of the processes of presence and telepresence to
modify in a controlled way our body experience. In this activity we will integrate neuroimaging (fMri and/or
MEDIUM TERM evoked potential mapping), psycho-physiological and behavioral indexes (Observer, Theme). Specifically, the

APPLICATIONS: process should work at three different levels (Biocca, 1997) :

• sensory engagement: the degree to which the senses are connected to the interface/simulator;

Health • motor engagement: the degree to which interface/simulator sensors map the motion of the body;

Persuasion • sensorimotor coordination: the degree to which changes in body position correlate immediately and naturally
with appropriate changes in sensory feedback.
Sport Training The final outcome will be a full body simulation tool, able to produce the experience of a totally new body, altered
Entertainment in shape and size - e.g., an obese subject could experience a thin body to increase his/her motivation to change.

This approach can be considered the ultimate form of "Somatic Displacement" (Holopainen & Meiers, 2000) - the
ability of a person to project the mental model of his or her own identity into another physical form

The project will also explore how to use the modifications induced to the somatic awareness across a range
of medical, training and entertainment applications.

A MULTIDISCIPLINARY
APPROACH:
FROM COMPUTER
SCIENCE, TO HAPTICS
AND ROBOTICS, TO
COGNITIVE SCIENCE

A final long-term goal of this proposal is developing new approaches to training.

It is known that the body is integrated with the mind as a representational system, or as the neuroscientist Antonio
Damasio puts it, "a most curious physiological arrangement ... has turned the brain into the body's captive
audience" (Damasio, 1994, p. xv). In some ways, the body is a primordial display device, a kind of internal mental
simulator. The body is a representational medium for the mind. Some would claim that thought is embodied or
modeled by the body. Johnson and Lakoff (1980) argue against a view of reasoning as manipulation of
prepositional representations (the "objectives position"), a tabulation and manipulation of abstract symbols. They
might suggest a kind of sensory-based "image schemata" that are critical to instantiating mental transformations
associated with metaphor and analogy. In a way virtual environments are objectified metaphors and analogies
delivered as sensory patterns instantiating "image schemata."

In his book, Decartes' Error, the neuroscientist Damasio explains how the body is used as a means of embodying
thought:

"...the body as represented in the brain, may constitute the indispensable frame of reference for the neural
processes that we experience as the mind; that our very organism rather than some absolute experiential reality
is used as the ground of reference for the constructions we make of the world around us and for the construction
of the ever-present sense of subjectivity that is part and parcel of our experiences; that our most refined thoughts
LONG TERM and best actions, our greatest joys and deepest sorrows, use the body as a yardstick" (Damasio, 1994, p. xvi).

APPLICATIONS: Thinking of the body as a kind of simulator for the mind opens new possibilities for training. From the “learning by
doing” approach we can move to the “learning by experiencing the simulation of the doing” approach.

Implicit Motor In particular the final long term goal could be the MATRIX training: as in the Matrix movie, a subject could acquire
a complex behavioral skill – mastering martial arts or driving a plane – by experiencing through a somatic
Training simulation the behavioral processes related to the skill.
 !

MATRIX REDONE
Synthetic Environments for the Modification of the Bodily
Self-Consciousness and Implicit Motor Training

Prof. Giuseppe Riva, Ph.D.

http://www.giusepperiva.com

The
Challenge:
from Fiction
to Reality
In the science-fiction thriller The Matrix,
the heroes "plugged in" to a new
generation virtual world for training.

While their bodies rested in reclining

chairs, they learned how to fight martial-
arts battles, drive motorcycles and
helicopters in a virtual world.
To see the original Matrix scene (http://www.youtube.com/watch?v=6AOpomu9V6Q)
 The Goal:
Bodily Self
Consciousness
The Matrix environment simulates bodily self
consciousness: a subject can acquire a complex
behavioral skill – mastering martial arts – by
experiencing through a somatic simulation
(passive interaction) the behavioral processes
related to the skill.

Main objective of the project is to replicate this

approach without the need of “jacking in” the
brain.

VIRTUAL REALITY VIRTUAL EMBODIMENT

AN ARTIFICIAL REALITY THAT AN ARTIFICIAL REALITY THAT

PROJECTS THE USER INTO A PROJECTS THE USER INTO A
3D SPACE GENERATED BY BODILY SELF-CONSCIOUSNESS
THE COMPUTER. GENERATED BY THE COMPUTER.

A PARADIGM SHIFT: FROM EXTERNAL TO INTERNAL WORLD

FROM VIRTUAL REALITY TO VIRTUAL EMBODIMENT
CYBERPSYCHOLOGY ROBOTICS

EMBODIMENT AFFECTIVE COMPUTING

A MULTIDISCIPLINARY APPROACH: FROM COMPUTER SCIENCE TO COGNITIVE SCIENCE

THE INVOLVED COMMUNITIES

3-5 Year
Application:
Experience a
different body
Applications:
• Health Care
• Persuasive Computing
• Sport Training
• Entertainment

consciousness: a subject can acquire a complex
mastering martial arts or just assembing a car –
by experiencing through a somatic simulation
(passive interaction) the behavioral processes
Main objective of the project is to replicate this
approach without the need of “jacking in” the
References
Barsalou, L. W. (2003). Situated simulation in the human conceptual system. Language and
Cognitive Processes, 18, 513-562. http://www.ingentaconnect.com/content/psych/plcp/
2003/00000018/F0020005/art00002;jsessionid=6jsg7kn0fpntv.victoria
Barsalou, L. W., Simmons, K. W., Barbey, A. K., & Wilson, C. D. (2003). Grounding conceptual
knowledge in modality-specific systems. Trends in Cognitive Sciences, 7(2), 84-91. http://
www.science.mcmaster.ca/Psychology/seminars/readings/barsalou-'03-tics.pdf
Biocca, F. (1997). The Cyborg's Dilemma: Progressive Embodiment in Virtual Environments, JCMC, 3
(2), http://jcmc.indiana.edu/vol3/issue2/biocca2.html
Biocca, F., Rolland J.P. (1998) Virtual Eyes Can Rearrange Your Body: Adaptation to Visual
Displacement in See-Through, Head-Mounted Displays, Presence, Vol. 7, No. 3, Pages 262-277.
http://www.mitpressjournals.org/doi/abs/10.1162/105474698565703
Blanke, O., Landis, T. et al. (2004). Out-of-body experience and autoscopy of neurological origin. Brain,
127 (2), 243-258. http://brain.oxfordjournals.org/cgi/content/abstract/127/2/243
Blanke O, Metzinger T. (2009). Full-body illusions and minimal phenomenal selfhood. Trends in
Cognitive Science, Jan;13(1):7-13. http://linkinghub.elsevier.com/retrieve/pii/
S1364-6613(08)00250-7
Castiello, U., Lusher, D., Burton, C., Glover, S. Disler, P. (2004). Improving left hemispatial neglect using
virtual reality. Neurology, 62(11):1958-62. http://www.ncbi.nlm.nih.gov/sites/entrez?
Db=pubmed&Cmd=ShowDetailView&TermToSearch=15184596&ordinalpos=7
Damasio, A.R. (1994). Descartes' Error: Emotion, Reason, and the Human Brain, Putnam Publishing,
1994. http://en.wikipedia.org/wiki/Descartes'_Error
De Ridder, D., Van Laere, K., Dupont, P., et al. (2007). Visualizing Out-of-Body Experience in the Brain.
New England Journal of Medicine, 357 (18), 1829-1833. http://content.nejm.org/cgi/content/full/
357/18/1829
Ehrsson, H.H. (2007). The Experimental Induction of Out-of-Body Experiences, Science 317 (5841),
1048. http://www.sciencemag.org/cgi/content/abstract/317/5841/1048
Gallese, V. (2005). Embodied simulation: From neurons to phenomenal experience. Phenomenology
and the Cognitive Sciences(4), 23-48. http://www.unipr.it/~gallese/PCS2005.pdf
Holopainen, J. & Meyers, S. (2000). "Neuropsychology and Game Design", Consciousness Reframed
III. http://stephan.com/NeuroBio.html,
Hommel, B., Müsseler, J., Aschersleben, G., & Prinz, W. (2001). The theory of event coding (tec): A
framework for perception and action planning. Behavioral and Brain Sciences, 24(5), 849-937.
http://www.bbsonline.org/Preprints/Hommel/Hommel.pdf
Knoblich, G., Thornton, I., Grosjean, M., & Shiffrar, M. (Eds.). (2005). Human body perception from the
inside out. New York: Oxford University Press. http://www.us.oup.com/us/catalog/general/subject/
Psychology/Cognitive/?view=usa&ci=9780195178371
Lakoff, G., Johnson, M. (1980/2003). Metaphors We Live By. University of Chicago Press. http://
en.wikipedia.org/wiki/Conceptual_metaphor
Lackner, L.R., DiZio, P. (2005). Vestibular, proprioceptive and haptic contributions to spatial orientation.
Annual Review of Psychology, 56: 115-147. http://arjournals.annualreviews.org/doi/abs/10.1146/
annurev.psych.55.090902.142023?cookieSet=1&journalCode=psych
Lenggenhager, B., Tadi, T., Metzinger, T. & Blanke, O. (2007). Video Ergo Sum: Manipulating Bodily
Self-Consciousness, Science 317 (5841), 1096. http://www.sciencemag.org/cgi/content/abstract/
sci;317/5841/1096
5-10 Year
Application:
Implicit Motor
Training
The Matrix environment simulates bodily self
behavioral skill – driving a motor bike,
related to the skill.
brain.
Lopez C, Lenggenhager B, Blanke O. (2010). How vestibular stimulation interacts with illusory hand
ownership. Conscious Cogn. 2010 Mar;19(1):33-47. http://linkinghub.elsevier.com/retrieve/pii/
S1053-8100(09)00189-5
Metzinger, T. (2009). The Ego Tunnel: The science of mind and the myth of the Self. Basic Books: New
York.
Prinz, W. (1997). Perception and action planning. European Journal of Cognitive Psychology, 9(2),
129-154. http://www.fil.ion.ucl.ac.uk/SocialClub/prinz_1996.pdf
Ramachandran, V. S. & S. Blakeslee (1998), Phantoms in the brain: Probing the mysteries of the human
mind., William Morrow & Company, http://ajp.psychiatryonline.org/cgi/content/full/157/5/841
Reason, J.T. & Brand, J.J. (1975). Motion Sickness. London: Academic Press. http://www.amazon.com/
Motion-Sickness-James-Reason/dp/0125840500/ref=sr_1_1/002-8390552-5128060?
ie=UTF8&s=books&qid=1194861553&sr=1-1
Riva G. (1998). Modifications of body-image induced by virtual reality.Perceptual & Motor Skills,
86:163-70. http://www.ncbi.nlm.nih.gov/sites/entrez?
Db=pubmed&Cmd=ShowDetailView&TermToSearch=9530726&ordinalpos=11
Riva, G. (2006). “Being-in-the-world-with”: Inner Presence Meets Social and Cognitive Neuroscience, in
G. Riva, M.T. Anguera, B.K. Wiederhold, F. Mantovani (Eds.), From Communication to Presence:
Cognition, Emotions and Culture towards the Ultimate Communicative Experience. IOS Press:
Amsterdam. http://www.emergingcommunication.com/volume9.html
Riva, G. (2007). Virtual Reality and Telepresence. Science 318: 1240-2. http://www.sciencemag.org/cgi/
content/full/318/5854/1240d”,
Riva, G. (2009). Is Presence a Technology Issue? Some insights from Cognitive Sciences
Virtual Reality,13 (3): 159-169.
http://sites.google.com/a/unicatt.it/giusepperiva/Riva%282009%29Ispresenceatechnologyissue.pdf?
attredirects=0
Riva, G., Bacchetta, M., et al. (2001). Virtual reality-based multidimensional therapy for the treatment of
body image disturbances in obesity: a controlled study. Cyberpsychology & Behavior, 4(4):511-26.
Sanchez-Vives MV, Spanlang B, Frisoli A, Bergamasco M, Slater M. (2010). Virtual hand illusion
induced by visuomotor correlations. Apr 29;5(4):e10381. http://www.plosone.org/article/info%3Adoi
%2F10.1371%2Fjournal.pone.0010381
Slater M, Spanlang B, Sanchez-Vives MV, Blanke O. (2010). First person experience of body transfer in
virtual reality. PLoS One. 2010 May 12;5(5):e10564. http://www.plosone.org/article/info%3Adoi
%2F10.1371%2Fjournal.pone.0010564
Waterworth J.A, Waterworth E.L., Mantovani F., Riva G., (2010).On Feeling (the) Present: An
evolutionary account of the sense of presence in physical and electronically-mediated
environments, Journal of Consciousness Studies, 17, No. 1–2: 167-178.
http://sites.google.com/a/unicatt.it/giusepperiva/Waterworth%282010%29Onfeelingthepresent.pdf?
attredirects=0
Zagalo, N., Torres, A., Branco, V. (2006). Passive Interactivity, an Answer to Interactive Emotion. Lecture
Notes in Computer Science. 4161: 43-52. http://www.springerlink.com/content/v403225386251752/