Mapping gesture across art forms

PhD by Project Roger Alsop

All the arts can be seen as a manifestation of one and the same impulse. (Robin Allott 1994) Remember that a picturebefore being a war horse or a nude woman or an anecdoteis essentially a flat surface covered with colours assembled in a certain order. (Maurice Denis 1890)

The computer is a multidimensional, interdisciplinary processing and communication machine. (Joan Truckenbrod 1992)

Presentation structure and project summary

Overview of the project considering the:
purpose theoretical base conceptual factors developmental factors Resolutions

Chapter outlines Demonstration of works

Project purpose
To develop a set of approaches to translating motion, text, and visual gestures into temporal, ephemeral, perceiver focused audio-visual (screen) works through a perceiver interactive processes.

Purpose
To create audio visual works that use:
Movement gesture Speech gesture Visual gesture

to form the works at their most fundamental, structural level

Purpose
It is essential that the interaction between the audio and visual aspects of the works be attributable, and not based on personal, aesthetic interpretations of the works considered in creating the new works. In doing, so demonstrate a direct translation between gesture from one form of communication to another.

Gestures
Three types of physical gesture of decreasing size are considered:
The movement gesture: is the conscious physical movement of the body. The speech gesture: is the intuitive motion of the mouth as, notated through the phoneme The visual gesture is saccadic motion, the unconscious motion of the oculomotor muscles

Theoretical base
Internal theories (what I am bringing to the project) The author expresses Allotts impulse through assembling elements (data) in a certain order That this data can be manipulated via multidimensional, interdisciplinary processing to create new works That the perceiver of any art work has equal & similar agency in the interpretation of the impulse to the author

Theoretical base
External theories There are two strands to the external theoretical bases behind this project;
specific discussions regarding communication, interaction, and process development specific art works that exemplify concepts influential in the project

This project considers an artwork per se as being a theoretical discussion

Theoretical base (artworks)

Specific artworks that inform this work Onos Box of smile Cages Theatre Piece No 1, Radio Music, and 433 Nonos Fragmente Stille, An Diotima Yuskavages True Blondes Pollocks Number 11

Theoretical base (discussions)

Specific discussions inform this work Lyotards Differend Jakobsons Tripartite schemata of verbal communication Kansei Notation Algorithmic processes

Conceptual factors
The gesture as the universal communicative tool
Some gestures are deliberate and conscious, some are intuitive, and some are automatic

Conceptual factors
The gesture as the universal communicative tool
Some gestures are deliberate and conscious, some are intuitive, and some are automatic

The majority of communication is aural and/or visual

Conceptual factors
The gesture as the universal communicative tool
Some gestures are deliberate and conscious, some are intuitive, and some are automatic

The majority of communication is aural and/or visual Aural and visual communications are broadcast through subtle changes in a medium (air molecules or photons/ electromagnetic waves)

Conceptual factors
The gesture as the universal communicative tool
Some gestures are deliberate and conscious, some are intuitive, and some are automatic

The majority of communication is aural and/or visual Aural and visual communications are broadcast through subtle changes in a medium (air molecules or photons/ electromagnetic waves) Communication is an oscillation between the origin (author) and the destination (perceiver)

Conceptual factors
These changes in a medium can be represented as numeric data (which can be manipulated) That the data produced by the creator is an interpretation of their intentions This data received is interpreted by the perceiver, and is separate from the creator

Conceptual factors
Fundamental tenets of the project All art is essentially ephemeral, existing only in its perception. All art can be interpreted as data. The data-fication of art allows its reinterpretation and translation (bridging Derridas horizon of a consensual sense, of a translatability)

Developmental factors in creating the works

Developing interface design processes for translating aspects of
verbal gesture into an audio visual result motion gesture into an audio visual result ocular gesture into an audio visual result

Exploring, developing and articulating conceptual frames that have intuitively informed my processes

Developmental factors in my practice

This project has developed my practice to include visual aspects to the formation of concrete relationships between visual and aural aspects to emphasize the ephemeral, perceiver centered nature of art works

Resolution
The creation of three interactive works based on perceiver interaction These works are created from
perceiver movement perceiver data input perceiver interpretation

and are essentially ephemeral, designed to exist only when being acted with by the perceiver

Project summary/description
This project explores ways that the results of different arts making processes can create novel results in a single art making process. Premises on which the work is based: that the points of intersection between creative processes are fundamentally greater than the points of divergence; that while an artworks experiential results may be dissimilar, their intellectual and instinctual causes have fundamental similarities. that the perceivers role in the making an art work is similar to the authors

Project summary/description
The project is the creation of systems that can be used by the artist in translating the results of one art form into results in another. These systems are similar to other systems used to translate concept and idea into a perceivable result (eg the piano or paintbrush). When creating the systems interaction with the perceiver was an essential part of the design

Chapter outline

Chapter 1
Introduction and document outline:
the conceptual/theoretical frames the research question and approach the approach to the literature background to the works and processes discussion of each of the three processes

Chapter 2
Conceptual frames
Role of the maker/perceiver Role of the process

Questions
What is a gesture How can gestures be translated
Interpreting gesture as data Computer as pivot

Chapter 3
Literature - Artworks Detailed discussion on
Onos Box of smile Cages Theatre Piece No 1, Radio Music, and 433 Nonos Fragmente Stille, An Diotima Yuskavages True Blondes Pollocks Blue Poles

Chapter 4
Literature - Concepts Lyotards Differend Jakobsons Tripartite schemata of verbal communication

Chapter 5
Literature - Processes Kansei Abstraction through notation Algorithmic processes

Background to works
Overall goals:
to create works that allow interaction that is novel but predictable

Interface design
that is simple but requires developing an intuitive understanding

Processes
that develop similar interfaces across the three different translators

Anticipated outcomes
processes that is engaging due to its outcomes

Chapter 6
Discussing IMAGE
Goals
To use the saccadic motion, an unconscious, small gesture, to create a computerized viewing of an image

Process
To develop interfaces that reflect the user through structures based on the saccadic motion of the eye

Outcomes

Chapter 7
Discussing TEXT
Goals
To use the speech motion, as defined through phonemic notation to create a computerized viewing of an image

Process
To develop interfaces that reflect the user through structures based on speech gesture

Outcomes

Chapter 8
Discussing MOTION
Goals
To use the physical gesture of the viewer to create a computerized image reflecting their motion

Process
To develop an interface that reflects the motion of the viewer in an abstract form.

Outcomes

Chapter 9
Conclusions Discussions of the works as a whole Potential developments

Discoveries
How stylized motion can create motion in another art form
Motion/interaction to AV outcome

How sonic gestural structure can be translated from one temporal art form to another
Text/interaction to AV outcome

How automatic motion creates a gesture

Ocular motion/interaction to AV outcome

Implications
Forming an open ended approach to art making Forming a new set of tools for art making/art perception POTENTIAL DEVELOPMENT OF NEW PARADIGMS FOR ART MAKING
The artist creates Field of play for audience

Outcomes
Creation of an open-ended making process developed from any input Development of an improvisational, perceiver focused approach to creating abstract audiovisual works Developing relationships between visual and sonic elements in art making in the viewer Developing processes for crossing boundaries between art forms Creating fulcrum or pivot around which art forms may orbit

Conclusions
Creation of real time ephemeral works Direct transformation between performative processes and concrete processes

(To be further explored here)

Tools are not benign Direct transformation between performative processes and concrete processes

(Value of art)
To expand the imagination beyond what can be imagined Direct transformation between performative processes and concrete processes My role is similar to that of a playwright, I create a set of possibilities that can be interpreted by the user A play is interpreted by a director/actor/etc within the paradigms of the play just as my work creates a paradigm for interpretation. the

Developing the discussion

Precedents
Performance art
The author creates a pointer to a creative event, the script mutates into a presentation.

Abstract expressionism
The author creates an event impossible to describe textually or verbally

Conceptual art
Art making as creating a pointer or representation to cause an interpretation in the perceiver

Issues 1
An artwork exists in its perception and interpretation The constantly shifting relationship between maker and perceiver Perceiver is an active agent in the art making process Maker and perceiver are simultaneously INTER/INTRA DEPENDANT in the creation of a work

Issues 2
All communication can be interpreted as data Most data is perceived extereoceptively, forming interoceptive responses Maker and perceiver are simultaneously INTER/INTRA DEPENDANT in the creation of a work

Issues 3
My work is Multidisciplinary & Multi Modal The focus is the relationship between the sender, object and perceiver. The perceiver is as active in the creation of a work as the creator The perceiver is the active receiver of the data Very slippery concept as it is very subjective Responses are subject to too many things Perceiver Art forms are transcending boundaries

The Maker Perceiver relationship

The maker creates an art gesture to which they are tangibly connected. The gesture affects the perceiver creating a virtual connection between the maker and the perceiver. The perceiver also connects to the gesture, remaking it with each viewing or recollection.

Virtual relationship between Perceiver and Maker

Maker

Gesture

Perceiver

Virtual relationship between Maker and Perceiver

Questions
What is a gesture
How can a gesture be used in transformational mapping

Role of the perceiver

In perceiving the artistic gesture Perception is a creative act

Role of the author

In creation of the artistic gesture

Questions: What

is a gesture

A gesture is a movement or change of state perceived over time .

Whatever the manifestation of the gesture it begins in an internalized, mental state. The gesture is externalized and made material by the author, broadcast through a medium, perceived, and then finally returned to a mental state in the perceiver

Questions: What

is a gesture

A gesture must be perceived Most gestures originate in the mind of the transmitter and are broadcast, creating a physical gesture transmitted as a
Visual gesture Sonic gesture,

which then creates an interpreted gesture in the mind of the perceiver

Questions: Gesture

transmission

To be perceived a gesture must be externalized from its source and transmitted. Visual gestures are transmitted through fluctuations in electromagnetic radiation perceived as light waves cycling between approximately 384 THz and 769 THz Sonic gestures are transmitted through mechanical fluctuations in air, received as sound waves travelling cycling approximately 20 Hz and 20000 Hz

4 areas of art
intersection
ART
image

sound movement

text

Questions: Role

of the perceiver

To paraphrase Barthes, from Death of the Author:

The perceiver is the anonymous space in which all gestures are inscribed, the unifying, impersonal destination, holding all gestures.

The perceiver creates the work through their own interpretation of the inscribed gestures All perception is unique, individual, and immediate to the perceiver in the act of perceiving

Questions: Role

of the perceiver in my work

The perceiver is creates the work, their input, regarding:

choice of interpretation processes (TEXT) choice of materials and interpretation processes (IMAGE) choice of interpretation processes and their motion (MOTION)

creates the work that they experience through applying gestures to the tools and subsequently interpreting the resultant AV gesture.

Questions: Role

of the author

The author is a pointer to the intentions of an art work. The author externalizes their intentions through translating them into sonic or visual data to be perceived by an external, anonymous perceiver. The author creates potential interpretations, of the authors intentions, in the perceiver through providing visual or sonic data

Questions: Role

of the author in my work

Here the author builds on their role as a pointer to the intention of the art work. By creating an environment in which the perceiver uses the paradigms of that environment created by the author to cause a re-viewing of visual and sonic input.

Methodology
To create pivots around which art works can be translated from one mode to another Explore and discuss approaches to art making and perception through considering specific artworks and specific theories

Methodology: Creating
Inspirations

Pivots

Histories

Previous work

Conceptual frames

Technologies

PROCESSES

OUTCOMES

Methodology: Creating

Pivots

The process creates the pivot on which the transformation can manifest The pivot is the liminal, interstitial point at which a mapping can be made

Methodology: Discussion
Elevate the process to the result Processes exits in art making
An artists model is a process, creating a template for a painting Serial composition, fugue, counterpoint are examples of templates in music

Make the process the result

Creative outcomes
Three interactive audiovisual works/processes: 1. TEXT 2. IMAGE 3. MOTION These use different input from external sources to create AV works based on processes created here. The user/perceiver can create a work based on the paradigms built into the processes.

Previous work
Master of arts Papers

Previous work
Master of arts

Previous work
Master of arts Papers

Location: Conceptual frames

Communication/making sense
Everything is data Lyotard Jakobson Cage

Conceptual frames: Everything

is

data

Conceptual frames: Lyotard

& the

Differend
the very point of The Differend is the very horizon of a consensual sense, of a translatability, of a possible to translate. (Derrida, 2000)

Conceptual frames: Jakobson

&

Communication
Insert image of communication environment here

Conceptual frames: Cage

& Theatre

Piece No 1

Conceptual frames: Communication

Conceptual frames: Making

Sense

Location: Influences
Artists
Non- Electronic arts Electronic arts

Systems
Approaches to art making

Creativity
Generation Recognition

Location:
Sound

Non- Electronic Artists

Nono, Webern, Cage

Vision
Ono, Yuskavage, Pollock

Text
Joyce, Bukowski, Gomringer

Location:
Sound

Electronic Artists

Ikeda, Lansky, Ashley, Lucier, Cage

Vision (Time based)

Polli, Marclay, Czarnecki, Barbara Kruger, Wearing, Viola there are also VJ artists as a point of contention, visual music

Location:

Systems

Cage Kansei Computers Notation

Location: Systems: Cage

iChing Indeterminacy Negation of ego

Location: Systems: Kansei

Location: Systems: Notation

Note-fication is data-fication Notation is an abstracted representation of tangible events, allowing those events to be considered algebraically This allows the artist to develop and interpret works through mathematical processes And to adjust events without reference to their materiality

Location: Notation: Sonic

Notation

Musical notation: Pitch and duration representation

Location: Notation: Sonic

Notation

Sonic notation: Frequency, amplitude and duration representation

Location: Notation: Sonic

Notation

Graphic notation: Representing form to be interpreted by the performer

Location: Notation: Motion

Notation

Laban notation

Location: Notation: Motion

Notation

visual representation

Location: Notation: Image

Notation

Paint by numbers

Location: Notation: Image

Notation

Paint by numbers
Completed image

Location: Notation: Image

Notation

The computer usually stores or generates an image as 4 numbers located at a specific point (pixel) within a screen, the first three numbers relate to the red, green, blue values of the pixel, and the fourth to the opacity of the resultant colour.

Location: Notation:

Image Notation

Adjusting numbers in computer stored images.

RGB to RGB (Red Green Blue to Red Green Blue)

Location: Notation:

Morphological Notation

Denis Smalley described a way of considering the morphological changes that occur over time in electro-acoustic music, using text as his communication tool Kevin Patton developed these ideas using visual images to represent morphological change in sound. This representation can be used to represent any kind of morphological change.

Location: Notation:

Morphological Notation

Location: Systems: Computers

Location:

Creativity (imagination)

Generation Recognition

Processes
Create systems to make AV works using
Motion (demonstrate) Text (demonstrate) Images (demonstrate)

based on physical gestures of the

Body/large/considered/voluntary Mouth/small/intuitive/learned/voluntary Eyes/minute/involuntary (saccadic motion)

Processes
These systems are creative works that have resonances with:
Playwriting, in which the playwright sets a text that may be reinterpreted Jazz improvisation, in which a sequence of harmonies are interpreted by a musician

Processes:

MOTION

Improvised creation of a AV work Ephemeral input Reactive/Collaborative

machine human human machine

Based on the kinetic/silent gesture The perceiver becomes the art work

Processes: Motion:

COMMENT

The perceiver becomes the art work; creating the work in the act of perceiving the work

Processes:

TEXT

Static/set input Concrete User interactive based on the sonic/temporal gesture

Processes: Text:

COMMENT

Static/set input Concrete User interactive based on the sonic/temporal gesture

Processes:

IMAGE

Static/set input Concrete User interactive Based on silent/non-temporal (IMMEDIATE) gesture

Processes: Image:

COMMENT

Static/set input Concrete User interactive Based on silent/non-temporal (IMMEDIATE) gesture