Melody creation with Brain-Computer Interface

ROBERTO MURGUIA E-mail: rm723@york.co.uk The purpose of this paper is to provide an overview of some of the most recent advancements in non-invasive Brain-Computer Interface technology and evaluate its feasibility as a practical tool for real-time composition of functional music. The specific goal is to be able to sequence single-note melodies in real time. Because of limitations in EEG decodification, there has been a lack of emphasis in melodic control until recently, therefore a special interest was taken in work that was published in the last couple years. Introduction Despite significant advancements in BCI in the past decade, mass media has just recently started to cover the technology more extensively in the form of occasional "novelty" items, such as a teenager playing Space Invaders with BCI alone (Murph, 2006). In this case, news articles depicted BCI as a mature technology that would soon be widely available, and some failed to mention that the BCI that the teenager employed was partially invasive and involved electrodes being surgically placed underneath the skullcap for clearer signal (Fitzpatrick, 2006). Non- invasive BCI currently employs EEG (electroencephalographic) devices, which unfortunately have a low signal-to-noise ratio, complicating pattern recognition and requiring hours of practice from the subject. However, these devices are preferred in research applications for reasons of practicality. The objective of this paper is to survey recent advancements that will allow the implementation of EEG as a compositional tool. What is desired is the ability to 1)create single monotonic melodies, 2) trigger sequences or loops while being able to create melodies on top and 3) to be able to select commands such as program change or transport control. Relevant BCI research There are several companies that develop and promote EEG technology to the main public as well as researchers. A well-known company that focuses on the main consumer is Emotive Systems, whose brain controller named EPOC has been getting coverage in video game forums and conventions. A good representative of the companies that focus on researchers is Guger Technologies, which promotes research by providing information and tutorials for integrating their products with MATLAB through their website http://www.gtec.at/products/g.BCIsys/bci.htm. Among their info is a good summary of the common current implementation techniques for BCI, which are worth mentioning: 1) The subject imagines the movement of different limbs to control a variable by changes in the sensorimotor areas of the central cortex which are classified by weighting spectral parameters of different frequency bands for different electrode positions(Guger Technologies, 2009). This technique was demonstrated by two people playing Pong (Peplow,2004) 2) Steady state visually evoked potentials (SSVEP) directing attention to sources with a specific flicker frequency to evoke similar EEG components. This technique was used at a concert in Prague by the Multimodal Brain Orchestra in April 2009. (Palmer, 2009)

3) P300 waves are event related potentials recorded by EEG when an unlike event occurs. This method was used by Mick Grierson with a high success rate of guessing inner BCI tones (Grierson 2008). He demonstrates this in a youtube video http://www.youtube.com/watch?v=kNp71xBDcMA which shows him triggering notes as he focuses on them showing up on the screen. This poses a timing problem and shows the current shortcomings of this method for real-time composition. 4) Slow shifts of cortical potentials when imagining an expected event such as a light turning to green (Guger Technologies 2009) 5) Other mental tasks such as mental arithmetic, mental cube rotation, attention versus relaxation. (Guger Technologies 2009) The last point in the list can be illustrated by the two most relevant works to the purposes of this paper, those of Schaefer/Desain and Klonowski/Duch/Perovi/Jovanovic. Schaefe/Desain present a method for decomposition of continuous EEG signals using multiple linear regression, offering a more precise look at EEG signatures than by the P300 method. Their paper is based on part of a larger experiment that was not just exclusively directed towards music and therefore they did not take into account the musical abilities of the subjects. Klonowski/Duch/Perovi/Jovanovic took special consideration in the musical abilities of the subjects and their work is the most relevant to real-time creation of inner tones and the purposes of this paper. Their experiments focused specifically on exposing the subjects to simple melodies, externally and internally (by memory). They simply used Fourier analysis on the EEG and related it to the tones. Even though the signal to noise ratio was much lower on the imagined inner tones, they did get positive evidence: in a significant number of experiments, spectral lines corresponding to imagined inner tones were detected. One of the ways they detected the inner tones was to train a neural network to recognize the fingerprints of the inner tones. It can hardly avoid spectograms as initial objects. Also, they note that the experiments with subjects lacking musical ability were negative. (Klonowski 2008) Conclusion There is great potential for BCI composition in the short run, especially if the operator can utilize techniques employed by musicians. For that, it is imperative to have a strong background in ear training. Voice and pitch recognition can be a stepping-stone to assist the performer in concentrating on the desired notes and therefore obtain signals from external stimuli rather than from inner tones exclusively. Other techniques could involve color association, akin to the experience of individuals with synesthetic tendencies, and associating certain auditory tones with color cues. These color associations can be related to either absolute pitches or to intervals between current and previous notes. While Schaefer focuses on absolute pitch as well as the intervals between pitches, a useful simplification of the research could focus on the latter, since this technique is used when teaching advanced ear training and could obviate the distraction of concentrating on a specific tonality. Other improvements may be technical, such as modifying the positioning of electrodes in order to reduce inner tone detection problems (W. Klonowski, 2009), . Klonowski, Jovanovic, Duch and Perovic conclude that improvements in EEG resolution will spur the development of multicommand systems. Klonowski et alia also agree that these innovations will mostly be taken advantage of by "musically gifted people" and believe that advancements in EEG resolution will lead to a greater understanding of the differences in neural processing of auditory input by musicians and non-musicians.

References
Fitzpatrick, T. Washington U., Oct 2006, Teenager moves video icons just by imagination http://news-info.wustl.edu/news/page/normal/7800.html Guger Technologies, Austria, 2009 http://www.gtec.at/products/g.BCIsys/bci.htm Grierson, M., Composing with brainwaves: minimal trial P300b recognition as an indication of subjective preference for the control of a musical instrument, in Proceedings of International Cryogenic Materials Conference (ICMC 08), Seoul, Korea, July 2008. Klonowski, W., Duch, W., Perovic, A, Jovanovic A., 2009 Some Computational Aspects of the Brain Computer Interfaces Based on Inner Music, Computational Intelligence and Neuroscience Volume 2009 Klonowski, W., From conformons to human brains: an informal overview of nonlinear dynamics and its applications in biomedicine, Nonlinear Biomedical Physics, vol. 1, article 5, pp. 119, 2007. Murph, D., Teenager plays Space Invaders with only his brain http://www.engadget.com/2006/10/11/teenager-plays-space-invaders-with-onlyhis- brain/ Palmer, J., 2009, World premiere of brain orchestra, Science and technology reporter, BBC News, Prague http://news.bbc.co.uk/1/hi/sci/tech/8016869.stm Peplow, M., Mental ping-pong could aid paraplegics, August 2004, Nature News http://www.nature.com/news/2004/040823/full/news040823-18.html Schaefer,R., Desain, P., Suppes, P., Structural decomposition of EEG signatures of melodic processing (August 2009) Biological Psychology 2009, Science Direct Schaefer, R.S., Guimaraes, M.P., Desain, P., Suppes, P., 2008. Detecting imagined music from EEG. In: Miyazaki, K., Hiraga, Y., Adachi, M., Nakajima, Y., Tsuzaki,M. (Eds.), Proceedings of the 10th International Conference of Music Perception and Cognition. Causal Productions, Sapporo, Japan.