Augmented Reality Scenarios for Guitar Learning

F. Liarokapis

Presented by: Andres Felipe Aguirre Quintero DECO7450 - Advanced Human-Computer Interaction Augmented Reality (AR) is a field of computer research which deals with the combination of real-world and computer-generated data. At present, most AR research is concerned with the use of live video imagery which is digitally processed and "augmented" by the addition of computer-generated graphics. To provide a higher level of immersion into the application, the user should physically interact with the audio-visual information in a natural way. To reach a natural audio system, we need to place the sound sources spatially, introducing the concept of 3d audio where we have to manipulate the speakers surrounding the user to give a virtual placement of the sound sources in the 3 dimensional space, usually we need more than 4 speakers to provide a decent natural effect. In this research they apply audio-visual AR to provide a self-teaching system for guitar players. They implemented a functional prototype, producing real time high definition AR images and a 3D sound system. The hardware used was: A laptop, a 3d speakers system, a HMD (head mounted display, with this one is possible to identify the location and the orientation of the user view), and a set of MARKER CARDS representing the guitar notes and chords.

Using different software libraries, they created the tutorial contents, like the prerecorded sounds, the annotations, the 3d models for the AR, and also training the image recognition system to recognize the marker cards. One of the most important tasks is the creation of the marker cards, they need to be big (10 cms approx.) and have good contrast because the image processing system needs to be trained to recognize them and use them as reference points and command inputs.

The prototype have 3 learning scenarios: 1 - Introduction to theoretical scenarios: This one involves the understanding of some basic components of the guitar, so each marker card contains an audio explanation and a 3d model of various guitars ranging from classical to electric guitars.

Also placing a marker card in a specific place of any guitar, the tutorial will place textual annotations on the parts of the guitar

2 - Theoretical and practical scenarios: The user selects a chord that wants to practice and places the correspondent marker card on the guitar, and the application shows where must place his/her fingers to play the specific chord.

To obtain more information, as well as practice the theory, the learner can get an interactive audio-visual augmentation about each specific note, showing textual information and a static image representing the tablature of it.

3 - Learning complete songs: One marker containing the name of the song will make appear a virtual tablature of the song and the lyrics, and will be played by the application so the user can listen to it and try to follow the rhythm.

Conclusions and Limitations: This app was easy to implement and the results are promising, but the computational power needed is heavy so for mobile apps is not possible yet, the image recognition also is not 100% accurate. The users liked it, they thought is easy to use and good looking, but the prototype still needs more development and content to be competitive against commercial magazines or learning tools. In the future they plan also to record the user guitar input and give score or feedback if the user is playing properly.