VIRTUAL PIANO

The project is basically meant to make fun with technology. A Piano is played without pressing any key(as in case of actual one) but just by waving hand. Construction & working: In order to work out this task, I have used MATLAB. A physical structure is made (fig.1). The top face of it has eight holes. A webcam is placed beneath it. The dimensions are so adjusted that the view angle of webcam can cover up all holes. The images are taken by webcam continuously and processed in MATLAB. An video input object vid is created. vid=

videoinput('winvideo',1,'YUY2_160x120'). Then images are taken continuously using a infinite loop.

The images are converted into black & white according to a predefined threshold value. bwimg=im2bw(a,.3);(.3 is a threshold value). A JAVA file has been imported in the script (coding) in order to press and release keyboard key automatically.

import java.awt.Robot; import java.awt.event.*; SimKey=Robot; A infinite loop is created to repeat steps continuously.

The process is triggered manually.

Initially when all the spots are white no key will be pressed and process will be executed hence no tune will be played. As soon as a hole is covered a corresponding key will be pressed and a tune will be played. In order to identify which hole has been darkened ,centre of each white spot is calculated and rounded.

centroids = cat(1, ct.Centroid); centroids=round(centroids);

When a specific hole is darkened a specific key is pressed and corresponding a tune will be played. When the dark spot again become white (after removing hand) the key is released. And the process continues. Code for virtual piano:
import java.awt.Robot; import java.awt.event.*; SimKey=Robot; vid= videoinput('winvideo',1,'YUY2_160x120'); set(vid,'ReturnedColorSpace','rgb'); set(vid, 'FramesPerTrigger', 1); set(vid, 'TriggerRepeat', Inf); triggerconfig(vid,'manual'); start(vid); pause(2); trigger(vid); a = getdata(vid,1); imshow(a); bwimg=im2bw(a,.1); mainimg = bwareaopen(bwimg,20); figure(2),imshow(mainimg);

ct= regionprops(mainimg, 'centroid'); centroids = cat(1, ct.Centroid); centroids=round(centroids); toggle = [0 0 0 0 0 0 0 0]; pause(10); while(1) trigger(vid); a1=getdata(vid,1); bwimg1=im2bw(a1,.8); if(bwimg1(centroids(1,2),centroids(1,1))==0 && toggle(1)==0) %disp('first'); SimKey.keyPress(KeyEvent.VK_Q); toggle(1)=1-toggle(1); elseif(bwimg1(centroids(1,2),centroids(1,1))==1 && toggle(1)==1) SimKey.keyRelease(KeyEvent.VK_Q); toggle(1)=1-toggle(1); end if(bwimg1(centroids(2,2),centroids(2,1))==0 && toggle(2)==0) %disp('second'); SimKey.keyPress(KeyEvent.VK_W); toggle(2)=1-toggle(2); elseif(bwimg1(centroids(2,2),centroids(2,1))==1 && toggle(2)==1) SimKey.keyRelease(KeyEvent.VK_W); toggle(2)=1-toggle(2); end if(bwimg1(centroids(3,2),centroids(3,1))==0 && toggle(3)==0) %disp('third'); SimKey.keyPress(KeyEvent.VK_E); toggle(3)=1-toggle(3); elseif(bwimg1(centroids(3,2),centroids(3,1))==1 && toggle(3)==1) SimKey.keyRelease(KeyEvent.VK_E); toggle(3)=1-toggle(3); end

if(bwimg1(centroids(4,2),centroids(4,1))==0 && toggle(4)==0) %disp('fourth'); SimKey.keyPress(KeyEvent.VK_R); toggle(4)=1-toggle(4); elseif(bwimg1(centroids(4,2),centroids(4,1))==1 && toggle(4)==1) SimKey.keyRelease(KeyEvent.VK_R); toggle(4)=1-toggle(4); end if(bwimg1(centroids(5,2),centroids(5,1))==0 && toggle(5)==0) %disp('fifth'); SimKey.keyPress(KeyEvent.VK_T); toggle(5)=1-toggle(5); elseif(bwimg1(centroids(5,2),centroids(5,1))==1 && toggle(5)==1) SimKey.keyRelease(KeyEvent.VK_T); toggle(5)=1-toggle(5); end if(bwimg1(centroids(6,2),centroids(6,1))==0 && toggle(6)==0) %disp('sixth'); SimKey.keyPress(KeyEvent.VK_Y); toggle(6)=1-toggle(6); elseif(bwimg1(centroids(6,2),centroids(6,1))==1 && toggle(6)==1) SimKey.keyRelease(KeyEvent.VK_Y); toggle(6)=1-toggle(6); end if(bwimg1(centroids(7,2),centroids(7,1))==0 && toggle(7)==0) %disp('seventh'); SimKey.keyPress(KeyEvent.VK_U); toggle(7)=1-toggle(7); elseif(bwimg1(centroids(7,2),centroids(7,1))==1 && toggle(7)==1) SimKey.keyRelease(KeyEvent.VK_U); toggle(7)=1-toggle(7); end

if(bwimg1(centroids(8,2),centroids(8,1))==0 && toggle(8)==0) % disp('eighth'); SimKey.keyPress(KeyEvent.VK_I); toggle(8)=1-toggle(8); elseif(bwimg1(centroids(8,2),centroids(8,1))==1 && toggle(8)==1) SimKey.keyRelease(KeyEvent.VK_I); toggle(8)=1-toggle(8); end end stop(vid);

--------------------------------Enjoying Playing Presented by: Rahul Kumar Btech. 2nd year Electrical Engg.