The guys from Cornell designed an interesting system that allows playing music on a paper with only fingers. The idea is simple but another talk with implementation. They took a camera to capture area where playing is performed. Then each area is divided in sectors with different instruments like keyboard, drum. FPGA has to detect finger by recognizing human skin and then determine its position.

As they say – the most challenging part of the project was detecting human skin. They escaped with pretty simple solution where normalized RGB values are analyzed. In order to get piano like music sound they used Karplus-Stron algorithm which models the behavior of string vibration. For drum signal they modeled a wave propagation on a mesh surface. They put all system in to two FPGA DE2 boards where one takes care of detecting fingers and generates piano sounds while another takes care of drum sound.