The task was to create a non-keyboard/mouse interface to control 'something'. I chose to look into colour tracking and ended up with this application that recognizes set colours and creates a virtual key-press depending on which colour has been found.
For our experiential module we were set the project of creating a non keyboard/mouse interface. I started by thinking about different interfaces that are already around; motion, voice, touch. After having a few sessions in using arduino chip boards, we were shown the possibility of connecting any hardware up to the computer all controlled with Processing. After some experimentation with Processing I started looking more into working with live video capture rather than using arduino based hardware.
My code loads the image from the webcam and then loads the image as a pixel array. This pixel array stores the RGB data for each pixel in a large three dimentional array. This then allows the code to run algorithms which loop through each and every pixel in each frame of the video and compare the colour data to the preset colours which are being tracked. Due to this process scanning each pixel in every frame of the video, the frame rate drops right down especially for larger video resolutions.
After testing out the program, it managed to find colour in most images unless the threshold was very low. As each pixel is scanned in the image, even on a white wall, individual pixels may pick up as a colour. Due to this I made a piece of code that takes a variable (video scale) and then essentially pixelates the image into blocks of this size. In this example image the video scale is set at 10, which means that any image captured will be reduced to a pixel array with 1oo times fewer pixels. This makes the colour detection a lot more reliable, as well as meaning the code runs much faster as seen in the output rate.
The interface menu allows you to change the threshold value for each individual colour. This allows you to set how close a colour needs to be before it counts as a detection. There are also controls which allow you to alter how long the key press will last. You can set it up to either press a key once each time a colour is detected with varying intervals, or mimic a key being continuously held down while a colour is being detected. Each of the colours can be assigned to any key on the keyboard. By default it is set to the directional keys.
During the D&AD New Blood show in London 2014, this project was awarded a best in show award. This is a great achievement in itself which also led me to getting one of the 10 wildcard entries into the D&AD academy. However I was sadly unable to attend due to the short notice. The interest in this project has made me think of new potential uses for this concept and I am planning to try and adapt the code so that I can run this online within the browser.