refract.me

“What is Computer Science,” a new research competition for my university’s ACM student chapter gave me an excuse to try Adam Kumpf‘s Trackmate system. Trackmate is a computer vision based tangible interface; it relies on different visual tags, affixed to items, to input to a computer.

I am interested in tangible interfaces. In addition, I honestly feel that diverse interactions with a machine is the heart of computer science. So, I began to work, dabbling in the language that Trackmate was designed for use with, Processing (a Java-derived language). I looked into Minim, a Processing library related to sound. I wanted to create something both visual and auditory to play with.

… I admit that I do not understand the mathematics necessary for the more creative work others have done in Processing.

Rather than speak directly to the Processing code I was developing, Trackmate goes through LusidOSC, an abstraction level between spatial interfaces and Processing. (LusidOSC was also developed by Kumpf.) LusidOSC does have a package of programs to get users started.

I then looked to Wikipedia for public domain or free and transformable licensed audio tracks. I decided on a series of percussion instruments, as they are abundent and mix well naturally.

I then sat down to create my own code. Some of the difficulties included differentiating the number 0 and the letter O in the Processing IDE (Integrated Development Environment). There is a LusidOSC simulator, which I used during the development.

Once my design was complete, I decided to build my Trackmate interface. That proved more difficult. First, I’m a Linux user and, at the time, there was no Linux support; I ended up borrowing a department Windows laptop to do my work.

I started with the PlayStation Eye camera, as recommended. I managed to get that working with the computer, after several downloads and attempts. Then, came trying to get it working with Trackmate; rather than print an error, the GUI (graphic user interface) became a static set of bars. I assumed it was my fault and reinstalled the camera. Repeatedly. I did eventually try another camera, which worked on the first try.

I then set up the system as per instructions, having to tweak the camera to work with my greater surface area (9″ x 7″). To get this to work beneath the fluorescent bulbs though, I was restricted to use blocks smaller than the tags: allowing overhead light to mark the edges of the circles. There was also a strong light bulb reflection on the acrylic surface I used; an area became a dead space to the camera. Best of all, the camera would see up to an inch above my surface.

Alright, I became quite frustrated, so I decided to look at the source code. LusidOSC is fairly simple looking: written in Java, many methods were mere stubs. Trackmate was a different matter all together. First, it’s written in C++, which seems strange when it goes into a Java program. But, it gets better. Much of it is hardcoded, needlessly repeated. It relies on OpenFrameworks, a library still in pre-release, and the CodeBlocks IDE. I tried to alter the code, but it was a lost cause; the instructions for using CodeBlocks included nonsensical statements like “Install” without an object.

… For “What is Computer Science,” there was a 1.5-2 hours of judging. This required having the computer running my program for quite a while. The laptop became quite hot; the CPU began to slow as the program progressed.

As for my program itself, I link the first tag to the frame rate, controlled by rotation. The other tags are each linked to an audio clip, which plays repeatedly if the tag is seen. A random color and sized circle indicates the location of these tags on the GUI.