I’ve been pretty impressed with some of the stuff ARKit can do, as well as the Microsoft Surface demos.
I decided that we need to do that on the web, too, so I build a virtual keypad, using HTML5 video and a webcam.
In HTML5, a common ( example 1, example 2) pattern is to take an image from a video, copy it to a canvas and manipulate it here using Javascript. That’s the general pattern I’ve used.
I use a colored marker (setup in a calibration phase) and track its location. In order to take account of brightness changes, etc, I do cosine similarity tests on average color vectors for each “key” on the keypad using the Sylvester Javascript vector library.
I think there’s a fair bit of potential for video based interfaces. I’d like to see a combination of this and the HTML5 drumkit, for example.
You can try it out here: http://demo.nicklothian.com/keypad/. You’ll need to be using Firefox 3.5+.
Here’s a demo video (ironically as an embedded flash video):