The Hangouts API gives you the ability to track a handful of facial features. For the drawing tool in Draw With Your Face, players move the tip of the brush using the center of their mouth. We wanted to keep the core drawing experience hands-free so finding a responsive way to begin and end a stroke was an important problem for us to solve.
The face tracker provides coordinates for upper and lower lip so our initial idea was to use the opening and closing of the mouth to start and finish a stroke. As a bonus, we thought we might control the stroke width with the widening of the mouth. Alas, in practice the lip position tracking didn’t give us the kind of responsiveness that we needed to get this to feel right so it was back to the drawing board (and API documentation pages).
As I scrolled down the documentation page, the getParticipantVolume() function leapt out at me. Using the sound level from the microphone seemed like it just might be the perfect hands-free solution! It only took a moment to get it up and running and soon we were all gathered around the computer, making weird noises and drawing with our faces.
Playing around and drawing, I was reminded of a section of Golan Levin and Zach Lieberman’s Messe di Voce where the two performers use their voices to draw on a projected canvas behind them. In Messe di Voce, the positioning of the brush has a relationship with the pitch of the performers voice while in Draw With Your Face, we give you direct control of the brush position. But somehow we’ve found so far that while drawing, there is a natural inclination to have your pitch ascend or descend as you move around and complete strokes.
Expect a video of us drawing soon!
Aaron — @aarontweets