Draw With Your Face

Development BLOG - about

Draw With Your Face is a game for Google+ Hangouts created by Aaron Meyers and OKFocus. Some of the mechanics will be familiar to players of other drawing games: players take turns trying to get other players to guess a word with a drawing.

As the name suggests, Draw With Your Face adds a new twist: using face tracking, players create their drawings hands-free using just their face! Its fast-paced and funny. And fun!

Draw With Your Face is a part of Google’s Hangouts Hackathon.

a Google+ Hangouts API experiment by Aaron Meyers & OKFocus

    6 / 07
  1. We’re excited to share Draw With Your Face with the world today. We created a video to show you a typical Draw With Your Face use case scenario. Once you check out the video, jump over to drawwithyourface.com and get a hangout started and invite some friends.

    Open up your mouth wide and say “AAAAAAHHHHHH” and you’ll be drawing in no time! Be sure to share your favorite drawings with the G+ button in the game and send them to us here.

  2. 5 / 30
  3. For the toolbar we used some snazzy CSS to create a 3D like quality on hover and active states. Checkout a static version here - http://labs.okfoc.us/draw_with_your_face_toolbar/

    The following CSS snippet is truly a godsend, it allows for width and height to remain true, not resized with padding or borders. More on this from Paul Irish

    The HTML is as follows :

    The CSS is as follows :

  4. 5 / 30
  5. Designing UI can not be done in a vacuum. At OKFocus we try to arrive at good UI though a process of deduction achieved from careful observation and the sharing of opinions. Good design is functional, as is culture. When designing a website like Draw With Your Face, an application which introduces an entirely new gesture, you have less to look towards than say, designing a restaurant website. When an application’s function has less historical precedent, it’s necessary to understand the nature of each UI element on a fundamental level. We looked towards the semiotics of past examples for each element. For instance - in designing a score board we looked towards the design in the game of darts and for designing a toolset we looked towards MacPaint. But our challenge ultimately was a new one - how to get people to learn how to draw with their face, and be excited about it. Through the marriage of function and aesthetics, social context is defined, with that in mind we try to design in the most direct way to ensure usability – new function promulgates new design. 

    Ryder Ripps@RYDER_RIPPS

  6. 5 / 29
  7. Jonny working hard on Memorial Day.

  8. 5 / 25
  9. Saving Images with HTML5 Canvas and App Engine’s Blobstore API

    So while we’re busy making the best hands-free, sound-activated drawing tool ever we figured saving your creations to share with your friends would be pretty important. I took some time yesterday morning to set our App Engine instance up with some image saving capabilites which I’ll share in this post. We’re doing some cool stuff here like sending an encoded image from an HTML5 canvas cross-domain and then saving it to the App Engine Blobstore (an experimental feature).

    Let’s start with our face drawing tool. The high-level mechanics of the tool are quite simple: we take data from the Hangouts API about face position and apply that data to a HTML5 ‹canvas› element about five times a second. You can draw to your heart’s content but that data will never get saved unless we do something about it. So here’s what we did about it: Once a drawing is complete, we stop the canvas from updating and save it with the native toDataUrl() method. Finally, we send it off to our Google App Engine instance for safe keeping.

    The below JavaScript is pretty simple. We’re just grabbing a canvas element, encoding it to a base64 PNG, and then sending that off to a server with the help of jQuery.

    What’s happening on our App Engine instance is a little more interesting. We use the Blobstore API, which allows us to create and serve data objects, called blobs. The Blobstore creates a blob from a file’s contents and returns a reference to the blob, called a blob_key, which can be used to serve the blob. Check out the UploadHandler class in the code snippet below, this is where everything gets blobby.

    You’ll note that we’ve set cross origin resource sharing header to accept data from wherever. This is potentially hazardous, although is suitable for our testing in development. On line 27 to 29 you’ll see we take the “imgdata” parameter from our JavaScript post request, parse it so we grab just the image data, and finally decode the encoded string.

    The next few lines (32 to 42) are where all the Blobstore API stuff happens. On line 32 we create a PNG and then in the following “with” statement we open the Blob that was just created and pipe in our decode image data. When that is complete, we finalize the file. I don’t really know what is happening here at a lower level, although I can take a guess. My guess is that a Google engineer takes a look at my code and laughs at my Python literacy with his buddies. After the file has been finalized we are safe to get the file’s blob key and save it to our App Engine datastore. We do this by creating a new Drawing object with our image’s blob key. In the very last bit of code in this method (lines 45 and 46) we simply respond to our JavaScript with the location of the image.

    I hope this sheds some light on how to use base64 data from a canvas and save it to an App Engine instance. All in all, I found the Blobstore API pretty easy to work with once I understood what was going on conceptually and technically. Its flexibility is pretty cool and could address a bunch of usecases, including drawing with your face.

    Jonathan Vingiano@jgv

  10. 5 / 24
  11. Local Development Environment, Cont’d

    In this post I describe our current Hangouts local development environment. It is set up to have multiple views, with JS and CSS loaded dynamically from a configurable hostname.

    Last week, Jonathan blogged about how we got around restrictions in the development version of App Engine that keep you from serving over HTTPS on localhost. To save us from having to deploy to App Engine each time, and also to keep from clobbering our App Engine remote, we instead deploy a small bootstrap and publish everything else locally. As Jonathan mentioned, we have gotten used to various Rails naming conventions, so we’ve tried to replicate some of these for our own convenience, as well as break our app apart so we can serve multiple views and edit them separately.

    We made a directory for serving HTML views from localhost, which could then be pushed directly into a div with the JQuery convenience function $("#div").load(url). We quickly discovered that while App Engine lets you set permissive CORS headers for your APIs, it is impossible to set similar headers for a static_files URL handler (as defined in app.yaml).

    The Python instance has some limited access to the filesystem, so instead we built a trivial API to slurp a file and dump it back at the user. Here, another restriction: the Python instance does not have access to any files in a static_files directory. Thus our solution is to make a directory called “views” in the same directory as main.py and serve our static HTML from there, which is sufficient for development.


    To use this new API, we made a small Javascript bootstrap that waits for the Hangouts API to load, caches the current hostname and view in localStorage, sets up some buttons so we can switch from view to view, binds to an input box (defined in our app.yml) for changing the hostname, and ultimately loads a view into the container.


    You may note the “javascripts” and “stylesheets” functions at the end — Each view loads its own javascript dynamically so we can write individual test environments for different features, and relies on a globally-defined serverPath so that content can be loaded in from a particular host. Acceptable values are “production” for your production App Engine server (defined at the top), “localhost” for a code repository local to each individual user, or a bare IP address. As with the current view, the current host is cached in localStorage so these things don’t have to be set with each app reload. There are a few reasons this might be useful — people who do not have the codebase can demo in a hangout with you, features may be iterated in a single hangout without everyone having to pull from git for minor changes, your local hangout may be swapped to run code from production if you rely on the server side, and so on.

    Finally, here is a brief test script that incorporates the Hangouts simple example (state changes, AppEngine API test) with Rehab Studio’s ping test (message send & receive).



    At this point, we have moved beyond this multi-page model toward making a more conventional single-page app, but this configuration is still useful while you are in the planning stages of an app, or breaking in those features peculiar to the Hangouts API. The rudiments of game logic can be worked out in one view, stress tests in another, layout and onboarding defined in a third. Furthermore, being able to set the hostname saves you from having to synchronize codebases when pair programming to debug the multi-user components of a hangout.


  12. 5 / 23
  13. Here is how you draw (write?) with your face…


  14. 5 / 23
  15. Yesterday I worked on transforming my Photoshop mockup of Draw With Your Face into CSS. Used a lot of utility classes for just about everything, which helps a lot for having less bloated CSS. I combine IDs and classes to keep element unique styles separate from those global - I find this really helps organize/structure a page. Stuff like this -

    Design is inspired by the Mac System 7 Calculator 

    The original font I wanted to use is owned by Adobe, but I found an alternative font in Google Webfonts. http://www.google.com/webfonts

    Ryder Ripps@RYDER_RIPPS

  16. 5 / 21
  17. The Hangouts API gives you the ability to track a handful of facial features. For the drawing tool in Draw With Your Face, players move the tip of the brush using the center of their mouth. We wanted to keep the core drawing experience hands-free so finding a responsive way to begin and end a stroke was an important problem for us to solve.

    The face tracker provides coordinates for upper and lower lip so our initial idea was to use the opening and closing of the mouth to start and finish a stroke. As a bonus, we thought we might control the stroke width with the widening of the mouth. Alas, in practice the lip position tracking didn’t give us the kind of responsiveness that we needed to get this to feel right so it was back to the drawing board (and API documentation pages).

    As I scrolled down the documentation page, the getParticipantVolume() function leapt out at me. Using the sound level from the microphone seemed like it just might be the perfect hands-free solution! It only took a moment to get it up and running and soon we were all gathered around the computer, making weird noises and drawing with our faces.

    Playing around and drawing, I was reminded of a section of Golan Levin and Zach Lieberman’s Messe di Voce where the two performers use their voices to draw on a projected canvas behind them. In Messe di Voce, the positioning of the brush has a relationship with the pitch of the performers voice while in Draw With Your Face, we give you direct control of the brush position. But somehow we’ve found so far that while drawing, there is a natural inclination to have your pitch ascend or descend as you move around and complete strokes.

    Expect a video of us drawing soon!


  18. 5 / 18
  19. Geting Up and Running With
    Google+ Hangouts API

    Optimizing for Developer Happiness

    At OKFocus, we think a lot about optimizing for developer happiness. It’s important to us that when we work we have the flexibility to experiment with different ideas without having to worry about the project at hand going up in flames or something terrible happening to the codebase. Since this is at the core of our philosophy around creating digital experiences, all of our code is tracked using Git. For those who aren’t familiar with Git, it is a very popular version control system that allows developers to work together without having to worry about stepping on each other’s toes. Furthermore, it’s important that developers can have the freedom to use their own tools (everyone on this project uses a different text editor) and work locally on their own machines.

    Our workflow goes something like this:

    • Write some code
    • Commit the code to the Git repository
    • Push the code to Github
    • All team members are notified in HipChat
    • Team pulls changes
    • Repeat!

    If you want a closer look at the tools we use at OKFocus, check out this project on Github.

    Working with the Google+ API and the Hangouts API is new for us. Going into this project we knew we wanted to bring our development methodologies with us to help us get the boring stuff out of the way and make something cool. Even better, we didn’t want to have to deploy to a server every time we committed a bit of code. Let’s look at how to make an awesome development environment that will get you hacking on a Google+ Hangout in no time.

    Hacking on a Google Hangout Locally

    To get started you will have to have the following installed:

    Most OKFocus projects happen using Rails or Express.js. Both of these frameworks have a very simliar workflow and let developers work in a test, development, or production environment. Running a local web server with both of these frameworks is quite simple. For Draw With Your Face, we knew we’d be hosting our application with Google’s App Engine which is new to us, so we created a development environment that feels similar to what we’re used to.

    Setting Up Your App Engine

    Our use of App Engine for this project is simply to serve static files, theres a good chance your Hangout App will only require this functionality from a web server. To set up an App Engine instance, log into App Engine and create a new application. Take a note of your application identifier.

    In your Hangouts API console, set your application URL to point to a path like http://[your-application-identifier].appspot.com/static/app.xml. This XML file tells your Hangouts App where to look for your app’s JavaScripts. Let’s worry about configuring it in a later section. Right now lets just configure your app.yaml for your App Engine instance to look like this:

    Note that we have set our Python runtime to use Python 2.7.x. When doing so, you must also set “threadsafe” to be either true or false. This also tells our app to serve all files in the “static” directory coupled with a little bit of Python from our main.py file.


    Once we had our App Engine instance deployed and running, we were still running into an issue where Hangouts wouldn’t let us serve our JavaScripts from localhost. It looks like almost all Google products have switched over to HTTPS (which is a good thing!); however, the development version of App Engine only allows HTTP locally. Thankfully, the open source community has a tool called stunnel that allows developers to tunnel HTTP connections through a secure layer as if the web server supported HTTPS.

    After you’ve installed and built stunnel, you can set “accept” to 8001 and “connect” to 8080 for HTTPS. You’ll find this setting at the bottom of stunnel.conf. We keep a copy of stunnel.conf in our Git repository for this project. To start our servers, we simply type:

    sudo stunnel stunnel.conf

    We run this command from the root of our project directory, referencing our customized stunnel.conf file as the first argument to the stunnel command from the command line.

    Serving the App

    Now let’s configure our Hangout App’s main XML so we are referencing our new stunnel configuration. Once everything is configured and looks okay, you can deploy to App Engine using the App Engine Launcher.

    Note that we are serving from port 8001 per our stunnel configuration. You’ll also note the link to the commented out path for production. Once we’re ready to start serving from App Engine, it just a matter of uncommenting that, removing our references to localhost, and deploying.

    Hacking Locally

    Now that your XML is referencing files that are being served from your localhost through stunnel, you are free to work locally and see changes immediately. This means you wont have to deploy to Google App Engine over and over. This also means that you can share code quickly with team members through Github and merge in changes with ease. Simply work with the JavaScript referenced in your main XML file, and visit the test hangout. You’ll see your changes immediately!

    One thing to note is that this setup allows for only one participant in a Hangout at a time. This isn’t a problem during early development stages but we will have to alter our environment next week to make multiple participants work. Look for a blog post early next week from Jules that will explore that and more.

    Happy hacking :)

    Jonathan Vingiano@jgv