This sketch recognition program could kick your ass at Pictionary

Sketching is an incredibly efficient way of creating a very rudimentary graphical depiction of our visual world. Most people are pretty good at figuring out what a sketch is trying to represent — but computers are absolutely awful at it. At least, until now.

Researchers at TU Berlin and Brown University have developed a sketch recognition program that utilizes the powers of crowd-sourcing to figure out what the heck it was that you just drew.

In what was the first large scale investigation into human sketch recognition, researchers Mathias Eitz, James Hays, and Marc Alexa analyzed the distribution of non-expert sketches of common objects like cars, teapots, and bunny rabbits. To do so, they recruited an army of volunteers who produced over 20,000 individual sketches across 250 different object categories.

Illustration for article titled This sketch recognition program could kick your ass at Pictionary

Using this dataset, the researchers concluded that humans have about a 73% proficiency at identifying the object category of a sketch.

Then, in an effort to get their computers to do the same, the researchers worked under the assumption that straight-ahead object recognition programs weren't going to work. And indeed, comparing sketched objects to real-world objects has proven useless; humans, when sketching, use shared, iconic representations of objects (like stick figures or simplified houses), or they make dramatic simplifications or exaggerations (like long ears on rabbits). Consequently, any successful computer program would have to refer to these sorts of depictions rather than the real thing.

To give their computers this same sort of recognition capacity, the researchers used multi-class support vector machines that were trained on the large sketch dataset. The result was a computer program that was successful at identifying object categories 56% of the time. The result was not as good as a human's, but still very good given that chance is 0.4%.


Looking ahead, the researchers are hoping to develop more sophisticated object recognition software applications. In addition, they want to compare sketches from around to the world to conduct a comparative analysis. More conceptually, the researchers are also considering the potential for sketching as an alternative way to communicate with computers — a development that could allow very young children, the illiterate, or someone who speaks a different language to work with computers.

You can read the entire paper here (pdf).


Share This Story

Get our newsletter


SIGGRAPH is a very important convention...I'd suggest the io9 editors pay more attention to it, there are some fantastic things presented every year...from stuff like this (crazy cutting edge vision and math) to the big VFX presentations from all the studios...Avengers etc

plus the parties are great!