Vote 2020 graphic
Everything you need to know about and expect during
the most important election of our lifetimes

Spies want to mine your tweets for signs of the next tsunami

Illustration for article titled Spies want to mine your tweets for signs of the next tsunami

The intelligence community has seen the future, and the future is Google Trends. Actually, more like a highly sophisticated version of Google Trends, with Twitter and YouTube thrown in for good measure.


Iarpa, the blue-sky research arm of the intelligence community, recently announced a new program that aims to monitor, collect and analyze publicly available data to predict future events. The Open Source Indicator Program would be so sensitive to changes in the zeitgeist that it could "beat the news," anticipating "political crises, disease outbreaks, economic instability, resource shortages and natural disasters" - to name a few.

The idea that web search trends, blogs, internet traffic and webcams could hint at the future is nothing new. Google Flu Trends was set up in 2009 to predict flu outbreaks, counting on the fact that hordes of people searching for "flu symptoms" might mean something. In some cases Google was faster than the government at identifying where the next outbreak would occur.


Last month, the Bank of England announced that it was using internet search data to understand economic trends. They found that searches for "estate agents" closely tracked house price inflation.

So it's no surprise the intelligence community has over the last few years become more interested in open source intelligence, or information that's freely available if you could just sort through the noise. Just recently, a West Point instructor teamed up with a Princeton researcher to investigate if Google search data could help analyze the situation in Egypt.

"Open source provides a critical lens, a critical opening into understanding the world around us in a much more dynamic way than traditional intelligence sources can provide," Lieutenant Colonel Reid Sawyer, the head of West Point's Combating Terrorism Center, told NPR.

Some of the latest investments in this more "open" direction include Visible Technologies, which scours the social web for posts and conversations on everything from Amazon to Flickr. Another Iarpa project, ALADDIN, would scan thousands of open source video clips and let you search for specific events of interest. Recorded Future taps thousands of websites, blogs and news alerts to somehow algorithmically calculate who knows whom, what's getting attention and what will happen next.


Of course, gathering data and making educated predictions is one thing -– having those predictions become reality is quite another. Recorded Future's prediction that Hosni Mubarak would flee to Saudi Arabia didn't exactly pan out. There are many events -– 9/11, Fukushima, Osama bin Laden's death -– that were unlikely to have popped up on a Google search, even one that underwent intense statistical analysis. As much as the internet can provide answers, without the right questions, it remains a flood of information. And there may be some questions we will never know to ask.

So the first step for Iarpa's most recent program is to try to get a grip on all that information. Other current technologies have their specialties, whether it's social media, search trends or YouTube videos. This program hopes to feed thousands of webcam streams, Twitter feeds, internet traffic numbers and search terms into a single pipeline, and have some semblance of sense come out the other side. Actually, more than just sense: a prediction about the future.


In other words, get every open source piece of information you can, and use it to predict every kind of outcome you can. The deadline for submitting an idea is July 27. But you probably predicted that already.

Photo: Lena Groeger

Illustration for article titled Spies want to mine your tweets for signs of the next tsunami

This post originally appeared on Wired's Danger Room. has been expanding the hive mind with technology, science and geek culture news since 1995.

Share This Story

Get our newsletter


This will work great for about 6 months, then someone will invent a spambot loaded with keywords that will trip the program on a false positive.