Neuroscientists have developed a way to listen to words you've heard, by translating brain activity directly into sound. Their findings represent a major step towards understanding how our brains make sense of speech, and are paving the way for brain implants that could one day translate your inner thoughts into audible sentences.

Every language on Earth is made up of distinct acoustic features. The volume or rate at which syllables are uttered, for example, allow our minds to make sense out of speech. How the brain identifies these features and translates them into relevant information, however, remains poorly understood.

UC Berkeley researcher Brian Pasley and his colleagues wanted to see what features of human speech, if any, could be reconstructed by monitoring brain activity.


Neuroscientists call this form of brain analysis — which is commonly construed as mind-reading — "decoding." If this study sounds familiar to you, it might be because last year, another team of scientists was able to decode images observed by volunteers by monitoring activity in the brain's primary visual cortex. What Pasley's team was trying to accomplish was quite similar, only they wanted to translate their volunteers' brain activity into auditory information. This, in turn, would require looking at a different region of the brain.

But that's not the only thing Pasley's team did differently. The scientists who last year reconstructed visual information used a popular brain-scanning method called functional magnetic resonance imaging (fMRI). And while fMRI is an incredibly useful way to monitor brain activation, it isn't actually the most direct method out there. Pasley's team wanted to get as close to their volunteer's brain waves as possible.

By seeking out patients already scheduled to undergo brain surgery, the researchers were able to place electrode nets directly onto the brains of 15 conscious volunteers (similar to the setup seen here) over a region called the posterior superior temporal gyrus (pSTG), which is believed to play a crucial role in speech comprehension. The volunteers then listened to a series of pre-recorded words for five to ten minutes, while their brain activity was recorded via the electrode nets. [Intracranial electrodes via]

Pasley then created two computational models that could convert the electrode readings back into sound. This allowed the researchers to predict what word the volunteer had been listening to when the brain activity was recorded.

You can listen to some examples of the recordings here. The first word you'll hear is "waldo" — it's the version the volunteers heard as they were having their brain activity monitored. The next two sounds you'll hear are the versions of "waldo" that were reconstructed using the researchers' two different algorithms. This process is then repeated for the words "structure," "town," "doubt," "property," and "pencil." For each word, the true sound will play first, followed by the versions that have been reconstructed from brain activity. [The image up top features spectrograms that were created to compare the accuracy of the six reconstructions you've heard here to their original sounds]

Pasley and his team reconstructed 47 words in total. Using these reconstructions, he and his colleagues were able to correctly identify the words that their volunteers had listened to almost ninety percent of the time. Of course, the researchers also knew which words to listen for — but the fact that they could reconstruct something from brain waves at all is very impressive.

The ability to convert brain activity into usable information — be it audio or imagery — has a long, long way to go before we're reading one another's thoughts, but its potential applications have scientists racing to make it happen; and that's because these applications are as inspiring as they are unsettling.

In its present state, this technology cannot eavesdrop on an internal monologue playing out in your head; it can't be used to squeeze information out of an uncooperative murder witness; and it can't translate the thoughts of a stroke patient struggling to speak — but it could, and soon. How soon will likely depend on how similarly the brain handles the tasks of perceiving auditory information, and imagining it.

"Potentially, the technique could be used to develop an implantable prosthetic device to aid speaking, and for some patients that would be wonderful," said Robert Knight — a senior member of Pasley's team — in an interview with The Guardian. "The next step is to test whether we can decode a word when a person imagines it."

"That might sound spooky, Knight says, "but this could really help patients. Perhaps in 10 years it will be as common as grandmother getting a new hip."

The researchers' findings are published in the latest issue of PLoS Biology.
All images, audio via Pasley et al unless otherwise indicated