Scientists trained a computer to analyze the brain activity of a person listening to music and, based only on those neuronal patterns, recreate the song.
The research, published on Tuesday, did a recognizable, if muffled version of Pink Floyd’s 1979 song, “Another Brick in the Wall (Part 1).”
Before this, researchers figured out how to use brain activity to reconstruct music that has similar features to the song a person is listening to. Now, “you can actually listen to the brain and play back the music that person heard,” said Gerwin Schalk, a neuroscientist who directs a research lab in Shanghai and collected data for this study.
The researchers also found an area in the temporal lobe of the brain that reacted when the volunteers heard the 16th note of the guitar groove of the song. They suggested that this particular area may be involved in our perception of rhythm.
The findings offer a first step toward creating more expressive devices to help people who cannot speak. In the last few years, scientists have made major achievements in extracting words from the electrical signals produced by the brains of people with muscle paralysis when they try to speak.
But the vast amount of information conveyed through speech comes from what linguists call “prosodic” elements, such as tone — “the things that make us a lively speaker and not a robot,” says Dr. Schalk.
By better understanding how the brain metabolizes music, scientists hope to develop new “speech prosthetics” for people with neurological disorders that affect their voice production. . The goal is for these devices to convey not only what a person is trying to say, but to retain some of the music, rhythm and emotion of organic speech.
To collect data for the study, researchers recorded from the brains of 29 epilepsy patients at Albany Medical Center in New York State from 2009 to 2015.
As part of their epilepsy treatment, patients had a net of nail-like electrodes implanted in their brains. This created a rare opportunity for neuroscientists to record from their brain activity while they were listening to music.
The team chose the Pink Floyd song because the elderly patients liked it. “If they said, ‘I’m not going to listen to this garbage,'” then the data would be terrible, Dr. Schalk. Plus, the song features 41 seconds of lyrics and two and a half minutes of moody instrumentals, a combination useful for teasing out how the brain processes words versus melody.
Robert Knight, a neuroscientist at the University of California, Berkeley, and the leader of the team, asked one of his postdoctoral fellows, Ludovic Bellier, to try to use the data set to reconstruct the music “because he was in one hand,” said Dr. Knight. The lab has done similar work reconstructing words.
By analyzing the data from each patient, Dr. Bellier what areas of the brain light up during song and what frequencies these areas respond to.
Just as the resolution of an image depends on its number of pixels, the quality of an audio recording depends on the number of frequencies it can represent. To clearly construct “Another Brick in the Wall,” the researchers used 128 frequency bands. That meant training 128 computer models, which collectively brought attention to the song.
The researchers then ran the output from four individual brains through the model. The resulting recreations are all recognizable Pink Floyd songs but with notable differences. The patient’s electrode placement likely explains most of the variation, the researchers said, but personal characteristics, such as whether a person is a musician, are also important.
The data extracted fine patterns from individual clusters of brain cells. But the approach is also limited: Scientists can only see brain activity where doctors have placed electrodes to look for seizures. That’s part of why the songs created sound like they’re being played underwater.
Other groups are doing similar experiments using noninvasive brain scanners, such as functional magnetic resonance imaging, or fMRI, which provide less detailed measures of activity but scan the entire brain.
Yu Takagi, a neuroscientist at Osaka University, worked together this year with scientists at Google who used fMRI data to determine the genre of music a volunteer was listening to while in a brain scanner.
said Dr. Takagi said the new study is significant because it shows that meaningful data can be collected from a relatively small number of neuronal clusters. “You don’t need a lot of electrodes to make something of quality,” he said.
New research also highlights why music differs from speech. When study volunteers heard a song, the right side of their brain tended to be more involved than the left, whereas the opposite happened when people heard simple speech. This finding, which replicates previous research, helps explain why some stroke patients who cannot speak well can sing sentences clearly.
“It’s a technical tour de force,” said Robert Zatorre, a neuroscientist at McGill University whose lab established how the brain separates lyrics from music using brain scans. But to play a song back from someone’s head? “That’s a very interesting contribution,” he said.