After multiple attempts, the setup successfully translated signals from his brain to the vocal tract in the form of words that appear as text on a screen.
Researchers at the University of California, San Francisco (UCSF) have come up with a unique neuroprosthetic device that has finally restored a paralyzed man’s speech.
After suffering severe brain stem stroke, the patient had been rendered speechless. But thanks to researchers at USCF, led by Dr. Edward Chang working alongside Dr. Karunesh Ganguly, he was able to communicate again in complete sentences.
Over the course of 48 sessions, where over 22 hours of cortical activity was monitored, the 36-year-old man, identified only as “BRAVO-1”, attempted to say individual words from a vocabulary set of 50 words. After multiple attempts, the setup successfully translated signals from his brain to the vocal tract in the form of words that appear as text on a screen, using a mix of artificial intelligence (AI) algorithms and natural language models.
This revolutionary study has been published in the July issue of the New England Journal of Medicine (NEJM).
What is Neuroprosthesis?
Neuroprosthetics or neuroprostheses refer to a variety of artificial devices or systems that can be used to help people suffering from severe, debilitating brain injuries or diseases like Parkinson’s disease, multiple sclerosis and amyotrophic lateral sclerosis (ALS, or Lou Gehrig’s disease). The technology can help restore their autonomy, at least partially, and enable them to continue with daily life activities. These devices or systems may include assistive technology, functional electrical stimulation, myoelectric prostheses, robotics, virtual reality gaming and brain stimulation.
The basic principle behind the technology is to use artificial methods to enhance the motor, sensory, cognitive, visual, auditory and communicative deficits caused by acquired brain injuries. Neuromodulation is achieved by using extracranial stimulation devices that generate electric or magnetic current to activate specific parts of the brain, or implanted devices such as brain-computer interfaces and deep brain stimulators.
Laying the Groundwork
USCSF had previously created a state-of-the-art brain-machine interface that could generate synthetic speech, sounding as natural as the person’s own. The interface was created using brain activity to control an anatomically detailed computer simulation of the vocal tract, including the lips, jaw, tongue and larynx, thus allowing it to map the brain’s speech center and help to generate artificial speech.
“For the first time, this study demonstrates that we can generate entire spoken sentences based on an individual’s brain activity. This is an exhilarating proof of principle that with technology that is already within reach, we should be able to build a device that is clinically viable in patients with speech loss,” Chang said.
There are still kinks to work out, however. The brain regions mapped through this technique did not directly represent the acoustic properties of speech sounds, but rather provided instructions for movements of the mouth and throat during speech.
An iteration of this study was conducted by Gopala Anumanchipalli, PhD, a speech scientist, and Josh Chartier, a bioengineering graduate student in the Chang lab. Electrodes were temporarily fit within the brains of five patients at the UCSF Epilepsy Center, all of whom had intact speech.
The electrodes in their brains were originally used to map the source of their seizures in preparation for neurosurgery. These patients were told to read several hundred sentences aloud while the researchers used the electrodes to record and map the activity of the brain center known to be involved in language production.
The next step was to use these audio recordings, along with linguistic principles, to reverse engineer the vocal tract movements (like pressing the lips together here, tightening vocal cords there, shifting and relaxing the tip of the tongue to the roof of the mouth, etc.)
By mapping the brain and vocal cord activity of each participant, researchers developed two neural network-based machine learning algorithms: a decoder to transform brain activity patterns produced during speech into movements of the virtual vocal tract, and a synthesizer to convert these vocal tract movements into a synthetic approximation of the participant’s voice.
The sentences produced using these algorithms were understandable to hundreds of human listeners in crowdsourced transcription tests conducted on the Amazon Mechanical Turk platform. This was a marked improvement from a previous attempt to decode speech only by mapping brain signals without decoding the brain activities and reverse engineering vocal tract movements.
The Present Study and Implications
BRAVO-1 suffered brain-stem stroke more than 15 years ago and has been living with anarthria (the loss of the ability to articulate speech) and spastic quadriparesis (a severe form of speech paralysis) ever since. He has had extremely limited head, neck and limb movements, and communicated by using a pointer attached to a baseball cap to poke letters on a screen.
BRAVO-1 worked with the researchers to create a 50-word vocabulary, which was identified by Dr. Chang’s team from brain activity using advanced computer algorithms. The vocabulary included words like ‘water’, ‘family’ and ‘good’ which underwent multiple permutations and combinations to create more than 1000 sentences expressing concepts applicable to BRAVO-1’s daily life.
Dr. Chang surgically implanted a high-density electrode array over BRAVO-1’s speech motor cortex, which is responsible for speech production. In each session, BRAVO-1 attempted to say each of the 50 vocabulary words many times while the electrodes recorded brain signals from his speech cortex.
To translate the patterns of recorded neural activity into specific intended words, the study leaders used custom neural network models, which are forms of artificial intelligence algorithms mimicking human neurons. When BRAVO-1 attempted to speak, these networks distinguished subtle patterns in brain activity to detect speech attempts and identify which words he was trying to say.
The approach was put to the test in multiple steps. The team first presented BRAVO-1 with sentences constructed from the 50 vocabulary words and asked him to try to repeat the same. As he made his attempts, the words were decoded from his brain activity, one by one, on a screen in real-time.
Then the team prompted him with questions such as “how are you today?” and “would you like some water?” Similar to his previous efforts, BRAVO-1’s attempted speech appeared on the screen: “I am very good,” and “no, I am not thirsty.”
The system was ultimately able to decode words from brain activity at a rate of up to 18 words per minute with up to 93% accuracy and a 75% median. An auto-correct function, similar to what is used by texting and speech recognition software, was applied to this system for further accuracy.
All in all, the team recorded 22 hours of neural activity in this brain region over 48 sessions and several months.
According to lead study author Dr. David Moses, the early trial results would be a good proof of concept. Moses shared that the team was able to detect 98% of BRAVO-1’s attempts to produce individual words and they classified words with 47.1% accuracy using cortical signals that were stable throughout the 81-week study period.
With this landmark proof-of-concept study proving to be a success, the team will now attempt to increase the size of the vocabulary and rate of speech. Neuroprosthesis has immense potential to help those with impaired speech or those suffering from complete loss of speech, to be part of regular conversations again.
“We were thrilled to see the accurate decoding of a variety of meaningful sentences,” Moses said. “We’ve shown that it is actually possible to facilitate communication in this way and that it has potential for use in conversational settings.”
Featured Jobs on BioSpace
