Stroke, brain injury, and neurological diseases rob thousands of people of their ability to speak. Scientists have now developed a technology that can decode words and sentences directly from brain waves in the cerebral cortex. This groundbreaking approach holds the potential to advance existing methods of assisted communication, improving the ability to communicate and improving autonomy and quality of life in paralyzed patients who are unable to speak.
Scientists at the University of California, San Francisco (UCSF), have applied this new technology in a man with severe paralysis to intercept signals from his brain to his vocal cords and translate these signals directly into sentences on a screen.
The findings with this novel neuroprosthetic technology, spearheaded by Edward Chang, MD, UCSF neurosurgeon, appear in an article in the New England Journal of Medicine titled, “Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria.” This clinical trial (NCT03698149) was funded by the National Institutes of Health, philanthropy, and a sponsored research agreement with Facebook Reality Labs.
“To our knowledge, this is the first successful demonstration of direct decoding of full words from the brain activity of someone who is paralyzed and cannot speak,” said Chang, the Joan and Sanford Weill chair of neurological surgery at UCSF, the Jeanne Robertson distinguished professor, and senior author on the study. “It shows strong promise to restore communication by tapping into the brain’s natural speech machinery.”
Work on communication neuroprosthetics has traditionally focused on restoring communication through spelling-based approaches to type out letters one-by-one in text. Chang’s team takes a different approach and translates signals intended for vocal muscles rather than signals to the arm or hand. Directly decoding speech signals promises quicker and more organic communication.
“With speech, we normally communicate information at a very high rate, up to 150 or 200 words per minute,” said Chang. Spelling-based approaches using typing, writing, and controlling a cursor are considerably slower and more cumbersome. “Going straight to words, as we’re doing here, has great advantages because it’s closer to how we normally speak.”
Studies on neurosurgery patients with normal speech, suffering from seizures at the UCSF Epilepsy Center who consented to have their brain recordings analyzed for speech-related activity, were largely responsible for the initial development of the technology and the current clinical trial in patients with paralysis.
Chang and his colleagues mapped the cortical activity patterns associated with vocal tract movements that produce each consonant and vowel. David Moses, PhD, a postdoctoral engineer in the Chang lab and lead author of the new study, developed new methods for real-time decoding of the patterns, incorporating statistical language models to improve the accuracy of translating cortical activity patterns into the ability to recognize complete words.
However, applying the method in patients with paralysis of the vocal tract presented a whole new set of challenges. “Our models needed to learn the mapping between complex brain activity patterns and intended speech,” said Moses. “That poses a major challenge when the participant can’t speak.”
Whether the brain generated signals to the vocal tract when the vocal muscles have been paralyzed for years was uncertain. Chang partnered with colleague Karunesh Ganguly, MD, PhD, an associate professor of neurology, to launch a study known as “BRAVO” (Brain-Computer Interface Restoration of Arm and Voice) to explore the potential of this technology in patients with paralysis.
A man in his late 30s who suffered a brainstem stroke more than 15 years ago that severely damaged the connection between his brain and his vocal tract and limbs is the first participant in the trial. The patient, with severely limited head, neck, and limb movements, communicates by using a pointer attached to a baseball cap to point at letters on a screen.
The volunteer patient who asks to be called “BRAVO1” helped researchers create a 50-word vocabulary that the team could identify in the brain activity using advanced computer algorithms. Using words such as “water,” “family,” and “good,” BRAVO1 created hundreds of sentences expressing concepts relevant to his daily life. Chang surgically implanted a high-density electrode array over BRAVO1’s speech motor cortex. After recovering from the surgery, Chang’s team recorded 22 hours of neural activity over 48 sessions and many months. During each session, BRAVO1 attempted to say each of the 50 vocabulary words many times while the electrodes recorded brain signals from his speech cortex.
Co-lead authors, Sean Metzger and Jessie Liu, bioengineering graduate students in the Chang Lab, used custom neural network models to translate the patterns of recorded neural activity into the specific words. The neural network was able to identify subtle changes in patterns in brain activity to detect speech attempts and identify each word BRAVO1 was trying to say.
The researchers first asked BRAVO1 to repeat short sentences constructed from the 50 vocabulary words to test the approach. They then asked him questions such as “How are you today?” and “Would you like some water?” BRAVO1’s attempted speech appeared on the screen: “I am very good,” and “No, I am not thirsty.”
The team measured that the system was able to decode words from brain activity at a rate of up to 18 words per minute with up to 93% accuracy. An “auto-correct” function, similar to that used in commercially available speech recognition software, contributed to the success of the technology.
“We were thrilled to see the accurate decoding of a variety of meaningful sentences,” said Moses. “We’ve shown that it is actually possible to facilitate communication in this way and that it has potential for use in conversational settings.”
Chang and Moses will expand the trial to include more participants affected by severe paralysis and communication deficits. The team is currently working to increase the vocabulary and the rate of speech.
“This is an important technological milestone for a person who cannot communicate naturally,” said Moses, “and it demonstrates the potential for this approach to give a voice to people with severe paralysis and speech loss.”
