Decoding Speech from Electrocorticography

2021 Seed Grant
Joseph G. Makin, Ph.D.
Purdue University

Losing the ability to speak can be devastating. Sometimes, though, people lose this ability without losing the ability to plan the proper mouth movements for speaking. For example, a stroke, or Lou Gehrig’s disease, can spare the parts of the brain responsible for issuing the right “commands” to the lips, jaws, and tongue, while damaging the parts of the nervous system through which those commands would normally travel. In these cases, it’s theoretically possible to restore speech by monitoring the brain cells associated with intended speech, and then using an algorithm that translates those patterns of brain activity directly into the words the person is trying to say.

However, to do that, an electronic monitoring device needs to be implanted on the surface of the brain, under the skull. Fortunately for speech research, some people already get these devices temporarily implanted: patients who are undergoing treatment for epilepsy. In recent work, I developed and implemented a system that translates this brain activity into speech with nearly perfect accuracy–but only when the participant was restricted to speaking just 50 predetermined sentences, using a total vocabulary of 250 words.

Now our lab is proposing to scale this speech decoding up to allow the participant to form any sentence he or she chooses, from a larger vocabulary of 1000 words. To do this, we plan to record brain activity from a group of epilepsy patients for several hours, while they speak sentences from a new set of sentences that are optimized for flexibility. Then we will develop and apply an improved speech-decoding algorithm, incorporating recent advances in “machine translation,” the technology behind (e.g.) Google Translate.

Other Grants

Lindsay M. De Biase, Ph.D., University of California Los Angeles
The Role of Microglial Lysosomes in Selective Neuronal Vulnerability
Synapses, the sites of signaling between neurons in the brain, play essential roles in learning, memory, and the health of neurons themselves. An enduring mystery is why some neurons are…
How the Nervous System Constructs Internal Models of the External World
As animals navigate their environments, they construct internal models of the external sensory world and use these models to guide their behavior. This ability to incorporate ongoing sensory stimuli into…
Xiaojing Gao, Ph.D., Stanford University
When Neural Circuits Meet Molecular Circuits: Quantitative Genetic Manipulation with Single-cell Consistency
Cells are the building blocks of our bodies. We get sick when the cells “misbehave”. The way modern gene therapies work is to introduce genes, fragments of DNA molecules that…
Rafiq Huda, Ph.D., Rutgers University
Conducting the Orchestra of Movement—Functional Role of Striatal Astrocytes in Health and Disease
Movement requires coordinated activity across a large brain-wide network. The striatum is a particularly important part of this circuit; it integrates motor-related information from many distinct brain regions to regulate…