Brain-Computer Interface Enables Coordinated Speech and Movement

ADVANCED brain-computer interface systems enable adults with paralysis to produce simultaneous speech and expressive body gestures. The clinical milestone demonstrates that a single cortical implant can coordinate multiple functional outputs concurrently, addressing a long-standing barrier in neurorehabilitation. While conventional brain-computer interface systems decode isolated speech or single-limb movements, human communication relies heavily on natural cospeech gestures to convey complete intent.

Restoring Multi-Effector Control With a Brain-Computer Interface

Investigators evaluated neural dynamics in three individuals presenting with severe vocal tract and limb paralysis resulting from brainstem stroke or amyotrophic lateral sclerosis. Each patient received a high-density subdural electrocorticography grid comprising 253 electrodes placed across the sensorimotor cortex. High-gamma cortical activity confirmed that speech articulators and upper-limb gestures recruit distinct yet partially overlapping neural populations within the precentral gyrus.

This spatial overlap initially created functional challenges. Neural decoders trained exclusively on isolated behaviors exhibited substantial performance drops during concurrent attempts, frequently misclassifying simultaneous actions as rest or generating false activations. Cortical ensembles altered their tuning dynamics during simultaneous execution, showing that concurrent motor behaviors produce unique population-level patterns rather than simple linear additions of isolated signals.

Engineering Robust Real-Time Avatars

To resolve this signal interference, researchers implemented context-inclusive model training that integrated both isolated and simultaneous motor trials. Additionally, they incorporated cross-modality negative sampling, training the speech decoder to recognize gesture-only activity as rest and vice versa. This training paradigm eliminated false positive activations across opposing modalities while preserving high classification accuracy.

When deployed in real-time conversational tasks, the parallel architecture translated cortical activity into coordinated outputs on a personalized full-body virtual avatar. In participant testing, the system achieved conversational decoding accuracies of 75.0% for speech and 85.0% for gestures, maintaining a speech false positive rate of 0.0%. Offline leave-one-out evaluations confirmed that decoders generalized effectively to previously unobserved phrase and gesture combinations. These results provide an adaptable brain-computer interface framework for comprehensive neuroprosthetics in severe paralysis.

Reference

Brosler SC et al. Simultaneous speech and gesture decoding for multimodal communication in paralysis. Nat Neurosci. 2026;29:2445-2455.