Journal papers (peer reviewed)
28. Willett FR, Avansino DT, Hochberg LR, Henderson JM**, Shenoy KV** (2020) High-performance brain-to-text communication via imagined handwriting. Nature. In press for 12 May 2021. When published, full paper available at: https://www.nature.com/articles/s41586-021-03506-2
- Prior version as submitted to bioRxiv. pdf supp-mats.pdf supp-methods.pdf supp-video1.mp4 supp-video2.mp4 supp-video3.mp4 supp-video4.mp4 url
- Data sharing:
- Willett FR, Avansino DT, Hochberg LR, Henderson JM**, Shenoy KV** (2021) Data from: High-performance brain-to-text communication via handwriting. Dryad. Dataset.
- Code sharing:
- This GitHub repo is associated with this manuscript and dataset. The code can be used to run an offline reproduction of the main result: high-performance neural decoding of attempted handwriting movements. The jupyter notebooks included here implement all steps of the process, including labeling the neural data with HMMs, training an RNN to decode the neural data into sequences of characters, applying a language model to the RNN outputs, and summarizing the performance on held-out data.
- Results from each step are saved to disk and used in future steps. Intermediate results and models are available with the data - download these to explore certain steps without needing to run all prior ones (except for Step 3, which you'll need to run on your own because it produces ~100 GB of files).
- BCI performance, calculations and tables
- If interested in BCI performance, calculations and tables, please see BCI Performance Tables on this site.
- We aim to periodically update, edit and evolve these NHP and human performance tables, and we are very open to input, corrections and suggestions (firstname.lastname@example.org).
27. Deo DR, Rezaii P, Hochberg LR, Okamura AM, Shenoy KV*, Henderson JM* (2021) Effects of peripheral haptic feedback on Intracortical brain-computer interface control and associated sensory responses in motor cortex. IEEE Transactions on Haptics. pdf url
26. Simeral JD, Hosman T, Saab J, Flesher SN, Vilela M, Franco B, Kelemen J, Brandman DM, Ciancibello JG, Rezaii PG, Rosler DM, Shenoy KV**, Henderson JM**, Nurmikko AV, Hochberg LR (2021) Home use of a wireless intracortical brain-computer interface by individuals with tetraplegia. IEEE Transactions in Biomedical Engineering. DOI 10.1109/TBME.2021.3069119. pdf url
25. Rastogi A, Willett FR, Abreu J, Crowder DC, Murphy B, Memberg WD, Vargas-Irwin CE, Miller JP, Sweet J, Walter BL, Rezaii PG, Stavisky SD, Hochberg LR, Shenoy KV, Henderson JM, Kirsch RF, Bolu Ajiboye A (2021) The neural representation of force across grasp types in motor cortex of humans with tetraplegia. eNeuro. 10.1523/ENEURO.0231-20.2020. pdf url
24. Wilson GH*, Stavisky SD*, Willett FR, Avansino DT, Kelemen JN, Hochberg LR, Henderson JM**, Druckmann S,** Shenoy KV** (2020) Decoding spoken English phonemes from intracortical electrode arrays in dorsal precentral gyrus. Journal of Neural Engineering. 17:066007 pdf url
23. Even-Chen* N, Muratore* DG, Stavisky SD, Hochberg LR, Henderson JM, Murmann** B, Shenoy** KV (2020) Power-saving design opportunities for wireless intracortical brain-computer interfaces. Nature Biomedical Engineering. 4:984-996. pdf supp_mats url
- Editorial (2020) The painstaking pace of bioelectronic interfaces. Nature Biomedical Engineering. 4:933–934. pdf url
- News & Views: Slutzky MW (2020) Increasing power efficiency. Nature Biomedical Engineering. 4:937–938 pdf url
- Associated paper: Nason SR, Vaskov AK, Willsey MS, Welle EJ, An H, Vu PP, Bullard AJ, Nu CS, Kao JC, Shenoy KV, Jang T, Kim H-S, Blaauw D, Patil PG, Chestek CA (2020) A low-power band of neuronal spiking activity dominated by local single units improves the performance of brain-machine interfaces. Nature Biomedical Engineering. 4:973–983. pdf url
22. Willett FR*, Deo DR*, Avansino DT, Rezaii Paymon, Hochberg LR, Henderson JM**, Shenoy KV** (2020) Hand knob area of motor cortex in people with tetraplegia represents the whole body in a compositional way. Cell. 181:396–409. pdf url
21. Stavisky SD, Willett FR, Avansino DT, Hochberg LR, Shenoy KV**, Henderson JM** (2020) Speech-related dorsal motor cortex activity does not interfere with iBCI cursor control. Journal of Neural Engineering. 17:016049 (13pp). pdf url
20. Rastogi A, Vargas-Irwin C, Willett F, Abreu J, Crowder DC, Murphy B, Memberg W, Miller J, Sweet J, Walter B, Cash S, Rezaii P, Franco B, Saab J, Stavisky SD, Shenoy KV**, Henderson J**, Hochberg LR, Kirsch R, Ajiboye AB (2020) Neural representation of observed, imagined, and attempted grasping force in motor cortex of individuals with chronic tetraplegia. Scientific Reports. 10:1429. pdf url
19. Stavisky SD, Willett FR, Wilson GH, Murphy BA, Rezaii P, Avansino D, Memberg WD, Miller JP, Kirsch RF, Hochberg LR, Ajiboye AB, Druckmann S, Shenoy KV**, Henderson JM** (2019) Neural ensemble dynamics in dorsal motor cortex during speech in people with paralysis. eLife. 8:e46015. pdf figure-supplements url
18. Willett FR, Young DR, Murphy BA, Memberg WD, Blabe CH, Pandarinath C, Stavisky SD, Rezaii P, Saab J, Walter BL, Sweet JA, Miller JP, Henderson JM, Shenoy KV, Simeral J, Jarosiewicz B, Hochberg LR, Kirsch RF, Ajiboye AB (2019) Principled BCI decoder design and parameter selection using a feedback control model. Scientific Reports. 9(1):8881. pdf supp mats url
17. Milekovic T, Bacher D, Sarma A, Simeral J, Saab J, Pandarinath C, Yvert B, Sorice B, Blabe C, Oakley E, Tringale K, Eskandar E, Cash S, Shenoy KV, Henderson JM, Hochberg LR, Donoghue JP (2019) Volitional control of single-electrode high gamma local field potentials (LFPs) by people with paralysis. Journal of Neurophysiology. 121:1428-1450. pdf url
16. Young D, Willett F, Memberg W, Murphy B, Rezaii P, Walter B, Sweet J, Miller J, Shenoy KV, Hochberg LR, Kirsch R, Ajiboye AB (2019) Closed-loop cortical control of virtual reach and posture using cartesian and joint velocity commands. Journal of Neural Engineering. 16:026011 (14pp). pdf url
15. Nuyujukian P*, Sanabria JA*, Saab J*, Pandarinath C, Jarosiewicz B, Blabe C, Franco B, Mernoff ST, Eskandar EN, Simeral JD, Hochberg LR**, Shenoy KV**, Henderson JM** (2018) Cortical control of a tablet computer by people with paralysis. PLoS One. 13:e0204566 pdf url-with-all-movies
14. Stavisky SD, Kao JC, Nuyujukian P, Pandarinath C, Blabe C, Ryu SI, Hochberg LR, Henderson JM, Shenoy KV (2018) Brain-machine interface cursor position only weakly affects monkey and human motor cortical activity in the absence of arm movements. Scientific Reports. 8:1635.7 pd
13. Pandarinath C, O'Shea DJ, Collins J, Jozefowicz R, Stavisky SD, Kao JC, Trautmann EM, Kaufman MT, Ryu SI, Hochberg LR, Henderson JM, Shenoy KV, Abbott LF, Sussillo D (2018) Inferring single-trial neural population dynamics using sequential auto-encoders. Nature Methods.15:805-815. pdf supp-mats-text-videos url
SuppVideo1 Generator initial states inferred by LFADS are organized with respect to kinematics of the upcoming reach. The video depicts the initial conditions vectors for each individual trial of the ‘Maze’ reaching task for monkey J, mapped onto a low-dimensional space (3D) via t-SNE (as in Fig. 2c). Each point represents the initial conditions vector for an individual trial (2,296 trials are shown). Colors denote the angle of the endpoint of the upcoming reach (colors shown in Fig. 2a), and marker types denote the curvature of the reach (circles, squares, and triangles for straight, counter-clockwise curved, and clockwise curved reaches, respectively). As shown, the initial conditions exhibit similarity for trials with similar kinematic trajectories (both for trials whose reach endpoints have similar angles and for trials with similar reach curvature). Since structure in the initial conditions implies structure at the level of the generator’s dynamics, this analysis implies that LFADS produces dynamic trajectories that show similarity based on the kinematics of the reach type for a given trial, despite LFADS not having any information about reaching conditions.
SuppVideo2 LFADS reveals consistent rotational dynamics on individual trials. The video contains two sequential movies showing the trajectories in neural population state space during individual reach trials for monkey J (Fig. 3). The first movie illustrates the single-trial trajectories uncovered by smoothing the data with a Gaussian kernel. The second movie illustrates single-trial trajectories uncovered by LFADS. 2,296 trials are shown, representing the 108 conditions of the ‘Maze’ task.
SuppVideo3 Multisession LFADS finds consistent representations for individual trials across sessions. The video contains six sequential movies showing the trajectories in state space during individual reach trials for monkey P (Fig. 4). The first video shows single-trial GPFA factor trajectories for all trials estimated for a single session. The second and third videos show single-trial LFADS factor trajectories estimated from all trials using a single-session model and from the stitched model, respectively. The fourth, fifth, and sixth repeat this sequence but show single-trial trajectories for 42/44 sessions (2 were omitted for ease of presentation). Colors represent eight reach directions. Multisession movies include approximately 14,500 trials, 38 separate electrode penetration sites and spanned 162 d from the first to the last session. Each trajectory begins at the go cue and proceeds for 510 ms into movement, which occurs at varying times due to reaction time variability. For GPFA and single-session. LFADS factors, the trajectories from individual sessions were concatenated and the projection yielding the CIS and first jPCA plane was estimated (Methods). This provided a set of common trajectories against which each individual session’s data were regressed. These regression coefficients provided projections of the individual sessions’ trajectories that were maximally similar to the common trajectories. In contrast, for stitched LFADS factors, we simply estimated the projection yielding the CIS and first jPCA plane from all of the sessions together, as the factors are already in shared space.
News & Views: Batista AP, DiCarlo JJ (2018) Deep learning reaches the motor system. Nature Methods. News & Views. 15:772-773. pdf
Seminar talk: Sussillo D (3/222018) LFADS seminar talk, Simons Institute for the Theory of Computing meeting, UC Berkeley. video
12. Milekovic T, Sarma A, Bacher D, Simeral J, Saab J, Pandarinath C, Sorice B, Blabe C, Oakley E, Tringale K, Eskandar E, Cash S, Henderson JM, Shenoy KV, Donoghue JP, Hochberg LR (2018) Stable long-term BCI-enabled communication in ALS and locked-in syndrome using LFP signals. Journal of Neurophysiology. 120:343-360. pdf url
11. Willett FR, Murphy BA, Young D, Memberg WD, Blabe CH, Pandarinath C, Franco B, Saab J, Walter BL, Sweet JA, Miller JP, Henderson JM, Shenoy KV, Simeral JD, Jarosiewicz B, Hochberg LR, Kirsch RF, Ajiboye AB (2018) A comparison of intention estimation methods for decoder calibration in intracortical brain-computer interfaces. IEEE Transactions in Biomedical Engineering. 65:2066-2078. pdf url
10. Even-Chen N, Stavisky SD, Pandarinath C, Nuyujukian P, Blabe CH, Hochberg LR, Henderson* JM, Shenoy* KV (2018) Feasibility of automatic error detect-and-undo system in human intracortical brain-computer interfaces. IEEE Transactions in Biomedical Engineering. 65:1771-1784. pdf url
9. Brandman D, Hosman T, Saab J, Burkhart M, Shanahan B, Ciancibello J, Sarma A, Milstein D, Vargas-Irwin C, Franco B, Kelemen J, Blabe C, Murphy B, Young D, Willett F, Pandarinath C, Stavisky S, Kirsch R, Walter B, Ajiboye A, Cash S, Eskandar E, Miller J, Sweet J, Shenoy KV, Henderson JM, Jarosiewicz B, 9. Harrison M, Simeral J, Hochberg, LR (2018) Rapid calibration of an intracortical brain computer interface for people with tetraplegia. Journal of Neural Engineering. 15:026007. pdf url
8. Pandarinath C*, Nuyujukian P*, Blabe CH, Sorice B, Saab J, Willett F, Hochberg LR, Shenoy KV**, Henderson JM** (2017) High performance communication by people with paralysis using an intracortical brain-computer interface. eLife. 6:e18554 pdf url-with-all-movies
7. Willett FR, Murphy B, Memberg W, Blabe C, Pandarinath C, Walter B, Sweet J, Miller J, Henderson JM, Shenoy KV, Hochberg LR, Kirsch R, Ajiboye AB (2017) Signal-independent noise in intracortical brain-computer interfaces causes movement time properties inconsistent with Fitts' law. Journal of Neural Engineering. 14:026010. pdf
6. Willett F, Pandarinath C, Jarosiewicz B, Murphy B, Memberg W, Blabe C, Saab J, Walter B, Sweet J, Miller J, Henderson J, Shenoy KV, Simeral J, Hochberg LR, Kirsch R, Ajiboye AB. (2017) Feedback control policies employed by people using intracortical brain-computer interfaces. Journal of Neural Engineering. 14:016001 (16pp). pdf
5. Jarosiewicz B, Sarma AA, Bacher D, Masse NY, Simeral JD, Sorice B, Oakley EM, Blabe C, Pandarinath C, Gilja V, Cash SS, Eskandar E, Friehs G, Henderson JM, Shenoy KV, Donoghue JP, Hochberg LR (2015) Virtual typing by people with tetraplegia using a self-calibrating intracortical brain-computer interface. Science Translational Medicine. 7:1-10. pdf supplement video1
4. Gilja V*, Pandarinath C*, Blabe CH, Nuyujukian P, Simeral JD, Sarma AA, Sorice BL, Perge JA, Jarosiewicz B, Hochberg LR, Shenoy KV**, Henderson JM** (2015) Clinical translation of a high performance neural prosthesis. Nature Medicine. 21:1142-1145. pdf supplement video1 video2 video3
3. Blabe C, Gilja V, Chestek CA, Shenoy KV, Anderson K, Henderson JM (2015) Assessment of brain-machine interfaces from the perspective of people with paralysis. Journal of Neural Engineering. 12:043002. pdf
2. Pandarinath C, Gilja V, Blabe CH, Nuyujukian P, Sarma AA, Sorice BL, Eskandar EN, Hochberg LR, Henderson JM*, Shenoy KV* (2015) Neural population dynamics in human motor cortex during movements in people with ALS. eLife. 4:e07436. pdf video1
1. Chestek CA, Gilja V, Blabe CH, Foster BL, Shenoy KV, Parvizi J, Henderson JM (2013) Hand posture classification using electrocorticography signals in the gamma band over human sensorimotor brain areas. Journal of Neural Engineering. 10:02602 (11pp). pdf
5. Willett FR, Avansino DT, Hochberg LR, Henderson JM**, Shenoy KV** (2020) High-performance brain-to-text communication via imagined handwriting. 1st BCI Un-Conference, Session 3: Language-based BCI-Communication. Talk 8. 12 July 2020. Video starts at 0min 15sec: url-with-video video.mp4
- Abstract: To date, a major focus of BCI research has been on restoring gross motor skills, such as reaching and grasping or point-and-click typing with a 2D computer cursor. However, rapid sequences of highly dexterous behaviors, such as handwriting or touch typing, might enable faster communication rates. Here, we demonstrate an intracortical BCI that can decode imagined handwriting movements from neural activity in motor cortex and translate it to text in real-time, using a novel recurrent neural network decoding approach. With this BCI, our study participant (whose hand was paralyzed) achieved typing speeds that exceed those of any other BCI yet reported: 90 characters per minute at >99% accuracy with a general-purpose autocorrect. These speeds are comparable to able-bodied smartphone typing speeds in our participant’s age group (115 characters per minute) and significantly close the gap between BCI-enabled typing and able-bodied typing rates. Finally, new theoretical considerations explain why temporally complex movements, such as handwriting, may be fundamentally easier to decode than point-to-point movements. Our results open a new approach for BCIs and demonstrate the feasibility of accurately decoding rapid, dexterous movements years after paralysis.
4. Stavisky SD*, Wilson GH*, Willett FR, Avansino DT, Kelemen JN, Hochberg LR, Henderson JM**, Druckmann S,** Shenoy KV** (2020) Decoding speech production using intracortical electrode arrays in dorsal precentral gyrus. 1st BCI Un-Conference, Session 3: Language-based BCI-Communication, Talk 9. 12 July 2020. Video starts at 19 min 42 sec: url-with-video video.mp4
- Abstract: Efforts to build brain-computer interfaces (BCIs) to restore lost speech have rapidly accelerated in recent years, with a number of impressive demonstrations using electrocorticography (ECoG). In parallel, BCIs that decode attempted arm and hand movements for controlling a robotic arm and point-and-click typing communication have had the highest performance to date using intracortical electrode arrays. We believe that an intracortical approach is equally promising for speech BCIs and can complement ECoG recordings. However, progress has been slower in this domain due to limited animal models for speech, leading to the need for preliminary work to be done in humans. We have recently made progress prototyping an intracortical speech BCI thanks to our discovery that individual neurons in the dorsal “hand knob” area of the precentral gyrus are active during speech (Stavisky et al., 2019, 2020). Here, we recorded from a BrainGate2 pilot clinical trial participant, who has tetraplegia but can speak, as he spoke visually prompted words that broadly sample a comprehensive set of 39 English phonemes. Phoneme identities could be classified from the neural activity recorded by two 96-electrode arrays with 33.9% accuracy using a recurrent neural network decoder. Performance did not saturate with increasing training data quantity or electrode count. By demonstrating offline performance comparable to previous ECoG studies – despite these arrays being in an area that is very likely suboptimal for speech decoding – these results help lay the groundwork for a concerted speech BCI effort using intracortical measurements from more ventral cortical areas which we expect to have stronger speech-related modulation.
3. Wilson G*, Stavisky SD*, Avansino D, Hochberg LR, Henderson JM**, Shenoy KV**, Druckmann S** (2020) Neural state space geometry in human motor cortex underlying speaking different phonemes. Frontiers in Neuroscience. Conference Abstract: Computational and Systems Neuroscience (COSYNE), Denver, CO. Poster number III-71. pdf
2. Willett F, Avansino D, Hochberg LR, Henderson JM**, Shenoy KV** (2020) Motor Cortical Representation and Decoding of Attempted Handwriting in a Person with Tetraplegia. Frontiers in Neuroscience. Conference Abstract: Computational and Systems Neuroscience (COSYNE), Denver, CO. Poster number III-1. pdf
1. Stavisky SD, Rezaii P, Willett FR, Hochberg LR, Shenoy KV*, Henderson JM* (2018) Decoding speech from intracortical multielectrode arrays in dorsal "arm / hand areas" of human motor cortex. 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Honolulu, HI: 93-97. pdf