Faculty Bibliography

Recent research has used large language models (LLMs) to study the neural basis of naturalistic language processing in the human brain. LLMs have rapidly grown in complexity, leading to improved language processing capabilities. However, neuroscience researchers haven't kept up with the quick progress in LLM development. Here, we utilized several families of transformer-based LLMs to investigate the relationship between model size and their ability to capture linguistic information in the human brain. Crucially, a subset of LLMs were trained on a fixed training set, enabling us to dissociate model size from architecture and training set size. We used electrocorticography (ECoG) to measure neural activity in epilepsy patients while they listened to a 30-minute naturalistic audio story. We fit electrode-wise encoding models using contextual embeddings extracted from each hidden layer of the LLMs to predict word-level neural signals. In line with prior work, we found that larger LLMs better capture the structure of natural language and better predict neural activity. We also found a log-linear relationship where the encoding performance peaks in relatively earlier layers as model size increases. We also observed variations in the best-performing layer across different brain regions, corresponding to an organized language processing hierarchy.

Syntax, the abstract structure of language, is a hallmark of human cognition. Despite its importance, its neural underpinnings remain obscured by inherent limitations of non-invasive brain measures and a near total focus on comprehension paradigms. Here, we address these limitations with high-resolution neurosurgical recordings (electrocorticography) and a controlled sentence production experiment. We uncover three syntactic networks that are broadly distributed across traditional language regions, but with focal concentrations in middle and inferior frontal gyri. In contrast to previous findings from comprehension studies, these networks process syntax mostly to the exclusion of words and meaning, supporting a cognitive architecture with a distinct syntactic system. Most strikingly, our data reveal an unexpected property of syntax: it is encoded independent of neural activity levels. We propose that this "low-activity coding" scheme represents a novel mechanism for encoding information, reserved for higher-order cognition more broadly.

Neural responses in visual cortex adapt to prolonged and repeated stimuli. While adaptation occurs across the visual cortex, it is unclear how adaptation patterns and computational mechanisms differ across the visual hierarchy. Here we characterize two signatures of short-term neural adaptation in time-varying intracranial electroencephalography (iEEG) data collected while participants viewed naturalistic image categories varying in duration and repetition interval. Ventral- and lateral-occipitotemporal cortex exhibit slower and prolonged adaptation to single stimuli and slower recovery from adaptation to repeated stimuli compared to V1-V3. For category-selective electrodes, recovery from adaptation is slower for preferred than non-preferred stimuli. To model neural adaptation we augment our delayed divisive normalization (DN) model by scaling the input strength as a function of stimulus category, enabling the model to accurately predict neural responses across multiple image categories. The model fits suggest that differences in adaptation patterns arise from slower normalization dynamics in higher visual areas interacting with differences in input strength resulting from category selectivity. Our results reveal systematic differences in temporal adaptation of neural population responses between lower and higher visual brain areas and show that a single computational model of history-dependent normalization dynamics, fit with area-specific parameters, accounts for these differences.

The ability to connect the form and meaning of a concept, known as word retrieval, is fundamental to human communication. While various input modalities could lead to identical word retrieval, the exact neural dynamics supporting this convergence relevant to daily auditory discourse remain poorly understood. Here, we leveraged neurosurgical electrocorticographic (ECoG) recordings from 48 patients and dissociated two key language networks that highly overlap in time and space integral to word retrieval. Using unsupervised temporal clustering techniques, we found a semantic processing network located in the middle and inferior frontal gyri. This network was distinct from an articulatory planning network in the inferior frontal and precentral gyri, which was agnostic to input modalities. Functionally, we confirmed that the semantic processing network encodes word surprisal during sentence perception. Our findings characterize how humans integrate ongoing auditory semantic information over time, a critical linguistic function from passive comprehension to daily discourse.

Contextual embeddings, derived from deep language models (DLMs), provide a continuous vectorial representation of language. This embedding space differs fundamentally from the symbolic representations posited by traditional psycholinguistics. We hypothesize that language areas in the human brain, similar to DLMs, rely on a continuous embedding space to represent language. To test this hypothesis, we densely record the neural activity patterns in the inferior frontal gyrus (IFG) of three participants using dense intracranial arrays while they listened to a 30-minute podcast. From these fine-grained spatiotemporal neural recordings, we derive a continuous vectorial representation for each word (i.e., a brain embedding) in each patient. Using stringent zero-shot mapping we demonstrate that brain embeddings in the IFG and the DLM contextual embedding space have common geometric patterns. The common geometric patterns allow us to predict the brain embedding in IFG of a given left-out word based solely on its geometrical relationship to other non-overlapping words in the podcast. Furthermore, we show that contextual embeddings capture the geometry of IFG embeddings better than static word embeddings. The continuous brain embedding space exposes a vector-based neural code for natural language processing in the human brain.

OBJECTIVE:New-onset refractory status epilepticus (NORSE) is a rare but severe clinical syndrome. Despite rigorous evaluation, the underlying cause is unknown in 30%-50% of patients and treatment strategies are largely empirical. The aim of this study was to describe clinical outcomes in a cohort of well-phenotyped, thoroughly investigated patients who survived the initial phase of cryptogenic NORSE managed in specialist centers. METHODS:Well-characterized cases of cryptogenic NORSE were identified through the EPIGEN and Critical Care EEG Monitoring Research Consortia (CCEMRC) during the period 2005-2019. Treating epileptologists reported on post-NORSE survival rates and sequelae in patients after discharge from hospital. Among survivors >6 months post-discharge, we report the rates and severity of active epilepsy, global disability, vocational, and global cognitive and mental health outcomes. We attempt to identify determinants of outcome. RESULTS:Among 48 patients who survived the acute phase of NORSE to the point of discharge from hospital, 9 had died at last follow-up, of whom 7 died within 6 months of discharge from the tertiary care center. The remaining 39 patients had high rates of active epilepsy as well as vocational, cognitive, and psychiatric comorbidities. The epilepsy was usually multifocal and typically drug resistant. Only a minority of patients had a good functional outcome. Therapeutic interventions were heterogenous during the acute phase of the illness. There was no clear relationship between the nature of treatment and clinical outcomes. SIGNIFICANCE/CONCLUSIONS:Among survivors of cryptogenic NORSE, longer-term outcomes in most patients were life altering and often catastrophic. Treatment remains empirical and variable. There is a pressing need to understand the etiology of cryptogenic NORSE and to develop tailored treatment strategies.

OBJECTIVE/UNASSIGNED:This study investigates speech decoding from neural signals captured by intracranial electrodes. Most prior works can only work with electrodes on a 2D grid (i.e., Electrocorticographic or ECoG array) and data from a single patient. We aim to design a deep-learning model architecture that can accommodate both surface (ECoG) and depth (stereotactic EEG or sEEG) electrodes. The architecture should allow training on data from multiple participants with large variability in electrode placements and the trained model should perform well on participants unseen during training. APPROACH/UNASSIGNED:We propose a novel transformer-based model architecture named SwinTW that can work with arbitrarily positioned electrodes, by leveraging their 3D locations on the cortex rather than their positions on a 2D grid. We train both subject-specific models using data from a single participant as well as multi-patient models exploiting data from multiple participants. MAIN RESULTS/UNASSIGNED:The subject-specific models using only low-density 8x8 ECoG data achieved high decoding Pearson Correlation Coefficient with ground truth spectrogram (PCC=0.817), over N=43 participants, outperforming our prior convolutional ResNet model and the 3D Swin transformer model. Incorporating additional strip, depth, and grid electrodes available in each participant (N=39) led to further improvement (PCC=0.838). For participants with only sEEG electrodes (N=9), subject-specific models still enjoy comparable performance with an average PCC=0.798. The multi-subject models achieved high performance on unseen participants, with an average PCC=0.765 in leave-one-out cross-validation. SIGNIFICANCE/UNASSIGNED:The proposed SwinTW decoder enables future speech neuroprostheses to utilize any electrode placement that is clinically optimal or feasible for a particular participant, including using only depth electrodes, which are more routinely implanted in chronic neurosurgical procedures. Importantly, the generalizability of the multi-patient models suggests the exciting possibility of developing speech neuroprostheses for people with speech disability without relying on their own neural data for training, which is not always feasible.

OBJECTIVE:This study was undertaken to conduct external validation of previously published epilepsy surgery prediction tools using a large independent multicenter dataset and to assess whether these tools can stratify patients for being operated on and for becoming free of disabling seizures (International League Against Epilepsy stage 1 and 2). METHODS:We analyzed a dataset of 1562 patients, not used for tool development. We applied two scales: Epilepsy Surgery Grading Scale (ESGS) and Seizure Freedom Score (SFS); and two versions of Epilepsy Surgery Nomogram (ESN): the original version and the modified version, which included electroencephalographic data. For the ESNs, we used calibration curves and concordance indexes. We stratified the patients into three tiers for assessing the chances of attaining freedom from disabling seizures after surgery: high (ESGS = 1, SFS = 3-4, ESNs > 70%), moderate (ESGS = 2, SFS = 2, ESNs = 40%-70%), and low (ESGS = 2, SFS = 0-1, ESNs < 40%). We compared the three tiers as stratified by these tools, concerning the proportion of patients who were operated on, and for the proportion of patients who became free of disabling seizures. RESULTS:The concordance indexes for the various versions of the nomograms were between .56 and .69. Both scales (ESGS, SFS) and nomograms accurately stratified the patients for becoming free of disabling seizures, with significant differences among the three tiers (p < .05). In addition, ESGS and the modified ESN accurately stratified the patients for having been offered surgery, with significant difference among the three tiers (p < .05). SIGNIFICANCE/CONCLUSIONS:ESGS and the modified ESN (at thresholds of 40% and 70%) stratify patients undergoing presurgical evaluation into three tiers, with high, moderate, and low chance for favorable outcome, with significant differences between the groups concerning having surgery and becoming free of disabling seizures. Stratifying patients for epilepsy surgery has the potential to help select the optimal candidates in underprivileged areas and better allocate resources in developed countries.

Cortical regions supporting speech production are commonly established using neuroimaging techniques in both research and clinical settings. However, for neurosurgical purposes, structural function is routinely mapped peri-operatively using direct electrocortical stimulation. While this method is the gold standard for identification of eloquent cortical regions to preserve in neurosurgical patients, there is lack of specificity of the actual underlying cognitive processes being interrupted. To address this, we propose mapping the temporal dynamics of speech arrest across peri-sylvian cortices by quantifying the latency between stimulation and speech deficits. In doing so, we are able to substantiate hypotheses about distinct region-specific functional roles (e.g. planning versus motor execution). In this retrospective observational study, we analysed 20 patients (12 female; age range 14-43) with refractory epilepsy who underwent continuous extra-operative intracranial EEG monitoring of an automatic speech task during clinical bedside language mapping. Latency to speech arrest was calculated as time from stimulation onset to speech arrest onset, controlling for individual speech rate. Most instances of motor-based arrest (87.5% of 96 instances) were in sensorimotor cortex with mid-range latencies to speech arrest with a distributional peak at 0.47 s. Speech arrest occurred in numerous regions, with relatively short latencies in supramarginal gyrus (0.46 s), superior temporal gyrus (0.51 s) and middle temporal gyrus (0.54 s), followed by relatively long latencies in sensorimotor cortex (0.72 s) and especially long latencies in inferior frontal gyrus (0.95 s). Non-parametric testing for speech arrest revealed that region predicted latency; latencies in supramarginal gyrus and in superior temporal gyrus were shorter than in sensorimotor cortex and in inferior frontal gyrus. Sensorimotor cortex is primarily responsible for motor-based arrest. Latencies to speech arrest in supramarginal gyrus and superior temporal gyrus (and to a lesser extent middle temporal gyrus) align with latencies to motor-based arrest in sensorimotor cortex. This pattern of relatively quick cessation of speech suggests that stimulating these regions interferes with the outgoing motor execution. In contrast, the latencies to speech arrest in inferior frontal gyrus and in ventral regions of sensorimotor cortex were significantly longer than those in temporoparietal regions. Longer latencies in the more frontal areas (including inferior frontal gyrus and ventral areas of precentral gyrus and postcentral gyrus) suggest that stimulating these areas interrupts a higher-level speech production process involved in planning. These results implicate the ventral specialization of sensorimotor cortex (including both precentral and postcentral gyri) for speech planning above and beyond motor execution.

PURPOSE/OBJECTIVE:Brain responsive neurostimulation (NeuroPace) treats patients with refractory focal epilepsy and provides chronic electrocorticography (ECoG). We explored how machine learning algorithms applied to interictal ECoG could assess clinical response to changes in neurostimulation parameters. METHODS:We identified five responsive neurostimulation patients each with ≥200 continuous days of stable medication and detection settings (median, 358 days per patient). For each patient, interictal ECoG segments for each month were labeled as "high" or "low" to represent relatively high or low long-episode (i.e., seizure) count compared with the median monthly long-episode count. Power from six conventional frequency bands from four responsive neurostimulation channels were extracted as features. For each patient, five machine learning algorithms were trained on 80% of ECoG, then tested on the remaining 20%. Classifiers were scored by the area-under-the-receiver-operating-characteristic curve. We explored how individual circadian cycles of seizure activity could inform classifier building. RESULTS:Support vector machine or gradient boosting models achieved the best performance, ranging from 0.705 (fair) to 0.892 (excellent) across patients. High gamma power was the most important feature, tending to decrease during low-seizure-frequency epochs. For two subjects, training on ECoG recorded during the circadian ictal peak resulted in comparable model performance, despite less data used. CONCLUSIONS:Machine learning analysis on retrospective background ECoG can classify relative seizure frequency for an individual patient. High gamma power was the most informative, whereas individual circadian patterns of seizure activity can guide model building. Machine learning classifiers built on interictal ECoG may guide stimulation programming.