Faculty Bibliography

OBJECTIVE/UNASSIGNED:This study investigates speech decoding from neural signals captured by intracranial electrodes. Most prior works can only work with electrodes on a 2D grid (i.e., Electrocorticographic or ECoG array) and data from a single patient. We aim to design a deep-learning model architecture that can accommodate both surface (ECoG) and depth (stereotactic EEG or sEEG) electrodes. The architecture should allow training on data from multiple participants with large variability in electrode placements and the trained model should perform well on participants unseen during training. APPROACH/UNASSIGNED:We propose a novel transformer-based model architecture named SwinTW that can work with arbitrarily positioned electrodes, by leveraging their 3D locations on the cortex rather than their positions on a 2D grid. We train both subject-specific models using data from a single participant as well as multi-patient models exploiting data from multiple participants. MAIN RESULTS/UNASSIGNED:The subject-specific models using only low-density 8x8 ECoG data achieved high decoding Pearson Correlation Coefficient with ground truth spectrogram (PCC=0.817), over N=43 participants, outperforming our prior convolutional ResNet model and the 3D Swin transformer model. Incorporating additional strip, depth, and grid electrodes available in each participant (N=39) led to further improvement (PCC=0.838). For participants with only sEEG electrodes (N=9), subject-specific models still enjoy comparable performance with an average PCC=0.798. The multi-subject models achieved high performance on unseen participants, with an average PCC=0.765 in leave-one-out cross-validation. SIGNIFICANCE/UNASSIGNED:The proposed SwinTW decoder enables future speech neuroprostheses to utilize any electrode placement that is clinically optimal or feasible for a particular participant, including using only depth electrodes, which are more routinely implanted in chronic neurosurgical procedures. Importantly, the generalizability of the multi-patient models suggests the exciting possibility of developing speech neuroprostheses for people with speech disability without relying on their own neural data for training, which is not always feasible.

Sound structures such as phonemes and words have highly variable durations. Thus, there is a fundamental difference between integrating across absolute time (e.g., 100 ms) vs. sound structure (e.g., phonemes). Auditory and cognitive models have traditionally cast neural integration in terms of time and structure, respectively, but the extent to which cortical computations reflect time or structure remains unknown. To answer this question, we rescaled the duration of all speech structures using time stretching/compression and measured integration windows in the human auditory cortex using a new experimental/computational method applied to spatiotemporally precise intracranial recordings. We observed significantly longer integration windows for stretched speech, but this lengthening was very small (~5%) relative to the change in structure durations, even in non-primary regions strongly implicated in speech-specific processing. These findings demonstrate that time-yoked computations dominate throughout the human auditory cortex, placing important constraints on neurocomputational models of structure processing.

In medication-resistant epilepsy, the goal of epilepsy surgery is to make a patient seizure free with a resection/ablation that is as small as possible to minimize morbidity. The standard of care in planning the margins of epilepsy surgery involves electroclinical delineation of the seizure onset zone (SOZ) and incorporation of neuroimaging findings from MRI, PET, SPECT, and MEG modalities. Resecting cortical tissue generating high-frequency oscillations (HFOs) has been investigated as a more efficacious alternative to targeting the SOZ. In this study, we used a support vector machine (SVM), with four distinct fast ripple (FR: 350-600 Hz on oscillations, 200-600 Hz on spikes) metrics as factors. These metrics included the FR resection ratio (RR), a spatial FR network measure, and two temporal FR network measures. The SVM was trained by the value of these four factors with respect to the actual resection boundaries and actual seizure free labels of 18 patients with medically refractory focal epilepsy. Leave one out cross-validation of the trained SVM in this training set had an accuracy of 0.78. We next used a simulated iterative virtual resection targeting the FR sites that were highest rate and showed most temporal autonomy. The trained SVM utilized the four virtual FR metrics to predict virtual seizure freedom. In all but one of the nine patients seizure free after surgery, we found that the virtual resections sufficient for virtual seizure freedom were larger in volume (p<0.05). In nine patients who were not seizure free, a larger virtual resection made five virtually seizure free. We also examined 10 medically refractory focal epilepsy patients implanted with the responsive neurostimulator system (RNS) and virtually targeted the RNS stimulation contacts proximal to sites generating FR at highest rates to determine if the simulated value of the stimulated SOZ and stimulated FR metrics would trend toward those patients with a better seizure outcome. Our results suggest: 1) FR measures can accurately predict whether a resection, defined by the standard of care, will result in seizure freedom; 2) utilizing FR alone for planning an efficacious surgery can be associated with larger resections; 3) when FR metrics predict the standard of care resection will fail, amending the boundaries of the planned resection with certain FR generating sites may improve outcome; and 4) more work is required to determine if targeting RNS stimulation contact proximal to FR generating sites will improve seizure outcome.

Neural responses in visual cortex adapt to prolonged and repeated stimuli. While adaptation occurs across the visual cortex, it is unclear how adaptation patterns and computational mechanisms differ across the visual hierarchy. Here we characterize two signatures of short-term neural adaptation in time-varying intracranial electroencephalography (iEEG) data collected while participants viewed naturalistic image categories varying in duration and repetition interval. Ventral- and lateral-occipitotemporal cortex exhibit slower and prolonged adaptation to single stimuli and slower recovery from adaptation to repeated stimuli compared to V1-V3. For category-selective electrodes, recovery from adaptation is slower for preferred than non-preferred stimuli. To model neural adaptation we augment our delayed divisive normalization (DN) model by scaling the input strength as a function of stimulus category, enabling the model to accurately predict neural responses across multiple image categories. The model fits suggest that differences in adaptation patterns arise from slower normalization dynamics in higher visual areas interacting with differences in input strength resulting from category selectivity. Our results reveal systematic differences in temporal adaptation of neural population responses between lower and higher visual brain areas and show that a single computational model of history-dependent normalization dynamics, fit with area-specific parameters, accounts for these differences.

Contextual embeddings, derived from deep language models (DLMs), provide a continuous vectorial representation of language. This embedding space differs fundamentally from the symbolic representations posited by traditional psycholinguistics. We hypothesize that language areas in the human brain, similar to DLMs, rely on a continuous embedding space to represent language. To test this hypothesis, we densely record the neural activity patterns in the inferior frontal gyrus (IFG) of three participants using dense intracranial arrays while they listened to a 30-minute podcast. From these fine-grained spatiotemporal neural recordings, we derive a continuous vectorial representation for each word (i.e., a brain embedding) in each patient. Using stringent zero-shot mapping we demonstrate that brain embeddings in the IFG and the DLM contextual embedding space have common geometric patterns. The common geometric patterns allow us to predict the brain embedding in IFG of a given left-out word based solely on its geometrical relationship to other non-overlapping words in the podcast. Furthermore, we show that contextual embeddings capture the geometry of IFG embeddings better than static word embeddings. The continuous brain embedding space exposes a vector-based neural code for natural language processing in the human brain.

In medication-resistant epilepsy, the goal of epilepsy surgery is to make a patient seizure free with a resection/ablation that is as small as possible to minimize morbidity. The standard of care in planning the margins of epilepsy surgery involves electroclinical delineation of the seizure-onset zone and incorporation of neuroimaging findings from MRI, PET, single-photon emission CT and magnetoencephalography modalities. Resecting cortical tissue generating high-frequency oscillations has been investigated as a more efficacious alternative to targeting the seizure-onset zone. In this study, we used a support vector machine (SVM), with four distinct fast ripple (FR: 350-600 Hz on oscillations, 200-600 Hz on spikes) metrics as factors. These metrics included the FR resection ratio, a spatial FR network measure and two temporal FR network measures. The SVM was trained by the value of these four factors with respect to the actual resection boundaries and actual seizure-free labels of 18 patients with medically refractory focal epilepsy. Leave-one-out cross-validation of the trained SVM in this training set had an accuracy of 0.78. We next used a simulated iterative virtual resection targeting the FR sites that were of highest rate and showed most temporal autonomy. The trained SVM utilized the four virtual FR metrics to predict virtual seizure freedom. In all but one of the nine patients who were seizure free after surgery, we found that the virtual resections sufficient for virtual seizure freedom were larger in volume (P < 0.05). In nine patients who were not seizure free, a larger virtual resection made five virtually seizure free. We also examined 10 medically refractory focal epilepsy patients implanted with the responsive neurostimulator system and virtually targeted the responsive neurostimulator system stimulation contacts proximal to sites generating FR at highest rates to determine if the simulated value of the stimulated seizure-onset zone and stimulated FR metrics would trend towards those patients with a better seizure outcome. Our results suggest the following: (i) FR measures can accurately predict whether a resection, defined by the standard of care, will result in seizure freedom; (ii) utilizing FR alone for planning an efficacious surgery can be associated with larger resections; (iii) when FR metrics predict the standard-of-care resection will fail, amending the boundaries of the planned resection with certain FR-generating sites may improve outcome and (iv) more work is required to determine whether targeting responsive neurostimulator system stimulation contact proximal to FR generating sites will improve seizure outcome.

Cortical regions supporting speech production are commonly established using neuroimaging techniques in both research and clinical settings. However, for neurosurgical purposes, structural function is routinely mapped peri-operatively using direct electrocortical stimulation. While this method is the gold standard for identification of eloquent cortical regions to preserve in neurosurgical patients, there is lack of specificity of the actual underlying cognitive processes being interrupted. To address this, we propose mapping the temporal dynamics of speech arrest across peri-sylvian cortices by quantifying the latency between stimulation and speech deficits. In doing so, we are able to substantiate hypotheses about distinct region-specific functional roles (e.g. planning versus motor execution). In this retrospective observational study, we analysed 20 patients (12 female; age range 14-43) with refractory epilepsy who underwent continuous extra-operative intracranial EEG monitoring of an automatic speech task during clinical bedside language mapping. Latency to speech arrest was calculated as time from stimulation onset to speech arrest onset, controlling for individual speech rate. Most instances of motor-based arrest (87.5% of 96 instances) were in sensorimotor cortex with mid-range latencies to speech arrest with a distributional peak at 0.47 s. Speech arrest occurred in numerous regions, with relatively short latencies in supramarginal gyrus (0.46 s), superior temporal gyrus (0.51 s) and middle temporal gyrus (0.54 s), followed by relatively long latencies in sensorimotor cortex (0.72 s) and especially long latencies in inferior frontal gyrus (0.95 s). Non-parametric testing for speech arrest revealed that region predicted latency; latencies in supramarginal gyrus and in superior temporal gyrus were shorter than in sensorimotor cortex and in inferior frontal gyrus. Sensorimotor cortex is primarily responsible for motor-based arrest. Latencies to speech arrest in supramarginal gyrus and superior temporal gyrus (and to a lesser extent middle temporal gyrus) align with latencies to motor-based arrest in sensorimotor cortex. This pattern of relatively quick cessation of speech suggests that stimulating these regions interferes with the outgoing motor execution. In contrast, the latencies to speech arrest in inferior frontal gyrus and in ventral regions of sensorimotor cortex were significantly longer than those in temporoparietal regions. Longer latencies in the more frontal areas (including inferior frontal gyrus and ventral areas of precentral gyrus and postcentral gyrus) suggest that stimulating these areas interrupts a higher-level speech production process involved in planning. These results implicate the ventral specialization of sensorimotor cortex (including both precentral and postcentral gyri) for speech planning above and beyond motor execution.

OBJECTIVE:Effective surgical treatment of drug-resistant epilepsy depends on accurate localization of the epileptogenic zone (EZ). High-frequency oscillations (HFOs) are potential biomarkers of the EZ. Previous research has shown that HFOs often occur within submillimeter areas of brain tissue and that the coarse spatial sampling of clinical intracranial electrode arrays may limit the accurate capture of HFO activity. In this study, we sought to characterize microscale HFO activity captured on thin, flexible microelectrocorticographic (μECoG) arrays, which provide high spatial resolution over large cortical surface areas. METHODS:We used novel liquid crystal polymer thin-film μECoG arrays (.76-1.72-mm intercontact spacing) to capture HFOs in eight intraoperative recordings from seven patients with epilepsy. We identified ripple (80-250 Hz) and fast ripple (250-600 Hz) HFOs using a common energy thresholding detection algorithm along with two stages of artifact rejection. We visualized microscale subregions of HFO activity using spatial maps of HFO rate, signal-to-noise ratio, and mean peak frequency. We quantified the spatial extent of HFO events by measuring covariance between detected HFOs and surrounding activity. We also compared HFO detection rates on microcontacts to simulated macrocontacts by spatially averaging data. RESULTS:We found visually delineable subregions of elevated HFO activity within each μECoG recording. Forty-seven percent of HFOs occurred on single 200-μm-diameter recording contacts, with minimal high-frequency activity on surrounding contacts. Other HFO events occurred across multiple contacts simultaneously, with covarying activity most often limited to a .95-mm radius. Through spatial averaging, we estimated that macrocontacts with 2-3-mm diameter would only capture 44% of the HFOs detected in our μECoG recordings. SIGNIFICANCE/CONCLUSIONS:These results demonstrate that thin-film microcontact surface arrays with both highresolution and large coverage accurately capture microscale HFO activity and may improve the utility of HFOs to localize the EZ for treatment of drug-resistant epilepsy.

Describing intracortical laminar organization of interictal epileptiform discharges (IED) and high frequency oscillations (HFOs), also known as ripples. Defining the frequency limits of slow and fast ripples. We recorded potential gradients with laminar multielectrode arrays (LME) for current source density (CSD) and multi-unit activity (MUA) analysis of interictal epileptiform discharges IEDs and HFOs in the neocortex and mesial temporal lobe of focal epilepsy patients. IEDs were observed in 20/29, while ripples only in 9/29 patients. Ripples were all detected within the seizure onset zone (SOZ). Compared to hippocampal HFOs, neocortical ripples proved to be longer, lower in frequency and amplitude, and presented non-uniform cycles. A subset of ripples (≈ 50%) co-occurred with IEDs, while IEDs were shown to contain variable high-frequency activity, even below HFO detection threshold. The limit between slow and fast ripples was defined at 150 Hz, while IEDs' high frequency components form clusters separated at 185 Hz. CSD analysis of IEDs and ripples revealed an alternating sink-source pair in the supragranular cortical layers, although fast ripple CSD appeared lower and engaged a wider cortical domain than slow ripples MUA analysis suggested a possible role of infragranularly located neural populations in ripple and IED generation. Laminar distribution of peak frequencies derived from HFOs and IEDs, respectively, showed that supragranular layers were dominated by slower (< 150 Hz) components. Our findings suggest that cortical slow ripples are generated primarily in upper layers while fast ripples and associated MUA in deeper layers. The dissociation of macro- and microdomains suggests that microelectrode recordings may be more selective for SOZ-linked ripples. We found a complex interplay between neural activity in the neocortical laminae during ripple and IED formation. We observed a potential leading role of cortical neurons in deeper layers, suggesting a refined utilization of LMEs in SOZ localization.

Neural responses to visual stimuli exhibit complex temporal dynamics, including sub-additive temporal summation, response reduction with repeated or sustained stimuli (adaptation), and slower dynamics at low contrast. These phenomena are often studied independently. Here, we demonstrate these phenomena within the same experiment and model the underlying neural computations with a single computational model. We extracted time-varying responses from electrocorticographic (ECoG) recordings from patients presented with stimuli that varied in contrast, duration, and inter-stimulus interval (ISI). Aggregating data across patients from both sexes yielded 98 electrodes with robust visual responses, covering both earlier (V1-V3) and higher-order (V3a/b, LO, TO, IPS) retinotopic maps. In all regions, the temporal dynamics of neural responses exhibit several non-linear features: peak response amplitude saturates with high contrast and longer stimulus durations; the response to a second stimulus is suppressed for short ISIs and recovers for longer ISIs; response latency decreases with increasing contrast. These features are accurately captured by a computational model comprised of a small set of canonical neuronal operations: linear filtering, rectification, exponentiation, and a delayed divisive normalization. We find that an increased normalization term captures both contrast- and adaptation-related response reductions, suggesting potentially shared underlying mechanisms. We additionally demonstrate both changes and invariance in temporal response dynamics between earlier and higher-order visual areas. Together, our results reveal the presence of a wide range of temporal and contrast-dependent neuronal dynamics in the human visual cortex, and demonstrate that a simple model captures these dynamics at millisecond resolution.SIGNIFICANCE STATEMENTSensory inputs and neural responses change continuously over time. It is especially challenging to understand a system that has both dynamic inputs and outputs. Here we use a computational modeling approach that specifies computations to convert a time-varying input stimulus to a neural response time course, and use this to predict neural activity measured in the human visual cortex. We show that this computational model predicts a wide variety of complex neural response shapes that we induced experimentally by manipulating the duration, repetition and contrast of visual stimuli. By comparing data and model predictions, we uncover systematic properties of temporal dynamics of neural signals, allowing us to better understand how the brain processes dynamic sensory information.