Faculty Bibliography

A clinical artificial intelligence (AI) system is often validated on data withheld during its development. This provides an estimate of its performance upon future deployment on data in the wild; those currently unseen but are expected to be encountered in a clinical setting. However, estimating performance on data in the wild is complicated by distribution shift between data in the wild and withheld data and the absence of ground-truth annotations. Here, we introduce SUDO, a framework for evaluating AI systems on data in the wild. Through experiments on AI systems developed for dermatology images, histopathology patches, and clinical notes, we show that SUDO can identify unreliable predictions, inform the selection of models, and allow for the previously out-of-reach assessment of algorithmic bias for data in the wild without ground-truth annotations. These capabilities can contribute to the deployment of trustworthy and ethical AI systems in medicine.

In kidney transplantation, day-zero biopsies are used to assess organ quality and discriminate between donor-inherited lesions and those acquired post-transplantation. However, many centers do not perform such biopsies since they are invasive, costly and may delay the transplant procedure. We aim to generate a non-invasive virtual biopsy system using routinely collected donor parameters. Using 14,032 day-zero kidney biopsies from 17 international centers, we develop a virtual biopsy system. 11 basic donor parameters are used to predict four Banff kidney lesions: arteriosclerosis, arteriolar hyalinosis, interstitial fibrosis and tubular atrophy, and the percentage of renal sclerotic glomeruli. Six machine learning models are aggregated into an ensemble model. The virtual biopsy system shows good performance in the internal and external validation sets. We confirm the generalizability of the system in various scenarios. This system could assist physicians in assessing organ quality, optimizing allograft allocation together with discriminating between donor derived and acquired lesions post-transplantation.

Meaningful real-world evidence (RWE) generation requires unstructured data found in electronic health records (EHRs) which are often missing from administrative claims; however, obtaining relevant data from unstructured EHR sources is resource-intensive. In response, researchers are using natural language processing (NLP) with machine learning (ML) techniques (i.e., ML extraction) to extract real-world data (RWD) at scale. This study assessed the quality and fitness-for-use of EHR-derived oncology data curated using NLP with ML as compared to the reference standard of expert abstraction. Using a sample of 186,313 patients with lung cancer from a nationwide EHR-derived de-identified database, we performed a series of replication analyses demonstrating some common analyses conducted in retrospective observational research with complex EHR-derived data to generate evidence. Eligible patients were selected into biomarker- and treatment-defined cohorts, first with expert-abstracted then with ML-extracted data. We utilized the biomarker- and treatment-defined cohorts to perform analyses related to biomarker-associated survival and treatment comparative effectiveness, respectively. Across all analyses, the results differed by less than 8% between the data curation methods, and similar conclusions were reached. These results highlight that high-performance ML-extracted variables trained on expert-abstracted data can achieve similar results as when using abstracted data, unlocking the ability to perform oncology research at scale.

PURPOSE/OBJECTIVE:Our objective was to describe differences in telemedicine use among women with metastatic breast cancer (mBC) by race, age and geographic region. METHODS:This was a retrospective cohort study of women with recurrent or de novo mBC treated in U.S. community cancer practices that initiated a new line of therapy between March 2020 and February 2021. Multivariable modified Poisson regression models were used to calculate adjusted rate ratios (OR) and robust 95% confidence intervals (CI) associated with telemedicine visits within 90 days of therapy initiation. RESULTS:Overall, among 3412 women with mBC, 751 (22%) patients had telemedicine visits following therapy initiation, with lower risks observed among older women (<50 years: 24%; 50-64 years: 22%; 65-74 years: 21%; ≥75 years: 20%). Greater telemedicine use was observed among Asian women (35%) compared to White (21%), Black (18%) and Hispanic (21%) women. Fewer telemedicine visits occurred in Southern (12%) and Midwestern (17%) states versus Northeastern (37%) or Western (36%) states. In multivariable models, women ages ≥75 years had significantly lower risks of telemedicine visits (RR=0.76, 95% CI 0.62-0.95) compared to ages <50 years. Compared to patients in Northeastern states, women in Midwestern (RR=0.46, 95% CI 0.37-0.57) and Southern (RR=0.31, 95% CI 0.26-0.37) states had significantly lower risks of telemedicine visits; but not women in Western states (RR=0.96, 95% CI 0.83-1.12). No statistically significant differences in telemedicine use were found between racial groups in overall multivariable models. CONCLUSIONS:In this study of community cancer practices, older mBC patients and those living in Southern and Midwestern states were less likely to have telemedicine visits. Preferences for communication and delivery of care may have implications for measurement of exposures and endpoints in pharmacoepidemiologic studies of cancer patients.

A vast amount of real-world data, such as pathology reports and clinical notes, are captured as unstructured text in electronic health records (EHRs). However, this information is both difficult and costly to extract through human abstraction, especially when scaling to large datasets is needed. Fortunately, Natural Language Processing (NLP) and Machine Learning (ML) techniques provide promising solutions for a variety of information extraction tasks such as identifying a group of patients who have a specific diagnosis, share common characteristics, or show progression of a disease. However, using these ML-extracted data for research still introduces unique challenges in assessing validity and generalizability to different cohorts of interest. In order to enable effective and accurate use of ML-extracted real-world data (RWD) to support research and real-world evidence generation, we propose a research-centric evaluation framework for model developers, ML-extracted data users and other RWD stakeholders. This framework covers the fundamentals of evaluating RWD produced using ML methods to maximize the use of EHR data for research purposes.

In prior work, Friends of Cancer Research convened multiple data partners to establish standardized definitions for oncology real-world end points derived from electronic health records (EHRs) and claims data. Here, we assessed the performance of real-world overall survival (rwOS) from data sets sourced from EHRs by evaluating the ability of the end point to reflect expected differences from a previous randomized controlled trial across five data sources, after applying inclusion/exclusion criteria. The KEYNOTE-189 clinical trial protocol of platinum doublet chemotherapy (chemotherapy) vs. programmed cell death protein 1 (PD-1) in combination with platinum doublet chemotherapy (PD-1 combination) in first-line nonsquamous metastatic non-small cell lung cancer guided retrospective cohort selection. The Kaplan-Meier product limit estimator was used to calculate 12-month rwOS with 95% confidence intervals (CIs) in each data source. Cox proportional hazards models estimated hazard ratios (HRs) and associated 95% CIs, controlled for prognostic factors. Once the inclusion/exclusion criteria were applied, the five resulting data sets included 155 to 1,501 patients in the chemotherapy cohort and 36 to 405 patients in the PD-1 combination cohort. Twelve-month rwOS ranged from 45% to 58% in the chemotherapy cohort and 44% to 68% in the PD-1 combination cohort. The adjusted HR for death ranged from 0.80 (95% CI: 0.69, 0.93) to 1.15 (95% CI: 0.71, 1.85), controlling for age, gender, performance status, and smoking status. This study yielded insights regarding data capture, including ability of real-world data to precisely identify patient populations and the impact of criteria on end points. Sensitivity analyses could elucidate data set-specific factors that drive results.

OBJECTIVES:Racial disparities in cancer care and outcomes remain a societal challenge. Medicaid expansion through the Affordable Care Act was intended to improve health care access and equity. This study aimed to assess whether state Medicaid expansions were associated with a reduction in racial disparities in timely treatment among patients diagnosed with advanced cancer. STUDY DESIGN:This difference-in-differences study analyzed deidentified electronic health record-derived data. Patients aged 18 to 64 years with advanced or metastatic cancers diagnosed between January 1, 2011, and January 31, 2019, and receiving systemic therapy were included. METHODS:The primary end point was receipt of timely treatment, defined as first-line systemic therapy starting within 30 days after diagnosis of advanced or metastatic disease. Racial disparity was defined as adjusted percentage-point (PP) difference for Black vs White patients, adjusted for age, sex, practice setting, cancer type, stage, insurance marketplace, and area unemployment rate, with time and state fixed effects. RESULTS:The study included 30,310 patients (12.3% Black race). Without Medicaid expansion, Black patients were less likely to receive timely treatment than White patients (43.7% vs 48.4%; adjusted difference, -4.8 PP; P < .001). With Medicaid expansion, this disparity was diminished and lost significance (49.7% vs 50.5%; adjusted difference, -0.8 PP; P = .605). The adjusted difference-in-differences estimate was a 3.9 PP reduction in racial disparity (95% CI, 0.1-7.7 PP; P = .045). CONCLUSIONS:Medicaid expansion was associated with reduced Black-White racial disparities in receipt of timely systemic treatment for patients with advanced or metastatic cancers.

OBJECTIVE/UNASSIGNED:This retrospective observational study described baseline characteristics, real-world treatment patterns, and outcomes among patients with metastatic breast cancer treated with abemaciclib in the United States. METHODS/UNASSIGNED:De-identified electronic health record-derived data were used to describe patients who began abemaciclib treatment on or after 30 June 2016 and ≥4 months before data cutoff (31 December 2018). Real-world response (rwR) and real-world progression assessments were abstracted from clinical documentation. Descriptive statistics were used to calculate the real-world best response. The Kaplan-Meier method estimated real-world time to first response (rwTTFR) and real-world progression-free survival (rwPFS). RESULTS/UNASSIGNED:The median age of 118 female patients at abemaciclib initiation was 66.5 years (interquartile range, 57.0, 73.0). The breakdown of patients who received abemaciclib in first, second, third, or later lines was 28.8%, 21.2%, 20.3%, and 29.7%, respectively. Patients received abemaciclib as monotherapy (12.7%) or in combination with endocrine therapy: fulvestrant (59.3%); aromatase inhibitor (22.9%); aromatase inhibitor and fulvestrant (5.1%). There were 68 patients (57.6%) with ≥1 rwR assessment: 41.2% with a real-world complete response or real-world partial response. Median rwTTFR was 3.6 months (95% confidence interval, 3.5, 5.2). Twelve-month rwPFS probability was 61.7%. CONCLUSIONS/UNASSIGNED:This study represents utilization and outcomes associated with abemaciclib approximately 1 year following FDA approval. Treatment patterns demonstrated heterogeneity and, as in clinical trials, patients appeared to benefit from abemaciclib treatment in the real world. More research investigating outcomes associated with abemaciclib treatment is needed, with larger samples and longer follow-up to enable closer evaluation by subgroup, regimen, and line of therapy.

Sorafenib has an antitumor activity in patients with radioactive iodine-refractory differentiated thyroid carcinoma (RAIR-DTC). Prior research has implicated signaling through the MAPK and AKT/PI3K pathways in the progression of DTC. To assess whether the activity of these pathways is predictive of response to sorafenib, we retrospectively studied molecular tumor markers from these two pathways from a phase 2 study of sorafenib in RAIR-DTC. Tumor samples from 40 of 53 DTC subjects obtained prior to initiation of sorafenib were immunostained with DAB-labeled antibodies to phospho-AKT (pAKT), phospho-ERK (pERK), and phospho-S6 (pS6). BRAFV600E genetic mutation analysis was performed on all samples. Expression levels and mutational status were compared to response and progression-free survival (PFS) for each patient. Low tumor expression of nuclear pAKT was associated with partial response to sorafenib (p < 0.01). Patients with nuclear pAKT expression that was below the median for our sample were more than three times as likely to have a partial response as patients with equal to or above median expression. There was no correlation between tumor expression of nuclear pERK or pS6 and response. Endothelial cell and pericyte expression of pERK, pAKT, and pS6 were not predictive of response. There was no correlation between BRAFV600E mutation status and partial response. No correlation was observed between either the expression of pAKT, pERK, or pS6, or the presence of the BRAFV600E mutation, and PFS. In conclusion, lower tumor expression of nuclear pAKT was associated with higher rate of response to sorafenib. This observation justifies evaluation of combination therapy with sorafenib and an inhibitor of the PI3K/AKT signaling pathway in RAIR-DTC.