Faculty Bibliography

Pediatric sepsis imposes a significant burden of morbidity and mortality among children. While the speedy application of existing supportive care measures can substantially improve outcomes, further improvements in delivering that care require tools that go beyond recognizing sepsis and towards predicting its development. Machine learning techniques have great potential as predictive tools, but their application to pediatric sepsis has been stymied by several factors, particularly the relative rarity of its occurrence. We propose an alternate approach which focuses on predicting the provision of resuscitative care, rather than sepsis diagnoses or criteria themselves. Using three years of Emergency Department data from a large academic medical center, we developed a boosted tree model that predicts resuscitation within 6 hours of triage, and significantly outperforms existing rule-based sepsis alerts.

BACKGROUND:It is estimated that up to 40% of Alzheimer's Disease and Related Dementias cases can be prevented or delayed by addressing modifiable factors including those that influence vascular risk (hypertension, obesity, smoking, physical activity, diabetes). Prevention may be particularly important in the vascular dementia subtypes. Despite the supporting evidence, the rates of medical therapy to reduce vascular risk are not well described. METHOD/METHODS:We assessed the utilization of statins, aspirin, and blood pressure (BP) medications in adults age ≥65 years cared for at NYU Langone Health, as recorded in the electronic health record. We included two cohorts: cohort 1 included patients who were diagnosed with vascular dementia (VaD) at NYU Langone Barlow Center for Memory Evaluation between January 1, 2015 and June 24, 2019. Cohort 2 extended the inclusion to seniors with VD diagnosis by any NYU Langone physician. Definitions for vascular dementia, the covariates assessed, and medications that we included in each category are shown in Tables 1-3. RESULT/RESULTS:We included 419 and 3745 patients in cohort 1 and cohort 2, respectively. Table 4 shows the characteristics and medication adherence in cohorts 1 and 2. In cohort 1, the prescription rates for statins, aspirin, and BP medications were 66%, 66%, 70%. In cohort 2, the rates for statin, aspirin, and BP medications were 56%, 46%, and 65%, respectively. The differences between prescription rates in cohort 1 and 2 for the three medication groups were statistically significant (p<0.05). CONCLUSION/CONCLUSIONS:Our analysis of the utilization of cardiovascular medications among patients with vascular dementia illuminates potential gaps both among patients who receive care at specialty clinics, as well as the overall population with vascular dementia. The rates of medication utilization are higher for patients under the care of cognitive neurologists. Electronic health records can help identify large cohorts of patients who may benefit from improved access to preventative measures including cardiovascular medications.

Objective/UNASSIGNED:The widespread deployment of electronic health records (EHRs) has introduced new sources of error and inefficiencies to the process of ordering medications in the hospital setting. Existing work identifies orders that require pharmacy intervention by comparing them to a patient's medical records. In this work, we develop a machine learning model for identifying medication orders requiring intervention using only provider behavior and other contextual features that may reflect these new sources of inefficiencies. Materials and Methods/UNASSIGNED:Data on providers' actions in the EHR system and pharmacy orders were collected over a 2-week period in a major metropolitan hospital system. A classification model was then built to identify orders requiring pharmacist intervention. We tune the model to the context in which it would be deployed and evaluate global and local feature importance. Results/UNASSIGNED:The resultant model had an area under the receiver-operator characteristic curve of 0.91 and an area under the precision-recall curve of 0.44. Conclusions/UNASSIGNED:Providers' actions can serve as useful predictors in identifying medication orders that require pharmacy intervention. Careful model tuning for the clinical context in which the model is deployed can help to create an effective tool for improving health outcomes without using sensitive patient data.

PURPOSE:Residency programs face overwhelming numbers of residency applications, limiting holistic review. Artificial intelligence techniques have been proposed to address this challenge but have not been created. Here, a multidisciplinary team sought to develop and validate a machine learning (ML)-based decision support tool (DST) for residency applicant screening and review. METHOD:Categorical applicant data from the 2018, 2019, and 2020 residency application cycles (n = 8,243 applicants) at one large internal medicine residency program were downloaded from the Electronic Residency Application Service and linked to the outcome measure: interview invitation by human reviewers (n = 1,235 invites). An ML model using gradient boosting was designed using training data (80% of applicants) with over 60 applicant features (e.g., demographics, experiences, academic metrics). Model performance was validated on held-out data (20% of applicants). Sensitivity analysis was conducted without United States Medical Licensing Examination (USMLE) scores. An interactive DST incorporating the ML model was designed and deployed that provided applicant- and cohort-level visualizations. RESULTS:The ML model areas under the receiver operating characteristic and precision recall curves were 0.95 and 0.76, respectively; these changed to 0.94 and 0.72, respectively, with removal of USMLE scores. Applicants' medical school information was an important driver of predictions-which had face validity based on the local selection process-but numerous predictors contributed. Program directors used the DST in the 2021 application cycle to select 20 applicants for interview that had been initially screened out during human review. CONCLUSIONS:The authors developed and validated an ML algorithm for predicting residency interview offers from numerous application elements with high performance-even when USMLE scores were removed. Model deployment in a DST highlighted its potential for screening candidates and helped quantify and mitigate biases existing in the selection process. Further work will incorporate unstructured textual data through natural language processing methods.

OBJECTIVE:Heightened inflammation, dysregulated immunity, and thrombotic events are characteristic of hospitalized COVID-19 patients. Given that platelets are key regulators of thrombosis, inflammation, and immunity they represent prime candidates as mediators of COVID-19-associated pathogenesis. The objective of this study was to understand the contribution of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) to the platelet phenotype via phenotypic (activation, aggregation) and transcriptomic characterization. APPROACH AND RESULTS/UNASSIGNED:In a cohort of 3915 hospitalized COVID-19 patients, we analyzed blood platelet indices collected at hospital admission. Following adjustment for demographics, clinical risk factors, medication, and biomarkers of inflammation and thrombosis, we find platelet count, size, and immaturity are associated with increased critical illness and all-cause mortality. Bone marrow, lung tissue, and blood from COVID-19 patients revealed the presence of SARS-CoV-2 virions in megakaryocytes and platelets. Characterization of COVID-19 platelets found them to be hyperreactive (increased aggregation, and expression of P-selectin and CD40) and to have a distinct transcriptomic profile characteristic of prothrombotic large and immature platelets. In vitro mechanistic studies highlight that the interaction of SARS-CoV-2 with megakaryocytes alters the platelet transcriptome, and its effects are distinct from the coronavirus responsible for the common cold (CoV-OC43). CONCLUSIONS:Platelet count, size, and maturity associate with increased critical illness and all-cause mortality among hospitalized COVID-19 patients. Profiling tissues and blood from COVID-19 patients revealed that SARS-CoV-2 virions enter megakaryocytes and platelets and associate with alterations to the platelet transcriptome and activation profile.

OBJECTIVES/OBJECTIVE:Predictive studies play important roles in the development of models informing care for patients with COVID-19. Our concern is that studies producing ill-performing models may lead to inappropriate clinical decision-making. Thus, our objective is to summarise and characterise performance of prognostic models for COVID-19 on external data. METHODS:We performed a validation of parsimonious prognostic models for patients with COVID-19 from a literature search for published and preprint articles. Ten models meeting inclusion criteria were either (a) externally validated with our data against the model variables and weights or (b) rebuilt using original features if no weights were provided. Nine studies had internally or externally validated models on cohorts of between 18 and 320 inpatients with COVID-19. One model used cross-validation. Our external validation cohort consisted of 4444 patients with COVID-19 hospitalised between 1 March and 27 May 2020. RESULTS:Most models failed validation when applied to our institution's data. Included studies reported an average validation area under the receiver-operator curve (AUROC) of 0.828. Models applied with reported features averaged an AUROC of 0.66 when validated on our data. Models rebuilt with the same features averaged an AUROC of 0.755 when validated on our data. In both cases, models did not validate against their studies' reported AUROC values. DISCUSSION/CONCLUSIONS:Published and preprint prognostic models for patients infected with COVID-19 performed substantially worse when applied to external data. Further inquiry is required to elucidate mechanisms underlying performance deviations. CONCLUSIONS:Clinicians should employ caution when applying models for clinical prediction without careful validation on local data.

BACKGROUND: Clinical reasoning (CR) is a core component of medical training, yet residents often receive little feedback on their CR documentation. Here we describe the process of developing a machine learning (ML) algorithm for feedback on CR documentation to increase the frequency and quality of feedback in this domain.
METHOD(S): To create this algorithm, note quality first had to be rated by gold standard human rating. We selected the IDEA Assessment Tool-a note rating instrument across four domains (I=Interpretive summary, D=Differential diagnosis, E=Explanation of reasoning, A=Alternative diagnoses explained) that uses a 3-point Likert scale without descriptive anchors. To develop descriptive anchors we conducted an iterative process reviewing notes from the EHR written by medicine residents and validated the Revised-IDEA Assessment Tool using Messick's framework- content validity, response process, relation to other variables, internal structure, and consequences. Using the Hofstee standard setting method, cutoffs for high quality clinical reasoning for the IDEA and DEA scores were set. We then created a dataset of expertrated notes to create the ML algorithm. First, a natural language processing software was applied to the set of notes that enabled recognition and automatic encoding of clinical information as a diagnosis or disease (D's), a sign or symptom (E or A), or semantic qualifier (e.g. most likely). Input variables to the ML algorithm included counts of D's, E/A's, semantic qualifiers, and proximity of semantic qualifiers to disease/ diagnosis. ML output focused on DEA quality and was binarized to low or high quality CR. Finally, 200 notes were randomly selected for human validation review comparing ML output to human rated DEA score.
RESULT(S): The IDEA and DEA scores ranged from 0-10 and 0-6, respectively. IDEA score of >= 6.5 and a DEA score of >= 3 was deemed high quality. 252 notes were rated to create the dataset and 20% were rated by 3 raters with high intraclass correlation 0.84 (95% CI 0.74-0.90). 120 of these notes comprised the testing set for ML model development. The logistic regression model was the best performing model with an AUC 0.87 and a positive predictive value (PPV) of 0.65. 48 (40%) of the notes were high quality. There was substantial interrater reliability between ML output and human rating on the 200 note validation set with a Cohen's Kappa 0.64.
CONCLUSION(S): We have developed a ML algorithm for feedback on CR documentation that we hypothesize will increase the frequency and quality of feedback in this domain. We have subsequently developed a dashboard that will display the output of the ML model. Next steps will be to provide internal medicine residents' feedback on their CR documentation using this dashboard and assess the impact this has on their documentation quality. LEARNING OBJECTIVE #1: Describe the importance of high quality documentation of clinical reasoning. LEARNING OBJECTIVE #2: Identify machine learning as a novel assessment tool for feedback on clinical reasoning documentation

While the need for interpretable machine learning has been established, many common approaches are slow, lack fidelity, or hard to evaluate. Amortized explanation methods reduce the cost of providing interpretations by learning a global selector model that returns feature importances for a single instance of data. The selector model is trained to optimize the fidelity of the interpretations, as evaluated by a predictor model for the target. Popular methods learn the selector and predictor model in concert, which we show allows predictions to be encoded within interpretations. We introduce EVAL-X as a method to quantitatively evaluate interpretations and REAL-X as an amortized explanation method, which learn a predictor model that approximates the true data generating distribution given any subset of the input. We show EVAL-X can detect when predictions are encoded in interpretations and show the advantages of REAL-X through quantitative and radiologist evaluation.