Faculty Bibliography

Introduction: Clostridium difficile infection (CDI) is one of the most common hospital-acquired infections leading to prolonged hospitalization and significant morbidity. Only a few prior studies have developed predictive risk models for CDI and all but one have utilized logistic regression (LR) models to identify risk factors. Automated machine learning (AutoML) programs consistently outperform standard LR models in non-medical contexts. This study aims to investigate the utility of AutoML methods in developing a model for post-operative CDI prediction.
Method(s): We used an AutoML system developed by Amazon, Autogluon v0.3.1, to evaluate the prediction accuracy of post-surgical CDI using the 2016-2018 ACS NSQIP database. A total of A total of 3,049,617 patients and 79 pre-operative features were included in the model. Post-operative CDI was defined as CDI within 30 days of surgery. Models were trained for 4 hours to optimize performance on the Brier score, with lower being better. Validation of all performance metrics was done using the 2019 NSQIP database.
Result(s): 0.36% of the patients (n = 11,001) developed post-operative CDI. Brier scores were calculated for each model with the top performing model being an ensembled neural net model having a Brier score of 0.0027 on the test set. The corresponding AUROC and AUC-PR was 0.840 and 0.015 respectively (Figure).
Conclusion(s): The models generated via AutoML to predict post-operative CDI had discriminatory characteristics greater than or equal to those models reported in the literature. Future post-operative CDI models may benefit from automated machine learning techniques

BACKGROUND:Residents receive infrequent feedback on their clinical reasoning (CR) documentation. While machine learning (ML) and natural language processing (NLP) have been used to assess CR documentation in standardized cases, no studies have described similar use in the clinical environment. OBJECTIVE:The authors developed and validated using Kane's framework a ML model for automated assessment of CR documentation quality in residents' admission notes. DESIGN, PARTICIPANTS, MAIN MEASURES/UNASSIGNED:Internal medicine residents' and subspecialty fellows' admission notes at one medical center from July 2014 to March 2020 were extracted from the electronic health record. Using a validated CR documentation rubric, the authors rated 414 notes for the ML development dataset. Notes were truncated to isolate the relevant portion; an NLP software (cTAKES) extracted disease/disorder named entities and human review generated CR terms. The final model had three input variables and classified notes as demonstrating low- or high-quality CR documentation. The ML model was applied to a retrospective dataset (9591 notes) for human validation and data analysis. Reliability between human and ML ratings was assessed on 205 of these notes with Cohen's kappa. CR documentation quality by post-graduate year (PGY) was evaluated by the Mantel-Haenszel test of trend. KEY RESULTS/RESULTS:The top-performing logistic regression model had an area under the receiver operating characteristic curve of 0.88, a positive predictive value of 0.68, and an accuracy of 0.79. Cohen's kappa was 0.67. Of the 9591 notes, 31.1% demonstrated high-quality CR documentation; quality increased from 27.0% (PGY1) to 31.0% (PGY2) to 39.0% (PGY3) (p < .001 for trend). Validity evidence was collected in each domain of Kane's framework (scoring, generalization, extrapolation, and implications). CONCLUSIONS:The authors developed and validated a high-performing ML model that classifies CR documentation quality in resident admission notes in the clinical environment-a novel application of ML and NLP with many potential use cases.

BACKGROUND: We previously developed and validated a predictive model to help clinicians identify hospitalized adults with coronavirus disease 2019 (COVID-19) who may be ready for discharge given their low risk of adverse events. Whether this algorithm can prompt more timely discharge for stable patients in practice is unknown. OBJECTIVES/OBJECTIVE: The aim of the study is to estimate the effect of displaying risk scores on length of stay (LOS). METHODS: We integrated model output into the electronic health record (EHR) at four hospitals in one health system by displaying a green/orange/red score indicating low/moderate/high-risk in a patient list column and a larger COVID-19 summary report visible for each patient. Display of the score was pseudo-randomized 1:1 into intervention and control arms using a patient identifier passed to the model execution code. Intervention effect was assessed by comparing LOS between intervention and control groups. Adverse safety outcomes of death, hospice, and re-presentation were tested separately and as a composite indicator. We tracked adoption and sustained use through daily counts of score displays. RESULTS: Enrolling 1,010 patients from May 15, 2020 to December 7, 2020, the trial found no detectable difference in LOS. The intervention had no impact on safety indicators of death, hospice or re-presentation after discharge. The scores were displayed consistently throughout the study period but the study lacks a causally linked process measure of provider actions based on the score. Secondary analysis revealed complex dynamics in LOS temporally, by primary symptom, and hospital location. CONCLUSION/CONCLUSIONS: An AI-based COVID-19 risk score displayed passively to clinicians during routine care of hospitalized adults with COVID-19 was safe but had no detectable impact on LOS. Health technology challenges such as insufficient adoption, nonuniform use, and provider trust compounded with temporal factors of the COVID-19 pandemic may have contributed to the null result. TRIAL REGISTRATION/BACKGROUND: ClinicalTrials.gov identifier: NCT04570488.

Testing multiple treatments for heterogeneous (varying) effectiveness with respect to many underlying risk factors requires many pairwise tests; we would like to instead automatically discover and visualize patient archetypes and predictors of treatment effectiveness using multitask machine learning. In this paper, we present a method to estimate these heterogeneous treatment effects with an interpretable hierarchical framework that uses additive models to visualize expected treatment benefits as a function of patient factors (identifying personalized treatment benefits) and concurrent treatments (identifying combinatorial treatment benefits). This method achieves state-of-the-art predictive power for COVID-19 in-hospital mortality and interpretable identification of heterogeneous treatment benefits. We first validate this method on the large public MIMIC-IV dataset of ICU patients to test recovery of heterogeneous treatment effects. Next we apply this method to a proprietary dataset of over 3000 patients hospitalized for COVID-19, and find evidence of heterogeneous treatment effectiveness predicted largely by indicators of inflammation and thrombosis risk: patients with few indicators of thrombosis risk benefit most from treatments against inflammation, while patients with few indicators of inflammation risk benefit most from treatments against thrombosis. This approach provides an automated methodology to discover heterogeneous and individualized effectiveness of treatments.

Machine learning applications promise to augment clinical capabilities and at least 64 models have already been approved by the US Food and Drug Administration. These tools are developed, shared, and used in an environment in which regulations and market forces remain immature. An important consideration when evaluating this environment is the introduction of open-source solutions in which innovations are freely shared; such solutions have long been a facet of digital culture. We discuss the feasibility and implications of open-source machine learning in a health care infrastructure built upon proprietary information. The decreased cost of development as compared to drugs and devices, a longstanding culture of open-source products in other industries, and the beginnings of machine learning-friendly regulatory pathways together allow for the development and deployment of open-source machine learning models. Such tools have distinct advantages including enhanced product integrity, customizability, and lower cost, leading to increased access. However, significant questions regarding engineering concerns about implementation infrastructure and model safety, a lack of incentives from intellectual property protection, and nebulous liability rules significantly complicate the ability to develop such open-source models. Ultimately, the reconciliation of open-source machine learning and the proprietary information-driven health care environment requires that policymakers, regulators, and health care organizations actively craft a conducive market in which innovative developers will continue to both work and collaborate.

Background: Risk calculators to predict perioperative major adverse cardiovascular and cerebrovascular events (MACCE) often rely on logistic regression (LR) analysis. Automated machine learning (AutoML) processes regularly outperform regular machine learning (ML) and LR methods for predictive accuracy. Commercial AutoML systems have not yet been applied to predict perioperative MACCE after non-cardiac and cardiac surgeries.
Method(s): We used a commercial AutoML system, Google AutoML Tables (GAMLT), to predict perioperative MACCE in the 2019 ACS NSQIP database. MACCE was defined as death, myocardial infarction, cardiac arrest, or stroke. Default AutoML settings were used, with 80% of cases randomly selected for training, 10% for validation, and 10% for testing. Global feature importance was determined through the Shapley method. Two models were generated: Model 1 included 81 pre-operative features; Model 2 included the top 21 features from Model 1 and was independently validated with 2016-2018 NSQIP data.
Result(s): Model 1 yielded an area under the receiver operating characteristic (AUROC) of 0.934 in the 2019 ACS NSQIP dataset. Model 2, an ensemble of 25 feedforward neural net models, yielded an AUROC of 0.914-0.920 for MACCE in 2016-2019 (Figure).
Conclusion(s): Compared to existing risk calculators, GAMLT-derived models offered novel feature detection and comparable predictive performance for MACCE. AutoML analyses should be considered for risk estimation of perioperative MACCE. [Formula presented]
Copyright

BACKGROUND:Residents and fellows receive little feedback on their clinical reasoning documentation. Barriers include lack of a shared mental model and variability in the reliability and validity of existing assessment tools. Of the existing tools, the IDEA assessment tool includes a robust assessment of clinical reasoning documentation focusing on four elements (interpretive summary, differential diagnosis, explanation of reasoning for lead and alternative diagnoses) but lacks descriptive anchors threatening its reliability. OBJECTIVE:Our goal was to develop a valid and reliable assessment tool for clinical reasoning documentation building off the IDEA assessment tool. DESIGN, PARTICIPANTS, AND MAIN MEASURES/UNASSIGNED:The Revised-IDEA assessment tool was developed by four clinician educators through iterative review of admission notes written by medicine residents and fellows and subsequently piloted with additional faculty to ensure response process validity. A random sample of 252 notes from July 2014 to June 2017 written by 30 trainees across several chief complaints was rated. Three raters rated 20% of the notes to demonstrate internal structure validity. A quality cut-off score was determined using Hofstee standard setting. KEY RESULTS/RESULTS:The Revised-IDEA assessment tool includes the same four domains as the IDEA assessment tool with more detailed descriptive prompts, new Likert scale anchors, and a score range of 0-10. Intraclass correlation was high for the notes rated by three raters, 0.84 (95% CI 0.74-0.90). Scores ≥6 were determined to demonstrate high-quality clinical reasoning documentation. Only 53% of notes (134/252) were high-quality. CONCLUSIONS:The Revised-IDEA assessment tool is reliable and easy to use for feedback on clinical reasoning documentation in resident and fellow admission notes with descriptive anchors that facilitate a shared mental model for feedback.

Predictive models may be particularly beneficial to clinicians when they face uncertainty and seek to develop a mental model of disease progression, but we know little about the post-implementation effects of predictive models on clinicians' experience of their work. Combining survey and interview methods, we found that providers using a predictive algorithm reported being significantly less uncertain and better able to anticipate, plan and prepare for patient discharge than non-users. The tool helped hospitalists form and develop confidence in their mental models of a novel disease (Covid-19). Yet providers' attention to the predictive tool declined as their confidence in their own mental models grew. Predictive algorithms that not only offer data but also provide feedback on decisions, thus supporting providers' motivation for continuous learning, hold promise for more sustained provider attention and cognition augmentation.

BACKGROUND:It is estimated that up to 40% of Alzheimer's Disease and Related Dementias cases can be prevented or delayed by addressing modifiable factors including those that influence vascular risk (hypertension, obesity, smoking, physical activity, diabetes). Prevention may be particularly important in the vascular dementia subtypes. Despite the supporting evidence, the rates of medical therapy to reduce vascular risk are not well described. METHOD/METHODS:We assessed the utilization of statins, aspirin, and blood pressure (BP) medications in adults age ≥65 years cared for at NYU Langone Health, as recorded in the electronic health record. We included two cohorts: cohort 1 included patients who were diagnosed with vascular dementia (VaD) at NYU Langone Barlow Center for Memory Evaluation between January 1, 2015 and June 24, 2019. Cohort 2 extended the inclusion to seniors with VD diagnosis by any NYU Langone physician. Definitions for vascular dementia, the covariates assessed, and medications that we included in each category are shown in Tables 1-3. RESULT/RESULTS:We included 419 and 3745 patients in cohort 1 and cohort 2, respectively. Table 4 shows the characteristics and medication adherence in cohorts 1 and 2. In cohort 1, the prescription rates for statins, aspirin, and BP medications were 66%, 66%, 70%. In cohort 2, the rates for statin, aspirin, and BP medications were 56%, 46%, and 65%, respectively. The differences between prescription rates in cohort 1 and 2 for the three medication groups were statistically significant (p<0.05). CONCLUSION/CONCLUSIONS:Our analysis of the utilization of cardiovascular medications among patients with vascular dementia illuminates potential gaps both among patients who receive care at specialty clinics, as well as the overall population with vascular dementia. The rates of medication utilization are higher for patients under the care of cognitive neurologists. Electronic health records can help identify large cohorts of patients who may benefit from improved access to preventative measures including cardiovascular medications.