Faculty Bibliography

BACKGROUND: We previously developed and validated a predictive model to help clinicians identify hospitalized adults with coronavirus disease 2019 (COVID-19) who may be ready for discharge given their low risk of adverse events. Whether this algorithm can prompt more timely discharge for stable patients in practice is unknown. OBJECTIVES/OBJECTIVE: The aim of the study is to estimate the effect of displaying risk scores on length of stay (LOS). METHODS: We integrated model output into the electronic health record (EHR) at four hospitals in one health system by displaying a green/orange/red score indicating low/moderate/high-risk in a patient list column and a larger COVID-19 summary report visible for each patient. Display of the score was pseudo-randomized 1:1 into intervention and control arms using a patient identifier passed to the model execution code. Intervention effect was assessed by comparing LOS between intervention and control groups. Adverse safety outcomes of death, hospice, and re-presentation were tested separately and as a composite indicator. We tracked adoption and sustained use through daily counts of score displays. RESULTS: Enrolling 1,010 patients from May 15, 2020 to December 7, 2020, the trial found no detectable difference in LOS. The intervention had no impact on safety indicators of death, hospice or re-presentation after discharge. The scores were displayed consistently throughout the study period but the study lacks a causally linked process measure of provider actions based on the score. Secondary analysis revealed complex dynamics in LOS temporally, by primary symptom, and hospital location. CONCLUSION/CONCLUSIONS: An AI-based COVID-19 risk score displayed passively to clinicians during routine care of hospitalized adults with COVID-19 was safe but had no detectable impact on LOS. Health technology challenges such as insufficient adoption, nonuniform use, and provider trust compounded with temporal factors of the COVID-19 pandemic may have contributed to the null result. TRIAL REGISTRATION/BACKGROUND: ClinicalTrials.gov identifier: NCT04570488.

Background: Optimization of mechanical ventilation (MV) in patients with cardiac arrest (CA) may help improve outcomes in these patients. We sought to investigate the association between hyperoxia, PCO2, and positive end-expiratory pressure (PEEP) with mortality in patients with CA.
Method(s): Patients admitted to our medical center CICU from 2001 through 2012 (MIMIC-III database) who received MV with available information on MV parameters and had arterial blood gases sampling were included. Hyperoxia was defined as time-weighted mean of PaO2 >120 mmHg and non-hyperoxia as PaO2 <=120 mmHg, while Hypercarbia was defined as PCO2 >35 mmHg during CICU admission. The primary outcome was inhospital mortality. Multivariable logistic regression was used to assess the association between hyperoxia and in-hospital mortality adjusted for age, female sex, Oxford Acute Severity of Illness Score, creatinine, lactate, pH, PaO2/FiO2 ratio, PCO2, PEEP, and time spent on PEEP.
Result(s): Among 136 patients, PaO2 = 139+/-55 mmHg, PCO2 = 39+/-10 mmHg, and PEEP = 6.4+/-2.2cmH2O. Unadjusted mortality was higher in the hyperoxic group (51.4%) compared to the non-hyperoxic group (29.0%) (long rank test p=0.0034, figure). In multivariable analysis, hyperoxia was independently associated with higher in-hospital mortality (OR 4.046, 95% CI: 1.501-10.907, p=0.0057). Additionally, there was no association between the presence of hypercarbia and in-hospital mortality (OR 0.896, 95% CI: 0.319 to 2.521, p=0.836) nor when PCO2 was analyzed as a continuous variable (OR 1.063 per 1 mmHg increase in CO2, 95% CI: 0.111- 10.145, p=0.957). Similarly, there was no assocation between PEEP and in-hospital mortality (OR 1.012 per 1cmH2O increase, 95% CI: 0.807 to 1.270, p=0.917). Post-hoc analysis with PaO2 as a continuous variable was consistent with the primary analysis (OR 1.214 per 10 mmHg increase in PaO2, 95% CI: 1.059-1.391, p=0.005).
Conclusion(s): In patients with CA, hyperoxia was associated with increased mortality, while PCO2 and PEEP levels were not. Optimal MV parameters are important in the management of patients with CA. Further research is warranted to confirm this association and explore the mechanisms behind these observations. These studies can help establish the best MV strategies for patients with CA

BACKGROUND:Automated systems that use machine learning to estimate a patient's risk of death are being developed to influence care. There remains sparse transparent reporting of model generalizability in different subpopulations especially for implemented systems. METHODS:A prognostic study included adult admissions at a multi-site, academic medical center between 2015 and 2017. A predictive model for all-cause mortality (including initiation of hospice care) within 60 days of admission was developed. Model generalizability is assessed in temporal validation in the context of potential demographic bias. A subsequent prospective cohort study was conducted at the same sites between October 2018 and June 2019. Model performance during prospective validation was quantified with areas under the receiver operating characteristic and precision recall curves stratified by site. Prospective results include timeliness, positive predictive value, and the number of actionable predictions. RESULTS:Three years of development data included 128,941 inpatient admissions (94,733 unique patients) across sites where patients are mostly white (61%) and female (60%) and 4.2% led to death within 60 days. A random forest model incorporating 9614 predictors produced areas under the receiver operating characteristic and precision recall curves of 87.2 (95% CI, 86.1-88.2) and 28.0 (95% CI, 25.0-31.0) in temporal validation. Performance marginally diverges within sites as the patient mix shifts from development to validation (patients of one site increases from 10 to 38%). Applied prospectively for nine months, 41,728 predictions were generated in real-time (median [IQR], 1.3 [0.9, 32] minutes). An operating criterion of 75% positive predictive value identified 104 predictions at very high risk (0.25%) where 65% (50 from 77 well-timed predictions) led to death within 60 days. CONCLUSION/CONCLUSIONS:Temporal validation demonstrates good model discrimination for 60-day mortality. Slight performance variations are observed across demographic subpopulations. The model was implemented prospectively and successfully produced meaningful estimates of risk within minutes of admission.

Objective/UNASSIGNED:One primary consideration when developing predictive models is downstream effects on future model performance. We conduct experiments to quantify the effects of experimental design choices, namely cohort selection and internal validation methods, on (estimated) real-world model performance. Materials and Methods/UNASSIGNED:Four years of hospitalizations are used to develop a 1-year mortality prediction model (composite of death or initiation of hospice care). Two common methods to select appropriate patient visits from their encounter history (backwards-from-outcome and forwards-from-admission) are combined with 2 testing cohorts (random and temporal validation). Two models are trained under otherwise identical conditions, and their performances compared. Operating thresholds are selected in each test set and applied to a "real-world" cohort of labeled admissions from another, unused year. Results/UNASSIGNED: = 92 148). Both selection methods produce similar performances when applied to a random test set. However, when applied to the temporally defined "real-world" set, forwards-from-admission yields higher areas under the ROC and precision recall curves (88.3% and 56.5% vs. 83.2% and 41.6%). Discussion/UNASSIGNED:A backwards-from-outcome experiment manipulates raw training data, simplifying the experiment. This manipulated data no longer resembles real-world data, resulting in optimistic estimates of test set performance, especially at high precision. In contrast, a forwards-from-admission experiment with a temporally separated test set consistently and conservatively estimates real-world performance. Conclusion/UNASSIGNED:Experimental design choices impose bias upon selected cohorts. A forwards-from-admission experiment, validated temporally, can conservatively estimate real-world performance. LAY SUMMARY/UNASSIGNED:The routine care of patients stands to benefit greatly from assistive technologies, including data-driven risk assessment. Already, many different machine learning and artificial intelligence applications are being developed from complex electronic health record data. To overcome challenges that arise from such data, researchers often start with simple experimental approaches to test their work. One key component is how patients (and their healthcare visits) are selected for the study from the pool of all patients seen. Another is how the group of patients used to create the risk estimator differs from the group used to evaluate how well it works. These choices complicate how the experimental setting compares to the real-world application to patients. For example, different selection approaches that depend on each patient's future outcome can simplify the experiment but are impractical upon implementation as these data are unavailable. We show that this kind of "backwards" experiment optimistically estimates how well the model performs. Instead, our results advocate for experiments that select patients in a "forwards" manner and "temporal" validation that approximates training on past data and implementing on future data. More robust results help gauge the clinical utility of recent works and aid decision-making before implementation into practice.

The COVID-19 pandemic has challenged front-line clinical decision-making, leading to numerous published prognostic tools. However, few models have been prospectively validated and none report implementation in practice. Here, we use 3345 retrospective and 474 prospective hospitalizations to develop and validate a parsimonious model to identify patients with favorable outcomes within 96 h of a prediction, based on real-time lab values, vital signs, and oxygen support variables. In retrospective and prospective validation, the model achieves high average precision (88.6% 95% CI: [88.4-88.7] and 90.8% [90.8-90.8]) and discrimination (95.1% [95.1-95.2] and 86.8% [86.8-86.9]) respectively. We implemented and integrated the model into the EHR, achieving a positive predictive value of 93.3% with 41% sensitivity. Preliminary results suggest clinicians are adopting these scores into their clinical workflows.