Applying A/B Testing to Clinical Decision Support: Rapid Randomized Controlled Trials
BACKGROUND:Clinical decision support (CDS) is a valuable feature of electronic health records (EHRs) designed to improve quality and safety. However, due to the complexities of system design and inconsistent results, CDS tools may inadvertently increase alert fatigue and contribute to physician burnout. A/B testing, or rapid-cycle randomized tests, is a useful method that can be applied to the EHR in order to rapidly understand and iteratively improve design choices embedded within CDS tools. OBJECTIVE:This paper describes how rapid randomized controlled trials (RCTs) embedded within EHRs can be used to quickly ascertain the superiority of potential CDS design changes to improve their usability, reduce alert fatigue, and promote quality of care. METHODS:A multistep process combining tools from user-centered design, A/B testing, and implementation science was used to understand, ideate, prototype, test, analyze, and improve each candidate CDS. CDS engagement metrics (alert views, acceptance rates) were used to evaluate which CDS version is superior. RESULTS:To demonstrate the impact of the process, 2 experiments are highlighted. First, after multiple rounds of usability testing, a revised CDS influenza alert was tested against usual care CDS in a rapid (~6 weeks) RCT. The new alert text resulted in minimal impact on reducing firings per patients per day, but this failure triggered another round of review that identified key technical improvements (ie, removal of dismissal button and firings in procedural areas) that led to a dramatic decrease in firings per patient per day (23.1 to 7.3). In the second experiment, the process was used to test 3 versions (financial, quality, regulatory) of text supporting tobacco cessation alerts as well as 3 supporting images. Based on 3 rounds of RCTs, there was no significant difference in acceptance rates based on the framing of the messages or addition of images. CONCLUSIONS:These experiments support the potential for this new process to rapidly develop, deploy, and rigorously evaluate CDS within an EHR. We also identified important considerations in applying these methods. This approach may be an important tool for improving the impact of and experience with CDS. TRIAL REGISTRATION/BACKGROUND:Flu alert trial: ClinicalTrials.gov NCT03415425; https://clinicaltrials.gov/ct2/show/NCT03415425. Tobacco alert trial: ClinicalTrials.gov NCT03714191; https://clinicaltrials.gov/ct2/show/NCT03714191.
The transformation of patient-clinician relationships with AI-based medical advice
The transformation of patient-clinician relationships with AI-based medical advice is discussed. many new tools are based on entirely new "˜black-box"™ AI-based technologies, whose inner workings are likely not fully understood by patients or clinicians. Most patients with Type 1 diabetes now use continuous glucose monitors and insulin pumps to tightly manage their disease. Their clinicians carefully review the data streams from both devices to recommend dosage adjustments. Recently new automated recommender systems to monitor and analyze food intake, insulin doses, physical activity, and other factors influencing glucose levels, and provide data-intensive, AI-based recommendations on how to titrate the regimen, are in different stages of FDA approval using "˜black box"™ technology, which is an alluring proposition for a clinical scenario that requires identification of meaningful patterns in complex and voluminous data.
Effectiveness of an Integrated Engagement Support System to Facilitate Patient Use of Digital Diabetes Prevention Programs: Protocol for a Randomized Controlled Trial
BACKGROUND:Digital diabetes prevention programs (dDPPs) are effective behavior change tools to prevent disease progression in patients at risk for diabetes. At present, these programs are poorly integrated into existing health information technology infrastructure and clinical workflows, resulting in barriers to provider-level knowledge of, interaction with, and support of patients who use dDPPs. Tools that can facilitate patient-provider interaction around dDPPs may contribute to improved patient engagement and adherence to these programs and improved health outcomes. OBJECTIVE:This study aims to use a rigorous, user-centered design (UCD) methodology to develop a theory-driven system that supports patient engagement with dDPPs and their primary care providers with their care. METHODS:at 6 and 12 months. Secondary outcomes will be patient engagement (use and activity) in the dDPP. The mediator variables (self-efficacy, digital health literacy, and patient-provider relationship) will be measured. RESULTS:The project was initiated in 2018 and funded in September 2019. Enrollment and data collection for phase 1 began in September 2019 under an Institutional Review Board quality improvement waiver granted in July 2019. As of December 2020, 27 patients have been enrolled and first results are expected to be submitted for publication in early 2021. The study received Institutional Review Board approval for phases 2 and 3 in December 2020, and phase 2 enrollment is expected to begin in early 2021. CONCLUSIONS:Our findings will provide guidance for the design and development of technology to integrate dDPP platforms into existing clinical workflows. This will facilitate patient engagement in digital behavior change interventions and provider engagement in patients' use of dDPPs. Integrated clinical tools that can facilitate patient-provider interaction around dDPPs may contribute to improved patient adherence to these programs and improved health outcomes by addressing barriers faced by both patients and providers. Further evaluation with pilot testing and a clinical trial will assess the effectiveness and implementation of these tools. TRIAL REGISTRATION/BACKGROUND:ClinicalTrials.gov NCT04049500; https://clinicaltrials.gov/ct2/show/NCT04049500. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)/UNASSIGNED:DERR1-10.2196/26750.
Giving Your Electronic Health Record a Checkup After COVID-19: A Practical Framework for Reviewing Clinical Decision Support in Light of the Telemedicine Expansion
BACKGROUND:The transformation of health care during COVID-19, with the rapid expansion of telemedicine visits, presents new challenges to chronic care and preventive health providers. Clinical decision support (CDS) is critically important to chronic care providers, and CDS malfunction is common during times of change. It is essential to regularly reassess an organization's ambulatory CDS program to maintain care quality. This is especially true after an immense change, like the COVID-19 telemedicine expansion. OBJECTIVE:Our objective is to reassess the ambulatory CDS program at a large academic medical center in light of telemedicine's expansion in response to the COVID-19 pandemic. METHODS:Our clinical informatics team devised a practical framework for an intrapandemic ambulatory CDS assessment focused on the impact of the telemedicine expansion. This assessment began with a quantitative analysis comparing CDS alert performance in the context of in-person and telemedicine visits. Board-certified physician informaticists then completed a formal workflow review of alerts with inferior performance in telemedicine visits. Informaticists then reported on themes and optimization opportunities through the existing CDS governance structure. RESULTS:Our assessment revealed that 10 of our top 40 alerts by volume were not firing as expected in telemedicine visits. In 3 of the top 5 alerts, providers were significantly less likely to take action in telemedicine when compared to office visits. Cumulatively, alerts in telemedicine encounters had an action taken rate of 5.3% (3257/64,938) compared to 8.3% (19,427/233,636) for office visits. Observations from a clinical informaticist workflow review included the following: (1) Telemedicine visits have different workflows than office visits. Some alerts developed for the office were not appearing at the optimal time in the telemedicine workflow. (2) Missing clinical data is a common reason for the decreased alert firing seen in telemedicine visits. (3) Remote patient monitoring and patient-reported clinical data entered through the portal could replace data collection usually completed in the office by a medical assistant or registered nurse. CONCLUSIONS:In a large academic medical center at the pandemic epicenter, an intrapandemic ambulatory CDS assessment revealed clinically significant CDS malfunctions that highlight the importance of reassessing ambulatory CDS performance after the telemedicine expansion.
Telemedicine and Healthcare Disparities: A cohort study in a large healthcare system in New York City during COVID-19
OBJECTIVE:Through the coronavirus disease 2019 (COVID-19) pandemic, telemedicine became a necessary entry point into the process of diagnosis, triage and treatment. Racial and ethnic disparities in health care have been well documented in COVID-19 with respect to risk of infection and in-hospital outcomes once admitted, and here we assess disparities in those who access healthcare via telemedicine for COVID-19 . MATERIALS AND METHODS/METHODS:Electronic health record data of patients at New York University Langone Health between March 19th and April 30, 2020 were used to conduct descriptive and multilevel regression analyses with respect to visit type (telemedicine or in-person), suspected COVID diagnosis and COVID test results. RESULTS:Controlling for individual and community-level attributes, Black patients had 0.6 times the adjusted odds (95%CI:0.58-0.63) of accessing care through telemedicine compared to white patients, though they are increasingly accessing telemedicine for urgent care, driven by a younger and female population. COVID diagnoses were significantly more likely for Black versus white telemedicine patients. DISCUSSION/CONCLUSIONS:There are disparities for Black patients accessing telemedicine, however increased uptake by young, female Black patients. Mean income and decreased mean household size of Zip code were also significantly related to telemedicine use. CONCLUSION/CONCLUSIONS:Telemedicine access disparities reflect those in in-person healthcare access. Roots of disparate use are complex and reflect individual, community, and structural factors, including their intersection; many of which are due to systemic racism. Evidence regarding disparities that manifest through telemedicine can be used to inform tool design and systemic efforts to promote digital health equity.
Implementation of a behavioral economics electronic health record (BE-EHR) module to optimize diabetes management in older adults [Meeting Abstract]
Using human-centered design to optimize shared multi-use clinical work spaces for clinicians [Meeting Abstract]
STATEMENT OF PROBLEM OR QUESTION (ONE SENTENCE): In the transition away from traditional doctors' offices, how can we optimize shared multi-use clinical spaces to serve clinicians' needs LEARNING OBJECTIVES 1: Identify ways in which a practice that relies upon shared clinical spaces can remain familiar and effective for clinical work. LEARNING OBJECTIVES 2: Determine how might technology help clinicians develop a sense of belonging, professional pride, and patient rapport in multi-use spaces by allowing them to display personal information and patient education materials related to their practice. DESCRIPTION OF PROGRAM/INTERVENTION, INCLUDING ORGANIZATIONAL CONTEXT (E.G. INPATIENT VS. OUTPATIENT, PRACTICE OR COMMUNITY CHARACTERISTICS): The traditional doctor's office is being rapidly replaced by multi-use clinical environments that combine exam rooms with shared touchdown spaces, promoting efficient use of space & team-based care approach while utilizing network technologies. While potentially efficient & lower-cost, there's a need to assess the impact of these configurations on clinician workflows, professional identity & explore opportunities to improve their build and aesthetics. We conducted need assessment interviews with 9 clinicians, health technologists, 2 operational leaders, shadowed 3 clinicians & conducted 4 site visits across various clinical practices. We then issued a 10-question survey and conducted 2 HCD workshops with 12 clinicians to understand the new conditions of clinical work, their impact on clinicians' professional & personal identity, practice habits, to identify areas for potential optimization to improve clinical workflow & experience. Workshops were divided in three phases: explore, ideate and create. MEASURES OF SUCCESS (DISCUSS QUALITATIVE AND/OR QUANTITATIVEMETRICSWHICHWILL BE USEDTOEVALUATE PROGRAM/INTERVENTION): We report qualitative success metrics used to evaluate the results of the HCD workshops: 1. Understanding of what shared multi-use work spaces mean to participating clinicians. 2. Identified needs, potential concerns and pain points of clinicians and stakeholders. 3. Group generation of potential solutions without bias towards feasibility. 4. Described solutions using quick prototyping tools. FINDINGS TO DATE (IT IS NOT SUFFICIENT TO STATE FINDINGS WILL BE DISCUSSED): Clinicians identified the lack of customization and capability for sharing information about their areas of expertise and tailored patient education materials as the most significant problem, and had privacy concerns about sharing personal information on a digital display. Potential solutions include customizable content display controlled by patients that fosters engagement, exploring education materials, patient testimonials, information about the care team and wait time as well as patient-specific information, such as labs and imaging. KEY LESSONS FOR DISSEMINATION (WHAT CAN OTHERS TAKE AWAY FOR IMPLEMENTATION TO THEIR PRACTICE OR COMMUNITY): The use of the HCD principles helped us better understand the challenges of multi-use spaces for clinicians, and identify potential technology solutions for data sharing, patient education, personalization, and efficiencies. It is crucial to design these spaces and choose appropriate technology solutions that will help reduce patients' anxiety by ensuring privacy, comfort, thorough understanding of care plans and boost collaborative care decision making between clinicians and patients
Impact of Clinical Decision Support on Antibiotic Prescribing for Acute Respiratory Infections: a Cluster Randomized Implementation Trial
BACKGROUND:Clinical decision support (CDS) is a promising tool for reducing antibiotic prescribing for acute respiratory infections (ARIs). OBJECTIVE:To assess the impact of previously effective CDS on antibiotic-prescribing rates for ARIs when adapted and implemented in diverse primary care settings. DESIGN/METHODS:Cluster randomized clinical trial (RCT) implementing a CDS tool designed to guide evidence-based evaluation and treatment of streptococcal pharyngitis and pneumonia. SETTING/METHODS:Two large academic health system primary care networks with a mix of providers. PARTICIPANTS/METHODS:All primary care practices within each health system were invited. All providers within participating clinic were considered a participant. Practices were randomized selection to a control or intervention group. INTERVENTIONS/METHODS:Intervention practice providers had access to an integrated clinical prediction rule (iCPR) system designed to determine the risk of bacterial infection from reason for visit of sore throat, cough, or upper respiratory infection and guide evidence-based evaluation and treatment. MAIN OUTCOME(S)/UNASSIGNED:Change in overall antibiotic prescription rates. MEASURE(S)/UNASSIGNED:Frequency, rates, and type of antibiotics prescribed in intervention and controls groups. RESULTS:33 primary care practices participated with 541 providers and 100,573 patient visits. Intervention providers completed the tool in 6.9% of eligible visits. Antibiotics were prescribed in 35% and 36% of intervention and control visits, respectively, showing no statistically significant difference. There were also no differences in rates of orders for rapid streptococcal tests (RR, 0.94; PÂ =â€‰0.11) or chest X-rays (RR, 1.01; PÂ =â€‰0.999) between groups. CONCLUSIONS:The iCPR tool was not effective in reducing antibiotic prescription rates for upper respiratory infections in diverse primary care settings. This has implications for the generalizability of CDS tools as they are adapted to heterogeneous clinical contexts. TRIAL REGISTRATION/BACKGROUND:Clinicaltrials.gov (NCT02534987). Registered August 26, 2015 at https://clinicaltrials.gov.
Implementation of a Behavioral Economics Electronic Health Record (BE-EHR) Module to Reduce Overtreatment of Diabetes in Older Adults
BACKGROUND:Intensive glycemic control is of unclear benefit and carries increased risk for older adults with diabetes. The American Geriatrics Society's (AGS) Choosing Wisely (CW) guideline promotes less aggressive glycemic targets and reduction in pharmacologic therapy for older adults with type II diabetes. Meanwhile, behavioral economic (BE) approaches offer promise in influencing hard-to-change behavior, and previous studies have shown the benefits of using electronic health record (EHR) technology to encourage guideline adherence. OBJECTIVE:This study aimed to develop and pilot test an intervention that leverages BE with EHR technology to promote appropriate diabetes management in older adults. DESIGN/METHODS:A pilot study within the New York University Langone Health (NYULH) EHR and Epic system to deliver BE-inspired nudges at five NYULH clinics at varying time points from July 12, 2018, through October 31, 2019. PARTICIPANTS/METHODS:Clinicians across five practices in the NYULH system whose patients were older adults (age 76 and older) with type II diabetes. INTERVENTIONS/METHODS:A BE-EHR module comprising six nudges was developed through a series of design workshops, interviews, user-testing sessions, and clinic visits. BE principles utilized in the nudges include framing, social norming, accountable justification, defaults, affirmation, and gamification. MAIN MEASURES/METHODS:Patient-level CW compliance. KEY RESULTS/RESULTS:CW compliance increased 5.1% from a 16-week interval at baseline to a 16-week interval post intervention. From February 14 to June 5, 2018 (prior to the first nudge launch in Vanguard clinics), CW compliance for 1278 patients was mean (95% CI)-16.1% (14.1%, 18.1%). From July 3 to October 22, 2019 (after BE-EHR module launch at all five clinics), CW compliance for 680 patients was 21.2% (18.1%, 24.3%). CONCLUSIONS:The BE-EHR module shows promise for promoting the AGS CW guideline and improving diabetes management in older adults. A randomized controlled trial will commence to test the effectiveness of the intervention across 66 NYULH clinics. NIH TRIAL REGISTRY NUMBER/UNASSIGNED:NCT03409523.
Good for the Many or Best for the Few? A Dilemma in the Design of Algorithmic Advice
Applications in a range of domains, including route planning and well-being, offer advice based on the social information available in prior users' aggregated activity. When designing these applications, is it better to offer: a) advice that if strictly adhered to is more likely to result in an individual successfully achieving their goal, even if fewer users will choose to adopt it? or b) advice that is likely to be adopted by a larger number of users, but which is sub-optimal with regard to any particular individual achieving their goal? We identify this dilemma, characterized as Goal-Directed vs. Adoption-Directed advice, and investigate the design questions it raises through an online experiment undertaken in four advice domains (financial investment, making healthier lifestyle choices, route planning, training for a 5k run), with three user types, and across two levels of uncertainty. We report findings that suggest a preference for advice favoring individual goal attainment over higher user adoption rates, albeit with significant variation across advice domains; and discuss their design implications.