Faculty Bibliography

BACKGROUND/UNASSIGNED:Usability testing is valuable for assessing a new tool or system's usefulness and ease-of-use. Several established methods of usability testing exist, including think-aloud testing. Although usability testing has been shown to be crucial for successful clinical decision support (CDS) tool development, it is often difficult to conduct across multisite development projects due to its time- and labor-intensiveness, cost, and the skills required to conduct the testing. OBJECTIVE/UNASSIGNED:Our objective was to develop a new method of usability testing that would enable efficient acquisition and dissemination of results among multiple sites. We sought to address the existing barriers to successfully completing usability testing during CDS tool development. METHODS/UNASSIGNED:We combined individual think-aloud testing and focus groups into one session and performed sessions serially across 4 sites (snowball group usability testing) to assess the usability of two CDS tools designed for use by nurses in primary and urgent care settings. We recorded each session and took notes in a standardized format. Each site shared feedback from their individual sessions with the other sites in the study so that they could incorporate that feedback into their tools prior to their own testing sessions. RESULTS/UNASSIGNED:The group testing and snowballing components of our new usability testing method proved to be highly beneficial. We identified 3 main benefits of snowball group usability testing. First, by interviewing several participants in a single session rather than individuals over the course of weeks, each site was able to quickly obtain their usability feedback. Second, combining the individualized think-aloud component with a focus group component in the same session helped study teams to more easily notice similarities in feedback among participants and to discuss and act upon suggestions efficiently. Third, conducting usability testing in series across sites allowed study teams to incorporate feedback based on previous sites' sessions prior to conducting their own testing. CONCLUSIONS/UNASSIGNED:Snowball group usability testing provides an efficient method of obtaining multisite feedback on newly developed tools and systems, while addressing barriers typically associated with traditional usability testing methods. This method can be applied to test a wide variety of tools, including CDS tools, prior to launch so that they can be efficiently optimized.

BACKGROUND:Laypeople have easy access to health information through large language models (LLMs), such as ChatGPT, and search engines, such as Google. Search engines transformed health information access, and LLMs offer a new avenue for answering laypeople's questions. OBJECTIVE:We aimed to compare the frequency of use and attitudes toward LLMs and search engines as well as their comparative relevance, usefulness, ease of use, and trustworthiness in responding to health queries. METHODS:We conducted a screening survey to compare the demographics of LLM users and nonusers seeking health information, analyzing results with logistic regression. LLM users from the screening survey were invited to a follow-up survey to report the types of health information they sought. We compared the frequency of use of LLMs and search engines using ANOVA and Tukey post hoc tests. Lastly, paired-sample Wilcoxon tests compared LLMs and search engines on perceived usefulness, ease of use, trustworthiness, feelings, bias, and anthropomorphism. RESULTS:In total, 2002 US participants recruited on Prolific participated in the screening survey about the use of LLMs and search engines. Of them, 52% (n=1045) of the participants were female, with a mean age of 39 (SD 13) years. Participants were 9.7% (n=194) Asian, 12.1% (n=242) Black, 73.3% (n=1467) White, 1.1% (n=22) Hispanic, and 3.8% (n=77) were of other races and ethnicities. Further, 1913 (95.6%) used search engines to look up health queries versus 642 (32.6%) for LLMs. Men had higher odds (odds ratio [OR] 1.63, 95% CI 1.34-1.99; P<.001) of using LLMs for health questions than women. Black (OR 1.90, 95% CI 1.42-2.54; P<.001) and Asian (OR 1.66, 95% CI 1.19-2.30; P<.01) individuals had higher odds than White individuals. Those with excellent perceived health (OR 1.46, 95% CI 1.1-1.93; P=.01) were more likely to use LLMs than those with good health. Higher technical proficiency increased the likelihood of LLM use (OR 1.26, 95% CI 1.14-1.39; P<.001). In a follow-up survey of 281 LLM users for health, most participants used search engines first (n=174, 62%) to answer health questions, but the second most common first source consulted was LLMs (n=39, 14%). LLMs were perceived as less useful (P<.01) and less relevant (P=.07), but elicited fewer negative feelings (P<.001), appeared more human (LLM: n=160, vs search: n=32), and were seen as less biased (P<.001). Trust (P=.56) and ease of use (P=.27) showed no differences. CONCLUSIONS:Search engines are the primary source of health information; yet, positive perceptions of LLMs suggest growing use. Future work could explore whether LLM trust and usefulness are enhanced by supplementing answers with external references and limiting persuasive language to curb overreliance. Collaboration with health organizations can help improve the quality of LLMs' health output.

INTRODUCTION/BACKGROUND:Early lung cancer screening (LCS) through low-dose CT (LDCT) is crucial but underused due to various barriers, including incomplete or inaccurate patient smoking data in the electronic health record and limited time for shared decision-making. The objective of this trial is to investigate a patient-centred intervention, MyLungHealth, delivered through the patient portal. The intervention is designed to improve LCS rates through increased identification of eligible patients and informed decision-making. METHODS AND ANALYSIS/METHODS:MyLungHealth is a multisite pragmatic trial, involving University of Utah Health and New York University Langone Health primary care clinics. The MyLungHealth intervention was developed using a user-centred design process, informed by patient and provider focus groups and interviews. The intervention's effectiveness will be evaluated through a patient-randomised trial, comparing the combined use of MyLungHealth and DecisionPrecision+ (a provider-focused shared decision-making intervention) against DecisionPrecision+ alone. The first study hypothesis is that among patients aged 50-79 with uncertain LCS eligibility (eg, 10-19 pack-years or unknown pack-years or unknown quit date for individuals who used to smoke), MyLungHealth eligibility questionnaires will result in increased identification of LCS-eligible patients (n~26 729 patients). The second study hypothesis is that among patients aged 50-79 with documented LCS eligibility (20+ pack-years, quit within the last 15 years if individuals who used to smoke, and no recent screening or screening discussion), MyLungHealth education will result in increased LDCT ordering (n~4574 patients). Primary outcomes will be identification of LCS-eligible patients among individuals with uncertain LCS eligibility and LDCT ordering rates among individuals with documented LCS eligibility. ETHICS AND DISSEMINATION/BACKGROUND:The protocol was approved by the University of Utah Institutional Review Board (# 00153806). The patient data collected for this study will not be shared publicly due to the sensitive nature of the patient health information and the fact that we will not be obtaining written informed consent to allow public sharing of their data. Results will be disseminated through peer-reviewed publications. TRIAL REGISTRATION NUMBER/BACKGROUND:Clinicaltrials.gov, NCT06338592.

Highly effective antiobesity and diabetes medications such as glucagon-like peptide 1 (GLP-1) agonists and glucose-dependent insulinotropic polypeptide/GLP-1 (dual) receptor agonists (RAs) have ushered in a new era of treatment of these highly prevalent, morbid conditions that have increased across the globe. However, the rapidly escalating use of GLP-1/dual RA medications is poised to overwhelm an already overburdened health care provider workforce and health care delivery system, stifling its potentially dramatic benefits. Relying on existing systems and resources to address the oncoming rise in GLP-1/dual RA use will be insufficient. Generative artificial intelligence (GenAI) has the potential to offset the clinical and administrative demands associated with the management of patients on these medication types. Early adoption of GenAI to facilitate the management of these GLP-1/dual RAs has the potential to improve health outcomes while decreasing its concomitant workload. Research and development efforts are urgently needed to develop GenAI obesity medication management tools, as well as to ensure their accessibility and use by encouraging their integration into health care delivery systems.

OBJECTIVE/UNASSIGNED:Our objective was to determine the feasibility and preliminary efficacy of a behavioral nudge on adoption of a clinical decision support (CDS) tool. MATERIALS AND METHODS/UNASSIGNED:We conducted a pilot cluster nonrandomized controlled trial in 2 Emergency Departments (EDs) at a large academic healthcare system in the New York metropolitan area. We tested 2 versions of a CDS tool for pulmonary embolism (PE) risk assessment developed on a web-based electronic health record-agnostic platform. One version included behavioral nudges incorporated into the user interface. RESULTS/UNASSIGNED: < .001). DISCUSSION/UNASSIGNED:We demonstrated feasibility and preliminary efficacy of a PE risk prediction CDS tool developed using insights from behavioral science. The tool is well-positioned to be tested in a large randomized clinical trial. TRIAL REGISTRATION/UNASSIGNED:Clinicaltrials.gov (NCT05203185).

IMPORTANCE/UNASSIGNED:Increasing numbers of unaffected individuals could benefit from genetic evaluation for inherited cancer susceptibility. Automated conversational agents (ie, chatbots) are being developed for cancer genetics contexts; however, randomized comparisons with standard of care (SOC) are needed. OBJECTIVE/UNASSIGNED:To examine whether chatbot and SOC approaches are equivalent in completion of pretest cancer genetic services and genetic testing. DESIGN, SETTING, AND PARTICIPANTS/UNASSIGNED:This equivalence trial (Broadening the Reach, Impact, and Delivery of Genetic Services [BRIDGE] randomized clinical trial) was conducted between August 15, 2020, and August 31, 2023, at 2 US health care systems (University of Utah Health and NYU Langone Health). Participants were aged 25 to 60 years, had had a primary care visit in the previous 3 years, were eligible for cancer genetic evaluation, were English or Spanish speaking, had no prior cancer diagnosis other than nonmelanoma skin cancer, had no prior cancer genetic counseling or testing, and had an electronic patient portal account. INTERVENTION/UNASSIGNED:Participants were randomized 1:1 at the patient level to the study groups at each site. In the chatbot intervention group, patients were invited in a patient portal outreach message to complete a pretest genetics education chat. In the enhanced SOC control group, patients were invited to complete an SOC pretest appointment with a certified genetic counselor. MAIN OUTCOMES AND MEASURES/UNASSIGNED:Primary outcomes were completion of pretest cancer genetic services (ie, pretest genetics education chat or pretest genetic counseling appointment) and completion of genetic testing. Equivalence hypothesis testing was used to compare the study groups. RESULTS/UNASSIGNED:This study included 3073 patients (1554 in the chatbot group and 1519 in the enhanced SOC control group). Their mean (SD) age at outreach was 43.8 (9.9) years, and most (2233 of 3063 [72.9%]) were women. A total of 204 patients (7.3%) were Black, 317 (11.4%) were Latinx, and 2094 (75.0%) were White. The estimated percentage point difference for completion of pretest cancer genetic services between groups was 2.0 (95% CI, -1.1 to 5.0). The estimated percentage point difference for completion of genetic testing was -1.3 (95% CI, -3.7 to 1.1). Analyses suggested equivalence in the primary outcomes. CONCLUSIONS AND RELEVANCE/UNASSIGNED:The findings of the BRIDGE equivalence trial support the use of chatbot approaches to offer cancer genetic services. Chatbot tools can be a key component of sustainable and scalable population health management strategies to enhance access to cancer genetic services. TRIAL REGISTRATION/UNASSIGNED:ClinicalTrials.gov Identifier: NCT03985852.

OBJECTIVES/OBJECTIVE:To evaluate demographic biases in diagnostic accuracy and health advice between generative artificial intelligence (AI) (ChatGPT GPT-4) and traditional symptom checkers like WebMD. MATERIALS AND METHODS/METHODS:Combination symptom and demographic vignettes were developed for 27 most common symptom complaints. Standardized prompts, written from a patient perspective, with varying demographic permutations of age, sex, and race/ethnicity were entered into ChatGPT (GPT-4) between July and August 2023. In total, 3 runs of 540 ChatGPT prompts were compared to the corresponding WebMD Symptom Checker output using a mixed-methods approach. In addition to diagnostic correctness, the associated text generated by ChatGPT was analyzed for readability (using Flesch-Kincaid Grade Level) and qualitative aspects like disclaimers and demographic tailoring. RESULTS:ChatGPT matched WebMD in 91% of diagnoses, with a 24% top diagnosis match rate. Diagnostic accuracy was not significantly different across demographic groups, including age, race/ethnicity, and sex. ChatGPT's urgent care recommendations and demographic tailoring were presented significantly more to 75-year-olds versus 25-year-olds (P < .01) but were not statistically different among race/ethnicity and sex groups. The GPT text was suitable for college students, with no significant demographic variability. DISCUSSION/CONCLUSIONS:The use of non-health-tailored generative AI, like ChatGPT, for simple symptom-checking functions provides comparable diagnostic accuracy to commercially available symptom checkers and does not demonstrate significant demographic bias in this setting. The text accompanying differential diagnoses, however, suggests demographic tailoring that could potentially introduce bias. CONCLUSION/CONCLUSIONS:These results highlight the need for continued rigorous evaluation of AI-driven medical platforms, focusing on demographic biases to ensure equitable care.

This study introduces a Generative Artificial Intelligence (GenAI) assistant designed to address key challenges in Remote Patient Monitoring (RPM) for hypertension. After a comprehensive needs assessment from clinicians and patients, we identified pivotal issues in RPM data management and patient engagement. The GenAI RPM assistant integrates a patient-facing chatbot, clinician-facing smart summaries, and automated draft portal messages to enhance communication and streamline data review. Validated through six rounds of testing and evaluations by ten participants, the initial prototype was positively received, highlighting the importance of personalized interactions. Our findings demonstrate GenAI's potential to improve RPM by optimizing data management and enhancing patient-provider communication.

BACKGROUND:Regular physical activity (PA) is a component of cardiovascular health and is associated with a lower risk of cardiovascular disease (CVD). However, only about half of US adults achieved the current PA recommendations. OBJECTIVE:The study purpose was to implement PA counseling using a clinical decision support tool in a preventive cardiology clinic and to assess changes in CVD risk factors in a sample of patients enrolled over 12 weeks of PA monitoring. METHODS:This intervention, piloted for 1 year, had 3 components embedded in the electronic health record: assessment of patients' PA, an electronic prompt for providers to counsel patients reporting low PA, and patient monitoring using a Fitbit. Cardiovascular disease risk factors included PA (self-report and Fitbit), body mass index, blood pressure, lipids, and cardiorespiratory fitness assessed with the 6-minute walk test. Depression and quality of life were also assessed. Paired t tests assessed changes in CVD risk. RESULTS:The sample who enrolled in the remote patient monitoring (n = 59) were primarily female (51%), White adults (76%) with a mean age of 61.13 ± 11.6 years. Self-reported PA significantly improved over 12 weeks ( P = .005), but not Fitbit steps ( P = .07). There was a significant improvement in cardiorespiratory fitness (469 ± 108 vs 494 ± 132 m, P = .0034), and 23 participants (42%) improved at least 25 m, signifying a clinically meaningful improvement. Only 4 participants were lost to follow-up over 12 weeks of monitoring. CONCLUSIONS:Patients may need more frequent reminders to be active after an initial counseling session, perhaps getting automated messages based on their step counts syncing to their electronic health record.