Faculty Bibliography

BACKGROUND:Online accessible patient education materials (PEMs) are essential for patient empowerment. However, studies have shown that these materials often exceed the recommended sixth-grade reading level, making them difficult for many patients to understand. Large language models (LLMs) have the potential to simplify PEMs into more readable educational content. OBJECTIVE:We sought to evaluate whether 3 LLMs (ChatGPT [OpenAI], Gemini [Google], and Claude [Anthropic PBC]) can optimize the readability of PEMs to the recommended reading level without compromising accuracy. METHODS:This cross-sectional study used 60 randomly selected PEMs available online from 3 websites. We prompted LLMs to simplify the reading level of online PEMs. The primary outcome was the readability of the original online PEMs compared with the LLM-simplified versions. Readability scores were calculated using 4 validated indices Flesch Reading Ease, Flesch-Kincaid Grade Level, Gunning Fog Index, and Simple Measure of Gobbledygook Index. Accuracy and understandability were also assessed as balancing measures, with understandability measured using the Patient Education Materials Assessment Tool-Understandability (PEMAT-U). RESULTS:The original readability scores for the American Heart Association (AHA), American Cancer Society (ACS), and American Stroke Association (ASA) websites were above the recommended sixth-grade level, with mean grade level scores of 10.7,10.0, and 9.6, respectively. After optimization by the LLMs, readability scores significantly improved across all 3 websites when compared with the original text. Compared with the original website, Wilcoxon signed rank test showed ChatGPT improved the readability to 7.6 from 10.1 (P<.001); Gemini, to 6.6 (P<.001); and Claude, to 5.6 (P<.001). Word counts were significantly reduced by all LLMs, with a decrease from a mean range of 410.9-953.9 words to a mean range of 201.9-248.1 words. None of the ChatGPT LLM-simplified PEMs were inaccurate, while 3.3% of Gemini and Claude LLM-simplified PEMs were inaccurate. Baseline understandability scores, as measured by PEMAT-U, were preserved across all LLM-simplified versions. CONCLUSIONS:This cross-sectional study demonstrates that LLMs have the potential to significantly enhance the readability of online PEMs while maintaining accuracy and understandability, making them more accessible to a broader audience. However, variability in model performance and demonstrated inaccuracies underscore the need for human review of LLM output. Further study is needed to explore advanced LLM techniques and models trained for medical content.

OBJECTIVES/OBJECTIVE:The study aimed to assess the usage and impact of a private and secure instance of a generative artificial intelligence (GenAI) application in a large academic health center. The goal was to understand how employees interact with this technology and the influence on their perception of skill and work performance. MATERIALS AND METHODS/METHODS:New York University Langone Health (NYULH) established a secure, private, and managed Azure OpenAI service (GenAI Studio) and granted widespread access to employees. Usage was monitored and users were surveyed about their experiences. RESULTS:Over 6 months, over 1007 individuals applied for access, with high usage among research and clinical departments. Users felt prepared to use the GenAI studio, found it easy to use, and would recommend it to a colleague. Users employed the GenAI studio for diverse tasks such as writing, editing, summarizing, data analysis, and idea generation. Challenges included difficulties in educating the workforce in constructing effective prompts and token and API limitations. DISCUSSION/CONCLUSIONS:The study demonstrated high interest in and extensive use of GenAI in a healthcare setting, with users employing the technology for diverse tasks. While users identified several challenges, they also recognized the potential of GenAI and indicated a need for more instruction and guidance on effective usage. CONCLUSION/CONCLUSIONS:The private GenAI studio provided a useful tool for employees to augment their skills and apply GenAI to their daily tasks. The study underscored the importance of workforce education when implementing system-wide GenAI and provided insights into its strengths and weaknesses.

BACKGROUND AND OBJECTIVES/OBJECTIVE:Healthcare is increasingly being delivered in the outpatient setting, but robust quality improvement programs and performance metrics are lacking in ambulatory care, particularly specialty-based ambulatory care. METHODS:To promote quality improvement in ambulatory care, we developed an infrastructure to create specialty-specific quality measures and dashboards that could be used to display providers' performance across relevant measures to individual providers and institutional leaders. RESULTS:The products of this program include a governance and infrastructure for specialty-specific ambulatory quality metrics as well as two distinct dashboards for data display. One dashboard is provider-facing, displaying provider's performance on specialty-specific measures as compared to institutional standards. The second dashboard is a leadership dashboard that provides overall and provider-level information on performance across measures. CONCLUSIONS:The Specialty-based Ambulatory Quality program reflects a systematic, institutionally-supported quality improvement framework that can be applied across diverse ambulatory specialties. As next steps, we plan to evaluate the program's impact on provider performance across measures and expand this program to other specialties practicing in the outpatient setting.

OBJECTIVES/OBJECTIVE: This study aimed to highlight the necessity of developing and implementing appropriate reference ranges for transgender and nonbinary (TGNB) patient populations to minimize misinterpretation of laboratory results and ensure equitable health care. CASE REPORT/METHODS: We describe a situation where a TGNB patient's abnormal laboratory values were not flagged due to undefined reference ranges for gender "X" in the Laboratory Information System (LIS). Implementation of additional reference ranges mapped to sex label "X" showed significant improvement in flagging abnormal lab results, utilizing sex-invariant reporting as an interim solution while monitoring developments on TGNB-specific reference ranges. CONCLUSION/CONCLUSIONS: Informatics professionals should assess their institution's policies for registration and lab reporting on TGNB patients as nonimplementation poses significant patient safety risks. Best practices include using TGNB-specific reference ranges emerging in the literature, reporting both male and female reference ranges for clinical interpretation and sex-invariant reporting.

We have entered a new age of health informatics—applied health informatics—where digital health innovation cannot be pursued without considering operational needs. In this new digital health era, creating an integrated applied health informatics system will be essential for health systems to achieve informatics healthcare goals. Integration of information technology (IT) and health informatics does not naturally occur without a deliberate and intentional shift towards unification. Recognizing this, NYU Langone Health’s (NYULH) Medical Center IT (MCIT) has taken proactive measures to vertically integrate academic informatics and operational IT through the establishment of the MCIT Department of Health Informatics (DHI). The creation of the NYULH DHI showcases the drivers, challenges, and ultimate successes of our enterprise effort to align academic health informatics with IT; providing a model for the creation of the applied health informatics programs required for academic health systems to thrive in the increasingly digitized healthcare landscape.

BACKGROUND:Healthcare crowdsourcing events (e.g. hackathons) facilitate interdisciplinary collaboration and encourage innovation. Peer-reviewed research has not yet considered a healthcare crowdsourcing event focusing on generative artificial intelligence (GenAI), which generates text in response to detailed prompts and has vast potential for improving the efficiency of healthcare organizations. Our event, the New York University Langone Health (NYULH) Prompt-a-thon, primarily sought to inspire and build AI fluency within our diverse NYULH community, and foster collaboration and innovation. Secondarily, we sought to analyze how participants' experience was influenced by their prior GenAI exposure and whether they received sample prompts during the workshop. METHODS:Executing the event required the assembly of an expert planning committee, who recruited diverse participants, anticipated technological challenges, and prepared the event. The event was composed of didactics and workshop sessions, which educated and allowed participants to experiment with using GenAI on real healthcare data. Participants were given novel "project cards" associated with each dataset that illuminated the tasks GenAI could perform and, for a random set of teams, sample prompts to help them achieve each task (the public repository of project cards can be found at https://github.com/smallw03/NYULH-Generative-AI-Prompt-a-thon-Project-Cards). Afterwards, participants were asked to fill out a survey with 7-point Likert-style questions. RESULTS:Our event was successful in educating and inspiring hundreds of enthusiastic in-person and virtual participants across our organization on the responsible use of GenAI in a low-cost and technologically feasible manner. All participants responded positively, on average, to each of the survey questions (e.g., confidence in their ability to use and trust GenAI). Critically, participants reported a self-perceived increase in their likelihood of using and promoting colleagues' use of GenAI for their daily work. No significant differences were seen in the surveys of those who received sample prompts with their project task descriptions. CONCLUSION/CONCLUSIONS:The first healthcare Prompt-a-thon was an overwhelming success, with minimal technological failures, positive responses from diverse participants and staff, and evidence of post-event engagement. These findings will be integral to planning future events at our institution, and to others looking to engage their workforce in utilizing GenAI.

Electronic health records have become an integral part of modern health care, but their implementation has led to unintended consequences, such as poor note quality. This case study explores how NYU Langone Health leveraged artificial intelligence (AI) to address the challenge to improve the content and quality of medical documentation. By quickly and accurately analyzing large volumes of clinical documentation and providing feedback to organizational leadership and individually to providers, AI can help support a culture of continuous note quality improvement, allowing organizations to enhance a critical component of patient care.