Faculty Bibliography

OBJECTIVE:To evaluate trends in telemedicine utilization overall and across clinical specialties, providing insights into its evolving role in health care delivery. STUDY DESIGN/METHODS:This retrospective cross-sectional study analyzed 1.9 million telemedicine video visits from a large academic health care system in New York City between 2020 and 2023. The data, collected from the health care system's electronic health records, included telemedicine encounters across more than 500 ambulatory locations. METHODS:We used descriptive statistics to outline telemedicine usage trends and compared telemedicine utilization rates and evaluation and management characteristics across clinical specialties. RESULTS:Telemedicine utilization peaked during the COVID-19 pandemic, then declined and stabilized. Despite an overall decline, 2 non-primary care specialties (behavioral health and psychiatry) experienced continued growth in telemedicine visits. Primary care and urgent care visits were mainly characterized by low-complexity visits, whereas non-primary care specialties witnessed a rise in moderate- and high-complexity visits, with the number of moderate-level visits surpassing those of low complexity. CONCLUSIONS:The findings highlight a dynamic shift in telemedicine utilization, with non-primary care settings witnessing an increase in the complexity of cases. To address future demands from increasingly complex medical cases managed through telemedicine in non-primary care, appropriate resource allocation is essential.

BACKGROUND:Laypeople have easy access to health information through large language models (LLMs), such as ChatGPT, and search engines, such as Google. Search engines transformed health information access, and LLMs offer a new avenue for answering laypeople's questions. OBJECTIVE:We aimed to compare the frequency of use and attitudes toward LLMs and search engines as well as their comparative relevance, usefulness, ease of use, and trustworthiness in responding to health queries. METHODS:We conducted a screening survey to compare the demographics of LLM users and nonusers seeking health information, analyzing results with logistic regression. LLM users from the screening survey were invited to a follow-up survey to report the types of health information they sought. We compared the frequency of use of LLMs and search engines using ANOVA and Tukey post hoc tests. Lastly, paired-sample Wilcoxon tests compared LLMs and search engines on perceived usefulness, ease of use, trustworthiness, feelings, bias, and anthropomorphism. RESULTS:In total, 2002 US participants recruited on Prolific participated in the screening survey about the use of LLMs and search engines. Of them, 52% (n=1045) of the participants were female, with a mean age of 39 (SD 13) years. Participants were 9.7% (n=194) Asian, 12.1% (n=242) Black, 73.3% (n=1467) White, 1.1% (n=22) Hispanic, and 3.8% (n=77) were of other races and ethnicities. Further, 1913 (95.6%) used search engines to look up health queries versus 642 (32.6%) for LLMs. Men had higher odds (odds ratio [OR] 1.63, 95% CI 1.34-1.99; P<.001) of using LLMs for health questions than women. Black (OR 1.90, 95% CI 1.42-2.54; P<.001) and Asian (OR 1.66, 95% CI 1.19-2.30; P<.01) individuals had higher odds than White individuals. Those with excellent perceived health (OR 1.46, 95% CI 1.1-1.93; P=.01) were more likely to use LLMs than those with good health. Higher technical proficiency increased the likelihood of LLM use (OR 1.26, 95% CI 1.14-1.39; P<.001). In a follow-up survey of 281 LLM users for health, most participants used search engines first (n=174, 62%) to answer health questions, but the second most common first source consulted was LLMs (n=39, 14%). LLMs were perceived as less useful (P<.01) and less relevant (P=.07), but elicited fewer negative feelings (P<.001), appeared more human (LLM: n=160, vs search: n=32), and were seen as less biased (P<.001). Trust (P=.56) and ease of use (P=.27) showed no differences. CONCLUSIONS:Search engines are the primary source of health information; yet, positive perceptions of LLMs suggest growing use. Future work could explore whether LLM trust and usefulness are enhanced by supplementing answers with external references and limiting persuasive language to curb overreliance. Collaboration with health organizations can help improve the quality of LLMs' health output.

IMPORTANCE/UNASSIGNED:Virtual patient-physician communications have increased since 2020 and negatively impacted primary care physician (PCP) well-being. Generative artificial intelligence (GenAI) drafts of patient messages could potentially reduce health care professional (HCP) workload and improve communication quality, but only if the drafts are considered useful. OBJECTIVES/UNASSIGNED:To assess PCPs' perceptions of GenAI drafts and to examine linguistic characteristics associated with equity and perceived empathy. DESIGN, SETTING, AND PARTICIPANTS/UNASSIGNED:This cross-sectional quality improvement study tested the hypothesis that PCPs' ratings of GenAI drafts (created using the electronic health record [EHR] standard prompts) would be equivalent to HCP-generated responses on 3 dimensions. The study was conducted at NYU Langone Health using private patient-HCP communications at 3 internal medicine practices piloting GenAI. EXPOSURES/UNASSIGNED:Randomly assigned patient messages coupled with either an HCP message or the draft GenAI response. MAIN OUTCOMES AND MEASURES/UNASSIGNED:PCPs rated responses' information content quality (eg, relevance), using a Likert scale, communication quality (eg, verbosity), using a Likert scale, and whether they would use the draft or start anew (usable vs unusable). Branching logic further probed for empathy, personalization, and professionalism of responses. Computational linguistics methods assessed content differences in HCP vs GenAI responses, focusing on equity and empathy. RESULTS/UNASSIGNED:A total of 16 PCPs (8 [50.0%] female) reviewed 344 messages (175 GenAI drafted; 169 HCP drafted). Both GenAI and HCP responses were rated favorably. GenAI responses were rated higher for communication style than HCP responses (mean [SD], 3.70 [1.15] vs 3.38 [1.20]; P = .01, U = 12 568.5) but were similar to HCPs on information content (mean [SD], 3.53 [1.26] vs 3.41 [1.27]; P = .37; U = 13 981.0) and usable draft proportion (mean [SD], 0.69 [0.48] vs 0.65 [0.47], P = .49, t = -0.6842). Usable GenAI responses were considered more empathetic than usable HCP responses (32 of 86 [37.2%] vs 13 of 79 [16.5%]; difference, 125.5%), possibly attributable to more subjective (mean [SD], 0.54 [0.16] vs 0.31 [0.23]; P < .001; difference, 74.2%) and positive (mean [SD] polarity, 0.21 [0.14] vs 0.13 [0.25]; P = .02; difference, 61.5%) language; they were also numerically longer (mean [SD] word count, 90.5 [32.0] vs 65.4 [62.6]; difference, 38.4%), but the difference was not statistically significant (P = .07) and more linguistically complex (mean [SD] score, 125.2 [47.8] vs 95.4 [58.8]; P = .002; difference, 31.2%). CONCLUSIONS/UNASSIGNED:In this cross-sectional study of PCP perceptions of an EHR-integrated GenAI chatbot, GenAI was found to communicate information better and with more empathy than HCPs, highlighting its potential to enhance patient-HCP communication. However, GenAI drafts were less readable than HCPs', a significant concern for patients with low health or English literacy.

The COVID-19 pandemic has boosted digital health utilization, raising concerns about increased physicians' after-hours clinical work ("work-outside-work"). The surge in patients' digital messages and additional time spent on work-outside-work by telemedicine providers underscores the need to evaluate the connection between digital health utilization and physicians' after-hours commitments. We examined the impact on physicians' workload from two types of digital demands - patients' messages requesting medical advice (PMARs) sent to physicians' inbox (inbasket), and telemedicine. Our study included 1716 ambulatory-care physicians in New York City regularly practicing between November 2022 and March 2023. Regression analyses assessed primary and interaction effects of (PMARs) and telemedicine on work-outside-work. The study revealed a significant effect of PMARs on physicians' work-outside-work and that this relationship is moderated by physicians' specialties. Non-primary care physicians or specialists experienced a more pronounced effect than their primary care peers. Analysis of their telemedicine load revealed that primary care physicians received fewer PMARs and spent less time in work-outside-work with more telemedicine. Specialists faced increased PMARs and did more work-outside-work as telemedicine visits increased which could be due to the difference in patient panels. Reducing PMAR volumes and efficient inbasket management strategies needed to reduce physicians' work-outside-work. Policymakers need to be cognizant of potential disruptions in physicians carefully balanced workload caused by the digital health services.

Visualizations of mass shooting incidents in the United States appearing in the media can influence people's beliefs and attitudes. However, different data sources each use their own definition of mass shootings, resulting in varying counts and trends of these incidents across the sources. To investigate the effects of these varying definitions on public perceptions, we conducted a crowdsourced study using data from four sources-Mother Jones, Mass Shooter Database, Everytown for Gun Safety, and The Washington Post. We used one or more line plots, with or without explicitly providing the definition, to see how these variations affect viewers' understanding of a 10-year trend in mass shooting frequency. We found that, depending on the data shown, participants' perceptions of the trend changed in both directions (i.e., more or less increasing) compared to their prestudy perceptions. We discuss how data from a single source can influence people's perceptions, and how visualizing data from multiple sources (e.g., superimposed line graphs) can enable more transparent communication. Our work has implications for other media and public visualizations, highlighting the importance of embracing pluralistic approaches to enquiry, especially when dealing with data of significant importance and consequence.

BACKGROUND:Remote patient monitoring (RPM) technologies can support patients living with chronic conditions through self-monitoring of physiological measures and enhance clinicians' diagnostic and treatment decisions. However, to date, large-scale pragmatic RPM implementation within health systems has been limited, and understanding of the impacts of RPM technologies on clinical workflows and care experience is lacking. OBJECTIVE:In this study, we evaluate the early implementation of operational RPM initiatives for chronic disease management within the ambulatory network of an academic medical center in New York City, focusing on the experiences of "early adopter" clinicians and patients. METHODS:Using a multimethod qualitative approach, we conducted (1) interviews with 13 clinicians across 9 specialties considered as early adopters and supporters of RPM and (2) speculative design sessions exploring the future of RPM in clinical care with 21 patients and patient representatives, to better understand experiences, preferences, and expectations of pragmatic RPM use for health care delivery. RESULTS:We identified themes relevant to RPM implementation within the following areas: (1) data collection and practices, including impacts of taking real-world measures and issues of data sharing, security, and privacy; (2) proactive and preventive care, including proactive and preventive monitoring, and proactive interventions and support; and (3) health disparities and equity, including tailored and flexible care and implicit bias. We also identified evidence for mitigation and support to address challenges in each of these areas. CONCLUSIONS:This study highlights the unique contexts, perceptions, and challenges regarding the deployment of RPM in clinical practice, including its potential implications for clinical workflows and work experiences. Based on these findings, we offer implementation and design recommendations for health systems interested in deploying RPM-enabled health care.

BACKGROUND:Chatbots are being piloted to draft responses to patient questions, but patients' ability to distinguish between provider and chatbot responses and patients' trust in chatbots' functions are not well established. OBJECTIVE:This study aimed to assess the feasibility of using ChatGPT (Chat Generative Pre-trained Transformer) or a similar artificial intelligence-based chatbot for patient-provider communication. METHODS:A survey study was conducted in January 2023. Ten representative, nonadministrative patient-provider interactions were extracted from the electronic health record. Patients' questions were entered into ChatGPT with a request for the chatbot to respond using approximately the same word count as the human provider's response. In the survey, each patient question was followed by a provider- or ChatGPT-generated response. Participants were informed that 5 responses were provider generated and 5 were chatbot generated. Participants were asked-and incentivized financially-to correctly identify the response source. Participants were also asked about their trust in chatbots' functions in patient-provider communication, using a Likert scale from 1-5. RESULTS:A US-representative sample of 430 study participants aged 18 and older were recruited on Prolific, a crowdsourcing platform for academic studies. In all, 426 participants filled out the full survey. After removing participants who spent less than 3 minutes on the survey, 392 respondents remained. Overall, 53.3% (209/392) of respondents analyzed were women, and the average age was 47.1 (range 18-91) years. The correct classification of responses ranged between 49% (192/392) to 85.7% (336/392) for different questions. On average, chatbot responses were identified correctly in 65.5% (1284/1960) of the cases, and human provider responses were identified correctly in 65.1% (1276/1960) of the cases. On average, responses toward patients' trust in chatbots' functions were weakly positive (mean Likert score 3.4 out of 5), with lower trust as the health-related complexity of the task in the questions increased. CONCLUSIONS:ChatGPT responses to patient questions were weakly distinguishable from provider responses. Laypeople appear to trust the use of chatbots to answer lower-risk health questions. It is important to continue studying patient-chatbot interaction as chatbots move from administrative to more clinical roles in health care.

Increasingly, laws are being proposed and passed by governments around the world to regulate artificial intelligence (AI) systems implemented into the public and private sectors. Many of these regulations address the transparency of AI systems, and related citizen-aware issues like allowing individuals to have the right to an explanation about how an AI system makes a decision that impacts them. Yet, almost all AI governance documents to date have a significant drawback: they have focused on what to do (or what not to do) with respect to making AI systems transparent, but have left the brunt of the work to technologists to figure out how to build transparent systems. We fill this gap by proposing a stakeholder-first approach that assists technologists in designing transparent, regulatory-compliant systems. We also describe a real-world case study that illustrates how this approach can be used in practice.

Peer production, such as the collaborative authoring of Wikipedia articles, involves both cooperation and competition between contributors. Cooperatively, Wikipedia's contributors attempt to create high-quality articles, and at the same time, they compete to align Wikipedia articles with their personal perspectives and "take ownership" of the article. This process is governed collectively by the community, which works to ensure the neutrality of the content. We study the interplay between individuals' cooperation and competition, considering the community's endeavor to ensure a neutral point of view (NPOV) on articles. We develop a two-level game-theoretic model: the first level models the interactions between individual contributors who seek both cooperative and competitive goals and the second level models governance of co-production as a Stackelberg (leader-follower) game between contributors and the communal neutrality-enforcing mechanisms. We present our model's predictions regarding the relationship between contributors' personal benefits of content ownership and their characteristics, namely their cooperative/competitive orientation and their activity profile (whether creators or curators of content). We validate the model's prediction through an empirical analysis, by studying the interactions of 219,811 distinct contributors that co-produced 864 Wikipedia articles over a decade. The analysis and empirical results suggest that the factor that determines who ends up owning content is the ratio between one's cooperative/competitive orientation (estimated based on whether a core or peripheral community member) and the contributor's creator/curator activity profile (proxied through average edit size per sentence). Namely, under the governance mechanisms, the fractional content that is eventually owned by a contributor is higher for curators that have a competitive orientation. Although neutrality-seeking mechanisms are essential for ensuring that ownership is not concentrated within a small number of contributors, our findings suggest that the burden of excessive governance may deter contributors from participating, and thus indirectly curtail the peer production of high-quality articles.

BACKGROUND:The surge of telemedicine use during the early stages of the coronavirus-19 (COVID-19) pandemic has been well documented. However, scarce evidence considers the utilization of telemedicine in the subsequent period. OBJECTIVE:This study aims to evaluate utilization patterns of video-based telemedicine visits for ambulatory care and urgent care provision over the course of recurring pandemic waves in one large health system in New York City, and what this means for healthcare delivery. METHODS:Retrospective electronic health record (EHR) data of patients between January 1st, 2020, and February 28th, 2022 were used to longitudinally track and analyze telemedicine and in-person visit volumes across ambulatory care specialties and urgent care, as well as compare them to a pre-pandemic baseline (June to November 2019). Diagnosis codes to differentiate COVID-19 suspected visits from non-COVID-19 visits, as well as evaluating COVID-19 based telemedicine utilization over time, were compared to the total number of COVID-19 positive cases in the same geographic region (city-level). The time-series data was segmented based on change-point analysis and variances in visit trends were compared between the segments. RESULTS:The emergence of COVID-19 prompted an early increase in the number of telemedicine visits across the urgent care and ambulatory care settings. This utilization continued throughout the pandemic at a much higher level than the pre-pandemic baseline for both COVID-19 and non-COVID suspected visits, despite fluctuation in COVID-19 cases throughout the pandemic and the resumption of in-person clinical services. Utilization of telemedicine-based urgent care services for COVID-19 suspected visits showed more variance in response to each pandemic wave, but telemedicine visits for ambulatory care have remained relatively steady after the initial crisis period. During the Omicron wave, the utilization of all visit types including in-person activities decreased. Patients between 25 and 34 years of age were the largest users of telemedicine-based urgent care. Patient satisfaction with telemedicine-based urgent care remained high despite the rapid scaling of services to meet increased demand. CONCLUSIONS:The trend of increased use of telemedicine as a means of healthcare delivery relative to the pre-COVID-19 baseline has been maintained throughout the later pandemic periods despite fluctuating COVID-19 cases and the resumption of in-person care delivery. Overall satisfaction with telemedicine-based care is also high. The trends in telemedicine utilization suggest that telemedicine-based healthcare delivery has become a mainstream and sustained supplement to in-person-based ambulatory care, particularly for younger patients, for both urgent and non-urgent care needs. These findings have implications for the healthcare delivery system, including practice leaders, insurers, and policymakers. Further investigation is needed to evaluate telemedicine adoption by key demographics, identify ongoing barriers to adoption, and explore the impacts of sustained use of telemedicine on healthcare outcomes and experience.