Faculty Bibliography

BACKGROUND:Unannounced standardized patients (USPs) have long been used to measure clinical performance in situ. These incognito actors capture data on clinician skills during an encounter, as well as patient experience more broadly. A robust USP program requires extensive preparation and standardization efforts. Given the widespread expansion of USPs for education, research, and improvement efforts, we conducted a systematic review with the goal of capturing the breadth of uses of USPs across settings, along with the standardization measures employed across studies. METHODS:In collaboration with a medical librarian, we conducted systematic searches across six databases. Two independent researchers screened each report for inclusion. Three coders extracted and reviewed study characteristics and data from the studies deemed eligible for inclusion. We extracted data on: target population, setting, and assessed skills. We also captured the reliability and fidelity measures described in each study, including USP detection, USP training methods, and assessment measures. RESULTS:128 articles were included. Individual clinicians were the most frequently targeted population (n = 114, 89 %). Common clinician roles included physicians (n = 92, 72 %) and pharmacists (n = 12, 9 %). The collective care team was the target in two studies (2 %), and systems and larger healthcare facilities were targeted in only 1 (1 %) and 13 (10 %) studies, respectively. Studies were primarily conducted in ambulatory settings (n = 118, 92 %). History gathering (n = 76, 59 %), communication (n = 55, 43 %), counseling (n = 51, 40 %), and patient education (n = 49, 38 %) were commonly assessed, as were correct diagnosis (n = 34, 27 %), appropriate ordering of labs/tests (n = 30, 23 %), referrals (n = 35, 27 %), and prescriptions (n = 36, 28 %). USP detection reporting was variable across studies; however, no detection information was provided for 48 studies. 62 % of articles reported incorporating a measure of reliability or fidelity into their study, while the remainder either failed to provide adequate information on use of these measures. CONCLUSIONS:We explored USP use across settings and describe the scope and limitations of the literature. USPs capture a range of data domains but a lack uniform report of reliability measures can potentially undermine findings. Future studies should incorporate and uniformly report out on detection, training, and assessment.

OBJECTIVES/OBJECTIVE:Communication and other clinical skills are routinely assessed in medical schools using Objective Structured Clinical Examinations (OSCEs) so routinely that it can be difficult to monitor and maintain validity. We report on the accumulation of validity evidence for the Clinical Communication Skills Assessment Tool (CCSAT) based on its use with 9 cohorts of medical students in a high stakes OSCE. METHODS:) based on continuous quality improvement and use of the CCSAT for feedback, remediation, curricular design, and research. RESULTS:Implementation of the CCSAT over time has facilitated our communication skills curriculum and training. Thoughtful case development and investment in standardized patient training has contributed to data quality. Item analysis supports our behaviorally anchored scale (not done, partly and well done) and the skills domains suggested by an a priori evidence-based clinical communication model were confirmed via analysis of actual student data. Evidence synthesized across the frameworks suggests consistent validity of the CCSAT for generalization inferences (that it captures the construct), responsiveness (sensitivity to change/difference), content validity/internal structure, relationships to other variables, and consequences/implications. More evidence is needed to strengthen validity of CCSAT scores for understanding extrapolation inferences and real-world implications. CONCLUSIONS AND PRACTICE IMPLICATIONS/CONCLUSIONS:This pragmatic approach to evaluating validity within a program of assessment serves as a model for medical schools seeking to continuously monitor the quality of clinical skill assessments, a need made particularly relevant since the US NBME no longer requires the Step 2 Clinical Skills exam, leaving individual schools with the responsibility for ensuring graduates have acquired the requisite core clinical skills. We document strong evidence for CCSAT validity over time and across cohorts as well as areas for improvement and further examination.

PURPOSE/OBJECTIVE:As the COVID-19 pandemic forced most colleges and universities to go online, student health centers rapidly shifted to telehealth platforms without frameworks for virtual care provision. An urban student health center implemented a needs assessment involving unannounced standardized patients (USPs) to evaluate the integration of a new telehealth workflow and clinicians' virtual communication skills. METHODS:From April to May 2021, USPs conducted two video visits with 12 primary care and four women's health clinicians (N = 16 clinicians; 32 visits). Cases included (1) a 21-year-old female presenting for birth control with a positive Patient Health Questionaire-9 and (2) a 21-year-old male, who vapes regularly, with questions regarding safe sex with men. Clinicians were evaluated using a checklist completed by the USP immediately following the visit and a systematic chart review of the electronic health record. RESULTS:USP feedback indicates most clinicians received high ratings for general communication skills but may benefit from educational intervention in several key telemedicine skills. Clinicians struggled with using nonverbal signals to enrich communication (47% well done), acknowledging emotions (34% well done), and using video for information gathering (34% well done). Low rates of standard screenings (e.g., 63% administered the PHQ-2, <50% asked about alcohol use) suggested protocols for in-person care were not easily incorporated into telehealth practices, and clinicians may benefit from enhanced care team support. Performance reports were shared with clinicians and leadership postvisit. DISCUSSION/CONCLUSIONS:Results suggest project design and implementation is scalable and feasible for use at other institutions, offering a structured methodology that can improve general student health care.

PURPOSE/UNASSIGNED:The Professional Identity Essay (PIE) is a theory and evidence-based Medical Professional Identity Formation (MPIF) measure. We describe trajectories of PIE-measured MPIF over a 4-year US medical school curriculum. METHODS/UNASSIGNED:Students write PIEs at medical school orientation, clinical clerkships orientation, and post-advanced (near graduation) clerkship. A trained evaluator assigns an overall stage score to narrative responses to nine PIE prompts (inter-rater ICC 0.83, 95% CI [0.57 - 0.96], intra-rater ICC 0.85). Distribution of PIE stage scores across time points were analyzed in the aggregate and individual students were classified as Increase, Stable (no score change) or Decrease based on the trajectories of PIE stage scores over time. RESULTS/UNASSIGNED: CONCLUSIONS/UNASSIGNED:Medical students' PIE stage scores increase over time with three distinctive trajectories. Further study is needed to explore the utility of this method for formative assessment, program evaluation, and MPIF research.

BACKGROUND:Although efforts are underway to address social determinants of health (SDOH), little is known about physicians' SDOH practices despite evidence that failing to fully elicit and respond to social needs can compromise patient safety and undermine both the quality and effectiveness of treatment. In particular, interventions designed to enhance response to social needs have not been assessed using actual practice behaviour. In this study, we evaluate the degree to which providing primary care physicians with feedback on their SDOH practice behaviours is associated with increased rates of eliciting and responding to housing and social isolation needs. METHODS:Unannounced standardised patients (USPs), actors trained to consistently portray clinical scenarios, were sent, incognito, to all five primary care teams in an urban, safety-net healthcare system. Scenarios involved common primary care conditions and each included an underlying housing (eg, mould in the apartment, crowding) and social isolation issue and USPs assessed whether the physician fully elicited these needs and if so, whether or not they addressed them. The intervention consisted of providing physicians with audit/feedback reports of their SDOH practices, along with brief written educational material. A prepost comparison group design was used to evaluate the intervention; four teams received the intervention and one team served as a 'proxy' comparison (no intervention). Preintervention (February 2017 to December 2017) rates of screening for and response to the scripted housing and social needs were compared with intervention period (January 2018 to March 2019) rates for both intervention and comparison teams. RESULTS:108 visits were completed preintervention and 183 during the intervention period. Overall, social needs were not elicited half of the time and fully addressed even less frequently. Rates of identifying the housing issue increased for teams that received audit/feedback reports (46%-60%; p=0.045) and declined for the proxy comparison (61%-42%; p=0.174). Rates of responding to housing needs increased significantly for intervention teams (15%-41%; p=0.004) but not for the comparison team (21%-29%; p=0.663). Social isolation was identified more frequently postintervention (53%) compared with baseline (39%; p=0.041) among the intervention teams but remained unchanged for the comparison team (39% vs 32%; p=0.601). Full exploration of social isolation remained low for both intervention and comparison teams. CONCLUSIONS:Results suggest that physicians may not be consistently screening for or responding to social needs but that receiving feedback on those practices, along with brief targeted education, can improve rates of SDOH screening and response.

PURPOSE:To describe patterns of clinical communication skills that inform curriculum enhancement and guide coaching of medical students. MATERIALS AND METHODS:Performance data from 1182 consenting third year medical students in 9 cohorts (2011-2019), on a 17-item Clinical Communication Skills Assessment Tool (CCSAT) completed by trained Standardized Patients as part of an eight case high stakes Comprehensive Clinical Skills Exam (CCSE) were analyzed using latent profile analysis (LPA). Assessment domains included: information gathering (6 items), relationship development (5 items), patient education (3 items), and organization/time management (3 items). LPA clustered learners with similar strength/weakness into profiles based on item response patterns across cases. One-way analysis of variance (ANOVA) assessed for significant differences by profile for CCSAT items. RESULTS:Student performance clustered into six profiles in three groups, high performing (HP1 and HP2-Low Patient Education, 15.7%), average performing (AP1 and AP2-Interrupters, 40.9%), and lower performing profiles (LP1-Non-interrupters and LP2, 43.4%) with adequate model fit estimations and similar distribution in each cohort. We identified 3 CCSAT items that discriminated among learner's skill profiles. CONCLUSION:Clinical communication skill performance profiles provide nuanced, benchmarked guidance for curriculum improvement and tailoring of communication skills coaching.

BACKGROUND/UNASSIGNED:The transition to telehealth during the COVID-19 pandemic revealed a lack of preexisting telehealth training for clinicians. As a workplace-based simulation methodology designed to improve virtual clinical skills, announced standardized patients (ASPs) may help meet evolving educational needs to sustain quality telehealth care. OBJECTIVE/UNASSIGNED:We describe the development and implementation of an ASP program to assess and provide feedback to resident and faculty clinicians in virtual practice, and report on performance, feasibility, and acceptability. METHODS/UNASSIGNED:From June 2021 to April 2022, resident and faculty clinicians at a VA primary care clinic participated in a video visit in which an ASP portrayed either a 70-year-old man with hearing loss and hypertension or a 60-year-old man with hypertension and financial stress. Following the visit, ASPs provided verbal feedback and completed a behaviorally anchored checklist to rate telehealth and communication skills, chronic disease management, and use of resources. Domain summary scores were calculated as the mean percentage of "well done" items. Participants completed a feedback survey on their experience. RESULTS/UNASSIGNED:Seventy-six televisits (60 primary care residents [postgraduate year 1-3], 16 internal medicine faculty) were conducted from August 2021 to April 2022. Clinicians performed well in communication skills: information gathering (79%, 60 of 76, well done), relationship development (67%, 51 of 76), education and counseling (71%, 54 of 76), and patient satisfaction (86%, 65 of 76). They performed less well in telemedicine skills (38%, 29 of 76). Participants agreed that the experience was a good use of their time (88%, 67 of 76). CONCLUSIONS/UNASSIGNED:An ASP-facilitated training for resident and faculty clinicians assessed telehealth skills and clinical practice and identified areas for intervention. Clinicians responded well to the training and feedback.

BACKGROUND:There were seven opioid overdoses in this New York City (NYC) federally qualified health center from December 2018 through February 2019, reflecting the rising rate of overdose deaths in NYC overall at the time. In response to these overdoses, we sought to increase the readiness of health center staff to recognize and respond to opioid overdoses and decrease stigmatizing attitudes around opioid use disorder (OUD). METHODS:An hour-long training focusing on opioid overdose response was administered to clinical and non-clinical staff of all levels at the health center. This training included didactic education on topics such as the overdose epidemic, stigma around OUD, and opioid overdose response, as well as discussion. A structured assessment was administered immediately before and following the training to evaluate change in knowledge and attitudes. Additionally, participants completed a feedback survey immediately after the training to assess acceptability. Paired t-tests and analysis of variance tests were used to assess changes in pre- and post-test scores. RESULTS:Over 76% of the health center staff participated in the training (N = 310). There were large and significant increases in mean knowledge and attitudinal scores from pre- to post-test (p < .001 and p < .001, respectively). While there was no significant effect of profession on attitudinal change scores, profession did have a significant effect on knowledge change scores, with administrative staff, non-clinical support staff, other healthcare staff, and therapists learning significantly more than providers (p < .001). The training had high acceptability among participants from diverse departments and levels. CONCLUSIONS:An interactive educational training increased staff's knowledge and readiness to respond to an overdose as well as improved attitudes toward individuals living with OUD. TRIAL REGISTRATION/BACKGROUND:This project was undertaken as a quality improvement initiative at the health center and as such was not formally supervised by the Institutional Review Board per their policies. Further, per the guidelines of the International Committee of Medical Journal Editors, registration is not necessary for clinical trials whose sole purpose is to assess an intervention's effect on providers.