Faculty Bibliography

OBJECTIVES/OBJECTIVE:Communication and other clinical skills are routinely assessed in medical schools using Objective Structured Clinical Examinations (OSCEs) so routinely that it can be difficult to monitor and maintain validity. We report on the accumulation of validity evidence for the Clinical Communication Skills Assessment Tool (CCSAT) based on its use with 9 cohorts of medical students in a high stakes OSCE. METHODS:) based on continuous quality improvement and use of the CCSAT for feedback, remediation, curricular design, and research. RESULTS:Implementation of the CCSAT over time has facilitated our communication skills curriculum and training. Thoughtful case development and investment in standardized patient training has contributed to data quality. Item analysis supports our behaviorally anchored scale (not done, partly and well done) and the skills domains suggested by an a priori evidence-based clinical communication model were confirmed via analysis of actual student data. Evidence synthesized across the frameworks suggests consistent validity of the CCSAT for generalization inferences (that it captures the construct), responsiveness (sensitivity to change/difference), content validity/internal structure, relationships to other variables, and consequences/implications. More evidence is needed to strengthen validity of CCSAT scores for understanding extrapolation inferences and real-world implications. CONCLUSIONS AND PRACTICE IMPLICATIONS/CONCLUSIONS:This pragmatic approach to evaluating validity within a program of assessment serves as a model for medical schools seeking to continuously monitor the quality of clinical skill assessments, a need made particularly relevant since the US NBME no longer requires the Step 2 Clinical Skills exam, leaving individual schools with the responsibility for ensuring graduates have acquired the requisite core clinical skills. We document strong evidence for CCSAT validity over time and across cohorts as well as areas for improvement and further examination.

PURPOSE/UNASSIGNED:The Professional Identity Essay (PIE) is a theory and evidence-based Medical Professional Identity Formation (MPIF) measure. We describe trajectories of PIE-measured MPIF over a 4-year US medical school curriculum. METHODS/UNASSIGNED:Students write PIEs at medical school orientation, clinical clerkships orientation, and post-advanced (near graduation) clerkship. A trained evaluator assigns an overall stage score to narrative responses to nine PIE prompts (inter-rater ICC 0.83, 95% CI [0.57 - 0.96], intra-rater ICC 0.85). Distribution of PIE stage scores across time points were analyzed in the aggregate and individual students were classified as Increase, Stable (no score change) or Decrease based on the trajectories of PIE stage scores over time. RESULTS/UNASSIGNED: CONCLUSIONS/UNASSIGNED:Medical students' PIE stage scores increase over time with three distinctive trajectories. Further study is needed to explore the utility of this method for formative assessment, program evaluation, and MPIF research.

Background Residents lack confidence in caring for transgender individuals. More exposure and practice throughout training is needed. Objective To explore whether and how prior exposure to transgender health skills during medical school impacted competency with these skills during residency. Methods In 2022, all 101 internal medicine residents at New York University Grossman School of Medicine participated in an objective structured clinical examination (OSCE) station as part of their annual formative assessment where they cared for a standardized patient (SP) who identified as transgender. Three SPs who were members of the transgender community were recruited through online and social media forums. Two resident groups (continuum vs noncontinuum) differed in their prior experiences with transgender OSCEs during medical school. We analyzed SPs"™ ratings of resident performance using checklist data and SP open-ended feedback to compare performance between groups and resident post-OSCE evaluations to understand residents"™ perceptions of the educational value of the case. Results Residents with prior experience with transgender SPs (continuum) were more frequently recommended by SPs (88% [21 of 24] vs 70% [54 of 77]) to a family member or friend, were all rated professional (100% [24 of 24] vs 94% [72 of 94]) and scored better in pain information-gathering (92% vs 65%, mean summary score) and gender-affirming care skills (67% vs 52%, mean summary score). Noncontinuum residents lacked experience, missed opportunities to ask about gender identity, and needed work on demonstrating comfort and using proper language. Most residents completing a post-OSCE evaluation (80%, 41 of 51) rated the case as "very valuable." Conclusions Spaced practice and feedback through early exposure to transgender OSCEs were valuable for skill acquisition, giving continuum residents a learning advantage compared to noncontinuum residents.

Purpose: To describe patterns of clinical communication skills that inform curriculum enhancement and guide coaching of medical students. Materials and methods: Performance data from 1182 consenting third year medical students in 9 cohorts (2011"“2019), on a 17-item Clinical Communication Skills Assessment Tool (CCSAT) completed by trained Standardized Patients as part of an eight case high stakes Comprehensive Clinical Skills Exam (CCSE) were analyzed using latent profile analysis (LPA). Assessment domains included: information gathering (6 items), relationship development (5 items), patient education (3 items), and organization/time management (3 items). LPA clustered learners with similar strength/weakness into profiles based on item response patterns across cases. One-way analysis of variance (ANOVA) assessed for significant differences by profile for CCSAT items. Results: Student performance clustered into six profiles in three groups, high performing (HP1 and HP2-Low Patient Education, 15.7%), average performing (AP1 and AP2-Interrupters, 40.9%), and lower performing profiles (LP1-Non-interrupters and LP2, 43.4%) with adequate model fit estimations and similar distribution in each cohort. We identified 3 CCSAT items that discriminated among learner"™s skill profiles. Conclusion: Clinical communication skill performance profiles provide nuanced, benchmarked guidance for curriculum improvement and tailoring of communication skills coaching.

Purpose: The Professional Identity Essay (PIE) is a theory and evidence-based Medical Professional Identity Formation (MPIF) measure. We describe trajectories of PIE-measured MPIF over a 4-year US medical school curriculum. Methods: Students write PIEs at medical school orientation, clinical clerkships orientation, and post-advanced (near graduation) clerkship. A trained evaluator assigns an overall stage score to narrative responses to nine PIE prompts (inter-rater ICC 0.83, 95% CI [0.57 − 0.96], intra-rater ICC 0.85). Distribution of PIE stage scores across time points were analyzed in the aggregate and individual students were classified as Increase, Stable (no score change) or Decrease based on the trajectories of PIE stage scores over time. Results 202 students completed 592 PIEs from 2018-2023. There was a significant change in the proportion of PIEs in stages over time (X² 84.40, p < 0.001), 47% (n = 95) students were categorized in the Increase trajectory, 45.5% (n = 92) as Stable and 7.4% (n = 15) as Decrease. Older age and time-predicted stage scores change within trajectories (p < 0.05). Conclusions Medical students"™ PIE stage scores increase over time with three distinctive trajectories. Further study is needed to explore the utility of this method for formative assessment, program evaluation, and MPIF research.

PURPOSE/OBJECTIVE:In 2014, the Association of American Medical Colleges defined 13 Core Entrustable Professional Activities (EPAs) that all graduating students should be ready to do with indirect supervision upon entering residency and commissioned a 10-school, 5-year pilot to test implementing the Core EPAs framework. In 2019, pilot schools convened trained entrustment groups (TEGs) to review assessment data and render theoretical summative entrustment decisions for class of 2019 graduates. Results were examined to determine the extent to which entrustment decisions could be made and the nature of these decisions. METHOD/METHODS:For each EPA considered (4-13 per student), TEGs recorded an entrustment determination (ready, progressing but not yet ready, evidence against student progressing, could not make a decision); confidence in that determination (none, low, moderate, high); and the number of workplace-based assessments (WBAs) considered (0->15) per determination. These individual student-level data were de-identified and merged into a multischool database; chi-square analysis tested the significance of associations between variables. RESULTS:The 2,415 EPA-specific determinations (for 349 students by 4 participating schools) resulted in a decision of ready (n = 997/2,415; 41.3%), progressing but not yet ready (n = 558/2,415; 23.1%), or evidence against student progression (n = 175/2,415; 7.2%). No decision could be made for the remaining 28.4% (685/2,415), generally for lack of data. Entrustment determinations' distribution varied across EPAs (chi-square P < .001) and, for 10/13 EPAs, WBA availability was associated with making (vs not making) entrustment decisions (each chi-square P < .05). CONCLUSIONS:TEGs were able to make many decisions about readiness for indirect supervision; yet less than half of determinations resulted in a decision of readiness to perform this EPA with indirect supervision. More work is needed at the 10 schools to enable authentic summative entrustment in the Core EPAs framework.

OBJECTIVE:To validate an approach to measuring professional identity formation (PIF), we explore if the Professional Identity Essay (PIE), a stage score measure of medical professional identity (PI), predicts clinical communication skills. METHODS:Students completed the PIE during medical school orientation and a 3-case Objective Structured Clinical Exam (OSCE) where standardized patients reliably assessed communication skills in 5 domains. Using mediation analyses, relationships between PIE stage scores and communication skills were explored. RESULTS:For the 351 (89%) consenting students, controlling for individual characteristics, there were increases in patient counseling (6.5%, p<0.01), information gathering (4.3%, p = 0.01), organization and management (4.1%, p = 0.02), patient assessment (3.6%, p = 0.04), and relationship development (3.5%, p = 0.03) skills for every half stage increase in PIE score. The communication skills of lower socio-economic status (SES) students are indirectly impacted by their slightly higher PIE stage scores. CONCLUSION/CONCLUSIONS:Higher PIE stage scores are associated with higher communication skills and lower SES. PRACTICE IMPLICATIONS/CONCLUSIONS:PIE predicts critical clinical skills and identifies how SES and other characteristics indirectly impact future clinical performance, providing validity evidence for using PIE as a tool in longitudinal formative academic coaching, program and curriculum evaluation, and research.

PURPOSE/OBJECTIVE:The purpose of this study was to evaluate the effectiveness of an annual oral-systemic health interprofessional education (IPE) clinical simulation and case study experience with nurse practitioner/midwifery (NP/MW), dental (DDS), medical (MD), and pharmacy (PharmD) students. METHODS:The Interprofessional Collaborative Competency Attainment Scale (ICCAS) was used to measure students' self-reported attainment of interprofessional competencies before and after the IPE experience. Pre- and post-test surveys were completed by NP/MW, DDS, MD, and PharmD student cohorts from 2017 to 2019. Students also had the opportunity to provide qualitative feedback about their experience at post-test. Data were collected from IPE faculty facilitators to assess their perception of the value of the Teaching Oral-Systemic Health (TOSH) program. RESULTS:Student ICCAS results demonstrated statistically significant improvement in self-reported interprofessional competencies among all types of students across all 3 years (P < 0.001); qualitative student comments reflected positive experiences with the TOSH program. Survey data from IPE faculty facilitators supported the value of the IPE experience for all students. CONCLUSIONS:The findings demonstrate the effectiveness of the TOSH program in using oral-systemic health as a clinical exemplar to develop interprofessional competencies. The 2017-2019 data reinforce the credibility of scaling the TOSH model for developing interprofessional competencies with students from different health professions.