Faculty Bibliography

PURPOSE/UNASSIGNED:Understanding medical students' readiness to perform basic entrustable professional activities (EPAs) informs tailored support during transition to residency. METHODS/UNASSIGNED:Night-onCall (NOC) was developed to assess medical student readiness to perform core EPAs on day one of internship. NOC is a complex, integrated simulation centered around three clinical cases. Assessments include standardized patient (SP), nurse (SN), physician attending (SA), and resident (SR) perspectives using clinical competency rating instruments mapped to EPAs and scored as Well Done (WD), Partly Done (PD) or Not Done (ND). Faculty rate clinical reasoning using a rubric evaluating written post-encounter notes as poor, beginning, competent, or strong. The ability to recognize lapses in patient safety is assessed based on written case responses. Medical librarians evaluate students' ability to formulate a clinical question and search for evidence. RESULTS/UNASSIGNED:Data was collected from 'near-graduates' from seven USA medical schools from 2020 to 2023 (n = 1116). Overall, SPs rated 75.0% of overall communication skills and only 56% of patient education items as WD. SNs rated interprofessional communication 57.0% WD, and SRs rated intraprofessional communication 61.0% WD. History gathering and physical exam skills varied by case. Faculty rated clinical reasoning as beginning (45%) or competent (44%) and librarians rated 16% of literature searches as WD. CONCLUSION/UNASSIGNED:Most near-graduates demonstrated competent basic patient communication skills but performed less well on patient education, communication with other team members, clinical reasoning, and accessing the evidence base to answer clinical questions. These overall trends, consistent across schools and year, provide benchmarks for clinical training.

OBJECTIVES/OBJECTIVE:Communication and other clinical skills are routinely assessed in medical schools using Objective Structured Clinical Examinations (OSCEs) so routinely that it can be difficult to monitor and maintain validity. We report on the accumulation of validity evidence for the Clinical Communication Skills Assessment Tool (CCSAT) based on its use with 9 cohorts of medical students in a high stakes OSCE. METHODS:) based on continuous quality improvement and use of the CCSAT for feedback, remediation, curricular design, and research. RESULTS:Implementation of the CCSAT over time has facilitated our communication skills curriculum and training. Thoughtful case development and investment in standardized patient training has contributed to data quality. Item analysis supports our behaviorally anchored scale (not done, partly and well done) and the skills domains suggested by an a priori evidence-based clinical communication model were confirmed via analysis of actual student data. Evidence synthesized across the frameworks suggests consistent validity of the CCSAT for generalization inferences (that it captures the construct), responsiveness (sensitivity to change/difference), content validity/internal structure, relationships to other variables, and consequences/implications. More evidence is needed to strengthen validity of CCSAT scores for understanding extrapolation inferences and real-world implications. CONCLUSIONS AND PRACTICE IMPLICATIONS/CONCLUSIONS:This pragmatic approach to evaluating validity within a program of assessment serves as a model for medical schools seeking to continuously monitor the quality of clinical skill assessments, a need made particularly relevant since the US NBME no longer requires the Step 2 Clinical Skills exam, leaving individual schools with the responsibility for ensuring graduates have acquired the requisite core clinical skills. We document strong evidence for CCSAT validity over time and across cohorts as well as areas for improvement and further examination.

PURPOSE/UNASSIGNED:The Professional Identity Essay (PIE) is a theory and evidence-based Medical Professional Identity Formation (MPIF) measure. We describe trajectories of PIE-measured MPIF over a 4-year US medical school curriculum. METHODS/UNASSIGNED:Students write PIEs at medical school orientation, clinical clerkships orientation, and post-advanced (near graduation) clerkship. A trained evaluator assigns an overall stage score to narrative responses to nine PIE prompts (inter-rater ICC 0.83, 95% CI [0.57 - 0.96], intra-rater ICC 0.85). Distribution of PIE stage scores across time points were analyzed in the aggregate and individual students were classified as Increase, Stable (no score change) or Decrease based on the trajectories of PIE stage scores over time. RESULTS/UNASSIGNED: CONCLUSIONS/UNASSIGNED:Medical students' PIE stage scores increase over time with three distinctive trajectories. Further study is needed to explore the utility of this method for formative assessment, program evaluation, and MPIF research.

PURPOSE:To describe patterns of clinical communication skills that inform curriculum enhancement and guide coaching of medical students. MATERIALS AND METHODS:Performance data from 1182 consenting third year medical students in 9 cohorts (2011-2019), on a 17-item Clinical Communication Skills Assessment Tool (CCSAT) completed by trained Standardized Patients as part of an eight case high stakes Comprehensive Clinical Skills Exam (CCSE) were analyzed using latent profile analysis (LPA). Assessment domains included: information gathering (6 items), relationship development (5 items), patient education (3 items), and organization/time management (3 items). LPA clustered learners with similar strength/weakness into profiles based on item response patterns across cases. One-way analysis of variance (ANOVA) assessed for significant differences by profile for CCSAT items. RESULTS:Student performance clustered into six profiles in three groups, high performing (HP1 and HP2-Low Patient Education, 15.7%), average performing (AP1 and AP2-Interrupters, 40.9%), and lower performing profiles (LP1-Non-interrupters and LP2, 43.4%) with adequate model fit estimations and similar distribution in each cohort. We identified 3 CCSAT items that discriminated among learner's skill profiles. CONCLUSION:Clinical communication skill performance profiles provide nuanced, benchmarked guidance for curriculum improvement and tailoring of communication skills coaching.

PURPOSE/OBJECTIVE:In 2014, the Association of American Medical Colleges defined 13 Core Entrustable Professional Activities (EPAs) that all graduating students should be ready to do with indirect supervision upon entering residency and commissioned a 10-school, 5-year pilot to test implementing the Core EPAs framework. In 2019, pilot schools convened trained entrustment groups (TEGs) to review assessment data and render theoretical summative entrustment decisions for class of 2019 graduates. Results were examined to determine the extent to which entrustment decisions could be made and the nature of these decisions. METHOD/METHODS:For each EPA considered (4-13 per student), TEGs recorded an entrustment determination (ready, progressing but not yet ready, evidence against student progressing, could not make a decision); confidence in that determination (none, low, moderate, high); and the number of workplace-based assessments (WBAs) considered (0->15) per determination. These individual student-level data were de-identified and merged into a multischool database; chi-square analysis tested the significance of associations between variables. RESULTS:The 2,415 EPA-specific determinations (for 349 students by 4 participating schools) resulted in a decision of ready (n = 997/2,415; 41.3%), progressing but not yet ready (n = 558/2,415; 23.1%), or evidence against student progression (n = 175/2,415; 7.2%). No decision could be made for the remaining 28.4% (685/2,415), generally for lack of data. Entrustment determinations' distribution varied across EPAs (chi-square P < .001) and, for 10/13 EPAs, WBA availability was associated with making (vs not making) entrustment decisions (each chi-square P < .05). CONCLUSIONS:TEGs were able to make many decisions about readiness for indirect supervision; yet less than half of determinations resulted in a decision of readiness to perform this EPA with indirect supervision. More work is needed at the 10 schools to enable authentic summative entrustment in the Core EPAs framework.

OBJECTIVE:To validate an approach to measuring professional identity formation (PIF), we explore if the Professional Identity Essay (PIE), a stage score measure of medical professional identity (PI), predicts clinical communication skills. METHODS:Students completed the PIE during medical school orientation and a 3-case Objective Structured Clinical Exam (OSCE) where standardized patients reliably assessed communication skills in 5 domains. Using mediation analyses, relationships between PIE stage scores and communication skills were explored. RESULTS:For the 351 (89%) consenting students, controlling for individual characteristics, there were increases in patient counseling (6.5%, p<0.01), information gathering (4.3%, p = 0.01), organization and management (4.1%, p = 0.02), patient assessment (3.6%, p = 0.04), and relationship development (3.5%, p = 0.03) skills for every half stage increase in PIE score. The communication skills of lower socio-economic status (SES) students are indirectly impacted by their slightly higher PIE stage scores. CONCLUSION/CONCLUSIONS:Higher PIE stage scores are associated with higher communication skills and lower SES. PRACTICE IMPLICATIONS/CONCLUSIONS:PIE predicts critical clinical skills and identifies how SES and other characteristics indirectly impact future clinical performance, providing validity evidence for using PIE as a tool in longitudinal formative academic coaching, program and curriculum evaluation, and research.

PURPOSE/OBJECTIVE:The purpose of this study was to evaluate the effectiveness of an annual oral-systemic health interprofessional education (IPE) clinical simulation and case study experience with nurse practitioner/midwifery (NP/MW), dental (DDS), medical (MD), and pharmacy (PharmD) students. METHODS:The Interprofessional Collaborative Competency Attainment Scale (ICCAS) was used to measure students' self-reported attainment of interprofessional competencies before and after the IPE experience. Pre- and post-test surveys were completed by NP/MW, DDS, MD, and PharmD student cohorts from 2017 to 2019. Students also had the opportunity to provide qualitative feedback about their experience at post-test. Data were collected from IPE faculty facilitators to assess their perception of the value of the Teaching Oral-Systemic Health (TOSH) program. RESULTS:Student ICCAS results demonstrated statistically significant improvement in self-reported interprofessional competencies among all types of students across all 3 years (P < 0.001); qualitative student comments reflected positive experiences with the TOSH program. Survey data from IPE faculty facilitators supported the value of the IPE experience for all students. CONCLUSIONS:The findings demonstrate the effectiveness of the TOSH program in using oral-systemic health as a clinical exemplar to develop interprofessional competencies. The 2017-2019 data reinforce the credibility of scaling the TOSH model for developing interprofessional competencies with students from different health professions.