Faculty Bibliography

In this introduction, the guest editors of the "Next Era in Assessment" special collection frame the invited papers by envisioning a next era in assessment of medical education, based on ideas developed during a summit that convened professional and educational leaders and scholars. The authors posit that the next era of assessment will focus unambiguously on serving patients and the health of society, reflect its sociocultural context, and support learners' longitudinal growth and development. As such, assessment will be characterized as transformational, development-oriented and socially accountable. The authors introduce the papers in this special collection, which represent elements of a roadmap towards the next era in assessment by exploring several foundational considerations that will make the next era successful. These include the equally important issues of (1) focusing on accountability, trust and power in assessment, (2) addressing implementation and contextualization of assessment systems, (3) optimizing the use of technology in assessment, (4) establishing infrastructure for data sharing and data storage, (5) developing a vocabulary around emerging sources of assessment data, and (6) reconceptualizing validity around patient care and learner equity. Attending to these priority areas will help leaders create authentic assessment systems that are responsive to learners' and society's needs, while reaping the full promise of competency-based medical education (CBME) as well as emerging data science and artificial intelligence technologies.

OBJECTIVES/OBJECTIVE:Communication and other clinical skills are routinely assessed in medical schools using Objective Structured Clinical Examinations (OSCEs) so routinely that it can be difficult to monitor and maintain validity. We report on the accumulation of validity evidence for the Clinical Communication Skills Assessment Tool (CCSAT) based on its use with 9 cohorts of medical students in a high stakes OSCE. METHODS:) based on continuous quality improvement and use of the CCSAT for feedback, remediation, curricular design, and research. RESULTS:Implementation of the CCSAT over time has facilitated our communication skills curriculum and training. Thoughtful case development and investment in standardized patient training has contributed to data quality. Item analysis supports our behaviorally anchored scale (not done, partly and well done) and the skills domains suggested by an a priori evidence-based clinical communication model were confirmed via analysis of actual student data. Evidence synthesized across the frameworks suggests consistent validity of the CCSAT for generalization inferences (that it captures the construct), responsiveness (sensitivity to change/difference), content validity/internal structure, relationships to other variables, and consequences/implications. More evidence is needed to strengthen validity of CCSAT scores for understanding extrapolation inferences and real-world implications. CONCLUSIONS AND PRACTICE IMPLICATIONS/CONCLUSIONS:This pragmatic approach to evaluating validity within a program of assessment serves as a model for medical schools seeking to continuously monitor the quality of clinical skill assessments, a need made particularly relevant since the US NBME no longer requires the Step 2 Clinical Skills exam, leaving individual schools with the responsibility for ensuring graduates have acquired the requisite core clinical skills. We document strong evidence for CCSAT validity over time and across cohorts as well as areas for improvement and further examination.

PURPOSE/UNASSIGNED:The Professional Identity Essay (PIE) is a theory and evidence-based Medical Professional Identity Formation (MPIF) measure. We describe trajectories of PIE-measured MPIF over a 4-year US medical school curriculum. METHODS/UNASSIGNED:Students write PIEs at medical school orientation, clinical clerkships orientation, and post-advanced (near graduation) clerkship. A trained evaluator assigns an overall stage score to narrative responses to nine PIE prompts (inter-rater ICC 0.83, 95% CI [0.57 - 0.96], intra-rater ICC 0.85). Distribution of PIE stage scores across time points were analyzed in the aggregate and individual students were classified as Increase, Stable (no score change) or Decrease based on the trajectories of PIE stage scores over time. RESULTS/UNASSIGNED: CONCLUSIONS/UNASSIGNED:Medical students' PIE stage scores increase over time with three distinctive trajectories. Further study is needed to explore the utility of this method for formative assessment, program evaluation, and MPIF research.

IMPORTANCE/UNASSIGNED:Racial implicit bias can contribute to health disparities through its negative influence on physician communication with Black patients. Interventions for physicians to address racial implicit bias in their clinical encounters are limited by a lack of high-fidelity (realistic) simulations to provide opportunities for skill development and practice. OBJECTIVE/UNASSIGNED:To describe the development and initial evaluation of a high-fidelity simulation of conditions under which physicians might be influenced by implicit racial bias. DESIGN, SETTING, AND PARTICIPANTS/UNASSIGNED:This cross-sectional study, performed on an online platform from March 1 to September 30, 2022, recruited a convenience sample of physician volunteers to pilot an educational simulation. EXPOSURES/UNASSIGNED:In the simulation exercise, physicians saw a 52-year-old male standardized patient (SP) (presenting as Black or White) seeking urgent care for epigastric pain, nausea, and vomiting. The case included cognitive stressors common to clinical environments, including clinical ambiguity, stress, time constraints, and interruptions. Physicians explained their diagnosis and treatment plan to the SP, wrote an assessment and management plan, completed surveys, and took the Race Implicit Association Test (IAT) and Race Medical Cooperativeness IAT. The SPs, blinded to the purpose of the study, assessed each physician's communication using skills checklists and global rating scales. MAIN OUTCOMES AND MEASURES/UNASSIGNED:Association between physicians' IAT scores and SP race with SP ratings of communication skills. RESULTS/UNASSIGNED:In 60 physicians (23 [38.3%] Asian, 4 [6.7%] Black, 23 [38.3%] White, and 10 [16.7%] other, including Latina/o/x, Middle Eastern, and multiracial; 31 [51.7%] female, 27 [45.0%] male, and 2 [3.3%] other), the interaction of physicians' Race IAT score and SP race was significant for overall communication (mean [SD] β = -1.29 [0.41]), all subdomains of communication (mean [SD] β = -1.17 [0.52] to -1.43 [0.59]), and overall global ratings (mean [SD] β = -1.09 [0.39]). Black SPs rated physicians lower on communication skills for a given pro-White Race IAT score than White SPs; White SP ratings increased as physicians' pro-White bias increased. CONCLUSIONS AND RELEVANCE/UNASSIGNED:In this cross-sectional study, a high-fidelity simulation calibrated with cognitive stressors common to clinical environments elicited the expected influence of racial implicit bias on physicians' communication skills. The outlined process and preliminary results can inform the development and evaluation of interventions that seek to address racial implicit bias in clinical encounters and improve physician communication with Black patients.

INTRODUCTION/UNASSIGNED:We must ensure, through rigorous assessment that physicians have the evidence-based medicine (EBM) skills to identify and apply the best available information to their clinical work. However, there is limited guidance on how to assess EBM competency. With a better understanding of their current role in EBM education, Health Sciences Librarians (HSLs), as experts, should be able to contribute to the assessment of medical student EBM competence. The purpose of this study is to explore the HSLs perspective on EBM assessment practices, both current state and potential future activities. METHODS/UNASSIGNED:We conducted focus groups with librarians from across the United States to explore their perceptions of assessing EBM competence in medical students. Participants had been trained to be raters of EBM competence as part of a novel Objective Structured Clinical Examination (OSCE). This OSCE was just the starting point and the discussion covered topics of current EBM assessment and possibility for expanded responsibilities at their own institutions. We used a reflexive thematic analysis approach to construct themes from our conversations. RESULTS/UNASSIGNED:We constructed eight themes in four broad categories that influence the success of librarians being able to engage in effective assessment of EBM: administrative, curricular, medical student, and librarian. CONCLUSION/UNASSIGNED:Our results inform medical school leadership by pointing out the modifiable factors that enable librarians to be more engaged in conducting effective assessment. They highlight the need for novel tools, like EBM OSCEs, that can address multiple barriers and create opportunities for deeper integration of librarians into assessment processes.

In competency-based medical education (CBME), which is being embraced globally, the patient-learner-educator encounter occurs in a highly complex context which contributes to a wide range of assessment outcomes. Current and historical barriers to considering context in assessment include the existing post-positivist epistemological stance that values objectivity and validity evidence over the variability introduced by context. This is most evident in standardized testing. While always critical to medical education the impact of context on assessment is becoming more pronounced as many aspects of training diversify. This diversity includes an expanding interest beyond individual trainee competence to include the interdependency and collective nature of clinical competence and the growing awareness that medical education needs to be co-produced among a wider group of stakeholders. In this Eye Opener, we wish to consider: 1) How might we best account for the influence of context in the clinical competence assessment of individuals in medical education? and by doing so, 2) How could we usher in the next era of assessment that improves our ability to meet the dynamic needs of society and all its stakeholders? The purpose of this Eye Opener is thus two-fold. First, we conceptualize - from a variety of viewpoints, how we might address context in assessment of competence at the level of the individual learner. Second, we present recommendations that address how to approach implementation of a more contextualized competence assessment.

PURPOSE:To describe patterns of clinical communication skills that inform curriculum enhancement and guide coaching of medical students. MATERIALS AND METHODS:Performance data from 1182 consenting third year medical students in 9 cohorts (2011-2019), on a 17-item Clinical Communication Skills Assessment Tool (CCSAT) completed by trained Standardized Patients as part of an eight case high stakes Comprehensive Clinical Skills Exam (CCSE) were analyzed using latent profile analysis (LPA). Assessment domains included: information gathering (6 items), relationship development (5 items), patient education (3 items), and organization/time management (3 items). LPA clustered learners with similar strength/weakness into profiles based on item response patterns across cases. One-way analysis of variance (ANOVA) assessed for significant differences by profile for CCSAT items. RESULTS:Student performance clustered into six profiles in three groups, high performing (HP1 and HP2-Low Patient Education, 15.7%), average performing (AP1 and AP2-Interrupters, 40.9%), and lower performing profiles (LP1-Non-interrupters and LP2, 43.4%) with adequate model fit estimations and similar distribution in each cohort. We identified 3 CCSAT items that discriminated among learner's skill profiles. CONCLUSION:Clinical communication skill performance profiles provide nuanced, benchmarked guidance for curriculum improvement and tailoring of communication skills coaching.

PURPOSE/OBJECTIVE:Traditional quality metrics do not adequately represent the clinical work done by residents and, thus, cannot be used to link residency training to health care quality. This study aimed to determine whether electronic health record (EHR) data can be used to meaningfully assess residents' clinical performance in pediatric emergency medicine using resident-sensitive quality measures (RSQMs). METHOD/METHODS:EHR data for asthma and bronchiolitis RSQMs from Cincinnati Children's Hospital Medical Center, a quaternary children's hospital, between July 1, 2017, and June 30, 2019, were analyzed by ranking residents based on composite scores calculated using raw, unadjusted, and case-mix adjusted latent score models, with lower percentiles indicating a lower quality of care and performance. Reliability and associations between the scores produced by the 3 scoring models were compared. Resident and patient characteristics associated with performance in the highest and lowest tertiles and changes in residents' rank after case-mix adjustments were also identified. RESULTS:274 residents and 1,963 individual encounters of bronchiolitis patients aged 0-1 as well as 270 residents and 1,752 individual encounters of asthmatic patients aged 2-21 were included in the analysis. The minimum reliability requirement to create a composite score was met for asthma data (α = 0.77), but not bronchiolitis (α = 0.17). The asthma composite scores showed high correlations (r = 0.90-0.99) between raw, latent, and adjusted composite scores. After case-mix adjustments, residents' absolute percentile rank shifted on average 10 percentiles. Residents who dropped by 10 or more percentiles were likely to be more junior, saw fewer patients, cared for less acute and younger patients, or had patients with a longer emergency department stay. CONCLUSIONS:For some clinical areas, it is possible to use EHR data, adjusted for patient complexity, to meaningfully assess residents' clinical performance and identify opportunities for quality improvement.