Faculty Bibliography

OBJECTIVES/OBJECTIVE:Evidence for the impact of racial implicit bias and clinical reasoning remains conflicting. Our inability to characterize the relationship between racial implicit bias and clinical reasoning (CR) precludes development of comprehensive interventions seeking to address the impact of racial implicit bias on clinical encounters. To address this gap, we conducted a simulation-based investigation with clinical presentations with known health disparities, cognitive stressors common to clinical environments, and Black and White standardized patients (SPs). METHODS:We recruited 75 early-career generalist physicians from five academic medical centers in New York, NY, USA. Physicians engaged in a three-station simulation. The first was for level-setting, familiarizing physicians with the online platform. The second was a diagnostic dilemma - an atypical presentation of acute coronary syndrome (ACS). The third tested treatment decision-making in acute pain. Immediately afterward, physicians completed the Race Implicit Association Test (IAT) and Race Medical Cooperativeness (RMC) IAT measuring affective and cognitive implicit biases, respectively. Two investigators assessed CR accuracy by applying a scoring rubric to physicians' post-encounter notes. RESULTS:Statistical analyses revealed no significant correlations between physicians' Race-IAT scores and CR by SP race. However, for ACS, a moderate correlation was suggested between physicians' RMC-IAT scores and CR accuracy when seeing Black (R=0.36, CI -0.04 to 0.6), but not White, SPs (R=0.1, CI -0.44 to 0.24). CONCLUSIONS:This study expands our understanding of the complex impact of racial implicit bias on clinical encounters. Future, larger studies should explore affective and cognitive implicit biases' effects on CR across varied clinical scenarios and contexts with diverse SPs.

OBJECTIVE/UNASSIGNED:We expect medical students to be able to apply evidence-based medicine (EBM) skills in the context of the clinical care of patients. Previous assessments of this domain have primarily utilized decontextualized knowledge tests, which provide limited insights into students' understanding of EBM skills in the context of patient care. New performance-based EBM competence assessments using Objective Structured Clinical Examinations (OSCEs) are being developed and tested. Understanding how students experience and interact with a simulation-based assessment of EBM competence would enable us to improve the modality. METHODS/UNASSIGNED:We recruited 13 graduating medical students from one medical school who had recently completed an immersive multi station readiness-for-residency OSCE (Night onCall) which included a case-based EBM assessment. We conducted individual interviews to explore their perceptions of participating in this OSCE as a method of EBM assessment. The interviews were transcribed, coded, and analyzed using Dedoose by three health science librarians. RESULTS/UNASSIGNED:Students discussed their experience and perceptions in six main areas: connection to clinical practice, curricular timing and content coverage, feedback, station instructions, awareness of their own limitations, and an OSCE as a format for assessing EBM. CONCLUSION/UNASSIGNED:Medical students appreciated the EBM OSCE because it enhanced their learning about how to integrate EBM into clinical practice. They proposed implementing multiple such opportunities throughout medical school because it would improve their competence and provide highly impactful opportunities to build toward EBM mastery. They endorsed that this would be well-accepted by medical students.

In this introduction, the guest editors of the "Next Era in Assessment" special collection frame the invited papers by envisioning a next era in assessment of medical education, based on ideas developed during a summit that convened professional and educational leaders and scholars. The authors posit that the next era of assessment will focus unambiguously on serving patients and the health of society, reflect its sociocultural context, and support learners' longitudinal growth and development. As such, assessment will be characterized as transformational, development-oriented and socially accountable. The authors introduce the papers in this special collection, which represent elements of a roadmap towards the next era in assessment by exploring several foundational considerations that will make the next era successful. These include the equally important issues of (1) focusing on accountability, trust and power in assessment, (2) addressing implementation and contextualization of assessment systems, (3) optimizing the use of technology in assessment, (4) establishing infrastructure for data sharing and data storage, (5) developing a vocabulary around emerging sources of assessment data, and (6) reconceptualizing validity around patient care and learner equity. Attending to these priority areas will help leaders create authentic assessment systems that are responsive to learners' and society's needs, while reaping the full promise of competency-based medical education (CBME) as well as emerging data science and artificial intelligence technologies.

OBJECTIVES/OBJECTIVE:Communication and other clinical skills are routinely assessed in medical schools using Objective Structured Clinical Examinations (OSCEs) so routinely that it can be difficult to monitor and maintain validity. We report on the accumulation of validity evidence for the Clinical Communication Skills Assessment Tool (CCSAT) based on its use with 9 cohorts of medical students in a high stakes OSCE. METHODS:) based on continuous quality improvement and use of the CCSAT for feedback, remediation, curricular design, and research. RESULTS:Implementation of the CCSAT over time has facilitated our communication skills curriculum and training. Thoughtful case development and investment in standardized patient training has contributed to data quality. Item analysis supports our behaviorally anchored scale (not done, partly and well done) and the skills domains suggested by an a priori evidence-based clinical communication model were confirmed via analysis of actual student data. Evidence synthesized across the frameworks suggests consistent validity of the CCSAT for generalization inferences (that it captures the construct), responsiveness (sensitivity to change/difference), content validity/internal structure, relationships to other variables, and consequences/implications. More evidence is needed to strengthen validity of CCSAT scores for understanding extrapolation inferences and real-world implications. CONCLUSIONS AND PRACTICE IMPLICATIONS/CONCLUSIONS:This pragmatic approach to evaluating validity within a program of assessment serves as a model for medical schools seeking to continuously monitor the quality of clinical skill assessments, a need made particularly relevant since the US NBME no longer requires the Step 2 Clinical Skills exam, leaving individual schools with the responsibility for ensuring graduates have acquired the requisite core clinical skills. We document strong evidence for CCSAT validity over time and across cohorts as well as areas for improvement and further examination.

PURPOSE/UNASSIGNED:The Professional Identity Essay (PIE) is a theory and evidence-based Medical Professional Identity Formation (MPIF) measure. We describe trajectories of PIE-measured MPIF over a 4-year US medical school curriculum. METHODS/UNASSIGNED:Students write PIEs at medical school orientation, clinical clerkships orientation, and post-advanced (near graduation) clerkship. A trained evaluator assigns an overall stage score to narrative responses to nine PIE prompts (inter-rater ICC 0.83, 95% CI [0.57 - 0.96], intra-rater ICC 0.85). Distribution of PIE stage scores across time points were analyzed in the aggregate and individual students were classified as Increase, Stable (no score change) or Decrease based on the trajectories of PIE stage scores over time. RESULTS/UNASSIGNED: CONCLUSIONS/UNASSIGNED:Medical students' PIE stage scores increase over time with three distinctive trajectories. Further study is needed to explore the utility of this method for formative assessment, program evaluation, and MPIF research.

IMPORTANCE/UNASSIGNED:Racial implicit bias can contribute to health disparities through its negative influence on physician communication with Black patients. Interventions for physicians to address racial implicit bias in their clinical encounters are limited by a lack of high-fidelity (realistic) simulations to provide opportunities for skill development and practice. OBJECTIVE/UNASSIGNED:To describe the development and initial evaluation of a high-fidelity simulation of conditions under which physicians might be influenced by implicit racial bias. DESIGN, SETTING, AND PARTICIPANTS/UNASSIGNED:This cross-sectional study, performed on an online platform from March 1 to September 30, 2022, recruited a convenience sample of physician volunteers to pilot an educational simulation. EXPOSURES/UNASSIGNED:In the simulation exercise, physicians saw a 52-year-old male standardized patient (SP) (presenting as Black or White) seeking urgent care for epigastric pain, nausea, and vomiting. The case included cognitive stressors common to clinical environments, including clinical ambiguity, stress, time constraints, and interruptions. Physicians explained their diagnosis and treatment plan to the SP, wrote an assessment and management plan, completed surveys, and took the Race Implicit Association Test (IAT) and Race Medical Cooperativeness IAT. The SPs, blinded to the purpose of the study, assessed each physician's communication using skills checklists and global rating scales. MAIN OUTCOMES AND MEASURES/UNASSIGNED:Association between physicians' IAT scores and SP race with SP ratings of communication skills. RESULTS/UNASSIGNED:In 60 physicians (23 [38.3%] Asian, 4 [6.7%] Black, 23 [38.3%] White, and 10 [16.7%] other, including Latina/o/x, Middle Eastern, and multiracial; 31 [51.7%] female, 27 [45.0%] male, and 2 [3.3%] other), the interaction of physicians' Race IAT score and SP race was significant for overall communication (mean [SD] β = -1.29 [0.41]), all subdomains of communication (mean [SD] β = -1.17 [0.52] to -1.43 [0.59]), and overall global ratings (mean [SD] β = -1.09 [0.39]). Black SPs rated physicians lower on communication skills for a given pro-White Race IAT score than White SPs; White SP ratings increased as physicians' pro-White bias increased. CONCLUSIONS AND RELEVANCE/UNASSIGNED:In this cross-sectional study, a high-fidelity simulation calibrated with cognitive stressors common to clinical environments elicited the expected influence of racial implicit bias on physicians' communication skills. The outlined process and preliminary results can inform the development and evaluation of interventions that seek to address racial implicit bias in clinical encounters and improve physician communication with Black patients.

INTRODUCTION/UNASSIGNED:We must ensure, through rigorous assessment that physicians have the evidence-based medicine (EBM) skills to identify and apply the best available information to their clinical work. However, there is limited guidance on how to assess EBM competency. With a better understanding of their current role in EBM education, Health Sciences Librarians (HSLs), as experts, should be able to contribute to the assessment of medical student EBM competence. The purpose of this study is to explore the HSLs perspective on EBM assessment practices, both current state and potential future activities. METHODS/UNASSIGNED:We conducted focus groups with librarians from across the United States to explore their perceptions of assessing EBM competence in medical students. Participants had been trained to be raters of EBM competence as part of a novel Objective Structured Clinical Examination (OSCE). This OSCE was just the starting point and the discussion covered topics of current EBM assessment and possibility for expanded responsibilities at their own institutions. We used a reflexive thematic analysis approach to construct themes from our conversations. RESULTS/UNASSIGNED:We constructed eight themes in four broad categories that influence the success of librarians being able to engage in effective assessment of EBM: administrative, curricular, medical student, and librarian. CONCLUSION/UNASSIGNED:Our results inform medical school leadership by pointing out the modifiable factors that enable librarians to be more engaged in conducting effective assessment. They highlight the need for novel tools, like EBM OSCEs, that can address multiple barriers and create opportunities for deeper integration of librarians into assessment processes.

In competency-based medical education (CBME), which is being embraced globally, the patient-learner-educator encounter occurs in a highly complex context which contributes to a wide range of assessment outcomes. Current and historical barriers to considering context in assessment include the existing post-positivist epistemological stance that values objectivity and validity evidence over the variability introduced by context. This is most evident in standardized testing. While always critical to medical education the impact of context on assessment is becoming more pronounced as many aspects of training diversify. This diversity includes an expanding interest beyond individual trainee competence to include the interdependency and collective nature of clinical competence and the growing awareness that medical education needs to be co-produced among a wider group of stakeholders. In this Eye Opener, we wish to consider: 1) How might we best account for the influence of context in the clinical competence assessment of individuals in medical education? and by doing so, 2) How could we usher in the next era of assessment that improves our ability to meet the dynamic needs of society and all its stakeholders? The purpose of this Eye Opener is thus two-fold. First, we conceptualize - from a variety of viewpoints, how we might address context in assessment of competence at the level of the individual learner. Second, we present recommendations that address how to approach implementation of a more contextualized competence assessment.

PURPOSE:To describe patterns of clinical communication skills that inform curriculum enhancement and guide coaching of medical students. MATERIALS AND METHODS:Performance data from 1182 consenting third year medical students in 9 cohorts (2011-2019), on a 17-item Clinical Communication Skills Assessment Tool (CCSAT) completed by trained Standardized Patients as part of an eight case high stakes Comprehensive Clinical Skills Exam (CCSE) were analyzed using latent profile analysis (LPA). Assessment domains included: information gathering (6 items), relationship development (5 items), patient education (3 items), and organization/time management (3 items). LPA clustered learners with similar strength/weakness into profiles based on item response patterns across cases. One-way analysis of variance (ANOVA) assessed for significant differences by profile for CCSAT items. RESULTS:Student performance clustered into six profiles in three groups, high performing (HP1 and HP2-Low Patient Education, 15.7%), average performing (AP1 and AP2-Interrupters, 40.9%), and lower performing profiles (LP1-Non-interrupters and LP2, 43.4%) with adequate model fit estimations and similar distribution in each cohort. We identified 3 CCSAT items that discriminated among learner's skill profiles. CONCLUSION:Clinical communication skill performance profiles provide nuanced, benchmarked guidance for curriculum improvement and tailoring of communication skills coaching.