Faculty Bibliography

PURPOSE/OBJECTIVE:In 2014, the Association of American Medical Colleges defined 13 Core Entrustable Professional Activities (EPAs) that all graduating students should be ready to do with indirect supervision upon entering residency and commissioned a 10-school, 5-year pilot to test implementing the Core EPAs framework. In 2019, pilot schools convened trained entrustment groups (TEGs) to review assessment data and render theoretical summative entrustment decisions for class of 2019 graduates. Results were examined to determine the extent to which entrustment decisions could be made and the nature of these decisions. METHOD/METHODS:For each EPA considered (4-13 per student), TEGs recorded an entrustment determination (ready, progressing but not yet ready, evidence against student progressing, could not make a decision); confidence in that determination (none, low, moderate, high); and the number of workplace-based assessments (WBAs) considered (0->15) per determination. These individual student-level data were de-identified and merged into a multischool database; chi-square analysis tested the significance of associations between variables. RESULTS:The 2,415 EPA-specific determinations (for 349 students by 4 participating schools) resulted in a decision of ready (n = 997/2,415; 41.3%), progressing but not yet ready (n = 558/2,415; 23.1%), or evidence against student progression (n = 175/2,415; 7.2%). No decision could be made for the remaining 28.4% (685/2,415), generally for lack of data. Entrustment determinations' distribution varied across EPAs (chi-square P < .001) and, for 10/13 EPAs, WBA availability was associated with making (vs not making) entrustment decisions (each chi-square P < .05). CONCLUSIONS:TEGs were able to make many decisions about readiness for indirect supervision; yet less than half of determinations resulted in a decision of readiness to perform this EPA with indirect supervision. More work is needed at the 10 schools to enable authentic summative entrustment in the Core EPAs framework.

PURPOSE:In undergraduate medical education (UME), competency-based medical education has been operationalized through the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). Direct observation in the workplace using rigorous, valid, reliable measures is required to inform summative decisions about graduates' readiness for residency. The purpose of this study is to investigate the validity evidence of 2 proposed workplace-based entrustment scales. METHOD:The authors of this multisite, randomized, experimental study used structured vignettes and experienced raters to examine validity evidence of the Ottawa scale and the UME supervisory tool (Chen scale) in 2019. The authors used a series of 8 cases (6 developed de novo) depicting learners at preentrustable (less-developed) and entrustable (more-developed) skill levels across 5 Core EPAs. Participants from Core EPA pilot institutions rated learner performance using either the Ottawa or Chen scale. The authors used descriptive statistics and analysis of variance to examine data trends and compare ratings, conducted interrater reliability and generalizability studies to evaluate consistency among participants, and performed a content analysis of narrative comments. RESULTS:Fifty clinician-educators from 10 institutions participated, yielding 579 discrete EPA assessments. Both Ottawa and Chen scales differentiated between less- and more-developed skill levels (P < .001). The interclass correlation was good to excellent for all EPAs using Ottawa (range, 0.68-0.91) and fair to excellent using Chen (range, 0.54-0.83). Generalizability analysis revealed substantial variance in ratings attributable to the learner-EPA interaction (59.6% for Ottawa; 48.9% for Chen) suggesting variability for ratings was appropriately associated with performance on individual EPAs. CONCLUSIONS:In a structured setting, both the Ottawa and Chen scales distinguished between preentrustable and entrustable learners; however, the Ottawa scale demonstrated more desirable characteristics. These findings represent a critical step forward in developing valid, reliable instruments to measure learner progression toward entrustment for the Core EPAs.

BACKGROUND:Nearly all U.S. medical students engage in a 4-8 week period of intense preparation for their first-level licensure exams, termed a "dedicated preparation period" (DPP). It is widely assumed that student well-being is harmed during DPPs, but evidence is limited. This study characterized students' physical, intellectual, emotional, and social well-being during DPPs. METHODS:This was a cross-sectional survey sent electronically to all second-year students at four U.S. medical schools after each school's respective DPP for USMLE Step 1 or COMLEX Level 1 in 2019. Survey items assessed DPP characteristics, cost of resources, and perceived financial strain as predictors for 18 outcomes measured by items with Likert-type response options. Open-ended responses on DPPs' influence underwent thematic analysis. RESULTS:A total of 314/750 (42%) students completed surveys. DPPs lasted a median of 7 weeks (IQR 6-8 weeks), and students spent 70 h/week (IQR 56-80 h/week) studying. A total of 62 (20%) reported experiencing a significant life event that impacted their ability to study during their DPPs. Most reported 2 outcomes improved: medical knowledge base (95%) and confidence in ability to care for patients (56%). Most reported 9 outcomes worsened, including overall quality of life (72%), feeling burned out (77%), and personal anxiety (81%). A total of 25% reported paying for preparation materials strained their finances. Greater perceived financial strain was associated with worsening 11 outcomes, with reported amount spent associated with worsening 2 outcomes. Themes from student descriptions of how DPPs for first-level exams influenced them included (1) opportunity for synthesis of medical knowledge, (2) exercise of endurance and self-discipline required for professional practice, (3) dissonance among exam preparation resource content, formal curriculum, and professional values, (4) isolation, deprivation, and anguish from competing for the highest possible score, and (5) effects on well-being after DPPs. CONCLUSIONS:DPPs are currently experienced by many students as a period of personal and social deprivation, which may be worsened by perceived financial stress more than the amount of money they spend on preparation materials. DPPs should be considered as a target for reform as medical educators attempt to prevent student suffering and enhance their well-being.

OBJECTIVE:To validate an approach to measuring professional identity formation (PIF), we explore if the Professional Identity Essay (PIE), a stage score measure of medical professional identity (PI), predicts clinical communication skills. METHODS:Students completed the PIE during medical school orientation and a 3-case Objective Structured Clinical Exam (OSCE) where standardized patients reliably assessed communication skills in 5 domains. Using mediation analyses, relationships between PIE stage scores and communication skills were explored. RESULTS:For the 351 (89%) consenting students, controlling for individual characteristics, there were increases in patient counseling (6.5%, p<0.01), information gathering (4.3%, p = 0.01), organization and management (4.1%, p = 0.02), patient assessment (3.6%, p = 0.04), and relationship development (3.5%, p = 0.03) skills for every half stage increase in PIE score. The communication skills of lower socio-economic status (SES) students are indirectly impacted by their slightly higher PIE stage scores. CONCLUSION/CONCLUSIONS:Higher PIE stage scores are associated with higher communication skills and lower SES. PRACTICE IMPLICATIONS/CONCLUSIONS:PIE predicts critical clinical skills and identifies how SES and other characteristics indirectly impact future clinical performance, providing validity evidence for using PIE as a tool in longitudinal formative academic coaching, program and curriculum evaluation, and research.

One of the main goals of the CoreEPA pilot has been to determine the feasibility of developing a process to make summative entrustment decisions regarding entrustable professional activities (EPAs). Five years into the pilot, we report results of a research study we conducted to explore approaches to the entrustment process undertaken by our ten participating schools. We sought to identify the choices that participating schools made regarding the entrustment process and why these decisions were made. We are sharing these results, highlighting ongoing challenges that were identified with the intent of helping other medical schools that are moving toward EPA-based assessment. We conducted semi-structured interviews with representatives of all 10 medical schools in the CoreEPA pilot to understand their choices in designing the entrustment process. Additional information was obtained through follow-up communication to ensure completeness and accuracy of the findings. Several common themes are described. Our results indicate that, while approaches to the entrustment process vary considerably, all schools demonstrated consistent adherence to the guiding principles of the pilot. Several common barriers to the entrustment process emerged, and there was a consensus that more experience is needed with the process before consequential entrustment decisions can be made. The CoreEPA pilot schools continue to address challenges identified in implementing entrustment processes and making entrustment decisions for our students graduating in the Class of 2020.

BACKGROUND /UNASSIGNED:Graduate medical education programs employ reflection to advance a range of outcomes for physicians in training. However, the most effective applications of this tool have not been fully explored. OBJECTIVE /UNASSIGNED:A systematic review of the literature examined interventions reporting the use of reflection in graduate medical education. METHODS /UNASSIGNED:. No language or date limits were applied. The search yielded 1308 citations between inception for each database and June 15, 2015. A total of 16 studies, encompassing 477 residents and fellows, met eligibility criteria. Study quality was assessed using the Critical Appraisal Skills Programme Qualitative Checklist. The authors conducted a thematic analysis of the 16 articles. RESULTS /UNASSIGNED:Outcomes studied encompassed the impact of reflection on empathy, comfort with learning in complex situations, and engagement in the learning process. Reflection increased learning of complex subjects and deepened professional values. It appears to be an effective tool for improving attitudes and comfort when exploring difficult material. Limitations include that most studies had small samples, used volunteers, and did not measure behavioral outcomes. CONCLUSIONS /UNASSIGNED:Critical reflection is a tool that can amplify learning in residents and fellows. Added research is needed to understand how reflection can influence growth in professional capacities and patient-level outcomes in ways that can be measured.

PROBLEM: To better prepare graduating medical students to transition to the professional responsibilities of residency, 10 medical schools are participating in an Association of American Medical Colleges pilot to evaluate the feasibility of explicitly teaching and assessing 13 Core Entrustable Professional Activities for Entering Residency. The authors focused on operationalizing the concept of entrustment as part of this process. APPROACH: Starting in 2014, the Entrustment Concept Group, with representatives from each of the pilot schools, guided the development of the structures and processes necessary for formal entrustment decisions associated with students' increased responsibilities at the start of residency. OUTCOMES: Guiding principles developed by the group recommend that formal, summative entrustment decisions in undergraduate medical education be made by a trained group, be based on longitudinal performance assessments from multiple assessors, and incorporate day-to-day entrustment judgments by workplace supervisors. Key to entrustment decisions is evidence that students know their limits (discernment), can be relied on to follow through (conscientiousness), and are forthcoming despite potential personal costs (truthfulness), in addition to having the requisite knowledge and skills. The group constructed a developmental framework for discernment, conscientiousness, and truthfulness to pilot a model for transparent entrustment decision making. NEXT STEPS: The pilot schools are studying a number of questions regarding the pathways to and decisions about entrustment. This work seeks to inform meaningful culture change in undergraduate medical education through a shared understanding of the assessment of trust and a shared trust in that assessment.

AIM: To assess the feasibility and utility of measuring baseline professional identity formation (PIF) in a theory-based professionalism curriculum for early medical students. METHODS: All 132 entering students completed the professional identity essay (PIE) and the defining issues test (DIT2). Students received score reports with individualized narrative feedback and wrote a structured reflection after a large-group session in which the PIF construct was reviewed. Analysis of PIEs resulted in assignment of a full or transitional PIF stage (1-5). The DIT2 score reflects the proportion of the time students used universal ethical principles to justify a response to 6 moral dilemma cases. Students' reflections were content analyzed. RESULTS: PIF scores were distributed across stage 2/3, stage 3, stage 3/4, and stage 4. No student scores were in stages 1, 2, 4/5, or 5. The mean DIT2 score was 53% (range 9.7?76.5%); the correlation between PIF stage and DIT score was rho = 0.18 (p = 0.03). Students who took an analytic approach to the data and demonstrated both awareness that they are novices and anticipation of continued PIF tended to respond more positively to the feedback. CONCLUSIONS: These PIF scores distributed similarly to novice students in other professions. Developmental-theory based PIF and moral reasoning measures are related. Students reflected on these measures in meaningful ways suggesting utility of measuring PIF scores in medical education.