73. ETIOLOGY OF UNABLE TO ASSESS ENTRUSTABLE PROFESSIONAL ACTIVITIES IN A NATIONAL STUDY [Meeting Abstract]
Background: Assessment using entrustable professional activities (EPAs) is currently under study as a joint effort of the American Board of Pediatrics and program director community. However, limited evidence exists for the feasibility of EPA-based assessment in pediatrics.
Aim(s): Among pediatric residency programs that implemented EPA-based assessment over 3 academic years (2015-16, 2016-17, 2017-18), determine how often programs were "unable to assess" a resident on a given EPA.
Method(s): 23 programs reported clinical competency committee (CCC) determined EPA supervision level assignments for all residents in their program for a subset of 5-6 of the 17 general pediatrics EPAs biannually (Fall and Spring), except interns who were only assessed at the end of the academic year (Spring). They indicated "unable to assess" whenever applicable.
Result(s): Across the 5 data collection cycles, there were 29,643 opportunities to assign an EPA supervision level to 1987 residents from all three post-graduate years. Of these, 4140 assignments (14%) were deemed "unable to assess." Across all EPAs, this determination was made less often in mid-year reports than year-end reports (p=.005). Post-graduate year 3 (PGY3) residents and PGY2 residents had significantly lower frequencies of "unable to assess" assignments when compared with PGY1 residents (Figure). EPAs with the lowest rates of "unable to assess" included: EPA 4 (patients with acute, common dx) at 0.4%, EPAs 10 (initial resuscitation/stabilization) and 16 (handovers) at 0.9%, EPA 11 (manage information) at 4%, and EPA 13 (practice management) at 5%. EPAs with the highest rates of "unable to assess" included EPA 8 (transition to adult care) at 43%, EPA 17 (procedures) at 27%, EPA 14 (quality improvement) at 20%, and EPAs 12 (make referrals) and 9 (behavioral and mental health) at 14%.
Conclusion(s): "Unable to assess" assignments are related to the given EPA, training year, and timing of assessment within the academic year. CCCs may be less likely to assign EPA supervision levels at timepoints viewed as higher stakes, such as transition from one training year to the next. Future work should explore this further and consider the etiology behind high levels of inability to assess some EPAs in particular.
Longitudinal Assessment of Resident Performance Using Entrustable Professional Activities
Importance/UNASSIGNED:Entrustable professional activities (EPAs) are an emerging workplace-based, patient-oriented assessment approach with limited empirical evidence. Objective/UNASSIGNED:To measure the development of pediatric trainees' clinical skills over time using EPA-based assessment data. Design, Setting, and Participants/UNASSIGNED:Prospective cohort study of categorical pediatric residents over 3 academic years (2015-2016, 2016-2017, and 2017-2018) assessed on 17 American Board of Pediatrics EPAs. Residents in training at 23 pediatric residency programs in the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network were included. Assessment was conducted by clinical competency committee members, who made summative assessment decisions regarding levels of supervision required for each resident and each EPA. Data were collected from May 2016 to November 2018 and analyzed from November to December 2018. Interventions/UNASSIGNED:Longitudinal, prospective assessment using EPAs. Main Outcomes and Measures/UNASSIGNED:Trajectories of supervision levels by EPA during residency training and how often graduating residents were deemed ready for unsupervised practice in each EPA. Results/UNASSIGNED:Across the 5 data collection cycles, 1987 residents from all 3 postgraduate years in 23 residency programs were assigned 25â€¯503 supervision level reports for the 17 general pediatrics EPAs. The 4 EPAs that required the most supervision across training were EPA 14 (quality improvement) on the 5-level scale (estimated mean level at graduation, 3.7; 95% CI, 3.6-3.7) and EPAs 8 (transition to adult care; mean, 7.0; 95% CI, 7.0-7.1), 9 (behavioral and mental health; mean, 6.6; 95% CI, 6.5-6.6), and 10 (resuscitate and stabilize; mean, 6.9; 95% CI, 6.8-7.0) on the expanded 5-level scale. At the time of graduation (36 months), the percentage of trainees who were rated at a supervision level corresponding to "unsupervised practice" varied by EPA from 53% to 98%. If performance standards were set to align with 90% of trainees achieving the level of unsupervised practice, this standard would be met for only 8 of the 17 EPAs (although 89% met this standard for EPA 17, performing the common procedures of the general pediatrician). Conclusions and Relevance/UNASSIGNED:This study presents initial evidence for empirically derived practice readiness and sets the stage for identifying curricular gaps that contribute to discrepancy between observed practice readiness and standards needed to produce physicians able to meet the health needs of the patient populations they serve. Future work should compare these findings with postgraduation outcomes data as a means of seeking validity evidence.
99. SIMULATED FIRST NIGHT-ONCALL (FNOC): ESTABLISHING COMMUNITY AND A CULTURE OF PATIENT SAFETY FOR INCOMING PEDIATRIC INTERNS [Meeting Abstract]
Background: The transition from medical student to intern presents a major patient safety concern. Our institution implemented an immersive First Night OnCall (FNOC) simulation to support transitioning trainees and cultivate a culture of safety.
Objective(s): Engage pediatric interns in a pediatric focused FNOC simulation to ensure readiness to recognize and address common safety issues in practice.
Method(s): Interns were asked to recognize patient safety hazards in a simulated patient room and participate in case based safety discussions. Interns then participated in GOSCEs (Group Observed Standardized Clinical Encounters). GOSCEs tasked trainees to obtain informed consent, evaluate a decompensating patient, recognize a mislabeled culture bottle, and give an effective patient handoff. Faculty debriefed all activities. Learners completed pre and post program assessments and a program evaluation.
Result(s): Twenty incoming interns completed FNOC. Only 11% reported any prior formal training in patient safety. Interns recognized 46% of the environmental patient safety hazards. Out of the 5 GOSCE groups, 3 called a rapid response team, 3 noted the label error for the culture bottle, and 3 obtained complete informed consent. After FNOC, 92% of interns reported increased comfort (4 or 5 on 1-5 scale) in speaking to a supervisor, escalating a situation, and reporting a medical error. All interns agreed that the case based safety discussions and the patient safety room increased readiness for internship. Almost all of the interns (85%) agreed or strongly agreed that FNOC was an effective way to learn patient safety, a good approach to improve readiness, fun, and engaging.
Conclusion(s): Incoming interns are not consistently able to demonstrate common safety practices. Engaging, immersive, simulation based experiences like FNOC may reduce this variability, while simultaneously instilling aspirational institutional norms, promoting a culture of safety, and providing a framework for effective on-boarding strategies for new trainees.
57. A "DENVER DEVELOPMENTAL" VIEW OF RESIDENT PERFORMANCE USING AN EPA FRAMEWORK [Meeting Abstract]
Introduction: The American Board of Pediatrics and pediatrics community developed entrustable professional activities (EPAs) defining the foundation of general pediatrics care. We sought to determine the developmental progression of pediatric residents across 3 years of training for the 17 General Pediatrics (GP) EPAs.
Method(s): Over 3 academic years (2015-18), 22 geographically representative residency programs in the APPD LEARN used scales with 5 or 8 anchors, describing levels of supervision from direct to indirect to entrustment, to assess EPAs biannually. Interns were assessed only at the end of Year 1. We fit growth curves to ordinal supervision levels for all 17 EPAs together using a linear mixed model and developed "Denver Developmental" charts to display the proportion of learners who have reached each given level of supervision on each EPA based on the growth curves.
Result(s): Figure 1 illustrates examples of the developmental progression for 5 of the 17 EPAs. The graphic demonstrates where 25% (beginning of white box), 75% (where white and blue boxes meet), and 90% (end of blue box) of residents achieve each supervision level for the EPA with the highest ratings (handovers) and 4 EPAs with the lowest ratings (quality improvement, resuscitate/stabilize, behavioral/mental health, and transition to adult care) across 3 years of training. Some levels were achieved by all residents by the end of the intern year (e.g., levels 1-3 for EPA 16, handovers). Most EPAs had one or more advanced levels where the 25%, 75%, or 90% data extended beyond the end of training. In those instances, the percent of residents achieving these levels is indicated on the right-hand side of Figure 1 (e.g., 12% of residents achieved level 8 for EPA 8, transition to adult care, by the end of training).
Conclusion(s): The GP EPA "Denver Developmental" progressions may help in early screening for underperforming residents. This is critical information for the learner and the program. For the former, providing feedback provides an opportunity for improvement and for the latter, if a number of learners are not progressing as expected this may indicate curricular gaps.
41. THE EFFECT OF AN ACADEMIC HALF DAY CURRICULUM ON ITE SCORES AND RESIDENT SATISFACTION WITH BOARD PREPARATION [Meeting Abstract]
Background: Resident didactics often occur during noon conferences. In 2017, the New York University School of Medicine Pediatric Residency Program transitioned from daily noon conferences to a weekly 3-hour Academic Half Day (AHD). While internal medicine residency programs have shown an association between AHD and higher in-training exam (ITE) scores1, the impact of AHD on pediatric resident ITE scores or satisfaction with preparation for the General Pediatric Board Exam is not well studied.
Objective(s): To investigate the impact of AHD on resident ITE scores and satisfaction with board preparation.
Method(s): We compared PGY2 raw ITE scores between the 2018 and 2019 graduating resident classes using a 2-sample t-test. We performed the same comparison for the PGY3 raw ITE scores. For the year between exams, the Class of 2018 received noon conferences, while the Class of 2019 received AHD. To assess resident satisfaction with board preparation, residents completed a survey prior to starting AHD and after 1 year. Satisfaction was reported on a Likert scale. Responses were categorized as "agree", "neutral", or "disagree" and analyzed with a chi-square test.
Result(s): There was no significant difference between the mean PGY2 ITE scores for the Class of 2018 (63.8 +/- 7.2) and 2019 (63.3 +/- 7.2) (p=0.8). The Class of 2019 had a significantly higher PGY3 ITE score (73.8 +/- 5.2) than the Class of 2018 (67.7 +/- 7.4) (p=0.01). When rating the statement "the current curriculum prepares me to take the boards", the percent of residents responding "agree" was 18% pre-AHD and 73% 1-year post-AHD. The association between initiation of AHD and satisfaction with exam preparation was statistically significant (X2=23.1, p<0.001).
Conclusion(s): The transition to an AHD had a positive impact on ITE scores and resident satisfaction with board preparation.
An individualized career exploration rotation: Can we impact career decision early in training? [Meeting Abstract]
BACKGROUND: The time during training that pediatric residents decide about career path is varied. Approximately half of residents plan to pursue fellowship; a percentage that is fairly stable across the 3 years of training. Individual priorities such as worklife balance, educational debt and career structure (i.e. time spent on direct patient care, research, education or administration) impact the decision. Exposure to possible career options early in training may help decision making.
OBJECTIVE(S): To determine interns perception of the usefulness of a 2-week individualized career exploration (CE) rotation.
METHOD(S): In July 2016 we implemented a 2-week CE rotation for interns that was individually designed to expose them to an area within pediatric medicine of their choosing that they are considering for their career. The experience was specifically tailored to emphasize aspects they may not typically be exposed to during an elective, such as faculty scholarship, career path and lifestyle. Interns were surveyed before and after their CE rotation. We used descriptive statistics to analyze intern's perceptions of preparedness to decide about post-residency career path.
RESULT(S): 24 interns completed pre and post CE rotation surveys. On the pre-rotation survey, when asked how prepared they felt to decide about their post-residency career 13% were neutral and 54% felt un-prepared. We found no association between timing of CE rotation during intern year and level of confidence to make a career decision. 79% felt positive that a CE rotation would help with making a career decision. On the post-rotation survey 83% of interns felt better prepared to make a career decision and 100% of interns felt the CE rotation was worthwhile. When asked which of the assigned rotation activities where most useful 42% of interns chose faculty shadowing, 32% chose conducting a faculty mini-interview, and 25% chose the self-reflection exercise.
CONCLUSION(S): A CE rotation during intern year can increase perceptions of preparedness to decide about post-residency career path