Faculty Bibliography

Competency-based medical education (CBME) has produced large collections of data, which can provide valuable information about trainees and medical education systems. Many organizations continue to struggle with accessing, collecting, governing, analyzing, and visualizing their clinical and/or educational data. This hinders data sharing efforts within and across organizations, which are foundational in supporting system-wide improvements. Challenges to data sharing within medical education include variability in legislation, existing data policies, heterogeneity of data, inadequate data infrastructure, and various intended purposes or uses. In this eye opener, the authors describe four case studies to illustrate some of the aforementioned challenges and characterize the complexity of data sharing within medical education along two dimensions: organizational (single vs. multiple) and data type (clinical and/or educational). With the goal of better supporting data sharing initiatives, the authors introduce an action-oriented blueprint that includes a three-stage process (i.e., preparation, execution, and iteration) to highlight crucial aspects of data sharing. This evidence-informed model incorporates current best practices and aims to support data sharing initiatives within their own organizations and across multiple organizations. Finally, organizations can use this model to conceptually guide and track their progression throughout the data sharing process.

BACKGROUND/UNASSIGNED:A lack of educational continuity creates disorienting friction at the onset of residency. Few programs have harnessed the benefits of coaching, which can facilitate self-directed learning, competency development, and professional identity formation, to help ease this transition. OBJECTIVE/UNASSIGNED:To describe the process of training faculty Bridge Coaches for the Transition to Residency Advantage (TRA) program for interns. METHODS/UNASSIGNED:Nineteen graduate faculty educators participated in a coaching training course with formative skills assessment as part of a faculty development program starting in January 2020. Surveys (n = 15; 79%) and a focus group (n = 7; 37%) were conducted to explore the perceived impact of the training course on coaching skills, perceptions of coaching, and further program needs during the pilot year of the TRA program. RESULTS/UNASSIGNED:Faculty had strong skills around establishing trust, authentic listening, and supporting goal-setting. They required more practice around guiding self-discovery and following a coachee-led agenda. Faculty found the training course to be helpful for developing coaching skills. Faculty embraced their new roles as coaches and appreciated having a community of practice with other coaches. Suggestions for improvement included more opportunities to practice and receive feedback on skills and additional structures to further support TRA program encounters with coaches. CONCLUSIONS/UNASSIGNED:The faculty development program was feasible and had good acceptance among participants. Faculty were well-suited to serve as coaches and valued the coaching mindset. Adequate skills reinforcement and program structure were identified as needs to facilitate a coaching program in graduate medical education.

PURPOSE/OBJECTIVE:To explore whether a machine-learning algorithm could accurately perform the initial screening of medical school applications. METHOD/METHODS:Using application data and faculty screening outcomes from the 2013 to 2017 application cycles (n = 14,555 applications), the authors created a virtual faculty screener algorithm. A retrospective validation using 2,910 applications from the 2013 to 2017 cycles and a prospective validation using 2,715 applications during the 2018 application cycle were performed. To test the validated algorithm, a randomized trial was performed in the 2019 cycle, with 1,827 eligible applications being reviewed by faculty and 1,873 by algorithm. RESULTS:The retrospective validation yielded area under the receiver operating characteristic (AUROC) values of 0.83, 0.64, and 0.83 and area under the precision-recall curve (AUPRC) values of 0.61, 0.54, and 0.65 for the invite for interview, hold for review, and reject groups, respectively. The prospective validation yielded AUROC values of 0.83, 0.62, and 0.82 and AUPRC values of 0.66, 0.47, and 0.65 for the invite for interview, hold for review, and reject groups, respectively. The randomized trial found no significant differences in overall interview recommendation rates according to faculty or algorithm and among female or underrepresented in medicine applicants. In underrepresented in medicine applicants, there were no significant differences in the rates at which the admissions committee offered an interview (70 of 71 in the faculty reviewer arm and 61 of 65 in the algorithm arm; P = .14). No difference in the rate of the committee agreeing with the recommended interview was found among female applicants (224 of 229 in the faculty reviewer arm and 220 of 227 in the algorithm arm; P = .55). CONCLUSIONS:The virtual faculty screener algorithm successfully replicated faculty screening of medical school applications and may aid in the consistent and reliable review of medical school applicants.

Medical schools and residency programs are increasingly incorporating personalization of content, pathways, and assessments to align with a competency-based model. Yet, such efforts face challenges involving large amounts of data, sometimes struggling to deliver insights in a timely fashion for trainees, coaches, and programs. In this article, the authors argue that the emerging paradigm of precision medical education (PME) may ameliorate some of these challenges. However, PME lacks a widely accepted definition and a shared model of guiding principles and capacities, limiting widespread adoption. The authors propose defining PME as a systematic approach that integrates longitudinal data and analytics to drive precise educational interventions that address each individual learner's needs and goals in a continuous, timely, and cyclical fashion, ultimately improving meaningful educational, clinical, or system outcomes. Borrowing from precision medicine, they offer an adapted shared framework. In the P4 medical education framework, PME should (1) take a proactive approach to acquiring and using trainee data; (2) generate timely personalized insights through precision analytics (including artificial intelligence and decision-support tools); (3) design precision educational interventions (learning, assessment, coaching, pathways) in a participatory fashion, with trainees at the center as co-producers; and (4) ensure interventions are predictive of meaningful educational, professional, or clinical outcomes. Implementing PME will require new foundational capacities: flexible educational pathways and programs responsive to PME-guided dynamic and competency-based progression; comprehensive longitudinal data on trainees linked to educational and clinical outcomes; shared development of requisite technologies and analytics to effect educational decision-making; and a culture that embraces a precision approach, with research to gather validity evidence for this approach and development efforts targeting new skills needed by learners, coaches, and educational leaders. Anticipating pitfalls in the use of this approach will be important, as will ensuring it deepens, rather than replaces, the interaction of trainees and their coaches.

PROBLEM:Ability to set goals and work with coaches can support individualized, self-directed learning. Understanding the focus and quality of graduating medical student and first-year resident goals and the influence of coaching on goal-setting can inform efforts to support learners through the transition from medical school to residency. APPROACH:This observational study examined goal-setting among graduating medical students and first-year residents from April 2021 to March 2022. The medical students set goals while participating in a Transition to Residency elective. The residents in internal medicine, obstetrics and gynecology, emergency medicine, orthopedics, and pathology set goals through meeting 1:1 with coaches. Raters assessed goals using a 3-point rubric on domains of specific, measurable, attainable, relevant, and timely (i.e., SMART goal framework) and analyzed descriptive statistics, Mann-Whitney U tests, and linear regressions. OUTCOMES:Among 48 medical students, 30 (62.5%) set 108 goals for early residency. Among 134 residents, 62 (46.3%) entered goals. Residents met with coaches 2.8 times on average (range 0-8 meetings, median = 3). Goal quality was higher in residents than medical students (average score for S: 2.71 vs 2.06, P < .001; M: 2.38 vs 1.66, P < .001; A: 2.92 vs 2.64, P < .001; R: 2.94 vs 2.86, P = .002; T: 1.71 vs 1.31, P < .001). The number of coaching meetings was associated with more specific, measurable goals (specific: F [1, 1.02] = 6.56, P = .01, R2 = .10; measurable: F [1, 1.49] = 4.74, P = .03, R2 = .07). NEXT STEPS:Learners set realistic, attainable goals through the transition to residency, but the goals could be more specific, measurable, and timely. The residents set SMARTer goals, with coaching improving goal quality. Understanding how best to scaffold coaching and support goal-setting through this transition may improve trainees' self-directed learning and well-being.

The explosion of medical information demands a thorough reconsideration of medical education, including what we teach and assess, how we educate, and whom we educate. Physicians of the future will need to be self-aware, self-directed, resource-effective team players who can synthesize and apply summarized information and communicate clearly. Training in metacognition, data science, informatics, and artificial intelligence is needed. Education programs must shift focus from content delivery to providing students explicit scaffolding for future learning, such as the Master Adaptive Learner model. Additionally, educators should leverage informatics to improve the process of education and foster individualized, precision education. Finally, attributes of the successful physician of the future should inform adjustments in recruitment and admissions processes. This paper explores how member schools of the American Medical Association Accelerating Change in Medical Education Consortium adjusted all aspects of educational programming in acknowledgment of the rapid expansion of information.

Background/UNASSIGNED:The American Medical Association Accelerating Change in Medical Education (AMA-ACE) consortium proposes that medical schools include a new 3-pillar model incorporating health systems science (HSS) and basic and clinical sciences. One of the goals of AMA-ACE was to support HSS curricular innovation to improve residency preparation. Objective/UNASSIGNED:This study evaluates the effectiveness of HSS curricula by using a large dataset to link medical school graduates to internship Milestones through collaboration with the Accreditation Council for Graduate Medical Education (ACGME). Methods/UNASSIGNED:ACGME subcompetencies related to the schools' HSS curricula were identified for internal medicine, emergency medicine, family medicine, obstetrics and gynecology (OB/GYN), pediatrics, and surgery. Analysis compared Milestone ratings of ACE school graduates to non-ACE graduates at 6 and 12 months using generalized estimating equation models. Results/UNASSIGNED:At 6 months both groups demonstrated similar HSS-related levels of Milestone performance on the selected ACGME competencies. At 1 year, ACE graduates in OB/GYN scored minimally higher on 2 systems-based practice (SBP) subcompetencies compared to non-ACE school graduates: SBP01 (1.96 vs 1.82, 95% CI 0.03-0.24) and SBP02 (1.87 vs 1.79, 95% CI 0.01-0.16). In internal medicine, ACE graduates scored minimally higher on 3 HSS-related subcompetencies: SBP01 (2.19 vs 2.05, 95% CI 0.04-0.26), PBLI01 (2.13 vs 2.01; 95% CI 0.01-0.24), and PBLI04 (2.05 vs 1.93; 95% CI 0.03-0.21). For the other specialties examined, there were no significant differences between groups. Conclusions/UNASSIGNED:Graduates from schools with training in HSS had similar Milestone ratings for most subcompetencies and very small differences in Milestone ratings for only 5 subcompetencies across 6 specialties at 1 year, compared to graduates from non-ACE schools. These differences are likely not educationally meaningful.

The acceptance of students to a medical school places a considerable emphasis on performance in standardized tests and undergraduate grade point average (uGPA). Traditionally, applicants may be judged as a homogeneous population according to simple quantitative thresholds that implicitly assume a linear relationship between scores and academic success. This 'one-size-fits-all' approach ignores the notion that individuals may show distinct patterns of achievement and follow diverse paths to success. In this study, we examined a dataset composed of 53 variables extracted from the admissions application records of 1,088 students matriculating to NYU School of Medicine between the years 2006-2014. We defined training and test groups and applied K-means clustering to search for distinct groups of applicants. Building an optimized logistic regression model, we then tested the predictive value of this clustering for estimating the success of applicants in medical school, aggregating eight performance measures during the subsequent medical school training as a success factor. We found evidence for four distinct clusters of students-we termed 'signatures'-which differ most substantially according to the absolute level of the applicant's uGPA and its trajectory over the course of undergraduate education. The 'risers' signature showed a relatively higher uGPA and also steeper trajectory; the other signatures showed each remaining combination of these two main factors: 'improvers' relatively lower uGPA, steeper trajectory; 'solids' higher uGPA, flatter trajectory; 'statics' both lower uGPA and flatter trajectory. Examining the success index across signatures, we found that the risers and the statics have significantly higher and lower likelihood of quantifiable success in medical school, respectively. We also found that each signature has a unique set of features that correlate with its success in medical school. The big data approach presented here can more sensitively uncover success potential since it takes into account the inherent heterogeneity within the student population.

Learning curves can support a competency-based approach to assessment for learning. When interpreting repeated assessment data displayed as learning curves, a key assessment question is: "How well is each learner learning?" We outline the validity argument and investigation relevant to this question, for a computer-based repeated assessment of competence in electrocardiogram (ECG) interpretation. We developed an on-line ECG learning program based on 292 anonymized ECGs collected from an electronic patient database. After diagnosing each ECG, participants received feedback including the computer interpretation, cardiologist's annotation, and correct diagnosis. In 2015, participants from a single institution, across a range of ECG skill levels, diagnosed at least 60 ECGs. We planned, collected and evaluated validity evidence under each inference of Kane's validity framework. For Scoring, three cardiologists' kappa for agreement on correct diagnosis was 0.92. There was a range of ECG difficulty across and within each diagnostic category. For Generalization, appropriate sampling was reflected in the inclusion of a typical clinical base rate of 39% normal ECGs. Applying generalizability theory presented unique challenges. Under the Extrapolation inference, group learning curves demonstrated expert-novice differences, performance increased with practice and the incremental phase of the learning curve reflected ongoing, effortful learning. A minority of learners had atypical learning curves. We did not collect Implications evidence. Our results support a preliminary validity argument for a learning curve assessment approach for repeated ECG interpretation with deliberate and mixed practice. This approach holds promise for providing educators and researchers, in collaboration with their learners, with deeper insights into how well each learner is learning.