Try a new search

Format these results:

Searched for:



Total Results:


The Research on Medical Education Outcomes (ROMEO) Registry: Addressing Ethical and Practical Challenges of Using "Bigger," Longitudinal Educational Data

Gillespie, Colleen; Zabar, Sondra; Altshuler, Lisa; Fox, Jaclyn; Pusic, Martin; Xu, Junchuan; Kalet, Adina
PROBLEM: Efforts to evaluate and optimize the effectiveness of medical education have been limited by the difficulty of designing medical education research. Longitudinal, epidemiological views of educational outcomes can help overcome limitations, but these approaches require "bigger data"-more learners, sources, and time points. The rich data institutions collect on students and residents can be mined, however, ethical and practical barriers to using these data must first be overcome. APPROACH: In 2008, the authors established the Research on Medical Education Outcomes (ROMEO) Registry, an educational data registry modeled after patient registries. New York University School of Medicine students, residents, and fellows provide consent for routinely collected educational, performance, quality improvement, and clinical practice data to be compiled into a deidentified, longitudinal database. As of January 2015, this registry included 1,225 residents and fellows across 12 programs (71% consent rate) and 841 medical students (86% consent rate). Procedures ensuring voluntary informed consent are essential to ethical enrollment and data use. Substantial resources are required to provide access to and manage the data. OUTCOMES: The registry supports educational scholarship. Seventy-two studies using registry data have been presented or published. These focus on evaluating the curriculum, quality of care, and measurement quality and on assessing needs, competencies, skills development, transfer of skills to practice, remediation patterns, and links between education and patient outcomes. NEXT STEPS: The authors are working to integrate assessment of relevant outcomes into the curriculum, maximize both the quantity and quality of the data, and expand the registry across institutions.
PMID: 26466377
ISSN: 1938-808x
CID: 1803682

Charting a Key Competency Domain: Understanding Resident Physician Interprofessional Collaboration (IPC) Skills

Zabar, Sondra; Adams, Jennifer; Kurland, Sienna; Shaker-Brown, Amara; Porter, Barbara; Horlick, Margaret; Hanley, Kathleen; Altshuler, Lisa; Kalet, Adina; Gillespie, Colleen
BACKGROUND: Interprofessional collaboration (IPC) is essential for quality care. Understanding residents' level of competence is a critical first step to designing targeted curricula and workplace learning activities. In this needs assessment, we measured residents' IPC competence using specifically designed Objective Structured Clinical Exam (OSCE) cases and surveyed residents regarding training needs. METHODS: We developed three cases to capture IPC competence in the context of physician-nurse collaboration. A trained actor played the role of the nurse (Standardized Nurse - SN). The Interprofessional Education Collaborative (IPEC) framework was used to create a ten-item behaviorally anchored IPC performance checklist (scored on a three-point scale: done, partially done, well done) measuring four generic domains: values/ethics; roles/responsibilities; interprofessional communication; and teamwork. Specific skills required for each scenario were also assessed, including teamwork communication (SBAR and CUS) and patient-care-focused tasks. In addition to evaluating IPC skills, the SN assessed communication, history-taking and physical exam skills. IPC scores were computed as percent of items rated well done in each domain (Cronbach's alpha > 0.77). Analyses include item frequencies, comparison of mean domain scores, correlation between IPC and other skills, and content analysis of SN comments and resident training needs. RESULTS: One hundred and seventy-eight residents (of 199 total) completed an IPC case and results are reported for the 162 who participated in our medical education research registry. IPC domain scores were: Roles/responsibilities mean = 37 % well done (SD 37 %); Values/ethics mean = 49 % (SD 40 %); Interprofessional communication mean = 27 % (SD 36 %); Teamwork mean = 47 % (SD 29 %). IPC was not significantly correlated with other core clinical skills. SNs' comments focused on respect and IPC as a distinct skill set. Residents described needs for greater clarification of roles and more workplace-based opportunities structured to support interprofessional education/learning. CONCLUSIONS: The IPC cases and competence checklist are a practical method for conducting needs assessments and evaluating IPC training/curriculum that provides rich and actionable data at both the individual and program levels.
PMID: 27121308
ISSN: 1525-1497
CID: 2092562

Scaffolding the Transition to Residency: A Qualitative Study of Coach and Resident Perspectives

Park, Agnes; Gillespie, Colleen; Triola, Marc; Buckvar-Keltz, Lynn; Greene, Richard E; Winkel, Abigail Ford
PURPOSE/OBJECTIVE:This study explores coaching during transition from medical school to residency through the perspectives of residents and faculty coaches participating in a coaching program from residency match through the first year of residency. METHOD/METHODS:From January to September 2020, 15 faculty coaches in internal medicine, obstetrics and gynecology, emergency medicine, orthopedics, and pathology participated in a synchronous, in-person coaching training course. All 94 postgraduate year 1 residents in these 5 training programs participated. Between November 2021 and March 2022, focus groups were held with interns from all residency programs participating in the program. Interviews were conducted with faculty coaches in February 2022. Faculty and residents discussed their experiences with and perceptions of coaching. De-identified transcripts were coded, and researchers organized these codes into broader categories, generated cross-cutting themes from the concepts described in both cohorts, and proposed a model for the potential of coaching to support the transition to residency. Descriptive themes were constructed and analytic themes developed by identifying concepts that crossed the data sets. RESULTS:Seven focus groups were held with 39 residents (42%). Residents discussed the goals of a coaching program, coach attributes, program factors, resident attributes, and the role of the coach. Coaches focused on productivity of coaching, coaching skills and approach, professional development, and scaffolding the coaching experience. Three analytic themes were created: (1) coaching as creating an explicit curriculum for growth through the transition to residency, (2) factors contributing to successful coaching, and (3) ways in which these factors confront graduate medical education norms. CONCLUSIONS:Learner and faculty perspectives on coaching through the transition to residency reveal the potential for coaching to make an explicit and modifiable curriculum for professional growth and development. Creating structures for coaching in graduate medical education may allow for individualized professional development, improved mindset, self-awareness, and self-directed learning.
PMID: 37683265
ISSN: 1938-808x
CID: 5628062

Bridging the Gap from Student to Doctor: Developing Coaches for the Transition to Residency

Winkel, Abigail Ford; Gillespie, Colleen; Park, Agnes; Branzetti, Jeremy; Cocks, Patrick; Greene, Richard E; Zabar, Sondra; Triola, Marc
BACKGROUND/UNASSIGNED:A lack of educational continuity creates disorienting friction at the onset of residency. Few programs have harnessed the benefits of coaching, which can facilitate self-directed learning, competency development, and professional identity formation, to help ease this transition. OBJECTIVE/UNASSIGNED:To describe the process of training faculty Bridge Coaches for the Transition to Residency Advantage (TRA) program for interns. METHODS/UNASSIGNED:Nineteen graduate faculty educators participated in a coaching training course with formative skills assessment as part of a faculty development program starting in January 2020. Surveys (n = 15; 79%) and a focus group (n = 7; 37%) were conducted to explore the perceived impact of the training course on coaching skills, perceptions of coaching, and further program needs during the pilot year of the TRA program. RESULTS/UNASSIGNED:Faculty had strong skills around establishing trust, authentic listening, and supporting goal-setting. They required more practice around guiding self-discovery and following a coachee-led agenda. Faculty found the training course to be helpful for developing coaching skills. Faculty embraced their new roles as coaches and appreciated having a community of practice with other coaches. Suggestions for improvement included more opportunities to practice and receive feedback on skills and additional structures to further support TRA program encounters with coaches. CONCLUSIONS/UNASSIGNED:The faculty development program was feasible and had good acceptance among participants. Faculty were well-suited to serve as coaches and valued the coaching mindset. Adequate skills reinforcement and program structure were identified as needs to facilitate a coaching program in graduate medical education.
PMID: 36351566
ISSN: 1087-2981
CID: 5357372

Using learning analytics in clinical competency committees: Increasing the impact of competency-based medical education

Carney, Patricia A; Sebok-Syer, Stefanie S; Pusic, Martin V; Gillespie, Colleen C; Westervelt, Marjorie; Goldhamer, Mary Ellen J
Graduate medical education (GME) and Clinical Competency Committees (CCC) have been evolving to monitor trainee progression using competency-based medical education principles and outcomes, though evidence suggests CCCs fall short of this goal. Challenges include that evaluation data are often incomplete, insufficient, poorly aligned with performance, conflicting or of unknown quality, and CCCs struggle to organize, analyze, visualize, and integrate data elements across sources, collection methods, contexts, and time-periods, which makes advancement decisions difficult. Learning analytics have significant potential to improve competence committee decision making, yet their use is not yet commonplace. Learning analytics (LA) is the interpretation of multiple data sources gathered on trainees to assess academic progress, predict future performance, and identify potential issues to be addressed with feedback and individualized learning plans. What distinguishes LA from other educational approaches is systematic data collection and advanced digital interpretation and visualization to inform educational systems. These data are necessary to: 1) fully understand educational contexts and guide improvements; 2) advance proficiency among stakeholders to make ethical and accurate summative decisions; and 3) clearly communicate methods, findings, and actionable recommendations for a range of educational stakeholders. The ACGME released the third edition CCC Guidebook for Programs in 2020 and the 2021 Milestones 2.0 supplement of the Journal of Graduate Medical Education (JGME Supplement) presented important papers that describe evaluation and implementation features of effective CCCs. Principles of LA underpin national GME outcomes data and training across specialties; however, little guidance currently exists on how GME programs can use LA to improve the CCC process. Here we outline recommendations for implementing learning analytics for supporting decision making on trainee progress in two areas: 1) Data Quality and Decision Making, and 2) Educator Development.
PMID: 36821373
ISSN: 1087-2981
CID: 5448242

Addressing social determinants of health in primary care: a quasi-experimental study using unannounced standardised patients to evaluate the impact of audit/feedback on physicians' rates of identifying and responding to social needs

Gillespie, Colleen; Wilhite, Jeffrey A; Hanley, Kathleen; Hardowar, Khemraj; Altshuler, Lisa; Fisher, Harriet; Porter, Barbara; Wallach, Andrew; Zabar, Sondra
BACKGROUND:Although efforts are underway to address social determinants of health (SDOH), little is known about physicians' SDOH practices despite evidence that failing to fully elicit and respond to social needs can compromise patient safety and undermine both the quality and effectiveness of treatment. In particular, interventions designed to enhance response to social needs have not been assessed using actual practice behaviour. In this study, we evaluate the degree to which providing primary care physicians with feedback on their SDOH practice behaviours is associated with increased rates of eliciting and responding to housing and social isolation needs. METHODS:Unannounced standardised patients (USPs), actors trained to consistently portray clinical scenarios, were sent, incognito, to all five primary care teams in an urban, safety-net healthcare system. Scenarios involved common primary care conditions and each included an underlying housing (eg, mould in the apartment, crowding) and social isolation issue and USPs assessed whether the physician fully elicited these needs and if so, whether or not they addressed them. The intervention consisted of providing physicians with audit/feedback reports of their SDOH practices, along with brief written educational material. A prepost comparison group design was used to evaluate the intervention; four teams received the intervention and one team served as a 'proxy' comparison (no intervention). Preintervention (February 2017 to December 2017) rates of screening for and response to the scripted housing and social needs were compared with intervention period (January 2018 to March 2019) rates for both intervention and comparison teams. RESULTS:108 visits were completed preintervention and 183 during the intervention period. Overall, social needs were not elicited half of the time and fully addressed even less frequently. Rates of identifying the housing issue increased for teams that received audit/feedback reports (46%-60%; p=0.045) and declined for the proxy comparison (61%-42%; p=0.174). Rates of responding to housing needs increased significantly for intervention teams (15%-41%; p=0.004) but not for the comparison team (21%-29%; p=0.663). Social isolation was identified more frequently postintervention (53%) compared with baseline (39%; p=0.041) among the intervention teams but remained unchanged for the comparison team (39% vs 32%; p=0.601). Full exploration of social isolation remained low for both intervention and comparison teams. CONCLUSIONS:Results suggest that physicians may not be consistently screening for or responding to social needs but that receiving feedback on those practices, along with brief targeted education, can improve rates of SDOH screening and response.
PMID: 35623722
ISSN: 2044-5423
CID: 5284022

Artificial Intelligence Screening of Medical School Applications: Development and Validation of a Machine-Learning Algorithm

Triola, Marc M; Reinstein, Ilan; Marin, Marina; Gillespie, Colleen; Abramson, Steven; Grossman, Robert I; Rivera, Rafael
PURPOSE/OBJECTIVE:To explore whether a machine-learning algorithm could accurately perform the initial screening of medical school applications. METHOD/METHODS:Using application data and faculty screening outcomes from the 2013 to 2017 application cycles (n = 14,555 applications), the authors created a virtual faculty screener algorithm. A retrospective validation using 2,910 applications from the 2013 to 2017 cycles and a prospective validation using 2,715 applications during the 2018 application cycle were performed. To test the validated algorithm, a randomized trial was performed in the 2019 cycle, with 1,827 eligible applications being reviewed by faculty and 1,873 by algorithm. RESULTS:The retrospective validation yielded area under the receiver operating characteristic (AUROC) values of 0.83, 0.64, and 0.83 and area under the precision-recall curve (AUPRC) values of 0.61, 0.54, and 0.65 for the invite for interview, hold for review, and reject groups, respectively. The prospective validation yielded AUROC values of 0.83, 0.62, and 0.82 and AUPRC values of 0.66, 0.47, and 0.65 for the invite for interview, hold for review, and reject groups, respectively. The randomized trial found no significant differences in overall interview recommendation rates according to faculty or algorithm and among female or underrepresented in medicine applicants. In underrepresented in medicine applicants, there were no significant differences in the rates at which the admissions committee offered an interview (70 of 71 in the faculty reviewer arm and 61 of 65 in the algorithm arm; P = .14). No difference in the rate of the committee agreeing with the recommended interview was found among female applicants (224 of 229 in the faculty reviewer arm and 220 of 227 in the algorithm arm; P = .55). CONCLUSIONS:The virtual faculty screener algorithm successfully replicated faculty screening of medical school applications and may aid in the consistent and reliable review of medical school applicants.
PMID: 36888969
ISSN: 1938-808x
CID: 5432762

SMARTer Goalsetting: A Pilot Innovation for Coaches During the Transition to Residency

Winkel, Abigail Ford; Chang, Lucy Y; McGlone, Pauline; Gillespie, Colleen; Triola, Marc
PROBLEM:Ability to set goals and work with coaches can support individualized, self-directed learning. Understanding the focus and quality of graduating medical student and first-year resident goals and the influence of coaching on goal-setting can inform efforts to support learners through the transition from medical school to residency. APPROACH:This observational study examined goal-setting among graduating medical students and first-year residents from April 2021 to March 2022. The medical students set goals while participating in a Transition to Residency elective. The residents in internal medicine, obstetrics and gynecology, emergency medicine, orthopedics, and pathology set goals through meeting 1:1 with coaches. Raters assessed goals using a 3-point rubric on domains of specific, measurable, attainable, relevant, and timely (i.e., SMART goal framework) and analyzed descriptive statistics, Mann-Whitney U tests, and linear regressions. OUTCOMES:Among 48 medical students, 30 (62.5%) set 108 goals for early residency. Among 134 residents, 62 (46.3%) entered goals. Residents met with coaches 2.8 times on average (range 0-8 meetings, median = 3). Goal quality was higher in residents than medical students (average score for S: 2.71 vs 2.06, P < .001; M: 2.38 vs 1.66, P < .001; A: 2.92 vs 2.64, P < .001; R: 2.94 vs 2.86, P = .002; T: 1.71 vs 1.31, P < .001). The number of coaching meetings was associated with more specific, measurable goals (specific: F [1, 1.02] = 6.56, P = .01, R2 = .10; measurable: F [1, 1.49] = 4.74, P = .03, R2 = .07). NEXT STEPS:Learners set realistic, attainable goals through the transition to residency, but the goals could be more specific, measurable, and timely. The residents set SMARTer goals, with coaching improving goal quality. Understanding how best to scaffold coaching and support goal-setting through this transition may improve trainees' self-directed learning and well-being.
PMID: 36652456
ISSN: 1938-808x
CID: 5502182

Longitudinal data support university-based biomedical entrepreneurship education programs

Vizgan, Gabriel; Hill-Whilton, Zachary; Achuonjei, Joy; Schweickart, Tucker; Chitale, Sadhana; Gillespie, Colleen; Gold-von Simson, Gabrielle
PMID: 36922690
ISSN: 1546-1696
CID: 5443502

The Biomedical Entrepreneurship Skills Development Program for the Advancement of Research Translation: Foundations of Biomedical Startups course, metrics, and impact

Schweickart, Tucker; Hill-Whilton, Zachary; Chitale, Sadhana; Cobos, Daniel; Gilon-Yanai, Michal; Achuonjei, Joy; Vizgan, Gabriel; Gillespie, Colleen; Gold-Von Simson, Gabrielle
Background/Objective: A growing number of biomedical doctoral graduates are entering the biotechnology and industry workforce, though most lack training in business practice. Entrepreneurs can benefit from venture creation and commercialization training that is largely absent from standard biomedical educational curricula. The NYU Biomedical Entrepreneurship Educational Program (BEEP) seeks to fill this training gap to prepare and motivate biomedical entrepreneurs to develop an entrepreneurial skill set, thus accelerating the pace of innovation in technology and business ventures. Methods: The NYU BEEP Model was developed and implemented with funding from NIDDK and NCATS. The program consists of a core introductory course, topic-based interdisciplinary workshops, venture challenges, on-line modules, and mentorship from experts. Here, we evaluate the efficacy of the core, introductory course, Foundations of Biomedical Startups, through the use of pre/post-course surveys and free-response answers. Results: After 2 years, 153 participants (26% doctoral students, 23% post-doctoral PhDs, 20% faculty, 16% research staff, 15% other) have completed the course. Evaluation data show self-assessed knowledge gain in all domains. The percentage of students rating themselves as either competent or on the way to being an expert in all areas was significantly higher post-course (P < 0.05). In each content area, the percentages of participants rating themselves as very interested increased post-course. 95% of those surveyed reported the course met its objectives, and 95% reported a higher likelihood of pursuing commercialization of discoveries post-course. Conclusion: NYU BEEP can serve as a model to develop similar curricula/programs to enhance entrepreneurial activity of early-stage researchers.
ISSN: 2059-8661
CID: 5446842