Faculty Bibliography

PROBLEM: Efforts to evaluate and optimize the effectiveness of medical education have been limited by the difficulty of designing medical education research. Longitudinal, epidemiological views of educational outcomes can help overcome limitations, but these approaches require "bigger data"-more learners, sources, and time points. The rich data institutions collect on students and residents can be mined, however, ethical and practical barriers to using these data must first be overcome. APPROACH: In 2008, the authors established the Research on Medical Education Outcomes (ROMEO) Registry, an educational data registry modeled after patient registries. New York University School of Medicine students, residents, and fellows provide consent for routinely collected educational, performance, quality improvement, and clinical practice data to be compiled into a deidentified, longitudinal database. As of January 2015, this registry included 1,225 residents and fellows across 12 programs (71% consent rate) and 841 medical students (86% consent rate). Procedures ensuring voluntary informed consent are essential to ethical enrollment and data use. Substantial resources are required to provide access to and manage the data. OUTCOMES: The registry supports educational scholarship. Seventy-two studies using registry data have been presented or published. These focus on evaluating the curriculum, quality of care, and measurement quality and on assessing needs, competencies, skills development, transfer of skills to practice, remediation patterns, and links between education and patient outcomes. NEXT STEPS: The authors are working to integrate assessment of relevant outcomes into the curriculum, maximize both the quantity and quality of the data, and expand the registry across institutions.

INTRODUCTION: People with a mental health diagnosis have high rates of tobacco use and encounter limited availability of tobacco treatment targeted to their needs. This study compared the effectiveness of a specialized telephone smoking-cessation intervention developed for mental health patients with standard state quit-line counseling. DESIGN: RCT. SETTING/PARTICIPANTS: The study was conducted at six Veterans Health Administration facilities in the Northeast U.S. Participants were 577 mental health clinic patients referred by their providers for smoking-cessation treatment. INTERVENTION: From 2010 to 2012, the study implemented a telephone program that included patient referral from a mental health provider, mailed cessation medications, and telephone counseling. Participants were randomized to receive a specialized multisession telephone counseling protocol (n=270) or transfer to their state's quit-line for counseling (n=307). MAIN OUTCOME MEASURES: Participants completed telephone surveys at baseline, 2 months, and 6 months. The study's primary outcome was self-reported 30-day abstinence at 6 months. Secondary outcomes were self-reported 30-day abstinence, counseling satisfaction and counseling content at 2 months, and self-reported use of cessation treatment and quit attempts at 6 months. Logistic regression was used to compare treatment groups on outcomes, controlling for baseline cigarettes per day and site. Inverse probability weighting and multiple imputation were used to handle missing abstinence outcomes. Data were analyzed in 2014-2015. RESULTS: At 6 months, participants in the specialized counseling arm were more likely to report 30-day abstinence (26% vs 18%, OR=1.62, 95% CI=1.24, 2.11). There was no significant group difference in abstinence at 2 months (18% vs 14%, OR=1.31, 95% CI=0.49, 3.49). Participants in the specialized arm were more likely to be assisted with developing a quit plan; receive follow-up calls after quitting; and receive counseling on several domains, including motivation, confidence, smoking triggers, coping with urges, and mental health symptoms (all p<0.05). Specialized counseling participants were more satisfied with treatment and more likely to find the counseling useful (p<0.05). CONCLUSIONS: The specialized counseling intervention was more effective at helping patients quit than transfer to a state quit-line. Patients were more satisfied with the specialized counseling program. TRIAL REGISTRATION: This study is registered at www.clinicaltrials.gov NCT00724308.

OBJECTIVE: The Test of Integrated Professional Skills (TIPS) is an objective structured clinical examination-simulation hybrid examination that assesses resident integration of technical, cognitive, and affective skills in Obstetrics and Gynecology (OBGYN) residents. The aim of this study was to analyze performance patterns and reactions of residents to the test to understand how it may fit within a comprehensive assessment program. DESIGN: A retrospective, mixed methods review of the design and implementation of the examination, patterns of performance of trainees at different levels of training, focus group data, and description of use of TIPS results for resident remediation and curriculum development. SETTING: OBGYN residents at New York University Langone Medical Center, a tertiary-care, urban academic health center. PARTICIPANTS: OBGYN residents in all years of training, postgraduate year-1 through postgraduate year, all residents completing the TIPS examination and consenting to participate in focus groups were included. RESULTS: In all, 24 residents completed the TIPS examination. Performance on the examination varied widely among individuals at each stage of training, and did not follow developmental trends, except for technical skills. Cronbach alpha for both standardized patient and faculty ratings ranged from 0.69 to 0.84, suggesting internal consistency. Focus group results indicated that residents respond to the TIPS examination in complex ways, ranging from anxiety about performance to mixed feelings about how to use the data for their learning. CONCLUSION: TIPS assesses a range of attributes, and can support both formative and summative evaluation. Lack of clear developmental differences and wide variation in performance by learners at the same level of training support the argument for individualized learning plans and competency-based education.

ISSUE/OBJECTIVE:Medical educators and educational researchers continue to improve their processes for managing medical student and program evaluation data using sound ethical principles. This is becoming even more important as curricular innovations are occurring across undergraduate and graduate medical education. Dissemination of findings from this work is critical, and peer-reviewed journals often require an institutional review board (IRB) determination. APPROACH/METHODS:IRB data repositories, originally designed for the longitudinal study of biological specimens, can be applied to medical education research. The benefits of such an approach include obtaining expedited review for multiple related studies within a single IRB application and allowing for more flexibility when conducting complex longitudinal studies involving large datasets from multiple data sources and/or institutions. In this paper, we inform educators and educational researchers on our analysis of the use of the IRB data repository approach to manage ethical considerations as part of best practices for amassing, pooling, and sharing data for educational research, evaluation, and improvement purposes. IMPLICATIONS/CONCLUSIONS:Fostering multi-institutional studies while following sound ethical principles in the study of medical education is needed, and the IRB data repository approach has many benefits, especially for longitudinal assessment of complex multi-site data.