Faculty Bibliography

BACKGROUND: At NYU, as part of a comprehensive objective structured clinical skills exam, experienced medical educators judge clinical knowledge, decision-making, and clinical reasoning skills of trainees based on their patient notes. Despite being rubric-driven, this task requires tremendous time and effort to establish consistent scoring, delaying and limiting individualized feedback. We conducted pilot machine learning text classification studies to establish if accurate automated scoring of clinical notes is possible. METHODS: As a use case, we tested 100 student written clinical notes from7 standardized patient cases (Vision Loss, Tel Diarrhea, Difficulty Sleeping, Shoulder Pain, Failure To Thrive, Abdominal, Pain, Palpitations) that had been scored for quality of clinical reasoning by faculty on a 1-4 scale. In order to assess performance of NLP strategies to categorize students in meaningful groups we dichotomized students based on their faculty given scores by case into "failing" (score of 1, 5-18 students per case) and "passing" (score 2,3,4). We treated each task as a binary classification task in a text classification pipeline. First, we treated each note as a bag of tokens and weight each token with term frequency-inverse document frequency (TFIDF) a numerical statistic that reflects howimportant aword is to a document. We then applied 3 different classification algorithms (random forests, support vector machines, and Bayesian logistic regression) and measured discriminatory performance using Area Under Curve (AUC) in a cross validation evaluation design. RESULTS: TFDIF performed with AUCs between 0.669 and 0.905. Logistic regression provided the highestAUC in four cases: Difficulty Sleeping (0.905), Shoulder Pain (0.618), Failure To Thrive (0.717) and Abdominal Pain (0.892). As we observed the highest AUCs in Difficulty Sleeping and Abdominal Pain cases, we have begun to refine the algorithm for these two cases by identifying the importance features that lead faculty to give students to a higher grade and improve the accuracy of NLP based scoring. Promising features include the presence and sequence of certainwords in the problem representation, sentence length in the management section, ranking of the differential diagnosis, sequence between key words (e.g. rule out appendicitis), and evidence of "thinkingness" or what many call semantic qualifiers. CONCLUSIONS: With additional effort to build targeted case specific classifiers for clinical content and reasoning, a validated machine-learning model may achieve partial or full automation of grading of the notes. This work, which builds on decades of clinical decision-making and critical reasoning research, may provide medical trainees with more and potentially better feedback; facilitating learning of clinical reasoning, freeing faculty to coach this process, and in the long run impacting healthcare quality and patient safety

BACKGROUND: Training residents to effectively practice value-based care is challenging. We hypothesized that residents with better communication skills would order fewer unnecessary tests and prescribe more appropriate care. We used a USP case of a patient with uncontrolled asthma to examine the relationship between value-based care and communication skills. METHODS: A 25 year-old female USP presented as a new patient to a medicine resident's clinic, reporting asthma since childhood with worsening symptoms over the past few months. At the time of the visit, she was using her albuterol inhaler multiple times daily, without any additional asthma treatment, and was unsure whether she was using it properly. Data was collected using two forms of assessment: a post-visit USP checklist and a systematic review of the corresponding clinic note to examine treatment recommendations including referrals and quality of documentation. The USP checklist measured communication, patient education, and assessment skills. Each response option included descriptive behavioral anchors and was rated as not done, partly done, or well done. Domain scores were calculated as percent items rated well done. RESULTS: 141 USP visits were made from 2009 to 2016 with a mean visit length = 88 min, SD= 28 min (range: 40 to 180 min). Almost all residents (92%) evaluated the patient's asthma with a pulmonary examination. The most common treatment prescribed was albuterol and an inhaled steroid, with or without a spacer (79%). The majority of residents (53%) did not order any additional studies; 21% ordered one study, and 26% ordered two or more studies. Study orders fell into one of three categories: gold (appropriate/recommended: PFTs, flu shot, HIV), grey (pulmonary consult, HCG), or inappropriate (TSH, A1C). Across the 141 visits, 129 studies were ordered; 46% were gold, 5% were grey, and 49% were inappropriate. The most common study ordered was a PFT (31%). 87% of single study orders were gold, but 92% of multiple orders included at least one inappropriate study. Residents who did not order any studies had significantly higher patient education and counseling skills than residents who ordered one or more studies (54% vs 34%, p = 0.00) and were more likely to explain how to correctly use an inhaler than residents who ordered one or more studies (48% vs 27%, p = 0.01). These residents also had significantly higher management and treatment skills (61% vs 39%, p = 0.00) and overall communication skills (68% vs 55%, p = 0.01). There were no significant differences between groups in medications prescribed or in quality of documentation. CONCLUSIONS: Effective communication skills may contribute to valuebased care through appropriate patient education and ordering of fewer inappropriate studies. Rigorous curricula and assessment of resident's patient education skills should be in place to help both patients and health care system achieve value-based care

BACKGROUND: All physicians need to be skilled at diagnosing, treating, and managing depression. We designed an unannounced standard patient (USP) case to assess residents' clinical skills in addressing depression and explored how those skills are associated with residents' general clinical skills in order to design targeted curriculum on depression. METHODS: The USP was a 26 y.o. male presenting as a new patient to a clinic complaining of fatigue and problems sleeping. Goals of the case were to diagnose a common presentation of depression and make a treatment/follow-up plan. The USP was trained to have a positive PHQ 2 &PHQ 9, family history of depression, and be willing to engage in medication and/or therapy if offered. A post-visit checklist was used by the SPs to assess communication, patient education, and assessment skills using behaviorally anchored items rated as not done, partly done, or well done. A systematic chart review was conducted to examine treatment, quality of documentation, and referrals. Case fidelity was checked by audiotape and confirmed by PHQ 9 score in the EHR. Evidence based treatment was defined as prescribing an SSRI and/or providing a psychiatric referral; if neither of those, scheduling follow-up for within 2 weeks. RESULTS: 122 residents saw the USP case from 2009-2015. Mean visit length = 45 min, SD 25 (14 to 183 min). The patient was screened for depression with a PHQ 2 in 93% of visits; 82% also had a PHQ 9. Overall, 77 residents (63%) provided appropriate treatment: 8% prescribed an SRRI, 23% provided a referral, 19% did both, 7% prescribed a sleep aid and <2 week follow-up, and 43% provided a combination of these treatments. 45 residents (37%) did not provide appropriate treatment: 27 (60%) prescribed a sleep aid and follow-up >2 weeks and 18 (40%) provided no treatment/referral and follow-up >2 weeks. There were no differences in exploration of medical history or substance use, but 83% of residents who treated appropriately had a PHQ 9 compared to 62% of residents who did not treat appropriately. 71% also included depression on the problem list compared to 13%of residents who did not treat appropriately. Residents who treated appropriately had significantly better clinical skills assessed by the USP including: overall communication (71% vs. 54%, p = 0.00), information gathering (72% vs. 55%, p = 0.01), relationship development (75% vs. 60%, p = 0.03), patient education (55% vs. 21%, p = 0.00), and patient activation skills (33% vs. 13%, p = 0.01). CONCLUSIONS: Although almost all residents obtained the relevant information, only about 50% of residents diagnosed depression. PHQ 9 appears to be associated with providing more effective treatment, supporting the importance of health system screening protocols. Residents' communication and depression-specific patient education and activation skills seem to be related to how they identify and manage depression, suggesting that interventions to build these skills may lead to higher quality care

NEEDS AND OBJECTIVES: To address rising education costs, physician shortages, and the need for educational reform, several medical schools have developed accelerated 3-year MD programs. In 2013, NYU School of Medicine began its new 3-year MD program with guaranteed acceptance into residency upon graduation. Using the AAMC's 13 Core Entrustable Professional Activities for Entry into Residency (CEPAER) framework, we designed an immersive 4-hour simulated "Night on Call" (NOC) experience to compare performance of our first graduating cohort of fifteen 3-year MD students (3A), with third (3T) and fourth year (4T) students in the traditional 4-year MD program. SETTING AND PARTICIPANTS: 73 medical students (39 women, age 26.5 (+2.6) years; 36 '3T', 12 '3A', 25 '4T') completed an IRB-approvedNOC at our simulation center 4 weeks prior to the end of their third or final year of medical school. DESCRIPTION: We developed NOC to measure competence and entrustment across all 13 CEPAERs from the perspective of patients, nurses, and attendings. During the simulation, a medical student rotated through a series of 8 clinical coverage scenarios including: 4 standardized patient (SP) cases with varying degrees of complexity, each of which require answering a call from a standardized nurse (SN), evaluating an SP with the SN in the room, making immediate management decisions and writing a coverage note; a phone call to an experienced clinician to orally present (OP) the case; formulation of a clinical question and finding the most appropriate evidence-based medicine (EBM) answer using digital library resources; a clinical vignette (CV) to test ability to recognize a pre-entrustable peer; and a handoff (HO) of 4 cases to a peer (a senior medical student). CEPAERs assessments based on validated tools included communication, physical exam, patient education and interprofessional teamwork skills assessed by an SP and SN, and clinical reasoning based on notes, OP, EBM, CV, HO. Each rater also provided an entrustment judgment. EVALUATION: Although overall student performance improved across cases and some interesting individual performance patterns emerged, there were no significant differences across the three groups in the core competency and entrustment measures evaluated across various NOC activities. DISCUSSION/REFLECTION/LESSONS LEARNED: The 13 CEPAERs are meant to define what students should be expected to perform (without direct supervision) prior to entering residency. Our results, based on multiple rater perspectives, suggest that our cohort of 3A students is as prepared for residency as their 4T counterparts

NEEDS AND OBJECTIVES: Standardized Patients (SPs) are integral to health care professions (HCPs) training. We must understand this workforce, make effective use of SPs' skills, and ensure they accurately portray cases and rate learners. To be authentic, simulation should reflect the demographics of the population served, while providing exposure to less commonly seen patients. We created an SP database to facilitate our work with SPs; review their demographic characteristics; and align information on SP performance -to better serve our educational mission. SETTING AND PARTICIPANTS: NYSIM (Simulation Center for NYU Langone and the City University of NY) serves hundreds of HCP training programs for learners at all levels. While sharing common resources, many programs independently recruit and train SPs. DESCRIPTION: We fielded a web-based survey for SPs and staff to populate the database. Survey items were iteratively reviewed by staff and SPs to ensure items elicited key information. Questions included basic demographics; SP experience/training; other professional background; and relevant physical findings (eg scars, cardiac findings). SPs also uploaded a headshot and resume. Staff separately input information about SPs' work on cases and programs; information about case portrayals; types of cases for which the SP is best suited, and other relevant information. EVALUATION: To date, we have 232 SP surveys, representing the majority of SPs at NYSIM. Demographics included gender (43% male, 56% female, 1% transgender), age range (x = 34.9 years, range teen to 75+) and selfidentified race (71% Caucasian, 17% African-American, 25.6% Asian/South Asian, 3.5% Middle Eastern, 3% Native American/Pacific Islander and 9% other). 22%are bilingual, with over 20 languages represented. SPs had a broad range of SP experience (x = 2.8 years, S.D. = 1.8, range 0-20 years). Almost all SPs were trained in basic case portrayal, with others being trained in aspects of the physical exam, emotional issues, giving feedback, and high stakes rating. SPs bring other skills to their work, including teaching (75%) or healthcare (12%). Survey information helps educators recruit SPs and identify (re)training needs. SPs who perform high stakes exams or are Unannounced Standardized Patients are shielded from general recruitment in order to maintain their anonymity. DISCUSSION/REFLECTION/LESSONS LEARNED: An SP database is useful for a high volume simulation center. Information in a searchable SP data base allows programs to understand the potential pool and expertise of SPs, and to track learners' exposure to specific SPs (this is relevant as our internal data reveal that SPs with more experience tend rate towards the middle of the scale). The demographic characteristics of our SPs broadly match the profile of our healthcare systems, and tracking the data allows us to maintain a good fit between SPs and our environment

AIM: To assess the feasibility and utility of measuring baseline professional identity formation (PIF) in a theory-based professionalism curriculum for early medical students. METHODS: All 132 entering students completed the professional identity essay (PIE) and the defining issues test (DIT2). Students received score reports with individualized narrative feedback and wrote a structured reflection after a large-group session in which the PIF construct was reviewed. Analysis of PIEs resulted in assignment of a full or transitional PIF stage (1-5). The DIT2 score reflects the proportion of the time students used universal ethical principles to justify a response to 6 moral dilemma cases. Students' reflections were content analyzed. RESULTS: PIF scores were distributed across stage 2/3, stage 3, stage 3/4, and stage 4. No student scores were in stages 1, 2, 4/5, or 5. The mean DIT2 score was 53% (range 9.7?76.5%); the correlation between PIF stage and DIT score was rho = 0.18 (p = 0.03). Students who took an analytic approach to the data and demonstrated both awareness that they are novices and anticipation of continued PIF tended to respond more positively to the feedback. CONCLUSIONS: These PIF scores distributed similarly to novice students in other professions. Developmental-theory based PIF and moral reasoning measures are related. Students reflected on these measures in meaningful ways suggesting utility of measuring PIF scores in medical education.

Becoming a doctor is fundamentally about developing a new, professional identity as a physician, which in and of itself may evoke many emotions. Additionally, medical trainees are increasingly moving from one cultural context to another and are challenged with navigating the resulting shifts in their professional identify. In this Article, the authors aim to address medical professional identity formation from a polyvocal, multidisciplinary, cross-cultural perspective. They delineate the cultural approaches to medical professionalism, reflect on professional identity formation in different cultures and on different theories of identity development, and advocate for a context-specific approach to professional identity formation. In doing so, the authors aim to broaden the developing professional identity formation discourse to include non-Western approaches and notions.

ISSUE/OBJECTIVE:Medical educators and educational researchers continue to improve their processes for managing medical student and program evaluation data using sound ethical principles. This is becoming even more important as curricular innovations are occurring across undergraduate and graduate medical education. Dissemination of findings from this work is critical, and peer-reviewed journals often require an institutional review board (IRB) determination. APPROACH/METHODS:IRB data repositories, originally designed for the longitudinal study of biological specimens, can be applied to medical education research. The benefits of such an approach include obtaining expedited review for multiple related studies within a single IRB application and allowing for more flexibility when conducting complex longitudinal studies involving large datasets from multiple data sources and/or institutions. In this paper, we inform educators and educational researchers on our analysis of the use of the IRB data repository approach to manage ethical considerations as part of best practices for amassing, pooling, and sharing data for educational research, evaluation, and improvement purposes. IMPLICATIONS/CONCLUSIONS:Fostering multi-institutional studies while following sound ethical principles in the study of medical education is needed, and the IRB data repository approach has many benefits, especially for longitudinal assessment of complex multi-site data.