Faculty Bibliography

BACKGROUND: Clinical reasoning (CR) is a core component of medical training, yet residents often receive little feedback on their CR documentation. Here we describe the process of developing a machine learning (ML) algorithm for feedback on CR documentation to increase the frequency and quality of feedback in this domain.
METHOD(S): To create this algorithm, note quality first had to be rated by gold standard human rating. We selected the IDEA Assessment Tool-a note rating instrument across four domains (I=Interpretive summary, D=Differential diagnosis, E=Explanation of reasoning, A=Alternative diagnoses explained) that uses a 3-point Likert scale without descriptive anchors. To develop descriptive anchors we conducted an iterative process reviewing notes from the EHR written by medicine residents and validated the Revised-IDEA Assessment Tool using Messick's framework- content validity, response process, relation to other variables, internal structure, and consequences. Using the Hofstee standard setting method, cutoffs for high quality clinical reasoning for the IDEA and DEA scores were set. We then created a dataset of expertrated notes to create the ML algorithm. First, a natural language processing software was applied to the set of notes that enabled recognition and automatic encoding of clinical information as a diagnosis or disease (D's), a sign or symptom (E or A), or semantic qualifier (e.g. most likely). Input variables to the ML algorithm included counts of D's, E/A's, semantic qualifiers, and proximity of semantic qualifiers to disease/ diagnosis. ML output focused on DEA quality and was binarized to low or high quality CR. Finally, 200 notes were randomly selected for human validation review comparing ML output to human rated DEA score.
RESULT(S): The IDEA and DEA scores ranged from 0-10 and 0-6, respectively. IDEA score of >= 6.5 and a DEA score of >= 3 was deemed high quality. 252 notes were rated to create the dataset and 20% were rated by 3 raters with high intraclass correlation 0.84 (95% CI 0.74-0.90). 120 of these notes comprised the testing set for ML model development. The logistic regression model was the best performing model with an AUC 0.87 and a positive predictive value (PPV) of 0.65. 48 (40%) of the notes were high quality. There was substantial interrater reliability between ML output and human rating on the 200 note validation set with a Cohen's Kappa 0.64.
CONCLUSION(S): We have developed a ML algorithm for feedback on CR documentation that we hypothesize will increase the frequency and quality of feedback in this domain. We have subsequently developed a dashboard that will display the output of the ML model. Next steps will be to provide internal medicine residents' feedback on their CR documentation using this dashboard and assess the impact this has on their documentation quality. LEARNING OBJECTIVE #1: Describe the importance of high quality documentation of clinical reasoning. LEARNING OBJECTIVE #2: Identify machine learning as a novel assessment tool for feedback on clinical reasoning documentation

STATEMENT OF PROBLEM OR QUESTION (ONE SENTENCE): Can user-centered design (UCD) facilitate the development of novel and effective training tools for the virtual ambulatory learning environment LEARNING OBJECTIVES 1: To identify the needs, preferences, and concerns of resident trainees and attending preceptors regarding the current virtual ambulatory care learning environment. LEARNING OBJECTIVES 2: To apply user-centered design (UCD) strategies to the development of effective tools to enhance the virtual learning experience of trainees and preceptors. DESCRIPTION OF PROGRAM/INTERVENTION, INCLUDING ORGANIZATIONAL CONTEXT (E.G. INPATIENT VS. OUTPATIENT, PRACTICE OR COMMUNITY CHARACTERISTICS): The COVID-19 pandemic spurred a rapid transition to virtual learning environments, the design of which may impact learning experiences and competency development for trainees. User-centered design (UCD) offers a framework to iteratively and collaboratively incorporate needs, preferences, and concerns of users (e.g. trainees and preceptors) in the development of acceptable and effective educational tools. This study applied UCD strategies of empathize, define, ideate, prototype, and test among Internal Medicine residents and outpatient attending preceptors to develop innovations for the virtual ambulatory care learning environment. MEASURES OF SUCCESS (DISCUSS QUALITATIVE AND/OR QUANTITATIVEMETRICSWHICHWILL BEUSEDTOEVALUATE PROGRAM/INTERVENTION): Using the UCD framework, we identified: 1) needs, preferences, and concerns of residents and preceptors in current virtual precepting practices (empathize) 2) key problem areas and pain points (define) 3) potential solutions (ideate) 4) specific products to develop (prototype), deploy, and evaluate (test) in practice FINDINGS TO DATE (IT IS NOT SUFFICIENT TO STATE FINDINGS WILL BE DISCUSSED): Qualitative needs-assessment interviews were conducted among 8 residents and 10 preceptors, which identified key areas of learner need: technical and workflow competency; the virtual precepting experience; patient rapport-building and communication; and documentation requirements. Subsequently, a Design Thinking Workshop focusing on virtual precepting was developed, and 3 workshops were conducted with 12 participants (residents and attendings). Using a three-phase interactive sequence of explore, ideate, and create, participants were divided into 2-or 3-person virtual breakout groups and asked to 1) identify a key problem in current virtual precepting, 2) brainstorm possible solutions, and 3) design and present a low-fidelity prototype of one solution. Key problems identified included: management of technical issues, goal setting for precepting sessions, clinic-specific information dissemination practices, and the loss of shared learning space with colleagues. Potential solutions included: a digital shared-learning plan for residents, a real-time virtual clinical bulletin board, an integrated virtual team huddle, and just-in-time digital chalk talks. Two prototypes are being developed for testing in the live precepting environment. KEY LESSONS FOR DISSEMINATION (WHAT CAN OTHERS TAKE AWAY FOR IMPLEMENTATION TO THEIR PRACTICE OR COMMUNITY): User-centered design can be deployed as an effective strategy to engage learners and preceptors in the design and development of educational innovations for the virtual training environment. We recommend collaborating with residents, preceptors, and other stakeholders in the iterative design of virtual learning tools

Background While there is overwhelming evidence for the beneficial role of hydroxychloroquine (HCQ) in SLE, little is known about its economic impact. We estimated annual direct, indirect, and total costs (DC, IC, TC) associated with HCQ use. Methods A subset of patients from the Systemic Lupus Erythematosus International Collaborating Clinics (SLICC) inception cohort were assessed annually between 2014 and 2019 for health resource use, lost work-force/non-work-force productivity and concurrent HCQ use. Resource use was costed using 2021 Canadian prices and lost productivity using Statistics Canada age-and-sex specific wages. At each assessment, HCQ dose over the past year and weight were documented and patients were stratified into 1 of 3 HCQ dosage groups: nonusers (0 mg/kg/day), low-intensity users (<= 5 mg/kg/day), or high-intensity users (>5 mg/kg/day). Costs associated with HCQ dose were calculated by averaging all observations within each dosage group. Multiple random effects linear regressions adjusted for the possible confounding of age at diagnosis, sex, race/ethnicity, disease duration, geographic region, education, alcohol use, and smoking on the association between annual DC and IC and HCQ dose. A possible mediating effect of disease damage (SLICC/ACR DI) on these associations was also investigated. Results 661 patients (89.4% female, 59.3% non-Caucasian race/ethnicity, mean age and mean disease duration at the start of economic assessments was 42.1 years and 9.5 years, respectively) were followed over a mean of 2.8 years. Across 1536 annual assessments, 36.1% of observations were provided by HCQ non-users, 43.1% by low-intensity users (mean dosage 3.4 mg/kg/day), and 20.8% by high-intensity users (mean dosage 5.9 mg/kg/day). Annual adjusted DC were higher in nonusers ($9599) versus low-intensity users ($6344) and highintensity users ($6333) (table 1). When disease damage was included in the regression, there were no significant differences in DC between dosage groups. While unadjusted IC were higher in non-users ($37,610) versus low-intensity users ($32,480) and high-intensity users ($31,418), adjusted IC did not differ. Adjusted TC were higher in non-users ($46,157) versus low-intensity users ($39,257) and high-intensity users ($37,634). Conclusion SLE patients reported higher adjusted annual DC and TC during periods of HCQ non-use versus periods of use, regardless of the intensity of use. There was no additional cost savings in those using high intensity dosages. The cost-savings effect of HCQ could potentially be partially mediated through reduced damage. In addition to its well-established therapeutic potential, there may be an economic imperative for HCQ use in SLE patients

LEARNING OBJECTIVES 1: Characterize internal medicine resident's inpatient clinical exposure across public, private, federal and community hospital settings. LEARNING OBJECTIVES 2: Identify disparities in resident's clinical exposure across sites to guide curriculum development and rational rotation scheduling. SETTING AND PARTICIPANTS: Experiential learning through patient care is the primary means by which Internal Medicine (IM) residents mature. Despite this, there is an unmet need to characterize how residents' diverse patient care activities inform their educational experience. This is especially true at NYU's IM Residency, which comprises over 200 residents across four distinct hospital systems, exposing residents to diverse, and often variable clinical content. We have previously described a 'crosswalk tool' which maps ICD10 diagnosis codes to one of 16 American Board of Internal Medicine (ABIM) medical content areas and one of 178 specific condition categories, to better characterize clinical exposure. Here, we translate resident-attributed principal ICD-10 discharge diagnosis codes from each of our Program's four training hospitals in Quarter 1 of AY2020 to profile the educational experience of residents at each site. DESCRIPTION: From July 1-Sept 30 2020, we mined principal ICD10 discharge diagnosis codes from resident teams at Bellevue Hospital (BH), a large public hospital; NYU Langone Hospital-Brooklyn (NYU-BK), an academic community hospital; NYU Lang one Hospitals-Manhattan (NYUMN), a large quaternary hospital; and VA NY Harbor Healthcare-Manhattan (VA), a Veteran's Affairs Hospital. We then applied diagnosis codes to the crosswalk tool to translate ICD10 codes into broad ABIM content areas and specific condition categories, yielding site-specific clinical content maps. EVALUATION: At each site there was notable enrichment in two specific content areas-Infectious Disease (ID) and Cardiovascular Diseases (CVD). However, there were striking differences in the frequency of these content areas across sites. Roughly 28% of all diagnoses fell under ID at NYU-BK and NYU-MN, with half that frequency at BH and VA. CVD diagnoses represented 40% of diagnoses at VA, while only 20% at NYU-BK, 25% at NYU-MN, and 30% at BH. For reference, CVD represents 14% and ID 9% of content on theABIM Certification Exam. There were uniformly low frequencies (<1%) of several less typical ABIM content areas, namely Optho, Derm, Allergy/ Immuno, OB/GYN, and ENT/Dental Med. The frequency of Psychiatry diagnoses, which houses substance use, was markedly higher at BH and NYU-BK than the other sites. There were several substantial differences in condition categories across sites, most notably within 'bacteremia and sepsis syndromes' and 'ischemic heart disease.' DISCUSSION / REFLECTION / LESSONS LEARNED: In this pilot we translate discharge data from four distinct hospital systems into an educational framework to characterize our resident's educational experience and in doing so unmask disparities in exposure that could drive rational changes in rotation schedules and didactic content selection

STATEMENT OF PROBLEM OR QUESTION (ONE SENTENCE): In order to describe quality improvement (QI) methods for health systems, we report on 10-years of using Unannounced Standardized Patient (USP) visits as the core of a program of education, training, and improvement in a system serving vulnerable patients in partnership with an academic medical center. LEARNING OBJECTIVES 1: Consider methods for supporting learning healthcare systems LEARNING OBJECTIVES 2: Identify performance data to improve care DESCRIPTION OF PROGRAM/INTERVENTION, INCLUDING ORGANIZATIONAL CONTEXT (E.G. INPATIENT VS. OUTPATIENT, PRACTICE OR COMMUNITY CHARACTERISTICS): The IOM defines a Learning Healthcare System (LHCS) as one in which science, informatics, incentives and culture are aligned for continuous improvement and innovation and where best practices are seamlessly embedded in the delivery process and new knowledge is captured as an integral by-product of the delivery experience. As essential as electronic health records are to LHCS, such data fail to capture all actionable information needed to sustain learning within complex systems. USPs are trained actors who present to clinics, incognito, to portray standardized chief complaints, histories, and characteristics. We designed and delivered USP visits to two urban, safety net clinics, focusing on assessing physician, team, and clinical micro system functioning. MEASURES OF SUCCESS (DISCUSS QUALITATIVE AND/OR QUANTITATIVEMETRICSWHICHWILL BE USEDTOEVALUATE PROGRAM/INTERVENTION): Behaviorally anchored assessments are used to assess core clinical skills (e.g., communication, information gathering, patient education, adherence to guidelines, patient centeredness, and patient activation). Team functioning assessments include professionalism and coordination. Micro system assessment focuses on safety issues like identity confirmation, hand washing, and navigation. Data from these visits has been provided to the residency, primary care teams, and to leadership and have been used to drive education, team training, and QI. FINDINGS TO DATE (IT IS NOT SUFFICIENT TO STATE FINDINGS WILL BE DISCUSSED): 1111 visits have been sent to internal medicine and primary care residents and their teams/clinics. At the resident level, needs for additional education and training in depression management, opioid prescribing, smoking cessation, and patient activation were identified and informed education. Chart reviews found substantial variation in ordering of labs and tests. At the team level, USPs uncovered needs for staff training, enhanced communication, and better processes for eliciting and documenting Social Determinants of Health (SDoH). Audit/feedback reports on provider responses to embedded SDoH combined with targeted education/resources, were associated with increased rates of eliciting and effectively responding to SDoH. In the early COVID wave, USPs tested clinic response to a potentially infectious patient. Currently, USPs are being deployed to understand variability in patients' experience of telemedicine given the rapid transformation to this modality. Finally, generalizable questions about underlying principles of medical education and quality improvement are being asked & answered using USP data to foster deeper understanding of levers for change. KEY LESSONS FOR DISSEMINATION (WHAT CAN OTHERS TAKE AWAY FOR IMPLEMENTATION TO THEIR PRACTICE OR COMMUNITY): A comprehensive USP program can provide unique insights for driving QI and innovation and help sustain a LHCS