Development of a Clinical Reasoning Documentation Assessment Tool for Resident and Fellow Admission Notes: a Shared Mental Model for Feedback
BACKGROUND:Residents and fellows receive little feedback on their clinical reasoning documentation. Barriers include lack of a shared mental model and variability in the reliability and validity of existing assessment tools. Of the existing tools, the IDEA assessment tool includes a robust assessment of clinical reasoning documentation focusing on four elements (interpretive summary, differential diagnosis, explanation of reasoning for lead and alternative diagnoses) but lacks descriptive anchors threatening its reliability. OBJECTIVE:Our goal was to develop a valid and reliable assessment tool for clinical reasoning documentation building off the IDEA assessment tool. DESIGN, PARTICIPANTS, AND MAIN MEASURES/UNASSIGNED:The Revised-IDEA assessment tool was developed by four clinician educators through iterative review of admission notes written by medicine residents and fellows and subsequently piloted with additional faculty to ensure response process validity. A random sample of 252 notes from July 2014 to June 2017 written by 30 trainees across several chief complaints was rated. Three raters rated 20% of the notes to demonstrate internal structure validity. A quality cut-off score was determined using Hofstee standard setting. KEY RESULTS/RESULTS:The Revised-IDEA assessment tool includes the same four domains as the IDEA assessment tool with more detailed descriptive prompts, new Likert scale anchors, and a score range of 0-10. Intraclass correlation was high for the notes rated by three raters, 0.84 (95% CI 0.74-0.90). Scores â‰¥6 were determined to demonstrate high-quality clinical reasoning documentation. Only 53% of notes (134/252) were high-quality. CONCLUSIONS:The Revised-IDEA assessment tool is reliable and easy to use for feedback on clinical reasoning documentation in resident and fellow admission notes with descriptive anchors that facilitate a shared mental model for feedback.
Development and Validation of a Machine Learning-Based Decision Support Tool for Residency Applicant Screening and Review
PURPOSE:Residency programs face overwhelming numbers of residency applications, limiting holistic review. Artificial intelligence techniques have been proposed to address this challenge but have not been created. Here, a multidisciplinary team sought to develop and validate a machine learning (ML)-based decision support tool (DST) for residency applicant screening and review. METHOD:Categorical applicant data from the 2018, 2019, and 2020 residency application cycles (n = 8,243 applicants) at one large internal medicine residency program were downloaded from the Electronic Residency Application Service and linked to the outcome measure: interview invitation by human reviewers (n = 1,235 invites). An ML model using gradient boosting was designed using training data (80% of applicants) with over 60 applicant features (e.g., demographics, experiences, academic metrics). Model performance was validated on held-out data (20% of applicants). Sensitivity analysis was conducted without United States Medical Licensing Examination (USMLE) scores. An interactive DST incorporating the ML model was designed and deployed that provided applicant- and cohort-level visualizations. RESULTS:The ML model areas under the receiver operating characteristic and precision recall curves were 0.95 and 0.76, respectively; these changed to 0.94 and 0.72, respectively, with removal of USMLE scores. Applicants' medical school information was an important driver of predictions-which had face validity based on the local selection process-but numerous predictors contributed. Program directors used the DST in the 2021 application cycle to select 20 applicants for interview that had been initially screened out during human review. CONCLUSIONS:The authors developed and validated an ML algorithm for predicting residency interview offers from numerous application elements with high performance-even when USMLE scores were removed. Model deployment in a DST highlighted its potential for screening candidates and helped quantify and mitigate biases existing in the selection process. Further work will incorporate unstructured textual data through natural language processing methods.
Hickam's dictum, Occam's razor, and Crabtree's bludgeon: a case of renal failure and a clavicular mass
OBJECTIVES/OBJECTIVE:Our discussant's thoughtful consideration of the patient's case allows for review of three maxims of medicine: Occam's razor (the simplest diagnosis is the most likely to be correct), Hickam's dictum (multiple disease entities are more likely than one), and Crabtree's bludgeon (the tendency to make data fit to an explanation we hold dear). CASE PRESENTATION/METHODS:A 66-year-old woman with a history of hypertension presented to our hospital one day after arrival to the United States from Guinea with chronic daily vomiting, unintentional weight loss and progressive shoulder pain. Her labs are notable for renal failure, nephrotic range proteinuria and normocytic anemia while her shoulder X-ray shows osseous resorption in the lateral right clavicle. Multiple myeloma became the team's working diagnosis; however, a subsequent shoulder biopsy was consistent withÂ follicular thyroid carcinoma. Imaging suggested the patient's renal failure was more likely a result of a chronic, unrelated process. CONCLUSIONS:It is tempting to bludgeon diagnostic possibilities into Occam's razor. Presumption that a patient's signs and symptoms are connected by one disease process often puts us at a cognitive advantage. However, atypical presentations, multiple disease processes, and unique populations often lend themselves more to Hickam's dictum than to Occam's razor. Diagnostic aids include performing a metacognitive checklist, engaging analytic thinking, and acknowledging the imperfections of these axioms.
Changing hats: Lessons learned integrating coaching into UME and GME [Meeting Abstract]
BACKGROUND: The transition from medical school to residency is characterized by an abrupt transition of learning needs and goals. Coaching is a promising intervention to support individual learning and growth trajectories of learners. It is uncommon for medical school faculty to have undergone training as coaches. We explored our faculty's perceptions and skills after instituting a new coaching program.
METHOD(S): Faculty advisors (N=12) and GME (N=16) participated in a coaching development program and in community of practice meetings where challenging coaching scenarios were shared. GME faculty also participated in a Group Objective Structured Clinical Exam (GOSCE) to practice and receive feedback on their skills. Peer-faculty observers and resident raters used behaviorally grounded checklists to assess faculty performance. We conducted 2 focus groups: 1) UME advisors engaged in longitudinal coaching (n=9) and 2) GME faculty participating in the coaching development program (n=8) to better understand how faculty make sense of and put into practice these new coaching roles and skills.
RESULT(S): Simple thematic coding showed that both groups emphasized the blurring of the many roles they serve when interacting with trainees and struggled with recognizing both which hat to wear (role to adopt) and which skills to call upon in specific situations. UME advisors who have dedicated advising/coaching roles reported assuming multiple roles at different times with their same students. Many of the GME coaches serve as Associate Program Directors, and described adopting a coaching frame of reference (mentality) and requiring external reinforcement for coaching skills. Some reported realizing after the fact that coaching would have been a valuable approach. Faculty newer to their role felt more successful in engaging in coaching mindset and coaching. Faculty were curious about how trainees would feel about this approach and anticipated that some would appreciate this more than others. 12 faculty participated in a three station Coaching GOSCE. Both resident raters and faculty peer raters suggested faculty coaches were able to establish trust and engage in authentic listening. Coaches negotiated the tension between empathetic listening with supporting goal-setting. Residents provided slightly lower ratings than peer observers on coaches' ability to ask questions and assume a coachee- focused agenda.
CONCLUSION(S): Medical educators may benefit from obtaining coaching skills, but deliberate training in how these skills complement, and differ, from existing skills requires both didactic and experiential learning. Cultivating a community of practice and offering opportunities for deliberate practice, observation and feedback is essential for medical educators to achieve mastery as coaches. LEARNING OBJECTIVE #1: Identify and perform appropriate learning activities to guide personal and professional development (PBL) LEARNING OBJECTIVE #2: Understand and apply core longitudinal coaching skills (Professionalism)
Notesense: development of a machine learning algorithm for feedback on clinical reasoning documentation [Meeting Abstract]
BACKGROUND: Clinical reasoning (CR) is a core component of medical training, yet residents often receive little feedback on their CR documentation. Here we describe the process of developing a machine learning (ML) algorithm for feedback on CR documentation to increase the frequency and quality of feedback in this domain.
METHOD(S): To create this algorithm, note quality first had to be rated by gold standard human rating. We selected the IDEA Assessment Tool-a note rating instrument across four domains (I=Interpretive summary, D=Differential diagnosis, E=Explanation of reasoning, A=Alternative diagnoses explained) that uses a 3-point Likert scale without descriptive anchors. To develop descriptive anchors we conducted an iterative process reviewing notes from the EHR written by medicine residents and validated the Revised-IDEA Assessment Tool using Messick's framework- content validity, response process, relation to other variables, internal structure, and consequences. Using the Hofstee standard setting method, cutoffs for high quality clinical reasoning for the IDEA and DEA scores were set. We then created a dataset of expertrated notes to create the ML algorithm. First, a natural language processing software was applied to the set of notes that enabled recognition and automatic encoding of clinical information as a diagnosis or disease (D's), a sign or symptom (E or A), or semantic qualifier (e.g. most likely). Input variables to the ML algorithm included counts of D's, E/A's, semantic qualifiers, and proximity of semantic qualifiers to disease/ diagnosis. ML output focused on DEA quality and was binarized to low or high quality CR. Finally, 200 notes were randomly selected for human validation review comparing ML output to human rated DEA score.
RESULT(S): The IDEA and DEA scores ranged from 0-10 and 0-6, respectively. IDEA score of >= 6.5 and a DEA score of >= 3 was deemed high quality. 252 notes were rated to create the dataset and 20% were rated by 3 raters with high intraclass correlation 0.84 (95% CI 0.74-0.90). 120 of these notes comprised the testing set for ML model development. The logistic regression model was the best performing model with an AUC 0.87 and a positive predictive value (PPV) of 0.65. 48 (40%) of the notes were high quality. There was substantial interrater reliability between ML output and human rating on the 200 note validation set with a Cohen's Kappa 0.64.
CONCLUSION(S): We have developed a ML algorithm for feedback on CR documentation that we hypothesize will increase the frequency and quality of feedback in this domain. We have subsequently developed a dashboard that will display the output of the ML model. Next steps will be to provide internal medicine residents' feedback on their CR documentation using this dashboard and assess the impact this has on their documentation quality. LEARNING OBJECTIVE #1: Describe the importance of high quality documentation of clinical reasoning. LEARNING OBJECTIVE #2: Identify machine learning as a novel assessment tool for feedback on clinical reasoning documentation
MIND THE GAP: TEACHING LIFELONG LEARNING THROUGH METACOGNITIVE AWARENESS [Meeting Abstract]
"I Cannot Take This Any More!": Preparing Interns to Identify and Help a Struggling Colleague
BACKGROUND:Few programs train residents in recognizing and responding to distressed colleagues at risk for suicide. AIM/OBJECTIVE:To assess interns' ability to identify a struggling colleague, describe resources, and recognize that physicians can and should help colleagues in trouble. SETTING/METHODS:Residency programs at an academic medical center. PARTICIPANTS/METHODS:One hundred forty-five interns. PROGRAM DESIGN/UNASSIGNED:An OSCE case was designed to give interns practice and feedback on their skills in recognizing a colleague in distress and recommending the appropriate course of action. Embedded in a patient "sign-out" case, standardized health professionals (SHP) portrayed a resident with depressed mood and an underlying drinking problem. The SHP assessed intern skills in assessing symptoms and directing the resident to seek help. PROGRAM EVALUATION/RESULTS:Interns appreciated the opportunity to practice addressing this situation. Debriefing the case led to productive conversations between faculty and residents on available resources. Interns' skills require further development: while 60% of interns asked about their colleague's emotional state, only one-third screened for depression and just under half explored suicidal ideation. Only 32% directed the colleague to specific resources for his depression (higher among those that checked his emotional state, 54%, or screened for depression, 80%). DISCUSSION/CONCLUSIONS:This OSCE case identified varying intern skill levels for identifying and assessing a struggling colleague while also providing experiential learning and supporting a culture of addressing peer wellness.
Renal failure and a clavicular mass: Don't cut yourself on occam's razor [Meeting Abstract]
Case Summary: A 66-year-old woman with hypertension presented to the hospital one day after arrival to New York City from Guinea with chronic daily vomiting, unintentional weight loss, progressive shoulder pain, and a subacute pruritic rash. On exam, patient was hypertensive, had limited range of motion in her right shoulder, scaling plaques over legs and trunk, and asterixis. Labs were notable for a creatinine of 15 with normocytic anemia. Calcium was normal. The patient was admitted to hospital for acute renal failure and further workup. Diagnoses: Primary end stage renal disease of unknown etiology; and follicular thyroid carcinoma with metastases to the clavicle and lungs with paraneoplastic rash.
Discussion(s): This case highlights the diagnostic principles of Occam's razor, Hickam's dictum, and Crabtree's bludgeon. The initial differential diagnosis and workup proceeded with the expectation of a unifying diagnosis to explain the wide constellation of presenting symptoms. But after common systemic unifying diagnoses, including multiple myeloma and other infiltrative processes, were ruled out, it became evident that two processes were at play. The first was end stage renal failure, likely long-standing given imaging findings suggestive of chronicity. The second, a common malignancy with uncommon metastases. Perhaps an absence of regular interactions with the healthcare system prior to presentation increases like the positive predictive value of Hickam's dictum. Additionally, common diseases with uncommon presentations are still more common than zebras. Lessons from this case allows us to expand our thinking beyond the dogma of any one diagnostic principle, to avoid type I thinking and help us to direct our diagnostic reasoning
Warm Handoffs: a Novel Strategy to Improve End-of-Rotation Care Transitions
BACKGROUND: Hospitalized medical patients undergoing transition of care by house staff teams at the end of a ward rotation are associated with an increased risk of mortality, yet best practices surrounding this transition are lacking. AIM: To assess the impact of a warm handoff protocol for end-of-rotation care transitions. SETTING: A large, university-based internal medicine residency using three different training sites. PARTICIPANTS: PGY-2 and PGY-3 internal medicine residents. PROGRAM DESCRIPTION: Implementation of a warm handoff protocol whereby the incoming and outgoing residents meet at the hospital to sign out in-person and jointly round at the bedside on sicker patients using a checklist. PROGRAM EVALUATION: An eight-question survey completed by 60 of 99 eligible residents demonstrated that 85% of residents perceived warm handoffs to be safer for patients (p < 0.001), while 98% felt warm handoffs improved their knowledge and comfort level of patients on day 1 of an inpatient rotation (p < 0.001) as compared to prior handoff techniques. Finally, 88% felt warm handoffs were worthwhile despite requiring additional time (p < 0.001). DISCUSSION: A warm handoff protocol represents a novel strategy to potentially mitigate the known risks associated with end-of-rotation care transitions. Additional studies analyzing patient outcomes will be needed to assess the impact of this strategy.
An uncommon presentation of streptococcus pneumoniae [Meeting Abstract]
LEARNING OBJECTIVE #1: Describe the mechanism of septic arthritis LEARNINGOBJECTIVE #2: Recognize the epidemiology, risk factors, and prognosis of patients with septic arthritis caused by Streptococcus pneumoniae CASE: A 43 year-old man with a history of Hodgkin's Lymphoma, treated with chemotherapy and radiation, complicated by avascular necrosis requiring bilateral hip replacements, and myelodysplastic syndrome requiring a splenectomy and stem cell transplant, presented with one month of productive cough, two days of myalgias, and acute on chronic bilateral shoulder and hip pain. On presentation to our institution, he was found to have profound leukocytosis with neutrophilia and bandemia and a computed tomography of the chest suggestive of a multi-lobar pneumonia. His physical exam was remarkable for a high fever, rigors, diminished breath sounds at the lung bases, and decreased range of motion in his right lower extremity due to right hip pain. He was treated empirically with intravenous vancomycin and piperacillin-tazobactam. Blood cultures grew pan-sensitive S. pneumoniae and his antibiotics were narrowed to intravenous ceftriaxone. His hospital course was complicated by persistent leukocytosis, fevers, and right hip pain. Magnetic resonance imaging and a gallium nuclear scan were performed, which suggested infection of his right hip. He underwent an arthrocentesis followed by a right hip wash out without prosthesis removal. He was discharged home on an extended course of antibiotics with resolution of his leukocytosis, fevers, and right hip pain. IMPACT: Our case highlights a relatively uncommon phenomenon, which is septic arthritis caused by S. pneumoniae. Clinicians should be aware of predisposing factors that place patients at risk for developing a joint infection from this organism. DISCUSSION: Septic arthritis occurs via bacterial invasion of the synovial membrane, most commonly by hematogenous spread, leading to an acute inflammatory response involving synovial membrane hyperplasia, local cytokine production, and cartilage destruction. Several studies report a widely disparate prevalence of septic arthritis caused by S. pneumoniae, with most studies reporting a low single digit prevalence. While relatively uncommon, S. pneumoniae arthritis should be considered in patients with extra-articular infections, such as pneumonia or meningitis, and predisposing factors, which include alcoholism, rheumatoid arthritis, immunosuppression, splenectomy, preexisting joint disease, or patients with prosthetic joints. Prosthetic joints pose a fifteen-time greater risk of infection by providing a surface for S. pneumoniae to multiply and form biofilms to defend against circulating leukocytes, complement, and antibiotics. Good functional outcome is seen in the majority of patients treated with appropriate antibiotic therapy and surgical intervention, and often patients with prostheses do not require removal