Faculty Bibliography

OBJECTIVES/OBJECTIVE:Public and private organizations are implementing systems for query-based health information exchange (HIE), the electronic aggregation of patient data from multiple institutions. However, existing studies of query-based HIE system usage have addressed a limited number of settings. Our goal was to quantify the breadth and depth of usage of a query-based HIE system implemented across multiple communities with diverse care settings and patient populations. METHODS:We performed a cross-sectional study in three communities in New York State using system access log files from January 2009 to May 2011 to measure usage patterns of a query-based HIE web portal system with respect to practice sites, users, patients, and data. RESULTS:System access occurred from 60% (n=200) of practice sites registered to use the system in Community A, 59% (n=156) in Community B, and 82% (n=28) in Community C. In Communities A and B, users were primarily non-clinical staff in outpatient settings, while in Community C inpatient physicians were the main users. Across communities, proportions of patients whose data were accessed varied with 5% (n=11,263) in Community A, 60% (n=212,586) in Community B, and 1% (n=1107) in Community C. In Community B, users updated patient consent through the HIE portal, whereas in the other communities, users updated patient consent through a separate system. Across communities, users most frequently accessed only patient summary data displayed by default followed by detailed laboratory and radiology data. CONCLUSIONS:This study is among the first to illustrate large-scale usage of a query-based HIE system implemented across multiple communities. Patient summary data displayed by default may be an important feature of query-based HIE systems. User role, practice site type, and patient consent workflow may affect patterns of query-based HIE web portal system usage in the communities studied and elsewhere.

OBJECTIVE:To develop a semantic representation for clinical research eligibility criteria to automate semistructured information extraction from eligibility criteria text. MATERIALS AND METHODS/METHODS:An analysis pipeline called eligibility criteria extraction and representation (EliXR) was developed that integrates syntactic parsing and tree pattern mining to discover common semantic patterns in 1000 eligibility criteria randomly selected from http://ClinicalTrials.gov. The semantic patterns were aggregated and enriched with unified medical language systems semantic knowledge to form a semantic representation for clinical research eligibility criteria. RESULTS:The authors arrived at 175 semantic patterns, which form 12 semantic role labels connected by their frequent semantic relations in a semantic network. EVALUATION/RESULTS:Three raters independently annotated all the sentence segments (N=396) for 79 test eligibility criteria using the 12 top-level semantic role labels. Eight-six per cent (339) of the sentence segments were unanimously labelled correctly and 13.8% (55) were correctly labelled by two raters. The Fleiss' κ was 0.88, indicating a nearly perfect interrater agreement. CONCLUSION/CONCLUSIONS:This study present a semi-automated data-driven approach to developing a semantic network that aligns well with the top-level information structure in clinical research eligibility criteria text and demonstrates the feasibility of using the resulting semantic role labels to generate semistructured eligibility criteria with nearly perfect interrater reliability.

CONTEXT/BACKGROUND:Public health services and systems research (PHSSR) focuses on the structure, organization, and legal basis of domestic public health activities and their effect on population health. An accurate description of the field is needed to empower funding agencies and other stakeholders to coordinate PHSSR activities and to foster the development of the field. The purpose of the study is to characterize the emerging community of researchers engaged in PHSSR. This study (1) describes dynamics of this growing community and (2) identifies research themes, subgroups within the field, and collaboration among groups. EVIDENCE ACQUISITION/METHODS:Coauthorship network visualization of selected research publications in the MEDLINE bibliographic database between 1988 and May 2010. EVIDENCE SYNTHESIS/RESULTS:PHSSR has emerged gradually with noticeable growth after 1994 and after 2004. The network of PHSSR research has a core-periphery structure. The core includes highly collaborative researchers focusing on topics pertaining directly to PHSSR, such as workforce, quality improvement and performance, law, and information infrastructure. The periphery consists of groups publishing either on general health services research topics or on epidemiologic and clinical topics. CONCLUSIONS:Although a nucleus group of productive and engaged individuals participate in PHSSR, most also publish broadly on health services research and population health. This trend suggests that this emerging field cannot yet support a singular focus on PHSSR. Lack of funding sources and defined career paths likely contribute to this pattern. An overview of collaboration in PHSSR is an important step in advancing a coordinated research agenda and attracting sustainable funding streams for this field.

Temporal constraints are present in 38% of clinical research eligibility criteria and are crucial for screening patients. However, eligibility criteria are often written as free text, which is not amenable for computer processing. In this paper, we present an ontology-based approach to extracting temporal information from clinical research eligibility criteria. We generated temporal labels using a frame-based temporal ontology. We manually annotated 150 free-text eligibility criteria using the temporal labels and trained a parser using Conditional Random Fields (CRFs) to automatically extract temporal expressions from eligibility criteria. An evaluation of an additional 60 randomly selected eligibility criteria using manual review achieved an overall precision of 83%, a recall of 79%, and an F-score of 80%. We illustrate the application of temporal extraction with the use cases of question answering and free-text criteria querying.

This paper describes a usability evaluation study of an innovative first generation system (Data Dig) designed to retrieve phenotypic data from the large SFARI data set of 2700 families each of which has one child affected with autism spectrum disorder. The usability methods included a cognitive walkthrough and usability testing. Although the subjects were able to learn to use the system, more than 50 usability problems of varying severity were noted. The problems with the greatest frequency resulted from users being unable to understand meanings of variables, filter categories correctly, use the Boolean filter, and correctly interpret the feedback provided by the system. Subjects had difficulty forming a mental model of the organizational system underlying the database. This precluded them from making informed navigation choices while formulating queries. Clinical research informatics is a new and immensely promising discipline. However in its nascent stage, it lacks a stable interaction paradigm to support a range of users on pertinent tasks. This presents great opportunity for researchers to further this science by harnessing the powers of user-centered iterative design.

This paper presents a novel approach to learning semantic classes of clinical research eligibility criteria. It uses the UMLS Semantic Types to represent semantic features and the Hierarchical Clustering method to group similar eligibility criteria. By establishing a gold standard using two independent raters, we evaluated the coverage and accuracy of the induced semantic classes. On 2,718 random eligibility criteria sentences, the inter-rater classification agreement was 85.73%. In a 10-fold validation test, the average Precision, Recall and F-score of the classification results of a decision-tree classifier were 87.8%, 88.0%, and 87.7% respectively. Our induced classes well aligned with 16 out of 17 eligibility criteria classes defined by the BRIDGE model. We discuss the potential of this method and our future work.

OBJECTIVE:To propose a centralized method for generating global unique identifiers to link collections of research data and specimens. DESIGN/METHODS:The work is a collaboration between the Simons Foundation Autism Research Initiative and the National Database for Autism Research. The system is implemented as a web service: an investigator inputs identifying information about a participant into a client application and sends encrypted information to a server application, which returns a generated global unique identifier. The authors evaluated the system using a volume test of one million simulated individuals and a field test on 2000 families (over 8000 individual participants) in an autism study. MEASUREMENTS/METHODS:Inverse probability of hash codes; rate of false identity of two individuals; rate of false split of single individual; percentage of subjects for which identifying information could be collected; percentage of hash codes generated successfully. RESULTS:Large-volume simulation generated no false splits or false identity. Field testing in the Simons Foundation Autism Research Initiative Simplex Collection produced identifiers for 96% of children in the study and 77% of parents. On average, four out of five hash codes per subject were generated perfectly (only one perfect hash is required for subsequent matching). DISCUSSION/CONCLUSIONS:The system must achieve balance among the competing goals of distinguishing individuals, collecting accurate information for matching, and protecting confidentiality. Considerable effort is required to obtain approval from institutional review boards, obtain consent from participants, and to achieve compliance from sites during a multicenter study. CONCLUSION/CONCLUSIONS:Generic unique identifiers have the potential to link collections of research data, augment the amount and types of data available for individuals, support detection of overlap between collections, and facilitate replication of research findings.

OBJECTIVE: Rigorous human-computer interaction (HCI) design methodologies have not traditionally been applied to the development of clinical trial participant tracking (CTPT) tools. Given the frequent us of iconic HCI models in CTPTs, and prior evidence of usability problems associated with the use of ambiguous icons in complex interfaces, such approaches may be problematic. Presentation Discovery (PD), a knowledge-anchored HCI design method, has been previously demonstrated to improve the design of iconic HCI models. In this study, we compare the usability of a CTPT HCI model designed using PD and an intuitively designed CTPT HCI model. METHODS: An iconic CPTP HCI model was created using PD. The PD-generated and an existing iconic CTPT HCI model were subjected to usability testing, with an emphasis on task accuracy and completion times. Study participants also completed a qualitative survey instrument to evaluate subjective satisfaction with the two models. RESULTS: CTPT end-users reliably and reproducibly agreed on the visual manifestation and semantics of prototype graphics generated using PD. The performance of the PD-generated iconic HCI model was equivalent to an existing HCI model for tasks at multiple levels of complexity, and in some cases superior. This difference was particularly notable when tasks required an understanding of the semantic meanings of multiple icons. CONCLUSION: The use of PD to design an iconic CTPT HCI model generated beneficial results and improved end-user subjective satisfaction, while reducing task completion time. Such results are desirable in information and time intensive domains, such as clinical trials management.

Building upon the foundation of the Structured Narrative Electronic Health Record (EHR) model, we applied theory-based (combined Technology Acceptance Model and Task-Technology Fit Model) and user-centered methods to explore nurses' perceptions of functional requirements for an electronic nursing documentation system, design user interface screens reflective of the nurses' perspectives, and assess nurses' perceptions of the usability of the prototype user interface screens. The methods resulted in user interface screens that were perceived to be easy to use, potentially useful, and well-matched to nursing documentation tasks associated with Nursing Admission Assessment, Blood Administration, and Nursing Discharge Summary. The methods applied in this research may serve as a guide for others wishing to implement user-centered processes to develop or extend EHR systems. In addition, some of the insights obtained in this study may be informative to the development of safe and efficient user interface screens for nursing document templates in EHRs.

Searches of bibliographic databases generate lists of articles but do little to reveal connections between authors, institutions, and grants. As a result, search results cannot be fully leveraged. To address this problem we developed Sciologer, a prototype search and visualization system. Sciologer presents the results of any PubMed query as an interactive network diagram of the above elements. We conducted a cognitive evaluation with six neuroscience and six obesity researchers. Researchers used the system effectively. They used geographic, color, and shape metaphors to describe community structure and made accurate inferences pertaining to a) collaboration among research groups; b) prominence of individual researchers; and c) differentiation of expertise. The tool confirmed certain beliefs, disconfirmed others, and extended their understanding of their own discipline. The majority indicated the system offered information of value beyond a traditional PubMed search and that they would use the tool if available.