Faculty Bibliography

In the last 5 years, regulatory agencies and drug monitoring centres have been developing computerised data-mining methods to better identify reporting relationships in spontaneous reporting databases that could signal possible adverse drug reactions. At present, there are no guidelines or standards for the use of these methods in routine pharmaco-vigilance. In 2003, a group of statisticians, pharmaco-epidemiologists and pharmaco-vigilance professionals from the pharmaceutical industry and the US FDA formed the Pharmaceutical Research and Manufacturers of America-FDA Collaborative Working Group on Safety Evaluation Tools to review best practices for the use of these methods.In this paper, we provide an overview of: (i) the statistical and operational attributes of several currently used methods and their strengths and limitations; (ii) information about the characteristics of various postmarketing safety databases with which these tools can be deployed; (iii) analytical considerations for using safety data-mining methods and interpreting the results; and (iv) points to consider in integration of safety data mining with traditional pharmaco-vigilance methods. Perspectives from both the FDA and the industry are provided. Data mining is a potentially useful adjunct to traditional pharmaco-vigilance methods. The results of data mining should be viewed as hypothesis generating and should be evaluated in the context of other relevant data. The availability of a publicly accessible global safety database, which is updated on a frequent basis, would further enhance detection and communication about safety issues.

PURPOSE: Several data mining algorithms (DMAs) are being studied in hopes of enhancing screening of large post-marketing safety databases for signals of novel adverse events (AEs). The objective of this study was to apply two DMAs to the United States FDA Adverse Event Reporting System (AERS) database to see whether signals of potentially fatal AEs with cancer drugs might have been identified earlier than with traditional methods. METHODS: Screening algorithms used for analysis were the multi-item gamma Poisson shrinker (MGPS) and proportional reporting ratios (PRRs). Data mining was performed on data from the FDA AERS database. When a signal was identified, it was compared with that in the year in which the event was added to package insert and/or the year a "case series" was published. A recent publication summarizing the time of dissemination of information on potentially fatal AEs to cancer drugs provided the data set for analysis. RESULTS: The peer-reviewed published analysis contained 21 drugs and 26 drug-event combinations (DECs) that were considered sufficiently specific for data mining. Twenty-four of the DECs generated a signal of disproportionate reporting with PRRs (6 at 1 year and 16 from 2 years to 18 years prior to either a published "case series" or a package insert change) and 20 with MGPS (3 at 1 year and 11 from 2 years to 16 years prior to either a published "case series" or a package insert change). Two DECs did not signal with either DMA. CONCLUSION: At least one commonly cited DMA generated a signal of disproportionate reporting for 24 of 26 DECs for selected cancer drugs. For 16 DECs, one could conclude that a signal was generated well in advance (> or =2 years) of standard techniques in use with at least one DMA. DMAs might be useful in supplementing traditional surveillance strategies with oncology drugs and other drugs with similar features. (i.e., drugs that may be approved on an accelerated basis, are known to have serious toxicity, are administered to patients with substantial and complicated comorbid illness, are not available to the general medical community, and may have a high frequency of "off-label" use).

BACKGROUND: Computer-assisted data mining algorithms (DMAs) are being studied to screen spontaneous reporting databases for signals of novel adverse events. The performance characteristics and optimum deployment of these techniques remain to be established. OBJECTIVE: To explore issues in the practical evaluation and deployment of DMAs by comparing findings from an empirical Bayesian DMA with those from a traditional drug safety surveillance program. METHODS: Published findings from early postmarketing safety surveillance of thalidomide were compared with findings from an empirical Bayesian DMA. Differential results were used to explore practical issues in the evaluation and deployment of DMAs. RESULTS: Most adverse events highlighted by each method were compatible with the product labeling or natural history/complications of reported treatment indications. Traditional surveillance highlighted 4 potentially serious and unexpected adverse events (Stevens-Johnson syndrome, toxic epidermal necrolysis, seizures, skin ulcers) warranting labeling amendments or close monitoring. None of these adverse event terms generated a signal using the DMA. CONCLUSIONS: The DMA would not have enhanced early postmarketing surveillance in this particular setting. While the results cannot be used to draw inferences about the global performance of DMAs, they illustrate the following: (1) DMA performance may be highly situation dependent; (2) over-reliance on these methods may have deleterious consequences, especially with so-called 'designated medical events'; and (3) the most appropriate selection of pharmacovigilance tools needs to be tailored to each situation, being mindful of the numerous factors that may influence comparative performance and incremental utility of DMAs

STUDY OBJECTIVE: To compare the results from one frequently cited data mining algorithm with those from a study, which was published in a peer-reviewed journal, that examined the association of pancreatitis with selected atypical antipsychotics observed by traditional rule-based methods of signal detection. DESIGN: Retrospective pharmacovigilance study. INTERVENTION: The widely studied data mining algorithm known as the Multi-item Gamma Poisson Shrinker (MGPS) was applied to adverse-event reports from the United States Food and Drug Administration's Adverse Event Reporting System database through the first quarter of 2003 for clozapine, olanzapine, and risperidone to determine if a significant signal of pancreatitis would have been generated by this method in advance of their review or the addition of these events to the respective product labels. MEASUREMENTS AND MAIN RESULTS: Data mining was performed by using nine preferred terms relevant to drug-induced pancreatitis from the Medical Dictionary for Regulatory Activities (MedDRA). Results from a previous study on the antipsychotics were reviewed and analyzed. Physicians' Desk References (PDRs) starting from 1994 were manually reviewed to determine the first year that pancreatitis was listed as an adverse event in the product label for each antipsychotic. This information was used as a surrogate marker of the timing of initial signal detection by traditional criteria. Pancreatitis was listed as an adverse event in a PDR for all three atypical antipsychotics. Despite the presence of up to 88 reports/drug-event combination in the Food and Drug Administration's Adverse Event Reporting System database, the MGPS failed to generate a signal of disproportional reporting of pancreatitis associated with the three antipsychotics despite the signaling of these drug-event combinations by traditional rule-based methods, as reflected in product labeling and/or the literature. These discordant findings illustrate key principles in the application of data mining algorithms to drug safety surveillance. CONCLUSION: The optimal place of data mining algorithms in the pharmacovigilance tool kit remains to be determined, requires consideration of numerous factors that may affect their performance, and is highly situation dependent.

INTRODUCTION: With increasing volumes of postmarketing safety surveillance data, data mining algorithms (DMAs) have been developed to search large spontaneous reporting system (SRS) databases for disproportional statistical dependencies between drugs and events. A crucial question is the proper deployment of such techniques within the universe of methods historically used for signal detection. One question of interest is comparative performance of algorithms based on simple forms of disproportionality analysis versus those incorporating Bayesian modelling. A potential benefit of Bayesian methods is a reduced volume of signals, including false-positive signals. OBJECTIVE: To compare performance of two well described DMAs (proportional reporting ratios [PRRs] and an empirical Bayesian algorithm known as multi-item gamma Poisson shrinker [MGPS]) using commonly recommended thresholds on a diverse data set of adverse events that triggered drug labelling changes. METHODS: PRRs and MGPS were retrospectively applied to a diverse sample of drug-event combinations (DECs) identified on a government Internet site for a 7-month period. Metrics for this comparative analysis included the number and proportion of these DECs that generated signals of disproportionate reporting with PRRs, MGPS, both or neither method, differential timing of signal generation between the two methods, and clinical nature of events that generated signals with only one, both or neither method. RESULTS: There were 136 relevant DECs that triggered safety-related labelling changes for 39 drugs during a 7-month period. PRRs generated a signal of disproportionate reporting with almost twice as many DECs as MGPS (77 vs 40). No DECs were flagged by MGPS only. PRRs highlighted DECs in advance of MGPS (1-15 years) and a label change (1-30 years). For 59 DECs, there was no signal with either DMA. DECs generating signals of disproportionate reporting with only PRRs were both medically serious and non-serious. DISCUSSION/CONCLUSION: In most instances in which a DEC generated a signal of disproportionate reporting with both DMAs (almost twice as many with PRRs), the signal was generated using PRRs in advance of MGPS. No medically important events were signalled only by MGPS. It is likely that the incremental utility of DMAs are highly situation-dependent. It is clear, however, that the volume of signals generated by itself is an inadequate criterion for comparison and that clinical nature of signalled events and differential timing of signals needs to be considered. Accepting commonly recommended threshold criteria for DMAs examined in this study as universal benchmarks for signal detection is not justified.

BACKGROUND: Statistical techniques have traditionally been underused in spontaneous reporting systems used for postmarketing surveillance of adverse drug events. Regulatory agencies, pharmaceutical companies, and drug monitoring centers have recently devoted considerable efforts to develop and implement computer-assisted automated signal detection methodologies that employ statistical theory to enhance screening efforts of expert clinical reviewers. OBJECTIVE: To provide a concise state-of-the-art review of the most commonly used automated signal detection procedures, including the underlying statistical concepts, performance characteristics, and outstanding limitations, and issues to be resolved. DATA SOURCES: Primary articles were identified by MEDLINE search (1965-December 2002) and through secondary sources. STUDY SELECTION AND DATA EXTRACTION: All of the articles identified from the data sources were evaluated and all information deemed relevant was included in this review. DATA SYNTHESIS: Commonly used methods of automated signal detection are self-contained and involve screening large databases of spontaneous adverse event reports in search of interestingly large disproportionalities or dependencies between significant variables, usually single drug-event pairs, based on an underlying model of statistical independence. The models vary according to the underlying model of statistical independence and whether additional mathematical modeling using Bayesian analysis is applied to the crude measures of disproportionality. There are many potential advantages and disadvantages of these methods, as well as significant unresolved issues related to the application of these techniques, including lack of comprehensive head-to-head comparisons in a single large transnational database, lack of prospective evaluations, and the lack of gold standard of signal detection. CONCLUSIONS: Current methods of automated signal detection are nonclinical and only highlight deviations from independence without explaining whether these deviations are due to a causal linkage or numerous potential confounders. They therefore cannot replace expert clinical reviewers, but can help them to focus attention when confronted with the difficult task of screening huge numbers of drug-event combinations for potential signals. Important questions remain to be answered about the performance characteristics of these methods. Pharmacovigilance professionals should take the time to learn the underlying mathematical concepts in order to critically evaluate accumulating experience pertaining to the relative performance characteristics of these methods that are incompletely defined

Pharmacovigilance serves to detect previously unrecognised adverse events associated with the use of medicines. The simplest method for detecting signals of such events is crude inspection of lists of spontaneously reported drug-event combinations. Quantitative and automated numerator-based methods such as Bayesian data mining can supplement or supplant these methods. The theoretical basis and limitations of these methods should be understood by drug safety professionals, and automated methods should not be automatically accepted. Published evaluations of these techniques are mainly limited to large regulatory databases, and performance characteristics may differ in smaller safety databases of drug developers. Head-to-head comparisons of the major techniques have not been published. Regardless of previous statistical training, pharmacovigilance practitioners should understand how these methods work. The mathematical basis of these techniques should not obscure the numerous confounders and biases inherent in the data. This article seeks to make automated signal detection methods transparent to drug safety professionals of various backgrounds. This is accomplished by first providing a brief overview of the evolution of signal detection followed by a series of sections devoted to the methods with the greatest utilisation and evidentiary support: proportional reporting rations, the Bayesian Confidence Propagation Neural Network and empirical Bayes screening. Sophisticated yet intuitive explanations are provided for each method, supported by figures in which the underlying statistical concepts are explored. Finally the strengths, limitations, pitfalls and outstanding unresolved issues are discussed. Pharmacovigilance specialists should not be intimidated by the mathematics. Understanding the theoretical basis of these methods should enhance the effective assessment and possible implementation of these techniques by drug safety professionals.