Faculty Bibliography

Spontaneous reporting is a crucial component of post-marketing drug safety surveillance despite its significant limitations. The size and complexity of some spontaneous reporting system databases represent a challenge for drug safety professionals who traditionally have relied heavily on the scientific and clinical acumen of the prepared mind. Computer algorithms that calculate statistical measures of reporting frequency for huge numbers of drug-event combinations are increasingly used to support pharamcovigilance analysts screening large spontaneous reporting system databases. After an overview of pharmacovigilance and spontaneous reporting systems, we discuss the theory and application of contemporary computer algorithms in regular use, those under development, and the practical considerations involved in the implementation of computer algorithms within a comprehensive and holistic drug safety signal detection program

BACKGROUND: Pharmacovigilance data-mining algorithms (DMAs) are known to generate significant numbers of false-positive signals of disproportionate reporting (SDRs), using various standards to define the terms 'true positive' and 'false positive'. OBJECTIVE: To construct a highly inclusive reference event database of reported adverse events for a limited set of drugs, and to utilize that database to evaluate three DMAs for their overall yield of scientifically supported adverse drug effects, with an emphasis on ascertaining false-positive rates as defined by matching to the database, and to assess the overlap among SDRs detected by various DMAs. METHODS: A sample of 35 drugs approved by the US FDA between 2000 and 2004 was selected, including three drugs added to cover therapeutic categories not included in the original sample. We compiled a reference event database of adverse event information for these drugs from historical and current US prescribing information, from peer-reviewed literature covering 1999 through March 2006, from regulatory actions announced by the FDA and from adverse event listings in the British National Formulary. Every adverse event mentioned in these sources was entered into the database, even those with minimal evidence for causality. To provide some selectivity regarding causality, each entry was assigned a level of evidence based on the source of the information, using rules developed by the authors. Using the FDA adverse event reporting system data for 2002 through 2005, SDRs were identified for each drug using three DMAs: an urn-model based algorithm, the Gamma Poisson Shrinker (GPS) and proportional reporting ratio (PRR), using previously published signalling thresholds. The absolute number and fraction of SDRs matching the reference event database at each level of evidence was determined for each report source and the data-mining method. Overlap of the SDR lists among the various methods and report sources was tabulated as well. RESULTS: The GPS algorithm had the lowest overall yield of SDRs (763), with the highest fraction of events matching the reference event database (89 SDRs, 11.7%), excluding events described in the prescribing information at the time of drug approval. The urn model yielded more SDRs (1562), with a non-significantly lower fraction matching (175 SDRs, 11.2%). PRR detected still more SDRs (3616), but with a lower fraction matching (296 SDRs, 8.2%). In terms of overlap of SDRs among algorithms, PRR uniquely detected the highest number of SDRs (2231, with 144, or 6.5%, matching), followed by the urn model (212, with 26, or 12.3%, matching) and then GPS (0 SDRs uniquely detected). CONCLUSIONS: The three DMAs studied offer significantly different tradeoffs between the number of SDRs detected and the degree to which those SDRs are supported by external evidence. Those differences may reflect choices of detection thresholds as well as features of the algorithms themselves. For all three algorithms, there is a substantial fraction of SDRs for which no external supporting evidence can be found, even when a highly inclusive search for such evidence is conducted

The US Food and Drug Administration (FDA) recently published a warning regarding pancreatitis in association with the use of exenatide, an incretin mimetic used for the treatment of patients with diabetes mellitus. We note that this safety issue is not associated with a signal of disproportionate reporting (SDR) in the FDA Adverse Event Reporting System (AERS) database or the World Health Organization (Uppsala Monitoring Centre) Vigibase for any of four data-mining algorithms we tested (proportional reporting ratio, the multi-item gamma-Poisson shrinker, an urn model and the Bayesian Confidence Propagation Neural Network). Exenatide and acute pancreatitis may thus represent a 'false-negative' result for disproportionality-based data-mining methodology generally. We evaluate the possibility that this lack of an SDR is caused by the phenomenon known as 'masking' (or 'cloaking') and reject this hypothesis. While positive findings are understandably more exciting, we discuss why publishing negative findings, such as in this example, is important for placing the capabilities and limitations of drug safety data mining into proper perspective

Data mining algorithms are increasingly being used to support the process of signal detection and evaluation in pharmacovigilance. Published data mining exercises formulated within a screening paradigm typically calculate classical performance indicators such as sensitivity, specificity, predictive value and receiver operator characteristic curves. Extrapolating signal detection performance from these isolated data mining exercises to performance in real-world pharmacovigilance scenarios is complicated by numerous factors and some published exercises may promote an inappropriate and exclusive focus on only one aspect of performance. In this article, we discuss a variation on positive predictive value that we call the 'number needed to detect' that provides a simple and intuitive screening metric that might usefully supplement the usual presentations of data mining performance. We use a series of figures to demonstrate the nature and application of this metric, and selected adaptive variations. Even with simple and intuitive metrics, precisely quantifying the performance of contemporary data mining algorithms in pharmacovigilance is complicated by the complexity of the phenomena under surveillance and the manner in which the data are recorded in spontaneous reporting systems

PURPOSE: To provide commentary and points of caution to consider before incorporating data mining as a routine component of any Pharmacovigilance program, and to stimulate further research aimed at better defining the predictive value of these new tools as well as their incremental value as an adjunct to traditional methods of post-marketing surveillance. METHODS/RESULTS: Commentary includes review of current data mining methodologies employed and their limitations, caveats to consider in the use of spontaneous reporting databases and caution against over-confidence in the results of data mining. CONCLUSIONS: Future research should focus on more clearly delineating the limitations of the various quantitative approaches as well as the incremental value that they bring to traditional methods of pharmacovigilance

OBJECTIVE: Recent epidemiological studies identifying an association between some classes of gastric acid suppressants and Clostridium difficile colitis and community-acquired pneumonia prompted our analysis. Our objective was to retrospectively apply data mining algorithms (DMAs) to the Food and Drug Administration (FDA) drug safety database to see if they might have directed/redirected attention to the reported association of gastric acid suppressive drugs with C. difficile colitis and community-acquired pneumonia, prior to the published epidemiological findings that supported the association. DESIGN: Two statistical DMAs, proportional reporting ratios (PRRs) and multi-item gamma Poisson shrinker (MGPS), were applied to a spontaneous reporting system (SRS) database to identify signals of disproportionate reporting (SDRs). RESULTS: SDRs related to community-acquired pneumonia were observed for two proton pump inhibitors (lansoprazole and omeprazole), two H(2) antagonists (famotidine and roxatidine), and one antacid (magnesium silicate hydroxide). For C. difficile colitis, an SDR was generated for one proton pump inhibitor (lansoprazole). CONCLUSIONS: Although our analysis suggests that there may be an association between the SDRs using SRS data and the epidemiological findings, these results may not have alerted public health professionals in advance of published studies to an association between proton pump inhibitors/gastric acid suppressants and C. difficile colitis or community-acquired pneumonia. However, the analysis reveals the potential utility of DMAs to direct attention to more subtle indirect drug adverse effects in SRS databases that as yet are often identified from epidemiological investigations

OBJECTIVE: Data mining algorithms (DMAs) are being applied to spontaneous reporting system (SRS) databases in the hope of obtaining timely insights into post-licensure safety data. Some DMAs have been characterized as 'objective' screening tools. However, there are numerous available modifiable configuration parameters to choose from, including choice of vendor, that may affect results. Our objective is to compare the data mining results on pre-selected drug-event combinations (DECs) between two commonly used software programs using similar protocols. METHODS: Two DMAs, using three thresholds, were retrospectively applied to the USFDA safety database through Q2 2005 to a set of eight pre-selected DECs. RESULTS: Differences between the two vendors were found for the number of cases associated with a signal of disproportionate reporting (SDR), first year of SDRs, and the magnitude of the SDR scores for the selected DECs. These were deemed to be potentially significant for 45.8% (11/24) of the data points. CONCLUSION: The observed differences between vendors could partially be explained by their differing methods of data cleaning and transformation as well as by the specific features of individual algorithms. The choices of vendors and available data mining configurations maximize the exploratory capacity of data mining, but they also raise questions about the claimed objectivity of data mining results and can make data mining exercises susceptible to confirmation bias given the exploratory nature of data mining in pharmacovigilance. When reporting results, the vendor and all data mining configuration details should be specified