How a $150 mistake can cost the scientific community billions

In late 2016, The Chemical Probes Portal created a category of small molecule called historical compounds. These small molecules are typically non-selective or at least not sufficiently potent compared with other available compounds to merit the “chemical probe” designation (i.e., a specific and selective tool that can be used to interrogate a particular protein). Indeed, we created this category of compounds to prevent their misuse by scientists as chemical probes.

We recently performed an analysis to estimate how much the misapplication of historic compounds as chemical probes costs the scientific community. While there is some good news, we were alarmed to find that what may appear a small purchasing error (i.e., spending $150 on a historic compound instead of a chemical probe) can cost the scientific community billions of dollars. We need to redouble our efforts to ensure scientists have access to the expert guidance they need when they are selecting critical reagents. When it comes to small molecules and chemical probes, we are growing The Chemical Probes Portal to help in this endeavor.

To perform our analysis, we first assessed PubMed to determine how often these historic compounds were being used in papers. We evaluated citations in PubMed (from 2016-present) that used any of 10 randomly selected historic compounds (LY294002, DNZep, AMI-1, Apomorphine, Vandetanib, Fedratinib, Axitinib, Epigallocatechin-3-gallate (ECGC), Apigenin, and Aloisine) from the close to 200 of these compounds that the Portal features. The results were troubling. Since 2016, these 10 compounds have been used in 2,090 papers (ranging from 0 to 803). LY294002 was the most widely used of the compounds we considered.

We next assessed the vendors who sell these compounds to scientists. In this part of our analysis, we were looking for two types of information. First, of the vendors who sell these compounds, how many of them are selling them as if they are a specific reagent (i.e., a chemical probe)? Second, how much do vendors charge for these compounds? This part of our analysis yielded some good news and some bad news.

All of the historic compounds we evaluated are available from multiple vendors (>250 opportunities to buy these compounds). We were pleasantly surprised to find that only a small subset of these vendors sells these historic compounds as if they were chemical probes (54/265 with each compound being advertised inappropriately by an average of 6 vendors). During this analysis we also noted that the vendors who sell these historic compounds as if they were chemical probes often sell the same compound for different purposes (i.e., one vendor sells apigenin as a PKC inhibitor and others sell it as a CYP29C inhibitor). While these findings indicate that some vendors are still contributing to and profiting from the misuse of historic compounds in research, the majority of vendors are making an effort to either provide more nuanced information about these compounds to their customers or sell them with no mention of their biological utility. This was an encouraging finding.

While reviewing vendor pages, we also assessed the prices they charge for these compounds. Using these numbers, we performed a rough calculation to estimate how much a scientist was likely to pay for these historic compounds; we then scaled that cost according to how many papers were published for each compound. In total, we estimate that scientists spent ~$312.5 K to obtain these compounds for the 2,090 papers (~$26K total per historic compound and ~$150 per study).

Considering that the misuse of a historic compound as a chemical probe is likely to yield irreproducible results, we also estimated the greater immediate cost of this misuse. We found a 2005 study that estimated the average cost of a scientific publication at $34K (Tracking publication outcomes of National Institutes of Health grants). Correcting for inflation, we estimate that these 2,090 papers and 10 historic compounds cost the scientific community ~$89M. To estimate the total cost of poor chemical probe selection on the scientific community, we will need a more complete catalog of historic compounds. We know there are at least 200 of these compounds; likely, there are many more. Extrapolating our estimate based on 10 compounds across just these 200, we reach an astounding $1.8B.

Cost per study = $150.00; Total cost to the international scientific community = $1.8B

Estimating the cost of poor quality reagent selection