Most medical researchers blindly adhere to the popular dogma of p-values. According to this dogma, the strategy of declaring statistical significance is based on a p-value alone (often a p-value below 0.05). To the practician of this religion, Statistics refer solely to the investigation of such values. However, the probability that an association is true given a statistically significant finding, depends not only on the estimated p-value but also on the prior probability of it being real, the research bias (the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced) and the statistical power of the test. More specifically, it can been seen that the positive predictive value (PPV) of a test (i.e. the post-study probability that the association is true) equals*:
where R is the ratio of the number of “true relationships” to “no relationships” among those tested in the field, α is the Type I error rate, β is the Type II error rate (and hence 1-β is the “power” of the test) and u the research bias. Hence, according to the equation above (assuming at this point insignificant bias) a research finding is more probable to be true than false iff (1 – β)R > α.
The graphs below highlight the relationship between the variables. As we can easily observe (click graphs to zoom in) the higher the R and the lower the type II error the higher the PPV. The red surface corresponds to the zero research bias case while the green and the yellow correspondingly to u=0.2 and u=0.6. The ball blue plane corresponds to PPV 0.5 i.e. the cut-off positive predictive value. The multicoloured floor of the graph indicates the levels of β and R (for u=0, 0.2, 0.6) for which research findings are more possible than not.