The number of studies that get published is a fraction of those submitted to journals. There is a distorted representation of data on a given subject – a filtering process – to which the public is largely blind. When there is a bias, the public has no choice but to trust the published results and distort that bias even more.
Why Most Published Research Findings Are False: by John Ioannidis in 2005, highlights this publication bias.
Journals look for studies with strong evidence
For a study to be published, it should be significant. In statistics, a p-value is a measure of how significant a study is. A low p-value – typically ≤ 0.05 – indicates strong evidence. Scientific studies are vulnerable to a concept called P-Hacking, where a number of variables are measured simultaneously to find a correlation with a low p-value. Any insignificant results are downplayed or discarded.
A good motto for aspiring scientists: Test novel and unexpected hypotheses and p-hack your way to publication.
Insignificant results or indeterminable conclusions are not of interest to journals
Insignificant results studies, “We thought X was true, but it really isn’t.” which doesn’t grab the attention of journals. Most studies result in this null conclusion. Scientific journals also don’t publish studies that disprove previous studies already published, leaving an erroneous finding unchecked.
Studies that get published could be erroneous or misleading
Among many shortcomings, published studies with small sample sizes can lead to errors in the findings, which produce incorrect conclusions. While scientific studies may refute the findings of published studies, original scientists/authors are incentivized to prevent being disproved by putting fences around the study’s findings.
The problem is that a potentially incorrect theory can catch hold in the zeitgeist. Laypeople will take a factoid they have heard widely used, especially by the media, and take it as truth. The factoid is most effective if it is short, utilizes a round number, provocative, and aspirational.
While published studies may be truthful, or at least thought provoking, we should be wary of them as there is a hidden game behind publication of studies.