4 Things That I Wish Everyone Knew About Scientific Research

There has been a lot of talk lately about scientific research. Science produces observable effects from a specific set of conditions, which are sometimes changed by the researchers. When done well, it can provide the needed evidence to approve medication, intervention, method of assistance, etc. It might be the link between key variables for reducing negative outcomes or increasing the positive ones. Scientific studies have made it into the news quite often in recent years, especially recently with vaccines for the novel Coronavirus being tested worldwide.

I find myself thinking of many things that I wish would be mentioned in media reports or social media discussions of scientific studies more often. So here are a few things that I wish everyone knew about scientific research.

Science Is Neither The Silver Bullet Nor Complete Garbage

Science is observable, measurable, researcher controlled, and structured. None of this means though that it is going to predict the future. The prediction it does allow for is general trends and likelihood of certain outcomes to happen. This sometimes leads people to believe that science is not worth it if there are examples of the results playing out differently than the way researchers say it should. So how should we look at scientific research and the results that come from it? How much can we rely on scientific studies?

The key is recognizing the scientific process as broader than a single study. Each study adds small pieces to an overall body of research on a given topic. As each of those pieces is added, then we have a more complete picture. The more we can see results replicated, the more we can have confidence in them. For this reason, a single study is not likely to hold ground truth. There are actually studies that combine the results of many studies in order to better understand the overall pattern of results called a meta-analysis.

Science has plenty to offer as long as we can understand how individual studies fit within the larger research area. We can rely on it when we have that perspective and understand it well. Across several studies and tests, we should be able to identify the main ideas despite not being able to predict everything 100% of the time. The implication of this is that although we cannot say that science is going to be perfect in prediction, it is very far from useless. Science will give us our best opportunity for advancement in many different areas of life. But it takes time to study it well and to draw accurate conclusions.

So, keep following the science. Just know that it there is likely more to come on the topic that may either change what we thought we knew or add meaningful depth to what we already know. That doesn’t mean that science is terrible. It means that science is progressing.

Statistical Significance Is Overrated

Statistical significance has become a common talking point since data analytics in the business world have become more prevalent. There are a few things about statistical significance that are misunderstood though.

The first is one of interpretation. All a statistically significant result means is that the effect is not zero. It does not indicate how much of an effect exists or how strong the effect is. To say that there is a difference is often not enough for any meaningful analysis or conclusion to be drawn. While helpful, what I have seen too often is an over-reliance on statistical significance. People will often take the stance of “it is statistically significant so we don’t need to analyze or interpret anything else”. However, the effect could still be a really weak one. Which brings me to my second thought on this…

Statistical significance does not mean that you have a meaningful effect. It is only that an effect exists. Effect size statistics (e.g., correlations) are what tell you how strong or good an effect it is. So, you can have a statistically significant result with a very weak effect size leading to essentially nothing. Statistical significance paired with a good effect size is what you should hope and look for.

My last point about significance is that statistical significance is highly dependent on the number of people in the analysis. A sample size of just 400 people can result in weak effect sizes being significant. If you’re a studying a large group (say 3M with their 90,000+ employees or the United States with over 330 million people), a sample of 400 people is not likely to be representative. And yet odds are, the results will be statistically significant. A non-representative sample that relies solely on significance is a recipe for error. In cases like these, statistical significance may be useless.

The Importance of Correlations

“Correlation does not mean causation” has become a common phrase in lay people terms. Let me first say that this is true. Just because one thing is present when another is also present doesn’t mean that one causes the other. What I fear is that so many people say this without understanding that correlations are still quite meaningful, valuable, and in fact necessary for the end goals of research.

First, a correlation between two variables is needed for there to be a causal relationship between them. If two variables are not correlated, then it is remarkably difficult (if not impossible) for one of those to cause the other. They have to show up at the same time for one to lead to the other. It is necessary but insufficient to imply causation. But the correlation is a very important piece to the causal relationship that many people discount because it doesn’t mean on its own that one variable caused another. So even though wearing bathing suits does not cause increased ice cream sales, high temperatures still need to exist in concert with those increased ice cream sales to conclude that it is the fundamental causal factor.

Second, correlations often provide more rich information since they provide the strength of a relationship. In most cases, is it important to know that one thing causes another? Or is just enough to know that if we improve one, then we also improve the other? No matter what the underlying reasons are for that correlation, improve one of those things and then the other will increase (or decrease as the desired case may be). The practicality of the question at hand is often one of correlation, not of causation.

Know The Publication Source

Believe it or not, the publication process can sometimes be sullied by politics or money. There are journals that are “pay to publish”. There are companies that pay groups to conduct research for them because they think that it can help their brand. Both of these methods introduce variance in the motivation for conducting studies and can lead to biased research. There is also nepotism that can sometimes go along with publishing research. The point is, the process still involves human judgment so it will still involve all the shortcomings that come with that. Seeing the overall body of research goes a long way towards mitigating this concern since the most accurate and consistent results can be replicated across many studies.

Final Thoughts

Science is worth it. It gives us the best and clearest chance of understanding our observable and measurable world. The next time you hear or read a report about a scientific study, ask yourself these questions:

Does the reporting tell you how the study fits into the rest of the research on the topic?
Does the analysis incorporate strength or magnitude of the relationship (i.e., effect size) in addition to statistical significance?
Who are all the parties involved with publishing the research? How might that affect the researchers’ interpretation of the data?
What is left to learn about the topic?

If you think critically about these questions, it can create the type of dialogue that I think many people are thirsting for on many important topics. Between social issues, global pandemic, climate change, questions about bias/fairness in decision making, we are in need of good, responsible, well executed science in our day and age more than ever. My hope is that we can all be a little savvier as research consumers in identifying what that means.

As always, thanks for reading!

–Brandon
You can subscribe to the blog below!