P-values: Why so slippery and yet an important concept

Hi Readers! I am sure along with the significance level, you have really heard about p-value.

Many researchers, statisticians, data analysts came across this p-value all the times, but my intention is to stress more on making you understand the intuitive concept behind the p-value so that it can remove the deadlock for everyone testing some sort of hypotheses problems.

Any time you see a p-value, you know you’re looking at the results of a hypothesis test. P-values determine whether your hypothesis test results are statistically significant. If your p-value is less than the significance level, you can reject the null hypothesis and conclude that the effect or relationship exists. In other words, your sample evidence is strong enough to determine that the effect exists in the population.

Statistics use p-values all over the place. You’ll find p-values in t-tests, distribution tests, ANOVA, and regression analysis. They have become so crucial that they’ve taken on a life of their own. They can determine which studies are published, which projects receive funding, and which university faculty members become tenured!

Ironically, despite being so influential, p-values are misinterpreted very frequently. What is the correct interpretation of p-values ? What do p-values really mean? That’s the topic of this blog! P-values are a slippery concept. I shall try to explain p-values using an intuitive, concept-based approach so you can avoid making a widespread misinterpretation that can cause serious problems.

It’s All About the Null Hypothesis

P-values are directly connected to the null hypothesis. In all hypothesis tests, the researchers are testing an effect or relationship of some sort. The effect can be the effectiveness of a new vaccination, the durability of a new product, and so on. There is some benefit or difference that the researchers hope to identify. However, its possible that there actually is no effect or no difference between the experimental groups. In statistics, we call this lack of an effect the null hypothesis. When you assess the results of a hypothesis test, you can think of the null hypothesis as the devil’s advocate position, or the position you take for the sake of argument. To understand this idea, imagine a hypothetical study for medication that we know already that it is entirely useless. In other words, the null hypothesis is true. There is no difference in patient outcomes at the population level between subjects who take the medication and subjects who don’t. Despite the null being accurate, you will likely observe an effect in the sample data due to random sampling error. It is improbable that samples will ever exactly equal the null hypothesis value.

Defining p-values

P-values indicate the believability of the devil’s advocate case that the null hypothesis is correct given the sample data. They gauge how consistent your sample statistics are with the null hypothesis. Specifically, if the null hypothesis is right, what is the probability of obtaining an effect at least as large as the one in your sample ?

• High p-values: Your sample results are consistent with a true null hypothesis.

• Low p-values: Your sample results are not consistent with a true null hypothesis.

If your p-value is small enough, you can conclude that your sample is so incompatible with the null hypothesis that you can reject the null for the entire population. P-values are an integral part of inferential statistics because they help you use your sample to draw conclusions about a population.

Here is the technical definition of p-value:

P-values are the probability of observing a sample statistic that is at least as extreme as your sample statistic when you assume that the null hypothesis is correct. 

Key Point: How probable are your sample data if the null hypothesis is correct ? That’s the only question that p-values answer.

P-values Are NOT an Error Percentage

Unfortunately, p-values are frequently misinterpreted. A common mistake is that they represent the likelihood of rejecting a null hypothesis that is actually true (Type I error). The idea that p-values are the probability of making a mistake is WRONG!

You can’t use p-values to calculate the error rate directly for several reasons. First, p-value calculations assume that the null hypothesis is correct. Thus, from the p-value’s point of view, the null hypothesis is 100% true. Second, p-values tell you how consistent your sample data are with a true null hypothesis. However, when your data are very inconsistent with the null hypothesis, p-values can’t determine which of the following two possibilities is more probable:

• The null hypothesis is true, but your sample is unusual due to random sampling error.

• The null hypothesis is false.

Thanks for reading :)