Lily V / UnsplashIf you’ve ever tried to read a scientific paper, you’ve almost certainly run into a p-value. And if you found it confusing, you’re in good company: plenty of working scientists misunderstand it too.The p-value is a number measuring the “statistical significance” of an experiment, and it’s often treated as the adjudicator of whether the experiment really found something or not. It decides whether researchers get to say that a drug did something, or that one group differed from another. Careers, publications and our understanding of the world hinge on p-values. So it’s worth understanding what these numbers actually mean, because it’s not what most people think.What does a p-value mean?Suppose we want to know whether adults in one country are taller, on average, than adults in another. We cannot measure everyone, so we sample 50 adults from each country. The first sample averages 1.80 metres, the second 1.77 metres. That 3cm gap tells us little on its own, because any two samples will differ to some degree simply because of who happened to be picked. So is the gap good evidence of a real difference in average heights across the whole population? Or is it just the kind of thing that turns up through the luck of the draw?To find out, a scientist runs a statistical test which returns a p-value. Say it comes back as p = 0.06. Here is what that means: if there were genuinely no difference between the countries, a gap at least this large would turn up about 6% of the time through the luck of sampling.Now notice what it does not mean. It does not mean there is a 6% chance the result was a fluke, or a 94% chance the two countries really differ. The p-value tells you how likely a difference this big is if nothing real is going on, not how likely it is that nothing real is going on. People swap the two constantly, and it is easy to do even when you know better.Why 0.05?By convention, a p-value below 0.05 is called “statistically significant”. Our height study, with a p-value of 0.06, would be written up as showing no significant difference.Where does the 0.05 threshold come from? Nowhere in particular. British statistician Ronald Fisher offered it in the 1920s as a rule of thumb, and the convenience hardened into a rule.Having 0.05 as the line for “significance” does do one useful thing. If there were no real effect at all, setting it at 0.05 means a study would still produce a “significant” result about one time in 20, purely by chance. That’s the rate of false alarms we have agreed to tolerate. It’s not a claim about how many published findings are wrong. What is arbitrary is one in 20 rather than one in 100 or one in ten, and it produces faintly absurd outcomes: if p = 0.049, it’s reported as a discovery; if p = 0.051 we act as though nothing happened.‘Not significant’ doesn’t mean ‘no difference’Coming back to our height experiment, we might be tempted to treat our p = 0.06 result as proof that people in the two countries are equally tall. It is nothing of the sort. A non-significant result means only that our gap was not unusual enough to clear the threshold. The real difference could be substantial. Our study may simply have been too small, or too noisy, to detect it. Noise means variation between individuals: the more heights differ from person to person within each country, the larger the p-value for the same 3cm gap.The reverse confusion is just as common. In ordinary English, significant means large or important; in statistics, it means only that a result cleared the threshold. Finding differences between groups of people is more complicated than just calculating a p-value. J Studio / Getty Images Imagine a study of 40,000 people finds a drug raises IQ by half a point, with p < 0.001. That clears the “statistical significance” threshold easily, and yet half an IQ point is a completely unimportant difference. A smaller study finding an average gain of five IQ points at p = 0.06 is “not significant”, even though the effect, if real, would be far more meaningful.This is why a p-value has to be read alongside the effect size, which asks the question that usually matters most: how big was the difference?The temptation to p-hackBecause so much rides on landing below 0.05, the threshold creates a temptation. Suppose researchers testing a supplement recruit 50 people and get p = 0.08. Frustratingly close. So they add another ten and test again: p = 0.06. Closer still. Another ten, and it finally dips below 0.05, at which point they stop and write it up as though the sample size had been fixed all along.Peeking at the data and stopping the moment you cross the line inflates the chance of a “significant” result even when nothing real is going on. The same happens when researchers try many outcomes or subgroups and report only the ones that worked: run enough tests and something will cross 0.05 by chance alone. This is called p-hacking, and there is evidence researchers do it quite a bit.So should we bin p-values?Some think so. In 2019 more than 800 scientists signed a call to retire the language of statistical significance. At least one journal has banned p-values outright.Others argue for more weight on effect sizes and confidence intervals, which show how large an effect may be and how precisely it has been estimated.Emphasising effect size and confidence intervals seems wise. But binning p-values altogether may be going too far.A p-value tells us something useful: how often a result at least as extreme as ours would appear if there were no real effect. The problem is treating it as a verdict on whether a hypothesis is true, or whether a finding matters.Kyle Hewitt previously received funding through an Australian Government Research Training Program Scholarship administered by the University of Wollongong.