Item — HN Reader

"However, the obtained effect sizes were not large enough to be statistically significant, and our study highlighted the need for more research around what performance thresholds indicate a meaningful increase in risk."

"We also discuss the limitations of statistical significance as an effective method of measuring model risk"

Seriously?

2 comments

It’s not a null result if the PR person writes the paper. It’s the fundamental mathematical nature of statistics that’s wrong!

I understand the second sentence but the first is flawed. Effects can be statistically significant at any size.

They can, but at any given sample size, there is a minimum effect size to achieve statistical significance. Larger effect sizes are always more significant, and smaller effect sizes are always less significant.

So if you assume they wrote the paper after doing their work, and not before, the sentence makes perfect sense: the work is already done, there is an effect size cutoff for statistical significance, and they didn't reach it.

One of Andrew Gelman's frequently-mentioned points is that a statistical significance filter in publishing means that published effect sizes are almost always wildly overestimated, precisely due to this effect.