Item — HN Reader

> The sample size is 22. According to this sample size calculator I found, a required sample size for just one thousand people would be 278

I'm all for criticizing a lack of scientific rigor, but this bit pretty clearly shows that the author knows even less about sample sizes than the GitHub guy, so it seems a bit pot calling the kettle black. You certainly don't need to sample more than 25% of any population in order to draw statistical information from it.

The bit about running the study multiple times also seems kinda random.

I'm sure this study of 22 people has a lot of room for criticism but this criticism seems more ranty than 'proper analysis' to me.

2 comments

> You certainly don't need to sample more than 25% of any population in order to draw statistical information from it.

Certainly? Now, who is ranting?

It’s a basic property of statistics. You need an extremely varied population to need a sample of 25%, and almost no human population is that varied in practice. Humans are actually very uniform in fact.

What if the group is 8 people

> The bit about running the study multiple times also seems kinda random.

Reproducibility? But knowing it comes from the CEO of Github, who has vested interests in that matter because AI is one of the things that will allow to maintain Github's position on the market (or increase revenue of their paid plans, once everyone is hooked on vibe coding etc.), anyone would anyway take it with a grain of salt. It's like studies funded by big pharma.