Item — HN Reader

I think a lot of studies are actually illegitimate. I think scientists all admit this, which is why peer-review, disclosing conflicts of interest, sharing your data, reporting all your measures BEFORE you collect data, not lying with statistics, etc are all being asked for (and often not done). This is why scientists often weight for meta-studies and replication before trusting any finding.

Laymen also correctly have an intuition that the people doing these studies aren't entirely trustworthy. What they don't have is a clear picture of how much work goes into these studies, who's doing it, what their motivations are, etc.

In my opinion studies when they can, should record videos of all data and make it publicly available online. Watching somebody do 1,000 hours of research is more proof-of-work to lay-people than some semi-coherent summary-for-a-layperson article.

2 comments

Laymen have been trained to be reflexively distrustful of everything because there's more profit and power in weaponizing fear and cynicism.

When a political party controls the science, you end up with say, Trump pushing one set of results, and Biden pushing another. It then becomes either pick the science that agrees with your politics, or throw up your hands in frustration. The average reader probably won't be able to dig into the fundamentals of the research and pull out the salient results, nor are they guaranteed it isn't policy pushed through overstated claims. It really undermines good science. It also falls back on the researchers who push science based on politics as well, so it isn't just the politicians.

That’s all true but it’s orthogonal to what i think you are responding to IMO.

I don’t think that studies are bunk because of corporate money, I think they are bunk because of how many studies I read. I am not a very fun person so when I see news reports about studies, I try to look them up. I find it more peaceful than memes or celebrity events. Think “coffee is good/bad for you again” style studies we read about daily.

These studies always suck. Ok, maybe 90% if I’m not being hyperbolic. It’s small sample sizes sure, but it’s also faulty logic, unsupported claims from evidence, lack of looking for alternatives, lack of ruling out confounding factors. And don’t get me started on soft-science an arts theses, I don’t have time.

I know that science moves forward mostly in millimeters, and I would agree that we have more and better scientific knowledge now than we have had in the past. But it certainly isn’t for the amount of publish-or-perish, p-hacking, storytelling, or outright fabrication.

In my experience the lay people aren't getting a great idea of the state of the field. Like you say coffee good/bad studies. It isn't that simple. People test all sorts of things in different contexts. But maybe some are genuinely bad studies. You don't know though because science journalism is so crappy. They don't care about the merits of the study. They go "people drink coffee, maybe that would drive engagement."

The most interesting papers are not going to get popular press releases because they are so many steps removed from the context that lay people understand. They can understand "coffee good/bad." They can't understand anything about the stories we are actually telling at the bleeding edge of a field, because even our undergrads working in our labs on these projects can scarcely understand them. Second year grad students struggle to understand them. How can a science journalist who only has a bs from communications department, or the lay public, possibly understand?

So, they don't reach for those papers when they seek to write articles for engagement. And the lay public doesn't learn the state of the art, and assumes the worst of the field from what they do read about.