back
9 comments
It's probably time to sunset peer review. It currently functions to keep outsiders and dissidents from publishing in a field.

I would rather see some sort of prediction market where scientists make claims betting against one another, and 3rd parties are incentivized to conduct experiments based on the total value at stake.

No offense, but your solution reads like a typical idea that Sillicon Valley startup could come up with, i.e., throwing technology at a wrong problem, ignoring human factor and applying monetization where it doesn't belong. It reminds me of this founder who wanted to replace hiring processes with people putting money on the person staying in the firm long enough.

Not all science is based on experiments. Furthermore, science is not usually done in two simple steps: creating a theory and validating it. In practice, it's a cycle of refirenement that often requires dozens of iterations and involves discovering new phenomena.

In your model, who decides if the 3rd party has invalidated the claim or simply conducted the experiment incorrectly? Everyone involved has a financial motivation, so there's no independent side to resolve this issue.

Perfectly said. Before attempting to solve a problem, it's important to consider the lenses/perspectives and that facts that you'll see nails everywhere with your hammers.

Peer review can be more structured, is already blinded in many cases, and could benefit from regular crowdsourced suggestions from a variety of scholars. In many cases, journals have begun many reform practices like valuing open sharing of materials and code, structured transparency checklists, and opportunities to publish null results. All wonderful advances that don't require hammers.

If we had a good way to pay people based on the value they add (basically solved for sales, but not for any other role), then you could construct a hiring process by betting on that value, and the most predictive interviewers would be rewarded. But until that's solved, I don't think there is a foundation for the betting. Time at the firm is the wrong metric.

> Not all science is based on experiments.

Okay, well I'm not talking about that kind of science. I'm talking about the science where the value of a theory is measured in it's ability to make testable predictions. I'm not really sure how the the other kind of scientists resolve disagreements.

I didn't claim to have solved the problem. I was outlining what I think a solution would look like, and contrasting that to the idea of peer review as the way to determine what is scientifically relevant, if we could only make it more fair.

I am curious: what is your background and experience? Please don’t take this as hostile as it’s not intended to be, I am honestly curious.
As a former faculty member who now directs science in industry. Agree.
On arxiv, of all places ;-)
To be fair though, it's not like the other places have a suitable ``meta track'' to discuss such issues...
This paper has been online at least since 2018.

It has since been published: https://proceedings.mlr.press/v98/stelmakh19a.html