back

by zeristor·9y ago·view on hn ↗
Hmmm.

Software Engineering has Continuous Integration, since it is so expensive to fix software later in the day.

Is there any such thing as Continuous Reproducibility?

Constantly checking that the science can be reproduced?

How prevalent is this in different branches of Science?

7 comments
In applied mathematics, the idea of having a standard platform for releasing numerical experiments and standard datasets have come and gone over the years. My advisor said that in the early 2000s, there was a push in some areas to standardize around Java applets for this in a few journals, but never really took hold. Nowadays I would think some form of VM or container technology could probably do the trick while avoiding configuration hell. Commercial licensing for things like MATLAB or COMSOL etc. would be the real challenge for totally open validation in a lot of disciplines. Proprietary software is way more prevalent in scientific and engineering disciplines than I think many general software developers realize.

The good news is that you can't really fake proofs or formal analysis. But the truth is, many folks in the area do cherry pick use case examples/numerical validation as much as you see in other disciplines. Perverse incentives to publish, publish, publish while the tenure clock is ticking keeps this trend going I think.

I doubt we have any container technology today that anyone would want to use in 10 years, just like nobody would want to use a Java applet today.

And I say this having used Docker myself to make one piece of computational research reproducible. I'm not sure it helped in the end. I do encounter people who want to reproduce it, and mostly I have to teach them how to use Docker and then apologize for all the ways it goes wrong.

>The good news is that you can't really fake proofs or formal analysis.

It's my understanding that most published mathematical proofs aren't "hey look at this theorem in first order logic that we reduced to symbol manipulation"; rather, they present enough evidence that other mathematicians are convinced that such a proof could be constructed.

Is that incorrect?

I'm not a mathematician, but I had a math prof tell me that most publications only contain proof sketches, not full proofs.

Can someone in the field comment?

I work as an applied mathematician. In general, I would say that this is incorrect. Virtually all of the papers that I read I would contend have full proofs. That said, I can sympathize with the sentiment in a certain sense.

Just because a paper contains a proof doesn't mean that the proof is correct nor that it's comprehensible. Further, even if a paper went through peer review, it doesn't mean that it was actually reviewed. I'll break each of these down.

First, a proof is just an argument that an assertion is true or false. Just like with day to day language, there are good arguments and bad arguments. Theoretically, math contains an agreed upon set of notation and norms to make its language more precise, but most people don't abide by this. Simply, very, very few papers use the kind of notation that's read by proof assistant tools like Coq. This is the kind of metalanguage really required for that precise. Now, on top of the good and bad argument contention, I would also argue that there's a kind of culture and arrogance associated with how the community writes proofs. Some years back, I had a coauthor screaming at me in his office because I insisted that every line in a sequence of algebraic reductions remain in the paper with labels. His contention was that it was condescending to him and the readers to have these reductions. My contention was that I, as the author of the proof, couldn't figure out what was going on without them and if I couldn't figure it out with all those details that I sincerely doubt the readers could either. Around the office, there was a fair amount of support for my coauthor and removing details of the proof. This gives an idea of the kind of people in the community. For the record, the reductions remained in the submitted and published paper. Now, say we removed all of these steps. Did we still have a full proof? Technically yes, but I would call it hateful because it would require a hateful amount of work by the readers to figure out what was going on.

Second, peer review is tricky and often incredibly biased. Every math journal I've seen asks the authors to submit to a single blind review meaning that the authors don't know their reviewers, but the reviewers know the authors. If you are well known and well liked in the field, you will receive the benefit of the doubt if not a complete pass on submitted work. I've seen editors call and scream at reviewers who gave "famous" people bad reviews. I feel like I was blacklisted from one community because I rejected a paper from another "famous" person who tried to republish one of their previous papers almost verbatim. In short, there's a huge amount of politics that goes into the review process. Further, depending on the journal, sometimes papers are not reviewed at all. Sometimes, when you see the words "communicated by so-and-so" it means that so-and-so vouched for the authenticity of the paper, so it was immediately accepted for publication without review. Again, it varies and this is not universal, but it exists.

What can be done? I think two things could be done immediately and would have a positive affect. First, all reviews should be double blind, including to the editor. Meaning, there is absolutely no good reason why the editor or the reviewers should know who wrote the paper. Yes, they may be able to figure it out, but beyond that names should be stripped prior to review and readded only at publication. Second, arbitrary page limits should be removed. No, we don't need rambling papers. If a paper is rambling it should be rejected as rambling. However, it removes one incentive to produce difficult to follow proofs since now all details can remain. Virtually all papers are published electronically. Page counts don't matter.

In the long run, I support the continued development of proof assistant tools like Coq and Isabelle. At the moment, I find them incredibly difficult to use and I have no idea how I'd use them to prove anything in my field, but someday that may change. At that point, we can remove much of the imprecision that reviewers introduce into the process.

Thanks for the insight!
The problem is that many branches of science don't have any immediate pressure to produce something that is usable by people outside the field, and sometimes not even peers. So they do what is needed to get out papers, and bothering about eliminating false results goes against that interest.

I'm going to bet that in competitive research branches, for practical applications, that have objectively verifiable results, most studies will, in fact, be reproducible.

In the physical sciences experiments can be enormously expensive to run and doing them "continuously" is impractical. Groundbreaking work is usually verified independently, but it varies across fields. E.g. physics is usually quite careful about reproducing new physics before accepting it, while in the biological sciences it seems that work isn't always reproduced.
> Is there any such thing as Continuous Reproducibility?

Only in principle. Most studies are episodic -- they have a beginning, an end, and publication of results only after the end. For continuous replication to exist, laboratories would have to be much more open about what they're working on, less competitive as to ideas and funding than they actually are.

> How prevalent is this in different branches of Science?

It's nonexistent everywhere but in "big" physics, where (because of the large number of people involved) people tend to know in advance what's being worked on. But I'm only saying physics opens the possibility for continuous replication, not that it actually exists, mostly because of cost.

Well often in behavioral sciences, the alternative hypothesis in experiment A is the null hypothesis in the follow-up, experiment B. If the results in alternative A do not act like the results in null in experiment B, then there is a failed replication. The researcher observing this, begins to lose confidence in the earlier finding, and pursues other ideas. This is the kind of implicit replications currently supported by funding agencies.
I completely agree with applying software engineering skills to this. Not just CI for deployment, but also open-source collaboration. I see research and development in other fields as largely like closed-source silos, and a centralized but evolving ecosystem for information collaboration in any topic is in the works.

I know that is definitely one of my driving visions for ibGib, and I can't be the only one. Open source, open data, open collaboration.

>Constantly checking that the science can be reproduced?

Scientific experiments usually need actual things to be manipulated in the real world. So I think a concept of Continuous Reproducibility may only applicable to a subset of science that can be done by robots given declarative instructions.