Item — HN Reader

Why exactly? We could also probably require a website to test the application, a mobile app on all platforms and then a native OS apps on all platforms. I apologise for being facetious but writing code is a serious, specialized task. It should be left to the people who specialise in it, scientists aren't the specialists for it.

I think it's important to understand that we only require information to generate algorithms written in the paper. As such, assumptions and limitations should be listed enough to generate the material in the paper. One way to do so is code and data, it is certainly not the only way.

2 comments

I empathise with what you say, but it's also true that maybe things could be better.

We don't expect universities to build their own laboratories and halls of residence from first principles - they will call on property developers, architects, and contractors. It would be good if there were more IT services, built by specialists, that academics could tap when required - so they would spend more time advancing their fields of expertise and less time writing mediocre code. Things like PythonAnywhere / Notebooks, Lambda etc are a step forward, but there is still ample space for advancement.

Getting software engineers interested in your science isn't easy. Though in this particular case there ought to have been enough funding. I'm not knowledgeable about its specific developments.

Do reviewers of such papers write their own implementations, and verify they get (roughly) the same results?

Not generally, no. Even if they had access to the code at the time of review, which is hardly ever the case in my experience, there's simply no time. I also don't see a point in doing so, most papers only describe minute changes to some state of the art thing that's well established so you trust in the academic integrity and experience of authors to get that part right and usually focus on parts that matter over things like the ones this issue brings up.

Even for real reproductions, that require work just like they do in any other scientific domain, the points in that issue seem pretty irrelevant. There's things like FAIR standards on reproducibility but as far as the status quo goes that repo doesn't look too bad. I could not care less if tests for some project are badly written, at least it's in written in a non-obscure language and shows a somewhat sane structure. What's next? Calling for redactions because somebody didn't follow the same tabs vs spaces paradigm?

There's nothing in that issue w.r.t. whatever scientific finding this was used for and this general phenomenon is really fascinating. Instead of a constructive discussion with the technical or scientific folk that put their work out there, or engagement with the politicians that drew conclusions based on those scientific findings, you get github issues and Twitter threads mixing a bunch of unrelated concerns.

I think the matter at hand is much less trivial than tabs v. spaces.

I’ve never heard of a reviewer writing 15K lines of code. That would be a serious undertaking for someone who is reviewing as an unpaid community service in their free time outside of doing their own research/teaching/etc.

Maybe a very dedicated reviewer might check a short simulation code that can be written in a few hours max, but even that is exceptionally rare in my experience. Reviewers usually look at the results and validation simulations in the manuscript, and if they look reasonable trust the code doesn’t have major bugs affecting accuracy.

Exactly, the person I replied to was saying there's no need to open source the code, it's enough for the paper to describe it. My point is that unless it's re-implemented and verified (which obviously it isn't) as part of reviewing the paper, that's not the same at all. It's like reviewing 'Assuming X, Y' and saying 'well I can't verify X, but yep, X->Y checks out. Y.'.

> My point is that unless it's re-implemented and verified (which obviously it isn't) as part of reviewing the paper, that's not the same at all.

Even if it is re-implemented and verified it won't be the same. There are a million possible conditions for "re-implementation". Where would you draw the line? How many "re-implementations" and "types of conditions" do you need to be sure?

You probably need to have expectation mismanagement from a scientific paper. A scientific article will only say "We tried this idea under conditions X,Y,Z and it works with A,B,C metrics and I, J, K assumptions". That is the crux of ANY empirical science - Computer Science or otherwise. That's all that they are paid (and incentivised) for. A computational scientific experiment is not a software product. If you want more, you need to pour in more funding and give them more resources explicitly for those purposes. Either that, or if you want to be 100% verified - you can choose to read theoretical papers where mathematical proofs are "verification".

Sure, but:

> "We tried this idea under conditions X,Y,Z and it works with A,B,C metrics and I, J, K assumptions".

Is what I'm calling the assumption (the whole sentence). The implementation is essentially the methodology, and IMO therefore needs to be included.

Note I'm arguing against 'academic, paper-supporting, code does or should not need to be open sourced', I'm not saying it needs to be 'software product' quality, written in the same way, to the same expectations, packaged, or anything like that. Just available for someone to say 'wait a second, you didn't try it under conditions, X,Y,Z, because Y gets negated here', or whatever.