back

by denzil_correa·6y ago·view on hn ↗
> My point is that unless it's re-implemented and verified (which obviously it isn't) as part of reviewing the paper, that's not the same at all.

Even if it is re-implemented and verified it won't be the same. There are a million possible conditions for "re-implementation". Where would you draw the line? How many "re-implementations" and "types of conditions" do you need to be sure?

You probably need to have expectation mismanagement from a scientific paper. A scientific article will only say "We tried this idea under conditions X,Y,Z and it works with A,B,C metrics and I, J, K assumptions". That is the crux of ANY empirical science - Computer Science or otherwise. That's all that they are paid (and incentivised) for. A computational scientific experiment is not a software product. If you want more, you need to pour in more funding and give them more resources explicitly for those purposes. Either that, or if you want to be 100% verified - you can choose to read theoretical papers where mathematical proofs are "verification".

1 comments
Sure, but:

> "We tried this idea under conditions X,Y,Z and it works with A,B,C metrics and I, J, K assumptions".

Is what I'm calling the assumption (the whole sentence). The implementation is essentially the methodology, and IMO therefore needs to be included.

Note I'm arguing against 'academic, paper-supporting, code does or should not need to be open sourced', I'm not saying it needs to be 'software product' quality, written in the same way, to the same expectations, packaged, or anything like that. Just available for someone to say 'wait a second, you didn't try it under conditions, X,Y,Z, because Y gets negated here', or whatever.