Academicia is suffering from a disease of uselessness where papers are written from a standpoint of getting credit. It’s not about other people understanding, reusing or learning how to use what they’ve found. Not to mention lacking in reproducability. Sadly this limits the usefulness of research for most. Yet for some reason we all still hold academia up on a pedestal
back
4 comments
People in academia always have a pressing career goal. Especially the best of them do not get bogged down by minor issues. I'm not talking about actual fraud here, just a bit of sloppiness that doesn't invalidate the results, but religiously adhering to meticulous standards is not profitable for researchers.
The primary goal is to publish the paper, go to conferences, meet others and network.
While it's not a dichotomy, who will be seen as "better"? Someone who spent tons of time to write reproducible clean code with tests, tracked down all minor details and pondered about things that may change the results by 0.1 percent, or someone who was a bit sloppy but got 3 publications in the meantime and got to know famous professors at conferences?
Attitudes like this determine your success. If nobody values or indeed even sees or knows about your efforts, those efforts are practically wasted. "I'm a really detail oriented person" doesn't have the same ring to it as another few papers on your CV.
I may sound cynical but the other extreme, idealism is not useful either.
> While it's not a dichotomy, who will be seen as "better"? Someone who spent tons of time to write reproducible clean code with tests, tracked down all minor details and pondered about things that may change the results by 0.1 percent, or someone who was a bit sloppy but got 3 publications in the meantime and got to know famous professors at conferences?
This is a common narrative, but I'm not certain if the example you give is typical. I've written about this before: https://news.ycombinator.com/item?id=18743531
Since 2018 I have tried to publish at least one paper debunking something every year. So far, if anything, being more careful has led me to publish more, not less. Admittedly, I don't think everyone should be as careful as I am as people would quickly run into diminishing returns, but the idea that being careful necessarily means publishing less has not been true in my experience.
Also, I don't think the typical case is something changing the results by 0.1%. In my experience when I catch an error it's usually larger than that. The first debunking paper I published was about something that was one or two orders of magnitude off the correct value in the typical case, but still received 300 citations...
Ultimately I think it would be best if an academic field adopts uniform standards for publication. That way, a sloppy researcher can't pump out 3 bad papers. Many academic communities have nominal standards that are not enforced. Some examples are discussed here: https://www.osti.gov/biblio/1141709
It often feels like a quixotic fight. Strong claims sell papers, but are hard to stand behind for a researcher with high standards, who is very critical of overall practices in a field. Their "best" recourse in the idealistic sense would be to publish an overarching meta analysis of the field, and pointing out problematic practices, gaining enemies and being labeled as a crusader.
In my field, computer vision, a major problem is uncomparable setups, far reaching claims etc. A major pattern is introducing some fancy model, showing that it gets a bit better results that could well be due to noise or some tuning on the test set (not even secretly, just trying many different things and putting the bold formatting on your best number and claiming it as SOTA). Whether it was really due to your fancy model or not is hard to prove. But my experience is that mundane, pedestrian changes in a model introduce way bigger performance changes than whether or not you use the fancy module or architectural tweak that is proposed in mediocre published works. Which also means you can absolutely not compare models created by different people, especially when the performance gap is small. Tiny details can screw up the whole ordering, so trying to interpret the order and draw conclusions from it is equivalent to reading tea leaves.
Now surely this applies mostly to mediocre research. But that is the majority. Top of the line research is more solid, but if you're a small unknown researcher, your best strategy is to start getting out there, publish and network.
This is not about code, but look at figure 4 here
[1] https://www.pnas.org/content/115/50/E11790
and tell me why i should ever trust something from someone who swaps the location of towns on a map, while the point of said paper is about the spreading of the plague along trade routes.
Instantly invalidated...(forever!)
Because people will make mistakes. If you expect papers to be 100% accurate, you'll always be disappointed. You have to account for possible mistakes. Whether they're accidental or misleading on purpose is to a degree irrelevant at first - we should always watch out for them.
Even in situations where mistakes cost lives, we put multiple failsafes and reviews and clear instructions in place - and still say they reduce the likelihood, rather than prevent accidents.
You know...
I mailed them, with CC to all involved. No reaction at all. Hamburg and Lübeck still swapped. What should i make of that?
Given the paper already appears to have a correction for something minor, one more would seem appropriate, yes - it’s a pretty jarring error.
But it’s just disingenuous to say it affects (let alone “invalidates”) the actual study. And shouldn’t a proper pedant spot that the top pin isn’t even where Lübeck is - that’s Rostock.
Good spot.
Does the swap in the position of the two cities compromise the findings in any way? If not, it's an irrelevant mistake like misspelling a word. What do you expect them to do? retract the paper because they misspelled a word?
The brokenness of academia starts with the exploitation of graduate students by educational institutions, which makes it a very lucrative enterprise to over-hire professors, each of whom is like an incubator for the school, which acts as an investor because they get a cut (upper five figures per year per student) no matter what. It's similar to college sports in a way. It's all driven by money.
Most research is useless. Most professors are unneeded given the size of the problem space. Students see this and in the end, besides the very few true geniuses, it's not the most purely motivated that become professors, but the ones who most aggressively play the game in trumping up their results.
> Most research is useless.
I agree.
> Most professors are unneeded given the size of the problem space.
I disagree.
First, most subjects are too deep for the professor to be knowledgable about more than just their specialty (Do you expect a single Computer Science teacher to have complete and up-to-the-date knowledge of formal methods, programming languages, and operating system virtualization?) Even then, they will either be out of date or have spent a lot of time keeping up-to-date.
The problem is in many cases, the size of the problem space is much too big for a single group of researchers to have any effect on it. Many 'simple' studies have thousands of factors that can affect the outcome, and almost all of them have too few people, with not enough time and energy to devote to isolating all of those factors. For that reason (and many others), most studies are not replicatable, and dubious at best.
This is the main reason why psychological, sociological, medical, and most other fields of research that aren't mathematical, are considered dubious. Not because their methods are inherently bad, or their fields inherently invalid, but because most studies do not have the manpower available to do a completely formally-correct ideal study, so they have to make-do with what they have, and trust that eventually we will have enough mediocre-evidence studies that together account for enough varying factors on the subject that we can eventually account and iron out the individual flaws through statistical methods.
If research were _truly_ a priority for humanity, and we really dumped all of our effort into scientific research as a society (i.e. governments and companies both prioritized R&D and gave the scientists enough resources to actually do the jobs properly), then we might see these fields as "hard science" rather than "soft science".
But that's like saying, if Jeff Bezos got up and actually objectively used his money properly, he would have billions left over and there would not be starvation or poverty in the modern world. It's an idealistic scenario that is extremely unlikely to happen.
>>> Most professors are unneeded given the size of the problem space.
>most subjects are too deep for the professor to be knowledgable about more than just their specialty
>in many cases, the size of the problem space is much too big for a single group of researchers to have any effect on it.
>most studies do not have the manpower available to do a completely formally-correct ideal study
I agree that the size of the problem space requires a lot of researchers. This is probably why there's so much attention focused on machine learning and big data, since they seem to have the potential to address the problems you've listed here. Of course, there are many technical and ethical issues to be confronted in developing and deploying them.
>If research were _truly_ a priority for humanity, and we really dumped all of our effort into scientific research as a society (i.e. governments and companies both prioritized R&D and gave the scientists enough resources to actually do the jobs properly)
It's not, because the greatest problem for humanity is still subsistence, which requires solving a massive resource distribution problem. There are some governments and some companies that do prioritise R&D and provide enough resources, but they are too few and far between.
>psychological, sociological, medical, and most other fields of research that aren't mathematical, are considered dubious.
>then we might see these fields as "hard science" rather than "soft science".
It seems to me that these fields are "soft" in part because the ethical issues surrounding the surveillance that's needed to collect the data to do a formally correct study are quite formidable.
You can view an undergraduate education (and especially a graduate education) as a significantly negatively paid (or at best unpaid) internship for the job of academia.
This has all the advantages of internships that employer's normally enjoy. The internship enables you to hire significantly better employees than you'd otherwise be able to get in the open market, because you can engage in more vetting and because of the power of defaults.
Many smart people go into academia who would have been much happier and more productive outside of it simply because going to school itself made pursuing a job in academia something much more of a default than it would have been for many people.
You should be able to contribute to these software projects, and receive credit akin to publishing a new result. Authors and contributors (and bug-fixers) should receive more credit for the subsequent results derived from their software. This aligns incentives more properly: The researchers struggling to gain insight using tools can trust the tools, and they are incentivized to make their new tools useful to the community, rather than publish-and-forget.
If you are a scientist and a programmer, you could make a software useful for others, publish an article about it, and provide it cost-free under the condition that when people use it for research they have to cite the article in their paper. This is how you could get lots of citations.
Many of the best-used robotics packages are handled in this way. The ROS paper itself has thousands of citations.
But it might be tough to get tenure doing that, and few jobs will support those efforts outside robotics startups or research labs, and often proprietary is the law of the land in those domains.
And outside tech R and D, where software still rules but isn't a core competency, you'll likely never have success with this model, sadly
Pop science coverage is always way overrated as a valuable exercise as well. So easy to cherry pick studies or fundamentally misunderstand the concepts or context.