back

by matheusmoreira·9y ago·view on hn ↗
I keep wondering why submitting papers to these journals is necessary. Does the research not stand on its own?

> I feel in those disciplines researchers are a big happy family

I feel the same way. ArXiv accepts submissions from any registered researcher¹ and it seems so much easier.

¹ https://arxiv.org/help/submit

1 comments
> Does the research not stand on its own?

Academic mathematician here. It does, sort of, but .....

Say a mathematics postdoc is applying for a job, and she works in underwater basketweaving. The two underwater basketweavers in her dream department read her application and can see that she's brilliant. She's developed a new methodology for weaving rattan under 100 feet of ocean without even getting it wet. She has letters from the most preeminent underwater basketweavers in her field. She should be a shoo-in to get the job.

One problem though. A majority of the hiring committee works in different areas of research. One thinks that underwater basketweaving is overhyped. Another thinks that great work was done in the eighties, but all the interesting questions have been solved already. And these people don't put much stock in the letter writers: they have a reputation for writing glowing letters even for mediocre candidates.

And, so far, the hiring meeting is deadlocked until the basketweavers pipe in: Our candidate has published in Journal für die reine und angewandte Mathematik and Inventiones Mathematicae and Publications Mathématiques de l'IHÉS.

And now, all of a sudden, that postdoc is getting the interview, because her competitors have only published in Mathematische Annalen and Indagationes Mathematicae. (These are all real names of math journals.)

That was a bit long winded, and please forgive me if it's a bit silly, but the point is that we need mechanisms to compare researchers working in different subdisciplines, for researchers to "prove" to the community at large that they're doing good work. I wish we could eschew these games, and I certainly wish we could ditch the commercial publishing industry, but we compete with each other for jobs, for grant funding, and for fame, and we need to establish some kind of metric for evaluating researchers which is as fair as possible.

It is not perfect; certainly it is not completely fair. But I believe it works reasonably well.

And that's in countries with relatively reasonable hiring practices, like the US.

In Spain it's even worse: for positions at universities there are so-called "objective" scales of assessment, so that a paper scores a number of points depending on the position of the journal it's published in in the ISI JCR journal ranking. Points are added and the top scoring candidate wins.

For hiring postdocs in projects, the PI typically has some more leeway to set the assessment scales, but not too much. As a PI who is hiring right now, I can use an interview or personal assessment of candidates' work to tip the scales a bit (at my place up to 15% of the grade or so, little more than a tiebreaker) but if I get a candidate with significantly more JCR-indexed papers than another, I basically have to hire them even if I think their track record is worse (which is often the case as one can get lots of "points" by submitting tons of very similar papers on an easy-to-publish area). Because doing otherwise is, you know, not "objective".

So here, basically, if you publish in a discipline where JCR-indexed journals are dominated by the likes of Elsevier, you absolutely need to publish there or not get hired, period. When people in my field from more fortunate countries say things like "let's just ditch the journals and publish everything in arXiv", I would agree with them in an ideal world, but in my world they would push my whole country out of the field...

Well, you're talking of a publicly-owned university, and the perception of "mamoneo" and "amiguismo" funded by taxpayer money is, in my opinion, already high enough that I welcome anything that tries to make the hiring system more objective.

Mind you: The hiring policy at Google is that managers don't get to interview the potential hires they will manage, nor have any say in the hiring process. For technical positions, five engineers interview the candidate independently, and submit their feedback and recommendation to a hiring committee that makes the decision. The process tries to be as objective as possible, and the manager has no leeway at all.

That doesn't mean the Spanish system is any good, just that I don't think giving the PI more say is a great solution. The points assigned to "easy-to-publish areas" can be reduced if PIs give that as feedback. A way to evaluate a candidate without the numerical scale silliness could be devised too.

Yes, that's a very good point.

What I am against is the rigid numerical scales. There are always ways to game them (even if easy-to-publish areas can be flagged, which would be politically difficult to do, there is the problem of people who manage to publish almost identical work in several journals, the problem of ghost authorship, etc.) I believe it's basically impossible to judge a researcher's work without actually reading it.

That said, indeed many PIs can't be trusted to make decisions based on meritocracy, as there is the perception you mention and it's often justified. Probably the CVs should be evaluated or at least checked by someone different from the PI (for example by experts in the field from a different country, I think in Italy they do that kind of evaluation for some projects and grants). I just expressed some frustration as PI because I may be forced to hire a suboptimal candidate due to the rigidness of the scales, but what I want is not necessarily more autonomy for hiring, but simply any solution (like external expert evaluation) that will (1) give me the best candidate, and (2) not support the oligopoly of the likes of Elsevier like the Spanish system does.

> we need mechanisms to compare researchers working in different subdisciplines, for researchers to "prove" to the community at large that they're doing good work.

But the acceptance into those journals is decided by editors, with the same thought process of the hiring committee. What makes their decision any more valuable?

Each journal focuses on a specific subfield of math/CS/$SUBJECT, so they can make more informed evaluations on the quality of the research than an expert in some other field can.
OK, but then why don't the editorial board of these journals just get together to do reviews and jettison the ridiculously high overhead of the publishers, and the concomitant games with copyrights and paywalls needed to sustain that overhead?

I mean, seriously, it's 2016. When was the last time you read a paper printed on actual paper?

When was the last time you read a paper printed on actual paper?

I know it's not what you meant, but I almost always print out papers before reading them.

Yes, I should have said: when was the last time you read a paper in a bound journal printed by a publisher?
Sorry but that's not the point GP was making:

> One thinks that underwater basketweaving is overhyped. Another thinks that great work was done in the eighties, but all the interesting questions have been solved already.

Then he cites three general Math journals as what the hiring committee can use to rank the candidates, not a journal specialized in underwater basket-weaving.

Do those sorts of differences in early publishing predict later success—say, in attracting grants or top grad students, when you adjust for the benefits of being at a top school? If not, maybe it doesn't matter which is hired, as with the undergraduates so often told that a whole admitted class could be set aside, the next N taken, and the world would go on.
> Do those sorts of differences in early publishing predict later success?

Absolutely yes. They are perhaps the most reliable yardstick for past success that we have available, and past success is the best predictor of future success.

Citation? Also, it's worth pointing out that the question was not "do they predict success", but "do they predict success controlling for pedigree". Ive been thinking for a while on how you might actually measure the effect of pedigree, beyond the identity of university. I believe that the specific research group matters, but that's a lot harder to prove (smaller N, harder to get reliable data about counterfacuals).
> it's worth pointing out that the question was not "do they predict success", but "do they predict success controlling for pedigree"

This is a valid social question. As a hirer, however, you are measured by your success, not your success benchmarked to some other thing.

  That was a bit long winded, and please forgive me if it's a bit silly
It was neither. For somebody not working in science it was a fascinating and very illustrative summary and represents very much one of the reasons why I'm here.

Thanks!

I feel like one step for fixing this would be a pledge by hiring committee members. Say the university of X thinks publisher Y is a burden on the scientific community and as such will ignore everything the applicants have published there (or even weight it negatively). It's never going to happen but these issues need to be attacked from all angles that affect the careers of scientists. Employers, publishers, peers and students.
I strongly disagree. You can't penalize young scientists for playing the game; they pretty much have to.

I believe it is much more appropriate to put pressure on senior, tenured scientists to publish their work in "ethical" venues. It's more challenging to pick a fight with people in positions of power, and of course they can simply ignore you, but their careers aren't going to suffer nearly as much if they avoid some of the big-name journals.

I honestly think that it would only take one or two high class universities to boycott the large publishing corps to bring the whole house of cards down. If MIT and Stanford (for instance) made it a policy to publicly recommend and prefer arxiv etc then it would set a trend that would be emulated by the rest of the universities.

It takes some high profile researchers in each field to start the ball rolling. They are the people whose work others will track down wherever it's published.