▲ 18 points

Ask HN: Is there any decent API to download a paper given its name?

by hexomancer·4y ago·13 comments·view on hn ↗

I am developing a PDF viewer designed for reading research papers[1]. One very useful feature that I would like to add is the ability to directly download and open a paper just by clicking on its name in the PDF file. I have implemented a version of this using PyPaperBot [2] which is not bad, but it is not as fast as I would like it to be (it uses scihub which doesn't have the best servers).

By parsing the html from google scholar, I wrote a python script that does this which was very fast and was working perfectly, however, after using it for a couple of minutes (maybe about 10-15 requests), I can no longer query google scholar using python requests (the returned html is a request for captcha). It appears that google disallows any programmatic use of google scholar (even though this was not spammy at all, the user has to manually click on a paper to send a request to google scholar).

Anyway I was wondering if there is any decent and free API to get the url of a paper given its name, I have found a couple of paid ones but they are way too expensive.

[1] https://sioyek.info

[2] https://github.com/ahrm/sioyek-python-extensions#-paper_downloader

13 comments

> it uses scihub which doesn't have the best servers

you could always download the scihub backup torrents from libgen and host them yourself somewhere. its probably like 100TB of data by now though so this isn't really a cheap approach.

Looks like OP is only interested in a mapping of Title->SciHubURL. In essence an index only, rather than the full contents of every paper. Maintaining a separate index would require significantly less storage than mirroring all the papers

Nice! I would easily use your tool for the next 2 years (my research deadline). I have a question tho. How would you bypass the fact that you need to be under some university vpn or special login to download some articles?

PyPaperBot uses scihub to download papers so there is not problem for that.

As for the google scholar, I don't care about non-open access articles. I am content with a download link only for the open-access ones.

One thing you could try is downloading the papers before the user requests them so that they will be ready instantly for a user's request.

Example: User opens paper on page 1. Page 1 has citations to 3 other papers. Your tool instantly begins downloading the other 3 papers. User goes on to Page 2 which cites 1 other paper. You begin downloading the new paper. User clicks on the citation and your tool now has the linked paper already downloaded and ready to open.

Might be a bit of an imposition on sci-hub though.

Yes, I have thought of that but that would use a lot of users' bandwidth and also as you said puts unnecessary pressure on scihub servers.

Not sure off the top of my head but it's possible that crossref DOI lookup has an API you can use for this.

https://openapc.github.io/general/openapc/2018/01/29/doi-rev...

I do use crossref for bibtex. It is not useful for getting a download link (as far as I know).

Once you have a DOI (from crossref or otherwise) you can get a legal download link simply by putting 'https://doi.org/' in front of it (though it will often go to a summary/login page rather than the PDF) and a SciHub PDF link by prepending 'https://sci-hub.ru/'. Or am I missing something?

I already use scihub (I explained in the post). However, it is much much slower than google scholar.

Then 'https://scholar.google.com/scholar?q=<doi>', where <doi> is URL escaped?

It seems to me that Crossref can give you the DOI, and once you've got that then getting to article text is pretty trivial. I must be missing something.

Google blocks you once you use python to request it multiple times (I explained in the post).

Ah sorry. I guess I don't understand what you are looking for.