Item — HN Reader

My guess is the estimate comes from term frequency index which is pretty easy to build. Estimates of that can come from HyperLogLog or similar.

Asking for page 10,000 is asking the search engine to search and _rank_ 10,000 * 10 results and give you the last one. That's very expensive and ultimately useless - search is about finding what you're looking for on page 1, not on page 10,000 :)

So it is true that there are 225k+ repositories using Golang (you can compute that with an index scan once a week), but searching them is an entirely different problem.

2 comments

If they cannot give you the page results how can you trust page 1 is better than page 34557

Huh? It's a ranked retrieval model. Each result has scored a little bit worse on their relevancy function than the one above it.

To not trust that the results on page 1 are better than those on page 34557 is the same as saying that their ranking function does not work at all, which would mean that it's at best as good as random chance. That's clearly not the case, therefore I can trust that page 1 indeed has more relevant results than page 34557.

With that said, Page 34557 doesn't exist. And that's fine. The result count estimate is not based on the actual ranking that has taken place (at least not directly). It would be an absolute waste of resources to rank that many results. If you cannot find what you're looking for on the first page, then it's much easier to reformulate your query. Easier for you because it gives you more control over what you want your search results to be and easier for google because it only needs to rank a couple hundred results instead of a bajillion.

Well, one could claim that indeed their ranking function does not work well - at least recently. Stuff that is relevant rarely is showing up on the first page as it is losing to various spam sites having articles written by AI so that it can match the typed search query and get a good ranking, but articles themselves being misleading and incorrect. I remember the days when you could really dig deep into results pages. Maybe not 34557, but above 100 you could get niche human written content on the interesting topic.

Yeah, I used to pretty frequently click through, and find relevant/useful content, some 20+ pages in on paginated web search results. Well, unfortunately the major search engines have changed their functionality such that there's usually no point in even looking beyond a couple pages (or, arguably, even looking at the first page for that matter).

If you search a famous name like Joe Biden or Donald Trump, it might be useful to read the 10,000th thing written about them, wouldn't it?

Like the others have said, Google wants you to refine your search (e.g. "Joe Biden foreign policy" instead of "Joe Biden"), rather than dumping low quality results on you. You don't have to agree with Google's rule by switching to other search engines, but if you do use Google, abide to that.

Ehhh, I write very specific search terms, with quoted elements, and I still get shockingly irrelevant results. I'd literally rather get a "no useful results" response rather than hundreds of pages of results that literally DO NOT CONTAIN the terms I typed into the search box.

If you put the search terms in quotation marks it will only show results where that term appears verbatim on the page. Also searching -keyword will exclude search results where that word is present.