back

by m-i-l·2y ago·view on hn ↗
> ... a project called "Nutch" would allow web users to crawl the web themselves. Perhaps that promise is similar to the promises being made about "AI" today. The project did not turn out to be used in the way it was predicted (marketed), or even used by web users at all.

Actually Nutch is used to produce the Common Crawl[0] and 60% of GPT-3's training data was Common Crawl[1], so in a way it is being used by a lot of web users today.

I did very briefly look into the possibility of self-hosting a search based on the Common Crawl data, but the latest size is 390Tb, and given the largest consumer drives are around 10-14Tb, that would be a lot of drives and a lot of money. Plus by all accounts most of Common Crawl is poor quality spam type of content, which OpenAI, Google etc. try to filter out before using it to train their LLMs. That's why I don't think the solution is a search engine for the whole internet any more, but one which just indexes the relatively small minority of good bits.

[0] https://commoncrawl.org/

[1] https://en.wikipedia.org/wiki/GPT-3

1 comments
Common Crawl's data is a bit inflated due to how it's stored. WARC is the opposite of a compression format ;-)

I think you are correct in that one of the primary problems faced in search is selecting interesting documents. I think I throw out like 75% of what I crawl because it's not very likely to ever be a good search result. You can probably filter it even more aggressively.

It's a smart avenue to approach search not in the sense of "how can I make this computer cluster big enough to fit the data" but "how can I make this data small enough to fit my computer"..