Nice. Hope somebody makes a torrent of it/ hosts it in a way that it can't be taken down.
Also, what are some estimates of how many tokens of text are out there? Seems like we are hitting that number pretty quick?
back
2 comments
> Seems like we are hitting that number pretty quick?
I don't think we're even close. Libgen's nonfiction archive alone is over 32 terabytes. Total size last year was over 120 terabytes. Between that, SciHub, and the internet, there's probably orders of magnitude more tokens out there.
I don't know about orders of magnitude left but we're definitely not close yet. This is just 5 languages(and frankly not even the 5 with the most text) and just as importantly, just what is crawlable from the web. There's tons of stuff in popular ebook archives you can't crawl from the web.
This is also relatively code/scientific corpora scant.
We're just getting started.