▲ 492 points
back
240 comments
Not to be intentionally contrarian, but I have thought a lot about this:
I think good fiction is actually the antithesis of AI, fundamentally. LLMs are really only capable of combining different things into output. This can give you some creative results, but it is ultimately kind of limited. I cannot imagine that prompting an AI a thousand times would give you something truly original in the way many high-quality fictional stories are. The magic and absurdity of life is a necessary thing to create such works.
> LLMs are really only capable of combining different things into output
> The magic and absurdity of life is a necessary thing to create such works.
These two points seem slightly conflicting. Why if a human is truly more creative than AI, must it experience "the magic and absurdity of life" if it can be creative without "combining things into output"? Is a creative human then just not splicing and recombining their experiences into their creative output? And therefore, if an AI had more data that mirrors that experience in some way would it be sufficiently creative to create something "truly original"?
My view is that intelligence and creativity is all just purely recombination. AI can do that well just as humans can. The gap is that the data humans have access to is much more analogue, emotional, and grounded in the real world compared to the data that AI is currently trained on.
"…truly original in the way many high-quality fictional stories are."
The older I get, the more "nothing new under the sun" returns again and again.
Agreed! I find AI to be pretty decent at documenting and explaining technical stuff, which is a subgenre of non-fiction.
I once tried telling it to write a fantasy novel for me because I didn't know what to read next. Gave it some examples of books I have enjoyed in the past so it would have an idea of my taste...God was that awful. Absolutely terrible idea.
AI is not coming for fantasy authors jobs any time soon!
Models are increasingly showing their ability to extrapolate into the human unexplored (math proofs being the most apparent). What gives you confidence the absurdity of life is uniquely difficult for models to source?
I have also thought about this a lot but have less developed views on fiction (short version: I don't currently want to read an LLM-written novel, but I expect that will change in the future and there will be some I enjoy)
But I want to say that non-fiction books are a lot more resistant to LLMs than you might think. I'm reading two non-fiction books currently. And both (https://www.goodreads.com/en/book/show/17801.Underground, https://www.goodreads.com/en/book/show/44824581-a-k-pop-live) could not have been written by an LLM. One is based on in-person interviews that were hard to set up, and the is based on a log of sources a number of which are going to live events.
Not every book fits this pattern of course. But even a more straight forward book (https://www.goodreads.com/series/139929-designers-dragons) will often rely on sources that are not easily accessed by an LLM, they may not be digitized, or they may not be accessible in your common digital collections.
I 100% think LLMs can do good synthesis work, don't really think we have the tooling/checking in place for them to write a competent non-fiction book yet, but that will certainly happen, but I still think the best and most interesting non-fiction remains firmly in the realm of humans.
(Also, a lot of non-fiction is memoirs, travelogues, etc. which LLM can't really replace unless you're dictating to it.)
While you are right that humans can probably find some original ideas that are not just combinations of inputs, the universe of things that humans will find interesting to read is not that vast, and that’s why there are only really like 7 types of stories, and those are the ones that survive through a process of natural selection.
Everything else is just not that interesting to a human.
LLMs are really only capable of combining different things into output
it's a real question. I've seen a lot of people stating that the human mind "is more than this" as if it was obvious
So now we'll have to investigate what "original" means?
From what I can see, AI is absolutely able to create new work. Things that are not in the dataset, but look like they could be. This is pretty much what generalising means, they can make a wide variety of things that aren't actually in the original set but can hide in that set pretty well.
When you are an original piece of culture, what do you mean by it?
Thank you for sharing this.
I do think book prizes are a better-than-average signal but I previously volunteered with a book award. I will caution that pretty much every publisher mass-submits these books for consideration in every remotely relevant prize. It is a cost of doing business (similar to how photographers pay to enter photography competitions to try and win the "award winning photographer" title, or businesses submit dossiers with consideration fees on why they're one of Michigan's top 100 places to work).
There are often so many books and so few willing qualified readers that which books get an award can either be completely arbitrary, or comically easy.
For example, the NCR Book Award in your book corpus faced a big scandal when it was revealed that the judges did not read the books themselves. [1] The PROSE award is so comically large that an ordinary category finalist or win is usually overstated in prestige and value.
[1] https://www.theguardian.com/news/2013/may/19/literary-prize-...
I would bet that how your brain stores information that you read from long-form text is very different from how it stores information you acquire from chatting with an LLM. When I read something challenging or new to me I spend a lot of time thinking about how what I'm reading matches my own experiences or knowledge. Although I'm a fairly fast reader, it often takes me a long time to get through difficult pages since I have to stop and think about what I'm reading. I seem to be doing a lot of integrating and reorganizing my thoughts. When interacting with LLMs it feels a lot more like I'm just receiving knowledge passively and I don't think it gets integrated as well. Not sure why this is and its somewhat counterintuitive since I don't think I'd have the same experience with a human tutor.
Thank you for this. I sorted the "technology" and "science" sections and saw a few excellent books that I've read and a few that I would really like to read. This motivates me to start setting aside "reading time" again every day, since I've lost that habit.
The recent "society & culture" books gave me some good book club ideas.
Bug report: filtering by "award" appears to be broken for some awards. If I select Pulitzer or National Book Award, no books show up, but I can find books with these awards by browsing.
Suggested additions: the Axiom Business Book Awards, Library Journal Best Books of the Year and Booklist Magazine's Editor's Choice Awards. There's also a few popular substacks that have big sideshows in regular book reviews, but I wouldn't even recommend my favourite for a public aggregator like this.
The article resonates, there is great serendipity in somewhat random exploration of books. But then the web app seems to be the antithesis of this? I clicked a few categories and the top ranked books all had the “popularity contest” smell, sort of the antithesis of random exploration.
The history section of the book price index is incredibly america-biased. History of america, american natives, japan after ww2 and vietnam in the last century.
At least one of the links on the post is still to localhost:3000 instead of the vercel page. I don't want to make a substack account to tell the author so hopefully this information finds its way over there.
Can I make a feature request? I would love to be able to search by author. I recently started Caro's LBJ series (it's amazing so far) and I'm curious how many awards he's won. Anywho, I love non-fiction so I plan to visit this website often. Thank you and great work!
Thanks for this!
I've been looking for a good book on the Spanish Civil War, in either Spanish or English. The one American book I was recommended by someone, I didn't like. I'd like to keep searching regardless of the book's language or what country it has won prizes in. Could you also include book awards from other non-english speaking countries?
On a related note, I was just noting to my co-founder, as we struggle to write good case studies for our website, that I find LLMs are astoundingly bad at writing good prose.
We all know the "AI-tics" that give away a sloppily AI-written piece, but even if you steer them, they still struggle to write consistently high-quality prose.
Somehow I feel that the work of a good copywriter has never been more noticeable.
Worth stating that this is one of the success stories from AI. Someone who has domain expertise outside of programming is able to create a really useful piece of software because the barrier to entry has been significantly lowered. Really nice.
I very much have had the same experience as the author with research libraries. I have learned so many life-changing techniques and derivations from books that were two or three down from the book that I was looking for. When I became a research associate, I would often have my librarians ship the books to the branch closest to my office, and I stopped browsing the shelves myself. It wasn't until I went to the main library that I checked out a book several over from the one I had found in the catalog that I had a major breakthrough. Now that same library has collapsible shelving that makes browsing feel like a chore, and it makes me think we lost something valuable.
That said, I had high hopes for the catalog that the author had put together, but for my technical fields, I found it lacking. Trying to find books that would be close to van Soest's Nutritional Ecology of the Ruminant or van der Werf's Animal Breeding: Use of New Technologies, yields a whole bunch of pop-sci titles. The joy of finding these titles at the library in the past was how remarkably understandable and engaging they were while remaining scientifically rigorous. I immediately feel that the pop-sci titles that I was recommended are going to try to sell me something or some idea, rather than letting me explore ideas myself. I guess that means I need to go back to browsing library shelves myself.
If you are interested in a non-fiction book, check out Veritas: Truth Across Cultures[1]. It's a collection of quotes/saying/proverbs that are similar between cultures, which I call truths. I am in interested in long term thinking, things that stood the test of time. This was primarily put together for myself as a lens to view the world.
Very neat site and write-up, but I found this part amusing:
> There is really nothing “AI” about this aside from the tool that collected the data and coded it, and, crucially, semantic search […]
So really, everything about it is AI. And that’s not a bad thing! It’s okay to simultaneously preach the superiority of award winning books over AI-generated garbage while also acknowledging the same AI as a valuable tool for other uses.
Fascinated that the author shares an experience of discovery in libraries. The internet, in its early days felt like that. Of course, no longer.
Libraries are certainly declining in their traditional form. I find it odd that everyone has a digital resources in their pocket yet libraries are squeezing out physical books to make way for more and more computers. Try to find paper copy of the Sony founder's book.... that will be £50 on amazon. Prohibitive. No library within a 10 mile radius has a copy.
I remember joyfully discovering Tony Royce's book in my library a few years ago. That enlightenment will never happen now. Primary knowledge is being lost, churned crude will forever lubricate the delusions of those who have no facility to collate the basis of our understanding.
My bona-fides here amount to little more than being a big time reader of non-fiction. But IMO, there's a nice synthesis here. Rather than going long rounds of asking LLMs about a subject, I usually end up asking it for book recommendations. It's much better than a google search and you can push it into some deep corners if you go past the surface level recs.
You can get quite specific. You can find texts you wouldn't discover unless you spent years studying the topic. Often these are completely approachable and give interesting perspectives they just get buried behind a wall of syllabi and listicles.
This is how I ended up reading Thompson's 'The Making of the English Working Class' and Graves' 'Goodbye to all That' among others.
This hits the nail on the head regarding the real utility of AI right now. The magic isn’t in using LLMs to generate endless reams of synthetic text; it’s in using embeddings and semantic search to navigate high-signal, human-curated data.
Is it even possible to find one these days? I would love to see some real honest amazon kindle stats on the amount of ebooks added over the last 3 years. Even a honest pie chart to show people its not worth it to get them to stop posting them.
This is wonderful. 30 seconds into trying the site and I’ve already found a couple of interesting books to read. The search I typed is ‘Stuff being made, how it was made, how history shaped its form’.
"To avoid the tedium, I decided that I would also flip to a random page of every book I shelved and read a random sentence from it."
Also, this is about the lost joy of "browsing".
I wonder how much serendipity played when wandering through a library (or other things in life—even window shopping). But now, everything on the internet, you more or less have to know what it is you're looking for—are rarely "surprised or delighted".
When I see lists like this I am surprised that they always live out school textbooks.
School textbooks are the kind of books that have the very precise purpose of giving you a foundation and roadmap to explore and understand a whole field of knowledge. And they're an industry that has been improving them on successive iterations to produce highly organized, comprehensive and accessible reading material through years.
This is pretty nice, I might use this if I'm looking to learn more on a topic.
I'm always cautious when reading nonfiction because it's hard for me to tell if the author knows what they're talking about when I'm not an expert myself. Using awards is a good metric. Maybe.
I am curious what the "score" for each book means. Is it calculated by giving each award a certain weight and adding them up?
I would really like this, but using academic reading lists instead of prizes. What are the important historians of (e.g.) medieval Italy?
Tangentially related, but what a snappy webpage
Penguin Random House seems to be dominating the leaderboard. What's the story here? Are there only very few non-fiction publishers capable of publishing high-quality work, or is Penguin Random House exceptionally good at it?
Thank you! Wonderful idea, and very well executed. Useful discovery tool.
I understand your passion is non-fiction, but would be wonderful to have the equivalent for fiction, of which there is a great deal of excellent literature. Perhaps the same site could incorporate both fiction and non-fiction, with a filter?
maybe I am being too small brained black/white about this, but something feels amiss about the idea of vibecoding a solution to an AI generated problem. How can we get better, doing the same thing that made us sick?
I like very very much that there's no user voting system here.
I just recently read a really mediocre book with a 4.7/5 rating on Goodreads. Shows being sloppy is not exclusively a privilege for neither the AIs or producers.
I love the "books for dads who like Pavement" prompt! It would be great if you could incorporate the same experience of "browsing" random page of some of the results that are returned
I read at night in bed. I love how, with ebooks (first Overdrive, now Libby), when I hear about an interesting book, I can within 30 seconds search for it across multiple libraries, borrow it, and send it to my Kindle.
Thanks for making this, I searched up economics and found at least 3 titles that , when reading through first pages, are really really interesting.
Is it only me or the titles of top books from History, Nature, and Politics sound very click-baiting? I'm really taken off by these titles.
Awesome stuff!
Now, can this interface with Libby? The worldcat interface is, well, confusing. I can't figure out if it knows libraries near me exist.
I've gotten AI to write things that made me teary eyed --- it manipulated me
I think your amazon links are broken, the ones i tried all 404.
Could you please add a link to the Goodreads page of each book?
Literally reading the books like an LLM would be trained on them!
Thank you for sharing. This will come in handy for my research!
I'm not sure how I feel about "Entangled Life" on the first page of books in "science". Isn't that slop? I mean, it's pop-science bullshit, and it's based on feelgood extrapolation from a thin evidence base. Maybe that doesn't quite fit the definition of slop, but it's not a signal of quality for the rest of the list.