▲ 564 points
back
472 comments
I suspect that a lot about what we call being very intelligent is ultimately out-remembering people around us. I think of all the times in my software career when I did something that others considered very high performance, it either came down to either having more energy than others at tackling a problem they thought was more trouble than it was worth, or just bringing back random knowledge from previous jobs or self study, and being able to apply it to the problem at hand.
I don't think I've had a truly original idea in my life. Combine A + B, when it's rare for people to know A and B at the same time. So from that perspective, what LLMs are doing is basically the same thing. Sometimes I am faster than the LLM because my context might be better organized, but it typically needs just a hint from me to steer itself correctly. It claims something is a memory leak, but smelling a rat, I suggest it to double check the garbage collection statistics too, at which point it's clear it's no leak, but a tuning error, at which point the LLM is better at tuning than me, because it has more energy than I do.
Maybe there's true brilliance out there, when something doesn't come out of combining data and building hypothesis until you get really lucky. My experience is not comprehensive. But I look around me, and it sure seems I've not been lucky enough to see it. Even the shiniest people I've worked with, which most of the audience here would recognize, have never shown me that they can go past this.
People have told me I was smart since I was a kid, but I can't remember for shit. I had a thought when I was fairly young that the only reason I was (maybe, sometimes) outperforming others intellectually is that I was habitually compensating for my poor memory by working things out on the fly, while others could rely more on rote memorization. Anyway, takes all kinds I guess!
> out-remembering people around us.
Doing an angular -> react refactor with Opus 5 at work.
Amazed at how good it is but also amazed at how dumb it is after I told it I want a very basic and specific code review:do the file names match the contents? is one functionality/concern split in multiple files? do the variable names match what they do? does a method read nicely?
Anyway, I was very surprised at how bad it can be at this.
I suspect it's because the LLM simply does not focus well enough on a single method.
My gut modelling is that the critical files are quite small, some 4k LOC and the LLM tries to read them all in one go and my requirement is surprisingly difficult but the LLM does not realize it, loads all the input into context and simply cannot find a good signal per token/group of tokens.
I guess I can do it because I literally cannot fit 4k LOC in my head and I actually read only tens of lines but I can reeealy dig deep in those lines.
The LLM can remember lots, it has 1M context window, was trained with needle in the haystack problems, but it can severely underestimate how much computation is required with TASTE/universal computation type tasks.
Yes, LLMs are incredible when it comes to delivering well defined outcomes, it's evident they were RL trained very well. But they are incredibly bad at understanding nuance in a text.
It's incredible how agentic coding benchmaxing seems to correlate with writing good poetry but at the same time finding good names for files and variables and wiring beautiful code does not.
Same with getting basic details of who said what in an "adversarial" conversation between 2 humans -- LLMs are incredibly bad at it, with all their "almost discovering new maths" skills
In line with the other comment on this post: I also am classified as “brilliant” by people and IQ tests. My country gave me an award for being a great inventor. But my memory is worse than anyone I know.
I can’t remember the lyrics to any songs, or the lines from any movies, or what I’m doing this week. I don’t think it’s a strong hypothesis.
My wife has a nearly eidetic memory, so marrying her was a game changer for counteracting mine :)
> I suspect that a lot about what we call being very intelligent is ultimately out-remembering people around us
There is a great episode of Inside the Actor's Studio where there is a Q&A with Mike Myers.
He was asked if going to college/getting more education was helpful to being a comedian.
He responded:
"Many times in writing comedy, you are on an 'island' and you are trying to figure out how to get to the next 'island'. Having more education and knowing more about the world can definitely help you do that"
I like the above quote because to me intelligence is a combination of remembering useful information but also being able to "connect the dots" between those facts and apply to them to the current situation.
As another example, I was watching a documentary about LIGO. One of the engineers had memorized the entire system in his head. The other engineers would come to him and ask "Will this change to component A affect anything else?". He would pause and think and then say "Yes, component F touches that so it will be affected." The other engineers mentioned how useful it was to haves someone who had the entire system in "one brain".
I've seen this in finance technology too where someone, usually a woman, has large parts of the system memorized and performs a similar function.
> it either came down to either having more energy than others at tackling a problem they thought was more trouble than it was worth, or just bringing back random knowledge from previous jobs or self study, and being able to apply it to the problem at hand.
There is an ability to hold ideas in the mind simultaneously. Some can hold a great deal more than others. It can be exercised, but is definitely bound by genetics.
If you want an example of someone at the near peak of human ability, check out Jon Von Neumann.
Then there is an ability to peer deeply into complex problems and somehow find the simplest truths that make sense of it all. Think of Einstein.
Both are incredibly intelligent, but in different ways. I'd say Von Neumann's memory was far greater than Einstein's though.
One can flawlessly ponder anything known to man, and the other could ponder completely original ideas (to an extent)
Memory is one aspect of intelligence, but hardly the only one. Making new connections, logical deduction, spatial intelligence and creativity are also aspects.
I have fairly poor memory, but I can solve problems that people with better memory can't.
Since it is so much easier to test for memory than understanding society grew to confuse the two. Only now faced with these other entities are people beginning to question it.
I know people that got to post grad math without understanding a thing but they could remember a lot easily, while many of those that understood but had a harder time remembering every last variation of everything got penalized.
> I don't think I've had a truly original idea in my life.
There’s no such thing as a truly original idea. It is all just combining A+B!
It's also "out-brute forcing them." It just never gets tired. If a mathematician picks a research direction and spends a whole week on it and it doesn't pan out, they will likely be annoyed, need a break for a while, etc. This thing just does not ever get tired or discouraged or care; it's just onto the next thing until something ends up working.
One thing about human mathematicians is that they only publish positive results. Professors etc might have file drawers full of "negative results", but the incentives and bandwidth of human mathematicians makes publishing these useful results impossible.
But AI agents have no such limitations and can publish and re-use negative traces easily. There have been some recent projects (https://www.theoremdb.org) aimed at exploiting this fact. https://news.ycombinator.com/item?id=49227505
In general though, LLMs do not have the same limitations and incentives as human mathematicians, and the next year's tsunami of change will make this abundantly. clear.
I can't help but think of Michael Nielsen's essay "Augmenting Long-Term Memory" [1].
> Many people's model of accomplished mathematicians is that they are astoundingly bright, with very high IQs, and the ability to deal with very complex ideas in their mind. A common perception is that their smartness gives them the ability to deal with very complex ideas. Basically, they have a higher horsepower engine.
> It's true that top mathematicians are usually very bright. But here's a different explanation of what's going on. It's that, per Simon, many top mathematicians have, through hard work, internalized many more complex mathematical chunks than ordinary humans. And what this means is that mathematical situations which seem very complex to the rest of us seem very simple to them. So it's not that they have a higher horsepower mind, in the sense of being able to deal with more complexity. Rather, their prior learning has given them better chunking abilities, and so situations most people would see as complex they see as simple, and they find it much easier to reason about.
I once tried out his Anki approach during a math lecture. Whenever I reiterated a card, say about some lemma, I noticed something interesting about it. This was delightful and many lemmas became much more streamlined over time. It's not a "solution" to mathematics, but I found it delightful while it lasted (before akrasia or lack of time kicked in and I stopped doing it).
It was duuring my PhD I realised that thinking is essentially
1. remembering all the different information to remember all the tricks
2. trying all the different tricks in the problem
3. optimizing deciding which to try based on different information
4. trying random things to discover patterns (and hence new tricks)
5. explaining your tricks to others so that they can do the first 4 steps independently and come up with even better tricsk
6. refactoring tricks into common and special parts to create a well organized theory
7. documenting for future generation in a language they can understand
that's all thinking is.
I thought this was fairly obvious. I don't consider any of the AI models I've worked with particularly smart, but they've read orders of magnitude more than I ever could in my entire lifetime. They have far better book knowledge than I have, so that's how I use them. I use them for things that I suspect other people out there would know, but I don't. But when I work on something that I suspect is truly new, the models rarely understand what I'm doing, and I've got to do it myself. Although I still poll them for basic principles, best practices, and other advice.
Yes. That's how LLMs do programming, mostly. It's also why LLMs don't need abstractions or parsimony as much as humans. They can work on something complicated without simplifying it first.
This has major implications that haven't been fully realized yet. On the math side, there are long machine generated proofs. On the code side, there are high volumes of code with similar code not being folded into functions.
While TFA itself makes sense I disagree with the title and the conclusion. I would not consider referencing working memory during thinking as “remembering” but as a part of thinking itself. Working memory is the RAM to the much larger but higher latency indexed database that is our long-term memory. As such I would say AI is out-thinking us, even if in a brute force sort of way.
I think where you could say it is out-remembering us is when it can contemplate the vast universe of patterns, gleaned from essentially all human disciplines, encoded in its weights, that may let it draw connections that a human could not, unless they just happen to be familiar with multiple disciplines.
Which is why I think the analogy with Von Neumann / Einstein is also a bit off. From TFA it seems Von Neumann was more akin to what AI does, than Einstein. I don’t get the impression that it was Einstein’s memory but his ability to look at things from a radically different perspective. So far I don’t know that we can categorically say that LLMs can or cannot do that.
A superhuman working memory is exactly how I've been describing the LLM advantage. Paired with the unreliability of its reasoning and judgment, it is what makes AI a supplement to human intelligence, not a replacement.
On the other side, LLMs make random mistakes and wrong choices and they have a bias toward writing more code instead of less. You can make up for this to some degree by running another LLM against their output, but with very diminishing returns. Even if they were perfect, there will be an ongoing cost to little or no human awareness and understanding of the codebase.
It may take some time for people to recognize the cost of AI code generation and their value for virtually everything else, but I believe we'll get there.
Sounds like another attempt to frame AI in a way that makes them feel better about themselves.
The simpler explanation is that a working memory is a requirement for intelligence, and a larger working memory will make you more intelligent. Hence the AI can in fact be more intelligent than the mathematician.
There are plenty of high value endeavors where being a superhuman knowledge remixer is right on target. But even capturing all of the knowledge is proving elusive.
I use coding agents. I think they're pretty good overall. They save me a lot of tedious coding. For example I probably wouldn't spend the time to implement native splash screens for all the build targets of a Flutter app, but I'll have the coding agent do it.
Nevertheless, for all the time that we've had coding agents, it's still trivially easy to find the jagged edges of their training. For example, Gemini evidently doesn't know if the Xcode part of a Flutter tool chain is misconfigured. That's not exactly a Millennium Prize problem. But it is shaped wrong for a training set for a coding agent.
Working memory is literally the L1 cache of the brain, which holds a mere 3-7 objects. The article seems to conflate w-m with regular memory, which is what a human knows at a specific time.
Computers, especially now with LLMs, have everything vastly larger than human like speed and complexity. What LLMs lack is more qualitative traits, like creativity, abstraction and intuition.
LLMs are still missing a part of working memory. Part of working memory is being able to attend to small amounts of information and then understand and parse all the pieces of that information. When LLMs use their "working memory" they just analyze different probabilities of tokens and there is no prioritization or understanding of the information in the way humans have it.
If there is no training data or data in the context that leads it to the correct result then it can't do it, whereas a human seems to be able to generalize and abstract a goal and then repeat an action or thought process in a 'recursive' manner to reach the result. AFAIK LLMs don't do this.
Just as an example to illustrate. I recently asked an LLM to organize a bunch of artists albums into whether they were released by a major label or an independent label, and for the most part it did a good job. But there were albums that it classified as independendent that weren't. I presume because it either didn't run into the right data when searching or it misunderstood the data it did find. A human would not do this because if a human had a list of all major labels, it could instantly detect whether an album was or wasn't indie, because it doesn't do any complicated parsing or token probabilities that LLMs do, it just recognizes a pattern (either an album is indie or it is not, a human brain needs simply one piece of information to decide this), an LLM is not that simple.
In a way human brains are simpler than LLMs. The algorithms it runs mentally can detect a piece of information and then see most / all of the consequences of that information whereas an LLM thrawls through megabytes of text and does a token probability distribution and so on without any simplicity.
It's probably worth checking the site history before posting. The guy who wrote this looks like some kind of "race science" crackpot.
Memory and intelligence are linked.
Someone with a better memory for ideas or concepts will be able to more quickly incorporate those into novel ideas or recall them when necessary to assist in solving a problem than someone with worse memory.
To those here challenging this with “yes but I’m smart and my memory is bad” - a) define smart and b) perhaps your memory for trivial things like life events, what you did two weeks ago on Monday or people’s names is bad, but I suspect your memory for “work” or problem solving is strong.
Another example I used to see (hear, rather) is how musicians rip off each others riffs and hooks without noticing (unintentionally - they claim), which I long suspected as simply “forgotten” riffs they heard in other songs that once they started playing themselves by chance they attributed to their own creativity. Creativity and intelligence are somewhat linked that way I suspect.
In any case, this all boils down to the same thing, you can think of yourself as a dynamic model made up of memories and biases to some degree, and your ability to store and recall useful information to solve problems increases what we call your intelligence.
I often think “what will ‘maintainable’ code look like in the future?” given this kind of thing. Much of ‘good coding’ is about optimising the codebase for workability for a human intellect with human working memory.
Though having said that, from the studies I’ve come across it seems like LLMs tend to generate more verbose code but perform better over the long-term when the code is maintained and not allowed to sprawl all over the place.
I’m mainly using AI for tools development and in that context I often wonder if I’ve just developed a career-long habit of over-abstracting. Like, the tools work without all the fuss I might have put in at the beginning and you just iterate and evolve as you go.
IMO, That's the case for all subject areas. It is also one area where AI excels at, and it could be of real use if we can find a way to stop hallucinations. The sum of all knowledge being at our finger tips would allow everyone to focus on the hard stuff.
True, but as it pieces together new mathematical truths from the pieces we have discovered ourselves, it then has more truths upon which to build new solutions. And so on, so while it is just remembering things we have forgotten, the amount of progression an LLM can make may still be several steps ahead and touch areas we have not yet been able to consider or make any progress on ourselves. It's a bit like a pyramid though, eventually it will have tiued together all teh things we know, found all the things we could have known, and then .. perhaps, be unable to actually come up with something genuinely new.
Outside of math you can basically take the entire corpus of research papers on any topic and have the AI read all of it and provide an analysis cross referencing everything all at once. This applies to everyone and everything.
This is why education used to start with rote memorization.
Functional intelligence isn't abstract, it is based on useful information you can quickly recall.
if you take any system and break it down you get to simple building blocks (like memory). Then you claim that it isn't "outthinking" but "out remembering". Sorry, but what do you think the brain is doing? Not using memory? Especially working memory. How about frequency, brain is operating in Hz whereas most processors are in GHz, etc.
Finally if we were to think in terms of a proper memory/latency hierarchy, what's a Model + CPU's equivalent of working memory? Registers? L1 cache?
What I'm looking forward to amidst all the negativity, fear, and loathing is for some 20something mathematician to outdo both humanity and machines by leaning hard into centauring to expand the frontiers of mathematics. Pretty much what I think the future will play out to be as well, but I don't think people are ready for that yet.
That intelligence is "just" retrieval within a huge corpus is an old one, and the one I am thinking about a lot these days: https://news.ycombinator.com/item?id=44060672
Does it matter? It's going to produce proofs far more intricate than humans can understand, outdoing humans and opening new frontiers.
The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.
Which is why love it! It can easily did through the code flows of our 250k file project and more often than not and understand what's happening much faster than I ever could.
This means some AI proofs might be impossible to comprehend by humans, right?
I guess AI still lacks human intuition for many concepts, but AI might beat humans in narrow areas, such as discrete math and combinatorics.
it's a paradox given how extremely complex each single neutron is. my current solution is that most of the complexity of neurons is just for maintenance
Exactly, it's making connection across vast set, not bringing the magic intuition.
Has anyone tried feeding all of human knowledge to an LLM prior to Einstein's work and tried to have it reinvent physics?
So does the cheapest calculator?
C'mon AI companies, pivot to lawyers or doctors already.
Trying to convince us that mathematics and software engineering are "solved" is getting very tiring.
The pushback would probably be too much for the soon-to-be IPO-ed companies.
Along with remembering facts, it is important to insightfully give ‘weight’ to the facts.
AI gives ‘weights’ to things depending on its own priorities or corporate interests. Our brain gives the weights to various facts depending on personal insights. (X is painful, Y is more effort but less painful etc etc)
In a real world scenario these both can act in complimentary ways and AI just supplements the human working memory at the end of the day.
Is there anyone on the planet who doesn’t think ”thinking” also includes memory?
I love the term "Out-Remembering"! I have been trying to find a way to communicate that "intelligence", "creativity" and so on might be misleading about the true nature of LLMs, and they would better be described as genious "reproducers" as in, they are very capable at reproducing what they have already seen - and they are a bit less capable, but for many use cases still good enough, at reproducing a mix of concepts seen previously.
This also nudges into how to use it best: By knowing where the "piles" of if training data are (i.e. when it comes to a CLI in rust, I just briefly describe the use cases, and I have a very high confidence the code will work exactly as intended by me since there will be a multitude of examples in the training data), one can predict where the LLM is likely to go wrong an prompt/guard accordingly. This skill grows with domain expertise, and is one of the many reasons LLMs can be (and probably should be) used to outsource busy work, but never understanding and learning. ("never" is a not meant literaly of course - I for one am glad that I do not have to wrap my head around CSS and other frontend topics and go straight to the topics that interest me most)
"Out-Remembering" captures that perfectly, I feel. Also goes nice along with "asking it leading questions" as we know how to do in real live; if you want a person (LLM) to confess (produce output tokens) something, sometimes you do that by leading the interogation (chat, context) to where you think the truth lies.
and yet very little access to common sense :)
the article cites the ability to keep a ton of details in working memory as an advantage, I'm not so sure that it is - perhaps it is quality over quantity; a compression of everything known into a smaller set of interlocking patterns should provide a more useful generalization (if correct, of course). The human's ability to perceive 7-10 concepts might just be the tips of the icebergs composed of a gazillion micro-concepts, i.e. our working memory is enormous, but not consciously accessible
also, the older neural capacity estimates are way wrong [1]: dendrites also compute
[1] https://www.yahoo.com/news/science/articles/neuroscientists-...
It is obvious that super intelligence comes from more working memory.
It is the scary thing actually. Cause once AI makes arguments that require a working memory of hundred items, then we as humans will have no way of understanding the arguments…
We can decompose and write things but only up to a point. when Ai can have a working memory that spans hundreds of books, we are necessarily going to have to trust the system.
So build a better proof/paper/technique search engine?
Choosing what to remember is my biggest challenge!
This is not surprising. I find LLMs in general have pretty limited utility, but there a niches where they are good and one of those niches is finding little snippets of information in a way that you used to have to sift through pages and pages of google results to find.
Mathematicians don’t forget things they haven’t learned. So is it that forgot history, or they didn’t learn parts of history to begin with?