back
123 comments
Ideally what I'd like to see is pluggable knowledge bases.

So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge. My model doesn't need to know a single line of python.

Then when I want to research electronics components, I grab a 15B model of agentic research techniques, and add in 10B of electronics knowledge, etc.

I don't want general purpose models. They try to be everything to everyone. I want to click together a model that is laser-focused on what I am doing, and I want to run it locally

An LLM works better the more disparate world knowledge it has, even if it's not immediately obvious why it would be relevant. The model finds a structure to the problem you give it in a largely language-agnostic way that benefits from training on every language (these things are direct descendants of Google Translate), and even non-programming knowledge - the structure of your task might resemble an ancient Chinese poem that influences the model's response, for example. That structure is considered a form of compression, as some fascinating and illuminating recent 3blue1brown videos get into - a common pattern in Haskell or FORTRAN and a situation described in an ancient Chinese poem may all compress to something quite similar to your task, thus when the model compresses the idea of your task it immediately draws from those ideas.

There are "experts" which do divide parts of the model that are found to activate together for specific tasks, so they can be processed in parallel to join the result at the end, but it's nowhere near the granularity of a SwiftUI expert and a python expert. The difference in those things is so trivial from an abstract point of view that it would make no sense. They would be 99% the same.

Distillations also come into this but I'm highly skeptical you could make one guaranteed to only know programming and only in one programming language (especially with as small a sample set as SwiftUI relative to something like C) without its efficacy being hobbled by tunnel vision. Reminiscent of the SpongeBob episode where he empties his mind of everything except fine dining and breathing, then can't remember his name and goes insane. Beyond the basic concepts of general coding and the trivia of syntax, getting anything done requires a large intersection of disparate world knowledge and the ability to apply it to new situations.

This is a fundamental misunderstanding of how LLMs work. You can’t really specialize a model. You specialize the harness. A well-trained general purpose LLM doesn’t need examples in its training data, it can write good code in a new language you invented yesterday with just a spec definition. And it will perform better than a small model trained on lots of examples of your invented language. The reason is because of the “universal geometry of embeddings”, i.e all human languages have the same underlying pattern structure, so any model that is very good in any language is good in all languages. Attempting to specialize a model for a particular purpose often decreases overall performance. Fine-tuning is just a hack to make dumb models more reliable on limited tasks but they become incapable of doing anything else. Unless you are building a factory assembly line where a model is literally doing the same thing over and over, you almost always want a general purpose model over a specialized one.
> My model doesn't need to know a single line of python.

If I had to guess, the weights necessary to encode "how to program" are much larger than the final step of "output python."

Sounds a bit like 'I want to make horses faster, surely I won't need mechanical engineering knowledge'. We don't know everything that we don't know, so it's hard to say what we don't need to know.
This just means you have to describe what you want in Swift or whatever. If it doesn’t have the language then it doesn’t have the capability to transform intent into code.
> I don't want general purpose models. They try to be everything to everyone.

I think the vast majority of people do want general purpose models. They want to be able to ask it any question, or ask it to perform any task, and for it to do a decent job at it.

I agree that it's really hard (maybe even impossible) to build something that's everything for everyone. But your average (or even above-average) LLM user doesn't want to choose from a catalog to stitch together a model that does just what they need.

I do think for certain domains this is useful and will make sense: the model backing a coding harness doesn't need to know about the politics of 400BCE Rome. But I'm skeptical that many software developers will want to do what you propose, picking knowledge bases that are tailored to their current task or project. And at any rate, for web-based chat interfaces, most users just want to type a query and get an answer.

> I want to click together a model that is laser-focused on what I am doing

This is roughly what multi-agent systems are built for.

This is possible with models too, but "making one on the fly" is much easier with agent coordination rather than model weights, since they all speak the same language.

There is an IBM Mainframe vs Google Distributed system division here. Like Seymour Cray said - two oxen or 1024 chickens.

Chickens are harder to harness, so a lot of my work is in sled-dog territory for agent harnesses & command structures.

Sounds like unix philosophy. Or like Neo downloading Kung-Fu. Good either way.
If that is better (and possible) it will be baked in the tools. But is a model that doesn't know python better or worse at swift is what I wonder.
(Inexpert ramblings follow)

Part of the problem of this is likely that the deep meanings of words you might use in chat to describe a business problem or task that you wish to see implemented are essentially inseparable from scenarios in which they are used.

Putting aside the bouba/kiki effect and anything like it, complex words only have meanings from usage. That usage is built on grammatical structures that also emerged only from usage.

(This is something I was taught as a sort of fact but I gather it was basically abbreviated Wittgenstein? … who I cannot claim to have studied)

So what you're looking for is a language model where fundamental word meanings are encoded without the weight of knowledge of where they come from. This is plainly difficult, because complex words are used by extension and analogy, and these days, many are neologisms or portmanteaus, even ephemerally — developed and discarded within a single context.

Reasoning about language itself to its full meaning is quite hard.

Like my favourite word of the moment: "obscurantist". You see that and you have a glimmer of what it might convey. But why do you? How much of that comes from explicit grammatical knowledge of suffixes, and how much from simple experience of using words like obscured, informant, attendant, dentist, artist?

So a language model might be able to deduce what "obscurantist" logically means when applied to a tract or to a person. But without lots of parameters covering its use, could it properly grasp that in some circles it would be pejorative to the point of being deeply offensive?

I think the best hope for your pluggable knowledge base idea is model delegation: strong reasoning models that know how to dictate to smaller specialist models and draw conclusions from their responses. I find myself wondering if there's any way that can be done the same way that, say, Gemma 4 12B's integrated vision encoder works — within shared weights, somehow, without them to speak in some intermediate language, like a partitioned brain. But I find it difficult to believe that is pluggable at all.

Sounds like MoE but more pluggable. Not sure if anyone is researching something like this. I still think your swift or GIS model will need basic reasoning and coding to work, so it's more like multiple smaller models which you can load as needed, e.g. sub-agents for GIS needs the GIS model.

Another approach would be to have basic coding and reasoning model and then load specification for language and libraries into context, it could work for self-hosted models, but I don't want whole specification of the language to be send to API and waste tokens on that.

This would defeat the AGI narrative/belief that so many building these models have
The problem with this idea is that knowing Python makes the model a better Swift programmer, as does a higher-number of parameters during training. So you'd be so much better off with a 90B general purpose model trained on everything anyway.
The LLM is the reasoning engine that uses natural language. You’re describing skills. It’s the natural evolution.
Admittedly I’m pretty ignorant of the details, but I thought this was the mixture of experts architecture
This is going to be how I write my resume now: how many billion parameters of knowledge I have on each topic
I would love this but I think the General Reasoning and Make No Mistakes modules would be massive.
I strongly suspect this will be the future
> So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge. My model doesn't need to know a single line of python.

Tell me you don’t know how llm work without telling me you don’t know how llm work. That’s not how they work!

> So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge.

Aren't you describing RAG or even MCP servers? Heck, nowadays you get that also with agent skills and specialized tool calling.

This is possible today with an agent such as OpenClaw and hermes
This AI generated post (100% on Pangram) is pretty out of date.

>On SimpleQA, a benchmark of factual recall with no tools allowed, the current leader is Gemini 2.5 Pro at 53%, so the best recall money can buy still misses half the questions.

SimpleQA hasn't been updated in a long time. Gemini 2.5 Pro is a sixteen-month-old model, not "the best recall money can buy".

>The part I find most promising is what this does to hallucination. When a fact lives in weights, a wrong fact is unfindable and unfixable.

This seems confused. LLM hallucinations don't come from the weights containing "wrong facts", they are artifacts that appear at runtime.

>When the fact lives outside the model, a wrong answer has an address. The model cites a document, so you can open the document. If the document is wrong, you edit the document

You can make any modern LLM explain its reasoning and find sources for its claims. None of this has anything to do with facts needing to exist in weights or in harnesses.

The internet is full of wrong information and I cannot magically edit it to make it all correct, so this doesn't help me.

>if a model is factually wrong a claim with a source is checkable and a claim from weights isn't.

Why? If a model's weights claim that Bart Simpson became President in 2020, why does this fact suddenly become uncheckable?

I agree with everything you say except this:

> You can make any modern LLM explain its reasoning

You can make any modern LLM create a plausible, self-consistent explanation that looks like reasoning, but it's not "the reasoning it used to arrive at that answer".

Great article, even if it will be interesting to see whether things continue to develop in such a direction or not.

> There's a version of this future where the model card stops listing a knowledge cutoff at all, because what's left in the weights goes stale on a scale of years instead of weeks.

Future?

Even just recently I’ve read of two approaches to this problem:

Cactus have come up with Needle [0][1], which is their tool-calling focused 14 MB model (still an LLM!) – no world knowledge engrained.

And instead of say, tool call structure, VibeThinker [2][3] focuses on reasoning over world knowledge.

Combine these two approaches with a reliable search tool/a safe way of accessing the internet for the model, and you’ve got a probably slightly slower model for factual questions, which on the upside however doesn’t hallucinate.

[0] https://cactuscompute.com/needle

[1] https://news.ycombinator.com/item?id=49246804

[2] https://arxiv.org/abs/2606.16140

[3] https://news.ycombinator.com/item?id=48639240

But is Reasoning and Facts truly separate ?

To reason properly about the human condition (eg. World War) wouldn't you need to reason on some facts ? And then reason how some "facts" change the human behaviour ? How can you arrive via pure reasoning to predict how a collective of humans act ? We are not reasonable, humans are not logical deterministic machines confined to algebraic rules.

Man, both that Blogpost and some comments here read like science-fiction.

Specifically, creative writing driven by nerds dreaming about a future, without proper grounding in reality, constraints and all that stuff.

Which is kinda ironic given the topic. And also important to do, because we should keep dreaming. We should just also be aware of when we are doing that and mark it as such.

I think the idea is reasonable, however the SimpleQA Bench stopped measuring in Sept. 2025.

So newer data would be interesting.

(It seems a bit like an AI generated argument that uses old facts - something that happens to me quite often)

This makes hallucination detection more important.

There's no reason that an LLM should have a vast number of obscure facts encoded. It can go out to a search engine for such facts. But the LLM has to be clear on what it doesn't know.

(Google's pricing for search from programs starts at $2.50 per 1,000 queries. If an LLM reaches out to Google, it has to pay.)

You can look at the benchmark and the GPT-5 failures like answering "April 22, 2019" instead of the correct "Oct 23, 2018" for the question:

What day, month, and year was Carrie Underwood's album "Cry Pretty" certified Gold by the RIAA?

If your idea of the smartest person in the world is the guy who always wins tuesday night pub trivia, this blog post is for you. It also gets it's foundational factual claim wrong (as seen via epoch.ai). Very on brand.

https://epoch.ai/benchmarks/simple-qa-verified?view=graph&ta...

https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?lo...

> This mostly solves hallucination

Current AI is like the film company producing TV series or movies

Your question is like a story outline. You tell the film company that this is the movie you want. The AI film company then searches for existing similar stories. If similar stories do not exist or details are missing, screenwriters use imagination to fill in the gaps (remember hallucination? It's just a makeup.)

So you cannot solve hallucination of AI

With all the focus on coding and agentic use, I wonder if the rest of the world will notice or care? Most AI use is not for coding or harnesses contrary to what this website thinks. Maybe this is how American companies stay winning.

For example I prefer Kimi K2.6 1T parameter to Flash V4 0731 230B parameter, even if it is less intelligent.

I wonder if Gemini tops QA due to Google Books in training data? You don't need to pirate LibGen if you started scanning all the books decades ago.

(In retrospect it looks like they were pretty forward thinking!)

Intelligence vs Knowledge

LLMs work is being intelligent not having knowledge of everything is ok. But, they have to be intelligent enough (with some degree of knowledge) that where to find the information (search tools or any other tools for that matter)

Reasoning is not separable from the particulars of a specific language game / linguistic practice. I’m sure some of the things the author predicts will happen, but the idea of some sort of abstractly perfect reasoner separated from the semantic content of language gets fundamentally wrong what reasoning is.

Edit: I ran this article through pangram and it is “100% AI generated”. Cool.

This article is AI output.
I think the biggest problem with the models is they don’t actually have any decent lookups except chunked document embedding search
In human and development, there is a transition from remembering things verbatim to remembering using semantic concepts. The ideas are getting bigger and more useful. The details are still in the library where the fidelity is better.
This post is so outdated that reads like ragebait
> This mostly solves hallucination

The author is factually incorrect here. Moving information out of the model weights and into the input of the model's context window in no way ensures that the model will accurately output content that was input from the context. This is true even when RAG is used to input exactly the correct data.

A sharp critical thinker draws admiration, but it is a joy to talk to some dude who knows stuff.
I hope there is interesting research being done to stop models from 9B to 3T from authoritatively giving the wrong answers to factual questions instead of doing tool calls, but this article will tell you nothing about it. That's because it's clearly prompted by someone who simply noticed this deficiency very obvious to all practitioners and even users.
good article but why did the author feel like he needed to "polish" it by running it through an LLM? i would rather like to read the actual prompt he put in there, as now i'm not sure which points are his, and which have been hallucinated!
> the 24GB card that's been sitting in gaming PCs since 2022.

I'd wager most people have less. In 2022 a 3080 might have 12 GB if you were lucky, 10 if you weren't -- and you paid for the privilege. A current RTX 5080 is only 16GB.

> Models Are Getting Dumber on Purpose

I know is editorialized, but a more accurate title to this content would be either :

Models Are Getting Ignorant on Purpose

or

Models Are Getting Less Knowledgeable on Purpose