> I don’t think so because those both live in the context window and as such pollute it when they’re not performing optimally.

There is nothing optimal about needing a few billion more parameters to be able to piece together probable answers that can be asserted by querying an oracle.

> I think having unused or rarely used weights doesn’t influence the results as poorly as RAG injecting irrelevant facts.

Those aren't free. The more parameters you add, the higher the computational cost required to train and prompt a mode.

And all for what? To piece together info that you can just query from a data source?