Item — HN Reader

It's currently a major area of research and an unsolved problem to find out what individual weights do – and the most recent research seems to suggest that there is not a one-to-one relationship between ideas and weights. In fact, one line of research is showing that each weight encodes multiple ideas with what is called polysemanticity [1], while another line of research seems to show that individual factoids are spread across thousands of weights and possibly even multiple layers [2].

[1]: https://transformer-circuits.pub/2022/toy_model/index.html [2]: https://rome.baulab.info/

2 comments

That really feels similar to genetics.

Early on, people assumed a 1:1 relationship between genes and traits. Some are indeed, but many phenotypes are smeared across thousands of genes in a way that is fully mysterious to us.

Sure ideas and facts would be spread out. The weights are cumulatively involved in generating the tokens. The error margins from them would also be accumulating for some paths while also decaying for some others. If the training and inference process could track, it should be possible to figure the error margins for the output tokens. However, such an approach would be too compute intensive to be practical.