Item — HN Reader

This was a little dense for me to grok. Are these well known results or is there an abstract-like summary?

The RYS (repeat yourself) hypothesis that duplicating (the right) layers is enough to improve performance (sorry for not reading closely enough, it's really just stacking the relevant layers?).

The ERD (encoding, reasoning, decoding) layer structure is a relatively robust observation? That the middle layers of the NN will reason in universal space, and this is kinda evidenced by cosine similarities of the hidden states at each layer given similar or dissimilar inputs. And that similar inputs converges by layer 5 and you can kinda watch that happen in the cosine similarities?

This post is incredible and I'm afraid it'll drop off the front page before people engage deeply with it. (The methodology was interesting, maybe there's other big ideas I'm missing.)

2 comments

I find the RYS result far more surprising than the ERD result. Encode-Reasoning-Decode is after all a very popular way to design neural networks (even an autoencoder is just that without the reasoning step), the same structure emerging from optimization isn't that surprising.

But the methodology to measure it and put numbers on which layers are most involved in encoding/decoding and where the reasoning takes place is very valuable.

The finding that the phases are more cleanly separated in large-ish models is interesting. I wonder what this could mean for embedding models? Usually we take small LLMs and chop off the last couple layers to get an embedding model. But I wonder if you could get better embedding models using something like the first five layers of Qwen3.5-27B, or the first X layers of Kimi K2.5? The methodology in the article seems to give a straight forward way to find the optimal cutting point

Yeah the ERD result was known before I think, see "The Remarkable Robustness of LLMs: Stages of Inference?" https://arxiv.org/abs/2406.19384

But the fact that the intermediary circuits are generic and robust enough that you can just loop them is unexpected. I mean maybe it sort of makes sense in retrospect, the above and other papers showed the middle layers of an LLM behave more like "iterative refinement", so to use a signal processing analogy maybe you just keep applying filters and suppress the noise.

But by that same analogy, I'd predict that you can't just keep repeating layers, at some point you'll suppress the signal as well. Not sure if there was an experiment conducted with how many times you can repeat RYS layers before performance goes back down.

Perhaps not widely known but certainly known in LLM research. There was a bunch of these experiments done 2 years ago and what's interesting is that it still seems to work on the latest models.

Though beware that the increased score on math and EQ could lead to other areas scoring less well; would love to see how these models score on all open benchmarks.

The author claimed that the models he modified with this layer repetition method topped the huggingface open llm leaderboard in his first post: https://dnhkng.github.io/posts/rys/

Do you remember the names of the previous experiments done on this? Would love to take a look.

Just learned about it the other day from this thread from Feb, 2024: https://old.reddit.com/r/LocalLLaMA/comments/1aqrd7t/i_made_...

Has some interesting github links.