This paper is honestly fairly weak, if you look at it closely. The theoretical justifications rely heavily on approximating attention with linear attention, and really just show that linear attention could be written as (A + A_delta)x, then try to imply (not show – they have now theoretical justification for this next step) that A_delta is a gradient. One could make the same implication about almost any matrix-vector multiplication, by splitting the matrix into two pieces.

The empirical studies are fairly weak as well – none of them suggest that there is actually any gradient descent occurring during in-context learning, only that if you fine-tune a model's attention parameters (and no other parameters) for a set of examples, its internal representations and attention patterns are slightly more similar (and it's really not by much!) to those of in-context learning than before fine-tuning. Is it really that surprising? You could most likely take any two techniques that optimize a model towards some target, artificial restrict both of them to only apply to the same set of parameters, use both, and find increased similarity of the two optimized models compared to the non-optimized model.

None of that would be what the paper claims, i.e., a shared mechanism of gradient descent, just a similarity in outcome – and we all already knew that ICL and fine-tuning have similar results!