This post isn't a scientific investigation. It's someone playing around with a black-box model with few to little controls – which is unfortunately the only thing we can do when experimenting with GPT-4, which does excuse this partially.

Unfortunately, though, the author seems unaware of the actual state of research on the actual mechanics of how LLMs store knowledge and specifically binary relations. The ROME paper[1], among others, shows that the feed-forward layers function as a key-value store, where the feed-forward's up projection of the last token in a noun phrase (say, "the Eiffel Tower") acts as a key, which when multiplied by the down projection, produces a value that contains information the model knows about the subject, which is then added into the residual stream/hidden representation.

A paper building on that work[2] then went on to show that it's usually the self-attention layers that use the relational phrase (say, "is in") to extract the relevant knowledge from the feed-forward layer's output (in this example, hopefully "Paris").

This mechanistic understanding makes it really obvious why the reversal curse occurs – using matrix multiplication as a key-value store requires having a fully separate key-value pair to look up the reversed relation.

[1] https://arxiv.org/abs/2202.05262 [2] https://arxiv.org/abs/2304.14767v1