Think of the "fast" network as being like a hashmap. It stores key-value pairs. You feed it a query, which is a key that you're looking for, and it gives you back the corresponding value. Unlike the hashmaps you're used to, however, this is sort of a "blurry hashmap". If you ask for a key you'll get sort of a blend of the values nearby it -- even if the key isn't in the hashmap.

> Are the key value pairs a vector and the last layer an actual token in a simple LLM, for example?

Almost.

Firstly, the image I linked depicts only one attention head in one layer of an LLM. So you have to imagine that replicated several times in parallel, and then repeated several times serially. The Fast Weight Programmer paper does a better job of explaining the fundamental unit; to see how they are replicated and repeated the block diagrams in Attention is All You Need are easy to understand.

Secondly, getting back to the "hashmap" analogy, you might imagine that the hashmap key-type and value-type are LLM tokens, but in fact that's not quite how it works. There are two other types. One is the embedding space and another is the key space, and the hash map keys and values are both of that type. Yeah this is why the key/value terminology really sucks. There is a feed-forward layer that translates input tokens into the embedding space, another that translates embedded space to key space (before the hashmap lookup) and a third one that translates hashmap-outputs back from key space to embedding space. I wouldn't get too hung up on all of these; it's sort of like if you learn a foreign language but still "think" in your native language... the embedding space is sort of like the LLM's native language that it invents for itself. The whole business with the key space is mostly sort of a hack... in order to have multiple attention heads and train on current GPUs efficiently you want the embedding space to be NUM_HEADS times larger than the key space. So the whole key space not being the embedding space is mostly just a kludge to make this work well on GPUs. Unless you're planning on writing your own LLM from scratch it's mostly safe to pretend that the key space and embedding space are the same thing. In practice people have tried taking apart small transformers and the key space appears to work like "parts of speech"... when it reads a sentence like "I walked the dog" the attention heads tend to create mappings vaguely similar to I=subject, walked=verb, dog=object, and then later on the slow network will produce queries like "what verb was applied to the dog?". This is an oversimplification, but it gives a general idea of what's going on.

Also if you like maths, equations (4)-(19) from https://arxiv.org/abs/2102.11174 are a really spectacular example of using math when it's the right way to explain something, instead of using it to hide things.