back

by numeri·2y ago·view on hn ↗
Almost all LLM inference these days includes a repetition penalty, to help prevent models from falling into this pattern, the so-called "boredom trap" [1]. Once the model repeats itself once, that greatly increases the chances that the next best token is another repetition (for a raw model like this, at least, that hasn't been chat/instruct/RLHF fine-tuned).

I ran the same prompt with a repetition penalty of 1.1 and got this:

The mountain in front of me was a little higher than the other two, and I could see the top of the mountain.

I'm going to climb it," I said. "It's a good place for a picnic."

You're not going to climb it?"

No!" I said. "But I'll climb it if you want! I can climb it with you—if we can get out there together—"

[1] https://arxiv.org/abs/2007.14966