The compression can be lossy though and the model often “predicts” complete garbage even with GPT-4. It would be great to get a probability for correctness “Mr. Spock” style in chat - especially so as in case of a telephone number…
back
1 comments
You can ask it to also provide a confidence score, though I found it biases heavily towards a high score.
There is an 84% chance we will encounter the force field and become vaporized instantly, Captain.
And rising.
What action would you recommend?
this is also hard for humans.
Even with a n shot example this needs complex understanding (same is true for similarity i think).
try yourself ask a random person the similarity between two sentences, without giving them a ruleset or examples.
Even with examples the result very from person to person.