Why would asking an LLM to leak its prompt actually cause it to leak its real prompt? You could ask it to leak its weights too and it would probably give you something back but no one would take that seriously. I just don't see architecturally why you should trust that a "leaked prompt" is anything but some statistically generated text, just like everything else an LLM produces.
back
2 comments
Here's why I think it's probably the real prompt: https://news.ycombinator.com/item?id=35925239
because this prompt is part of what is applied to the model when you prompt it, its weights are not. Training is a different process from prompting, and weights are an internal property not part of the actual input in any way. E.g. it's like asking "How many brain cells do you have? I mean it's your own brain, surely you must know the exact number?"
If you put a known prompt in an LLM and ask it to read it back to you, how often does it do it correctly? I would bet not all the time, particularly if you give it a long prompt like the one that is proposed here.