I've been thinking that they have to be doing this. It seems like a fun problem, actually - all you're trying to encode is a 1-bit message within a text with the least amount of necessary changes possible, but in a way that arbitrary fragments will show it.

My intuition is that this would be very possible, in a way that makes false positives so unlikely as to be virtually nonexistent (at a certain fragment length.) Basically all you would be trying to do is to defeat people who would deliberately screw up the signal below the fragment length, and you would try to get that fragment length to at least the size that intentional obscuring of the signal would be obvious. I could see it being possible to detect even from non-contiguous fragments interspersed with noise.

It's just 1 bit, and you don't really care if a sentence or two is slop. I'd be surprised if a PhD interested in steganography couldn't come up with a good scheme in a week. It's a QR code.

What would be scary is if they could come up with a way to detect advice from Claude i.e. you get Claude to review your work as an editor, read the output, then as a result make non-verbatim changes, and that signal still gets through. If you could do that, you could do things like tell if a pundit speaking on television has read a particular Wikipedia page. Seems impossible, but LLMs seemed impossible.

edit: there are so many unimportant language choices; ones that are even hallmarks of AI use already, like the fact that it generally picks the mode. Not always picking the mode or picking at precise distances from the mode could hide signals without significantly affecting the quality of the content.