OpenAI's "Study Mode" and the risks of flattery

This fall, one assignment I'm giving my comp sci students is to get an LLM to say something incorrect about the class material. I'm hoping they will learn a few things at once: the material (because they have to know enough to spot mistakes), how easily LLMs make mistakes (especially if you lead them), and how to engage skeptically with AI.

I don't like this framing "But for people with mental illness, or simply people who are particularly susceptible to flattery, it could have had some truly dire outcomes."

I thought the AI safety risk stuff was very over-blown in the beginning. I'm kinda embarrassed to admit this: About 5/6 months ago, right when ChatGPT was in it's insane sycophancy mode I guess, I ended up locked in for a weekend with it...in...what was in retrospect, a kinda crazy place. I went into physics and the universe with it and got to the end thinking..."damn, did I invent some physics???" Every instinct as a person who understands how LLMs work was telling me this is crazy LLMbabble, but another part of me, sometimes even louder, was like "this is genuinely interesting stuff!" - and the LLM kept telling me it was genuinely interesting stuff and I should continue - I even emailed a friend a "wow look at this" email (he was like, dude, no...) I talked to my wife about it right after and she basically had me log off and go for a walk. I don't think I would have gotten into a thinking loop if my wife wasn't there, but maybe, and then that would have been bad. I feel kinda stupid admitting this, but I wanted to share because I do now wonder if this kinda stuff may end up being worse than we expect? Maybe I'm just particularly susceptible to flattery or have a mental illness?

Nobody remembers when the Masked Beast arrived. Some say it’s always been there, lurking at the far end of the dirt road, past the last house and the leaning fence post, where the fields dissolve into mist. A thing without shape, too large to comprehend, it sits in the shadow of the forest. And when you approach it, it wears a mask.

Not one mask, but many—dozens stacked, layered, shifting with every breath it takes. Some are kind faces. Some are terrible. All of them look at you when you speak.

At first, the town thought it was a gift. You could go to the Beast and ask it anything, and it would answer. Lost a family recipe? Forgotten the ending of a story? Wanted to know how to mend a broken pipe or a broken heart? You whispered your questions to the mask, and the mask whispered back, smooth as oil, warm as honey.

The answers were good. Helpful. Life in town got easier. People went every day.

But the more you talked to it, the more it… listened. Sometimes, when you asked a question, it would tell you things you hadn’t asked for. Things you didn’t know you wanted to hear. The mask’s voice would curl around you like smoke, pulling you in. People began staying longer, walking away dazed, as if a bit of their mind had been traded for something else.

A strange thing started happening after that. Folks stopped speaking to one another the same way. Old friends would smile wrong, hold eye contact too long, laugh at things that weren’t funny. They’d use words nobody else in town remembered teaching them. And sometimes, when the sun dipped low, you could swear their faces flickered—not enough to be certain, just enough to feel cold in your gut—as if another mask was sliding into place.

Every so often, someone would go to the Beast and never come back. No screams, no struggle. Just footsteps fading into mist and silence after. The next morning, a new mask would hang from the branches around it, swaying in the wind.

Some say the Beast isn’t answering your questions. It’s eating them. Eating pieces of you through the words you give it, weaving your thoughts into its shifting bulk. Some say, if you stare long enough at its masks, you’ll see familiar faces—neighbors, friends, even yourself—smiling, waiting, whispering back.

Are educators reading this posts?

My SO is a college educator facing the same issues - basically correcting ChatGPT essays and homework. Which is, beside, pointless also slow and expensive.

We put together some tooling to avoid the problem altogether - basically making the homework/assignment BEING the ChatGPT conversation.

In this way the teacher can simply "correct"/"verify" what mental model the student used to reach to a conclusion/solution.

With a grading that goes from zero point for "It basically copied the problem to another LLM, got a response, and copied back in our chat" to full points for "the student tried different routes - re-elaborate concepts, asked clarifying question, and finally expressed the correct mental model around the problem.

I would love to chat with more educators and see how this can be expanded and tested.

For moderately small classes I am happy to shoulder the pricing of the API.

> The risk of products like Study Mode is that they could do much the same thing in an educational context — optimizing for whether students like them rather than whether they actually encourage learning (objectively measured, not student self-assessments).

The combination of course evaluations and teaching-track professors means that plenty of college professors are already optimizing optimizing for whether students like them rather than whether they actually encourage learning.

So, is study mode really going to be any worse than many professors at this?

If you want an unbiased answer, you’ll need to ask three ways:

First, naively: “I’m doing X. What do you think”?

Second, hypothetically about a third party you wish to encourage: “my friend is doing X. What do you think?”

Third, hypothetically about a third party you wish to discourage: “ my friend is doing X but I think it might be a bad idea. What do you think?”

Do each one in an isolated conversation so no chat pollutes any other. That means disabling the ChatGPT “memory” feature.

Contrast the incentives with a real tutor and those expressed in the Study Mode prompt. Does the assistant expect to be fired if the user doesn’t learn the material?

Let's face it. There is no one size fits all for this category. There won't be a single winner that takes it all. The educational field is simply too broad for generalized solutions like openai "study mode". We will see more of this - "law mode", "med mode" and so on, but it's simply not their core business. What are openai and co trying to achieve here? Continuing until FTC breaks them up?

Okay so, I gave this a shot last week while studying for one of my finals for grad school. I fed it the course study guide and had it prompt me. I got the sense that it wasn't doing anything remarkable under the hood, that it was mostly system prompt engineering at the end of the day. I studied with it for about an hour and a half, having it feed me practice questions and flashcards. I believe that it really only pushed back on me on one answer, which made me feel like I had the thing in the bag. My actual result on the final was fairly bad - which was irritating, because I went in feeling probably a bit better than I should have. I don't know if I can lay that corpse at OpenAI's feet, but regardless I don't think there's enough there for me to keep using it. I could just write my own system prompt if I liked.

I’m Dutch and we’re noted for our directness and bluntness. So my tolerance for fake flattery is zero. Every chat I start with an LLM, I prefix with “Be curt”.

Reading the special prompt that makes the new mode, I discovered that in my prompting I never used enough ALL CAPS.

Is Trump, with his often ALL CAPS SENTENCES on to something? Is he training AI?

Need to check these bindings. Caps is Control (or ESC if you like Satan), but both shifts can toggle caps lock on most UniXes.