▲ 118 points
back
122 comments
Obviously there's a focus on how LLMs will play into the main narrative... but what I'm more excited for, honestly, is how we can integrate them with NPCs and background narratives. An LLM could generate traits, backstories, narratives, etc for any number of NPCs; rewrite those narratives based on player actions when they encounter that NPC, or other related NPCs; use those updated narratives to create new NPCs, etc.
Very exciting times ahead in game design!
I can't wait for a subscription based open world where the paths of other players caches the generations for the next ones.
So your subscription pays for generations, but also fills out the persistent narrative and lore.
I actually like the 3 choices dialogue system and don't want to come up with my own NPC small talk on my couch, but the idea that any NPC that's been talked to by any player gradually has a whole catalogue of interactions to pull from means there would be compounding depth to the world as more players interact.
Like Minecraft but where it's collective world building by exploration and interaction rather than actual building it yourself.
Humans tend to be pretty predictable, so I feel like the universe would quickly feel like it's insanely deep after just a short while with a somewhat decent player base.
Even if the dialogue wouldn't be generic like ChatGPT tends to be, I would find this incredibly boring. With so many options all the options become meaningless to me. But to each their own I guess - I've always liked games with linear storylines more than the ones where you get to choose your own path.
Also I have a feeling that the NPC narratives when enriched by all player interactions would tend to go towards some average, i.e. become more and more generic and dull. The truly rich experiences have always been on the fringes in all art for me. I'm interested to see a counter example to this though.
No! This is a terrible, terrible idea. At best you can have the LLM generate minor variations of the nothings they say.
1) you need to be able to ensure that the characters don’t hallucinate new details. No big deal.
2) you need to be able to ensure that the characters don’t suggest they can or will do anything that they can’t do. For NPCs, this is typically literally nothing. They can’t do anything. It’s actually way worse for realism and immersion if every character has a unique reason why they can’t do anything than every single guard citing their arrow to the knee and leaving it at that.
3) NPCs for story reasons often should know something but don’t tell you it. It’s very difficult to execute any sort of story flow if your characters are suddenly free agents. At best they merely lack self consistency. Stories are not going to become better if you can meta game them as genre savvy speculators.
4) NPCs with irrelevant backstories are very dull. They are absolutely annoying to talk to. We have these in DND. And you may say “hey, I loved my DND NPC”, but that character probably started to become very important as consistent because of the DM’s attention. Which is cool for DND, but it’s not cool for a structured game.
Like, would Harry Potter be a better experience if you could personally grill dumbledore for information in the middle of the first book? I think 100% no.
Dumb NPCs are an aspect of the suspension of disbelief of game writing. And it largely works. Asking for LLM agents is like asking to remove the need for the suspension of disbelief. But most narratives are too fragile for this to happen.
Narratives generally follow a flow that ultimately concludes after a climax. If you give players more agency the most likely outcome is that they follow a path that does not hit the climax.
This seems very pessimistic to me. For a linear storyline, highly dynamic NPCs are likely not going to add to the experience, that is correct. There are however other ways of creating interesting narratives.
A good example is dwarf fortress where the narratives are created very dynamically and the player is free to interact with them as they want. LLMs could add immensely to these types of games.
LLMs need to either be extremely good with insane context or LLM output has to be conditioned by and translated into formal logic, like Prolog or Datalog to make sure it's compatible with the rules and state of the world, and determine which effect player and NPC actions have on the world.
Odd comment. NPCs inventing details for themselves is the draw.
NPCs will be limited by the game requiring the LLM to generate an action ("Move 5 units forward"), and then the game checks that it's a valid action, and rejects invalid ones.
If you don't like metagaming LLMs, don't do that, and enjoy a regular roleplaying experience
NPCs won't have irrelevant backstories. The backstories generated on the fly for them by an LLM will be incorporated into the overall story.
The overall story climax will happen based on player actions, indicating they're ready for the climax, based on the world created by the player's actions and specific instructions.
I think you're trying to cram the square peg of limitless, custom narration into the round hole of a premade story. You can experience a decent roleplaying session already with an LLM you can run on your computer. Compute is only going to get cheaper, and LLM capabilities will only increase.
That's all as true as the fact there's a gun over the mantelpiece, at least if you're making Murder on the Orient Express or, you know, Horror on the Orient Express, and I'm all for some good old railroading with Victorian lampshades but I suspect that the designers interested in LLMs-as-NPCs are interested in some kind of open-world, immersive experience where PCs get to poke and prod every inch of the game world.
The truth is that most people are used to games as these formulaic simulations with rigid rules. That's partly making a virtue out of necessity (very hard to play a game without rules; and some of us just like rules), and partly habit. Maybe LLMs can act as something new in that sense.
Bang bang.
Games with substantial roles for LLMs may be more like human GMed TTRPGs and less like traditional static-structured computer games, with less fixed narratives.
I would have thought so too, until about 2023.
But, if recent attitudes to AI are anything to go by, this will be immediately seen as boring and derivative at best, or at worst people will accuse the back stories as having been "stolen" from other works of fiction.
While my expectations for the capabilities of AI have never been higher, my expectations for how the public will perceive this amazing technology has never been lower. It's all very disappointing.
And that's without specifically placing blame by the way, perhaps it's big techs fault for going about this the wrong way, I don't know, but the public blowback against AI is just making working in this field that I used to love super depressing.
My 2 cents on why I think at the moment that I would find AI-driven games boring in the future (to clear confusion I think games are art):
- Amazing art is rare. Finding this amazing art nowadays is hard, with AI generated mumbojumbo increasing the noise it will be harder. It will be easier to just skip all AI generated stuff to find the real good stuff.
- Art has human touch or a message almost always in it. Purely AI generated stuff lacks meaning, because there's no creator.
- Socially sharing experiences that are unique only to you will be hard. I like to talk with my friends about the newest Nolan film or about Elden Ring's best bosses. If the whole game experience would be unique to me I wouldn't be able to relate with my friends at all and it would be the same as playing a different game.
- Valuing human work. I idolize people who have the skills to make great art. AI takes away from it.
But to these points, I can also see the future where
- AI could generate amazing art a lot, and would make me reconsider my categorical exclusion of AI art.
- AI could make it seem there's meaning behind the art, or an artist would use the AI as a tool to convey his meaning. - Unique game experiences would be so wild that you would be sharing them and relating via that way.
- AI becomes so common that valuing human skills isn't important anymore
Last point, I think novelty is something I value a lot and that's why in my opinion amazing art is rare. If AI would start creating a lot of novel art, novelty and rarity would just become common, and common is boring.
Unfortunately, creative and interesting use of the technology is both not how its being portrayed and also, critically, not how its being marketed and/or used.
The current expectation is that these models allow the creation of equivalent artwork to what artists are currently producing, but at larger scale / lower cost / shorter training. All of these boil down to less effort. And perceived effort is a large part of what brings value to art.
Imagine having NPCs with prompt set character traits and set boundaries of knowledge of the game world.
The player can interact freely with the NPC rather than through predetermined conversation decision trees, potentially driving a lot of unique emergent gameplay.
Downsides are that it is a lot quicker just to skip through conversations with the decision tree format and having to think about and type in questions could get tiresome. Another downside is players jailbreaking the LLM...
Yes, I look forward to not have the NPC repeat the same exact thing every time, as you pass them by.
How much framerate are you willing to give up so that NPCs don't repeat themselves? The elephant in the room with LLMs in games is that the users GPU is usually already being maxed out by pushing pixels, so any amount of compute allocated to running AI models instead is going to have a very tangible cost.
They could be offloaded to the cloud instead but that brings its own set of issues (ongoing upkeep costs, finite capacity, servers will inevitably shut down...)
Instead you'll just get variations on "Sorry, but as a large language model trained by OpenAl, I cannot condone the shooting of adventurers in the knees with arrows."
Any production deployment would probably do pre-fetching of potential initial dialogues as some kind of background job.
Now just imagine what we could do with a purpose-built ML model!!!
It could also work well for modding, for example, creating a natural dialogue system in Morrowind (replacing the vanilla "interactive signpost" NPCs).
I don't believe this is going to actually make it into games I have any interest in playing. Ubisoft might include it for a bit in some awful games, but it will fizzle out quick.
For the sake of argument, let's imagine it actually did make it into some game you'd want to play. It's well known that the quality of generation from these LLM's is heavily dependent on the quality of the text you put into it. Does that mean the quality of the game narrative is now going to be dependent on how I play it? Will normal people who just like to mess about get a worse game experience than the guy following the traffic laws in GTA? What's the creative talent even doing in that world?
In contrast to runtime stuff (like an AI dungeon master) I've long wished for authoring tools to help people create rich game-worlds and quest interactions. On reflection, that's something I wouldn't expect LLMs to deliver either: It relies heavily on modeling cause-and-effect, with choices which are blocked or enabled by other choices.
I'm talking about stuff that could throw up a warning like: "Warning: Broken quest steps in For The Want of a Nail. Invariant Character:Blacksmith may have been killed by 5 other paths."
I wonder if models trained on the real world will unintentionally destroy the game experience for people. In (NPC structured) video games things always work out ideally, you invest and you get a return. Any risk is illustrated for the user.
With LLM training, the NPC might take a more realistic approach and inform you that 4 of 6 the beetle nuts you collected for them are cracked and useless, and next time you should be more careful packing them in your backpack.
> authoring tools to help people create rich game-worlds and quest interactions
I built one of those things. It used random generation from datasets to create everything from gods and their pantheons to their followers, cities, and the cultures that make up the cities.
None of it used LLMs and it created better content than any model could, but it was hard to debug and harder still to figure out "What even is a culture?"
That's a more interesting usecase. But what you're describing is just an IDE, there's nothing AI about that.
There's a bigger (but related) problem than text quality:
With a usual game environment, the devs create a whitelist of actions the player is allowed to perform. If there's no combat, they simply don't give the player the option to perform violent actions. If they don't want the player to vault barriers they just don't include a jump button. Vehicles and buildings are props by default and only become intractable because the devs consciously chose to make them so. Devs have generally gotten very good at implying what's out of scope, so players generally don't even attempt actions that are intended to be impossible.
In LLM world with free-text entry, controlling what's on-theme becomes a blacklist rather than a whitelist, as the player can choose to input anything. Including things for flavour or lore reasons now becomes more difficult, as a whitelist is finite, while the blacklist of undesired actions is practically infinite.
I don't quite buy that contrast. I agree as long as we are talking story. The developers will have a very difficult time trying to control what some character is like, and if that matters for the gameplay it's a big problem. On the gameplay front the available actions will always be a whitelist, because somebody has to implement them. It's essentially the same problem the crypto kids had with their "cross game non-fungible items", where they just assumed that items would somehow be portable. Just because you can get some NPC to tell you about a dragon in the nearby cave doesn't make that cave actually appear. You're never going to get to fight a darksouls boss in forza horizon. The game might tell you that you get to, that doesn't make it true.
That's still a problem though, right?
If the NPCs are writing cheques that the engine can't cash, the player won't necessarily be able to trust any NPC dialogue.
Something related is compellingly explored in an interesting trilogy of LitRPG comedy books called the Vaudevillain series, consisting of Top Hat Express, Black Tie Villainy, and Cane Whirling Lunacy. In the series, there is a VR video game called “World of Supers” where players can type in any superpower they can dream of into a game-master LLM, and the LLM will then create a character for them which has that power, but reworked a little bit to fit into game balance. For example, the player could type “I can stop time and still move around and take actions for up to 90 seconds” and the game-master LLM may approve this power but alter 90 seconds to 10 seconds, and give the player a major weakness, such as “However, stopping time in the vicinity of a gamma ray source will result in immediate loss of 75% of HP due to the interaction of the time stop with the radiation’s effects. This HP loss can be mitigated if you are encased in a significant amount of lead, such as a lead vehicle or heavy lead armor.” It's a pretty neat concept for how LLMs could allow custom player abilities while maintaining game balance. That way, player agency extends to actual powers/abilities, not just dialogue.
That would of course require an LLM to understand game balance, as opposed to just writing things that sound like they might balance a hypothetical game. At that point it's less similar to our LLMs and more like an AGI. And at that point we may as well have it create the neurotech necessary for the sort of VR tech I assume is used in the referenced books' world.
Agree. This point is a big concern and is explored in the stories (the protagonist creates a "mad scientist" character that can use the LLM to invent new equipment, and then uses a lot of trial and error to find prompts that generate insanely overpowered equipment). It sounds fun to play though! Balance doesn't have to be .. well .. balanced if there are other ways to maintain fun, such as leaning heavily into the creativity & self-actualization angle. As an example, if I could have any superpower in real life, I'd love to be able to fly, even though that's actually a very weak power compared to things like teleporting that are mostly a superset of flying, but not as fun and whimsical.
Encouraging "living your dream" rather than being the most powerful might be easier to achieve in a game with coop or open-world objectives rather than one focused on PvP, but if "living your dream" could be compelling made into the primary player objective, it would pair very well with the custom powers mechanic envisioned in the story.
As an example of how this is explored in the story, one of the protagonist's friends who loves surfing has a character whose superpower is "the ability to surf anything". It's an extremely weak power but the character really loves it because they get to surf the most extreme things imaginable, just for the fun of it, including in outlandish scenarios such as surfing along the blast wave of an explosion, or even the beam of a laser - silly stuff that is fun to explore & try.
I think it wouldn't have to be VR to be fun - imagine if it was something like a Roblox-based game played on laptops. Even then, the "surf anything" power and other outlandish powers would be incredibly fun to use. Imagine if you could do almost anything! You don't have to be the most powerful player in the world to have fun. Maybe you just really want to pilot a mecha-Godzilla, or something - even though an Iron Man or Thor level super could smash it in seconds. It's still awesome!
Finally we can have interesting games with real meaningful choices that actually change the narrative of the game. Modern MMORPGs are barely better than the Choose Your Own Adventure books.
Every time I watch Chris Crawford's Dragon Speech: https://www.youtube.com/watch?v=VwZi58u1FjI I think about what could've been, and what games actually turn out to be.
Honestly, if the next The War Within (next World of Warcraft expansion) makes me do 10 daily quests per day again, I will cancel my subscription for the first time since 2005.
You need focus and direction in any media. Books, movies, games, etc.... they all need it.
The first NPC that you encounter with such freedom of interaction is interesting, the 45th will be annoying and you'll just want to get to the point. The holodeck of Star Trek is often brought up as an excellent example of AI generated interactions, but people seem to be forgetting that the stories we see unfold on TV aren't random interactive stories. They're scripted stories written by show's writers.
Unless you want to make a simulation without a real goal, I don't see how this could lead to fun and interesting gameplay past the first hour. Realism is, always will be, a bad gameplay pillar just for the sake of it.
As my kids got some recent Choose Your Own Adventure books, I'd say that the quests in modern MMORPGs tend to be significantly worse than that.
I've also done research in the use of LLMs for RPGs, using Disco Elysium as a testbed (rather than the small made up scenario seen in this paper).
Ah, my friend created guides for soloing (or play with a friend without a GM) TTRPGs using ChatGPT, here's his site: https://oracle-rpg.com/
The Discord is pretty active too.
I forwarded him this study, he'll appreciate it!
The criticisms people are laying out about being able to say anything making games boring would apply to D&D as well. Since D&D is really fun despite the freedom to (attempt) to say or do whatever your imagination can dream up means it's at least plausible that a game with generative dialogue and narrative could work well.
Just because you're free to speak doesn't mean NPCs will listen, nor does it mean there aren't rules or that there won't be consequences.
D&D is fun because it is a social experience. There’s also a level of expected player etiquette. If you want to go off the rails and completely ignore the party and plot, you’ll probably be kicked out of the group. D&D is fun, but the vast majority of campaigns are cut short and unfinished.
A game with generative dialogue could work well. But one should expect that slapping it into most games will work horribly.
Slapping any mechanic in without thought rarely produces a good game.
> If you want to go off the rails and completely ignore the party and plot, you’ll probably be kicked out of the group.
Sounds like a great place an AI can fill in as is DM for otherwise toxic players.
The real bummer is that if you want to get financing to make a game right now, putting a PDF onto arxiv.org will get you closer to that goal than actually making a game.
I have not sat down and thought through how LLMs would be made use of in the World of Baldurs Gate 3. But when I do, my mind will be boggled.
It will surely also mean always online games with a hefty subscription fee.
Unless!!! The llm can run locally on the player gpu... Whoa mind boggled. This is coming, I'm sure of it.
/end internal thought process
All LLMs in that regime can be run on mid-range gaming hardware today.
LLMs run off-prem/off-desk are already niche for enthusiasts. You need Opus or 0125 sometimes, but not often.
Apple just announced their intention to make running them in the cloud niche generally, at least for mundane consumer use cases, work out of the box.
Can you write a single line of dialogue that you think would make the game better?
Good job. More LLMs in Games please!
A lot of the dialogue that OpenAI's models write is incredibly bland.. I really think we'll need less censored models trained on how to act different roles other than just 'super safe and friendly assistant'.
Not as in depth as the abstract, but I found this to be a well integrated use of LLM's into a game: https://youtu.be/sUL0Z7PwanY
Hey, this currently has a bug but I want to share here.
The framework should be clear.
I’ve written a React pattern for writing interactive fiction using LLMs.
Here’s a demo with Taskmaster roleplay implemented.