▲ 248 points
back
163 comments
An important caveat from the paper
>Moreover, we follow previous work in accelerating block breaking because learning to hold a button for hundreds of consecutive steps would be infeasible for stochastic policies, allowing us to focus on the essential challenges inherent in Minecraft.
Like all things RL, it is 99.9% about engineering the environment and rewards. As one of the authors stated elsewhere here, there is a reward for completing each of 12 steps necessary to find diamonds.
Mostly I'm tired of RL work being oversold by its authors and proponents by anthropomorphizing its behaviors. All while this "agent" cannot reliably learn to hold down a button, literally the most basic interaction of the game.
And a relevant piece of ancient wisdom (exact date not known, but presumably before 1970):
> In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.
> “What are you doing?”, asked Minsky.
> “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied.
> “Why is the net wired randomly?”, asked Minsky.
> “I do not want it to have any preconceptions of how to play”, Sussman said.
> Minsky then shut his eyes.
> “Why do you close your eyes?”, Sussman asked his teacher.
> “So that the room will be empty.”
> At that moment, Sussman was enlightened.
Well, to be fair... I (a human) had to look it up online the first time I played as well. I was repeatedly clicking on the same tree for an entire minute before that. I even tried several different trees just in case.
In my mind, this generalizes to the same problem with other non-stochastic (deterministic) operations like logical conclusions (A => B) .
I have a running bet with friend that humans encode deterministic operations in neural networks, too, while he thinks there has to be another process at play. But there might be something extra helping our neural networks learn the strong weights required for it. Or the answer is again: "more data".
"accelerating block breaking because learning to hold a button for hundreds of consecutive steps "
This is fine, and does not impact the importance of figuring out the steps.
Anybody that has done any tuning on systems that run at different speeds, the adjusting for the speed difference is just engineering, and allows you to get on with more important/inventive work.
I'm not sure it's a serious caveat if the "hint" or "control" is in the manual.
Turns out that AI are much better at playing video games if they're allowed to cheat.
"It allows AI to understand its physical environment and also to self-improve over time, without a human having to tell it exactly what to do."
Key to Dreamer’s success, says Hafner, is that it builds a model of its surroundings and uses this ‘world model’ to ‘imagine’ future scenarios and guide decision-making.
Can you look at the world model, like you can look at Waymo's world model? Or is it hidden inside weights?
Machine learning with world models is very interesting, and the people doing it don't seem to say much about what the models look like. The Google manipulation work talks endlessly about the natural language user interface, but when they get to motion planning, they don't say much.
Yes, you can decode the imagined scenarios into videos and look at them. It's quite helpful during development to see what the model gets right or wrong. See Fig. 3 in the paper: https://www.nature.com/articles/s41586-025-08744-2
I implemented an acoustic segmentation system in FPGA recently. The whole world model was a long list of known events and states with feasible transitions. Plus novel things not observed before. Basically rather dumb state machine with machine learning part attached to acoustic sensors. Of course, both parts could be hidden behind weights. But state machine was easily readable and that was the biggest advantage of it.
I’d say it’s more like Waymo’s world model. The main actor uses a latent vector representation of the state of the game to make decisions. This latent vector at train time is meant to compress a bunch of useful information about the game. So while you can’t really understand the actual latent vector that represents state, you do know it encodes at least the state of the game.
This world model stuff is only possible in environments that are sandboxed. Ie you can represent the state of the world in an and have a way of producing the next state given a current state and action. Things like Atari games, robot simulations, etc
> Can you look at the world model, like you can look at Waymo's world model? Or is it hidden inside weights?
I imagine it's the latter, and in general, we're already dealing with plenty of models with world models hidden inside their weights. That's why I'm happy to see the direction Anthropic has been taking with their interpretability research over the years.
Their papers, as well as most discussions around them, focus on issues of alignment/control, safety, and generally killing the "stochastic parrot" meme and keeping it dead - but I think it'll be even more interesting to see attempts at mapping how those large models structure their world models. I believe there's scientific and philosophical discoveries to be made in answering why these structures look the way they do.
Article makes it seem like finding diamonds is some kind of super complicated logical puzzle. In reality the hardest part is knowing where to look for them and what tool you need to mine them without losing them once you find them. This was given to the AI by having it watch a video that explains it.
If you watch a guide on how to find diamonds it's really just a matter of getting an iron pickaxe, digging to the right depth and strip mining until you find some.
Hi, author here! Dreamer learns to find diamonds from scratch by interacting with the environment, without access to external data. So there are no explainer videos or internet text here.
It gets a sparse reward of +1 for each of the 12 items that lead to the diamond, so there is a lot it needs to discover by itself. Fig. 5 in the paper shows the progression: https://www.nature.com/articles/s41586-025-08744-2
While I agree with your comment, this sentence:
"This was given to the AI by having it watch a video that explains it."
This was not as trivial as it may seem just a few months ago...
>This was given to the AI by having it watch a video that explains it.
That is not what the article says. It says that was separate, previous research.
I don't get it. How can you reduce this achievement down to this?
Have you gotten used to some ai watching a video and 'getting it' so fast that this is boring? Unimpressive?
The AI is able to learn from video and you don't find that even a little bit impressive? Well I disagree.
The “holding a button” thing actually resonated. It feels like the real work here is engineering the reward structure to make exploration even remotely viable. Dreamer’s world model might be cool, but most of the heavy lifting still seems to come from how forgiving the Minecraft environment is for training.
I do wonder though: if you swapped Minecraft for a cloud-based synthetic world with similar physics but messier signals, like object permanence or social reasoning, would Dreamer still hold up? Or is it just really good at the kind of clean reward hierarchies that games offer?
Characterizing finding diamonds as "mastering" Minecraft is extremely silly. Tantamount to saying "AI masters Chess: Captures a pawn." Getting diamonds is not even close to the hardest challenge in the game, but most readers of Nature probably don't have much experience playing Minecraft so the title is actually misleading, not harmless exaggeration.
> Getting diamonds is not even close to the hardest challenge in the game
Mining diamonds isn't even necessary if you build, e.g., ianxofour's iron farm on day one and trade that iron[0] with a toolsmith, armourer, and weaponsmith. You can get full diamond armour, tools, and weapons pretty quickly (probably a handful of game weeks?)
[0] Main faff here is getting them off their base trade level.
I agree with you, this is just the start and Minecraft has a lot more to offer for future research!
I didn't know that Nature did movie promotions.
Reinforcement learning is very good with games.
>> In Minecraft, the team used a protocol that gave Dreamer a ‘plus one’ reward every time it completed one of 12 progressive steps involved in diamond collection — including creating planks and a furnace, mining iron and forging an iron pickaxe.
And that is why it is never going to work in the real world: games have clear objectives with obvious rewards. The real world, not so much.
For a lot of things, VLMs are good enough already to provide rewards. Give them the recent images and a text description of the task and ask whether the task was accomplished or not.
For a more general system, you can annotate videos with text descriptions of all the tasks that have been accomplished and when, then train a reward model on those to later RL against.
Plenty of real world situations have clear objectives with obvious rewards.
> And that is why it is never going to work in the real world: games have clear objectives with obvious rewards. The real world, not so much.
I encourage you to read deepmind's work with robots.
Who would have thought you could get your TAS run published in Nature if you used enough hot buzzwords. (they have been using various old-school-definition "artifical intelligence" algorithms for a long time)
Minecraft is ubiquitous now.
But I remember the alpha version, and NOBODY knew how to make a pick ax. Humans were also very bad at figuring out these steps.
People were de-compiling the java and posting help guides on the internet.
How to break a tree, get sticks, make a wood pick. In Alpha, that was a big deal for humans also.
Slightly off-topic from the article itself, but… does anyone else feel like Nature’s cookie banner just never goes away? I have vivid memories of trying to reject cookies multiple times, eventually giving up and accepting them just to get to the article only for the banner to show up again the next time I visit. I swear it’s giving me déjà vu every single visit.. Am I the only one experiencing this, or is this just how their site works?
Could this perform better by having the internal representation of Minecraft instead of raw pixels?
It seems rather tenuous to keep pounding on 'training via pixels' when really a game's 2D/3D output is an optical trick at best.
I understand Sergey Brin/et al had a grandiose goal for DeepMind via their Atari games challenge - but why not try alternate methods - say build/tweak games to be RL-friendly? (like MuJoCo but for games)
I don't see the pixel-based approach being as applicable to the practical real world as say when software divulges its direct, internal state to the agent instead of having to fake-render to a significantly larger buffer.
I understand Dreamer-like work is a great research area and one that will garner lots of citations for sure.
Finally a use case for AI
Isn't this DeepMind achievement from 2023?
Pretty impressive. Minecraft’s a complex environment, so for an AI to figure out how to find diamonds on its own shows real progress in learning through exploration — not just pattern recognition.
Attempting to train this on a real workload I converted over the weekend after, "step" 8M~ so far and rarely scores above 5% and most are 0% but has scored 60% once 7M~ steps ago.
Adding more than 1 GPU didn't improve speed but that's pretty standard as we don't have fancy interconnect. Bit annoying they didn't use tensorboard for logging, but overall seems like a pretty cool lib - will leave it a few days and see if it can learn (no other algo has so I dont have much hope).
This looks like an article about the recent Nature publication. Was confused at first because DreamerV3 is a couple of years old now
There's a YouTube channel that does a lot of videos focused on LLMs in Minecraft:
https://www.youtube.com/@EmergentGarden
I very much like the comparative approach this guy takes looking at how different LLMs fare... including how they interact together. Worth a look.
It’s still being a stochastic parrot. Now it’s just parroting the human creativity and imagination so I’m still not impressed.
If all you’re going to do is parrot things like human consciousness or human ingenuity then I will never be impressed so long that it’s just parroting.
They write: "Below, we show uncut videos of runs during which Dreamer collected diamonds."
... but the first video only shows the player character digging downwards without using any tools and eventually dying in lava. What?
How robust is this?
Isn't something like finding dimonds in minecraft something that old-school AI could already do decently?
Isn't "masters" when you build a working copy of Minas Tirith or something like that?
So can i and no one needed to teach me either, but you dont see nature writing articles on it...
I guess we can look forward
to a bright future
where we focus 100% on work
and AI will play our games
/s