Item — HN Reader

> But I am a bit reasurred that at least my job won't be fully replaced with AI :)

I honestly can't comment with certainty that training from videos alone and whatever tokenization scheme they're using will ever get perfect dynamics.

However it is worth noting that transformers can do a pretty good job at learning dynamics with the right pipeline (not video): https://arxiv.org/pdf/2605.15305 https://arxiv.org/pdf/2605.09196

My point here being that representationally, it might be possible to learn good dynamics without a radically different approach/arch. There are already models that extract 3D tracking points from videos, so they could possibly be leveraged for learning dynamics (which on its own gives precedent for end-to-end approaches also possibly working).

1 comments

Thanks for the additional reading. I've often thought about LLMs and their ability to represent the physical world with its laws. And always concluded it is not really possible to do so with "just" text tokens and their relations in a latent space. It looks to me there are different approaches being taken to tackle this:

* You could instruct your LLM to interact with a simulator to run experiments and infer behaviour

* You could edit the transformer model and inject spatially relevant data rather than text as is done in above paper

* You could change the architecture to be more condusive for representating a world state. I.e., LeCun's JEPA world model.

* You could further enhance some of the above by using a differentiable physics engine (eg. NVIDIA Newton) to calculate losses directly.

But at the end of the day if a model has any hope to always produce realistic physics, it HAS to learn the laws of nature in some form or other. It looks to me that the next big leap could be achieved by combining the last two approaches.

P.S.: I like discussing such topics. If anyone knows a forum or discord with like-minded people, please let me know :)

> P.S.: I like discussing such topics. If anyone knows a forum or discord with like-minded people, please let me know :)

Unironically twitter (and only use the "Following" tab as opposed to the "For You")

Make an account that only follows university affiliated researchers with less than 1000 followers. In my experience discord servers get suffocated by beginners and crackpots because conversations don't naturally self-organize into their own threads.

Thanks, I'll try using the "Following" tab. I have a lurker account but never really used it because I only ever saw crap in "For You".