▲ 188 points
back
146 comments
I think that most people are underestimating Nvidia strategy.
Their bet is that AI will unlock robotics use and they don't want to be simply compute providers, they want to innovate on the whole chain, software, hardware, services, everything.
Their position is quite unique as their R&D is basically financed by their future competitors, they are making bank while going where the puck will be.
I think Nvidia should try to create an compute-analogue of wifi routers where computation will be offloaded to smart-home gpu-servers. This strategy will cement their future for eternity.
The hard part to pull off with this strategy is that a truly wide spread "robot" platform I think depends on the commodification of the IP driving the platform.
I think there are many early innovators that fail in later stage growth because of this issue.
> they don't want to be simply compute providers
I'm still surprised they did not create an App Store for AI. Basically lock everything down and make developers pay a % of their revenue, Apple style.
What is the new breakthrough in robotics that is gpu driven ? There are subsets of the overall problem that can be solved by a gpu (eg object detection) but the whole planning and control algo scheme seems to be more or less the same as it has been for the past decades. These typically involve non-convex optimization so not much gpu benefit.
Two decades ago, I was trying to use classical machine vision to tell the difference between cut and uncut grass, to guide a self-driving lawnmower.
I concluded that it couldn't be done with classical machine vision, and that this "neural network" nonsense wasn't going to catch on. Very slow, computationally inefficient, full of weirdos making grandiose claims about "artificial intelligence" without the results to back it up, and they couldn't even explain how their own stuff worked.
These days - you want to find the boundary between cut and uncut grass, even though lighting levels can change and cloud cover can change and shadows can change and reflections can change and there's loads of types of grass and grass looks different depending on the angle you look from? Just label some data and chuck a neural network at it, no problemo.
> What is the new breakthrough in robotics that is gpu driven ? There are subsets of the overall problem that can be solved by a gpu (eg object detection) but the whole planning and control algo scheme seems to be more or less the same as it has been for the past decades. These typically involve non-convex optimization so not much gpu benefit.
In the past two years two very important developments appeared around imitation learning and LLMs. Some starting points for this rabbit hole:
1. HuggingFace LeRobot: https://github.com/huggingface/lerobot
2. ALOHA: https://aloha-2.github.io/
There are search spaces that are quite large that are used in optimal control. GPUs can be used to drastically accelerate finding a solution.
As an example, imagine you are given a height map, a 2D discrete search space overlayed in the height map, 4 legs, and robot dynamics for every configuration of the legs in their constrained workspace. Find the optimal toe placement of the 4 legs. Although a GPU isn't designed exactly to deal with this sort of problem, if it's framed as a reduction problem it still significantly out performs a multi core CPU.
I think it is what nobody has answered yet.. virtualized training/testing. I watched a presentation by their research team. This is a HUGE force multiplier. Don't underestimate how much this changes robotic foundational model training.
VLA - Vision Language Action models
https://arxiv.org/abs/2406.09246
It turns out you can take a vision language foundational model that has a broad understanding of visual and textual knowledge and fine tune it to output robot actions given a sequence of images and previous actions.
This approach beats all previous methods by a wide margin and transfers across tasks.
The article lists 2: firstly, simply that ML models are now feasible at a scale they weren't only a few years ago. Secondly, compute power is now better enough that it can now simulate more realistic environments which enables sim-based (pre)training to work better. That second one is potentially particularly alluring to nvidia given how it plays on two of their unique strengths - AI and graphics.
> What is the new breakthrough in robotics that is gpu driven ? There are subsets of the overall problem that can be solved by a gpu (eg object detection) but the whole planning and control algo scheme seems to be more or less the same as it has been for the past decades
I think the "object detection" goes quite far beyond the classic "objection detection" bounding boxes etc we're used to seeing. So not just a pair of x,y coords for the bounding box for e.g. a mug of coffee in the robot's field of view, but what is the orientation of the mug? where is the handle? If the handle is obscured, can we infer where it might be based on what we understand for what a mug typically looks like and plan our gripper motion towards it (and at 120hz etc)? Is it a solid mug, or a paper cup (affects grip strength/pressure)? Etc etc. Then there is the whole thing about visually show the robot once what you are doing, and it automatically "programs" itself to repeat the tasks in a generalised way etc. Then you could probably spawn 100 startups just on hooking up a LLM to tell a robot what to do in a residential setting (make me a coffee, clear up the kitchen, take out the trash etc)
This has all been possible before of course, but could it be done "on device" in a power efficient way? I am guessing they are hoping to sell a billion or two chips + boards to be built directly into things to do so so that your next robotic vacuum or lawn mower or whatever will be able to respond to you yelling at it and not mangle your pets/small children in the process.
I eagerly await the day when I have a plug and play robot platform that can tell the difference between my young children and a fox, and attack the fox shitting/shredding something small and fluffy in the garden but ignore the kids
The fun thing with DL is that you don't have to optimize stuff with complicated math. You just train it, and it will generate solutions. Maybe not the perfect solutions, but don't let perfect be the enemy of good.
My understanding is that it's in vogue to use deep learning for complex control problems, and the results are fairly impressive. The idea is to train robotic motion end to end with RL. Not an expert so I don't know the strength and weaknesses versus classical approaches.
Unstructured Sensory input driven by these large neural networks, if I had to guess.
To be able to visually determine weight, texture, and how durable something is can be done with those systems so long as we have a training set.
I've been doing this for 10+ years, and have seen GPU-based calculations slowly eat away at the following problems:
* Mapping. Nowadays generating a dense grid of costs can be done insanely fast on GPU. There's just no excuse to not use a GPU on every robot so it can build a fast map, unless you move at snail speed.
* Computer Vision. Classical depth mapping is best done on a GPU. Classical computer vision object detection has fallen away to the rise in ML-based CV for segmentation. Some (IMHO) overzealous practitioners are trying to eat away at estimation and tracking, which IMHO will recede a little since there was nothing wrong with the estimators (just Bayesian stats) to begin with, it was always the measurements. Still, for detection (and sometimes association), ML on GPU is the way to go and that will very likely not change. It has gotten so good that you can get away without using other sensors and just deploying a vision system (though I don't recommend it, but this is what Tesla does). This is an obvious case for one (or one more) GPU on every robot.
* Planning - End to end planning is eating traditional planning now, similar to CV. There are some areas where this is an obvious win (e.g., complex manipulation tasks), and some areas where some overzealous overreach is happening (e.g., simpler planning tasks like routing). But ML on GPUs is here to stay for all planning tasks, especially when estimating costs from complex data, even if a classical planner uses those costs. And I'd be remiss if I didn't mention policy-based planning, which does a huge amount of training to generate essentially a fast lookup table for actions. Deployment of these types of planners often requires a very good estimator to determine what state you are in - and this is a great area for ML, mapping real world messy data to a clean state lookup. I think this can typically be done without a GPU, due to training prior to deployment, but if you have a GPU already (see prior two), you will find this is a good use of it.
* Low-level planning / Controls - Shares a small overlap with above, but mostly concerned with fast responses to transient data and stabilizing the system. I've heard, but not seen directly, that learned policies are coming into vogue here. But regardless, it is a common thread that a network can assist with estimating costs and states to allow a traditional controls system to operate more reliably. I doubt this will necessitate a GPU, but like above, will gladly use it if required and available.
To add to this, consider that we're generally not talking about discrete, gaming-type GPUs, we're talking about purpose built robotics-targeted embedded systems that speak native CUDA. The Jetson family, in particular.
Actually, it is quadratic programming that is big in robotics. QP is powerful enough that you can formulate your task, but also fast enough that you can run it in the control loop in real time.
It's the edge computing.
Similar to autonomous vehicles, doing complex multi sensor things very quickly.
Surgical robotics is a great example, lots of cool use cases coming out in that field.
I personally think apart from GPU and compute for intelligence for meaningful robotics to take off we still have lot of things to crack like better battery, better affordable sensors, microelectronics etc, I'm pretty sure we will get there but I don't think one company can do it.
Better battery isn't really an issue for factories. Same with sensors if you're saving the cost of employing a human, especially for dangerous work.
It’s hard to say when we’re still looking for a first real household robot. But a car-priced (60k?) housekeeper bot will be very popular.
And those duties can be achieved with today’s mechanics — they just need good control, which is now seeing ferocious progress
The ChatGPT, LLMs, generative AI, and other hyped usecases have been the driving force for Nvidia: it injected huge sums of money into their R&D, which also stimulated the economy as developers ran to build build build in order to keep up with the demand for datacenters, which in turn required more infrastructure building to satiate the thirst and power needs of datacenters, etc. Before, ChatGPT, I recall the hype was blockchain, crypto, and NFTs; and maybe before that, it was "big data."
As the LLM, generative AI, etc. bubble begins to deflate due to investors and companies finding it hard to make profits from those AI usecases, Nvidia needs to pivot. This article indicates that Nvidia is hedging on robotics as the next driving force that will continue to sustain the massive interest in their products. Personally, I don't see how robotics can maintain that same driving force for their products, and investors will find it hard to squeeze profit out of it, and they'll be back to searching for another hype. It's like Nvidia is trying to create a market to justify their products and continued development, similar to what Meta has tried, to spectacular failure, with the Metaverse for their virtual products.
After the frenzy that sustained these compute products transitioned from big data, to crypto, and now, to AI, I'm curious what the next jump will be; I don't think the "physical AI" space of robotics can sustain Nvidia in the way that they're hoping.
The part that is hard for me to parse is there is hype but there is also a significant amount of value being extracted by using LLMs and other products coming from this new wave. Everytime I read opinions like yours it’s hard to make sense of it because there is value in the tooling that exists. It cannot be applied to everything and anything but it does exist.
Comparing AI to crypto doesn't really work due to the utility of AI. If you believe that there haven't been meaningful use cases from the recent generative AI surge, then you might be out of touch.
On the investment side, it's hard to say that since ROIC is still generally up and to the right. As long as that continues, so will investment.
Then biggest gap I see is expected if you look at past trends like mobile and the internet: In the first wave of new tech there's a lot of trying to do the old things in the new way, which often fails or gives incremental improvements at best.
This is why the 'new' companies seem to be doing the best. I've been shocked at so many new AI startups generating millions in revenue so quickly (billions with OpenAI, but that's a special case). It's because they're not shackled to past products, business models, etc.
However, there are plenty of enterprise companies trying to integrate AI into existing workflows and failing miserably. Just like when they tried to retrofit factories with electricity. It's not just plug and play in most cases, you need new workflows, etc. That will take years and there will be plenty more failures.
The level of investment is staggering though, and might we see a crash at some point? Maybe, but likely not for a while since there's still so much white space. The hardest thing with new technologies like this is not to confuse the limits of our imagination with the limits of reality (and that goes both ways).
I have worked with Jetson Orin platform, and honestly Nvidia has something that is really easy to work with there. The Jetsons are basically a full GPU (plus some stuff) at very low power. If I were tasked with building a robot it would likely be the first place I look.
They are OK. If you need advanced vision - yes, because CUDA.
But off the shelf mini PCs are much more user friendly for existing software IME.
Thankfully ARM being so wide spread and continuing to grow this wont matter as much.
After using FSD 13 for 2 weeks I'm convinced we are close to solving self driving. Too bad the everyone lost interest and now robotics is the hot new thing.
As someone who worked in V&V for AV systems for a decade, it’s exactly the kind of thinking displayed here that has held back real assessment of AV safety for years.
There is absolutely no meaningful signal about a system’s safety that can be derived from one person using a system for two weeks.
At best it can only demonstrate that a system is wildly unsafe.
There is a very large chasm of 9s between one person being able to detect an unsafe system in two weeks of use and actually having a truly safe system.
And it only takes a (near) accident in 5 more minutes' driving to completely negate that.
Your observation from this short time window isn't enough to prove the usefulness of something as serious as life and death.
I’m not sure if you’re generalizing to a specific region in your assessment but regardless, I doubt this is anywhere close to a solved problem given the crashes/incidents (so far) still associated with the tech and the dependencies IIRC on street signs and other markers.
re: region, I’d like to see it take on more challenging conditions, like in India for example where things are chaotic even for human drivers. I doubt that it’ll survive over here.
> Too bad the everyone lost interest and now robotics is the hot new thing.
Self driving is robotics. Simple as that.
Note that Nvidia is also working on self driving. The Jetson robotics platform is based on the same SoC as the DRIVE platform, but is a separate product.
Although the idea of self driving is obviously cool I think it's good that robotics take priority (if such a thing is possible) e.g. think of it like the invention of the washing machine as a liberating force on the world.
Have you been a passenger in a Waymo? My only ride felt safer than every uber / Lyft driver I have ever had pretty much, so wondering how it compares to a beta thing you have to be able to take over in an instant.
Cars are robots without arms
Waymo already solved self driving years ago. Tesla still has a long way to go.
Now if we could get a robotics platform like ROS that actually cares about modern dev patterns and practices from dev's slapping keyboards through production deployment with decent smoke tests, easy versioned artifacts and no need to understand linux packaging details...
Coming from web / app dev this was my very least favorite part of working on the software side of robotics with ROS.
Robotics has long been an area of promise but (I think) limited returns
I just finished reading Daron Acemoglu and Simon Johnson’s book “Power and Progress” where they talk about how the leaders in the technology space is (unfortunately) able to set the direction of the technology according to their goals, not humanity’s goals. This is an excellent example of such power. NVIDIA wants to expand its business and pushes the industry to use more and more AI, which highly depends on their cards. Now all the VCs put billions of dollars towards this goal, thousands of Phds spend all their time, and companies change direction of business to catch the AI hype. Not necessarily because we decided this is the best for humanity, just because it’s the best for NVIDIA.
Sounds like NVIDIA doesn't know what the hell the future is going to look like but hopes it's something to do with robotics, and is taking some the boatloads of money from the past few years to build out product lines for every conceivable robotics need. Good for them I guess.
The article references a "ChatGPT moment" for physical robotics, but honestly I think the Chat GPT moment has kind of come and gone, and the world still runs largely as it ever did. Probably not the best analogy, unless they're just talking about buckets of VC money flowing into the space to fund lots of bad ideas, which would be good for NVIDIA financially.
As an admitted non-expert in this field, I guess the one thing that really annoys me about articles like this is the lack of a concrete vision. It's like Boston Dynamics and their dancing robots, which while impressive, haven't really amounted to much outside of the lab. The last thing I remember reading was a military prototype to carry stuff for infantry that ended up being turned down because it was too loud.
The article even confirms this general perspective, ending with "As of right now, we don’t have very effective tools for verifying the safety and reliability properties of machine learning systems, especially in robotics. This is a major open scientific question in the field,” said Rosen."
So whatever robot you're developing is incredibly complex, to be trusted with heavy machinery or around consumers directly, while being neither verifiably safe nor reliable.
Sorry, but almost everything in this article sounds like a projection of AI-hype onto physical robotics, with all the veracity of "this is good for Bitcoin". Sounds like NVIDIA is doing right by its shareholders though.
took them 16 years to fix night light bug in the driver but yeah robots are the future
if they really want to bet on robotics, I want them to release a $10 variant of jetson board.
What breakthrough are people expecting in robotics I wonder?
Since cops, guards and military officers are itching to get autonomous guns it's probably a reasonable move. The genocide of palestinians has showed that people operated gun drones aren't distance enough, the operators cost a lot in psych treatment and personnel churn.