Was raytracing a psyop by Nvidia to lock out amd? Games today don't look that much nicer than 10 years ago and demand crazy hardware. Is raytracing a solution looking for a problem?
back
10 comments
I've kind of wondered about this a bit too. The respective visual quality side of it that is. Especially in a context where you're actually playing a game. You're not just sitting there staring at side by side still frames looking for minor differences.
What I have assumed given then trend, but could be completely wrong about, is that the raytracing version of the world might be easier on the software & game dev side to get great visual results without the overhead of meticulous engineering, use, and composition of different lighting systems, shader effects, etc.
For the vast majority of scenes in games, the best balance of performance and quality is precomputed visibility, lighting and reflections in static levels with hand-made model LoDs. The old Quake/Half-Life bsp/vis/rad combo. This is unwieldy for large streaming levels (e.g. open world games) and breaks down completely for highly dynamic scenes. You wouldn't want to build Minecraft in Source Engine[0].
However, that's not what's driving raytracing.
The vast majority of game development is "content pipeline" - i.e. churning out lots of stuff - and engine and graphics tech is built around removing roadblocks to that content pipeline, rather than presenting the graphics card with an efficient set of draw commands. e.g. LoDs demand artists spend extra time building the same model multiple times; precomputed lighting demands the level designer wait longer between iterations. That goes against the content pipeline.
Raytracing is Nvidia promising game and engine developers that they can just forget about lighting and delegate that entirely to the GPU at run time, at the cost of running like garbage on anything that isn't Nvidia. It's entirely impractical[1] to fully raytrace a game at runtime, but that doesn't matter if people are paying $$$ for roided out space heater graphics cards just for slightly nicer lighting.
[0] That one scene in The Stanley Parable notwithstanding
[1] Unless you happen to have a game that takes place entirely in a hall of mirrors
Yep. I worked on the engine of a PS3/360 AAA game long ago. We spent a long of time building a pipeline for precomputed lighting. But, in the end the game was 95% fully dynamically lit.
For the artists, being able to wiggle lights around all over in real time was an immeasurable productivity boost over even just 10s of seconds between baked lighting iterations. They had a selection of options at their fingertips and used dynamic lighting almost all the time.
But, that came with a lot of restrictions and limitations that make the game look dated by today’s standards.
I get the pitch that it is easier for the artists to design scenes with ray-tracing cards. But I don’t really see why we users need to buy them. Couldn’t the games be created on those fancy cards, and then bake the lighting right before going to retail?
(I mean, for games that are mostly static. I can definitely see why some games might want to be raytraced because they want some dynamic stuff, but that isn’t every game).
The player can often have a light, and is usually pretty dynamic.
One of the effects I really like is bounce lighting. Especially with proper color. If I point my flashlight at a red wall, it should bathe the room in red light. Can be especially used for great effect in horror games.
I was playing Tokyo Xtreme Racer with ray tracing, and the car's headlights are light sources too (especially when you flash a rival to start a race). My red car will also bounce lighting on the walls in tunnels to make things red.
It doesn't even have to be super dynamic either, I can't even think of a game that has opening a door to the outside sun to change the lighting in a room with indirect lighting (without ray tracing it). Something I do every day in real life. It would be possible to bake that too, assuming your door only has 2 positions.
When path tracing works, it is much, much, MUCH simpler and vastly saner algorithm than those stacks of 40+ complicated rasterization hacks in current rasterization based renderers that barely manage to capture crude approxinations of the first indirect light bounces. Rasterization as a rendering model for realistic lighting has outlived its usefulness. It overstayed because optimizing ray-triangle intersection tests for path tracing in hardware is a hard problem that took some 15 o 20 years of research to even get to the first generation RTX hardware.
>When path tracing works, it is much, much, MUCH simpler and vastly saner algorithm than those stacks of 40+ complicated rasterization hacks in current rasterization based renderers that barely manage to capture crude approxinations of the first indirect light bounces.
It's ironic that you harp about "hacks" that are used in rasterization, when raytracing is so computationally intensive that you need layers upon layers of performance hacks to get decent performance. The raytraced results needs to be denoised because not enough rays are used. The output of that needs to be supersampled (because you need to render at low resolution to get acceptable performance), and then on top of all of that you need to hallu^W extrapolate frames to hit high frame rates.
And you still need rasterization for ray traced games (even "fully" path traced games like Cyberpunk 2077) because the ray tracing sample count is too low to result in an acceptable image even after denoising. So the primary visibility rendering is done via rasterization (which has all the fine texture and geometry detail without shading), and the ray traced (and denoised) shading is layerd on top.
You can see the purely ray traced part in this image from the post: https://substack-post-media.s3.amazonaws.com/public/images/8...
This combination of techniques is actually pretty smart: Combine the powers of the rasterization and ray tracing algorithms to achieve the best quality/speed combination.
The rendering implementation in software like Blender can afford to be primitive in comparison: It's not for real-time animation, so they don't make use of rasterization at all and do not even use denoising. That's why rendering a simple scene takes seconds in Blender to converge but only milliseconds in modern games.
Not quite correct.
For primary visibility, you don't need more than 1 sample. All it is is a simple "send ray from camera, stop on first hit, done". No monte carlo needed, no noise.
On recent hardware, for some scenes, I've heard of primary visibility being faster to raytrace than rasterize.
The main reasons why games are currently using raster for primary visibility:
1. They already have a raster pipeline in their engine, have special geometry paths that only work in raster (e.g. Nanite), or want to support GPUs without any raytracing capability and need to ship a raster pipeline anyways, and so might as well just use raster for primary visibility. 2. Acceleration structure building and memory usage is a big, unsolved problem at the moment. Unlike with raster, there aren't existing solutions like LODs, streaming, compression, frustum/occlusion culling, etc to keep memory and computation costs down. Not to mention that updating acceleration structures every time something moves or deforms is a really big cost. So games are using low-resolution "proxy" meshes for raytracing lighting, and using their existing high-resolution meshes for rasterization of primary visibility. You can then apply your low(relative) quality lighting to your high quality visibility and get a good overall image.
Nvidia's recent extensions and blackwell hardware are changing the calculus though. Their partitioned TLAS extension lowers the acceleration structure build cost when moving objects around, their BLAS extension allows for LOD/streaming solutions to keep memory usage down as well as cheaper deformation for things like skinned meshes since you don't have to rebuild the entire BLAS, and blackwell has special compression for BLAS clusters to further reduce memory usage. I expect more games in the ~near future (remember games take 4+ years of development, and they have to account for people on low-end and older hardware) to move to raytracing primary visibility, and ditching raster entirely.
Meanwhile raserization is fundamentally incapable of producing the same image.
As a sibling post already mentioned, rasterization based hacks are incapable of getting as accurate lighting as path tracing can, given enough processing time.
I will admit that I was a bit sly in that I omitted the word "realtime" from the path tracing part of my claim on purpose. The amount of denoising that is currently required doesn't excite me either, from a theoretical purity standpoint. My sincere hope is that there is still a feasible path to a much higher ray count (maybe ~100x) and much less denoising.
But that is really the allure of path tracing: a basic implementation is at the same time much simpler and more principled than any rasterization based approximation of global illumination can ever be.
This doesn't hold at all. Path tracing doesn't "just work", it is computational infeasible. It needs acceleration structures, ray traversal scheduling, denoisers, upscalers, and a million other hacks to work any close to real-time.
Except that it isn't like that at all. All you get from the driver in terms of ray tracing is the acceleration structure and ray traversal. Then you have denoisers and upscalers provided as third-party software. But games still ship with thousands of materials, and it is up to the developer to manage lights, shaders, etc, and use the hardware and driver primitives intelligently to get the best bang for the buck. Plus, given that primary rays are a waste of time/compute, you're still stuck with G-buffer passes and rasterization anyway. So now you have two problems instead of one.
I think there's two ways of looking at it. Firstly that raster has more or less plateaued, there haven't been any great advances in a long time and it's not like AMD or any other company have offered an alternative path or vision for where they see 3d graphics going. The last thing a company like nvidia wants is to be a generic good which is easy to compete with or simple to compare against. Nvidia was also making use of their strength/long term investment in ML to drive DLSS
Secondly, nvidia are a company that want to sell stuff for a high asking price, and once a certain tech gets good enough that becomes more difficult. If the 20 series was just a incremental improvement from the 10, and so on then I expect sales would have plateaued especially if game requirements don't move much.
I don't believe we have reached a raster ceiling. More and more it seems like groups are cahoots to push rtx and ray tracing. We are left to speculate why devs are doing this. nvidiabux? easier time to add marketing keywords? who knows... i'm not a game dev.
There's no need for implications of deals between nvidia and game developers in smoke filled rooms. It's pretty straightforward: raytracing means less work for developers, because they don't have to manually place lights to make things look "right". Plus, they can harp about how it looks "realistic". It's not any different than the explosion of electron apps (and similar technologies making apps using html/js), which might be fast to develop, but are bloated and feel non-native. But it's not like there's an electron corp, giving out "electronbux" to push app developers to use electron.
>>We are left to speculate why devs are doing this.
Well, I am a gamedev, and currently lead of a rendering team. The answer is very simple - because ray tracing can produce much better outcomes than rasterization with lower load on the teams that produce content. There's not much else to it, no grand conspiracy - if the hardware was fast enough 20 years ago to do this everyone would be doing it this way already because it just gives you better outcomes. No nvidiabux necessary.
I'm a gamedev as well, also in rendering.
> There's not much else to it, no grand conspiracy
True, in that raytracing is the future. Though I don't think it's a conspiracy rather than just the truth that "RTX" as a product was Nvidia creating a 'new thing' to push AMD out of. Moat building, plain and simple. Nvidia's cards were better at it unsurprisingly, much like mesh shaders they basically wrote the API standard to match their hardware.
And just to make sure Nvidia doesn't get more credit than it deserves, the debut RTX cards (RTX 20 series) were a complete joke. A terrible product generation offering no performance gains over the 10 series at the same price with none of the cards really being fast enough to actually do RT very well. They were still better at RT than AMD though so mission accomplished I guess.
Raster quality is limited by how much effort engine developers are willing to put into finding computationally cheap approximations of how light/materials behave. But it feels like the easy wins are already taken?
All the biggest innovations in "pure" rasterization renderers in the last 10-15 years have actually been some form of raytracing in a very reduced, limited form.
Screenspace Ambient Occlusion? Marching rays (tracing) against the depth buffer to calculate a terrible but decent looking approximation of light occlusion. Some of the modern SSAO implementations like GTAO need to be denoised by TAA.
Screenspace Reflections? Marching rays against the depth buffer and taking samples from the screen to generate light samples. Often needs denoising too.
Light shafts? Marching rays through the shadow map and approximating back scattering from whether the shadowed light is occluded or not.
That voxel cone tracing thing UE4 never really ended up shipping? Tracing's in the name, you're just tracing cones instead of rays through a super reduced quality version of the scene.
Material and light behavior is not the problem. Those are constantly being researched too, but the changes are more subtle. The big problem is light transport. Rasterization can't solve that, it's fundamentally the wrong tool for the job. Rasterization is just a cheap approximation for shooting primary rays out of the camera into a scene. You can't bounce light with rasterization.
Yeah I agree with the last paragraph.
For rasterization to be useful it must approximately do the same thing that light does in the real world. Therefore rasterization that wants to get closer and closer to the real world will have to emulate more and more of the real world.
It will have to cast exactly the rays that rasterization is hoping to avoid.
I don't think it's just about looks. The advantage of ray tracing is the real time lighting done rather than the static baked maps. One of the features I feel that was lost with modern game lighting is dynamic environments. But as long as the game isn't only ray tracing, these types of interactions will stay disabled for the game. Teardown and The Finals are examples of a dynamic environment game with raytraced lighting.
Another example is when was the last time you've seen a game with a mirror that wasn't broken?
Hitman, GTA, both of which use a non-raytraced implementation. More to the point, lack of mirrors doesn't impact the gameplay. It's something that's trotted out as a nice gimmick, 99% of the time it's not there, and you don't really notice that it's missing.
Hitman is an example that contradicts your point about gameplay, guards will see you in mirrors and act appropriately. They'll be doing that for gameplay with a non-graphical method, but you need to show it to the player graphically for them to appreciate the senses available
>Hitman is an example that contradicts your point about gameplay, guards will see you in mirrors and act appropriately.
See:
>It's something that's trotted out as a nice gimmick, 99% of the time it's not there, and you don't really notice that it's missing.
Yeah, it's a nice detail for the 1% of time that you're in a bathroom or whatever, but it's not like the immersion takes a hit when it's missing. Moreover because the game is third person, you can't even accurately judge whether you'll be spotted through a mirror or not.
GTA V's implementation did not work in their cars. Rear view and side view mirrors in cars are noticeably low quality and missing other cars while driving, which is pretty big for gameplay purposes.
Working mirrors are limited to less complex scenes in GTA. Hitman too I believe.
Not if you want better fidelity: the VFX industry for film moved from rasterisation to raytracing / pathtracing (on CPU initially, and a lot of final frame rendering is still done on CPU due to memory requirements even today, although lookdev is often done on GPU if the shaders / light transport algorithms can be matched between GPU/CPU codepaths) due to the higher fidelity possible starting back in around 2012/2013.
It required discarding a lot of "tricks" that had been learnt with rasterisation to speed things up over the years, and made things slower in some cases, but meant everything could use raytracing to compute visibility / occlusion, rather than having shadow maps, irradiance caches, pointcloud SSS caches, which simplified workflows greatly and allowed high-fidelity light transport simulations of things like volume scattering in difficult mediums like water/glass and hair (i.e. TRRT lobes), where rasterisation is very difficult to get the medium transitions and LT correct.
I don’t think the last two decades of computer graphics research are an Nvidia psyop. Ray tracing is relatively simple and also reflects reality (you’re literally doing a light simulation to approximate how we really see). It’s always been the gold standard for rendering, we just didn’t know how to make it fast enough for real time rendering.
lol, go play Cyberpunk 2077 with pathtracing and compare it to raster before you call it a gimmick.
i own an rtx 4090 and yes cyberpunk looks amazing with raytracing - but worth the $2000k and nvidia monopoly over the tech? a big resounding no (for me).
If you think of either crypto or gaming and not accelerated compute for advanced modeling and simulation when you hear Nvidia, you won't have sufficient perspective to answer this question.
What does RTX do, what does it replace, and what does it enable, for whom? Repeat for Physx, etc. Give yourself a bonus point if you've ever heard of Nvidia Omniverse before right now.
It's a transition happening.
Research and progress is necessary, Ray tracing is a clear advancement.
AMD could just easily skip it if they want to reduce costs, we could just not by the gpus. Non of it is happening.
It does look better and it would be a lot easier if we would only do ray tracing
If suggested usage means upscaling, it's a dubious trade off. That's why I'm not using it in Cyberpunk 2077, at least with RDNA 3 on Linux, since I don't want to use upscaling.
Not sure how much RDNA 4 and on will improve it.
Here is a good video by Digital Foundry looking at Metro Exodus Enhanced edition with devtools. Where they show what raytracing is and how it differs from regular lighting.
simplified tldr: with raytracing you build the environment, designate which parts (like sun, lamps) emit light and you are done. With regular an artist has to spend hours to days adding many fake lightsources to get same result.