▲ 118 points
back
123 comments
This might be a stupid question, because I don't really know what I'm talking about.. But is there a way this could be used to improve physics engines? My understanding is that a physics engine generally uses continuous collision detection. I don't know if noise plays a roll in casting for collisions, but if you could cast less rays, and fill in the holes in the same way... that seems like it could improve performance there too?
Aren't the collision models already pretty simple as is? Simple shapes like boxes, spheres and lines, for which the collisions are fast to calculate. Don't know how it works if you have full blown meshes colliding though...
I wouldn't be surprised to see something like that.
Remember that Krishevsky's software implementation for Alexnet of making neural networks work with GPUs, wasn't initially intended by Nvidia either. But, look how that worked out.
The problem is that you might get a lot of instability.
The most obvious case: stack 20 boxes one slightly above other and let them fall.
If you will have random noise in there the simulation can become unstable/explode.
Yes. Here's a video that describes an approach using neural networks to estimate physics instead of using a traditional simulation: https://www.youtube.com/watch?v=atcKO15YVD8
Yes. I think I’ve seen some preliminary papers with related ideas (because that seems like where ML is at) but I don’t remember anything substantial.
It could be an interesting area of research. Start branching out from ray tracing to other integration problems.
I dont know, it depends on implementation. Collision detection doesn't involve any graphics, it might not paralellise well. so it might still be easier on the CPU.
The tech is cool but I still can't shake the feeling that from a practical perspective, in the context of videogame graphics, that's a lot of work for a meager difference in the end result compared to good old rasterizer shading. Sure, you get more accurate shadows and reflections... But I'd gladly trade that for more detailed models and larger, more complex environments for instance.
Maybe it'll become the new standard for real time 3D graphics in the future but for the time being I file it next to HairWorks as an nvidia gimmick whose main purpose is to make the competition look worse in benchmarks because they don't implement that API when it practice the visual difference is fairly subtle.
In addition to the examples other have given where it improves the visuals, raytracing can make developing a game easier.
Currently, game developers spend huge amounts of time cooking up scene specific lighting hacks, things that make no physical sense but result in the visual appearance they are after.
Another hack is dynamically computing an environment map for a shiny object (that is, rendering the scene from the vantage point of the object) so it can be used as a texture to model first order reflections. It is infeasible to compute that dynamic environment map for every object, so the art department picks and chooses which assets get reflections and which don't. Or maybe an environment map is computed from the vantage point of the centroid of a car, which is used for all reflections (windshield, bumpers, roof, hood), and that works OK for objects which are far away from the car. But if there are any objects near the car, the reflection angles are wrong in different ways on different parts of the car.
With raytracing, a lot fewer hacks are needed. Yes, what nvidia is offering here is a hack in its own way (denoising, neural net upscaling too), but is one hack which applies to all cases, vs the old way of needing to create a different hack for each situation.
> scene specific lighting hacks, things that make no physical sense but result in the visual appearance they are after.
The flip side of this is that physically realistic shadows aren't always what you want. Any amateur photographer who has noticed just how awful hard shadows are on faces on clear sunny days has learned this lesson the hard way.
To quote a graphics professor of mine (tongue in cheek), "Physically accurate rendering is a crutch for people who aren't smart enough to cheat their way to the look they want".
If you want more detailed models and more complicated scenes, you should really look into raytracing.
The problem with raytracing is that it mostly depends on your resolution, not on your scene. Right now, we're just on the edge of being able to do raytracing at practical resolutions in realtime. But once that's possible, the rest is basically free. Rendering 10M polygons is almost as fast as rendering 1K polygons.
And writing a path tracer isn't that hard - most of the math is quite trivial. You get all the magic shadow / reflection / caustic / lighting stuff for free. We're getting quite close to being able to do movie-level graphics on consumer machines in realtime, and all that without any of the tricks required for okay-ish looking rasterizer shading. Best of all, it's even easier to work with for artists as well!
"Rendering 10M polygons is almost as fast as rendering 1K polygons"
That is correct for ray tracing.
But modeling and animating 10M polygons requires a Hollywood movie budget, while I can model and convincingly animate 1K polygons myself.
If you have static lights in a static world with pre-computed raytraced lights, yes it doesn't change that much. But once you have a dynamic world and dynamic lights, it's a game changer.
Minecraft with raytracing is a great example. It's so much better with raytracing compared to simple shaders.
I also heard that it will make game development a lot smoother and faster in the futur once most hardware will be powerful enough. No need to build lightmasses, no ambiant occlusion hacks and similar, no reflections with approximate results.
Full RT will be amazing for some things like a DOOM game where shadows in corridors make you stay alert, but otherwise I don't think it is a game changer in terms of gameplay.
What is a given is that there will be a flood of games overusing it, like it happened with Bloom and other effects.
This[1] demo does pretty convincing dynamic lighting without the need for RTX relying just on regular shaders. To me nvidias RTX looks like a gimmick with the sole purpose of enforcing vendor lock-in.
It is already being used in movie studios for a better rendering experience.
https://home.otoy.com/render/octane-render/
Fun fact, at GTC 2020 Otoy revealed that they are moving away from Vulkan and adopting CUDA instead (via Optix 7) due to better compute power and tooling for ray tracing algorithms.
Funny that you mention HairWorks, because that one causes a lot of issues in practice because the hair is rendered in post-processing. Therefore, it is not present in the z-buffer so that alpha-blended transparent effects (e.g. smoke) may accidentally overlap with the hair. For an example of this issue, see the excellent Shadow of the Tomb Raider game.
A lot of Nvidia's new RTX tech may be marketed towards gaming but it's actually having an impact in 3D rendering. Cycles supports RTX out of the box now and I'm sure other renderers will/do as well. And the denoising tech is amazing, while not up to the challenge on final renders it's perfect for getting a quick frame upstream.
I'd gladly trade that for more detailed models and larger, more complex environments for instance.
No doubt nVidia could make a card that enabled games to support models with double the polycount of today's games, but if companies refuse to ship games with them because the art effort is too great then it's not going to get players to update their PCs. I suspect the push towards raytracing is a way of selling more graphics cards without radically increasing the amount of investment needed by games companies in the way that higher fidelity models or more complex environments would.
Have a look at the game Control. I think this is a game where I feel they've used the technology really well. Maybe the office building location is a good one for raytracing, as it's full of reflective surfaces, but the graphics feel substantially better with it switched on. Remedy Games also managed to maintain a reasonable framerate with lower end RTX hardware. Other games look less impressive, however, such as minor changes in the shadows. It obviously requires some thought before it can be used.
Character animation is still a field that is lacking, compared to otherwise more and more realistic environments. So Raytracing may help make interiors and exteriors really impressive, but you'll be stuck with characters that have the same looping animation when they run and move around.
A lot of computational work yes (with the currently available hardware at least), but many visual effects are much much easier to implement with ray tracing than with rasterization techniques.
Dynamic off-screen reflections, soft shadows, ambient occlusion, dynamic global illumination, caustics etc. can all be implemented more easily and realistically with ray tracing than with previous techniques.
I think Minecraft RTX pretty much seals the deal: ray tracing will definitely be the future of lighting in real-time computer graphics.
I think the most important thing to do with RTX will be indirect illumination. Maybe RTX will be capable of computing relatively high quality lightmaps on the fly, store the results for as long as necessary, and keep recomputing for new parts of the scene.
just wondering is it possible to implemented in FPGA for ioquake3?
Same. You can always spend those transistors/cycles/watts on something else that is more compelling (more frames, better AA or higher rennder resolution). For most games doing 4K/120 without raytracing would probably be a better experience than 4K/60 with raytracing.
Ray tracing is not powerful enough nor are games built yet from the ground up with ray tracing in mind. With the newer generation consoles expected to arrive with ray tracing that will change then probably we will get to see something different. Personally I feel raytracing will make the most difference in vr rather than normal games.
What APIs are being used to perform ray tracing in these new games that support it?
Edit: Found a starting point of an answer, covering Nvidia’s hardware at least, at https://developer.nvidia.com/rtx:
> Ray tracing acceleration is leveraged by developers through NVIDIA OptiX, Microsoft DXR enhanced with NVIDIA ray tracing libraries, and the upcoming Vulkan ray tracing API.
I wonder if the same concept of running the output image through a fast neural network could be applied to typical rasterized game engines. For instance, applying antialiasing or anisotripic filtering or shadow blurring?
This is more or less what DLSS is
I watched some videos comparing ray traced and rasterised graphics and in real life the benefit seems negligible.
On top of that rasterised graphics use all kinds of smart techniques to improve the performance. Ray tracing the whole scene is kind of brute forcing, brick and mortar method. It should be used selectively and not on each frame IMO
The problem with comparison videos is that most of them are focused on comparing games with RTX on/off. But this means that the game is developed with raytracing as an afterthought: it cannot depend on it for core features, because the majority of consumers don't have it yet.
Imagine a shooter with, say, a subway entrance level. Initially the indoor part is lighted by overhead lamps. During a shootout, the defending team can destroy those to provide cover. while they can still see backlighted silhouettes of people walking in. The attackers might choose to park a semitruck at the entrance to provide some cover in shadow.
Or imagine the end of an escalator. How do you look for people coming in? Obviously, looking directly into the shaft exposes you to anyone coming in, but perhaps you can use the slight reflectivity of the marble floor to look at it indirectly.
Should you use the east of west side of a valley to go to a target? Well, that depends on the position of the sun, of course. You want to hide in the shadow. But what if a day only lasts 5 minutes on your world? Shadows are constantly changing, and so are the hiding places.
Want to hide in a room? Turn off the lights to make the window into a one-way mirror. Be sure to turn off your laser pointer and flashlight, though!
And have you seen the mirror scene in John Wick (https://www.youtube.com/watch?v=7-TZCEyok_o) ? Try doing that with a rasterizer!
So no, using it selectively isn't the way forward, if you ask me. We'll only see the full magic of it if we're able to use it fully in every scenario. We'll only figure out how to use it when we see how gamers interact with the effects of it. All the things I described above aren't something the developer explicitly programmed, they're just properties inherent to raytracing left for the player to discover. Only after a few years of that will developers be able to fully make use of the benefits it provides.
From the fundamental point of view, the cost of raytracing grows proportional to the number of pixels rendered times log(scene size), whereas rasterization grows linearly with the scene size. So raytracing enables much more complex scenes at a fraction of the cost of rasterizing, which explains why it's being used more and more in real time rendering.
For others interested in why ray tracing scales O(log N), this is covered in an earlier video in the NVIDIA "Ray Tracing Essentials" series, specifically this one:
https://youtu.be/ynCxnR1i0QY?t=173
It's timestamped to the discussion of why this is true, but the whole video (like the series, IMO) is very informative, this one focused on "Rasterization vs Ray Tracing".
This is only true from a naive approach to visibility. With rasterization you can rasterize geometric values like normals and positions to an image buffer and shade only what is visible. Rasterization in old versions of renderman would do a lot of shading that was thrown away once scenes became more complex. Real time games eventually followed a similar pattern and now do 'deferred shading'.
Rasterization can also be used to cull polygons that aren't visible when they are going to end up hidden by opaque objects.
In theory ray tracing the first hit visibility can scale better, but in practice that part isn't a big deal and rasterization will probably win anyway. Not only that, but the idea that more polygons will make something look better is another trap. High quality lighting and high resolution textures become more important once polygonal geometry has enough polygons to not have faceting artifacts.
The problem is that although it scales better than rasterization, the computations involved in tracing are much more expensive (on current hardware at least).
Another big drawback of ray tracing is that you can't use conventional occlusion and view frustum culling [1] with it. So significantly reducing the scene size, as you do in a rasterization renderer, is just not possible. If you have access to the GDC Vault I can recommend DICE's 2019 talk about reflections in Battlefield V. The slides are available for free [2].
[1] https://media.giphy.com/media/xUPGcgiYkD2EQ8jc5O/source.gif From the game Horizon Zero Dawn. Only geometry that might end up on the screen is actually sent through the rasterization pipeline.
[2] https://gdcvault.com/play/1026282/It-Just-Works-Ray-Traced Talk about culling starts on slide 56. Occlusion and Frustum culling aren't an option, so new techniques had to be developed.
The main benefit of raytracing is primarily for the CG artist pipeline/workflow. Complex, realistic lighting effects just drop out of the integrator, and changing scene lighting is much more akin to a lighting director on a movie set. Whereas with rasterization an 3D artist has to know the correct hack (i.e. "smart technique") to use in each situation--along with the failure modes for that hack.
If we had enough power though we could do away with the performance hacks and just use one 'correct' way to draw the scene. This combined with accurate representations of materials would make building scenes easier and result in more coherent worlds.
All kinds of smart techniques are emerging for ray tracing, too; you don't have to shoot the rays directly from the camera per se. Both rasterization and ray tracing will coexist to provide better graphics, and it was just that ray tracing didn't have the place it deserves on the chip.
While this works OK-ish for static images, it is almost unusable for animations. But for static images, blur+sharpen in Photoshop has worked OK-ish, too, for many years.
So the practical benefit of this is negligible.
What we need is a denoising technique where the denoising artifacts move convincingly with the features of the scene, so that you can use it for movies and games.
Oh and for games, as long as this is NVIDIA-exclusive, developers have to treat it as an optional add-on. For multiplayer games, that implies that Ray Tracing may never show details (such as a reflection of an enemy) that would give a strategical advantage.
Plus the real issue with contemporary game development is that consoles make a majority of the revenue (due to less piracy) but they choke when you have 50k+ polygons on an animated character. And you'll be limited to 2 GB GPU RAM on 30% of your PC player base, because they use laptop GPUs.
In the end, then, you usually don't have enough detail to make ray-tracing look good. It looks amazing for high-poly curved surfaces, such as those used for offline-rendered cinema movies. But on a blocky realtime game model, ray-tracing may also highlight artifacts.
Here's a ray-traced low poly bunny: https://i.imgur.com/MGotRC7.png
Notice how clearly you can see that it is low poly. In a rasterization engine, one would "fix" this by blurring the edges with shaders and bending the corners with normal maps.
So in a sense, ray-tracing is too honest to work well with current video game models.
>Oh and for games, as long as this is NVIDIA-exclusive
It isn't. [0][1]
The APIs are intelligently laid out such that hardware accelerated raytracing can be used on popular APIs regardless of GPU vendor, if the vendor has made their drivers correctly.
0. https://devblogs.microsoft.com/directx/announcing-microsoft-...
At the moment, there is no AMD GPU which supports the full ray-tracing spec. So effectively, it is NVIDIA-only.
There's no reason why ray-tracing can't work with bent normals. That's how Blender's "smooth shading" works, and Blender uses raytracing for rendering.
That is correct for Cycles, but the kind of raytracing that is possible with hardware-acceleration inside a GPU is severely restricted in comparison to a general-purpose CPU raytracer like Blender's Cycles.
I believe Eevee, the Blender GPU ray-tracer, does also currently not support having reflection rays collide with surface generated by normal/displacement operations, but instead they will approximate flight path by using the unmodified geometry.
You're a decade behind on consoles, 50k animated triangles haven't been an issue for years. Even early PS4 games had more triangles per character, in some cases more triangles simply for hair, let alone the rest of a character.
"Tom Clancy's Rainbow Six Siege"
40,000 Triangles + 3 2048 x 2048 Texture Maps
https://www.artstation.com/artwork/0eKg8
"Fortnite" LOD 0: 23,041 Triangles https://i.imgur.com/y8fOip0.png
I'll give you that purely GPU-skinned on PS4 can go up to 700k for the entire scene, but that means 14 or less people visible at a 50k budget.
So for a game like Assassin's Creed, the per-character and per-item poly counts need to be significantly lower, to make sure the combined sum of the scene is still manageable.
Just as an explanation, I was referring to "Denoising for Ray Tracing", because that's what the link points to.
I do not oppose ray-tracing in general, but merely the over-hyped denoiser.