How Ray Tracing Works - Computerphile

you've probably heard of Ray tracing because it's a big big thing at the moment um but you've definitely seen it at some point in your life because it's um heavily used in games uh rendered art and films and visual effects it's massive um over there and it's it can render things in very very high quality detail and I want to go over how it works and also why we eventually went from rasterization to Ray tracing I mean rasterization still used but why they eventually said we need something that can do a little bit more detail

to start off with I want to show you how good RA tracing can look so here's a video of someone playing cyber Punk with the highest ra tracing mods now this is absolutely insane to me this is photorealistic why does this look so good and that is because of things like the Shadows down here the Reflections in the car that is all done thanks to Ray tracing so traditionally we would use something called rization now to give an overview of 3D Graphics what we have is we have a series of polygons in 3D space and

what we're trying to do is generate an image of these polygons right and what we would use is something called rasterization which is a way of almost painting pixels onto your screen now rasterization is very very very optimized and very quick but it's struggles with handling things more complex light interactions like you saw on there like the reflections and the shadows and stuff that so I want to go over quickly how that works so that we can then show you why we eventually start using Ray tracing and things like that so let's create a very

simple scene right so let's say we have a floor here oh God I'm really not very good at drawing um and let's just have very basic triangle here and just so I can really show off how this works we'll have a red square behind it right what do we do now right we have this scene so we need to have a camera in this scene so we have a camera up here and now you're probably going how do we get the camera here I'm not going to go into that right now but you basically have

a series of matrices that you multiply each of these points by and you'll get that camera I believe there's a video on that on this channel already what you'll then get is you'll get an image up here like this and you'll have these vertices for uh these shapes um in 2D basically they'll look something like this um one hit one hit and let say you have your triangle here so you have your vertices on your image plane how do we actually generate the pixels for these shapes right that is when you ruize it so it

would look something like this you'd have your vertices here for this triangle and you want to basically color in the pixels that are inside this triangle so let's just draw the outline here essentially go okay this pixel looks good this pixel this pixel this pixel and yeah that pixel's in enough that one's not in enough and basically what you do is you'll color that all in and you'll get a triangle there and that is what you will do for each of these now you're not generating a pixel per se you're generating something called a fragment

at the moment because what you do is you do it for every single shape in you're seeing so you'll do that for the triangle and you'll also do that for the square well even if they both in the same pixel yeah yeah because you're not generating a pixel quite yet it's a fragment and what you will then do they have all your fragments go okay which one's close to the camera you reference your Zed buffer it's called and you'll go okay green triangles first let's just render that fragment as your pixel so then what you

get is eventually you get I'm trying to represent these as pixels I'm just shade in Life's Too Short Life's Too Short yeah so you then get something like that what I'm trying to show green triangle in front of the red triangle okay so what you can do better than that we can do better than that so what another thing you'll do is you'll use textures so you rather than um interpolating between colors on your shapes you would use a texture map and then you basically just plun your texture on top it's dead easy right but

let's do some more complex things let's do lighting right and I'll show you how that works so let's just say we have a light source here for example we have a little Sun there typically for lighting you have let's say some ambient values some diffus vales and spectular values and what happens is that if your camera is let's say here and the light rays are bouncing off this object and essentially you use some calculations to um figure out what what that lighting should be at each fragment so you go okay light source is here our

normal Vector is here our camera is here therefore these are what the diffusion spectral um stuff look and you get a nice little thing that and it looks it looks pretty good but the issue is that all you're doing is saying I like sources here um let's just guess sort of what the the lighting effects are and and the issue is that let's say there is another let's say little like flying sorcer here okay when you calculate your lighting values you're not accounting for objects in the way essentially you would still render these lighting effects

on your on your shapes and that's not right it doesn't look correct and you can do things like Shadows you could generate these things called Shadow maps that do that but they're an extra step um that you have to do and then what happens if you do let's say Shadows or Reflections or refractions is that they're so incredibly hard to calculate with um rization that you get these Shader functions which are millions and millions of lines along okay maybe not that much but you get these massive bloated Shader functions uh just trying to figure out

what what these should be and then what happens is that your your quality um just isn't lifelike is it it it's good but it's not as good as it can be it doesn't have that you know special source to it now maybe what we should do is we should maybe go outside and I can show you how light works in real life and show you how we can Implement that uh in modern Graphics ah it's lunchtime thankfully the weather's actually been very kind to us uh here in Britain so how are we seeing this right

now what is happening is that the light is coming down from the Sun or wherever other light sources here and it's bouncing off the ground around the lake it's it's bouncing around all that and then it's hitting the back of our eye and then we are technically rendering an image in our brains now if you go back to how what I was talking about of rasterization that did not do any of that it it's not like it is in real life while Ray tracing is almost a little bit like the the opposite of how light

works in real life so as I say light comes from your light source it's bouncing around and hitting your eye what we do in in Ray tracing is that you have your camera you then shoot Rays out for your camera and track where those Rays go for example if I wanted to render the ground here what we would do is we' shoot a ray out it would hit the ground and we go okay that's that's where our surface is and we'll go okay that's where our light is therefore we can calculate the lighting effects and

it's why I'm able to have a shadow here because what we would do if we're going to render this Shadow is we'd shoot a ray out in our camera it would hit the ground and then we go hang on our Sun is up here when we shoot a ray from the ground to the sun there's an object in the way I am in the way so therefore we know that that light cannot hit that surface therefore we don't render anything and you see how that's much more intuitive than generating these crazy Shadow maps and having

these large crazy rasterization shaders you see how interesting the light how that's bouncing off that that water over there and sometimes with stuff like you can see a rainbow um and that is because the light is interacting with those water particles in in a very interesting way so trying to model all this in rasterization is is difficult to say the least um but what we can we can do is is use raay tracing to follow the light and get a much better rendered image because of it so now I've I've gone on about how R

tracing works so maybe let's go inside and I can show you draw some diagrams and show you how that that works um on the computer I've redrawn this scene again start from scratch and I'll show you how you can render this with Ray tracing so I mentioned outside that you shoot Rays out into your scene there are different ways you can shoot the your ray out the angles you can shoot your rays out so there's like orthogonally which is they're all straight or from a perspective where they're like that like how the human eye renders

an image basically so let's shoot three arays out here so let's do one like that let's do one like like that for example and then let's just do one like that now the first thing you got to figure out if you're doing some sort of Ray tracing is what objects have we hit once we shot the ray out so here we've hit this object we've hit that object on this Ray we've hit that object here we've hit that object here and we've hit that object here the main thing that slows Ray tracing down is figuring

out what objects we have hit in this scene so as you can see on the here we've hit various objects and how do we figure out what oh there's one should be one here how do we figure out what objects we' hit now there's many different ways of doing that and that's a whole separate video in itself potentially because that is how we optimize this this um process but let's just say we can figure it out for now okay so we figured out that we' fit these objects here so what we do is go okay

which was was the first object that we hit okay we hit that object we hit that object we hit that object Okay cool so we know now know the colors because we've hit these objects so that one would be black let set a black surface hit black green green okay when we did rasterization that was the first thing that we did here was figuring out these colors we've just done that by shooting Rays out and seeing what it's hit so how do we calculate lighting okay we have our light source up here what we then

do uh typically is we'll shoot out another ray up here towards our light source and go okay firstly can it reach the light okay these two can this one can't cuz it's hit the tri Le therefore that is a shadow okay see how much easier that is than having to render a separate Shadow map and incorporate all that into your Shader functions you just go I'm going to shoot out another a didn't didn't hit it okay there's something in the way that's it right that is all you need to do and to calculate the lighting

is actually quite similar to how uh the equations we use for um rasterization it's just you calculate your diffuse and your spectular values and then you just go okay that's the final color and that's the color of your pick pixel basically you figure out the color of your pixel by following that Ray through your scene calculating the lighting calculating if there shadows and then basically backtrack and go that's our pixel color now hopefully I've I've explained that reasonably well um let's do s slightly more complex here let's imagine that down here let's say that this

is no longer a floor and that we have a mirror here okay this is what I was going on about outside that R Trac can do very well how do we handle something like a reflection so what you do you shoot your your ray off down here it hits that surface and then you go okay it's a mirror let's bounce this Ray off boom now that Ray has gone up here and it's hit that triangle here and you go okay now I've hit the triangle you calculate your lighting uh it's not it's um uded by

that therefore it's you don't render any light from that light source and you backtrack there and you basically what you'll see when you render the image is you'll see the underside of the this triangle because that is how it would be you have a mirror you have a triangle you hit that triangle you render that image you see how much easier that is and having to faf around with these Shader functions and create something crazy in rasterization you it's built R tracing is so good because it mimics how real life works so things like Reflections

refractions um subsurface scattering suddenly become a lot more easy can I ask a question yeah um obviously in in the light source where we were outside the sun is so bright things are bouncing off all sorts of things in graphics they often have more than one light source don't they how do you deal with that so if you have more than one light source what you let's say it's one here what you do when you check and you go okay where what's our lighting if we're if we're let's say um a shadow you shoot off

array there and you shoot off array there and you go okay let's say you're this one and this isn't a mirror anymore we're back to uh flat object you go okay uh that is in um uded so you don't render any light from there but that one isn't therefore what you get is you get like a shadow going like that so just basically some kind of summation of what different things are at that point exactly yeah so that's how you can render really cool um multiple light uh scenes um because it can do that right

but with every light source you've then got to figure out an extra calculation of shooting another ray out and you see how that can get very uh very complicated but it's a lot easier than rasterization um for calculating light and just one other question before we go to whatever's next yeah um each Ray is one pixel so a pixel is a ray and a ray is a pixel typically yeah typically yeah normally there there are other more complicated things you can do but whole different subject entirely we won't go into that otherwise it'll be an

hour long video what I want to show you is a comparison between a ray Tracer and a rasterizer now we have our rizer here which is um open gel and on the right we have a raid Tracer now this is a RAID Tracer that I built while I was at the University of Leeds just to make sure everyone is aware my Ray Tracer is not very good it's very unoptimized um it it's not as nowhere near as good as as it can be however it is a good comparison and it's good and I can show

you the the benefits of using a ray Tracer and the drawbacks at our lighting here first thing I want to just talk about while I'm setting this scene up look how slow it is to do anything here the reason why this is so slow is that for every single um Ray that you shoot off it checks every single triangle in the scene to see if it's hit it now let's say you got a 600x 600 pixel image here that is 600x 600 Rays checking every single triangle in this scene let's say it's 10 triangles so

600 * 600 * 10 calculations every time which is just why it just is why it's slow and that's not including the lighting calculations and checking shadows and stuff like that my point being with that is that while raid tracing can render very nice images it's a lot slower than rization which is why for a game like Minecraft Minecraft now uses R tracing I've heard but the very early versions of Minecraft only use rasterization because why not you're just rendering blocks there is no crazy Reflections refractions crazy lighting effects and you wouldn't need to because

it's just a nice game with some blocks so I've finally set up this scene here and and what I'm trying to do is is make it like what we were doing down here so we have a triangle over a ground plane here now on the left you have your open G on the right you have my Ray Tracer they look about the same and they should because we're not doing anything crazy here the light is is is pretty much the same in both scenes we're not doing any Reflections refractions or anything crazy like that but

what I can do on the ray Tracer is do Shadows there are light sources up here how this works is the camera would shoot a ray out here it hits this ground plane here and we go okay let's shoot another ray up to your light source oh wait there's a triangle in the way don't render any light there that's how you do a shadow and that is how you the the whole scene is rendered by shooting Rays out let's say you do Point put Point here shoot a ray out all let hit our ground plane

shoot another R out to the the light nothing in the way therefore we can just calculate the the lighting effects there obviously I've gone on and spoken about oh that this is how ra tracing works but there ra tracing is a big sub object heavily researched there are many many ways of of calculating your like calculating different um effects like this but essentially the underlying thing is you are handling Rays shooting Rays out and tracking them that is that is what Ray tracing is okay this is just one way of doing it we've booted up

a more complex scene here now this is called the Cornell box um which is basically if you want to compare how good a graphics engine is to other scenes other other Ray tracers and and rest rides what you do is you set up the scene of the Corell box here and then you can get a direct comparison which is what we're doing here between open G and my um Ray Tracer now this framework is built by the University of leads I'm just using it to show a comparison so on the right is my raay Tracer

that I built and on the left is just open gel one thing I want to mention here is that I am using probably the worst um way of using open gel the lighting is directional light so it looks pretty muggy you can use positional lights to make this a little bit better um and it will look similar to to the um R Tracer here but what it can't do unless you go into your Shader functions and change stuff is this Shadow down here so what is happening is we have a light source here a shadow

here and it's and it's rendered um like that and I think it looks pretty decent right but what's something cool we can do now what would happen if we set I the the walls either side to a mirror well I'll show you how that works it takes a while to to calculate which think proves my point of while Ray tracing can render stuff like this it is a little slower than open gel which can render like that um so here we go we've set both of these walls to be mirrors and as you can see

here it's rendering the side of the the the box here now let me show you how it can render that shadow there what is happening is we are shooting a ray through your camera into your scene it is hitting that mirror and going okay it's a mirror let's reflect off it it is reflecting off that mirror and hitting the floor let's say here now we have our light source up there we are shooting another array towards our light source and it's going ah we've hit the side of this box here therefore there is no light

hitting here so don't render any diffuse or spectular lighting and then that's how we get that final pixel color you see so it's all about following those those rays and figuring out where they are in the scene what the color should be on the surface and then what the lighting should be if it's in a shadow if it's refracted through an object if it's done anything like that Ray tracing is very good for rendering very high quality scenes but tradition it is much slower than traditional rization so if you're a games developer or an artist

you've got to decide which if you want speed but maybe a little less quality or you uh you want it to be maybe a bit slower but you can render stuff like this seamlessly the further away you get from where your images are actually captured the more nois your galaxians are going to get exactly so if you look over here can you see and so if we just sort of feel our way towards it by taking off little bits of noise at a time we can actually produce an image right so you start off with

a noisy image