Adam Savage Learns About Gaussian Splats!

Hey everybody, Adam Savage. I am here with Chris in FBFX's capture department. You guys have captured me a couple of different times with your photoggramometry rig.

And I want to go to this zoom out and talk about like this is a department where you take real things, turn them into digital things and also where you take digital things and make them able to be real things. Is that right? Yeah.

It's the cycle of life. But today we're talking about you using this rig for something different. You you got a different scanning technique to talk about.

So yeah, Gaussian splats. I've been hearing about Gaussian splats for a long time. Yeah.

So that's an evolution of nerves, neural radiance fields. So it's interesting for us because we come from a photogometry background where load of cameras take pictures, build a 3D model and they're each taking it's the same camera, same lens times a couple hundred times a couple hundred. So on our big rig, we have multiple lenses, multiple cameras because we need to get the focal length, we need to zoom in on the head, zoom on the hands.

So that's very different. But because we have the infrastructure when gauian or Gaussian splatting came along, we could take our original data sets and throw them through software and pull out a Gaussian splat. So we were looking at our existing stuff.

Yeah. And running it through just to see, oh, hang on a minute, it's working. So we could use our old stuff in the old Gazian splatting workflow.

So if you have a photoggramometry scan of one of your costumes for instance, you've been able to take that scan, put it into a program and have it turned into a Gaussian splat. Turn into a Gaussian splat and what are the advantages of turning it into a gaussian splat? What does that give you?

So well what is a Gaussian splat? Is Yeah, let's start there. No, so that is the question I was asking when it was coming on because you know you've got a 3D model is points.

So there's just points in space that are tied together as a plane creating a polygon and you got a 3D model which you can then light and shade and etc etc. We all know that a Gaussian splat is using points but on this point there is a direction there is color and then there is like a a Gaussian distribution of of transparency. Okay.

So if you have millions of those and those interlock with each other, you get a blended image and that can create a what is perceived to be a 3D realtime asset. Oh, real time asset that's at a very high resolution. High resolution.

It's lightweight, which when you say lightweight, you mean in terms of the the computing load to turn it in real time. Exactly. So, you know, you could have a 3 million splat Gaussian splat and a 3 million polygon asset and one is going to run in a browser.

That's the Gaussian and one is definitely not going to most home computers. Exactly. So it's it's quite revolutionary in what you can now do with this data because if you can run it in a browser, you've now got a huge area in the world where they are looking into how they can use a Gaussian splat to to showcase it.

So does this help you guys for instance where you've got a production say in the desert in the Middle East filming but you need to get approval for some production piece from Los Angeles. The Gaussian splat allows you to send a much higher resolution. I mean that is one use case that can be used because yeah it doesn't take us long to capture the data takes around two three hours to process the data so the same amount of time really to for a model to a Gaussian splat and that's let's talk about that for a second there's always post-processing it's not like you photograph something and you have an asset yeah exactly so with a 3D model there is a lot of post-processing I was I would imagine like we have played with a bunch of your scans you know you guys carve me out of bead foam you've sent us some STLs and I'm really impressed with the post-processing on the hair work, the the gentle transitions you did.

And we were really clear having looked at raw scans of me, how much work you guys did. Yeah. There's a lot of work.

It takes an artist. I mean, I'm a Zbrush artist and you know, so taking a scan Yeah. And a Zbrushed artist, they are they go hand in hand because Zbrush artists use high poly data sets to be able to sculpt, right?

So to sculpt your hair is literally a sculpting technique. is using the tools in the software to to create pores, hair, all these other fine things that make up a human head. And you guys have built that pipeline over uh over the course of the company.

Gaussian spots are brand new. Are you building a new pipeline for them and what they can do? 100% definitely.

So we are looking at one how to capture it more efficiently. So we build technology, we build our rigs, we build our capture systems. So, this is not too far removed from our current capture systems to be able to optimize the splat, right?

But it will need building and we're working on ways to incorporate that at the moment. So, that's quite exciting. Are there other people working in this space that you're benefiting from some of their institutional knowledge or is it all sort of right now siloed and different folks kind of experimenting?

I think it's experimentation from different groups at the moment because no one really knows how to use it and no one knows why they want to use it, right? So, we've been working with a couple of, you know, big players within the advertising industry and they they have used some of our splats within their marketing. Yeah.

And there was obviously a lot of teething problems with that because we're experimenting, they're experimenting, but out of the end of it will come something quite amazing because the only way you move forward is to experiment, right? and you learn from your mistakes, you refine it and you learn the pitfalls of it because like with 3D modeling or photoggramometry, there are pitfalls that if you don't do X Y and Zed, it's going to cause problems later down the line. And some of them you can't avoid, right?

It's like if you have a beard, for example, I can't get photoggramometry scan. I don't come out very well. Um, but with with splats, all this would would be renderable within the scene.

So there are huge advantages to Gaussian splats, but how they get used is is going to be for the future to decide. Can you walk me through some of how your computers work with them currently? So I've got these up on the screen because these are quite interesting.

So I don't know what I'm looking at. No, no, I don't know either. But these points are essentially a Gaussian splat.

So each of these threads that I can see, it looks like I'm inside like I don't know a mattress. Yeah. Yeah.

So each of these threads I can see is a Gaussian splat. I I what you said a point a trajectory or a direction. Yes.

It's got a direction. There's a point in space. There's a direction and there's a color with opacity.

So you imagine like a bell curve. Yeah. That is a gausian curve.

Oh wow. So if you now take that bell curve and make it into 3D, you've got a a concentrated point in the middle and then it blends blends out which is why you've got gaussian blurring in Photoshop and these applications because it's taking an average from points and Right. Right.

So yeah, so this is what you're seeing. And as you as I zoom out of this, this is like the molecular and I love doing this. You can see them appearing on God.

It's like we're flying out of something. And then as you go further, further out. Holy.

Go back. Go back. That was go back in.

What? I don't understand. It's pretty mad.

So, who who would have thought inside a Gaussian splat would be more exciting than than the surface? But it is pretty crazy. That is so fascinating.

You said there's no savings here in terms of post-processing. It's always takes post-processing after. Yeah.

To to actually get to this point. It's a you can run it for two hours, but you can run it for 10 hours and it'll just get more refined and more refined and more refined because it's there's a thing called the steps. So, it just looks at the pixel, applies its algorithms to each gaussian splat and refineses it.

Refines. So, the longer you let it go, the better it is. And then this is a lightweight spinable every possible detail.

So, this is like 60 fps on this machine. Good lord. And you know, you can start to read the Yeah.

So, the longer I leave it and the more photos I go closer, you will be able to read that. But for now, like if you wanted a mid-range shot that you could add camera passes to, throw it into a 3D scene. Wow.

This is pretty impressive. And what's also impressive is as you rotate, you'll see that the lighting is baked. Oh, it's changing.

It's baked in. So, this is interesting. With photoggramometry, you generally want the flatter lighting as possible.

So we run cross polarized data sets. So there's no highlights which is perfect for a 3D model because you don't want highlights. You want to put your materials on post, right?

Whereas with gorian splats, the lighting is baked in which means you can now light to suit your output. Wow. So if you wanted, as an example, if you had a scene where you wanted your your asset in it and it was lit in a very dramatic way, rather than going photoggramometry, you could light it in that dramatic way.

Amazing. And then you would build your gorgeous splat and it would be lit in that exact way having all of the the basically the reflections. You can see you can move around as a reflection.

I don't have to apply material shader to that. It it naturally has it on it. Same with all the metal surfaces.

But also, I'm even seeing how light reacts at different angles across these specular fabrics, which is like I don't know another way you could communicate that to an art director from, you know, another location. So, you know, as a as a you know, for our end, if we could take that and send it to a digital team, they now have a perfect reference of how this should look under a certain light condition. And what's also interesting and it handles glass.

I mean photoggramometry say goodbye to that. That is not picking that up whatsoever. But Gazian splits don't seem to have any problem.

It's having is it's doing something right. I mean it's a bit janky here and there but it's found a glass dome. You've been immersed in the the the photoggramometry technology which is again out at the edge surfing at the edge of this technology.

how it seems like even though you're not quite sure what photoggramometry is ultimately good for, it's clear it's really important. Something really amazing is happening here. Is that why you're wheeling it in this early?

I think so. I think I was saying earlier on, I I think you're going to see a lot of this in music videos. You're going to you already seeing in the BAFTAs.

They used it on the bafters to showcase the winners. But I think where this is going to come in, it's going to be all the backend pipelines where people are going to start using it for their backend work. We're already seeing people so amazingly with Gaussian splatting works incredible with video.

So if you get a iPhone, phone, camera, video camera, and you walked through this building, recorded it, you could take out every 25th frame, so every second of your 25 frames, build your Gaussian splat of your walkthrough. No. Yeah.

Then because it's pointbased, yeah, you could feed that into your 3D software. You could add a camera to your first camera, you could run the camera through all the cameras. Thus, now you've got a perfectly tracked camera shot matching your video footage.

So, you've now bypassed the idea of putting your film in to then manually track it, to track it forward, track it backward, to then plot it into 3D space. you already have it because you built your 3D scene out of it and it looks photoreal. So that's just one thought of how this might change a pipeline within just the backend industry.

That's amazing. There's so much I think where this is going to just just go into and change how people do things. Incredible.

What So you also have this screen up. Tell me what's going on over here. So because it works really well with video, drones have really good video.

And because it works really well with smooth footage, drones also produce smooth footage. And I can't climb 100 foot into the air with one camera on one ladder and then move it. I wouldn't like to do that.

Yeah. So, drones are really good. So, once again, we have this crazy thing that no one knows really what you're looking at until I love doing this.

So cool. What? You end up with that, right?

No. So, because it's taking it from a shot, you also get the background. So, as I rotate around, you get parallax.

It's hard for me to even wrap my head around how it's so cool. Good that looks. It's so cool.

So, you know, now you now you're thinking, okay, well, how can this be used? Well, cultural heritage. I'm sure they would love to be able to preserve a moment in time of a church, a monastery, a cathedral.

Cuz not only that, you could do this. You could then go onto the ground. You could take photos going into the door.

You can take images of all the inside. Make go to town photoggramometry. And now you've got a completely full walkthrough up and down inside and out of an area.

And all this is generated from a single drone shot. This is this is generated from from three drone passes. So be three or four rings.

Three ormazing. And um you know if you go on set in film sets, they want stuff like this. You want to be able to take a shot of your film set so the director can spin it around and go actually no that that's pretty good over here.

If we can go zoom in here be great. Maybe we need the rigging over here. So the back end of this to be able to build sets and to see the time of day and to see the horizon.

If you need to map paint that out or add something block it, you know, I mean pretty pretty impressive. No. So, one of the early one of the early executions for the Boston Dynamics dog was to be a construction site uh chronicler and it would just walk the construction site basically and everything.

Exactly. This does that in a fraction of the time and gets the same result. Exactly.

So, it's very exciting. Um, can we scan me with Gaussian splats? I think we can.

Okay. Let's turn you into a splat. Let's turn into a splat.

All right, Chris. I I know I stand here, but is there having been scanned by photoggramometry, is there anything different about doing it with a Gaussian splat? Yeah, Gaussian splatting, you really want one camera, one lens, and just keeping it very simple.

Copy. With a photogometry setup, you want to be able to capture your head closeup, but your full torso. So, if we had one camera, one lens take forever.

And we wouldn't be able to do it because you'd need to be able to zoom in on the head. So, on our rig, we have dedicated set of head cameras and they are pretty cropped in. Mhm.

And then we'd have some lower cameras that are cropped in on the feet, but then the majority of the rig is zoomed out to capture the full body or 3/4. Uh, so for gaussian splat, do I should do I should I stop breathing while you're No, no, no, no. Do do fill your boots.

I mean, the advantage of gaussian splatting is that you can move. Amazing. You can you can do some cool things.

I really want to try fire, but not now. I'd like to try cuz you know I ready to just jump right in. Yeah.

just just capturing some fire and seeing how the fire I mean how does that how will that look in 3D right yeah I'm totally interested in that so I'm interested in when you get to that when you send me some of that I just I just love to see how that looks yeah same right yeah you can do what you want you want okay you ready here we go oh wow that was fast all right how did that go pretty good Regal. Yeah, I was doing a superhero pose. Very regal.

This is your view. You're always looking at 200 plus shots of Yeah, I'm just, you know, I scroll down just to see if there's any any dropouts. Obviously, when you got a rig that's been sort of homemade, there's the occasional black image and you get that, you know, you can always capture again, but this is currently my favorite one of you.

I mean, not Oh, like if we can get that built as a splat, that is going to be incredible. I cannot wait. So, these go into the program and get slowly processed.

Yeah, they do. Amazing. Uh, you'll send us something to look at in the US.

I Oh, cannot wait to see. The whole world will see it. Amazing.

Amazing. Chris, this is really exciting stuff. It is.

I've been My friends at Apple have been telling me about Gazian splats for a long time and that they're important and interesting. I love seeing some real world applications. For sure.

Amazing. Thank you, sir. Been a pleasure.

See you next time. Indeed. Heat.

Heat.