I think uh gausian splats will be a big part of CG and VFX work very soon and I wanted to master splats and learn everything there is to learn about techniques and tools available out there. In today's video, you're going to learn how to capture and train your splats like a boss. I'm not going to go into detail of what gausian splats are or how they operate.
Assuming you already know that because you found this video in VFX work, especially with engines such as Octane with ability to do a clean relight, shadow casting and things like that. The benefits are obvious. They show up in reflections.
They interact with refractions and they also render lightning fast. They can also be manipulated like any other geometry at some extent and I intend to use them as much as possible moving forward. What I want to show you is how to capture stuff beyond dandelions in your park and more importantly how to deal with shiny objects.
How to tune the training settings and what are the tools available today. First off, gausian splats are as good as the data that you feed into the trainer. So, you need to understand everything about the data.
With splats, more doesn't mean better quality. And it very much comes down to the type of the scenery you're training. Counterintuitively, uh, shiny objects require less splats and not as much training as you would think.
objects with many details like trees require ton of splats and the splats themselves are memory hungry. What do you do with that? Let's divide the video in several se segments.
Capture, prep, and training. I will explain the underlying principles of splat training in the later chapter. Spoiler alert, splats are better trained from videos.
They require the frames consistency and continuity and that ensures the best possible result. There are two primary factors that will influence the quality of your splats. Lighting and coverage.
If you're doing it for fun or for a very specific lighting scenario, then lighting is not as much of a concern. Just make sure you have enough of it. However, if the model is to be relit in Octane, manipulated in Houdini and reused in future, appropriate due diligence is required, especially if you're striving for production quality.
First, let's understand what uh ganian splat training does not tolerate. It does not like motion blur. It does not like depth of field.
It does not like noise. All the traits of not enough light situation. Also, go splatting is not fond of wide lenses because of the radial distortion.
Even though all other aspects may be perfect, this one optical factor will ruin your splat. Use 35 ms minimum. When scanning the object in a controlled environment, I found that even powerful video lights are not enough for details and sharpness and splats.
We need a lot of consistent light because I want to close my lens as much as possible while keeping my shutter speed quite high and ISO quite low. Sounds very much like a photo shoot. And I found that the photos are in fact the best solution.
Highspeed sync strobes and a fast shooting camera. It is basically shooting a video but with raw high-res photos. I use Godox tropes and Sony A1 camera.
I do understand it may be a bit out of budget. So I tested it with Sony A74 which I'm shooting with right now and smaller Godox lenses. Works fine but will require a bit more time because smaller flashes can't recharge as fast as studio strobes can.
And in terms of light placements, uh, I first used a lot of different soft boxes, big small ones, but then realized blasting the flash into the wide wall is the best solution. I fire three flashes on all my white walls and get the flattest light possible, which is perfect for our purpose. When scanning a location or object outdoors, it's best to choose an overcast day for the same reason, flat lighting.
Outdoors, lighting options are limited, so most likely you will have to use whatever mother nature gives you. Overcast daylight is already more powerful than any video light you'll use indoors, so you're all set. Close the lens as much as possible.
Don't use NDs. Lower your ISO, bring up your shutter speed, and off you go. This aspect took me a few dozen tests to figure out and get used to.
You can capture your photos and videos with whatever you want, but don't expect razor sharp results from something like DJI Nano, for example. Although I used it to scan a tractor and then turn it into cinematic shot with effects, it's not what this video is about. We are after macro detail here.
When capturing, we need parallax, not look around the lax. So, whenever you're shooting, make sure you're moving your body with the camera, not just changing the angle of the camera. When scanning a location, walk around more than look around, but make sure to constantly move the camera in space.
That's what uh gausian splat requires. Also, important point, don't stop moving. Gausian splats hate those moments when the camera just stops.
It confuses it. Amount of coverage will come down to the type of object you're scanning really. matte objects training will be much more forgiving to the gaps in the co coverage.
So handheld shoot will be just fine and will be the quickest option. Again make sure to move all the time and change heights and angles around the object and try to cover as many angles around the object as possible. When scanning a shiny object, the system will be much less forgiving to lack of c coverage.
It can only be saved by providing trainer with all possible angles of view. If you would have a coverage gap, it will result in heavy artifacts. The trainer will fail to interpolate between glaring angles and that's what you're going to get.
Usually subconsciously you're doing one level good and then you start mess around and you miss a lot of spots. I chose this subject specifically because it was my lab rat for many many tries. I've done dozens if not hundreds at this point tests of shiny objects.
And this running rig, which I'm using before Edwards, by the way, is the only item I failed to scan scan properly. I did have some decent results, but most of the times it's failing either from the top or from the bottom. And the shiny surfaces are always failing.
This is a tricky subject in many aspects. Not only we have super shiny surfaces, but a lot of elements that obscure each other. with complex objects such as this one.
I found that handheld is not optimal at all. Instead, I'm using this uh dual axis uh turntable from DF Cine if I'm not mistaken and it's a mechanical one which is important because motorized one would make the process even longer. This one allows me to mount the camera here and I can restrict the axis on which I'm currently shooting.
I won't be firing flash right now because it's night. I don't want to bother my family or my neighbors. I just wanted to show you the rig and how I'm doing those production level uh scans.
And I'm just literally pointing my camera and start to go off like this. What I'm also doing in addition to that uh main axis rotating around the column on which the subject is, I'm pivoting the camera on this axis as well on one circle. I can capture many angles.
Then just lower the arm and repeat the motion. I wasn't scanning the model right now because I did it earlier in the day. This is just the coverage demo.
When I'm actually shooting, I'm making the light as flat as possible. As discussed in previous chapter, the lights, the strobes are turned to walls. They bounce back, giving me quite flat light.
And I have three strobes. Right now, I have a really harsh spotlight lighting the rig. This is not what you want.
You want flat lighting. After you're done with the rigged scan, you can pick up the camera handheld and and do some finishing shots in the spots that you didn't uh cover and you can just go around around your subject and cover it as if it would be a handheld scan. Other tips, uh obviously you wouldn't want to um have anything changing in your scenery.
No people walking around. Uh the arm should be ideally out of the shot, though I did have successful solves with it in the shot. You don't want anything playing on your TV.
You don't want any flashing Christmas trees on the background. And the subject supposed to be super still, statue still. This is another tricky aspect of this particular subject because it's heavy.
And this column here, it's not as solid as I would like it to be. So, it's prone to wobble like this. Key takeaways of this demo.
Try to cover as many angles of views as you possibly can. Think of it like light field. You need to cover cover any possible scenario.
Rigs like this one allows you to be more precise and actually remember which parts you covered and which you don't. It ensures you do at least the full spherical coverage and everything else is a nice bonus. Don't wobble or move the model itself and don't change your environment.
The lighting as well. No passing people, no passing pets, no TV playing, no strobing Christmas tree on the background. This is just a decorated demo.
I did a proper shoot with flat lighting earlier in the day. This tool is something I wanted to bring across. It's not very popular.
I think these particular ones are discontinued. Avoid motorized ones. They allow you to scan much smaller objects only and it's really slow.
And remember, this is a very specialized case when you need the splat to be production ready. I myself see a lot of value in something like this. Scanning, you know, a hard surface object, showing it off either for client commercial work or for YouTube reviews.
Who knows? This stuff when converted into splats renders momentarily. It receives light, receives shadows.
I see a big potential in future in this. And honestly, when you master all this stuff, it doesn't take that much time. Th this turntable is permanently set in my studio in my office.
And when when necessary, I just turn on my lights. Those are permanently set up as well. And I just do those uh motions, scan something.
And now you know what to do with coverage, how to capture stuff like a boss. When scanning objects or locations outdoors, same principles applies. However, you're less likely to encounter super shiny objects outdoors, so handheld shoot would work just fine, unless you want to scan a big tree.
For that, we of course need a drone. Manual drone flying will be comparable to handheld shooting we discussed a couple of minutes ago. With the only unfortunate difference that the system will be much less tolerant to lack of co coverage even on matte objects with trees especially, it's too many details.
So any abrupt movement will throw the reconstruction off and you will end up with no splats at all. So, I developed a Houdini tool that allows me to pre-plan and design the flight in advance. The tool allows me to pull up the satellite lighter data and work around accurate 3D uh representations of the objects I want to scan.
So, I know exactly the height and branches span of the trees. So, this method with automated fast drone flight allows me to scan big objects outdoors the same way I do rigged uh scans indoors uh ensuring no coverage gaps and continuity. If you're not using Houdini, there are other uh flight planning tools available online, although they are of course not 3D and quite useless because you can't plan where the camera is looking exactly.
It's time to talk about data prep stage and I have to do a spoiler before we talk about actual training. So Joe said postshot is the best gausian splat trainer existing in there today. I tested research grade trainers such as uh Nvidia G-Splat Lickfeld, Lickfield uh Studio, but Postshot ticked many more boxes for me, especially when I found out how to finally get razor sharp splats out of it.
Poshot is paid though. Others are open source, so there's that. It's Cinema 4D of Splat's world in terms of convenience and pricing, unfortunately.
Postshot is uh an end to end convenient package uh and all you will ever need for gausian splatting. So end to end means you can take your videos or photos drag and drop them uh to postshot and it will do all the stages required and get you the best possible splat from your data. First, it will reconstruct the scene and cameras using its own SFM method.
SFM stands for structure from motion. And this method is honestly working like some kind of black magic. The most uh used system is called co map.
There is no way for me to know what postshot uses exactly, but my assumption is it is probably based on co map. In simple terms, uh, COM map takes a set of overlapping photos and figures out where each camera was in 3D space when the images were taken. As part of this process, it also reconstructs a sparse 3D point cloud of the scene.
These cameras and points are then used to train splats. Anyway, the reconstruction step in postShot is one of the most reliable ones, but super long with anything beyond 500 images. Not to mention very conservative point cloud if you need to manipulate your splat somehow.
After ideally for that, you would need a proper point representation of your scan. And postshot just doesn't give you that. It only reconstructs what it needs to train.
Naturally, I started to explore what other reconstruction tools are out there. And I have quite a few options for you, paid and free. Starting from Colap and Glowap uh source codes on the GitHub.
Colap is the heart of this whole magic and it's open source. You can either download the source code and compile your own executables or download pre-ompiled executables. Although they don't ship all the dependencies and a lot of users are complaining about Q CUDA support, I went ahead and compiled my own version that ensures CUDA and GPU utilization and made an HDA out of it to be able to reconstruct anything whenever I need directly in Houdini.
So call mapap is the heart and glow mapap is a GPU powered speed booster to do global adjustments much faster and it's all conveniently working in our HDA uh that is available in our store. I just specify where my source is uh video or pre pre-filtered images doesn't matter the tool will solve it and most of the times I will have a perfect cameras and point cloud representing my scene. I can also recreate very precise camera motion with this method in Houdini directly.
There are a few other reasons to use alternative reconstruction tools apart from speed. Often for crispy splats I need uh more than thousand images especially if the model has intricate details you know elements obscuring each other and things like that. Glow Map does work with thousands of images very efficiently.
And I exposed a few controls on the HDA to control the precision reliability as well as be able to control the density of the point cloud. For shiny hard surface scans, you need a tight, precise point cloud. As tight and clean as you can get.
Most of the times my HDA does a great job, but sometimes it doesn't. It almost guaranteed that I effed up somewhere at the capture stage out of focus image or something like that can put the reconstruction off. In that case, we have a free alternative called reality scan.
This one is meant for photoggramometry that Mega Scan guys are using for their assets. It also works wonders, especially with hard surface object. And finally, there's a paid software called Agisoft Metashape.
It is equally magical. This one is significantly faster with massive data sets with thousands and thousands of images. It's a top tier professional software for industrial photoggramometry and as such often it does the best reconstruction I've seen so far.
I myself do not own the license. I used the 30 days free trial uh because most of the times either my glow map HDA in Houdini does the job or reality scan. If you know any other SFM reconstructors, let me know in the comments because I would love to try them out.
All the mentioned uh softwares share the construction principles. So all the steps are the same just UI is a little bit different. In this video I will use reality scan to show you how reconstructions are made because it is uh available to anyone and not all of you are using Houdini.
I have this lovely Lego model that we will use as a demo. Typically for a simple object like this Lego le Lego model you wouldn't need anything beyond uh 1500 images. My recommendation in case if you have more images, prefilter them, reduce the amount of images.
If you are using postShot for everything, you can select how many images you want to use for training. If you're using my HDA in Houdini, you can select the stepping and the tool will use the right amount of images for reconstruction. If you're using reality scan, however, or metash, you will have to prefilter.
I use Nuke for that. If you just load your image, just put X2 after your frame range and it will render every second frame. You always want your images to be in images folder for proper structure further down the line.
Once you have the frames all sorted, just load them into Reality Scan and press align on the workflow tab. Finish estimation time will always be discouraging, but in reality, it finishes much sooner. Once you have the point cloud and cameras, you need to export them in comup compliant format because that's what postshot expects comb data.
If you are using metash, the process will be exactly the same. Even from UI perspective, they are really similar with reality scone. Remember these are not the splats.
This is the skeleton on which the splats will be trained later. The better the skeleton, the better the splats. I already mentioned I experimented with other trainers.
Posshot stands out as end to end solution meaning other trainers will not provide you with reconstruction features. So you would have to do the reconstruction from previous steps anyway before you can use them. Nvidia G-splat and licked field studio are research grade trainers.
They can adjust uh point cloud and cameras extrinsics as they train which potentially could produce better splats from worse data. But so far I didn't have any luck in proving that compared to past shot they're very scientific with no apparent benefit. I will obviously keep experimenting because I think adjusting the point cloud and extrinsics can be beneficial.
I just couldn't see the point yet. So postshot does not adjust the point cloud or cameras when it trains. It's locked.
That's why everything we discussed previously about reconstruction and the quality of the data you're feeding is so important. Anyway, when you have your reconstruction, just drop the reconstruction and your images into postshot. Most parameters on postshot are flexible, meaning we can change them along the way, like the amount of splats we allow and training steps.
Usually, I leave at defaults and then observe as training progresses. The SSIM graph stands for structural similarity index measure. It tells you how much your splats resemble uh with the real thing.
The higher the value, the better. We can see out of the bat our index is quite high, which is very good. Meaning we did a good job at prep stage.
Uh when training, as discussed, shiny objects are better off with as less splats as possible. Just avoids all those weird glare glare interpolations. Usually I just load the reconstruction in and limit the splats amount to what it initialized with and just let the initialized amount of splats resolve better.
If your SFM is decent, the amount of splats you're getting at initialization stage is usually enough. Just refine those a bit and you're done. These types of scans benefit from high resolution images.
helps to get all those macro details like dust on my Lego models. See not a ton of splats, not crazy amount of steps, glares on the surface look great and no artifacts from any angle because this was a rigged shoot. I covered everything in a perfect sphere.
On the opposite, if you're scanning extremely detailed objects, uh, like trees, you may benefit from downsampling your images and fitting as many splats as you can into VRAM. You may even lower the spherical harmonics degree to save VRAMm space and pack the training with as many splats as possible. Harmonics generally help better with reflective surfaces which trees are not.
So you can lower those and save on memory big time. In fact, harmonics have a very very substantial influence on the memory consumption for rendering and scene assemblies. I of course of course use Octane.
Octane was the first to support proper lighting and shadow casting with splats. Real-time spectral path render is not such a sci-fi anymore to be honest. In future videos, we will explore how to assemble scenes with splats and our pipeline, how to make digital splats of I don't know heavy models you previously made with Octane, how to make those heavy models render real time, and much more.
As usual, thank you for watching and I'll see you soon. Oh, while I remember I launched uh LMI's Discord server. If you need any support with our tools or have any general questions about the pipeline or any questions really, join the Discord.
I'm quite active there. Any questions, I'll answer there. Bye.
I think uh gausian splats will be a big part of CG and VFX work very soon and I wanted to master splats and learn everything there is to learn about techniques and tools available out there. In today's video, you're going to learn how to capture and train your splats like a boss. I'm not going to go into detail of what gausian splats are or how they operate.
Assuming you already know that because you found this video in VFX work, especially with engines such as Octane with ability to do a clean relight, shadow casting and things like that. The benefits are obvious. They show up in reflections.
They interact with refractions and they also render lightning fast. They can also be manipulated like any other geometry at some extent and I intend to use them as much as possible moving forward. What I want to show you is how to capture stuff beyond dandelions in your park and more importantly how to deal with shiny objects.
How to tune the training settings and what are the tools available today. First off, gausian splats are as good as the data that you feed into the trainer. So, you need to understand everything about the data.
With splats, more doesn't mean better quality. And it very much comes down to the type of the scenery you're training. Counterintuitively, uh, shiny objects require less splats and not as much training as you would think.
objects with many details like trees require ton of splats and the splats themselves are memory hungry. What do you do with that? Let's divide the video in several se segments.
Capture, prep, and training. I will explain the underlying principles of splat training in the later chapter. Spoiler alert, splats are better trained from videos.
They require the frames consistency and continuity and that ensures the best possible result. There are two primary factors that will influence the quality of your splats. Lighting and coverage.
If you're doing it for fun or for a very specific lighting scenario, then lighting is not as much of a concern. Just make sure you have enough of it. However, if the model is to be relit in Octane, manipulated in Houdini and reused in future, appropriate due diligence is required, especially if you're striving for production quality.
First, let's understand what uh ganian splat training does not tolerate. It does not like motion blur. It does not like depth of field.
It does not like noise. All the traits of not enough light situation. Also, go splatting is not fond of wide lenses because of the radial distortion.
Even though all other aspects may be perfect, this one optical factor will ruin your splat. Use 35 ms minimum. When scanning the object in a controlled environment, I found that even powerful video lights are not enough for details and sharpness and splats.
We need a lot of consistent light because I want to close my lens as much as possible while keeping my shutter speed quite high and ISO quite low. Sounds very much like a photo shoot. And I found that the photos are in fact the best solution.
Highspeed sync strobes and a fast shooting camera. It is basically shooting a video but with raw high-res photos. I use Godox tropes and Sony A1 camera.
I do understand it may be a bit out of budget. So I tested it with Sony A74 which I'm shooting with right now and smaller Godox lenses. Works fine but will require a bit more time because smaller flashes can't recharge as fast as studio strobes can.
And in terms of light placements, uh, I first used a lot of different soft boxes, big small ones, but then realized blasting the flash into the wide wall is the best solution. I fire three flashes on all my white walls and get the flattest light possible, which is perfect for our purpose. When scanning a location or object outdoors, it's best to choose an overcast day for the same reason, flat lighting.
Outdoors, lighting options are limited, so most likely you will have to use whatever mother nature gives you. Overcast daylight is already more powerful than any video light you'll use indoors, so you're all set. Close the lens as much as possible.
Don't use NDs. Lower your ISO, bring up your shutter speed, and off you go. This aspect took me a few dozen tests to figure out and get used to.
You can capture your photos and videos with whatever you want, but don't expect razor sharp results from something like DJI Nano, for example. Although I used it to scan a tractor and then turn it into cinematic shot with effects, it's not what this video is about. We are after macro detail here.
When capturing, we need parallax, not look around the lax. So, whenever you're shooting, make sure you're moving your body with the camera, not just changing the angle of the camera. When scanning a location, walk around more than look around, but make sure to constantly move the camera in space.
That's what uh gausian splat requires. Also, important point, don't stop moving. Gausian splats hate those moments when the camera just stops.
It confuses it. Amount of coverage will come down to the type of object you're scanning really. matte objects training will be much more forgiving to the gaps in the co coverage.
So handheld shoot will be just fine and will be the quickest option. Again make sure to move all the time and change heights and angles around the object and try to cover as many angles around the object as possible. When scanning a shiny object, the system will be much less forgiving to lack of c coverage.
It can only be saved by providing trainer with all possible angles of view. If you would have a coverage gap, it will result in heavy artifacts. The trainer will fail to interpolate between glaring angles and that's what you're going to get.
Usually subconsciously you're doing one level good and then you start mess around and you miss a lot of spots. I chose this subject specifically because it was my lab rat for many many tries. I've done dozens if not hundreds at this point tests of shiny objects.
And this running rig, which I'm using before Edwards, by the way, is the only item I failed to scan scan properly. I did have some decent results, but most of the times it's failing either from the top or from the bottom. And the shiny surfaces are always failing.
This is a tricky subject in many aspects. Not only we have super shiny surfaces, but a lot of elements that obscure each other. with complex objects such as this one.
I found that handheld is not optimal at all. Instead, I'm using this uh dual axis uh turntable from DF Cine if I'm not mistaken and it's a mechanical one which is important because motorized one would make the process even longer. This one allows me to mount the camera here and I can restrict the axis on which I'm currently shooting.
I won't be firing flash right now because it's night. I don't want to bother my family or my neighbors. I just wanted to show you the rig and how I'm doing those production level uh scans.
And I'm just literally pointing my camera and start to go off like this. What I'm also doing in addition to that uh main axis rotating around the column on which the subject is, I'm pivoting the camera on this axis as well on one circle. I can capture many angles.
Then just lower the arm and repeat the motion. I wasn't scanning the model right now because I did it earlier in the day. This is just the coverage demo.
When I'm actually shooting, I'm making the light as flat as possible. As discussed in previous chapter, the lights, the strobes are turned to walls. They bounce back, giving me quite flat light.
And I have three strobes. Right now, I have a really harsh spotlight lighting the rig. This is not what you want.
You want flat lighting. After you're done with the rigged scan, you can pick up the camera handheld and and do some finishing shots in the spots that you didn't uh cover and you can just go around around your subject and cover it as if it would be a handheld scan. Other tips, uh obviously you wouldn't want to um have anything changing in your scenery.
No people walking around. Uh the arm should be ideally out of the shot, though I did have successful solves with it in the shot. You don't want anything playing on your TV.
You don't want any flashing Christmas trees on the background. And the subject supposed to be super still, statue still. This is another tricky aspect of this particular subject because it's heavy.
And this column here, it's not as solid as I would like it to be. So, it's prone to wobble like this. Key takeaways of this demo.
Try to cover as many angles of views as you possibly can. Think of it like light field. You need to cover cover any possible scenario.
Rigs like this one allows you to be more precise and actually remember which parts you covered and which you don't. It ensures you do at least the full spherical coverage and everything else is a nice bonus. Don't wobble or move the model itself and don't change your environment.
The lighting as well. No passing people, no passing pets, no TV playing, no strobing Christmas tree on the background. This is just a decorated demo.
I did a proper shoot with flat lighting earlier in the day. This tool is something I wanted to bring across. It's not very popular.
I think these particular ones are discontinued. Avoid motorized ones. They allow you to scan much smaller objects only and it's really slow.
And remember, this is a very specialized case when you need the splat to be production ready. I myself see a lot of value in something like this. Scanning, you know, a hard surface object, showing it off either for client commercial work or for YouTube reviews.
Who knows? This stuff when converted into splats renders momentarily. It receives light, receives shadows.
I see a big potential in future in this. And honestly, when you master all this stuff, it doesn't take that much time. Th this turntable is permanently set in my studio in my office.
And when when necessary, I just turn on my lights. Those are permanently set up as well. And I just do those uh motions, scan something.
And now you know what to do with coverage, how to capture stuff like a boss. When scanning objects or locations outdoors, same principles applies. However, you're less likely to encounter super shiny objects outdoors, so handheld shoot would work just fine, unless you want to scan a big tree.
For that, we of course need a drone. Manual drone flying will be comparable to handheld shooting we discussed a couple of minutes ago. With the only unfortunate difference that the system will be much less tolerant to lack of co coverage even on matte objects with trees especially, it's too many details.
So any abrupt movement will throw the reconstruction off and you will end up with no splats at all. So, I developed a Houdini tool that allows me to pre-plan and design the flight in advance. The tool allows me to pull up the satellite lighter data and work around accurate 3D uh representations of the objects I want to scan.
So, I know exactly the height and branches span of the trees. So, this method with automated fast drone flight allows me to scan big objects outdoors the same way I do rigged uh scans indoors uh ensuring no coverage gaps and continuity. If you're not using Houdini, there are other uh flight planning tools available online, although they are of course not 3D and quite useless because you can't plan where the camera is looking exactly.
It's time to talk about data prep stage and I have to do a spoiler before we talk about actual training. So Joe said postshot is the best gausian splat trainer existing in there today. I tested research grade trainers such as uh Nvidia G-Splat Lickfeld, Lickfield uh Studio, but Postshot ticked many more boxes for me, especially when I found out how to finally get razor sharp splats out of it.
Poshot is paid though. Others are open source, so there's that. It's Cinema 4D of Splat's world in terms of convenience and pricing, unfortunately.
Postshot is uh an end to end convenient package uh and all you will ever need for gausian splatting. So end to end means you can take your videos or photos drag and drop them uh to postshot and it will do all the stages required and get you the best possible splat from your data. First, it will reconstruct the scene and cameras using its own SFM method.
SFM stands for structure from motion. And this method is honestly working like some kind of black magic. The most uh used system is called co map.
There is no way for me to know what postshot uses exactly, but my assumption is it is probably based on co map. In simple terms, uh, COM map takes a set of overlapping photos and figures out where each camera was in 3D space when the images were taken. As part of this process, it also reconstructs a sparse 3D point cloud of the scene.
These cameras and points are then used to train splats. Anyway, the reconstruction step in postShot is one of the most reliable ones, but super long with anything beyond 500 images. Not to mention very conservative point cloud if you need to manipulate your splat somehow.
After ideally for that, you would need a proper point representation of your scan. And postshot just doesn't give you that. It only reconstructs what it needs to train.
Naturally, I started to explore what other reconstruction tools are out there. And I have quite a few options for you, paid and free. Starting from Colap and Glowap uh source codes on the GitHub.
Colap is the heart of this whole magic and it's open source. You can either download the source code and compile your own executables or download pre-ompiled executables. Although they don't ship all the dependencies and a lot of users are complaining about Q CUDA support, I went ahead and compiled my own version that ensures CUDA and GPU utilization and made an HDA out of it to be able to reconstruct anything whenever I need directly in Houdini.
So call mapap is the heart and glow mapap is a GPU powered speed booster to do global adjustments much faster and it's all conveniently working in our HDA uh that is available in our store. I just specify where my source is uh video or pre pre-filtered images doesn't matter the tool will solve it and most of the times I will have a perfect cameras and point cloud representing my scene. I can also recreate very precise camera motion with this method in Houdini directly.
There are a few other reasons to use alternative reconstruction tools apart from speed. Often for crispy splats I need uh more than thousand images especially if the model has intricate details you know elements obscuring each other and things like that. Glow Map does work with thousands of images very efficiently.
And I exposed a few controls on the HDA to control the precision reliability as well as be able to control the density of the point cloud. For shiny hard surface scans, you need a tight, precise point cloud. As tight and clean as you can get.
Most of the times my HDA does a great job, but sometimes it doesn't. It almost guaranteed that I effed up somewhere at the capture stage out of focus image or something like that can put the reconstruction off. In that case, we have a free alternative called reality scan.
This one is meant for photoggramometry that Mega Scan guys are using for their assets. It also works wonders, especially with hard surface object. And finally, there's a paid software called Agisoft Metashape.
It is equally magical. This one is significantly faster with massive data sets with thousands and thousands of images. It's a top tier professional software for industrial photoggramometry and as such often it does the best reconstruction I've seen so far.
I myself do not own the license. I used the 30 days free trial uh because most of the times either my glow map HDA in Houdini does the job or reality scan. If you know any other SFM reconstructors, let me know in the comments because I would love to try them out.
All the mentioned uh softwares share the construction principles. So all the steps are the same just UI is a little bit different. In this video I will use reality scan to show you how reconstructions are made because it is uh available to anyone and not all of you are using Houdini.
I have this lovely Lego model that we will use as a demo. Typically for a simple object like this Lego le Lego model you wouldn't need anything beyond uh 1500 images. My recommendation in case if you have more images, prefilter them, reduce the amount of images.
If you are using postShot for everything, you can select how many images you want to use for training. If you're using my HDA in Houdini, you can select the stepping and the tool will use the right amount of images for reconstruction. If you're using reality scan, however, or metash, you will have to prefilter.
I use Nuke for that. If you just load your image, just put X2 after your frame range and it will render every second frame. You always want your images to be in images folder for proper structure further down the line.
Once you have the frames all sorted, just load them into Reality Scan and press align on the workflow tab. Finish estimation time will always be discouraging, but in reality, it finishes much sooner. Once you have the point cloud and cameras, you need to export them in comup compliant format because that's what postshot expects comb data.
If you are using metash, the process will be exactly the same. Even from UI perspective, they are really similar with reality scone. Remember these are not the splats.
This is the skeleton on which the splats will be trained later. The better the skeleton, the better the splats. I already mentioned I experimented with other trainers.
Posshot stands out as end to end solution meaning other trainers will not provide you with reconstruction features. So you would have to do the reconstruction from previous steps anyway before you can use them. Nvidia G-splat and licked field studio are research grade trainers.
They can adjust uh point cloud and cameras extrinsics as they train which potentially could produce better splats from worse data. But so far I didn't have any luck in proving that compared to past shot they're very scientific with no apparent benefit. I will obviously keep experimenting because I think adjusting the point cloud and extrinsics can be beneficial.
I just couldn't see the point yet. So postshot does not adjust the point cloud or cameras when it trains. It's locked.
That's why everything we discussed previously about reconstruction and the quality of the data you're feeding is so important. Anyway, when you have your reconstruction, just drop the reconstruction and your images into postshot. Most parameters on postshot are flexible, meaning we can change them along the way, like the amount of splats we allow and training steps.
Usually, I leave at defaults and then observe as training progresses. The SSIM graph stands for structural similarity index measure. It tells you how much your splats resemble uh with the real thing.
The higher the value, the better. We can see out of the bat our index is quite high, which is very good. Meaning we did a good job at prep stage.
Uh when training, as discussed, shiny objects are better off with as less splats as possible. Just avoids all those weird glare glare interpolations. Usually I just load the reconstruction in and limit the splats amount to what it initialized with and just let the initialized amount of splats resolve better.
If your SFM is decent, the amount of splats you're getting at initialization stage is usually enough. Just refine those a bit and you're done. These types of scans benefit from high resolution images.
helps to get all those macro details like dust on my Lego models. See not a ton of splats, not crazy amount of steps, glares on the surface look great and no artifacts from any angle because this was a rigged shoot. I covered everything in a perfect sphere.
On the opposite, if you're scanning extremely detailed objects, uh, like trees, you may benefit from downsampling your images and fitting as many splats as you can into VRAM. You may even lower the spherical harmonics degree to save VRAMm space and pack the training with as many splats as possible. Harmonics generally help better with reflective surfaces which trees are not.
So you can lower those and save on memory big time. In fact, harmonics have a very very substantial influence on the memory consumption for rendering and scene assemblies. I of course of course use Octane.
Octane was the first to support proper lighting and shadow casting with splats. Real-time spectral path render is not such a sci-fi anymore to be honest. In future videos, we will explore how to assemble scenes with splats and our pipeline, how to make digital splats of I don't know heavy models you previously made with Octane, how to make those heavy models render real time, and much more.
As usual, thank you for watching and I'll see you soon. Oh, while I remember I launched uh LMI's Discord server. If you need any support with our tools or have any general questions about the pipeline or any questions really, join the Discord.
I'm quite active there. Any questions, I'll answer there. Bye.
