Meshroom: 3D models from photos using free photogrammetry software

[Music] welcome to another video from explaining computers this time we're going to check out a photogrammetry application called meshroom photogrammetry turns a series of photographs into a 3d object so for example a few years ago i used a free photogrametry application called autodesk 123d catch to turn this wooden elephant into this 3d print however since that time one 123d catch has been replaced with an application called recap pro that costs 340 dollars a year which gets us to meshroom which is a free open source photogrammetry application so let's go and take a closer look right

here we are on the website for alice vision where we can download mushroom alice vision is a framework that provides computer vision tools for turning photographs into 3d models with meshroom being a photogrammetry application built around alice vision all of this is supported by the alice version association which is a non-profit with the ambition to democratize 3d digitization technologies from photographs and here as you can see users of meshroom can make a donation to support the work of alice vision if they wish back on the main website if we click on meshroom to download the

software i'm sure you've guessed that we can scroll down and we will find two download links one for windows and one for linux also as it notes mesh room requires an nvidia gpu with cuda support and here on my test rig we're running with a two gigabyte gt1030 note that the broader recommended hardware specification is an i7 pc or equivalent with 32 gigabytes of ram however here we're running on a quad core three gigahertz amd a8 3870 k with 16 gigabytes of ram on which mesh room runs with no issues finally on the hardware side

it's worth noting that regardless of what's stated here it is possible to run mushroom without an nvidia gpu and to do this we go across some of the support pages and we discover we can do it if we use draft meshing and if we click on this you'll find that using this is very well documented but everything to do with mushroom is very well documented but if you do use draft meshing you'll get lower quality results and it's not something i've tried out but i just thought i'd mention this in case you haven't got an

nvidia gpu anyway let's download the software so i'll click here on the windows link as i'm currently on a windows 10 machine so i'll do that and save the file which is 327 megabytes in size and with the file downloaded if we go across to the folder in which it's contained we need to extract the file so we'll extract there and with the extraction completed no more installation is required because all we have to do is to open up the folder where mushroom has ended up there we are let's just give ourselves a bit more

space on the screen and all we have to do to run the package is to click on the meshroom xc file and here we are in this amazing software application meshroom is incredibly well designed to incorporate both a high level and a low-level user interface specifically at the top of the screen we have the high-level ui which should allow almost anybody to produce a 3d object from a set of still images and then at the bottom of the screen we find the low-level ui which is node based and should allow researchers and advanced users to

take complete control of the photogrammetry process and i find all the stuff going on down here very interesting indeed but in this video we're going to stick with the high level interface here at the top of the screen and of course you want some demonstrations i'm going to do two in this video and i thought i'd start out by creating a 3d model of the wooden elephant like the one i previously generated in autodesk 123d catch for this i still have the set of 36 images that i took as i rotated around the object to

photograph it from all possible angles so to start the process here in mesh room the first thing we need to do is to name our files i'm going to do a file and i'm going to do a save and we'll go to a folder i created called photogrammetry where you can see i've already got a folder which contains the elephant images and i'm going to save this project with the name elephant test and it's worth noting that it's very important to keep track of files and folders and their locations when you're working in meshroom as

output from the program is saved in an auto-generated folder called meshroom cache which will sit alongside your project files so with our file set up we'll now load in the images of the elephant so we'll go across to file and import images and again back to photogrammetry here are the images and we'll just select all of these and bring them in to the program like this and here they all are sitting there elephant photographed from lots of different angles and all we now have to do to begin the photogrammetry process is to click on start

and unlike many other photogrammetry applications meshroom does all of its processing locally it doesn't rely on a cloud server so this is going to take a rather long time and we can see the progress being made in two places we can see a progress bar here at the top of the screen this will take a while to move across and we can also see down here in this node view we can see which node is doing something and again these will be curled in as things move across and whilst mushroom is getting on with its

task it's very important to stress that how well the photogrammetry process will work depends on the quality of the images it's provided with so just what do you need to do to achieve photogrammetry success well the first thing is to take a lot of images an absolute minimum of 20 and to get a really good scan of a detailed object several hundred photographs need to be sharp with no motion blur as little noise as possible and good depth of field so that no part of the target object is out of focus lighting is also critical

and needs to be as flat and diffuse as possible with minimal shadows and no reflections in an ideal world an object will be photographed with lots of identical cameras all taking a picture at exactly the same point in time as can happen in a booth like this one i saw at the eye maker store in london however most of us have to work with one camera which is rotated around the object and it's worth stressing that rotating the object itself can be problematic if this changes how different parts of it are lit oh and two

final tips firstly make sure you don't change the orientation of your camera or in other words shoot everything landscape or everything portrait and also don't change the focal length of your lens or in other words shoot every picture on the same zoom setting right back in mesh room about 90 minutes has passed and the photogrammetry process has completed as we can see and in the 3d viewer we've now got some data and if i just zoom in on that a bit in fact i'll give us a bit more space on the screen for it we'll

just reorganize things a little bit like that there we are and you can see what we have here and i can move around with the mouse we can see a representation of the camera position for every one of the images if i click on there you'll see it actually moves them around let's give us a bit more space to see more those as well it'll flick between cameras and we can also zoom in on the elephant which you can see is sitting in the middle of this let's just go in a bit closer on the

elephant there it is it's upside down that doesn't doesn't really matter we can still cope with that we can change the size of the points on the elephant the ones that have been created there as you might see coming up down there and we can also change the size of the cameras if it makes it easier to see things and something else we can do here which is very interesting and we can click on sync with image selection if i do that we see the elephant based upon the image that we've actually got selected over

here so it's showing the point data that's coming from that particular image which i i find fascinating so that's a that's just a cool little feature here anyway what we're most interested in of course is actually using this model in other applications and you might be thinking how do we export this model from mesh room and there aren't any export functionality things here in the file menu and the reason for that is because mesh room saves all of its output in its cache folder as i said earlier so if we go across to the photogrammetry

folder you'll see it's created a mushroom cache file here and if we go into that and we go into meshing you will see we have a very long name folder there and inside that we have a mesh object which here is for the elephant and if we open this up it'll open up in the windows 3d viewer and i'll f9 to use the whole screen and here we are we have the elephant still upside down and there's a bit of a extra date a few extra polygons and things around the top but we can sort

this out we can actually remove these extra polygons and things fairly easily as i'll show you in the next segment of the video but just before we do that it's worth pointing out that here in this video all i'm going to be looking at is the geometry created by meshroom the actual object we're looking at on the screen here without any color on it but mushroom has also saved loads and loads of texture data so if we take a look at that let's close down that and that and go into text string in the cache

and again we found a file for this particular project hello look there's the texture we open it up and this is a png file containing all of the textures all of the color data for the model and also down here we've got a textured mesh object let's just open up that as well so if you're interested in having a textured object and working with that you can create one using mesh room i'm very impressed with the results the elephant might still be upside down but other than that it's having a fun time here in the

windows 3d viewer to clean up our mushroom output we're going to use a free program called mesh mixer which can be downloaded from meshmixer.com meshmixer has been an autodesk product since 2011 although it's no longer in development because a lot of its features now found in fusion 360 the flagship product from autodesk this said mesh mixer still remains very handy standalone tool especially for preparing 3d objects for 3d printing so let's close down the website i won't show you a download i'm sure you can sort that one out i've got meshmixer installed there so we'll

run it up and maximize the program and we'll import our output from mesh room going into that cache again going into meshing and the project file and open up the elephant like that and here we can either move around using the mouse in the normal sort of way or we can use this little cube at the top to move things around as well and as you can see the elephants come out pretty well there are a few problems on the top of the object and under the belly because inevitably the photographs don't show that bit

of detail so well but certainly things need to be sorted out a bit because for start the elephant on its side so let's sort out that by going to edit and transform we've now got some little handles we can pull to turn the elephant around and that looks pretty good to me so we'll accept and the second thing i'm going to do is to do a plane cut which allows us to cut a plane through the object well named tool and if we just drag this down to take it down towards the base of the

feet above all the spurious polygons we'll leave it about there i think again that looks about okay and we'll accept that and there we are we've got a much neater elephant and later in the video i'll demonstrate several other mesh mixer tools but before that let's go in search of another photogrammetry subject guess what look what i found a rather large stone lion a chinese guardian lion no less and i've got my camera so i'm going to take lots of pictures of the lion all around it so we can turn it into a 3d object

and by the magic of filmmaking here we are in mesh room where i've loaded in the 118 pictures of the lion that i took and i've also run the photogrammetry process which took about seven hours i run it overnight and we've got some output here in the 3d viewer and as you can probably see we've got lots of data here we've got the lion in the middle but also lots of other things we've got bits of wall here all types of stuff and if we zoom in on the lion like that you can see it

is sitting there let's just take the camera slice down so we can see it a bit more clearly we have actually got our final object sitting there and if we go across to the mesh room cache and we go into meshing and we open up the folder it's created for this project and also the object called mesh obj like that here we have the data we've got to work with and the first time i saw this i went oh dear that's a bit of a mess but do not fear the lion is sitting in the

middle of all this if we zoom in there it is and the problem we've had here is one that you often have with photogrammetry if you're photographing an object outside because you get lots of data around the object which isn't the object you want but let's try and get the line a bit closer to us on the screen there we are and as you can see what we do have of the lion is nice and detailed getting a bit close there it's a it's not bad is it the actual model looks looks pretty good so

what i now need to do is to take this into mesh mixer and to clean it up in order to extract the line from the middle of this spurious data greetings here i am back again in autodesk mesh mixer where i've cleaned up the final object as you can see and it's come out pretty well i'm very pleased with it it's not perfect there are issues in particular around mouth because the photographs didn't reveal detail inside the mouth so inevitably that's a problem with the geometry and there's also a problem on this back leg where

there's an internal cavity that i've not actually cleaned up i've spent about two hours to get the model into this state and if we look at the level of detail here let's just bring up the wireframe view you can see we've got a very very dense mesh masses massive detail in this model and i think it's very interesting to compare it with one of the photographs of the real stone lion from which the model was derived it's fantastic i think what meshroom has achieved with its photogrammetry process but i'm sure some of you are wondering

chris how did you get to this final model from the initial meshroom output so let's go back to that output which looked like this when i first loaded it into meshmixer and the first thing i did was to get it aligned with the ground plane using edit and transform and then i had to start getting rid of all the extra polygons we don't actually need in the model and to do that i use select and then selected lasso and drew on the screen to pick up polygons that weren't needed like that and clicked on delete

and after a lot of rotating around and selecting and deleting i got to this point where there's pretty much only the lion left in the scene we've got a bit of wall over here a bit of ground but i soon got rid of those and ended up with a line like this next i dealt with scaling by going to edit and transform again and i set the height of a line to 105 millimeters next i went to analysis and inspector to try and make sure our object has got a water tight mesh because if you

want to 3d print something the geometry has got to be watertight and as you can see there's a lot of issues with the particular object but fortunately in meshmixer there is an auto repair all option so i just clicked on that which fortunately sorted everything out this said the liner still got a bit of a problem because on the top of its head it's got this strange body of polygons a strain sort of hat on the top of it and this is because the actual stone line that i photographed was over seven feet tall so

even lifting my camera above my head i couldn't take good pictures of the top of it so once again what i needed to do was to go to select and to select all of these polygons and to delete them and with that process completed i ended up here with a lion without all the strange polygons on the top of its head but was a hole in the top of its head which needed to be filled in and a hole like this could be filled in in various ways but i went for the simplest possible solution

which was to go to edit and make solid and this as you can see does fill in the hole if in a rather rudimentary fashion although here we've lost a lot of resolution a lot of detail in the object with the current settings but we can fix that by changing the settings for make solid and i think i picked something like that and then press the button at the bottom of the screen which is difficult to see here on my current scaling settings but i can click it down here and by jingo i now ended

up with a far happier lion to which all i needed to do was to go and edit and a plane cut which gave a nice smooth base as we can see here in the final object and so the final thing to do was a little bit of exporting first i exported an stl file for 3d printing and then secondly an obj file so i could import the lion into the lightwave 3d modeling package in which i make all of my 3d animation and here i first rendered the lion with a flat shade just to see

what it looks like and then i gave it a gold shader and rendered out a bit of animation spinning around the model and i think it's come out very well indeed certainly a lot more time could have been spent cleaning up the top of the model and that internal cavity around the back of the leg but there's no doubt at all we've proved here that meshroom is a great piece of software for performing photogrammetry for making a 3d object from a series of still photographs guess what i'm now 3d printing the guardian lion to achieve

this i loaded the stl file that we generated in mesh mixer into the curious slicing software which in turn produced a g-code file for the 3d printer obviously extruding the line into physical reality will take many hours so we'll now let the printer get on with its task and here we are with all the support structures removed we have a final 3d print and it's really great to hold in my hand an object that i last saw in the physical world as a two and a half ton piece of stone however the lion's journey is

not quite complete yet as i'm going to do a little filling and sanding and painting so that we end up with a final model that can guard something of importance on my desk and i for one think this is a great final output from our photogrammetry process as we've seen in this video meshroom is an amazing piece of software that allows us to turn something like this into something like this but now that's it for another video if you've enjoyed what you've seen here please press that like button if you haven't subscribed please subscribe and

i hope to talk to you again very soon [Music] you