we present morphable diffusion 3D consistent diffusion for single image Avatar creation the introduction of large 3D scan collections allowed building of expressive articulated models of 3D human bodies faces and hands one of the primary applications for such models is to reconstruct photorealistic avatars from highly underconstrained setups such as monocular video or a single image recent Works have tackled this problem with novel neural rendering techniques and generative modeling in this work we build on the state-of-the-art multi view consistent diffusion model and enhance the Reconstruction quality while enabling additional control for generating animatable photorealistic avatars from a
single image given a single input image and a morphable mesh with a desired facial expression our method directly generates 3D consistent and photorealistic images from novel viewpoints using these generated images We Could reconstruct a Coors 3D model using off-the-shelf neural surface reconstruction methods such as News 2 a single Den noising step of our method consists of four parts first a noise feature volume is constructed by lifting and processing the 2D noise features into a 3D Volume which is processed by a 3DC in and to produce a spatial feature grid next a morphable noise volume is
instructed by attaching the 2D noise features onto mesh vertices that are processed by a sparse convet to Output a 3dmm aware feature volume then the resulting feature grids are Blended via addition the Blended noise volume is further interpolated to the frustum of a Target view that we wish to synthesize lastly the noisy Target image the input image and the target feature frustum are processed by a pre-trained 2du net to predict the denoised image at the next iteration we evaluate our method on the task of Novel view synthesis of faces given a single portrait image our
method produces more plausible and realistic novel views while maintaining better Mt view consistency compared to state-of-the-art methods quantitative comparison on the face scape data set demonstrates that our method produces the best perceptual metrics we show the meshes of our method and the Bas lines reconstructed with News 2 please pay attention to the subject's facial expression and side view renderings our method could also be applied to novel view synthesis of full bodies with explicit conditioning on simple x vertices our method generates fullbody human images and more accurate body poses compared to the baselines quantitatively our method
demonstrates a considerable improvement over all the baselines across all metrics our method also supports novel facial expression synthesis with the proposed training steam that disentangles the Reconstruction Guided by the input image and the animation Guided by the under lying morphable model note that columns with dagger symbols are models with per subject fine-tuning our method is the only one that successfully synthesize novel views for a novel facial expression while retaining High Fidelity quantitatively both of our models with or without fine-tuning outperform the corresponding Bas lines on most of the metrics We compare the reconstructed mesh of
our method with mofen Earth both models synthesize novel facial expressions from a single input image our method produces better resemblance we find to in our method on 16 views of the test subject in neutral expression and compare the fine-tune model with to Fusion Rick our fine-tune model improves the resemblance slightly compared to the results with single input image and produces images with better Fidelity compared to the Baseline we show that our method could be applied to animate a full body image using simple meshes as driving signals while our Mor able model exhibits promising capabilities it
is essential to acknowledge its inherent limitations that could impact its widespread applicability such as identity flickering during expression interpolation and general identity deterioration in certain cases our method is also unable to generalize well to in the wild input images due to data set bias in ethnicity and hair styles understanding and mitigating these occasional identity discrepancies and switching to a more diverse multi view human face data set would undoubtedly enhance the overall robustness and reliability of our morphable model future work can also involve adapting the method for realtime animation of the model from the video stream
thanks for watching
