we present a new approach that allows to capture diverse static 3d scenes from image data and to render them from new viewpoints our approach uses point clouds to represent scene geometry such point clouds can be obtained by stereo matching registered RGB image sets alternatively they can be obtained by fusing depth maps from registered RGB D videos even with a noisy and incomplete underlying point cloud our approach can generate complete and realistic views for novel viewpoints to facilitate rendering from new viewpoints our approach assigns each point a neural descriptor of low dimensionality the descriptor contains information
about local photometric and geometric properties in the experiments our neural descriptors are eight dimensional here we color code each point with the first three PCA components of the corresponding descriptor [Music] to render a new view we project points onto the novel view at several resolutions using descriptor values as pseudo color a rendering convolutional network is then used to transform the multi resolution rendering into a photorealistic image to capture the appearance of a scene we fit the descriptors and the rendering network parameters to a given set of views the fitting uses back propagation and minimizes the
loss between the rendered views and the ground truth frames here if we compare our method on the right with the pics to pix ablation which essentially replaces neural descriptors with the original colors [Music] here we compare our method to the deferred neural rendering system which uses a mesh rather than a point cloud is a geometric proxy in general both methods achieve similar results yet ours perform better wherever meshing fails for example on thin object parts [Music] when the images in the training set suffer from inconsistent exposure or have inaccurate registration our method can suffer from
temporal flickering this effect can be alleviated by applying temporal smoothing and spatial anti-aliasing to the renderings of the point cloud before they are processed by the rendering network among other objects we assess the performance of our method on human portraits here we show one of the portraits that we have created we can combine two scenes into a single one for that we first fit both of them while keeping the coefficients of the rendering Network the same we then merge the point clouds after rough geometric alignment the rendering network can then be applied to the combined
point cloud [Music]