In recent years, neural rendering has emerged as a prominent and prolific field within computer graphics research. One groundbreaking innovation in this domain is the concept of Neural Radiance Fields (NeRFs), which has revolutionized 3D modeling. NeRFs enable the creation of intricate, high-fidelity 3D scenes from novel viewpoints using only a sparse set of images and corresponding camera positions. This neural network architecture leverages connections between these images and foundational computer graphics techniques like ray tracing to produce realistic scenes.
NeRF
In NeRFs, the scene is represented using fully connected network architectures. These networks process 5D coordinates comprising camera positions and spatial locations to generate the color and volume density of each point within the scene. The loss function of NeRFs draws inspiration from conventional volume rendering techniques, incorporating color information for every ray traversing the scene directly into the neural network’s weights.
While NeRF architecture excels at generating high-quality renders of new viewpoints in static scenes, it encounters challenges, primarily due to the time-intensive process of encoding object shapes into neural network weights. Training and inference with NeRF models can be prohibitively time-consuming, limiting their applicability in real-time scenarios.
Gaussian Splatting
In contrast, Gaussian Splatting (GS) offers comparable render quality with faster training and inference times. GS achieves this by dispensing with neural networks and instead encoding object information into a set of Gaussian distributions. Gaussian Splatting model 3D scene by a collection of 3D Gaussians defined by a position (mean), covariance matrix, opacity, and color represented via spherical harmonics (SH). GS algorithm creates the radiance field representation by a sequence of optimization steps of 3D Gaussian parameters (i.e., position, covariance, opacity, and SH colors). The key to the efficiency of GS is the rendering process, which uses projections of Gaussian components. These Gaussians can be utilized similarly to classical meshes, allowing for the swift development of models, especially for dynamic scenes. However, conditioning GS is challenging, requiring a large number of Gaussian components.
Both NeRFs and GS present distinct advantages and drawbacks. Our team’s primary objective is to develop new representations for both NeRFs and Gaussian Splatting to address a fundamental challenge in neural rendering.
– Przemysław Spurek