Geometric Algebra Planes: Convex Implicit Neural Volumes

We thank the reviewer for their thoughtful engagement with our work.

Response to other comments and suggestions: Thank you for noticing the inconsistency with figure 1. This is a typo, the full figure is in the appendix (figure 6) due to space limitations in the main text.

Response to questions:

Q1 (full architecture figure in appendix): We did not have enough space.

Q2 (performance of Tri-Planes with concatenation): The performance is very similar to original Tri-Planes (with feature addition). For instance, in the 3D segmentation with 2D supervision task (task 1), Tri-Planes with concatenation using the same channel dimensions and resolutions (leading to slightly more parameters as the input feature size to the MLP decoder is increased) achieves the following IOU results: 0.691 (convex), 0.870 (semiconvex), 0.868 (nonconvex). The results achieved by addition of the features were: 0.681 (convex), 0.868 (semiconvex), 0.863 (nonconvex). Since the difference is very minor, we present Tri-Planes in its original form.

Q3 (guidelines for when to use convex, semi-convex, vs. nonconvex GA-Planes): The radiance field reconstruction task is inherently nonconvex due to the nonlinear forward model. Thus, using a semiconvex or convex model does not provide any theoretical or practical advantages. For the other 2 tasks, convexity and semi-convexity improve stability with respect to random initialization and training procedure. However, empirically we find that nonconvex models also work well once model size is large enough; convexity is most essential for small models that are most sensitive to random initialization. Convex models can also be trained very quickly with specialized convex solvers (a future direction for our project). Semiconvex models don’t have fast, dedicated solvers like convex models, but they also achieve the global minimum of the objective and are thus more stable compared to nonconvex models with similar performance.

Thank you for your comments and thoughtful engagement with our work. We address your concerns below.

Response to E1 (other baselines and nonconvex decoders): Thank you for referencing [1]. In our 3D segmentation experiments, we use a different dataset compared to data types mentioned by the survey paper. The task we consider is to lift the 2D segmented images of a particular scene to get the 3D segmentation of the object, which is very similar to the normal radiance field reconstruction task, with the only difference being the lack of color prediction. We chose this task as it has a linear forward operation and can be formulated as a convex optimization problem with the use of a convex loss function (like mean-squared error). In the referenced paper, the segmentation is done on different representations of the 3D space. By focusing on optimizing a method for reconstructing 3D scenes from 2D images, we allow for the use of any highly performant, open source pre-trained segmentation model to be used to produce binary masks to be lifted to 3D by our method. Thus, any improvement of 2D segmentation models can be directly reflected in our 3D segmentation output. Regarding decoders, our GA-Planes representation can be used as a feature embedding with any desired decoder including existing INRs as decoders; indeed our nonconvex GA-Planes model uses a standard nonconvex MLP decoder. For generative modeling, a similar approach like EG3D paper (using Tri-Plane representation) could be used with GA-Planes as the underlying representation (this is an exciting direction for future work).

Response to E2 (clarification on video segmentation): The video segmentation task we consider is the same as 3D segmentation with 2D+time (xyt) replacing 3D (xyz). We show performance on temporal superresolution in this task, which is interpolating missing frames, similar to rendering missing view angles. Again, here we are taking per-frame segmentation masks as training input and our goal is to essentially interpolate these 2D masks into a coherent 3D mask across the video (including missing test frames).

Response to E3 (shape/image representation): Our 3D segmentation experiment is performing shape fitting (with either direct 3D supervision via space carving, or indirect 2D “tomographic” supervision); we will clarify our description of this task in the revision. We also include image fitting experiments to support our theoretical analysis (figure 7 in appendix of original submission).

Response to GA references: Thank you for pointing us to these models using geometric algebra. We are happy to include them in our revised related works discussion.

Response to weaknesses:

W1 (clarification on existing experiments): Please refer to our responses above to E1, E2, and E3.

W2 (speed and memory): No, our method does not come with a speed or memory overhead. We compare the number of parameters (memory) and performance metrics such as PSNR in figure 3. We also added a plot comparing memory and training times of GA-Planes and K-Planes with the configurations as in figure 3 (https://imgur.com/a/Kgm12Pk). Note that the variations in training time are directly influenced by the input feature dimension that gets decoded by the MLP. Comparing GA-Planes and K-Planes with the same feature dimensions, their training times are on par. As noticeable on the plot, GA-Planes had larger feature dimensions on some models, leading to longer training times (we selected the configurations to optimize memory vs. PSNR, which we list on appendix table 5).

Response to suggestions: Thank you for noticing the typo about referring to figure 1. We will put the full figure in the main text.

Thank you for your thorough review.

Response to 1 (comparison with volume-based method): The pink dashed line marked as “GA-Planes ablation (volume only)” is a volume-based method. In the notation we use, this is D($e_{123}$). We believe the inferior performance is caused by the coarser resolution of the 3D grid compared to the higher resolution that 2D and 1D grids can afford with the same total parameter count. Increasing the resolution of the volume features is expensive in terms of memory. Based on our 2D theory, the use of an MLP decoder enables high effective rank (limited by the resolution) representations even through “low-rank” components such as lines. Thus, the volume-only model does not achieve the expressivity of a combination of high-resolution lines/planes at the same memory budget.

Response to 2 (local minima vs model expressiveness for volume-based vs factorized and implicit models): We note that for NeRF training, the objective function itself is nonconvex, so regardless of the volume parameterization there is a risk of getting stuck in a local minimum. However, if we consider convex objectives such as shape/video/image fitting, then a 3D grid or a concatenation of tensor features (e.g. our convex GA-Planes model) should be able to optimize globally, whereas a tensor factorization involving feature products (e.g. our nonconvex GA-Planes model, and K-Planes) would still be nonconvex. As described above, we believe that the primary reason behind the performance improvement we observe compared to a 3D grid is based on the difference in resolution, which improves the representation capacity or effective rank of our models, rather than a difference in optimizability.

Response to 3 (analysis of MLP-based models): With a purely MLP-based model, we cannot define an equivalent matrix factorization optimization problem like the ones we provide in our theoretical analysis of GA-Planes. However, some prior works [1] have analyzed the NeRF model with Fourier features through the lens of neural tangent kernels (NTKs).
[1] https://arxiv.org/pdf/2006.10739