Geometric Algebra Planes: Convex Implicit Neural Volumes

Thanks for the thoughtful review and helpful comments. We have revised our paper with extensions to our theoretical and experimental results; changes are summarized in the joint comment to all reviewers. We respond to reviewer-specific comments and questions below.

$**Use of Geometric Algebra**$

We have edited the discussion of geometric algebra in the revised paper, to clarify our use of the algebra in geometric algebra, and provide a more intuitive introduction since we agree many readers may be unfamiliar with geometric algebra (GA). Thanks for the suggestion. In particular, we restrict our (nonconvex) GA-Planes model to use only products of features that are valid in GA. Therefore, our modeling choice heavily relies on the geometric product in GA. In three-dimensional GA, the geometric product is used to construct higher grade objects, such as bivectors and trivectors. As an example consider the canonical basis vectors $e_1, e_2, e_3$. The geometric product $e_1e_2$ is a bivector, i.e., an area form, representing the plane spanned by $e_1$ and $e_2$. The geometric product of $e_1e_2$ and $e_3$ is the trivector $e_1e_2e_3$ representing the volume form. On the other hand, the geometric product ($e_1e_2$) and ($e_2e_3$) is $e_1e_2e_2e_3 = e_1e_3$ since $e_2e_2 = e_2^2 = 1$ due to the fact that all vectors squared result in Euclidean norm squared.

In our model we allow all the features that obey GA, e.g., we allow line $\times$ plane features but not plane$\times$plane, because in GA line $\times$ plane = volume but plane $\times$ plane is not. For comparison, the K-planes method uses features like plane $\times$ plane $\times$ plane not obeying GA. The TensoRF model does not use all the GA based features, but a subset of them only. Specifically, TensoRF uses either line $\times$ line $\times$ line or line $\times$ plane (the main method) but not both of them at once, and also lacks the volume grid feature.

$**Piecewise constant interpolation in the theorems**$

Although the theorem statements in the initial submission specify use of piecewise constant (nearest neighbor) interpolation, this is actually not required for the theorem and has been removed in the revision. We have also streamlined the presentation of the theorems, and included a more thorough discussion of interpolation in the revised appendix A.2 (before the proofs). In the theorem statements, the outer product of low-rank components U and V actually has the same resolution as the target matrix M, so there is no interpolation needed for these parameters. We have also experimentally verified that, in the context of 2D image fitting (matching our theorems), our models perform very similarly whether they use nearest neighbor (right) or more realistically linear/bilinear interpolation (left) (the figure is in revised appendix A.3 as well as here: https://imgur.com/a/523NGVa). In our 3D experiments, we use linear/bilinear/trilinear interpolation of features.

$**Theory for MLP decoders**$

In our original submission, we included theorems using linear decoders in the main text, and analogous results with convex and nonconvex MLP decoders in the appendix (due to space limitations, since the linear versions are a bit simpler). However, we agree with the reviewer that the results with MLP decoders are closer to practice and thus more interesting, and we have accordingly moved them to the main paper in the revision. Table 1 in the revised paper summarizes key takeaways from these theoretical results, comparing the maximum rank of a 2D model depending on how it combines features (addition, concatenation, or multiplication) and how it decodes features (linear, convex MLP, or nonconvex MLP). These results are then validated by the 2D image fitting experiments in the revised paper (and the link https://imgur.com/a/523NGVa). We believe this rearrangement of the MLP decoder theory and accompanying experimental validation greatly improves our paper; thanks for the suggestion.

$**Experiments with convex objectives**$

The review states “many of the experiments in section 5 are unable to use convex objective functions”...although it is true that our radiance field experiments are inherently nonconvex (due to the nonlinear forward model), all other experiments in section 5 are using convex objectives (i.e. 2 out of 3 of our experimental settings are based on convex optimization; we include the nonconvex radiance field experiment to show versatility of GA-Planes).

I thank the authors for their considerate reply and for addressing my concerns and suggestions. I believe the extended experiments and the improved presentation further strengthen the paper.

I would additionally suggest the authors reconsider the color scheme of the plots. This spectrum of blue-purple colors is problematic with certain color blindness and grayscale printing. Please consult publishing practices for formatting figures.

Thanks for the thoughtful review and helpful comments. We have revised our paper with extensions to our theoretical and experimental results; changes are summarized in the joint comment to all reviewers. We respond to reviewer-specific comments and questions below.

$**Experiments**$ Our revised paper includes experiments in the radiance field setting on all 8 scenes from the NeRF dataset, rather than only the Lego scene (average results are in the revised main paper and at https://imgur.com/a/O4NQrGY , and per-scene results and renderings are in the revised appendix as well as at https://imgur.com/a/psnr-ssim-lpips-pareto-optimal-curves-of-compared-models-bsXUXIL, https://imgur.com/a/MQTua7H). Results are similar when we consider all scenes compared to our initial results on only the Lego scene, suggesting that performance is not overfit to the specific scene we originally tested. Thanks for the suggestion; including more complete experiments certainly strengthens our revised paper.

$**Design choices and model illustration**$ Our revised paper includes a new figure (figure 1 in the revision) to illustrate the nonconvex, semiconvex, and convex GA-Planes models we use in our experiments, including our use of multiresolution grids (also accessible here: https://imgur.com/a/FVSIhW4). We have also added a table in the appendix that details the specific model configurations we used in our experiments, since there are indeed interesting design choices in terms of allocating limited parameters among the different feature grids (appendix A.8). We found these specific models via a grid search on the Lego scene, from which we selected the pareto-optimal configurations with respect to PSNR and model size. We have clarified this procedure in the revised paper (i.e. amended the wording to clarify that the model family can be adapted to different settings, but this adaptation is not automatic).

$**Limitations**$ We have added a discussion of limitations to the revised paper; thanks for the suggestion. In particular, currently we have shown GA-Planes only for 3D (or smaller) representations, not for higher dimensions (e.g. dynamic volumes), and currently we demonstrate GA-Planes for reconstruction rather than also generation. We expect both of these extensions to be possible, but beyond the scope of the present paper. We also expect that our convex GA-Planes model may benefit from optimization with specialized convex solvers, though we do not explore this in the present paper.

$**Benefits of convexity**$ In our experiments we don’t experience major difficulties in nonconvex training, though we observe its sensitivity to initialization increases as model size decreases. We added a new figure (https://imgur.com/a/6ZiD1rF) that compares IOU curves of convex, semiconvex and nonconvex GA-Planes models with very small size, initialized with 3 different seeds. While (semi)convex models display robustness across initializations, the IOU curve of the nonconvex model suffers from large variations. In particular, for one seed the nonconvex model completely fails to fit the video. Importantly, although in our experiments we use the same first-order optimization algorithm for all models, our convex GA-Planes formulation is compatible with any convex solver (e.g. cvxpy), allowing it to inherit decades of research in efficient convex optimization algorithms (which would e.g. avoid the need to tune algorithm hyperparameters). We appreciate the reviewer for highlighting this important point, which we have also made more prominent in the revision.

We hope our responses above and our revised paper address your comments; please let us know if you have any further questions. Thanks so much for your time and input!

Thank you for your feedback! We indeed have changed the color scheme, and tested it with grayscale to ensure visual clarity. The updated figure for the average results for the Blender dataset can be seen here: https://imgur.com/a/sPmRWkH. The per-scene plots are updated in the revised paper.

Thanks for the thoughtful review and helpful comments. We have revised our paper with extensions to our theoretical and experimental results; changes are summarized in the joint comment to all reviewers. We respond to reviewer-specific comments and questions below.

$**2D equivalence to matrix completion (theoretical results)**$ Our theorems implicitly assume that the GA-Planes objective is also to minimize the Frobenius norm of the error. We appreciate the clarification question, and we have made this assumption explicit in the revised paper. This assumption is valid (albeit in 3D) for our experiments on 3D and video segmentation, for which we use direct supervision to minimize mean squared error (space carving supervision in 3D segmentation). However, our radiance field experiments use indirect pixel-level supervision so the setting there is not identical to the theorems.

$**Benefits of convexity**$ Although in our experiments we use the same first-order optimization algorithm for all models, our convex GA-Planes formulation is compatible with any convex solver (e.g. cvxpy), allowing it to inherit decades of research in efficient convex optimization algorithms (which would e.g. avoid the need to tune algorithm hyperparameters). We appreciate the reviewer for highlighting this important point, which we have also made more prominent in the revision. Thank you for bringing up the question about sensitivity to initialization. In our main experiments, we didn’t encounter such an issue as the model sizes were large enough for the nonconvex model to be able to generalize and be trained reliably. However, we trained semiconvex, nonconvex and convex GA-Planes models with a very small number of parameters for the video segmentation task, to show robustness of our (semi)convex models. We added a figure in the revised appendix which compares test frame IOU score curves of nonconvex, semiconvex and convex formulations (all using concatenated features) that are initialized with three different seeds (we keep the randomness in the gating weights constant). This preliminary experiment shows that the variation in the IOU scores of (semi)convex models is minimal, while the nonconvex model suffers (it completely fails to fit the video in one seed). Figure link: https://imgur.com/a/6ZiD1rF

$**Figure 1a dots**$ The curves show pareto-optimal model configurations; the disconnected dots are other (suboptimal) model configurations that we included for completeness. We have removed these suboptimal models from the figure in the revision, since they are not really necessary and may not be clear to other readers. Thanks for the feedback!

We hope our responses above and our revised paper address your comments; please let us know if you have any further questions. Thanks so much for your time and input!

Thanks for the thoughtful review and helpful comments. We have revised our paper with extensions to our theoretical and experimental results; changes are summarized in the joint comment to all reviewers. We respond to reviewer-specific comments and questions below.

$**Clarifications in the methods section**$

• We have introduced a new figure to clarify our method, including the use of multiresolution grids. (https://imgur.com/a/FVSIhW4)
• Regarding dimensions: the dimensions of the feature grids are introduced in the bullet points at the beginning of section 3.1. We have added explicit dimensions for the extracted feature vectors $**e**$ (which are derived from these feature grids, so their length matches the feature dimension in the grids).
• Thanks for pointing out places where the description of our method can be simplified/clarified. We have edited the text following your suggestions so that the method will be clear to future readers. In particular, we removed the unnecessary definition of $f_c$ in eq 5, and moved eq 5 to be before eq 3 so that the $**e**$ notation is clear (note that some of the equation numbers have therefore changed in the revised paper). Here, $c$ was indexing over the different sets of coordinates in each of the grid features, but these are enumerated explicitly in eq 6 so the shorthand is not really necessary and has been removed.
• We have clarified the definition of semiconvexity in the revised paper, as well as provided a reference which proves that all local optima are also globally optimal, meaning that first-order optimization methods will succeed. Basically, semiconvexity here refers to the Burer-Monteiro factorization of a convex objective. Such problems are biconvex (separately convex in each parameter group), and moreover, it can be shown that local optima are global [1].

[1] Arda Sahiner, Tolga Ergen, Batu Ozturkler, John M Pauly, Morteza Mardani, and Mert Pilanci. Scaling convex neural networks with burer-monteiro factorization. In ICLR, 2024.

$**How GA-Planes generalizes prior work**$ This is discussed in more detail in appendix A.1 (in the original submission as well as the revision). GA-Planes is a family of models that includes many previously proposed models as special cases; this is what we mean by generalization. However, the specific models in this family that we use in our experiments have not been used in prior work, because no prior methods (to our knowledge) have combined line, plane, and volume features, and an MLP decoder, in the same model. Further, we adopt feature combinations (in the multiplicative model) that approximate a volume element (trivector) based on geometric algebra operations. For instance, we do not use plane products (unlike K-Planes), as this is not a valid operation in geometric algebra. Compared to TensoRF, GA-Planes represents the 3D volume with line products (like TensoRF-CP) in addition to line-plane products (like TensoRF-VM) and a low-resolution volume feature, generalizing TensoRF factorizations.

$**Connection between GA-Planes and convex optimization**$ We designed the GA-Planes architectures specifically to be compatible with convex optimization. Any model that involves multiplication of features (e.g. K-Planes) cannot be convex, and not all models that can be trained by convex optimization will achieve good quality. For example, in our 3D and video segmentation experiments we find that the GA-Planes models perform comparably well regardless of whether we use the convex, semiconvex, or nonconvex versions, while the previously proposed Tri-Planes model (from EG3D) performs markedly worse under convex or semiconvex formulations. Although in our experiments we use the same first-order optimization algorithm for all models, our convex GA-Planes formulation is compatible with any convex solver (e.g. cvxpy), allowing it to inherit decades of research in efficient convex optimization algorithms. We appreciate the reviewer for highlighting this important point, which we have also made more prominent in the revision. The empirical advantage of having a (semi)convex formulation is improved robustness to initialization (especially in small models), and fast optimizability with specialized solvers. We provide preliminary robustness analysis comparing nonconvex, semiconvex and convex GA-Planes in the video segmentation setting in the revised appendix, and at https://imgur.com/a/6ZiD1rF for convenience.

We hope our responses above and our revised paper address your comments; please let us know if you have any further questions. Thanks so much for your time and input!