LinPrim: Linear Primitives for Differentiable Volumetric Rendering

We thank the reviewer for the insightful comments. We address the questions and concerns below:

Population Control. Our aim was to keep as many factors apart from the primitives themselves fixed when comparing to 3DGS to make the evaluation as fair as possible, and focus on the underlying primitive type. As such, our population control paradigms closely mirror the behavior of 3DGS, with changes necessitated by the change of primitive.

To analyze the impact of the population control on performance, we implement a completely separate population control technique from the 3D-MCMC paper $NeurIPS 2024$ , which rethinks population control as drawing samples from the distribution of the scene. This not only improves performance but also allows precise control over the final population size. We once again adopt their approach as close as possible to minimize undesired performance impacts. In this case, we essentially act as if our primitives are Gaussian-like with standard deviations equal to the linear distances divided by 2.6. This value was chosen as it results in a close match in appearance between the original shape and its Gaussian substitute.

As can be seen in Table 11, this approach is able to consistently improve performance when evaluating on the first three ScanNet++ scenes. We further believe that this illustrates how approaches made as improvements to 3DGS can be similarly applied to LinPrim. We will add these results to the revised manuscript.

Table 11: Ablation of the performance of MCMC-based densification on linear primitives with a fixed number of 250k and 1M primitives in the scene on the first three ScanNet++ scenes.

Comparison between Octahedra and Tetrahedra. We will include the results of our experiments on comparing octahedra and tetrahedra-based approaches in the final manuscript. For purposes of readability, we include only the results on three ScanNet++ and Mip-NeRF 360 scenes in Table 12.

Based on the quantitative and qualitative results, we find that octahedra and tetrahedra generally show similar visual characteristics in equally populated areas of a scene. Differences arise when considering the population control paradigms. They are required to enhance the initial coverage of the scene and enable higher fidelity in well-observed regions. Using the symmetry of the octahedra primitives allows us to mirror established approaches for Gaussian primitives to place new primitives which is not directly possible for tetrahedra.

Nonetheless, we believe that tetrahedra promise a large improvement in runtime efficiency (see Table 8) when compared to octahedra due to their reduced face count, with the current drawbacks largely concerning them being less related to established approaches. As such, applying novel or well-suited population control approaches to tetrahedra offers a promising avenue for future work.

Table 12: Comparison of the performance of LinPrim’s octahedron and tetrahedron-based approaches on three ScanNet++ and three Mip-NeRF 360 scenes.

Used Baselines. We thank the reviewer for his feedback and will update our experiments section to make it clearer and more readable. Keeping space limitations in mind, the baselines were chosen following the availability of results and relevance to our work. We will ensure that a more complete overview of results and introduction of the datasets will be available in the final manuscript.

We thank the reviewer for their positive feedback and insightful questions. We address the reviewer’s questions below:

Comparison to 3DCS. We will add a full quantitative and qualitative comparison of 3D Convex Splatting to our evaluation and discuss the approach. For the sake of brevity, we include only a partial comparison of the approaches on the first three ScanNet++ scenes in Table 7. Keeping in mind that 3DCS was not tuned on this dataset, we are able to achieve comparable performance. Including densification paradigms via Markov Chain Monte Carlo (as further described in our comment to reviewer Z51C), we are able to slightly outperform their results. In the revised paper, we will extend this comparison to all ScanNet++ and Mip-NeRF 360 scenes and include additional visualizations.

Table 7: Quantitative evaluation on three ScanNet++ scenes.

Performance. To provide deeper insights into our method’s runtime characteristics, we profiled the evaluation of our method and 3DGS. We evaluate the methods as-is, as well as using the MCMC population control to ensure similar populations for both approaches, as can be seen in Table 8.

Our results show that the large majority of the cost is incurred in the rendering stage, where per-tile lists are traversed to aggregate color contributions. In this stage, 3DGS only requires evaluating the function value of a Gaussian, while LinPrim calculates intersections between the pixel ray and the primitives faces. As a consequence, even though LinPrim traverses fewer primitives per pixel than 3DGS (as shown in Table 6), the current implementation is simply less optimized. Since ray-triangle intersections are commonly required, we believe leveraging existing optimized implementations for this use presents a significant opportunity for runtime improvements and will discuss the findings in the revised manuscript.

Storage Requirements. As shown in Table 8, our representation not only uses fewer primitives in general it also employs smaller bounding boxes and consequently holds far fewer primitives in memory at any given time. Concerning total storage requirements, the actual disk size of an optimized scene is almost solely reliant on the number of primitives. Looking at the file sizes of the optimized scenes, we find that given unrestricted optimizations, the LinPrim scene is around 1/3 of the 3DGS size.

Table 8: Analysis of the runtime of individual steps in the rendering pipeline and corresponding primitive counts and file sizes on the ScanNet++ scene 39f36da05b.

Depth Maps and Mesh Reconstruction. We evaluate the accuracy of depth maps predicted by our method and by 3DGS on all images from the first three ScanNet++ scenes. For each pixel, the predicted depth is defined as the depth of the first primitive along the ray where cumulative opacity exceeds 0.5. The resulting depth maps look convincing for both approaches.

Ground‑truth depths are extracted from the ScanNet++ meshes via the official toolkit. We compute error metrics only at pixels with valid ground‑truth and prediction values - pixels missing either are excluded. Both methods achieve similarly high coverage, with 3DGS covering 99.97 % of pixels and LinPrim covering 99.79 %. Results can be seen in Table 9.

Table 9: Pixel-wise depth map errors on three ScanNet++ scenes.

To assess the 3D consistency and extract surfaces from the predicted depth maps, we apply TSDF-Fusion with a voxel size of 5cm and apply Marching Cubes to reconstruct meshes from the images. We then compute the Chamfer Distance to the ground-truth meshes from ScanNet++ by uniformly sampling 5000 surface points, as can be seen in Table 10.

Table 10: Chamfer distances between mesh meshes reconstructed on three ScanNet++ scenes by fusing predicted depth maps.

The results indicate that LinPrim produces slightly more accurate depth estimates than 3DGS, suggesting better geometric reconstruction. The Chamfer Distance evaluation shows that LinPrim achieves better 3D reconstruction in two out of three scenes. This indicates that while depth predictions are generally of higher quality, exact results can vary based on scene characteristics. We will add qualitative and quantitative results and discuss them in our revised manuscript.

We thank the reviewer for the precise and perceptive analysis and comments. We address the comments below:

How and why is EWA splatting applied? We employ EWA splatting by using the ray-space approximation to cut down per-primitive work during rasterization. Although our primitives remain full 3D objects, projecting them into ray space lets us treat each triangle as an effectively 2D shape for intersection and gradient computations. Concretely, this simplifies intersection calculations as the ray directions are always orthogonal to the image plane and have only one nonzero component. This effectively means that testing a ray against a triangle reduces to a 2D problem. Additionally, this applies similarly to the gradient calculations, where position gradients collapse to two dimensions until transformed out of ray space.

We agree that an ablation study comparing the rendering speed and quality, each with and without the ray-space approximation, would help clarify the trade-offs and will include the corresponding quantitative results and discussion in the final manuscript.

Optimizing Densities. Despite the representations being largely equivalent in our use case, optimizing a density directly is certainly plausible and might help performance by avoiding unnecessary transformations. We will run an ablation on this to see if there are positive effects in practice.

For the submission, we used opacities for two main reasons:

1. They are bounded, and we thus assume them to be more numerically stable during optimization.
2. Compared to densities, opacities are more easily interpretable, making it easier to understand distributions of primitives or contributions to final appearance.

Sensitivity to Initialization. To evaluate the sensitivity of our approach and 3DGS to initialization, we apply changes to the initialization scheme and evaluate the relative performance impact. We perform this ablation on the first ScanNet++ scene (39f36da05b), on which both approaches performed similarly well.

These are the specifics of the changes made:

1. Noisy Points: We add noise to the position of the SfM points used for initialization. For this, we sample random offsets in each dimension with a standard deviation of 5% of the maximum distance between cameras, which is around 14cm for the scene.
2. Half #Points: We randomly drop half of the SfM points used for initialization.
3. Half-Size Primitive Initialization: The initial size of the primitives in either approach depends on the distance between neighbouring SfM points. For this test, we reduce the initial size of primitives by half, leading to sparse initial coverage of the 3D space.

As can be seen in Table 5, all of the above methods reduce performance across all metrics and for either approach. Noticeably, reducing the coverage through smaller initial primitives hurts 3DGS's performance more than ours. This suggests that even though LinPrim boundaries are distinct and intersections are fewer, relevant visual cues can be transferred onto the primitives. In general, while the scope of the experiment is limited, we believe it showcases that our approach is not more dependent on a strong initialization and will include the results in the revised manuscript.

Table 5: Ablation on the impact of worse initialization schemes on the final reconstruction quality on ScanNet++ scene 39f36da05b.

Memory Usage and Bounding Boxes. We evaluate the memory demands of the methods by analyzing how many primitives are kept in memory during each part of the rendering process. For this, we run the evaluation on the first ScanNet++ scene (39f36da05b) and present the mean values on the tested views in Table 6. We also include the values when limiting either method to a similar number of total primitives by leveraging the corresponding MCMC-based approach as introduced in our reply to reviewer Z51C.

As can be seen, the number of primitives in the view frustum is generally similar for both primitive types when the scene's population is similar. However, looking at the total length of the per-tile sorted lists clearly shows that the bounding boxes of LinPrim are generally much smaller, as they intersect fewer tiles.

This trend is further exemplified by the total number of primitives that are iterated over. Due to the distinct boundaries of linear primitives, we can further differentiate between primitives that were only iterated over and those that were intersected and contributed to the output image. In total, linear primitives create an image with only around 5-10% of the intersections of Gaussians.

As Table 5 shows that this does not translate to issues with gradient flow, it suggests that the large majority of intersections in 3DGS only contribute minor effects to the output image and similarly do not enable significant gradient flow. We believe these values present a significant opportunity for future work to optimize the rendering process and leverage the concise nature of our representation. And will include the findings and discuss opportunities in our revised manuscript.

Table 6: Analysis of the number of primitives in memory and in the tile-list, as well as the count of total primitives iterated over and intersected by pixel rays. Output images are of size 1752x1168 with patch size 16x16, and values are evaluated on ScanNet++ scene 39f36da05b.

Benefits of Linear Primitives over 3DGS. We kindly refer to our reply to reviewer EARy about geometry and surface reconstruction and reviewer Pgum about semantic-wise visual quality. In summary, we show that our primitives provide improvements in geometric reconstruction and are better able to reconstruct view-dependent appearance when compared to the Gaussian baseline.

We thank the reviewer for the thorough and insightful review. We address the individual points and questions below:

Complexity Trade-off. To estimate the break-even point between the rendering speed of our octahedron approach and 3DGS, we optimize several variations of the same scene with different primitive counts and track the average rendering speed in Table 1. Please refer to our reply to reviewer Z51C for more details on how primitive counts were limited.

While subsampling primitives for evaluating rendering speed would allow us to more easily test rendering speeds at different primitive counts, it would not accurately reflect an optimized scene since the optimization will compensate for smaller populations with larger primitives, which in turn increase rendering times. Additionally, rendering speed is largely dependent on the number of visible primitives, not the population, as clearly demonstrated by 3DGS-MCMC taking longer to render while having only around a third the population size of 3DGS.

While an exact measure for relative rendering efficiency is difficult, when optimizing with the same MCMC heuristics, the ratio seems to be rendering 5 Gaussians for every octahedron or rendering 1.6 faces for each Gaussian. We kindly refer to Table 6 and Table 8 and the accompanying comments for additional insights on rendering efficiency. We will add this discussion and evaluation to the final manuscript.

Table 1: Comparison of the rendering speed of ScanNet++ scene 39f36da05b at different primitive populations. All experiments were run on an RTX 3090. *We subsample the initial SfM points to 1/3 for the 50k and 1/2 for the 100k version since we cannot remove primitives when using MCMC densification.

Surface Extraction. We kindly refer to our comment to reviewer EARy, where we analyze the quality of depth maps and surface reconstructions. In summary, we find that our method exhibits slightly stronger geometric reconstruction than the 3DGS baseline.

We agree with the reviewer that our triangle-based primitives are a more natural fit for direct iso-surface extraction, and exploring specialized mesh-extraction algorithms is a promising direction for future work. As shown by prior works (e.g., 2D Gaussian Splatting $SIGGRAPH 2024$ ), there is often a trade-off between geometric accuracy and visual fidelity in reconstructed scenes. We argue that LinPrim’s explicit triangular representation aligns more closely with the goal of producing high-quality meshes, and that incorporating additional optimization constraints - such as opacity requirements or encouraging face-normal alignment - could further improve surface quality, possibly without or with only little sacrifice in terms of appearance. Investigating these surface-aware regularizers may unlock both sharper geometry and maintained rendering performance.

SH Bandwidth. We evaluate the impact of the spherical harmonics degree on performance for LinPrim and 3DGS by optimizing scenes for either approach with reduced maximal degrees and comparing test-view performance. We show results in Table 2 and omit SSIM and LPIPS results for brevity reasons, and since they are less impacted by the changes to spherical harmonics than PSNR.

Noticeably, higher SH-degrees don’t necessarily improve the quality of reconstructions for unobserved regions, and can instead hurt performance by obfuscating underlying geometry and overfitting on known views. We can observe this by comparing performance on ScanNet++, where test views are more out-of-distribution, to the results on Mip-NeRF 360.

In our ablation, we find that 3DGS generally benefits more from spherical harmonics than LinPrim. As seen in Table 3, LinPrim nonetheless reaches higher visual quality in more view-dependent areas. In conjunction, this can likely be attributed to a better understanding of the underlying surfaces and geometry instead of an affinity to spherical harmonics. We will add a more expansive analysis and discussion in the revised manuscript.

Table 2: Ablation on the impact of reduced SH-degrees on the PSNR on three ScanNet++ and two Mip-NeRF 360 scenes, and compared to the degree 3 baselines.

Dynamic Scenes. We expect our rasterizer to be applicable to object motion and dynamic scenes, following approaches introduced to Gaussians. As an example, 4D Gaussian Splatting $CVPR 2024$ leverages a Spatial-Temporal Encoder to derive primitive deformations based on the timestep and could similarly be used to extend our work. In the context of applying motion to an existing scene, the explicit nature of our representation would allow us to select and move primitives that make up an object. In theory, the bounded nature of our primitives could aid motion by more distinctly separating individual appearances, and it would be interesting to see the performance in practice. We will discuss it as future work in our revised manuscript.

Edge-case Failures. To provide a deeper understanding of which regions and types of objects are well-suited to our approach and which ones are reconstructed better by 3DGS, we created semantic images for ScanNet++ renderings and calculated the per-class PSNR for both approaches. We again consider the first three ScanNet++ scenes, but since coverage is not required as in the depth case, we just use test images. We present the top 3 categories, some select results, and specific failure cases we identified in Table 3.

Table 3: Comparison of visual errors on specific classes when evaluating on three ScanNet++ scenes.

In absolute terms, the failure cases of our approach were the classes of “Storage Cabinet” and “Computer Tower,” in which LinPrim achieved the lowest relative performance compared to 3DGS. On the contrary, the classes that exhibit the strongest view-dependent appearance in the scenes - windows (+0.41), doors (+1.59), and monitors (+2.77) - are better reconstructed by LinPrim. There further seem to be types of geometry and appearance LinPrim is better able to capture, e.g., whiteboards (+1.95), blinds (+0.64), and heaters (+0.96).

Values in brackets describe the relative performance of LinPrim compared to 3DGS as measured by PSNR.

The results suggest that linear and Gaussian primitives are better able to reconstruct different types of appearance and geometry. Following this, we believe the hybrid primitives proposed by the reviewer could indeed alleviate edge-case failures of either primitive type. Exploring how to bridge the gap between the types of primitive and manage their representations and populations is an interesting avenue for future work. We will add a discussion in the revised paper.

Anti-Aliasing. To evaluate the zoom-out capabilities of the proposed anti-aliasing approach, we evaluate the visual quality of single-scale training at multiple smaller scales following Mip-Splatting in Table 4. While the improvements over the baseline are minor, they are consistent and generalize to zoom-out scenarios. We will add this ablation to our existing discussion on the anti-aliasing approach.

Table 4: Evaluation of zoom-out performance on NeRF synthetic scenes. Results are PSNR values on test set images. Approaches are optimized at full resolution and evaluated on smaller resolutions simulating zoom-outs.