Q1) The details about the maximum size of the Gaussian

A) We applied different maximum size thresholds depending on the elevation angles. Looking at Eq. (7), in omnidirectional projection, the Gaussian located in the polar region is rasterized to a wider area of the image than the equatorial region. Reflecting these characteristics, we applied the split size limit of Gaussian differently depending on the upper and lower angles at which Gaussian is located. Reflecting these characteristics, we applied the size limit of Gaussian differently depending on the angle at which Gaussian is located. Specifically, the larger the absolute value of the angle, the larger the size threshold becomes, and it becomes the minimum in the equatorial region. When this size exceeds the threshold, Gaussian is split into two smaller Gaussians and optimized separately.
In Figure R3, we qualitatively compare the images created with and without using the above method. We understand that it could be difficult to fully grasp this concept based solely on the contents written in the paper. Therefore, we will provide a detailed explanation to make it easier to understand.

Q2) Performance comparison with 360-GS

A) We attempted to compare with 360-GS before submission, but were unable to do so. Since 360-GS is currently only available on arXiv and the official code implementation has not been made public, we endeavored to reproduce 360-GS by ourselves. Using the code we reproduced, however, Gaussian splats are hardly optimized, producing ludicrous results. After a thorough analysis of the issue, we noticed that there was an error in the equation calculating the Jacobian matrix $J$ in 360-GS. Specifically, the signs of the diagonal terms of the matrix should be -, in Eq. (7) of 360-GS paper [2]. This could be due to a differentiation error or a mistake in the description within the paper. While our reproduction might have been incorrect, we believe it would not be appropriate to include this comparison in the main table under this situation. If the official code for 360-GS becomes available, we could include the comparison in the main table.

Q3) Is it feasible to render using a perspective camera without fine-tuning?

A) ODGS delivers high-quality perspective camera rendering results without any additional fine-tuning.

Our method is fundamentally based on projecting onto the tangent plane of a sphere, which is conceptually compatible to the perspective rendering of each Gaussian. As a result, the Gaussian splats trained with ODGS produce high-quality perspective images when rasterized with a pinhole camera model without fine-tuning. Some sample images rendered using a perspective camera model are shown in Figure R4. 3DGS(P6) and 3DGS(P18) were optimized using perspective images and rendered with a perspective camera, while ODGS was optimized using omnidirectional images and rendered with a perspective camera. Despite the difference between the camera model used during optimization and the one used during inference, ODGS demonstrates significantly sharper and superior results.

Dear reviewer sZSB,

We are pleased to hear that our response has addressed your concerns well.

Your invaluable feedback has greatly helped us improve our draft. We promise to include the points you raised and enhance clarity in the updated manuscript.

Finally, we sincerely thank you for highly valuing the contributions of our work.

Best regards,

Authors.

Q) Are there any specific technical novelty we (readers and reviewers) should care about?

A) We hope that our work will not be dismissed as merely an engineering effort. The idea of applying a local affine approximation to a sphere may seem simple at first glance. However, aside from concurrent work, this approach is being attempted for the first time. We hope that the process of developing and validating this idea will not be dismissed as merely an engineering effort. We want to highlight the following unique contributions and technical innovations:

• New Omnidirectional Rasterizer: We introduce a novel rasterization technique specifically designed for omnidirectional images, which is not a straightforward extension of existing 3DGS methods and provides a more accurate and efficient representation of 3D scenes captured from 360-degree cameras.

• Advanced Densification and Splitting Policy: ODGS employs a sophisticated policy for managing the densification and splitting of Gaussians in the omnidirectional domain. Our work is not just an engineering effort, but also a significant technical consideration that enhances the quality and efficiency of the 3D reconstruction. This is a key aspect that we believe sets our work apart.

• Comprehensive Evaluation Across Diverse Datasets: Unlike many other approaches, the paper thoroughly tests various datasets, covering both egocentric and roaming scenes. This comprehensive evaluation showcases the robustness and versatility of ODGS.

Fortunately, all other reviewers have acknowledged the strengths of our work, and none have raised concerns about its novelty. Specifically:

• Reviewer cMFb noted that our work is "quite relevant to the research community, simple, and efficient."
• Reviewer iEY4 highlighted as "practically useful and valuable."
• Reviewer sZSB described it as "simple but effective, and inspiring."

Therefore, while we can understand the concerns raised by reviewer oCBJ to some extent, we hope that our work will be evaluated positively.

Q1) Inconsistencies between key sections and the contents

A) We strongly contend that the iEY4's review is based on a substantial misapprehension of our study, thus leading to erroneous conclusions. Our work is based on 3DGS [25] and is not related to 2DGS at all. Eq. (9) describes a density function of a 3D Gaussian projected on the image plane, modeled as a 2D Gaussian distribution in the image plane through local affine approximation; the actual 3D space consists of 3D Gaussians. (Please refer to the paragraph in 3DGS [25] paper, Section 4, Equation 5) This is completely different from the 2DGS work that originally composes a 3D space as a set of disk-shaped 2D Gaussian primitives.

Q2) Comparison with other methods

A) The contributions of the listed papers are outside the scope of the problem we are trying to tackle. First, Sdfstudio, Neuralangelo, NeuS, and NeusFacto aim at neural 'surface' reconstruction, which is different from the problem we are addressing. The surface normal maps generated by these models do not contain texture information and, therefore, cannot be directly compared with high-quality rendered images obtained from ODGS. In addition, the datasets used in these works for training and inference do not contain omnidirectional images.

Meanwhile, we excluded classic 3D reconstructions using Structure-from-Motion (SfM) and Multi-view Stereo (MVS) from the main table because they produce significantly inferior results compared to the recent reconstruction methods. Here, we present these results in Table R2 for better understanding. The SfM results are obtained using the OpenMVG library, then the MVS results are obtained from the OpenMVS library, respectively. (We used OpenMVG instead of COLMAP because COLMAP currently does not support camera pose estimation from omnidirectional images.) Both methods fail to create a complete point cloud, resulting in blank areas in the projected image, as observed in the samples shown in Figure R2. The valid area ratio in the table indicates the proportion of pixels among images that receive 3D information and have valid pixel values. The results of SfM shows a valid area ratio under 5%, which implies that most of the regions remain empty. Although the high accuracy of the estimated points shows high PSNRs among the valid area, the number of matched points is small, and the PSNRs of the whole region are measured to be less than 5dB. Applying MVS to the results of SfM increases the number of generated points, but still, at least 40% of pixels are not filled after projection. In addition, by comparing the PSNR measured in the valid area, we can see that the accuracy of the points created by applying MVS has decreased. ODGS, in contrast, shows consistently much higher PSNR than those methods while maintaining a 100% valid area ratio for all datasets.

Q1) How to implement alpha blending?

A) We conducted our work based on the 3DGS framework, utilizing the same tile-based rasterization and alpha-blending pipeline. Our work suggests how to render with an omnidirectional (omni in short) camera model instead of a perspective (persp in short) camera model, and the remaining methods are identical to those of 3DGS. For clarity, the rasterization process is as follows:

1. Projecting 3D Gaussians to 2D Gaussians in the planar pixel space (as described in Section 3.2).
2. Tile-based culling against the spherical shell and sorting by depth.
3. For a given pixel, accumulating color and alpha values by traversing the list front-to-back until we reach a target saturation of alpha in the pixel. (alpha-blending)

$$C = \sum_{j\in*N*} {c_j \alpha_j T_j}, \quad {T_j = \prod_{k=1}^{j-1} (1-\alpha_k)},$$

$C$ is the pixel color, $c_j$s are the colors of splats, and $\alpha$ is computed by multiplying the learned opacity of Gaussian with the power of rasterized Gaussian in the pixel (calculated using Eq. (9)).

We omitted the details of the tile-based culling and alpha-blending process because those are not the contributions of our work. However, we acknowledge the concern that the omission may make the paper appear incomplete to some readers and reviewers. We will include this information in the final version of the paper to enhance its completeness and clarity.

Q2.1) What happens if we increase the number of persp cameras

A) While using more views can yield slightly better results than the 6-view setup, it still falls short of the performance achieved by ODGS. Cubemap projection is a well-known method for converting an omni image to six persp images, where each corresponds to the face of a cube, and it is widely used in many studies involving 360 cameras [A,B,C]. However, as suggested by cMFb, we also believe it would be an interesting experiment to observe the trends when using more persp cameras for optimization. While the standard approach uses six perspective cameras, we optimized the model by adding 12 perspective cameras (18 in total) facing the edges of the cube.
Table R1 shows the performance of optimized results according to the number of persp viewpoints. When using the 18-views, the performance is comparable to the 6-views at the 10-minute mark but surpasses the 6-view results at the 100-minute mark. First, the increased number of images for training keeps the model from sufficiently learning from all views in the early stages (10 minutes), resulting in slightly lower performance. However, after sufficient optimization time (100 minutes), the additional views allow for further optimization, leading to improved results. Still, ODGS shows the highest performance in most metrics, even considering 3DGS using 18-views.

Q2.2) Is the rasterizer properly handling the different persp coordinate frames?

A) In the case of the 6-view cubemap, artifacts can occur at the edges of the cube due to the rasterizing method used by 3DGS. Since 3DGS omits the 3D Gaussians at image boundaries for stable optimization, discontinuities may occur at the edges of the cube, as illustrated in Figure R1.(c). (The orange dashed lines in Figure R1.(b) represent the edges of the cubemap after being combined into one omni image.) The image optimized with 18-view persp images has fewer artifacts, as shown in Figure R1.(d). However, artifacts still appear even at points other than the cube's edges, resulting in less favorable outcomes than ODGS.

Q2.3) How ODGS runs more than three times faster than 3DGS applied on persp images?

A) A majority of time is spent stitching the six persp images into the omni image. Since the total resolution of an omni image (2000 x 1000) is similar to six persp images (512 x 512 x 6), we empirically measured that the time for generating six perspective images is similar to generating one omni image. However, producing the final omni image from the six persp images takes a large proportion of of inference time (about 80%) since every persp image should be non-linearly warped to compose the final omni image.

Q3) The details of the densification heuristics

A) We acknowledge that the concept might be challenging to fully understand from the paper alone, so we will include a detailed explanation for clarity. As represented by $\mathbf{C}_o$ in Eq. (7), Gaussians of the same size in 3D space are rasterized to different sizes in an omnidirectional image depending on the angle at which they are located. Consequently, from a pre-defined size threshold for densification, we dynamically change the threshold according to the vertical location of the Gaussian. As depicted in Figure R3, this approach effectively reduces artifacts when representing lanes of the road.

Q4) Missing a related work: OmniGS

A) We now recognize OmniGS as a concurrent work. We thoroughly reviewed the suggested paper, OmniGS, and noted that they employ a similar approach. Unfortunately, their work is only available on arXiv; no code or implementation details have been provided. We would have liked to compare their method with ours, but we will have to settle for mentioning it as a concurrent work in our references.

Reference

[A] H. Jang et al., "Egocentric scene reconstruction from an omnidirectional video," SIGGRAPH 2022.
[B] M. Gond et al., "LFSphereNet: Real-time spherical light field reconstruction from a single omnidirectional image," SIGGRAPH 2023.
[C] F. Wang et al., "Bifuse: Monocular 360 depth estimation via bi-projection fusion," CVPR 2020.