R$^2$-Gaussian: Rectifying Radiative Gaussian Splatting for Tomographic Reconstruction

We thank the reviewer for the detailed review. The comments and suggestions are helpful in improving our paper.

Q3.1: Novelty comparison w.r.t. X-Gaussian.

Our method demonstrates considerable novelty compared to the concurrent work X-Gaussian for the following reasons:

• Broader task scope: Our R$^2$-Gaussian is designed for both X-ray view synthesis and direct 3D CT reconstruction, whereas X-Gaussian only supports 2D X-ray novel view synthesis.

• Theory-supported model design: Our R$^2$-Gaussian has successfully extended 3DGS to 3D CT reconstruction with a theoretically sound approach, including new Gaussian kernels, new splatting equations, and a voxelization strategy. All of these are grounded with careful theoretical derivations. In contrast, X-Gaussian empirically modifies and extends 3DGS for X-ray view synthesis applications, with limited novel theoretical contribution.

• Theory contribution: Our method provides novel and original theoretical results, including the new derivation of X-ray rasterization and the identification (and the remedy) of the previously overlooked integration bias in the standard 3DGS technique. X-Gaussian, on the other hand, does not offer theoretical contributions in this regard, despite that it indeed represents the first successful yet empirical adaptation and application of 3DGS to X-ray view synthesis.

• Efficient CT reconstruction: Our method can directly output 3D CT volumes. In contrast, X-Gaussian augments novel-view projections first, and then relies on other existing CT algorithms for CT reconstruction.

In summary, our method offers notable technical and theoretical contributions compared to X-Gaussian. The methodological comparison between the two methods has been discussed in L89-94. We will further highlight our novelty and contributions w.r.t. X-Gaussian in the revised manuscript.

Q3.2: X-Gaussian achieved a PSNR of 43, whereas this paper achieved a PSNR of 36.

Actually, the reported 43 (in Tab. 1 of the X-Gaussian paper) is the 2D PSNR of novel-view image rendering quality, rather than the 3D PSNR of the CT volume reconstruction quality. As a mater of fact, X-Gaussian only reported their 3D PSNR as 30.56 dB (from 5+95 view, as shown in Tab. 2 in the X-Gaussian paper), which is significantly lower than our 3D PNSR at 36 (or 36.89 dB, for human organ, in Tab. 3 of our paper).

Q3.3: It would be helpful to augment the related works and baseline models, such as C$^2$-RV (CVPR'24).

Thank you for this suggestion. We will include these SOTA related works in our revised manuscript. These will add value and authority to our current paper.

Regarding baseline selection, we did not include supervised learning methods like C$^2$-RV because they require external datasets for pre-training. Our primary focus is on evaluating the method's representation capability for arbitrary objects without pre-training. For a fair comparison, we choose self-supervised learning methods that require only X-ray projections of objects (L483-484). To our knowledge, SAX-NeRF [7] (CVPR'24) is the latest SOTA work, and we have included it in our baseline methods. Experiments show that our method also outperforms SAX-NeRF, with a 0.93 PSNR increase and 78$\times$ faster training speed.

Please also note that C$^2$-RV has not released their code and models (empty GitHub repository). We tried to contact the authors but received no response. Due to the limited time available for rebuttal, it is unfortunate that we could not reproduce their method and compare it with our method experimentally.

Q3.4: Could you elaborate on the performance gap between SAX-NeRF papers and your paper?

We use the official code of SAX-NeRF [7] and NAF [62] to perform experiments without changing the network or hyperparameters. We show human organ results from the SAX-NeRF paper [7] and our paper, which use the same source (Tab. C). SAX-NeRF performs consistently in both papers, while NAF in our paper has a higher PSNR. Nevertheless, both the SAX-NeRF paper and our paper conclude that SAX-NeRF achieves better results than NAF. Therefore, the slight performance inconsistency does not harm the arguments made in our paper.

Table C. PSNR values in SAX-NeRF paper [7] and our paper.

I appreciate the authors for their detailed rebuttal. It resolved most of the critical concerns. Therefore, as long as the authors incorporate the explanations from the rebuttal into the final version, I will raise my score accordingly.

We thank the reviewer for the detailed review.

Q4.1: The performance in real-world clinical or industrial scenarios is not thoroughly examined. For instance, real X-ray images are given to reconstruct the CT scan.

We further evaluate our method on real-world data. We use FIPS [b], a public dataset providing real 2D X-ray projections. FIPS includes three objects (pine, seashell, and walnut). Each case has 721 projections in the range of $0^{\circ}\sim 360^{\circ}$ captured by Hamamatsu Photonics C7942CA-22. Since ground truth volumes are unavailable, we use FDK to create pseudo-ground truth CT volumes with all views and then subsample 75/50/25 views for sparse-view experiments. We report the quantitative and qualitative results in Tab. 1 and Fig. 1 in the attached PDF file. Our method outperforms baseline methods by a large margin in 75- and 50-view scenarios. In 25-view, our method slightly underperforms IntraTomo but is 11$\times$ faster. Overall, our method shows superior performance and efficiency in the presence of real-world noise and scattering effects. We will include these results in the revised manuscript.

Q4.2: I am not sure if 75, 50, and 25 views of X-rays are considered sparse-view reconstruction and if it is practical to have 75, 50, or 25 views of X-rays for CT reconstruction.

• Practicality: As mentioned by Reviewer sKx6, in industrial and medical applications, CT machines typically take hundreds to thousands of X-ray projections for high-quality details [a]. Therefore, modifying existing CT machines to 75/50/25 views is practical and convenient by setting different scanning intervals.
• Rationale: The community considers fewer than 100 projections as sparse-view CT (SVCT). We list the numbers of projections used in some published papers, as shown in Tab. D. These papers use 20-180 views. Accordingly, we set our projections to 75, 50, and 25 views. Additionally, there are works investigating extremely sparse-view CT (ESVCT), which uses 2-10 views [58,30,10]. ESVCT is a severely ill-posed problem that cannot be solved without fine-grained prior knowledge, such as pretraining with external datasets. Therefore, ESVCT is out of the scope of our study, as we only use projections (completely self-supervised). Overall, our experimental setting aligns with previous work and should be considered sparse-view.

Table D. Number of projections used in published papers.

Q4.3: It cannot model the scattering effect in X-ray.

We follow most CT reconstruction work [13,2,50,61,62,7], assuming that the target radiodensity field to be isotropic, and treating scattering as a noise source on the 2D detector.

Although we do not explicitly model scattering effects, we take it into consideration in the experiments. When generating X-ray projections for synthetic datasets, we model scattering noises with Poisson. We also evaluate our method in the real-world data which contains scattering effects (Q4.1). All results demonstrate our method's superior performance and robustness to scattering noise.

Q4.4: It is unclear how to get X-3DGS slices and what "queried from three views" means in detail.

We will improve Fig. 7 captions and relevant descriptions for better understanding.

• X-3DGS slice: After recovering the 3D density of each Gaussian (L251-252), we use the same voxelizer (Sec. 4.2.2) as R$^2$-Gaussian to extract CT volumes. We then show slices of these volumes in Fig. 7 to demonstrate the reconstruction quality.
• "Queried from three views" means that we show reconstruction results from three different views to demonstrate the view inconsistency in X-3DGS.

Q4.5: Why not implement the voxelization on X-3DGS (or vanilla 3DGS) for a fair comparison?

For X-3DGS, we implement the same voxelizer (Sec. 4.2.2) in R$^2$-Gaussian to extract a CT volume. We will clarify it in the revised manuscript.

Q4.6: After the voxelization from Gaussians, can the final density volume be compatible with traditional CT scans and be viewed in CT software?

Yes the reconstructed volume is compatible with traditional CT scan viewers. We show a screenshot of inspecting reconstructed volumes with the Weasis DICOM medical viewer in Fig. 4 (PDF).

Q4.7: Ethics review.

All data used in our experiments are from open-source datasets. We have properly cited the relevant references (Appx. B) and adhered to the respective data licenses.

Reference

[a] Villarraga-Gómez, Herminso, and Stuart T. Smith. "Effect of the number of projections on dimensional measurements with X-ray computed tomography." Precision Engineering 66 (2020): 445-456.

[b] Zhang, Zhicheng, et al. "A sparse-view CT reconstruction method based on combination of DenseNet and deconvolution." IEEE transactions on medical imaging 37.6 (2018): 1407-1417.

[c] Rückert, Darius, et al. "Neat: Neural adaptive tomography." ACM Transactions on Graphics (TOG) 41.4 (2022): 1-13.

We appreciate your positive review and valuable feedback.

Q2.1: Did the authors preprocess the CT volumes?

Yes we convert raw volumes from HU to attenuation coefficients. Following [62, 7, 27], we then normalize voxel values to [0,1] for balanced evaluation across modalities. We will add more details in the revised manuscript.

Q2.2: How does anisotropic residual noise impact the results?

Q2.3: The paper would benefit from a real-world evaluation.

We address two questions together since they all relate to scattering effects. We agree that anisotropic effects, such as Compton scattering, occur in real-world X-ray imaging. We follow most CT reconstruction work [13,2,50,61,62,7], assuming that the target radiodensity field to be isotropic, and treating scattering as a noise source on the detector.

We do consider scattering in the experiments.

• When preparing X-ray projections, we follow conventions [a] to model scattering noise with Poisson.
• As requested, we further evaluate our method on real-world data containing scattering effects. We use FIPS [b], a public dataset providing real 2D X-ray projections. FIPS includes three objects (pine, seashell, and walnut). Each case has 721 projections in the range of $0^{\circ}\sim 360^{\circ}$. Since ground truth volumes are not available, we use FDK to create pseudo-ground truth with all views and then subsample 75/50/25 views for sparse-view experiments. We report the quantitative and qualitative results in Tab. 1 (PDF) and Fig. 1 (PDF). Our method outperforms baseline methods in 75- and 50-view scenarios. In 25-view, our method slightly underperforms IntraTomo but is 11$\times$ faster. Overall, our method shows superior performance and efficiency in the presence of real-world noise and scattering effects.

Q2.4: The authors should better reference existing X-ray 3DGS works.

We will enhance the comparison with existing X-ray 3DGS methods by adding more details in the related work section. Please note that all X-ray 3DGS works were preprinted/under review before the NeurIPS submission deadline.

We would also like to clarify the following points regarding the comments:

• Radiative Gaussian: Although X-Gaussian and our method coincidentally use the same term, the motivations and formulations are quite different.

  • X-Gaussian replaces view-dependent spherical harmonics with a view-independent feature vector based on the isotropic assumption. It retains color and opacity, which do not physically represent radiodensity field. Besides, it uses alpha-blending, which contradicts the unordered nature of X-ray imaging.
  • We define the Gaussian kernel as a local radiodensity field and derive new rendering equations (Eq. 7), demonstrating that summation should be used instead of alpha-blending (L176-178). See Q3.1 (Reviewer P8A3) for more details.

• Total variation (TV): Our work does not claim contributions to TV regularization. Instead, we use TV to demonstrate the possibility of applying voxel-based supervision to Gaussians, thanks to the differentiable voxelizer. To our knowledge, we are the first to do so.

Q2.5: References w.r.t. TV are missing.

We will add reference [c] w.r.t. TV.

Q2.6: I do not see novelty in the radiodensity voxelization.

We conclude the technical novelty of our voxelizer as the first differentiable CUDA-accelerated one for 3D Gaussians. The voxelizer offers opportunities to apply other voxel-based losses (such as SDF supervision) to 3D Gaussians, which can benefit the community.

Q2.7: Changes to the adaptive control are unclear/minor.

We made minor modifications to adaptive control to suit X-ray imaging. We do not intend to claim a contribution regarding adaptive control. We will clarify it in the revised manuscript.

Q2.8: Some qualitative results could help grasp the contribution of TV regularization.

We show qualitative results with and without TV in Fig. 12. It is clear that adding TV promotes smoothness and homogeneity. However, needle-like artifacts still persist, consistent with the general acknowledgment that high-level priors such as TV do not significantly improve performance. As mentioned in Q2.6, we do not claim contributions to TV but rather to a novel strategy for applying 3D losses.

Q2.9: Larger CT datasets are available.

While there are larger CT datasets, such as CTPelvic1K, they only cover human organs with similar structures and materials. We focus on method's representation capability of arbitrary objects. Therefore, we chose data across various modalities, aiming at diversity rather than quantity. Besides, NeRF-based baseline methods typically require hours for training (SAX-NeRF needs 13 hours). We unfortunately do not have sufficient resources to support thousand-level experiments.

Compared with previous work, our dataset has the same size as SAX-NeRF (15), and is larger than NAF (5) and X-Gaussian (5).

Q2.10 Additional results could better contextualize the claims of integration bias correction.

We show more quantitative and qualitative results regarding integration bias in Tab. 2 (PDF) and Fig. 2 (PDF). Our method achieves better results than X-3DGS in both 2D and 3D. This suggests that correcting integration bias improves both image rendering and volume reconstruction in CT. Note that this conclusion slightly differs from L253-254, and we will update it in the revised manuscript.

Q2.11: Typo: "Compton scatter" or "ponton scatter"?

We use Poisson to model photon statistics on the detector, which also includes Compton scattering. We will use "photon statistics" or "Compton scattering" for clarity.

Reference

[a] Zhu, Lei, et al. "Noise suppression in scatter correction for cone‐beam CT." Medical physics (2009).

[b] Siltanen, Samuli, et al., "FIPS: Open X-ray Tomographic Datasets.", Zenodo (2022)

[c] Rudin, Leonid I., et al. "Nonlinear total variation based noise removal algorithms." Physica D: nonlinear phenomena (1992).

I thank the authors for their thorough response, as well as my fellow reviewers for their insightful comments. I appreciate the author's effort to address my (overall minor) concerns and questions, and I lean towards maintaining my current score (accept).

I do hope that, were the paper accepted, the authors would account for the reviewers' remarks, as summarized by the authors in their global response, e.g.:

• Including results on real-world data c.f. DGKr and w7Ci (me). These new experiments gathered by the authors demonstrate the real-world applicability of their work (c.f. Tab. 1 and Fig. 1 of rebuttal PDF).
• Better discussing/referencing existing XR-GS works c.f. P8A3 and w7Ci (me). The results and discussion provided in the authors' response would benefit the readers, by better contextualizing their work.

I would also suggest:

• Clarifying the isotropic simplification at the core of some claims/contributions. E.g., the authors' claims that "[X-Gaussian] uses alpha-blending, which contradicts the unordered nature of X-ray imaging" [response] and that "we can individually integrate each 3D Gaussian to rasterize an X-ray projection" [L162 + Equation 5] are only correct in the context of the isotropic simplification of X-ray imaging. I.e., if actual physics effects, such as Compton scattering, were to be considered, then the ordering of the Gaussians would matter (see preprint [i] for contributions to XR-GS orthogonal to R$^2$-Gaussian's, as well as [ii, iii] w.r.t. why ordering matters in GS-based scattering simulation). I do agree with the authors that most CT reconstruction models adopt the isotropic simplification; and, therefore, that their rasterization simplification is legitimate. However, readers should be more explicitly made aware of the basis of the authors' claims (isotropic approximation [L100, L140, L162]).
• Clarifying the benefits of proposed GS voxelizer compared to existing solutions, e.g., PyTorch3D PC-to-voxel solution, which is also CUDA-based and differentiable but may require a few tweaks to work on Gaussians.
• Including discussed references, e.g. [c] (TV).

Reference:

[i] Gao, Zhongpai, et al. "DDGS-CT: Direction-Disentangled Gaussian Splatting for Realistic Volume Rendering." arXiv preprint arXiv:2406.02518 (2024).

[ii] Zhou, Yang, Songyin Wu, and Ling-Qi Yan. "Unified Gaussian Primitives for Scene Representation and Rendering." arXiv preprint arXiv:2406.09733 (2024).

[iii] Condor, Jorge, et al. "Volumetric Primitives for Modeling and Rendering Scattering and Emissive Media." arXiv preprint arXiv:2405.15425 (2024).

We appreciate your recognition of our work and your valuable feedback.

Q1.1: How does the proposed method compare to FDK when a large number of projections are used?

We further evaluate FDK and our method with 500 to 2000 views. Results in Tab. A show that our method outperforms FDK by a large margin in all settings. Additionally, our method achieves a peak PSNR of around 39.85 dB while FDK is at approximately 37 dB.

Table A. Quantitative results of FDK and our method under dense-view scenarios.

Q1.2: What are the implications of correcting the integration bias in image-based 3D reconstruction tasks?

While RGB-based 3D reconstruction is out of our scope, we share some preliminary findings. We compare vanilla 3DGS and rectified one (R-3DGS) on NeRF synthetic dataset. We define the geometry field as the sum of Gaussian opacities, the same as SUGAR (CVPR'24). For vanilla 3DGS, we compute the mean of recovered 3D opacities of all training views. We then use our voxelizer (Sec. 4.2.2) to query opacity volumes and extract meshes using marching cubes (MC). Note that because the actual iso-value of the surface is unknown, we report chamfer distances (CD) with three MC thresholds.

Results in Tab. B and Fig. 3 (PDF) show that correcting the integration bias does not harm 2D rendering. Furthermore, it improves 3D reconstruction, though less significantly than in volumetric CT reconstruction. We suspect three reasons. First, for opaque objects, Gaussians are trained to be flat, so integration values ($\mu$ in L182) do not change significantly in front views. Second, Gaussians are close to the surface, allowing for reasonable surface extraction using only positions. Third, the splatting technique involves many simplifications in rendering equations, which may have more impact than integration bias on 3D reconstruction.

Since these findings are preliminary, we do not include them in this paper. We will make efforts to develop a bias-free 3DGS for RGB-based reconstruction in future research.

Table B. Quantitative results of vanilla 3DGS and rectified one (R-3DGS) on NeRF-synthetic dataset.