Are Pixel-Wise Metrics Reliable for Computerized Tomography Reconstruction?

Thank you very much for your thoughtful review and constructive feedback.

---

Q1. Insufficient numbers of literature reviews for non-pixel metrics and evaluations.

• We have expanded the related works section to discuss a broader range of non-pixel-based metrics and evaluation methods, such as FID, LPIPS, KID, MS-SSIM, GLIPS, GMSD, etc.

---

Q2. It’s very hard to be convinced when the authors proposed a frozen-model-based metric trained on ~3000 cases from a single institution, and maybe single brand of CT machines. In addition, the testing set is only ~60 cases, which might be even lower than expectation if the proposal is related to a general metric.

• Our segmentator was trained on ~3,000 CT scans from four different machines (GE, Siemens, Philips, Toshiba) covering ~45% of the clinical market, ensuring diversity.
• For the ~60 cases in reconstruction, it is so far the largest CT reconstruction testing set. All baseline methods in Tab. 1 used the following numbers of CT cases for testing:

• The reason for these small testing set sizes is: for each CT volume, each reconstruction method must be trained individually.

---

Q3. The proposed metric was not applied to multiple important organs, like brain, prostate, lung, heart, etc. The generalizability and applicability of the metric is relatively limited.

• We intentionally focused on abdominal organs due to the availability of reliable segmentation methods essential for high-quality evaluation and enhancement.
• Our approach can be readily extended to datasets and segmentation models involving other organs in future work.

---

Q4. The proposed metric is not evaluated in comparison with the observation and evaluation from radiologists, which is important.

• Our Segmentator used for evaluation is trained on ground-truth annotations provided by 21 radiologists over 3 years.
• We have now also conducted a blinded study: all 21 radiologists preferred CARE-enhanced reconstructions.
• Fig. 6 in the paper and Supp. Sec. E.2 visually compare the image quality with and without CARE, making the improvement evident even to non-experts.

---

Q5. Evaluation is only conducted with respect to organs, ignoring the importance of the surrounding tissues, which are also important.

• To clarify, we include vessels as an example of relevant surrounding tissue.
• We have now updated the manuscript to clarify this limitation and emphasize the potential for broader tissue evaluation in future work, as follows: “At present, the evaluation is limited by the lack of high-quality segmentation datasets and models for many surrounding tissues. CARE can readily incorporate additional tissues as soon as such datasets become available, ...”

---

Q6. For the related works of non-pixel evaluation, I would like to request the authors to provide deeper investigations and results comparisons. It is very easy to find similar works, for example but not limited to: (Scholz et al., 2023), (Stein et al., 2023), FID, IS, etc.

• We have expanded the related works section to discuss non-pixel metrics. (details in Q1)
• Regarding the specific works you mentioned:
  • (Scholz et al., 2023): This metric is designed for MRI, and the domain gap with CT limits its direct applicability. Moreover, the lack of released code makes comparison difficult.
  • (Stein et al., 2023): This metric uses features from neural networks trained on natural images, which may not transfer well to CT due to significant domain differences.
  • FID and IS: Both rely on InceptionV3 features pre-trained on ImageNet, which may not be meaningful for CT images.
• Existing metrics generally lack direct assessment of local anatomical detail, which our approach targets.

---

Q7. I would like to ask the authors the reasoning why you believe a frozen-model-based metric trained on ~3000 cases can be reliable.

• Our segmentator achieves expert-level accuracy on a 1,958-case radiologist-annotated test set (see Rev. Wkf4-Q4), and metrics derived from it are highly correlated with human annotation (see Fig. 3).
• CARE’s modular design allows easy integration of future segmentation improvements to further enhance reliability.

Q8. In Fig. 3, what is the meaning of large/small, and also what are the meanings of the numbers shown in x-axis an y-axis?

• In Fig. 3, “large” refers to large organs, and “small” refers to small organs. And regarding the x and y axes:

---

Q9. I’m very curious why the authors take the vessel and the intestine out for extra evaluations. Also, are the “big” & “small” organs defined by physicians following any clinical guidelines? Or is it just arbitrarily grouped? I think this might be too arbitrary if the later case holds.

• We evaluate vessels and the intestine separately because

  • They are elongated, tubular structures whose primary clinical relevance lies in preserving connectivity.
  • The spatial relationship between vessels and tumors is critical for surgical planning and subsequent treatment strategies, making accurate reconstruction of these structures particularly valuable.

• Our grouping strategy for large and small organs is based on clinical guidelines. We partition organs by typical in vivo volumes reported in radiologic volumetry studies. | Classification | Representative mean volume (mL)| Key references | | ------- | --- | ---- | | Large organs ( > ~100 mL)| Liver ≈1200–1700; Kidney ≈150–200 ; Spleen ≈215 ; Pancreas ≈71–83 | (Geraghty, E M et al., 2004) | | Small organs / structures (< ~50 mL or < 10 mm diameter) | Gallbladder ≈30–50 mL ; Each adrenal gland ≈4–6 mL ; Celiac trunk diameter ≈6–12 mm ; Duodenum lumen 2–3 cm, wall volume << 50 mL | (Schick, 2022) |

• The 100 mL / 10 mm cutoff aligns with standard thresholds for distinguishing major organs from accessory structures in abdominal atlases.

---

Q10. If you want to propose a metric for CT quality measurement in general, I think it would be more reasonable if you can include all organs & tissues that will be tested by CT, like brain, lung, fat, prostate, bladder, etc.

• No dataset currently covers all organs/tissues with expert-annotated segmentations.
• To ensure accuracy and reliability, we focus on abdominal structures due to data quality and model performance, but CARE is extensible as the field improves.

---

Q11. I would like to ask the authors to briefly mention CT parameters, and patient cohort attributes in the manuscript to let the reviewers see if they are generalizable enough, or diverse enough.

• In our revised manuscript, we have included the following information for the segmentation and reconstruction datasets to ensure both diversity and generalizability:

---

Q12. Why didn’t the authors compare CARE’s assessments with radiologists’ visual evaluations? Such comparison is crucial for medical imaging and should include not only organs but also other clinically relevant tissues like muscle, fat, and contrast.

• We did conducted a radiologist reader study, as the 3,151 ground truth segmentations used to train the Segmentator were manually annotated by 21 experienced radiologists over 3 years, ensuring CARE assessments align with expert evaluation across organs, vessels, and other clinically relevant structures.
• We have now added results from a blind study with 21 radiologists and added in the revision: “A blind reader study with 21 radiologists showed consistent preference for CARE-enhanced reconstructions in terms of overall image quality, supporting the clinical relevance of our method.”
• About other clinically relevant tissues (e.g., muscle, fat),
  • A comprehensive evaluation is currently limited by available annotated datasets and segmentation accuracy for these tissues. (details in Q10)
  • As segmentation models advance, CARE can be readily extended to more tissues and structures.
• Our current implementation encodes CT contrast via a text encoder for targeted enhancement.

---

Reference

• Geraghty, E M et al. “Normal organ volume assessment from abdominal CT.” Abdominal imaging, 2004.
• Schick, Fritz. “Automatic segmentation and volumetric assessment of internal organs and fatty tissue: what are the benefits?.” Magma (New York, N.Y.), 2022.

Thank you so much for raising your rating. Below are our responses to the remaining three questions, and we hope they address your concerns.

---

Q3 | More organ evaluation for a “generic” metric

• We agree that a generic CT reconstruction metric should ideally cover all clinically relevant organs and tissues. Our current focus on abdominal organs, vessels, and intestines is not because CARE cannot handle more, but because the public domain currently lacks large-scale, voxel-wise annotated CT datasets (e.g., n > 2,000) for many other structures—such as fat, muscle, prostate, and bladder—at the accuracy needed for reliable metric computation. This is a limitation of available data in the field, not of CARE itself.

• For the structures we do evaluate, we leveraged a private large-scale dataset of over >3,000 expert-annotated CT scans from four major scanner vendors. The segmentation model trained on this dataset will be made publicly available, so others can reproduce our metrics and build upon them. This is, in itself, a significant contribution toward enabling reliable, anatomy-aware CT evaluation in the community.

• Once accurate segmentation models for more types of anatomical structures become publicly available, they can be integrated into CARE without changing its design or training process.

---

Q6 | Other metrics

• Following your suggestions, we have now added Stein et al. (2023), FID, and IS as baseline metrics for comparison. FID and IS are good for judging overall visual similarity between images, but they don’t check whether the anatomical structures are actually complete (the main focus of our paper).

• Our metric is different. It uses a segmentation model trained on over 3,000 voxel-wise annotated CT scans from four scanner brands, so its features are tied directly to organ boundaries and connectivity. We then measure NSD for organ surface accuracy and clDice for vessel/intestine continuity, which are built to detect exactly the kinds of structural errors that FID and IS will miss.

---

Q10 | Generalization to tissues like fat and muscle

• CARE is already demonstrated on diverse anatomical classes, including non-solid tubular structures like vessels and intestines, achieving up to +40% clDice improvement (Table 2). This shows the framework’s ability to handle varied morphology. When robust segmentation models for fat and muscle become available, they can be dropped into CARE with no change in method—making the system directly extensible to these tissues.

Thank you for recognizing the clinical relevance and methodological clarity of our work.

---

Q1. The clDice metric is adopted from prior work and not novel to this paper, which the authors acknowledge. While appropriate, this should be clearly emphasized to avoid overstating originality in metric design.

• We would like to clarify that our work does not propose the clDice metric itself; rather, we utilize the clDice metric as an established tool to support our evaluation framework.

• We have now revised the manuscript to more clearly emphasize that clDice is adopted from prior work and not a novel contribution of this paper. For example, we now state:

  “We adopt the clDice metric as introduced in [29] to provide a topology-aware evaluation of segmentation performance. The definition and usage of clDice remain unchanged. ”

---

Q2. clDice is adopted from prior work ([29]) and not novel to this paper. The authors should clarify any differences in its use, as the limited contribution to metric design may reduce the evaluation novelty.

• To clarify, our use of clDice for evaluating vessel and intestine segmentation follows exactly the same methodology as described in the prior work ([29]), without any modification.
• Our intent is not to claim novelty in the design or adaptation of the clDice metric itself. Rather, the novelty of our work lies in being the first to apply the clDice metric—originally developed for segmentation tasks—to assess reconstruction quality in the context of CT reconstruction.
  • Similarly, for large and small organs, we adopt the NSD metric in the same spirit, introducing it as an evaluation tool for this new domain.

---

Q3. Since the metrics rely on nnU-Net predictions rather than manual annotations, segmentation errors may bias evaluation and cause error accumulation. The paper should explicitly discuss this limitation, its impact on metric reliability, and possible mitigation strategies.

• We agree that relying on nnU-Net predictions rather than manual annotations introduces the risk of segmentation errors, which may bias metric estimation and potentially lead to error accumulation in both evaluation and enhancement. This is an inherent limitation of any evaluation framework based on automated models.

• To address this, we have analyzed the impact of segmentation errors in Fig. 3 and observed that segmentator-based metrics are highly correlated with GT-based metrics, indicating strong reliability within our experimental setting.

• We have added the following explicit discussion to clarify this limitation:

  “A limitation of our approach is that metric reliability is contingent on the quality of automated segmentation; errors in segmentator predictions may propagate to metric estimation and evaluation. While our experiments show high correlation between segmentator-based and ground-truth-based metrics, ongoing advances in segmentation models are expected to further mitigate this risk.”

---

Q4. CARE introduces additional complexity due to the diffusion model and segmentation supervision. The paper briefly mentions this but could provide more detailed analysis on computational cost, scalability, and applicability to large 3D volumes or real-time scenarios.

• Regarding computational cost,

  • We have provided implementation details, including computational requirements, in Supp. Sec. A.3 of the supplemental materials. We have now clearly referenced this section in the main text. For example:

    “The implementation details, including computational costs and hardware configurations, are provided in Supp. Sec. A.3.”

  • The segmentation model contributes to computational cost only during CARE’s training phase, and the overall increase in training cost is minimal—just +3.28% in the case of SAX-NeRF. (details in Rev. Wkf4 - Q1)

• In terms of scalability, our framework is designed to be flexible.

  • In terms of models, the Segmentator component can be replaced with other segmentation models as needed; the diffusion model can be replaced with other latent diffsuion backbones.
  • In terms of data, CARE can be trained using more CT reconstructions if available, which will further improve its generalization and performance.

• With respect to applicability to large 3D volumes,

  • We would like to clarify that while we compute pixel-wise and anatomy-aware metrics in 3D, the diffusion process itself is performed in 2D. As a result, large 3D volumes do not pose a bottleneck for our approach.
  • Our test set includes 61 CT scans, some of which are large volumes (the biggest one has a shape of 512 x 512 x 1229). The primary difference for larger volumes is simply longer inference times.

• As for real-time scenarios, it is important to note that, in the field of medical image analysis, accuracy is prioritized over speed.

  • For example, after a CT scan, both physicians and patients generally prefer to wait for a higher-quality reconstruction rather than receiving a lower-quality result immediately. Therefore, real-time efficiency is not the main focus in CT reconstruction applications.

  • We have now added this discussion to the revised manuscript, as follows:

    “In clinical CT reconstruction, accuracy is typically prioritized over real-time efficiency, as ...”

---

Q5. The current focus is on abdominal CT scans. The generalization to other anatomies or modalities is suggested as future work but not demonstrated.

• Thank you for this valuable comment. We agree that this limitation should be more explicitly stated in the manuscript.

• We have now made this point clearer in the Limitations and Future Work section. For example, we added:

  “The current study is limited to abdominal CT scans. Extending the proposed framework to other anatomies or imaging modalities remains an important direction for future research.”

Thank you for your constructive comments and for highlighting important aspects of our framework.

---

Q1. Weaknesses include reliance on nnU-Net’s segmentation accuracy (risking error propagation), added computational cost from latent diffusion and segmentation, and sensitivity to hyperparameter settings. The paper should further discuss hyperparameter tuning, generalization, and the proposed anatomy-aware metrics in both training and testing.

• Segmentation error propagation: Our segmentator achieves state-of-the-art performance on a large CT dataset, reducing such risks; CARE’s modular design allows easy integration of future segmentation improvements to further enhance reliability; the use of imperfect models during training is common practice in fields like GANs and reinforcement learning. (details in Q2)
• Additional computation: The CARE framework involves extra computation due to the segmentation model and latent diffusion process during training, but this additional cost remains relatively modest and is justified by the significant gains in reconstruction quality and anatomical fidelity.
  • Segmentation model in CARE: The segmentation model only brings extra computational cost during CARE training. At inference time, our CARE model operates equivalently to a standard latent diffusion model, requiring no additional computations for segmentation.
  • 'Reconstruction+CARE' pipeline: Most computational overhead is incurred by the underlying reconstruction method, and the subsequent CARE model training typically constitutes only a small fraction of the total GPU hours and wall time.
  • For example, training a SAX-NeRF model on a CT scan takes about 12 hours, totaling 732 hours for our 61 CT scans on a GPU. In contrast, our CARE model uses 25 CTs reconstructed by SAX-NeRF, requiring roughly 24 hours (50k iterations) on a GPU. This results in only a 24/732=3.28% increase in computation.
• Hyperparameter settings and generalization capacity: All implementation details, including training hyperparameters, are provided in Supp. Sec. A.3; The loss weights (λₙ = 1, λₚ = 1, λₛ = 0.001) were empirically determined, with λₛ carefully tuned to balance anatomical guidance and reconstruction quality. (details in Q4)
• More discussion of metrics in training/inference: We have included detailed descriptions of the metrics’ integration into both the training and evaluation pipelines in Supp. Sec. D, along with the design rationale for each loss component.
  • Training: CARE uses an auxiliary segmentation task to guide reconstruction toward preserving anatomical structures, with segmentation-based supervision applied during training to evaluate organ fidelity.
  • Inference: During inference, CARE performs 2D slice-by-slice diffusion without the segmentation module; metrics are applied afterward to assess anatomical fidelity without affecting the reconstruction process.

We hope these clarifications address your concerns, and we appreciate your helpful feedback on how to improve the clarity of our manuscript.

---

Q2. The framework and the metric relied on the nnU-Net's segmentation accuracy, with risks of error propagation. CARE’s anatomy-guidance loss tends to reinforce incorrect structures when given noisy segmentations.

• We acknowledge that inaccurate segmentations may impact CARE’s ability to learn robust anatomy-aware representations. However, our segmentator achieves state-of-the-art (SOTA) performance on a dataset with 1958 CT scans, minimizing such risks in practice. (details in Rev. Wkf4 - Q4)

• The CARE framework has a modular design—future advances in segmentation models can be easily incorporated to further improve the reliability of anatomical guidance.

• We would like to highlight that the use of imperfect models during training is not uncommon in the literature.

  • In GANs, the discriminator is continuously improved throughout training but is never truly perfect.
  • In reinforcement learning algorithms such as PPO, the reward model and value model are pretrained and may inherently carry biases from their training data.
  • As long as these models are sufficiently informative or well-calibrated, they can still provide meaningful guidance and contribute positively to the overall learning process.

---

Q3. This paper proposes anatomy-aware CT reconstruction metrics, using L1 loss for pixels and cross-entropy for anatomy guidance during training. However, it is unclear if these metrics aid training or are used only for evaluation, in which case they do not improve reconstruction.

• Our proposed metrics utilize NSD and clDice, both of which originate from the segmentation domain and are non-differentiable by definition; thus, directly using them as anatomy-guidance losses during training is unpractical.

• As segmentation metrics, NSD and clDice will naturally improve if we optimize towards accurate anatomical segmentation.

  • Aiming for general applicability within our CARE framework, we selected the widely used cross-entropy loss as the segmentation-guided loss function.
  • Dice loss is another common segmentation loss, but it often performs poorly when dealing with small or thin structures. For instance, if a thin vessel is completely missed, Dice loss immediately reaches its maximum value (1.0), providing little meaningful gradient information for training.

• Although the anatomy-aware metrics themselves are used primarily for evaluation purposes, their segmentation essence has been integrated into our training pipeline via the anatomy-guidance loss Ls, which actively guides the reconstruction toward better anatomical fidelity.

---

Q4. The hyperparameter setting is crucial to balance the anatomy-guidance loss and the pixel loss. It is unclear whether the proposed model can avoid risks under-prioritizing structural completeness.

• We have provided a comprehensive list of all implementation details in Supp. Sec. A.3, including training hyperparameters.
• The noise-prediction loss Ln has a weight of 1; the pixel-space reconstruction Lp weights of $\lambda_p=1$, the anatomy-guidance loss Ls weights of $\lambda_s=0.001$. These parameters are chosen based on thorough empirical studies:
  • During training, we observed that both Ln and Lp losses decrease from approximately 0.1 to 0.01, while the cross-entropy segmentation loss Ls drops from around 3 to 0.1, necessitating a reduced weight for Ls in the overall loss.
  • Through ablation, we found that setting λs > 0.001 (e.g., 0.01) leads to the generation of meaningless noisy images, whereas too small a value diminishes the anatomical effect. Ultimately, λs = 0.01 provided the best balance between anatomical preservation and reconstruction quality.

---

Q5. Fig. 6 visualizes results with and without semantic supervision, but for methods like TensoRF and R2-GS, differences are minimal. Highlighting these differences would be helpful.

• This high similarity occurs because the baseline CT reconstructions already achieve reasonable quality, making further improvements less visually distinguishable in the segmentation maps.

• Even though their shape might look alike (Dice score 92.6 and 93.1), their NSD metric, which highlights the difference in boundary, is more prominent (NSD score 90.1 and 93.0).

• To further address your concern, we computed a difference map by subtracting the two reconstructed volumes (with and without CARE), which more clearly highlights the regions of change.

  • Although we are unable to show this figure here in the response, our analysis revealed that the differences are most prominent in (1) the shape and contrast of bony structures, (2) organ grayscale values and boundary delineation against surrounding tissues, and (3) global texture patterns—for instance, the horizontal and vertical artifacts frequently observed in TensorRF without CARE are not present when CARE is used.

  • We have now supplemented the manuscript with these detailed observations and difference map visualizations to better highlight the improvements brought by CARE, as follows:

    “We include a difference map between reconstructions with and without CARE in the revised manuscript. These maps reveal that the most prominent changes occur at organ boundaries and within bone structures, where CARE more effectively reduces artifacts and enhances contrast.”

Thank you for giving the Accept score and acknowledging the potential of our proposed metric for guiding anatomy-aware methods in CT reconstruction.

---

Q1. Based on my understanding, the metric focuses on abdominal area only. This information should be stated in the abstract and introduction, since there are other regions such as brain and whole-body CT.

• We agree that our current manuscript does not clearly state that the proposed metric is specifically designed for abdominal CT reconstruction.

• We have revised the abstract, introduction, and method sections to explicitly clarify that our work focuses on the abdominal region, distinguishing it from applications in other anatomical areas. For example, the revised abstract now includes:

  “Our proposed anatomy-aware metric is specifically developed for evaluating abdominal CT reconstruction, leveraging organ-level segmentation within the abdominal region.”

---

Q2. Caption Fig4. Add more information since the losses appear in the figure but not in the caption. Is Lc a typo?

• “Lc” is a typo and should refer to “Ls” as defined in Equation (7).

• We have now clarified the meaning of all the losses shown in Fig. 4, both in the caption and in the main text. The updated caption reads in part:

  “...CARE can be integrated into any reconstruction method to perform its enhancement capability. The overall training is supervised by three loss terms: the noise-prediction loss $L_n$, pixel-space reconstruction loss $L_p$, and anatomy-guidance loss $L_s$.”...

---

Q3. Lines 247–251 overstate the reordering effect of the new metric. While it is useful, top-performing methods under SSIM/PSNR, like SAX-NeRF, remain top-ranked under the new metric, so the claim should be toned down.

• We agree that the statement in Lines 247–251 was overstated, and we have revised the text to remove any exaggerated claims.
  • While the overall ranking of methods may not change drastically, we believe the anatomy-aware metric provides a more sensitive differentiation between methods. For example, FDK and SART differ by only ±0.1 in SSIM/PSNR, but their differences reach -1.0 and +1.4 percent, respectively, on small organs and vessel scores under our proposed metric.
  • The revised text now reads, in part: “While the overall ranking is largely consistent under the new metric, the anatomy-aware evaluation provides finer granularity—such as the small-organ and vessel scores—enabling more meaningful comparisons between methods with similar SSIM/PSNR values.”

---

Q4. Another concern is in terms of the quality of the Segmentator which can influence the overall performance negatively. A study testing different segmentation models should be performed.

• We have reported the segmentation performance of our nnUNet-based Segmentator on a large test set of 1,958 CT scans in Supp. Sec.B Tab. 3.
• Due to the modular design, as the field of CT segmentation continues to advance, our framework can easily incorporate future improvements in segmentation methods.
• To further address your concern, we compared our Segmentator with the top three state-of-the-art segmentation models highlighted in the Touchstone 1.0 Benchmark (Bassi, et al., NeurIPS 2024), evaluating all models on our 1,958 CT scans dataset. The complete comparison is shown below with IQR:

• As shown, our Segmentator achieves highly competitive performance compared to these SOTA methods. We therefore believe that our choice of Segmentator for building the CARE framework is justified.

---

Reference

• Bassi, Pedro RAS, et al. "Touchstone benchmark: Are we on the right way for evaluating ai algorithms for medical segmentation?" NeurIPS 2024.
• Roy, Saikat, et al. "Mednext: transformer-driven scaling of convnets for medical image segmentation." MICCAI 2023.
• Wang, Yihe, et al. "Medformer: A multi-granularity patching transformer for medical time-series classification." NeurIPS 2024.
• Huang, Ziyan, et al. "Stu-net: Scalable and transferable medical image segmentation models empowered by large-scale supervised pre-training." arXiv 2023.