Introducing 3D Representation for Dense Volume-to-Volume Translation via Score Fusion

Dear reviewer XUPB,

Thank you for your detailed and constructive feedback. Here is our response to the raised questions and concerns:

Weakness (1-2): Generalizability beyond brain MRI:

Thank you for your insightful review. We highly agree that results on datasets beyond brain MRI can significantly strengthen our paper. We evaluated our method on sparse-view reconstruction with CT data:

Here are the quantitative results for sparse-view CT:

We perform the sparse-view reconstruction task on the CHAOS[1] with 36 views. The CHAOS dataset contains CT/MRI images of the liver. Each CT volume has 78-289 slices of resolution 512x512. During training, we cropped the third axis to have a common shape of 64. During inference, we adopted a simple sliding window strategy along the third axis. We resized each CT slice to 256x256, following TPDM[2]. These results on the CT data demonstrate our method's effectiveness beyond MRI, showing the generalizability of Score-Fusion to other modalities (CT). It also demonstrates our method's effectiveness on a different organ (liver). The results help support the claim of “medical image” in the title.

We acknowledge that there are many other modalities, such as ultrasound, and there are other organs, such as the breast. The results on these datasets will also enhance our claims. Unfortunately, given the limited time during rebuttal, we are not able to include even more dataset results and we leave these applications to future work.

However, we believe it is a common practice to use two modalities, especially CT and MRI, for papers targeting volumetrics medical image translation, according to multiple previous works, including TPDM[2], ScoreSDE[3], DiffusionMBIR[4], etc.

[1] CHAOS - Combined (CT-MR) Healthy Abdominal Organ Segmentation Challenge Data, A.E. Kavur et al

[2] TPDM.

[3] Solving Inverse Problems in Medical Imaging with Score-Based Generative Models, Song et al.

[4] Solving 3D Inverse Problems using Pre-trained 2D Diffusion Models, Chung et al.

Weakness 3: Non-isotropic medical imaging:

As mentioned above, our new experiments on CT data are trained on 256x256x64 data due to the nature of the CHAOS dataset. Due to the time limit of the rebuttal, we did not get results on the fastMRI dataset yet, where the resolution is 320x320x20. We plan to include the fastMRI results in the second round of rebuttal. We hope this can still be considered.

As an extension of score-fusion, we tried to extend Score-Fusion for video super-resolution. We used MGLD-VSR[5] as the 3D model and used two perpendicular models on the time-space slices (i.e., (x-t) slices and (y-t) slices). Due to the capacity requirement of the video model, we are only using 5 consecutive slices in this problem, resulting in a 1280 x 960 x 5 or 2560 x 1920 x 5 resolution for the volume. This can potentially help address the concern of extreme resolution. Here are the quantitative results on the VideoLQ dataset[6]. We use DOVER[7] as a metric, which is a spatial-temporal metric for video quality assessment.

Although the video results might be beyond the scope of the rebuttal, we would be glad if these preliminary results could help address the concern and could be included in the current paper.

[5] Motion-Guided Latent Diffusion for Temporally Consistent Real-world Video Super-resolution

[6] Investigating tradeoffs in real-world video super-resolution

[7] Exploring Video Quality Assessment on User-Generated Contents from Aesthetic and Technical Perspectives

Question 1:

See weakness 1-2

Question 2-3:

Network architecture: We have already included a detailed model architecture in Supplementary Material section F and Tables 10, 11, 12. We will emphasize this by pointing to the Supplementary Material section in the experiment section of the main paper for clarity.

Question 4:

We acknowledge that a lot of modality translation is a task setting that is not directly useful in the medical imaging domain, such as MRI to CT, where MRI lacks certain information due to physical properties, as the reviewer mentioned.

However, from a multi-modal learning perspective, efficiently training a multi-modal model by fusing knowledge from multiple single-modal models can be useful due to the difficulty in training a foundation model. We will modify the discussion on this part in our final version of the paper.

Also, we want to emphasize that our problem setting ( i.e., low resolution flair + T1ce => high resolution flair) does not have the issue that the reviewer mentioned.

I hope the above rebuttal addresses your concerns. Again, we would like to express our sincere gratitude for taking the time to review our paper and providing insightful questions.

Dear Reviewer vsot,

Thank you for your insightful review of our work. Here is our response:

Experimental Question 1, LDM for diffusion framework:

The reason why we are not using LDM is related to our task setting.

Intuitively, the main aim of Super-resolution is to recover/generate the high-frequency detail from a lower resolution image. In both LDMs, the high-frequency details are mainly generated with an autoencoder, which may result in worse performance in super-resolution. Therefore, we chose DDPM over LDM, in which the high-frequency detail is carefully modelled with a diffusion process, although LDM could potentially improve computational efficiency.

Empirically, we find that the 3D LDM's autoencoder is usually unreliable. At an early stage of this project, we tried 3D LDM using medical diffusion[1]. Although the model can reconstruct the image with semantically correct and realistic results, the reconstruction error is already pretty large. resulting in a ~25 reconstruction PSNR. (We get a reconstruction PSNR of 26.23 on a subset of the validation set of the BraTs dataset). We do acknowledge there are more advanced 3D LDM recently, e.g. MAISI[3], but they rely on much more data (more than 20 datasets combined for MAISI) to train a reliable auto-encoder model, which makes an unfair comparison between models. In addition, if we use perpendicular 2D LDM, their diffusion process will be in different perpendicular 2D latent spaces, which makes the fusion of 2D scores difficult. Therefore, we do not use 2D LDMs such as Make-A-Volume[2].

Another reason is that Score Fusion is the first work to explore a learning-based method for score function fusion to the best of our knowledge. We believe that using a more standard and vanilla version of the diffusion model can provide more solid and generalizable insights in terms of score-function fusion.

In future work, we may consider extending score fusion to latent diffusion models in a text-to-image setting, where semantic information is better supported by the LDM. We will also include the negative results for 3D LDM and the discussion in the future version of our paper. Thank you for your insightful review.

[1] Medical Diffusion: Denoising Diffusion Probabilistic Models for 3D Medical Image Synthesis.

[2] Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D Brain MRI Synthesis.

[3] MAISI: Medical AI for Synthetic Imaging

Experimental Question 2, Statistical significance:

We have already included the std values in the Supplementary Material in Table 5, confirming that our improvements are statistically significant. In the future version of our paper, we will show the results more directly in the experiment section for clarity.

Again, we appreciate your recognition of our work's potential for real-world applications. We would like to express our sincere gratitude for taking the time to review our paper

Dear reviewer AnYS,

We want to thank you for your feedback noting that "the topic is interesting and method is timely." and providing constructive suggestions for our paper.

Experimental Question 1, multi-scale SR:

Thanks for raising this point. We agree that multi-scale super-resolution is common practice. We tried to address your concern about performing super-resolution at multiple scales and to demonstrate the generalizability of different super-resolution scales. We performed experiments on 2x and 8x SR. Here are the quantitative metrics. (The 4x results are the original results presented in the paper.)

Quantitative Results for 2x Super-resolution:

Quantitative Results for 8x Super-resolution:

We can see that our model consistently outperforms baselines across multiple super-resolution scales for all three metrics, showing the generalizability to different super-resolution scales.

Question 2 Missing references:

We appreciate your suggestions [A, B] and will incorporate these in our final version:

Question 3, Typos:

Thank you for pointing this out. We will ensure consistency in section title capitalization in the final version and also check for other typos.

Again, we greatly appreciate your time and effort in reviewing our work and providing positive feedback and constructive advice.

Dear reviewer ELBW,

We sincerely thank you for taking the time to review our paper and for your insightful and positive assessment, noting that "the paper is very well written and easy to follow with good synthesis of literature" and "the contribution is well motivated." Here is our response to your questions and reviews.

Weakness 1: challenge-specific metrics for tumor segmentation:

We have added lesion-wise Dice scores and 95% Hausdorff Distance (HD95) metrics below. Following [1], we set thresholds (150, 250, and 100 for TC, WT, and ET) to remove small predicted segmentations across all the evaluations.

Both Condition Results:

Modality Translation Results:

Super Resolution Results:

Our methods show improvements over the baselines in most cases, with a particularly notable gain for WT lesion segmentation under the Modality Translation task. Moreover, lowered Hausdorff distances suggest that our method helps the segmentation model capture subtle lesion boundaries more accurately, indicating more precise morphological alignment. This further demonstrates that our model's generated results are more friendly for downstream segmentation models.

[1] Ferreira, A.,et al. How we won brats 2023 adult glioma challenge? just faking it! enhanced synthetic data augmentation and model ensemble for brain tumour segmentation. arXiv preprint arXiv:2402.17317.

Weakness 2: sparse-view reconstruction in CT:

Thanks for raising this. We highly agree that sparse-view reconstruction in CT can substantially add weight to the claim of better representation learning.. We evaluated our method on sparse-view reconstruction with CT data:

Quantitative results for sparse-view CT:

We perform a sparse-view reconstruction task on the CHAOS dataset[1] with 36 views. The CHAOS dataset contains CT/MRI images of the liver. Each CT volume has 78-289 slices, each slice 512x512. During training, we cropped the third axis to have a common shape of 64. During inference, we adopted a simple sliding window strategy along the third axis. We resized each CT slice to 256x256, following TPDM[2].

These results on CT data demonstrate our method's effectiveness beyond MRI, enhance the generalizability of Score-Fusion to other modalities, and help support the claim of “medical image” in the title.

Again, we sincerely appreciate your thorough review and constructive suggestions, which have helped us improve our work.

[1] CHAOS - Combined (CT-MR) Healthy Abdominal Organ Segmentation Challenge Data, A.E. Kavur et al

[2] TPDM.