X-Diffusion: Generating Detailed 3D MRI Volumes From a Single Image Using Cross-Sectional Diffusion Models

We thank the reviewer for their insightful comments and relevant questions, which we tried to address to the best of our ability. We updated the paper PDF with section D on Additional Discussions in supplementary material for the rebuttal.

Many of the approaches that are used for MRI reconstruction rely on the notion of data consistency which can ensure that there is consistency between the reconstructed imaging data and the extent of signals that are actually obtained. The proposed approach provides no sense of such confirmation.

While our X-Diffusion is indeed a feed-forward reconstruction method and not an optimization-based method ( improved speed advantage), it does incorporate multi-view consistency since it generates the MRI volume from independent different views, each having a novel perspective and output and the final output is the average volume of these independent volumes. However, we agree with the reviewer that optimization-based methods have an edge in terms of incorporating a feedback loop to improve the output. We run an experiment to do Test Time Augmentation to improve the output of X-Diffusion based on feedback from the output and input signals and report the results in Table XI and Sec D.9 in the updated Appendix.

It is mostly an impractical scenario where a single high-resolution slice will be obtained instead of obtaining multiple undersampled slices. So the overall premise of the work unfortunately seems more academic rather than something that would have a clinical impact.

We kindly agree with the reviewer that these generated “pseudo MRISs” are not a subsite for actual MRI scans as we described briefly on page 2 “It is important to note that the generated MRIs are not clinical replacements for true MRIs yet, but could provide a quick, affordable, and informative “pseudo-MRI" before conducting a full MRI examination.”. So we envision in the future this can be used in an active setup, while the patient is conducting a scan the model reconstructs the MRI and informs the technician where to scan next, reducing reducing in the scanning process, and hence reducing the time in conducting the MRI scan, reducing the cost of running MRI scans, and potentially saving countless lives. We hope that our X-Diffusion (especially if published at ICLR 2025) will act as the first step towards this future we envision. We include Sec. D1, and D.2 in the updated Appendix to highlight this discussion.

In prior work, there is a very incomplete analysis of work in image reconstruction and super-resolution. It is unclear what “classical” means based on the references, does classical mean anything without foundation models? This doesn’t seem accurate.

We apologize for the minor issue in the related works section. We have modified the related works section to be more comprehensive and include the pre-learning works and is now integrated into the paper PDF.

In figure 4, the acquisition is actually axial and the study depicts the acquisition to be coronal. Since there is always a fully sampled dimension readout dimension in MRI, the reformatting of this 3D volume does not accurately convey the representational capacity of the model. Actual slice traversal needs to be in the true slice direction.

While it's possible to limit the slice traversal to only the true slice collection plane, there is no specific reason why other directions will not be helpful given the model is a learning-based model and would benefit greatly from scaling, diversity, and augmentation. In fact, the multi-view slicing is “of particular interest to the medical imaging community and constitutes - to the best of my knowledge - a unique contribution of the paper” as reviewer F5BK finds it. We run an ablation study to clarify this particular point about the original view vs other views effect in UK Bioban, which shows there is no significant difference between the different planes, but using the multi-view volume aggregation indeed yields improved performance. This is explained by the fact that deep learning models like X-Diffusion benefit from the increase and variety of the size of training data to generalize, which benefits from exposure to as many views as possible. We show the results in the appendix updated PDF in sec D.15 in Table XIV.

A better evaluation would be to show a progression of PSNR values the further away a test slice is being computed from a conditioning slice.

We agree with the reviewer on this point and we show error plots as a function of distance from the conditioning slice in updated Appendix Sec D.8 Fig XV. We can see that as the distance increases the error increases before slowly decreases as the information content is reduced at the boundary and the model can predict this accurately.

For the example provided in Figure 5, it is interesting to think about why the method should work in the first place. Given that this is an ill-posed problem, why should a model be able to predict a slice that is 20 slices away accurately?

This can be answered easily as follows. X-Diffusion is pre-trained with 5 billion images in LAION and fine-tuned on 48 K MRI samples with almost 17.9M 2D images. This gives the model the ability to generalize to samples that are especially similar to the domain in which it was trained. For example, Zero123 paper showed how finetuning Stable Diffusion on 3D object models, allows the diffusion model to generate 3D objects from a SINGLE IMAGE, even though it is an ILL-POSED problem, as there might be many ways the images can be formed. However, people in the 3D computer vision community have verified these approaches on 3D datasets, and actually, the results are very convincing and even 3D geometrically accurate [a][b]. The same rationale is applied here with X-Diffusion generating MRI 3D volumes from a single/few slices, and that is actually the original motivation for why this whole project came into play.

[a] Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors, (ICLR 2024)

[b] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation, (ECCV 2024)

The better test would be to pick a conditioning slice that does not have a tumor but from a patient with a tumor. Then, will the model correctly produce a tumor? That is the real test. I hypothesize that there is no reason a model should be able to predict that. A brief sentence or two is included in the paper but this analysis can be made more rigorous.

We indeed performed this experiment in the original paper appendix, it is in appendix Sec B,3 and Figure VI and Table I. We on purpose picked the conditional slice to not include the tumor and report the results as input gets closer to the tumor. The study briefly shows that as the input conditioning slice gets closer to the center of the tumor, its prediction of the entire tumor increases. However, even with no intersection, the performance is still decent.

Spine curvature: What is the distribution of spine curvature? One may think this is a regression to the mean of healthy spines. showing a Bland-Altmann or regression pot where would be better.

A healthy spine in coronal view would be straight, If not healthy there would be a curve in the spine, and curvature we can measure. We apologize for not being clear, we provide Pearson’s correlation, which is a measure of how well the curve matches. We show the requested additional plots in the updated appendix sec D.16 Fig XIX (Bland-Altman) and sec D.8 Fig XIV (scoliosis severity categories). We can see that the curvature creates varied distributions of scoliosis severity and the X-Diffusion generations preserve these categories of scoliosis severity.

All datasets had 155 slices. In the case of using 31 conditioning slices, it is not clear why one should use this approach instead of just undersampling the 3D data by 5x and then reconstructing it with variational methods. These methods can easily provide accelerations of 8-10x with much higher metrics.

This is exactly true, the baselines can do this (easy) job well, and actually, by looking at Table 1 we can see the difference between the baselines and our X-Diffusion shrinks to almost 0 and actually in plot fig X in the updated appendix, if input slices go 100, the baselines beats our X-Diffusion. That is why the main focus of our work, different from all previous works in MRI reconstruction is that we reconstruct accurate MRIS from extremely sparse input, as clearly explained in the introduction and Fig 1 of the paper. This is a new area of research not typical in MRI reconstruction, and we hope our X-Diffusion paper ( if accepted to ICLR 2025 for example ) will open this door for other works in the field.

In Figure 6, up until 10-13 slices, there are significant hallucinations in the structure and contrast of the grey and white matter. These results would cause radiologists quite a bit of anxiety! Like the previous point, at this point, why not just use a variational reconstruction method?

While the variation reconstruction method will work for a large number of slices like 31, or more, it fails miserably with very few slices as can be seen in the same Fig6 with 2-slice input for the baselines in the top two rows.

The results in Figure 6 and the massive impact in hallucinated details entirely reduce confidence in all 1-5 slice results. This further shows how poor a metric PSNR is.

While PSNR is not a perfect metric, we include SSIM like in Table II in the appendix. Additionally, we include LPIPS metrics for the main results of the paper and add them in the updated appendix PDF in Section D.10 Table X.

Total brain volume is not the correct metric to assess fidelity. Cortical volumes and folding patterns would be better. One would hypothesize that it is likely trivial to regress total brain volume from a single brain MRI slice anyway.

We perform the requested cortical volume measurements and folding patterns and report the results in the updated Appendix PDF Section D.14.

It is pretty shocking to me why the model should be able to work in the OOD setting to synthesize knee MRI scans. Can the authors please provide a rationale for why this should work in the first place? The UKB dataset includes IDEAL fat-water sequences that are MUCH lower resolution than the 2D TSE knee MRI fast MRI sequences that have a proton density contrast. Why should the model know this distribution Section 6.3 attempts to answer this but simply claiming Stable Diffusion training and large non-domain specific datasets unfortunately does not seem convincing.

We are not the first to show the OOD capability of diffusion models. Score-MRI showed that their model trained solely on proton density fat suppression (PDFS) coronal knee scans achieves remarkable performance in generating brain MRI slices (axial plane) (see Fig 9 of Score-MRI paper (https://arxiv.org/pdf/2110.05243).) The idea advanced in Score-MRI comes from the posterior sampling in the diffusion being very robust to distribution shifts. Moreover, we compared the PSNR of two randomly sampled knee and brain MRIs and obtained an average PSNR of 18.34 over 30 sampling pairs. This is a relatively high PSNR value which suggests that the knee and brain MRI share to some extent patterns in their features.

Since H = W = D in this study, what would happen when the data is non-isotropic, as is most common in MRI applications and protocols.

Zero padding and resizing cropping are all common and naive approaches to handling such scenarios. Actually, in our experiments with UKBB, we pad with zeros all dimensions to make them equal.

The lack of data consistency of proposed approach.

The authors have added test time augmentation as well as multi-view consistency. However, all of these are data-driven metrics and do not necessarily enforce data consistency with the acquired raw data. Note that the optimization-based approaches mentioned here are actually commonly unrolled variational models, which are more efficient than diffusion models.

Pseudo-MRIs:

Localizer/scout scans are always performed first and provide this ability anyway, so it is not clear what the additional overhead of the proposed method adds.

Hallucinations when <13 slices are used for conditioning:

As shown in the figures in the manuscript, there are extensive hallucinations in the generated images. Independent of the metrics, which are low to begin with, these hallucinations would not be high-fidelity enough for any clinical application.

Thank you for adding the new analysis regarding cortical volumes and the spinal curvature. These results make the paper stronger.

Figure 5 Discussion:

Thank you for sharing the insights from the natural 3D imaging community. However, there are some differences because in a medical image the image that is being reconstructed is dense. In contrast, for general-purpose 3D imaging, we are simply reconstructing the outer surfaces of images which are sparser. In any case, it is not clear that there is any theoretical explanation possible at this particular point, but the comment was mostly made to get the author's perspectives.

Overall, from a methodological perspective, there are some nice new innovations that this work provides. However, the lack of applicability to clinical settings, as mentioned by the authors, ultimately limits the impact that the proposed method could add. I will raise my score from a 3 to a 5.

Thank you for sharing the insights from the natural 3D imaging community. However, there are some differences because in a medical image the image that is being reconstructed is dense. In contrast, for general-purpose 3D imaging, we are simply reconstructing the outer surfaces of images which are sparser. In any case, it is not clear that there is any theoretical explanation possible at this particular point, but the comment was mostly made to get the author's perspectives.

Yes, indeed there is no theoretical evidence of why such diffusion models generalize well on dense predictions ( like the recent SORA video diffusion models generating completely novel sequences[a] ) or why LLMs trained on next token prediction can solve puzzles and Math Olympic questions and even achieve an IQ higher than the human average [b]. However, LLMs and Diffusion models are being used all over the world currently to create media, write programs, advance Science, and enhance human productivity in all domains of life. We believe our X-Diffusion is a good baby step in this direction of Generative AI helping the struggling costly healthcare sector worldwide.

[a] https://openai.com/index/sora

[b] https://klu.ai/llm-leaderboard

Overall, from a methodological perspective, there are some nice new innovations that this work provides. However, the lack of applicability to clinical settings, as mentioned by the authors, ultimately limits the impact that the proposed method could add. I will raise my score from a 3 to a 5.

We thank the reviewer for the supporting comments. While we acknowledged potential clinical future work that can be done to verify X-Doffusion feasibility on clinical setup on actual patients, we believe that would be suitable for a medical journal clinical study and is out of the scope of the current submission to ICLR conference which is about the method novelty, technical execution, and reconstructions results, where the reviewer agrees on (also reviewer fhnz just mentioned that). Hence we don’t see this clinical future work as a reason to lower the score of our paper which could result in a rejection. Hence we kindly ask the reviewer if all of their other concerns were addressed properly and if they believe the paper is worthy of acceptance to reconsider raising the score to at least 6 which is still borderline acceptance but at least lowers the risk of rejecting the paper solely based on the clinical feasibility aspect of the paper.

We thank the reviewer for increasing the score to 5. Here is the response to the additional concerns raised during the discussions.

The authors have added test time augmentation as well as multi-view consistency. However, all of these are data-driven metrics and do not necessarily enforce data consistency with the acquired raw data. Note that the optimization-based approaches mentioned here are actually commonly unrolled variational models, which are more efficient than diffusion models.

We added in Section D.17 Table XV an actual optimization-based approach of the X-Diffusion model for each acquired raw data during test time. The results show a slight improvement in the quality of the reconstructed MRI compared to the feed-forward only method (+0.5 PSNR), but with an additional 3X time required during inference. This explains why we resort to a feed-forward diffusion instead of an optimization approach. variational models indeed are more efficient than Diffusion, but they typically struggle with detailed output generalization (e.g. VNNs [a] compared to Stable Diffusion )

[a] A Variational neural network for image restoration based on coupled regularizers, Multimedia Tools and Applications, 2023

Localizer/scout scans are always performed first and provide this ability anyway, so it is not clear what the additional overhead of the proposed method adds.

In MRI, the process of acquiring images involves filling k-space, a matrix that stores raw data before image reconstruction. Traditional MRI scans often employ Cartesian trajectories, where k-space is filled line by line. This method can be time-consuming, as each line requires a separate phase-encoding step, contributing to longer scan durations [a]. Localizer or scout scans are preliminary images used to determine the position and orientation of subsequent imaging slices or volumes. These scans are typically acquired quickly, often within a few seconds, and provide a reference for planning the imaging sequence. However, they usually offer low detail and may cover only a small volume, limiting their diagnostic utility [b]. In contrast, our X-Diffusion by requiring one/few fully sampled slices, reduces the time requirement of covering the original volume and allows us to reach the precision very close to obtaining the complete MRI scan. The whole paper is dedicated to verifying the precision of these generated MRIs (from brain tumour to spine scoliosis). Unlike the current scout MRIs that contain low details or cover small volumes [b], X-Diffusion “pseudo-MRIs” cover large volumes and are detailed as shown in the paper.

[a] Rinck PA. Magnetic Resonance in Medicine. The Basic Text­book of the European Magnetic Resonance Forum., 2024.

[b] https://mrimaster.com/mri-localizer-scout-or-survey-scan/

As shown in the figures in the manuscript, there are extensive hallucinations in the generated images. Independent of the metrics, which are low to begin with, these hallucinations would not be high-fidelity enough for any clinical application.

We agree that only PSNR/SSIM are not enough to quantify the usefulness of the generated MRIs. This is why almost half of the paper is about validating the generated MRIs. Figure 5 in the main paper and Table I in the Appendix show the brain tmour detection in the generated MRIs almost matches that of the original MRIs, even though this tmour information was never explicitly used in training X-Diffusion on BRATS. When testing this BRATS-trained model on the healthy brain MRIs of IXI dataset the predicted tumor is in 9.9% of the real healthy brains and in 11.3% of the generated brain MRIs (sec 6.4), which means the X-Diffusion model is mostly not hallucinating tumors when there is no tumor in the sample. On the other hand for the full body MRIs the spine metrics (Figure XIV and Figure XIX) and the body fat metrics ( figure IX) all further validate these meta information about the generated MRIs, without ever being used in training X-Diffusion model.

Thank you for adding the new analysis regarding cortical volumes and the spinal curvature. These results make the paper stronger.

We appreciate the recognition by the reviewer.

More philosophically speaking, I fail to understand how the authors see what they propose used in practice, which is what they state they aim for. In their scenario, patients would come to the imaging center, have 2D slices acquired and pseudo-3D volumes reconstructed, and then what? Would they go home and maybe be called to come back and have a normal exam later? If so in what case? This does not seem cost-efficient nor practical.

Currently, our generated “pseudo MRIs” are not a substitute for actual MRI scans as we described briefly on page 2 “It is important to note that the generated MRIs are not clinical replacements for true MRIs yet, but could provide a quick, affordable, and informative “pseudo-MRI" before conducting a full MRI examination.” So we envision in the future this can be used in an active setup, while the patient is conducting a scan and the model reconstructs the MRI and informs the technician where to scan next, reducing redundancy in the scanning process, and hence reducing the time in conducting the MRI scan, reducing the cost of running MRI scans, and potentially saving countless lives. Moreover, in case of corrupted scans which regularly happen in practice, our model could act to generate pseudo-MRIs in the scenario if only one or few slices are usable. We hope that our X-Diffusion (especially if published at ICLR 2025) will act as a first step towards this future we envision. We added more discussions in updated Appendix Section D.1 and D2.

The related work section should cover super-resolution approaches. Also, the Full-Body MRI Analysis paragraph is very generic and pretty empty. See for instance Tunariu et al., British Journal of Radiology, 2020 on the use of whole-body MRI in clinical practice and Küstner et al., Radiology: Artificial Intelligence, 2020 for the automatic analysis methods related to these applications.

We apologize for the minor issue in the related works section. We have modified the related works section to be more comprehensive and include the mentioned works and is now integrated in the paper PDF.

Some references are too generic or do not correspond to the statement they are supposed to comfort, e.g. Tran et al., 2015; Sohl-Dickstein et al., 2015 or Kawar et al., 2022 in the second paragraph of the introduction.

We apologize for the minor issues in the related works section and introduction references. They are all now fixed in the updated PDF .

Questions:

Please fix the hyperref links, many references in the Appendix do not point to the good figure, this makes the paper even more difficult to follow.

All figures references are now fixed. Please see the updated PDF paper.

We thank the reviewer for their insightful comments and relevant questions, which we tried to address to the best of our ability. We updated the paper PDF with section D on Additional Discussions in supplementary material for the rebuttal.

The experiments are confusing. It is as if the authors tried all they could think of and let the readers do what they can with the results.

We kindly disagree with the statement as other reviewers ( reviewer Haqk and reviewer F5BK) find the paper “ is written in a very clear, concise, and comprehensive way.” and “ problem, motivation, and contributions are clearly stated”. However, we took into account your comment and we added a new subsection in the updated Appendix Section D.3 that explains the evaluation protocol further.

A major point that is often unclear in the experiments/results is what slice(s) was/were used as input. If the authors show in Figure VI that it has an impact on the results, it is not specified in other sections, e.g. 5.1, 5.2, 5.3, 5.4.

The test slices are randomly sampled from the entire datasets and fixed throughout the experiments for all the methods including our X-Diffusion and the baselines shown in Table I, etc. as explained in section 4.3 of the paper. We add a detailed subsection about the evaluation slice choice in sec D3 in the appendix

The way the authors split the BraTS dataset is not clear. On the one hand, they state that the dataset includes 5,880 MRI scans from 1,470 patients (roughly four MRI sequences per patient) and that they split it into Train (n=4704), Validation (n=588), and Test (n=588) sets, without mentioning if it is done at the subject level or not, which could mean data leakage. On the other hand, they say that they just use the FLAIR sequence, so then the numbers do not add up. Please clarify.

We used the standard test split of BRATS evaluation following previous works. The statement about the 5,880 MRI scans is a description of the dataset, not our training samples, we use the FLAIR sequence. We apologize for any confusion caused by this.

When difference maps are displayed there is no scale, on top of the colormap poorly chosen, so they are impossible to analyze.

We added the bar color maps to Figure 4 and updated the PDF.

No estimate of variation is provided in the tables.

In Figure 7 we showed the standard deviation on the plot which depicts the same results in Table 1 (very few variations anyway). However, as requested, we added STD on the main Table I, and added it in the Appendix as Table IX in Section D.8.

The evaluation by medical experts, though valuable, confirms that images generated are realistic but not that they show what they are supposed to, i.e. an image could be realistic but not correspond to the reality of a patient.

We agree with the reviewer that a proper clinical study that actively relates the accuracy of the generated MRIs with the actual clinical diagnosis of the patients would be ideal. However, we believe it is outside of the scope of this work and can be indeed a perfect further work. We add Sec D.2 in the Appendix to detail possible clinical study procedures to be followed to evaluate the clinical validity of X-diffusion. We intend to do the clinical medical study soon. Please see below the summary of the study objectives :

Objective: To see if recent generative AI technologies for MRIs (X-Diffusion) are relevant to knee diagnosis. The goal is to validate our pipeline for reconstructing full MRIs from one/few slices with high precision and evaluate its usefulness for clinical assessment of knees. Given expert grading of how abnormal a knee is, we will compare the score given for degeneration of knees between the two sets of knees with and without AI-generated knees.

Specific Aim. Grade how abnormal a knee is on an external set of 50 reconstructed MRIs from single slices or two slices each using X-Diffusion. Then, compare the score for knee degeneration between the annotated original set of 50 samples from humans and see how they are correlated. First, experts will grade degenerative knees on MRI. Then we test X-Diffusion on the same sampled grade. Given any single slice(s) from a degenerate knee, we generate synthetic knees. Finally, we compare the score given as grading for the degeneration of knees between the two sets of knees with and without AI-generated knees. Hypothesis. The trained X-Diffusion model for generating MRIs is capable of reconstructing unseen MRIs from one/a few slices in the clinic with high precision. The generated MRIs also maintain the diagnosis differentiability that makes them useful for physicians in the clinical setup.

I still fail to see how such approach would be used in practice. Patients come for an imaging exam with a specific indication that triggers a specific acquisition protocol, so I do not see how the synthetic images generated would inform "the technician where to scan next" (and please avoid going so far as "potentially saving countless lives"). For example, predicting the next sequences to acquire to refine the diagnosis would match closer the objective targeted by the authors.

We evaluate this proposed protocol of active selection of slices on brain MRIs as follows. We use the first slice to reconstruct with X-Diffusion the full volume , then we use the tumour segmentation network to detect the brain tumour at the full 1-slice generated MRI, and the center of the tumour prediction determines the next slice we use , and repeat until the budget of slices is reached, these actively selected slices are then used with the X-Diffusion model trained to reconstruct the full MRI and report this results on test set compared to the naive random baseline reported in Table 1. The results are shown on the following table A (for PSNR reconstruction quality) and Table B (for Dice score of tumour compared to ground truth labels). It shows that such active selection simulating what can actually be performed in an MRI scanning session in the clinic creates better reconstruction quality for the active selection and obtains a better tumour detection compared to the naive randomly selected slices.

Table A: PSNR Reconstruction Quality of using Active Selection strategy

Table B: Dice Score for Tumour Detection of using Active Selection strategy

The hallucinations are also a problem that would need to be further explored to assess the value of the presented work.

We addressed hallucinations briefly in Section 6.4 of the paper in which we showed when testing RATS-trained model (all BRATS samples have tumour) on the healthy brain MRIs of IXI dataset, the predicted tumor (using tumour segmentation network) is in 9.9% of the real healthy brains and in 11.3% of our generated brain MRIs. This means the X-Diffusion model is mostly not hallucinating tumors when there is no tumor in the sample despite accurately reconstructing the healthy brains with a PSNR of f 35.86 .

To test this hypothesis of dataset bias on the hallucination of X-Diffusion, we retrain X-Diffusion on BRATS but excluding all the slices that contain tumour (we have GT on BRATS) , and then test on the healthy brains of IXI. The resulting generated brains have a PSNR of 36.41 in reconstruction quality ( + 1.45 improvement in PSNR compared to training on the full dataset) , and a detection of tumour in 10.1 % of the generated samples in the test set ( -1.2% reduction in hallucinated tumour) . This gives an indication of why this hallucination occurs and how to mitigate it.

Nonetheless, as the focus of ICLR is more on the methods than on the applications, and that the methods proposed are quite novel, I will raise my score from 3 to 5.

We thank the reviewer for their considerations. We kindly ask the reviewer if all of their other concerns above were addressed properly and if they believe the paper is worthy of acceptance to reconsider raising the score to at least 6 which is still borderline acceptance but at least lowers the risk of rejecting the paper solely based on the clinical feasibility aspect of the paper (which is also addressed above).

We thank the reviewer for their insightful comments and relevant questions, which we tried to address to the best of our ability. We updated the paper PDF with section D on Additional Discussions in supplementary material.

1.Clinical relevance of the presented method:

We kindly agree with the reviewer that these generated “pseudo MRISs” are not a substitute for actual MRI scans as we described briefly in page 2 “It is important to note that the generated MRIs are not clinical replacements for true MRIs yet, but could provide a quick, affordable, and informative “pseudo-MRI" before conducting a full MRI examination.” We envision in the future this can be used in an active setup, while the patient is conducting a scan the model reconstructs the MRI and informs the technician where to scan next, reducing redundancy in the scanning process, and hence reducing the time in conducting the MRI scan, reducing the cost of running MRI scans, and potentially saving countless lives. We hope that our X-Diffusion (especially if published at ICLR 2025) will act as a first step towards this future we envision. We changed the tone a bit in the updated paper PDF to reflect your suggestion and added extended discussions about these points in Sections D.1 and D.2 of the updated appendix. While the experiments in Table I when the input slice is 31 are acting like a super-resolution problem with high-density input, and because our input aggregation layer is agnostic to the structure of the input, we indeed can perform SR from anisotropic to isotropic scans. We agree that the K-space setup is indeed a challenging case for our X-Diffusion, because the focus of the reconstruction from sparsity demands generalization and fewer details, while the K-space methods focus on high-frequency details.

2.Difficulty in assessing how much is owed to conditioning with the "right" slices and the capability of the model:

We sincerely thank the reviewer for their insightful comment regarding the evaluation challenges of X-Diffusion compared to existing methods. We fully agree that conditioning on high-quality 2D slices that are part of the ground truth (GT) introduces unique considerations in assessing our model's capabilities versus the influence of the conditioning data. This point was briefly touched in the introduction of the paper and Fig1 of the main paper which highlights the difference in the setup. However, to address this important point further, we have added a new subsection in D.3. In this subsection, we thoroughly discuss the evaluation challenges posed by the differences in input data and reconstruction objectives between X-Diffusion and traditional MRI reconstruction methods. We highlight how these differences impact the assessment of reconstruction quality and the comparative analysis with state-of-the-art (SOTA) baselines that were not originally designed to operate under the same conditions as X-Diffusion. We believe that this addition enhances the transparency of our work and provides a clearer understanding of how we have approached the evaluation of X-Diffusion, acknowledging the limitations and the steps taken to ensure a fair comparison.

3. Downstream Evaluation:

We perform the requested analysis on white matter and we report the results in updated appendix sec D.14.

4. Please share the SSIM for all tables and experiments; these values are often more relevant to PSNR values for reconstruction.

We added SSIM and LPIPS in Tables that have only PSNR and included standard deviation along with PSNR values for main Table I and included it in Section D.7 in Tables X,XI,XII and XIII in the updated Appendix.

5. A1 and A3 are almost identical. Consider merging or rewriting them to have one cohesive section.

We fixed this in the updated PDF.

How many tumors do we see if we only provide slices at the outer parts of the brain where tumors will not lie?

We run a similar study on BRATS in Table I in the Appendix.

For Figure 3, please clarify how many input slices are utilized from the target image. Considering the leftmost image is considered as input, am I right to assume that single-slice conditioning was utilized?

Yes, it is from the input image slice on the left. The tumour coincided with some of the inputs but not all, Table I in the Appendix details.

For Figure 6, which images were used to condition X-Diffusion

Because of space limitations, we are not able to include all the input images and how results and comparisons on the same figure. We show the input images now on a new figure XVI in appendix Sec D.6

For Figure 9, Figure II/II Appendix (and more) - how many slices were used to condition the model?

For Figure II/III of the appendix, we used our best X-Diffusion model conditioned on 31 slices at training to generate these volumes.

Given that I see the clinical utility still somewhat limited (even though the authors transparently discuss it) I would like to keep my initial rating of the paper.

We thank the reviewer for supporting this manuscript. While we acknowledged potential clinical future work that can be done to verify X-Doffusion feasibility on clinical setup on actual patients, we believe that would be suitable for a medical journal clinical study and is out of the scope of the current submission to ICLR conference which is about the method novelty, technical execution, and reconstructions results, where the reviewer agrees on (also reviewer fhnz just mentioned this observation). Hence we don’t see this clinical future work as a reason to lower the score of our paper which could result in a rejection given the current scores.

However, we simulate such a clinical setup proposed protocol of active selection proposed by reviewer fhnz. We use the first slice to reconstruct with X-Diffusion the full volume , then we use the tumour segmentation network to detect the brain tumour at the full 1-slice generated MRI, and the center of the tumour prediction determines the next slice we use , and repeat until the budget of slices is reached, these actively selected slices are then used with the X-Diffusion model trained to reconstruct the full MRI and report this results on test set compared to the naive random baseline reported in Table 1. The results are shown on the following table A (for PSNR reconstruction quality) and Table B (for Dice score of tumour compared to ground truth labels). It shows that such active selection simulating what can actually be performed in an MRI scanning session in the clinic creates better reconstruction quality for the active selection and obtains a better tumour detection compared to the naive randomly selected slices.

Table A: PSNR Reconstruction Quality of using Active Selection strategy

Table B: Dice Score for Tumour Detection of using Active Selection strategy

Based on the above, we kindly ask the reviewer if all of their other concerns were addressed properly and if they still believe the paper is worthy of acceptance to reconsider raising the score to 8 to lower the risk of rejecting the paper solely based on the clinical feasibility aspect of the paper.

We thank the reviewer for their insightful comments and relevant questions, which we tried to address to the best of our ability. We updated the paper PDF with section D on Additional Discussions for the rebuttal.

There are no measurements on how fast the X-diffusion is. It would be beneficial to include comparisons with previous approaches. E.g., Figure 3 shows that for good quality reconstruction, almost T=1000 steps are required.

Table VI in the original appendix page 26 shows compute cost and runtimes that are measured on a computer with a single GPU a6000, 48GB of RAM. The full sample would require slightly over 2 minutes. The 1000 steps are standard followed on LDM, Stable Diffusion, etc. Section D.11 and Figure XVII in the updated appendix include additional analysis of the time steps T, indicating we do not need 1000 steps to actually converge to good solutions.

The limitations discussed in Sections 6 and 7 appear to be strong, potentially limiting the practical value of the work.

While we agree that the diffusion model can hallucinate in some cases to account for missing information, we conduct a conformal prediction study to assess the confidence of the predictions that can be used by doctors and practitioners to assess the confidence in the diffusion output. We show results in the updated Appendix section D.13.

The authors dismiss the study of whether it is possible to reduce the number of T in diffusion without sacrificing the quality? And how would this reduction be specific to the medical imaging domain? (can simply substitute ideas of speeding up from text-to-image models?)

We conduct an ablation study on the number of time steps in Section D.12 and Figure D.12 in the updated appendix, which includes additional analysis. We also try to use the recent diffusion models like SD 2.1 and SD XL to speed up the diffusion process or increase performance and report results in Section D.10 and Table 12 of the updated Appendix.

Questions:

It would be interesting to see how X-Diffusion works with other types of augmentations, like blurring, changing colorspace etc. Intuitively, this should boost the ability to generalize OOD or at least improve the algorithm compared to using only SO3.

The SO(3) transformation is not only used as an augmentation but also generalizes the learning of volumes because it exposes the diffusion model to internal structure from different views and we aggregate the final prediction from all different directions. We try different augmentations: rotation, flip, and scale to improve the diffusion output at test time and report the results in Sec D.9 and Table XI.

Author’s do not discuss what are the benefits of using pretrained VAE from SD without any additional finetuning (i.e VAE is frozen) to specific MRI domain. Additional discussion would be great.

While we indeed use VAE from SD, it is trained to reconstruct any image from the latent space with high accuracy. More modern approaches have been proposed like in SD2.1 and SD XL. We added this in the discussion and X-Diffusion results on these modern approaches in Sec D.10 and Table 12 in the updated appendix.

Is it possible to integrate and check performance using existing text-to-3D models? Seems like utilizing directly 3D generative model provides more inductive bias, which should improve learning.

The use of modern Text-to-3D or image-to-3D that is based on Video Diffusion models or Multi-view diffusion models (eg.g SDV3D[a]) to reconstruct the MRI in the X-Diffusion pipeline is indeed a very promising direction to improve the performance. We expect this to work slightly better than the current X-Diffusion despite struggling with computing and memory cost issues, and we leave the dedicated investigation of this direction for future work.
[a] Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion, (ECCVC 2024).

Thank you for the rebuttal. I believe the new experiments mitigate my critiques but the limitations are still strong for practical implementation of the method. Reviewer fhnz writes along these lines too w.r.t. clinical validation (if only a bit too relentlessly to my taste). I thus keep my original score of the borderline accept.

We thank the reviewer for supporting this manuscript. While we acknowledged potential clinical future work that can be done to verify X-Doffusion feasibility on clinical setup on actual patients, we believe that would be suitable for a medical journal clinical study and is out of the scope of the current submission to ICLR conference which is about the method novelty, technical execution, and reconstructions results, where the reviewer agrees on (also reviewer fhnz just mentioned this observation). Hence we don’t see this clinical future work as a reason to lower the score of our paper which could result in a rejection given the current scores.

However, we simulate such a clinical setup proposed protocol of active selection proposed by reviewer fhnz. We use the first slice to reconstruct with X-Diffusion the full volume , then we use the tumour segmentation network to detect the brain tumour at the full 1-slice generated MRI, and the center of the tumour prediction determines the next slice we use , and repeat until the budget of slices is reached, these actively selected slices are then used with the X-Diffusion model trained to reconstruct the full MRI and report this results on test set compared to the naive random baseline reported in Table 1. The results are shown on the following table A (for PSNR reconstruction quality) and Table B (for Dice score of tumour compared to ground truth labels). It shows that such active selection simulating what can actually be performed in an MRI scanning session in the clinic creates better reconstruction quality for the active selection and obtains a better tumour detection compared to the naive randomly selected slices.

Table A: PSNR Reconstruction Quality of using Active Selection strategy

Table B: Dice Score for Tumour Detection of using Active Selection strategy

Based on the above, we kindly ask the reviewer if all of their other concerns were addressed properly and if they still believe the paper is worthy of acceptance to reconsider raising the score to 8 to lower the risk of rejecting the paper solely based on the clinical feasibility aspect of the paper.

Based on the discussions below, all of the reviewers initial concerns and questions are cleared during rebuttal except for the clinical aspect of the paper. While we acknowledged potential clinical future work that can be done to verify X-Doffusion feasibility on clinical setup on actual patients, we believe that would be suitable for a medical journal clinical study and is out of the scope of the current submission to ICLR conference which is about the method novelty, technical execution, and reconstructions results, where the reviewers agree on (also reviewer fhnz just mentioned this observation). Hence we don’t see this clinical future work as a reason to lower the score of our paper.

However, we simulate such a clinical setup proposed protocol of active selection proposed by reviewer fhnz. We use the first slice to reconstruct with X-Diffusion the full volume , then we use the tumour segmentation network to detect the brain tumour at the full 1-slice generated MRI, and the center of the tumour prediction determines the next slice we use , and repeat until the budget of slices is reached, these actively selected slices are then used with the X-Diffusion model trained to reconstruct the full MRI and report this results on test set compared to the naive random baseline reported in Table 1. The results are shown on the following table A (for PSNR reconstruction quality) and Table B (for Dice score of tumour compared to ground truth labels). It shows that such active selection simulating what can actually be performed in an MRI scanning session in the clinic creates better reconstruction quality for the active selection and obtains a better tumour detection compared to the naive randomly selected slices.

Table A: PSNR Reconstruction Quality of using Active Selection strategy

Table B: Dice Score for Tumour Detection of using Active Selection strategy