Leveraging Side Information with Deep Learning for Linear Inverse Problems: Applications to MR Image Reconstruction

We thank the reviewer for their constructive feedback and valuable suggestions, which helped us improve the manuscript. Below, we provide our responses to the raised concerns and questions.

Weaknesses:

• limitation of experiments in terms of the methods that have been compared

In fact, we compared our method against E2E-VarNet in the submitted version, as shown in Table 1. However, we did not include E2E-VarNet in the figures, as its performance is significantly worse than the methods utilizing side information. Based on your feedback, we have now added E2E-VarNet reconstructions to the figures as well.

• Imposing additional constraints is not a new topic.

Thank you for your feedback. Besides this work, we have already referenced more than ten prior studies utilizing side information. However, we missed this particular work and have now mentioned it in our introduction.

• Since it is a model based recon paper, authors need to add optimization based deep learning methods such as ADMM-net, ISTA-net, in the related work

In the related work section, we discussed only MR image reconstruction methods that utilize side information. The methods you mentioned are already covered in the introduction, where we provide a broader explanation of MR image reconstruction. We have changed the title of the related work section to Related Work: Side Information in MR Image Reconstruction to avoid confusion.

• Authors didn't provide a network structure or whole framework of this work.

We have added a figure (Fig. 1 on page 2) illustrating the network structure and how the side information is incorporated into the reconstruction problem using TGVN.

Questions:

1. From the figure 3 and 12, DMSI provides more details of the tissue and close to the ground truth.

Thank you pointing out this issue. A common challenge in end-to-end reconstructions compared to generative approaches such as DMSI is a residual smoothing of fine image features or background textures. To enhance the subjective perception of sharpness in images, known as acutance in photography, low levels of complex-valued Gaussian noise were added back to the reconstructed TGVN output in a new figure in Fig. 16 in Appendix D. We illustrated how this procedure easily changes the acutance. Note that noise levels in DMSI are substantially higher than in ground-truth, which may give the appearance of sharpness but can also obscure key structures, including pathologies.

2. Please indicate the box of the zoom in region in all of the figures.

We have indicated the box for the zoomed-in region as suggested.

3. Conduct a doctor study for evaluating the performance of these compared results should strengthen robustness of the study.

We are certainly interested in validating our results in prospective physician graded studies. However, there is not sufficient time to perform such studies appropriately in this revision.

4. In equation 3 and 4, author didn't explain what is \phi.

On line 178 on page 4 of the original submission, and on line 190 on page 4 of the revised version, $\Phi$ is explained as a neural network. If you mean what the exact architecture is, then it is explained in Appendix B, Training and Evaluation Details.

5. H(s, \gamma) is assumed to be learnable, and s is fixed variable, how does author choose s? Why not updating s iteratively with s^t and x^t alternatively?

It is explained, e.g., on line 258 of the original and revised submission, that $\mathbf{s}$ refers to the side information. The task we are working on involves reconstruction with side information, which refers to utilizing side information to reconstruct the main information. This is why $\mathbf{s}$ is not updated. A related problem is joint multi-contrast reconstruction, where multiple contrasts are reconstructed jointly; however, this is outside the scope of the current work.

Dear Reviewer TG2Y,

As the extension period for discussion nears its end, we kindly remind you that we have addressed all your comments and questions in the revised manuscript and provided additional clarifications where needed.

We would greatly appreciate it if you could revisit your evaluations in light of our updates. Please let us know if there is anything further we can clarify before the deadline.

Thank you again for your valuable feedback.

Best regards,

The Authors

We thank the reviewer for their constructive feedback and valuable suggestions, which helped us improve the manuscript. Below, we provide our responses to the raised concerns and questions.

Weaknesses:

• Clarity

Based on your feedback, we have expanded Sec. 4 explaining our novel work. We have also added a figure on page 2 (Fig. 1) to explain our architecture along with the general framework.

• Type of side information

In the original manuscript, as well as the revised version, we have included a brief discussion of complementary information at the beginning of Sec. 4. As long as the side information $\mathbf{s}$ is conditionally dependent on the image being sought ($\mathbf{x}$) given the under-sampled measurements $\mathbf{k}$, we can exploit this side information in resolving ambiguities. In the literature, we have observed that multi-contrast MRI is sometimes referred to as multi-modal MRI. However, functional MRI, diffusion MRI, and MR spectroscopy are occasionally also referred to as multi-modal MRI. Our work is generic enough to incorporate either, but the datasets we use contain only multi-contrast data. To avoid confusion, we have opted to use the term multi-contrast instead of multi-modal. Additionally, joint multi-contrast MRI reconstruction is sometimes referred to as multi-modal MRI reconstruction. While our work can be adapted to perform joint reconstruction, it is not the focus of this manuscript.

• Compared methods

Based on your feedback, we have discussed these single-contrast methods in the introduction section. However, since the main information is accelerated at levels up to $20\times$, the use of side information is crucial in our experiments. All of the suggested methods, on the other hand, are designed for single-contrast reconstruction and perform similarly to E2E-VarNet, which does not exploit side information. For clarification, we want to note that the baseline MTrans is transformer-based, and the baseline DMSI is diffusion-based, both utilizing side information.

• Generalization evaluation on different datasets

This is an interesting point that we have not explored yet. Considering the time required for each experiment, (up to two weeks for each knee experiment), we have not been able to test this approach in time of the current submission. It will guide our future research direction.

• Under-sampling mask

As Cartesian under-sampling is by far the most frequently used in practice, we have focused on the Cartesian case. We fully expect our method to work with other sampling trajectories; however, nontrivial modifications would be required, which would take longer than the time available for this revision.

• Experimental results 1

Thank you for this insightful question. Exp. 3 was performed using the M4Raw dataset, which was acquired at $0.3$T with $4$ coils and has a small matrix size and low SNR. In contrast, Exp. 1 and 2 were performed using the fastMRI dataset, acquired at $1.5$T and $3$T with $15$ coils and a much larger matrix size and higher SNR. Considering these factors, achieving high acceleration without side information is significantly more challenging in Exp. 3, compared to the first two experiments. Furthermore, E2E-VarNet is the only method that does not use the side information, which demonstrates how important side information becomes in challenging cases.

• Experimental results 2

We have added PSNR and NRMSE scatter plots in the ablation studies. Furthermore, we have significantly expanded our ablation studies and Appendices.

Dear reviewer,

Previously, we interpreted your point differently. For E2E-VarNet and TGVN of the same capacity, which were trained to reconstruct 20x under-sampled main information (knee PDFS), we performed inference on the M4Raw brain FLAIR test split. The statistics are provided in the table below:

While side information appears to enhance generalization by significantly improving the metrics, we would like to emphasize that the image quality degrades substantially (cf. Table 1 in the manuscript) and the reconstructions remain far from being diagnostic and are not acceptable in this generalization experiment, even with side information.

We thank the reviewer for their constructive feedback and valuable suggestions, which helped us improve the manuscript. Below, we provide our responses to the raised concerns and questions.

Weaknesses:

• There exist already many attempts to incorporate side information.

While there have been some attempts to incorporate side information, to the best of our knowledge, no previous approach has investigated an ambiguous space framework with end-to-end trainable variational networks. Furthermore, we have achieved statistically significant improvements over a variety of baselines utilizing side information. In fact, we have not seen acceleration levels as high as those demonstrated in this work using other side information approaches.

• The paper includes a lot of general information in the paper but a quite short explanation of the novel work.

Based on your suggestion, we shortened the literature review and extended the explanation of the novel work in Sec. 4. Furthermore, we have illustrated the general framework, a TGVN cascade, and a full TGVN on page 2 (Fig. 1).

• In this paper, four research questions are mentioned but only two are discussed whereby also the other ones, especially Q3, would be also considered as important as it is part of the main contribution of this paper.

Due to page limitations, two research questions were answered in the main text while the other two were discussed in the Appendix. Essentially, our experiments demonstrated that (I) The use of side information significantly improves the reconstruction quality. (II) TGVN is more effective in leveraging side information than the relevant DL baselines of the same capacity which also utilize side information. (III) Projection onto the ambiguous space improves the reconstruction quality significantly. (IV) TGVN is robust to moderate under-sampling or small misregistration of the side information.

• The authors advertise that different types of side information can be included but not exactly how. Furthermore, it is mentioned that this is part of future work

In our proposed architecture, the side information is utilized in the $H$ block. Hence, different types of side information can be incorporated by modifying the $H$ block. For instance, if the side information comes from other imaging modalities such as CT, ultrasound, or PET, the same architecture can be used. For non-image data, such as textual or tabular information, embeddings obtained from a pre-trained, frozen encoder can be used to train a decoder end-to-end, along with all the other parameters of TGVN. In this case, the $H$ block will function as a decoder that maps features to image space.

• The generalizability is not shown across different anatomies, as stated. The brain and knee dataset are handled separately.

Our intention was not to claim that one network can be trained and used for all anatomies. Rather, we intended to convey that the approach can work across various anatomies. Based on your feedback, we have changed the wording from generalizability to robustness.

• A lot of stuff is described generally but the key points are missing.

We have addressed this general concern to the best of our ability, by shortening the introductory text and expanding the descriptions of the methods, results, and Appendices.

Questions:

• Why should we use this approach and not another one to incorporate side information?

With our method, we have been able to achieve acceleration levels substantially higher than what has been feasible using other approaches. Furthermore, our approach is designed to minimize the risk of hallucinations while effectively leveraging side information to maximize the desired similarity metric between the reconstructions and reference images. The main experiments we presented demonstrate the effectiveness of TGVN compared to relevant deep learning baselines of the same capacity (which also utilize side information).

• How does the threshold effect the results?

As the threshold gets smaller, the ambiguous space converges to the null space. Since the null space is trivial for acceleration rates lower than the number of coils in parallel imaging, this means the optimal threshold will likely be large. In our experiments, we in fact learned the threshold as we have used the approximate projection explained in Sec. 4.2 in the original as well as in the revised submission.

• Which projection was chosen?

The approximate projection was chosen due to computational complexity of using the exact projection. This was noted in Appendix B of the original submission as well as of the revised version.

• How is the module learned?

The module is learned end-to-end with all the other parameters of TGVN, as we explained in Sec. 4.2. Specifically, $\Omega$ is selected to maximize the multi-scale structural similarity (MS-SSIM) between the reconstructions and the reference images.

Dear Reviewer hAQd,

As the extension period for discussion nears its end, we kindly remind you that we have addressed all your comments and questions in the revised manuscript and provided additional clarifications where needed.

We would greatly appreciate it if you could revisit your evaluations in light of our updates. Please let us know if there is anything further we can clarify before the deadline.

Thank you again for your valuable feedback.

Best regards,

The Authors

We thank the reviewer for their constructive feedback and valuable suggestions, which helped us improve the manuscript. Below, we provide our responses to the raised concerns and questions.

Weaknesses:

1. The presentation and story telling is not clear and there isn't any figure illustration of the problem/method/novelty, which makes the paper very difficult to follow, especially the null-space and theory part.

We have added a figure illustrating the general framework, a TGVN cascade, and a full TGVN on page 2 (Fig. 1).

2. Using multi-contrast information for joint reconstruction has been a popular topic, but the authors didn't compare with any-of-them.

We acknowledge that using multi-contrast information for joint reconstruction has been a popular topic. In the original submission, we compared our method against three baselines that also utilize side information. These baselines can also be applied in a joint reconstruction setting; however, their performance deteriorates in this context. Specifically, the end-to-end baselines process both contrasts and output two reconstructions. In joint reconstruction, the network parameters are optimized to minimize the sum of losses from both contrast reconstructions and their respective references. In contrast, in reconstruction with side information, the parameters are optimized to minimize the loss between only the target contrast and the reference image. Consequently, the baselines we provided are more likely to be stronger than the same baselines trained in a joint reconstruction setting in terms of target contrast reconstruction performance. Besides all this, the diffusion baseline (DMSI) is applied in the joint reconstruction setting, as the side information in the knee experiments is $2\times$ or $3\times$ under-sampled while the score models are trained with fully sampled images.

3. Following the previous weaknes, instead of using Null-space, one way to extract information from side contrast is by learning an representation

In the original submission, as well as in the revised version, ablation study A.1 in Appendix investigates the effect of projection. In this experiment, we utilized the side information without applying the ambiguous space projection by learning representations and incorporating them into the E2E-VarNet architecture. Hence, we have presented this result in the original manuscript and significantly expanded them in the revised version to make this clearer.

4. Instead of using aliased side information ($3\times$ undersampled image from side contrast), it makes sense to use a reconstructed version of other contrast, like using E2E, have you done this experiments?

We have done this experiment but did not present it in the original submission. Following your suggestions, we have appended it to Appendix A.2.2. While reconstructed side information might seem intuitive, we observed no statistically significant improvements in reconstruction metrics compared to a single-stage approach that directly utilizes the under-sampled side information.

5. ~20x undersmapling doesn't make sense at all, in practice, or in fastMRI dataset, only a few lines are sampled beside the ARC region, how do you expect recovering any details? especially for pathologies.

Without any side information, there is no hope of recovering details at these acceleration levels. However, the side information in our experiments is only moderately under-sampled (at most $3\times$), providing significant anatomical and potentially pathological detail, as demonstrated by the clear depictions of meniscal tears in our results. Additionally, in our experiments, we used smaller ARC regions than usual, as explained in Sec. 5 in our manuscript. We are ready to make the code publicly available. Since the datasets are already accessible, our results can be reproduced easily.

Questions:

1. Do you use paired data for training - like the main and side contrasts from the same object/ same anatomy? how do you do the alignment?

We used the fastMRI and M4Raw datasets, which are paired but occasionally contain small misregistrations. Additionally, we performed an ablation study (Appendix A.2.1) to investigate the effect of deliberately introducing misregistrations. The results indicated that small misregistrations can be tolerated if training also introduces such random misregistrations, and reconstruction with misregistered side information still performs significantly better than reconstruction without any side information.

2. I need more ablation study to understand how the side information contributes.

Based on your suggestion, we have added a comparison between two-stage and single-stage (proposed) reconstruction, presented in Appendix A.2.2. Furthermore, we significantly expanded the other ablation studies to provide a more comprehensive understanding of the impact of side information on reconstruction performance.