CrossModalNet: Multimodal Medical Segmentation with Guaranteed Cross-Modal Flow and Domain Adaptability

I recommend the authors withdraw their submission as it does not meet the necessary standards and may waste reviewers' time.

1. Insufficient Loss Function Analysis: The authors need to conduct a thorough analysis of the loss function within the nnUNet framework across multiple cross-modal datasets. Referencing studies like "nnu-net revisited: A call for rigorous validation in 3D medical image segmentation" and "Loss Odyssey in Medical Image Segmentation" would help validate that their approach results in genuine improvements.
2. Unoriginal Theoretical Proofs: The theoretical proofs (e.g. Eq.6 to Eq. 8,) lack novelty and logical, appear to replicate content from established sources such as "Pattern Recognition and Machine Learning" (PRML) or may have been generated using AI tools like GPT-4. This raises concerns about the originality and contribution of the theoretical aspects presented. The Complexity of CrossAttention is wierd to be a lemma.
3. Non-Innovative Loss Balancing for Multimodal Data: The approach to balancing the loss function for two modalities does not introduce a new concept and has been previously addressed in existing literature. The authors should clarify how their method offers a distinct advantage or improvement over existing techniques.
4. It wierd that the classifier present in the Fig. 1, since the task is segmentation. The term jaccard loss, graph mapping loss are not mentioned any where in the paper.
5. The Sec 2.1.4 is wierd, the deformable is usually used for medical registration task, maybe the authors just use the GPT in a wrong way by coping (CrossModalNet: exploiting quality preoperative images for multimodal image registration).

Minor Issues:

1. Missing Citations: The manuscript lacks numerous essential citations, which undermines the depth of the literature review and fails to acknowledge relevant prior work in the field.
2. Incorrect Citation Formatting: There is inconsistent use of citation commands, such as mixing \citep and \citet, leading to formatting errors and potential confusion for readers.
3. Misleading Network Structure Definitions: Providing definitions and proofs related to the network structure is misleading. Definitions and proofs should specifically address the unique aspects of the proposed network without conflating them with general architecture principles.
4. Inaccurate or Incorrect Citations: References cited on lines 216, 220, and 233 appear to be invalid or nonexistent, suggesting that some citations may have been improperly generated, possibly using GPT. These need to be verified for accuracy and legitimacy.

1. This paper mentioned their advantages in model generalization and robustness in multiple places, but did not provide any quantitative analysis to solidify this strength.

2. It is important to clarify that the problem setting in this paper utilized unpaired cross-modality data, i.e., the CT data and MR data come from different patient cohorts, and it is unknown whether one CT sample and one MR sample belong to the same patient or not.

Applying the proposed approach in scenarios like that could weaken the clinical significance of leveraging the complementary information embedded in multi-modality data. After all, different patients commonly have different health statuses. It could be hard to say whether the differences between any two patients' images are complementary to each other and how this complementary information (if any) could benefit the image understanding of either of them.

A more recommended application scenario in the medical field could be multi-modal brain tumor segmentation (e.g., the Brats 2018 dataset). Each patient case is paired and well-registered, containing 4 MRI volumes as inputs and 1 segmentation mask as ground truth. The clinical importance of leveraging complementary information among these MRI volumes has been clearly stated and emphasized in multiple literature.

Ref:

[1] S. Bakas, M. Reyes, A. Jakab, S. Bauer, M. Rempfler, A. Crimi, et al., "Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge", arXiv preprint arXiv:1811.02629 (2018)

[2] B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, et al. "The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)", IEEE Transactions on Medical Imaging 34(10), 1993-2024 (2015) DOI: 10.1109/TMI.2014.2377694

3. It needs to clarify how theoretical findings like generalization bounds and robustness analysis could assist the practical usage of medical applications (e.g., whole heart segmentation with the MM-WHS dataset), such as the performance improvements after using them, wise model selection according to the generalization bounds or better preprocessing steps guided by robustness analysis, etc.

4. No table is presented in this paper. The performance comparison in Fig. 2 did not provide the exact numerical result of each approach. Also, the text font size is too small.

The experimental validation is currently limited to a single dataset (MM-WHS). While this dataset is challenging and relevant to the task, further evaluation on other datasets is necessary to demonstrate the generalizability of the model's performance. For example, multi-class segmentation datasets like BraTS and ACDC can be used.

The authors proposed the Cross-Modal Information Flow (CMIF) metric, which measures the information exchange between modalities. However, the evaluation results on the proposed metrics are not included in the main paper.

The content of the paper can be further improved by having a discussion section by providing statistical insights.

Figure 1 and Figure 2 can be further improved. Captions are not helpful for a reader who is not in the context. Improving the architecture diagram of the proposed method to provide an overall view of the training process, along with relevant mathematical notations, would make it more informative and easier to understand. Figure 2 should be revised, it may be more effective as a table (include key performance metrics), with an additional figure to present qualitative results.

While the authors discussed the computational complexity of transformer blocks and other components, they did not explicitly discuss in the main paper that using transformers can lead to increased computational complexity compared to simpler convolutional architectures. It would be great if the authors could provide insights on this by comparing computational complexity with baseline models.

1、The paper is difficult to follow. The authors dedicate a large portion to deriving formulas with a lot of assumptions and time complexity, making it more like a scientific outreach rather than a research paper. Further, the paper does not clearly demonstrate how the derived theoretical gaurantee translates into an experimental performance gain. 2、The paper does not provide concrete implementation details of the proposed network design (U-shaped Parallel Feature Network, Cross Transformer Block and Deformable Operator), and the provided figure (i.e. Figure 1) does not clearly illustrate the connections between the visualized structure and the methods discussed in the text. These makes it challenging for readers to understand. 3、The experiments are insufficient （e.g. only single dataset MMWHS used for the experiment), which makes it difficult to substantiate the claimed advantages of the proposed method. 4、The comparisons are likely unfair; the methods used for comparison are all single-modal approaches. There is no multimodal approach comparison. Thus, the results fail to demonstrate the advantages highlighted in the paper. 5、The paper lacks ablation studies to support the claimed benefits of the adaptive loss balancing method. Including visualized training convergence curves and an analysis of how adaptive balancing contributes to training stability could enhance the credibility of this section.