We thank the reviewer for the insightful comments, and we address the questions below:

(W1). The novelty appears to be somewhat incremental, as the method may seem to be a direct application of Attentive Mask CLIP proposed by Yang et al. (2023).

(Q3) Please elucidate the distinctions between the proposed method and Attentive Mask CLIP proposed by Yang et al. (2023).

We re-elaborate more here on the novelty of our method, which is quite different from Attentive Mask CLIP (ACLIP).

First, we target an unified VL framework capable of handling various medical modalities (e.g. 2D X-rays, 3D CT), and Figure 2 shows that merging representations of different medical image modalities into a language-guided common semantic space is challenging. Naively using language guidance and VL contrastive learning (Figure 2a) and simply unifying in one model (Figure 2b) cannot achieve the goal of merging. To this end, we introduce several designs for UniMedI, and all of the designs aim at better merging 2D and 3D medical images into a language-guided common semantic space. To best of our knowledge, we are the first one merging 2D and 3D medical images into a unified VL framework (we also provide a table to summarize the difference between our work and existing methods in this field at the end of this answer). Particularly, ACLIP cannot handle multi-modal images, and it can only be used for single modal images. So, ACLIP cannot be directly applied to our problem.

The first design is incorporating the guidance from report via VL contrastive learning. Generally, the $[CLS]$ token in the image side is optimized with representation of the paired report. And this image $[CLS]$ token is used in the following parts of UniMedI to ensure that the integrating is guided by language. We also have a novel motivating observation here for our problem (shown in Figure 1 and introduction). We see that despite big differences, medical images from various modalities share a common semantic latent space, which captures the underlying features of an individual's health status, and such status are reflected in medical reports via language. This motivating observation is insightful in our problem and has not been mentioned in previous works as far as we know.

The second design is our pipeline. Since no paired 2D and 3D image data exists, so we first select informative 2D slice from 3D volume as the bridge via the proposed attentive selection strategy. And then 2D slices, 2D X-ray and 3D volume are sent to the vision encoder together, enabling better merging. ACLIP does not have this pipeline and the attentive selection strategy, since bridging multimodal images is not handled in ACLIP. We have updated our paper by including a subsfigure in Figure 3 to show this pipeline.

The third design is the auxiliary task of mask and recovery implemented by self-distillation. Since 2D and 3D tokens are sent to the visual encoder at the same time and a large proportion of tokens are masked, this auxiliary task can help to enhance interactions among 2D and 3D tokens, facilitating merging representations. The mask in ACLIP is used for removing redundant information, while the auxiliary task in UniMedI is enhancing dimensional interactions.

Besides, the vision encoder has separate tokenizer for 2D and 3D data and a shared backbone, which is also designed for representation integration, and not used by ACLIP.

In summary, the problem we addressed in this paper is quite novel, and all designs in UniMedI aim at better solving the targeted problem. Maybe some losses are similar with other methods, but they are introduced with different purpose here, and thus brings different effects. And there are lots of difference compared with ACLIP, as we listed above.

We thank the reviewer for the detailed feedback on our paper!

(W1) The framework appears to be a multi-stage learning process rather than a true end-to-end 2D/3D multi-modal learning framework. It seems to involve selecting high-attention slices from 3D images and then training a 2D image-text encoder. If this understanding is correct, it is not as useful as a unified 2D/3D multi-modal learning framework.

We apologize for causing misunderstanding here. UniMedI is not a multi-stage learning process and it takes 2D slices and 3D images at the same time. Specifically, after extracting 2D slices, the 3D volume and 2D slices are sent to the vision encoder as inputs at the same time. With separate tokenizers, 2D and 3D tokens exists in one pass. In this way, we can use the mask and recovery task to enhance dimensional interactions. We do not have separate training stage for attentive slice selection and the whole model is updated with backprogration once each batch. To clarify more clearly, we provide explanations of UniMedI's pipeline below. In our revised paper, we have included a subfigure in Figure 3 to illustrate, and also adjusted corresponding descriptions.

Pipeline of UniMedI

As we have described in the first paragraph of Section 3, to overcome the challenges that no paired 2D and 3D image data exists, during the training process, UniMedI first extracts a portion of informative 2D slices from a 3D volume according to language guidance via the proposed attentive slice selection strategy (shown in Figure 4). This creates 2D-3D data pairs to bridge the disparity between the two modalities. Then, selected 2D slices and 2D X-rays, together with original 3D data, are sent to a unified vision encoder (shown in Figure 3) to obtain image representations.

Overall, UniMedI is a vision-language pre-training framework. The paired report of each X-ray/CT is encoded by the text encoder, and VL contrastive learning (shown in Figure 3) are employed to obtain a language-guided common semantic space for both 2D and 3D medical images.

(W2) While the use of t-SNE for problem definition is interesting, the paper lacks a comparative t-SNE plot in the results section to illustrate the impact of 2D and 3D co-learning. This could have provided clearer visual evidence of the model's effectiveness.

We would like to clarify that the impact of 2D and 3D co-learning is already shown in Figure 2. Generally, UniMedI target the goal of mapping data from various medical image modalities into the shared semantic space, which is guided by language in reports, and Figure 2 (visualizations of representations) is used to show that this goal is hard to achieve. We provide the information in Figure 2 in the following table. In summary, Figure 2 shows that naively using language guidance (2a) and simply unifying in one model (2b) cannot achieve the our challenging goal. In contrast, UniMedI, which involves 2D and 3D co-learning, can better integrate different medical image modalities.

Moreover, per suggestion from Reviewer zUjR, we have added visualization of data representations of two different classes (i.e., pneumonia and cardiomegaly) in Figure 6 in Appendix C.4. As can be seen, even data coming from different classes, UniMedI can bring together representations for medical data of different modalities, while not making features of all classes similar, well preserving semantic information from reports in the representation space. This nicely demonstrates the effectiveness of UniMedI on integrating medical multi-modal images into a language-guided common semantic space.

Most listed models [1, 2, 4, 5] are trained via self-supervised learning (SSL). Comparing SSL models with the model by vision-language (VL) pre-training is not very reasonable, since VL pre-training is dealing with some multi-modality problem, which is quite different with the SSL problem. Moreover, UniMedI also handles multimodal medical images at the same time. Therefore, the comparison might not be as fair as what we listed in the paper. Nevertheless, we provide compared results in the below table (CheXpert$^\ast$ denoting full backbone fine-tuning and CheXpert$^\diamond$ representing linear classification, only fine-tuning the classification head). Here, we additionally include some latest SOTA SSL methods in the general domain, further demonstrating the superiority of our method.

It is evident that UniMedI outperforms all other methods, even without fine-tuning for the evaluation on CheXpert. Other methods not included in the above table are specially designed for some downstream tasks. SAM [2] is for landmark detection and Lesion matching with designed backbone, and it use separate models for 2D and 3D data, respectively. Swin UNETR [4] is designed for semantic segmentation of brain tumors in MRI Images, which cannot be applied to the same downstream tasks as our method. KAD [3] is designed for classification tasks, incorporating external knowledge and a classification head related to the disease. Med3D [6] is for transfer learning with a coarse-to-fine structure. Comparing with these specially designed methods is not a fair setting. Besides, all these designs for specific tasks are orthogonal with our methods, and can be involved in our framework for further performance improvement. Unifying all these designs can be our future work.

(Q4) The authors did not provide sufficient reference for the baseline methods presented in Table 1.

Thanks for pointing out. We have added the references in the updated version.

[1] Ma, DongAo, Mohammad Reza Hosseinzadeh Taher, Jiaxuan Pang, Nahid UI Islam, Fatemeh Haghighi, Michael B. Gotway, and Jianming Liang. "Benchmarking and boosting transformers for medical image classification." In MICCAI Workshop on Domain Adaptation and Representation Transfer, pp. 12-22. Cham: Springer Nature Switzerland, 2022.

[2] Yan, Ke, Jinzheng Cai, Dakai Jin, Shun Miao, Dazhou Guo, Adam P. Harrison, Youbao Tang, Jing Xiao, Jingjing Lu, and Le Lu. "SAM: Self-supervised learning of pixel-wise anatomical embeddings in radiological images." IEEE Transactions on Medical Imaging 41, no. 10 (2022): 2658-2669.

[3] Zhang, Xiaoman, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. "Knowledge-enhanced visual-language pre-training on chest radiology images." Nature Communications 14, no. 1 (2023): 4542.

[4] Tang, Yucheng, Dong Yang, Wenqi Li, Holger R. Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. "Self-supervised pre-training of swin transformers for 3d medical image analysis." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20730-20740. 2022.

[5] Haghighi, Fatemeh, Mohammad Reza Hosseinzadeh Taher, Michael B. Gotway, and Jianming Liang. "DiRA: Discriminative, restorative, and adversarial learning for self-supervised medical image analysis." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20824-20834. 2022.

[6] Chen, Sihong, Kai Ma, and Yefeng Zheng. "Med3d: Transfer learning for 3d medical image analysis." arXiv preprint arXiv:1904.00625 (2019).

[7] Chen X, Fan H, Girshick R, et al. Improved baselines with momentum contrastive learning[J]. arXiv preprint arXiv:2003.04297, 2020.

[8] Chen X, Xie S, He K. An empirical study of training self-supervised vision transformers. In 2021 IEEE[C]//CVF International Conference on Computer Vision (ICCV). 9620-9629.

[9] Caron M, Touvron H, Misra I, et al. Emerging properties in self-supervised vision transformers[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 9650-9660.

Thanks for your constructive comments on our work. We appreciate the opportunity and have addressed all your concerns point by point as follow, and also revised our manuscript accordingly.

(W1) The motivation of the paper, bridging features in 2D and 3D, is not validated (see Q1).

(Q1) From Figure 2, I fail to observe $c$ is particularly better than $b$ in terms of feature quality. I understood that the authors try to present the features of pneumonia get closer between 2D and 3D images, but this needs features of another class (e.g., nodule) as a comparison. It is possible that the proposed UniMedI makes features of all classes similar. I think the discrepancy between 2D and 3D modalities could be much larger than that among classes.

First, in the revised version, we have highlighted the differences among each subfigures of Figure 2 as much as possible. In 2(b), overall, the 2D X-ray representations are distributed in the upper part of the figure, while the 3D CT representations are distributed in the lower part of the figure. In 2$c$, representations of 2D and 3D images are generally merged together and closer compared with 2(b). In summary, Figure 2 shows that naively using language guidance (2a) and simply unifying in one model (2b) cannot achieve the our challenging goal, which is using text information in reports as the guidance to map data from various medical image modalities into the shared semantic space. In contrast, UniMedI can better integrate different medical image modalities.

Also, to further address your concerns, we have visualized data representations of two different classes (i.e., pneumonia and cardiomegaly) in Figure 6 in Appendix C.4. As can be seen, even data coming from different classes, UniMedI can bring together representations for medical data of different modalities, while not making features of all classes similar, well preserving semantic information from reports in the representation space.

We really agree with the reviewer on the viewpoint that discrepancy between 2D and 3D modalities could be much larger than that among classes. Therefore, Figure 6 in Appendix address this point and nicely demonstrate the effectiveness of UniMedI on integrating medical multi-modal images into a language-guided common semantic space, since Figure 6 shows semantic meaning is preserved while modalities are merged in the representation space, as we expected.

(W2) The details of the method are confusing (see Q2).

(Q2) The illustration of Figure 3 and Figure 4 is unclear to me. What are the light orange boxes used for? What is the operation after T_3D and T_2D? The main text in the method section is not corresponding to Figure 3 or Figure 4. It is unclear how Equations (1-2) were integrated into the framework.

We appreciate the reviewer's suggestion to depict our method more clearly. Firstly, we would like to elaborate more on the pipeline of UniMedI to make the whole process clear. Based on the pipeline, we further explain contents in Figure 3 and Figure 4 clearly. After that, we address your question point-to-point in order to clarify any confusion. Lastly, we have adjusted Figures 3 and 4 without changing the content in the updated version of our paper, including clearer annotations and some accompanying explanations. We believe these enhancements will greatly improve the clarity and comprehensibility of our framework.

Pipeline of UniMedI

As we have described in the first paragraph of Section 3, to overcome the challenges that no paired 2D and 3D image data exists, during the training process, UniMedI first extracts a portion of informative 2D slices from a 3D volume according to language guidance via the proposed attentive slice selection strategy (shown in Figure 4). This creates 2D-3D data pairs to bridge the disparity between the two modalities. Then, selected 2D slices and 2D X-rays, together with original 3D data, are sent to a unified vision encoder (shown in Figure 3) to obtain image representations.

Overall, UniMedI is a vision-language pre-training framework. The paired report of each X-ray/CT is encoded by the text encoder, and VL contrastive learning (shown in Figure 3) are employed to obtain a language-guided common semantic space for both 2D and 3D medical images. To make this pipeline clear, we add a subfigure to Figure 3 for illustration.

We thank the reviewer for providing insightful comments on our paper. We provide clarifications to the concerns below:

(W1) The description of the framework is not clear. It would be better if you could mark the (1)-(4) in Figure 3.

Thank you for your valuable suggestion. We apologize if the description of our framework was unclear in the initial submission. We have revised Figure 3 in our paper to give a clearer presentation without changing the content. The left of Figure 3 demonstrates the overall pipeline of UniMedI, and the right part of Figure 3 and Figure 4 present details on the modules used in the pipeline. Also, we have marked (1)-(4) in Figure 3 upon your request. To further address your concerns, we would like to briefly review UniMedI here.

Pipeline of UniMedI

As we have described in the first paragraph of Section 3, to overcome the challenges that no paired 2D and 3D image data exists, during the training process, UniMedI first extracts a portion of informative 2D slices from a 3D volume according to language guidance via the proposed attentive slice selection strategy (shown in Figure 4). This creates 2D-3D data pairs to bridge the disparity between the two modalities. Then, selected 2D slices and 2D X-rays, together with original 3D data, are sent to a unified vision encoder (shown in Figure 3) to obtain image representations.

Overall, UniMedI is a vision-language pre-training framework. The paired report of each X-ray/CT is encoded by the text encoder, and VL contrastive learning (shown in Figure 3) are employed to obtain a language-guided common semantic space for both 2D and 3D medical images.

Key module 1: the unified vision encoder (Figure 3)

For the vision encoder, first, 2D data (selected 2D slices and 2D X-rays) and 3D data (original 3D volumes) are processed by 2D tokenizer $T_{2D}$ and 3D tokenizer $T_{3D}$, respectively. Seperate tokenizers encode input data from different modalities into a series of tokens. 3D CT tokens and corresponding 2D slices tokens are concatenated together for further procesing by a shared backbone. Note here input contains both 2D and 3D data, so we have 2D and 3D tokens at the same time. 2D X-rays tokens directly processing by a shared backbone.

Then, to enhance the cross-dimensional communication among 2D and 3D tokens, we adopt an auxiliary task, i.e., mask and recovery, and this task is implemented by the self-distillation method. Specifically, our visual encoder contains the teacher network $\overline{E}_v$ and the student network $E_v$, where $\overline{E}_v$ is updated by exponential moving averaged (EMA) over $E_v$. Tokens, once encoded by the tokenizer, are input into the teacher network. This process yields global and local embeddings and simultaneously outputs attention scores for each patch of the X-rays or CT scans, as outlined in Equation 1 in main text. Depending on the scores of each patch, we mask the less significant areas (those with lower scores) for the input into the student network. Since the student network only inputs important areas of the medical images, the resulting visual encoding is more closely matched with the semantics included in the report. Finally, loss of the self-distillation method is applied to both the global $[CLS]$ token (i.e., $L_{icl}$) and local patch tokens (i.e., $L_{pcl}$).

Key module 2: the attentive slices selection strategy (Figure 4)

The aim of attentive slice selection is selecting informative 2D slices from the 3D CT volume. The 3D CT volume is processed by the tokenizer $T_{3D}$ and the teacher network, and thus, attention scores for patches in the CT can be obtained. Then, the scores of all tokens within one slice are averaged to derive the score of the corresponding slice. This process is referred to as the Inter-Slice Average, as outlined in Equation 2 in main text. Slices with top $k$ score are selected as the connection between 2D and 3D data.

Note that these scores are attention weights between the $[CLS]$ token and other patch tokens, and the $[CLS]$ token is used for VL contrastive learning. Therefore, this slice selection process is guided by text information in the report, and 2D slices containing important information in the report can be selected accordingly.

(Q1) Different language of medical reports from 2D and 3D datasets:

Thank you for your question. Indeed, you are correct. Given the language differences in the radiology reports from the BIMCV and MIMIC-CXR datasets, we have opted to translate all Spanish reports from the BIMCV dataset into English.

(Q2) How to utilizes 2D visual encoder for 3D CT input:

Thank you for your valuable suggestion. We apologize if the description of our framework was unclear in the initial submission. For the vision encoder, first, 2D data (selected 2D slices and 2D X-rays) and 3D data (original 3D volumes) are processed by 2D tokenizer and 3D tokenizer, respectively. Separate tokenizers encode input data from different modalities into a series of tokens. These tokens are then fed into the ViT-B architecture. Thanks to the presence of the 3D tokenizer, our CT scans are converted into 512 tokens, ensuring a manageable input size for the visual encoder.