Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words

We are grateful for your thorough review and constructive feedback, which have been very important in refining our manuscript.

• Novelty: We wish to emphasize the novel aspects of our work in comparison to the existing literature:
  • Robustness of ViT to Missing Channels: Prior research has explored various robustness measures for ViT models, such as resilience to spurious correlations and adversarial attacks. Our work pioneers the investigation into the channel robustness of ViTs, addressing realistic scenarios in multi-channel imaging applications where sensor discrepancies between training and testing are common.
  • ChannelViT for modeling multi-channel images: ChannelViT is designed to create patch tokens from each channel, enabling effective cross-channel and positional reasoning. Unlike the main literature, we employ weight sharing across channels and leverage learnable channel embeddings to encapsulate channel-specific information. In Appendix B, we present additional comparisons with recent ViT variants for multi-channel imaging, where our simple ChannelViT demonstrates significant performance enhancements.
  • HCS as a Training Regularizer: HCS not only improves training efficiency but also substantially bolsters model robustness when evaluated on varying channel combinations. It is true that it is largely inspired by channel dropout, and we have delineated the similarities and distinctions between HCS and channel dropout within the paper. Figure 3 showcases its superior efficacy, even when combined with a standard ViT.
• Limited to Data with Multiple Channels: ChannelViT is indeed tailored for modeling cross-channel interactions, a critical yet previously neglected issue in multi-channel imaging. As shown in Table 2, the advantages of ChannelViT amplify with the diversity of input sources. In the paper we have also conducted an exhaustive evaluation of ChannelViT on conventional imaging datasets:
  • On ImageNet, ChannelViT outperforms a standard ViT even when both are trained and evaluated on all channels. ChannelViT's robustness is evident when tested with missing channels.
  • On Camelyon17-WILDS, ChannelViT's parameter sharing across channels results in enhanced generalization to new hospitals compared to regular ViT.
• Computational Cost:
  • A new section (Section 4.2) has been added to the manuscript to address computational efficiency, with an extensive runtime analysis in Table 8. Although training ChannelViT requires approximately 3.64 times more resources, the inference time is merely 1.6 times longer than that of ViT. We anticipate future integration of ChannelViT with more efficient attention mechanisms, such as Linformer or LongNet, will lead to further efficiency gains.
  • It is also noteworthy that the additional training time is counterbalanced by eliminating the need to train separate expert models for each channel combination. ChannelViT's performance at processing inputs with varying channels translates to significant practical savings.
• Title Revision: We really appreciate your careful consideration of the title's interpretation. The original ViT paper examined patch sizes of both 16x16 and 14x14, with the latter yielding superior performance. Given their use of a 224x224 training resolution, this equates to 16x16 patch tokens (224/14=16) for each input image. In our work, as we create image patches for each input channel, this results in Cx more patch tokens (words). We acknowledge the potential for confusion and are considering updating the title for better clarity.
• Hierarchical Channel Sampling: In HCS, we first determine the number of channels to utilize (m). Subsequently, based on m, we select the specific channel combination. The term "hierarchical" is derived from the two-stage nature of the sampling process.
• We have amended the notation in the formula as per your suggestion—thank you for bringing this to our attention.

We hope that this rebuttal comprehensively addresses the concerns you have raised. If you have any other questions or concerns, please let us know. We thank you once again for your valuable time and input.

We sincerely appreciate the detailed review and the constructive suggestions provided. Your feedback has been instrumental in enhancing the quality of our manuscript.

• Baselines: We are grateful for the recommendation to include additional baselines. In response, we have expanded our paper to incorporate the following new results, detailed in Appendix B:
  • CNNs (Appendix B.3): We conducted experiments with ResNet-50 and ResNet-152 on the JUMP-CP dataset. The outcomes are presented in Table 7. Our analysis reveals that both ResNet models exhibit performance comparable to ViT-S/8, with ResNet-152 slightly outperforming ResNet-50. However, when testing on different channel combinations, the ResNets' performance diminishes. Attempts to integrate ResNet with HCS during training resulted in instability, culminating in a final 8-channel accuracy of approximately 5%. Finally, the performance disparity between the new ResNet baselines and ChannelViT is consistent and substantial.
  • MultiViT (Appendix B.1): Hussein et al. recently proposed the Multi-Channel ViT architecture for multi-channel EEG data analysis (https://pubmed.ncbi.nlm.nih.gov/35884859/). Their approach involves learning individual ViTs for each channel and fusing the outputs of their final layers for classification. Unlike MultiViTs, ChannelViT models cross-channel dependencies directly from the very first layer. Our comparative analysis of MultiViT, ViT, and ChannelViT, presented in Table 5, demonstrates that MultiViT, when trained without HCS, is more robust than ViT. Nonetheless, it consistently underperforms compared to ChannelViT. Incorporating HCS with MultiViT adversely affects its performance.
  • Fully Attentional Networks (Appendix B.2): Zhou et al. introduced Fully Attentional Networks (FANs) that enhance a standard transformer encoder with channel-wise attention (arxiv.org/abs/2204.12451). Contrary to FANs, ChannelViT executes cross-channel and cross-location attention jointly, allowing each patch token to attend to different channels at distinct locations. FANs restrict attention to different channels within the same location. Utilizing the official implementation, we reported FANs' performance in Table 6. Training FANs without HCS significantly improves performance over a standard ViT (52.06 vs. 65.42). However, ChannelViT still slightly outperforms FANs (66.22 vs. 65.42). When trained with HCS, ChannelViT's performance markedly increases to 68.09, while FANs experience a significant performance decline (from 65.42 to 20.31).
  • Other baselines: We acknowledge the reviewer's suggestions regarding other ViT variants for video analysis and 3D imaging. We argue that the data distribution in multi-channel imaging applications is fundamentally distinct from that of video or 3D data. Specifically, the images we considered are centered around the same object but captured with different sensors. We believe that while ChannelViT could potentially be adapted for sensor fusion beyond multi-channel imaging, such applications are beyond the scope of our current work.
• Computational Cost:
  • We have incorporated a new section (Section 4.2) to discuss computational efficiency, including a more comprehensive runtime analysis in Table 8. While training ChannelViT is approximately 3.64 times more resource-intensive, the inference time is only 1.6 times longer than that of ViT. We anticipate that integrating ChannelViT with more efficient attention mechanisms, such as Linformer or LongNet, could yield further improvements in efficiency.
  • It is also noteworthy that the additional training time is counterbalanced by eliminating the need to train separate expert models for each channel combination. ChannelViT's performance at processing inputs with varying channels translates to significant practical savings.
• Task of Treatment Classification: In line with the initial release of the JUMP-CP benchmark (https://www.biorxiv.org/content/10.1101/2022.01.05.475090v3), we have framed the task of perturbation detection (or treatment classification) as a means to assess representation models for cell imaging. We concur that the primary objective of cell image analysis extends beyond perturbation prediction to encompass other biologically pertinent tasks. We have clarified this point in our manuscript (Page 5, JUMP-CP section) and welcome the opportunity to reference any relevant prior work that you may suggest.
• Standard Deviation and Error Bars:
  • The results presented in Table 3 and Figure 5 were derived from multiple runs using different random seeds for both the partitions and the model initializations. We have now included a descriptive text to clarify these results.
  • Table 14 details the standard deviations across different channel combinations. Additionally, Table 15 provides the mean and standard deviation of the improvements observed for each channel combination with ChannelViT over ViT.

[following previous comment]

• Typos: We are grateful for your assistance in identifying typographical errors. These have been corrected in the revised version of our manuscript.

We trust that this rebuttal comprehensively addresses the concerns you have raised. Should you have any further inquiries, please do not hesitate to contact us. We thank you once again for your valuable time and input.

Thank you for the detailed review and the constructive suggestions, which we have found to be very helpful.

• Scope of ChannelViT: We acknowledge the references you provided, which pertain to cell imaging. It is important to clarify that ChannelViT is designed as a versatile image representation backbone suitable for a broad range of multi-channel imaging applications, not exclusively for cell imaging. To demonstrate this, we have selected four diverse datasets for a thorough evaluation of ChannelViT:
  • ImageNet: As a foundational image representation benchmark, ImageNet is critical for assessing ChannelViT's performance in standard computer vision tasks. Our results on ImageNet demonstrate that ChannelViT not only surpasses the standard ViT in full-channel scenarios by enabling cross-channel reasoning but also exhibits superior robustness when channels are missing.
  • JUMP-CP: The JUMP-CP benchmark, released by the Broad Institute, represents the largest public cell imaging dataset available at the time of our submission. While the mentioned Rxrx datasets and TissueNet are interesting, our comprehensive analysis on JUMP-CP covers similar ground. Also we were unable to access TissueNet via their link in the paper (https://netbio.bgu.ac.il/tissuenet3).
  • So2Sat: Satellite imaging, with its multi-channel, multi-resolution data, is an ideal application for ChannelViT. Our empirical results confirm that ChannelViT outperforms ViT across various data splits and channel availability.
  • Camelyon17-WILDS: Although Camelyon17 features fewer channels compared to JUMP-CP and So2Sat, the challenge of cross-hospital generalization remains compelling. ChannelViT's robust performance across varying staining color distributions from different hospitals is noteworthy. Due to space constraints, we have included this application in our appendix.
  • Finally, we again appreciate the cell imaging references and will consider them for future work for life science journals. We hope this clarification addresses your concern regarding the scope of our paper.
• Model Scale: We have expanded our results to include larger models: ViT-Base, ViT-Large, ChannelViT-Base, and ChannelViT-Large, detailed in Appendix C.7. Unsurprisingly, larger backbones result in superior performance. Our findings also reveal that despite the increased parameter count in ViT-Base and ViT-Large, ChannelViT maintains a significant performance lead. For instance, ViT-Large/16, with 13.9x more parameters, still lags behind ChannelViT-Small/16 (57.96 vs. 68.09), underscoring the importance of explicitly modeling cross channel interactions. We believe this addition should also address your concern regarding the efficiency of learning channel embeddings in larger ViT models.
• Time Efficiency:
  • We have added a new paragraph in Section 4.2 to discuss time efficiency, with a comprehensive runtime analysis presented in Table 8. Training ChannelViT is indeed 3.64x more resource-intensive, but the inference time is only 1.6x longer than ViT. We anticipate further efficiency gains by integrating ChannelViT with more efficient attention mechanisms like Linformer or LongNet.
  • It is also important to highlight that the additional training time is offset by the avoidance of training separate expert models for each channel combination. ChannelViT's performance in handling inputs with varying channels results in considerable savings in practical applications.

We trust that this rebuttal has thoroughly addressed the concerns you have raised. We welcome any further questions and thank you once again for your meticulous review.

Thank you for the detailed review and constructive suggestions. We have found them to be invaluable.

• Information Loss & Channel Alignment: ChannelViT employs learnable positional and channel embeddings to align image patches effectively. The shared positional embeddings across channels enable ChannelViT to easily identify patch tokens from different channels at identical positions.
  • Empirical Evidence 1: Figure 7 illustrates ChannelViT's ability to concentrate on the same region across RGB channels when identifying a zebra.
  • Empirical Evidence 2: We conducted a sanity check to validate the alignment. By using ChannelViT's last layer output for each patch token to predict the original position and channel identity for each patch, we observed that ChannelViT achieves 100% accuracy on both ImageNet and JUMP-CP datasets within a single epoch of training. This underscores the effectiveness of the positional and channel embeddings.
• Distinguishing ChannelViT from Traditional ViT: We appreciate your suggestion to delineate the differences between ChannelViT and the conventional ViT. We had initially discussed these distinctions in the second paragraph of our introduction. Following your advice, we have now included a comparative analysis in the related work and methodology sections of our revised manuscript.
  • We would also like to emphasize that, although ChannelViT's patch and positional embeddings are conceptually similar to those in traditional ViT, several nuanced yet critical modifications have been made to enhance performance significantly.
    • Patch Embedding: Traditional ViT acquires patch embeddings through a linear transformation of 3-channel image patches, with distinct weights for each channel. Conversely, ChannelViT processes each channel's image patches separately, employing shared weights across channels, which we demonstrate leads to improved performance (see Appendix C.3).
    • Positional Embedding: Given that ChannelViT generates individual image patches for each channel, we opted to share positional embeddings, thereby equipping the model with cross-channel reasoning capabilities based on identical positions.
• Clarification of HCS in Section 3.2:
  • The variable m is uniformly sampled from the total number of channels (as described in step 1 of HCS), ensuring that the number of channels of the images processed by HCS are uniformly distributed over {1, 2, … C}. We have appended this clarification to the relevant paragraph to alleviate any confusion.
  • We have clarified in Section 3.2 that HCS is utilized solely during the training phase.
• Evaluation in Table 2: For an 8-channel image, we have considered all 255 possible channel combinations. The exhaustive results, including mean and standard deviation, are presented in Appendix C.8. Notably, despite each row having a consistent number of channels, the informational content across different channel combinations can vary markedly. This variation is reflected in the substantial standard deviation observed in Table 14. To further investigate this variance, we analyzed ChannelViT's performance gains over ViT for each channel combination, reporting both the mean improvement and standard deviation in Table 15. These findings affirm that ChannelViT's performance enhancements are not only uniform but also substantial.
• Evaluation on Datasets with Varying Channel Availability:
  • In Tables 3 and Figure 5, we have trained ViT and ChannelViT on datasets with diverse channel combinations, reporting their accuracy on both full and partial channels.
  • In Tables 2 and 4, we trained ViT and ChannelViT on full channels and assessed their performance under various channel availability. We did not test them on datasets with a mix of partial and full channels. Since the models are fixed post-training, the evaluation scenario you described can be viewed as a blend of different channel combinations. Consequently, we can interpolate performance metrics from Table 2, which already encompasses all potential channel combinations.

We trust that this rebuttal has addressed the concerns you have raised. Please let us know if you have other questions. Thank you once again for your thorough review.