Scaling Channel-Invariant Self-Supervised Learning

Thank you for taking the time to review our paper! Your question regarding the novelty of our method was raised by other reviewers, in addition to our comments below, please also see the general response section.

I do not consider DINO-BoC to be a novel method. It process channels separately using a ViT, then concatenates embeddings.

Addressing your concerns specifically, our main contribution is that, contrary to recent works that argued in favor join encoding strategies (Bourriez et al 2024, Bao et al 2024, Kraus et al 2024, Pham & Plummer 2024), we find that (surprisingly) at scale the winning channel-invariant strategy is to combine independent channel encoding (BoC approach) with ViTs! We think this is rather striking result given that (a) Xun et al 2024, in their Fig.S1, found that ViTs underperformed CNNs when training on single channels (BoC approach) and (b) Bouriez et al 2024, in their Fig.4, reported that the BoC approach in turn underperforms channel-adaptive ViTs.

Channel-ViT (the baseline) beats DINO-HA on 17 of 26 test sets on page 8. And DINO-BoC beats Channel-ViT on 16 of 26 test sets on page 8. I'm not convinced that DINO-HA is valuable to the community beyond these fairly negative results.

To your question about the value of the Hierarchical Attention model, it achieves a trade-off between jointly encoding the channels (Channel-ViT) and independently encoding them (BoC strategy), as such, it offers an intermediate point of analysis between joint and independent channel encoding. In our work, we find that inter-channel reasoning is detrimental to the performance and generalization capability of channel-invariant models. Therefore, the HA model results are pertinent because it strongly outperforms Channel-ViT, while not reaching the performance of BoC, supporting the conclusion that moving from joint to independent channel encoding is indeed the most promising direction for channel-invariant models. Please also see our response to reviewer TY5V regarding our HA model.

Regarding your questions about the features' dimension:

As far as I can tell, DINO-BoC concatenates channel embeddings to form the full-sample embedding using the CLS tokens. This embedding is a D * N-dimensional vector (where D is the ViT width and N is the number of channels). And the full-sample embedding of Channel-ViT is a D-dimensional vector, taken from the CLS token.

Although in the appendix it says the CLS token is concatenated across the last 4 layers for linear probing.

Can you please clarify how you build full-sample embeddings for each of the three methods?

We have updated Appendix G.2 (now Appendix D.2) to more clearly describe how the features are obtained for each of the methods. Let:

• $D$: embedding dimension
• $K$: number of channels
• $L$: number of “last” layers we take the CLS tokens from

For all evaluations -- with the exception of the evaluations on the CHAMMI benchmark -- we take $L=4$ and the features are obtained by concatenating the CLS tokens across the last $L$ layers, as well as the channel-wise average pooled patch tokens. In more details:

• For Channel-ViT we concatenate the CLS tokens from the last L layers and the channel-wise average pooled patch tokens, so $LD + KD$ dimensional.
• For BoC we concatenate the CLS tokens, for each channel, from the last L layers and the channel-wise average pooled patch tokens, so $LKD + KD$ dimensional.
• For HA we concatenate the global and channel CLS tokens from the last L layers and the channel-wise average pooled patch tokens, so $L(1+K)D + KD$ dimensional.

For the evaluations on the CHAMMI benchmark, only the CLS tokens are used and $L=1$. Therefore the feature size for Channel-ViT is $D$; for BoC it is $KD$ and for HA it is $(1+K)D$.

If Channel-ViT's embeddings are indeed D-dimensional and DINO-BoC are D * N-dimensional, I'd like to see results when they have the same sized embeddings.

This can be done by taking the mean between channels (for DINO-BoC) to achieve D-dimensional embeddings.

This can be done by concatenating the mean-pooled patch representations for Channel-ViT to achieve N * D-dimensional embeddings (specifically, you'd pool the patches for each channel, then concatenate these pooled channel-wise embeddings).

Thanks for this suggestion! We are adding those two comparisons to Appendix K. We present the results for the following two scenarios:

1. For Channel-ViT we use only the CLS token from the last layer, so $D$ dimensional; while for BoC we average pool the CLS tokens for each channel, so $D$ dimensional as well.
2. For Channel-ViT we concatenate the CLS token from the last layer to the channel-wise average pooled patch tokens, so $D + KD$ dimensional; while for BoC we concatenate only the CLS tokens for each channel, so $KD$ dimensional.

The results we obtained on the CHAMMI benchmark for those configurations are:

From those partial results, we observe that the superior performance of BoC is indeed due to the better quality of the features that are learned, rather than a consequence of a larger embedding size.

Still, as the ChannelViT $KD$ results are a bit better than the ChannelViT $D$ results, we will update the other CHAMMI benchmark tables of the paper with this baseline which is more fair.

Thank you for your detailed comments and suggestions! Your concerns regarding the novelty of our work and applicability across various imaging domains (your cited weaknesses 1 & 2) were shared by other reviewers, we hence address them in the general response section.

The technical novelty (contribution (ii)) is limited, as the proposed Hierarchical Attention module appears to be a combination of existing methods (Channel-ViT and Bag of Channels) with the addition of extra "channel" tokens and corresponding attention masking. Furthermore, the clarity of presentation of the technical contribution needs improvement, as I explained in the next point. It is also unclear to me the motivation behind introducing DINO HA since, in Tables 1, 2, and 3, the best method is usually DINO Bag of Channels and not DINO Hierarchical Attention. What is the purpose of proposing two methods, with one clearly better than the other?

Regarding the Hierarchical Attention (HA) model, let us separately address the two concerns that were raised: its technical novelty, and its purpose. With respect to its novelty, it achieves a trade-off between jointly encoding the channels (Channel-ViT) and independently encoding them (Microsnoop and BoC strategy). Yet, it is not a simple combination. HA is technically distinct from both methods.

The only common aspect between Channel-ViT and HA is that they employ single-channel patches. However, Channel-ViT differentiates between tokens coming from different patches through channel embeddings, and there is no restriction on cross-channel attention. On the other hand, HA employs a novel attention scheme that only allows cross-attention between patch tokens belonging to a same channel, then cross-channel interactions are restricted to the global CLS token that attends to the channel CLS tokens. This way, the HA model offers both independent channel encodings through the channel CLS tokens, and an image-level encoding, through the global CLS token.

In regards to the purpose of the HA model, it offers an intermediate point of analysis between joint and independent channel encoding. In our work, we find, surprisingly, that at scale inter-channel reasoning is detrimental to the performance and generalization capability of channel-invariant models. Therefore, the HA model results are relevant because it strongly outperforms Channel-ViT, although not reaching the performance of BoC. This counters the consensus of a series of recent papers and suggests instead that moving from joint to independent channel encoding is the most promising direction to train foundation models in domains with variable input channels.

The technical presentation of contributions (i) and (ii) is somewhat unclear and misleading. More precisely, the main technical section 3, which is named "Problem Formulation" (renaming needed), should instead focus on the description of the introduced modules and clearly discern the contributions of the paper (sections 3.3 (?) and 3.4) against the already existing strategies (i.e., sections 3.1, 3.2, 3.5). To clarify the contributions, it would help to split Section 3 into two sub-sections (Preliminaries vs Methodology) to help the reader better grasp the actual contributions. I would also suggest a better design of Figure 2, e.g., in a single panel, highlighting the differences between previous and proposed work.

The related work section concerning SSL with microscopy images and section 4.1 are comprehensive but verbose and would need some summarization and re-writing.

Thank you for the suggestions regarding the structure of the paper, we have made broad revisions to the text that we believe significantly clarify the contributions and results of our work.

Please let us know if we have addressed your questions, and if you have any additional suggestions.

Thank you so much for taking the time to review our paper! Many of your concerns were shared by other reviewers. We specifically comment on your major concerns below, but please also see our general response section.

The title describes a generic method for multi-channel images, but the paper focuses only on microscopy images. Considering that different multi-channel images may exhibit different behaviors, the title seems to be a bit big. The authors shall either broaden their experiments to include other types of multi-channel images, or narrow the title and framing to focus specifically on microscopy images.

We agree that our focus on the domain of microscopy images should be reflected in the title and we have narrowed it. Nevertheless, our method is indeed generic, and we now show with new experiments that it works very well out of the box for aerial and remote sensing imagery (see general responses and Appendix J).

The title states this work focuses on self-supervised learning. However, after reading the whole paper, the method is not generic. On the contrary, it seems to be designed based on DINO v2. No discussion on the choice of DINO v2 at all. The authors shall provide a clear justification for choosing DINO v2 as their base method. Additionally, please discuss how your approach might generalize to other self-supervised learning frameworks, or explain why your method is specific to DINO v2.

The contributions on self-supervised learning is not clear to me. In my opinion, the paper seems to be a multi-channel variant of DINO v2, which is an incremental work. But, the paper tries to tell a different story. If this work is about self-supervised learning while leveraging multi-channel images, please specify why this work differs from studies of self-supervised learning on RGB images. If this work focuses on adapting existing self-supervised learning works on multi-channel images, please clarify it from the title to introduction.

To your questions on self-supervised learning (SSL): as the title suggests, our paper is not concerned with SSL per se, but rather with the scaling properties of “channel-invariant” SSL. In particular, contrary to RGB images, many scientific imaging modalities, such as microscopy images, vary in number and content of channels (see Fig. 1); e.g. WTC-11 has 3 channels, HPA-FOV has 5. Such datasets are also highly heterogeneous with respect to available labels. To explore scaling of channel-invariant methods, SSL is hence a natural choice.

Indeed, we show that our method is compatible with any SSL framework, not just DINOv2. For instance in Table 5 of the original manuscript we compare the performance of BoC pre-trained with DINOv2 and MAE, the dominant paradigm for microscopy foundational models. Noticeably, BoC pre-trained with MAE still performs on par or better than Channel-ViT. Still, we find that DINOv2 substantially outperforms MAE, even when handicapped by a smaller model size. We present these results more prominently now, in a revised Table 1 (see also general responses).

Yet, our main objective is to study the behavior of different channel-invariant strategies at scale. Contrary to findings by several high-profile papers that have explored channel-invariant learning strategies at smaller scale (Bourriez et al 2024, Bao et al 2024, Kraus et al 2024, Pham & Plummer 2024), and noticeably, in further stark contrast to established best-practices for RGB images, encoding channels independently is the winning strategy for large-scale channel-invariant learning with SSL. We thus empirically refute previous claims about the importance of joint channel encoding and, with DINO BoC, present the best performing channel-invariant model for microscopy to date.

The technical writing is a bit hard to follow. The presentation of the method proposed is unclear. The paper spends quite some space the background while the methodological details are provided in the appendix. It is recommended to reorganize the related works and method sections to show key methodological results.

We have revised structure, writing, tables (Tables 1-2), and figures (Fig. 2), as well as added additional experiments that together, we think, substantially strengthen the presentation.

Does this method work on other self-supervised learning framework? Follow the same logic, will better self-supervised learning backbone show better results?

As described above, BoC is compatible with any SSL method. We compare DINOv2 to MAE.

How does this method work on multi-channel images such as remote sensing or multi-spectral images?

As mentioned above and in the general response section, we demonstrate Appendix J that our method generalizes very well to aerial and remote sensing imagery, out of the box.

Thank you very much for your review! Your questions regarding how our method is different from Microsnoop and whether it works on other domains was shared by other reviewers. In addition to our comments below, please also see the general response section.

It is not clear how the bag of channels approach claimed as yours is different to Microsnoop form Xun et al.

Regarding the difference between DINO Bag of Channels (BoC) and Microsnoop, there are three main components to be considered in a channel-invariant encoder:

1. the strategy that technically allows the model to handle channel heterogeneous data;
2. the network architecture itself;
3. the training strategy.

DINO BoC and Microsnoop share the channel-invariant strategy, i.e., independently encoding each channel, however they differ significantly on the other aspects: DINO BoC leverages a ViT and the DINOv2 SSL method, while Microsnoop consists of a convolutional U-Net pre-trained with a Masked Auto-Encoder (MAE) SSL objective.

We note here that, Xun et al 2024 (see Fig.S1), found that ViTs underperformed convolutional U-Nets when training on BoC and Bouriez et al 2024 (see Fig.4), reported that the BoC approach in turn underperforms channel-adaptive strategies with ViTs. Rather surprisingly then, our results show that combining ViTs with BoC outperforms all other methods at scale!

In Table 5 of the original manuscript (now featured more prominently in Table 1), we show that, for BoC, the choice of DINOv2 further yields significant improvements over MAE, that is, the SSL learning objective used in Microsnoop.

Finally, we added a new comparison to Microsnoop, evaluating our approach on the Cyclops dataset, which is highly out of domain in our case, in which DINO BoC outperforms Microsnoop by a large margin (8 points on average; up to 30 points on rare classes).

The proposed hierarchical attention approach rarely beat the bag of channel approach

With respect to the Hierarchical Attention (HA) approach, its main value is providing an intermediate point of comparison between the joint and independent encoding of channels. By using a hierarchical attention scheme that limits the inter-channel interactions, it outperforms Channel-ViT, while complete independent encoding (BoC) is the best approach at scale, contrary to previous findings at small scale (Bourriez et al 2024). Therefore, the performance of the HA model further corroborates our finding that moving from joint to independent channel encoding is the most promising direction to train foundation models in domains with variable input channels.

Do you expect this approach to work for other application domains with image modalities of different numbers of channels with limited channels matching across datasets like it is often the case in remote sensing?

Yes, our approach is generic and we now show with a new experiment that it works very well out of the box for aerial and remote sensing imagery, see the new section in Appendix J.

Figure 2 legend can be improved by adding more details to it.

We have revised Figure 2 and provided a more detailed description.

Thank you for acknowledging positive aspects of our work (well written, with a rich experimental section, usefulness of the work). Your primary concern, regarding the significance of our contribution, was shared by other reviewers, and we hence describe our clarifications, revisions, and additional experiments in detail in the general response section above.

Addressing your concerns specifically, compared to Microsnoop (Xun et al 2024), DINO Bag of Channels (BoC) is different in both architecture (convolutional U-Net vs ViT) and SSL strategy (Masked Auto-Encoder (MAE) vs DINOv2). Our main contribution is that, contrary to recent works that argued in favor join encoding strategies (Bourriez et al 2024, Bao et al 2024, Kraus et al 2024, Pham & Plummer 2024), we find that (surprisingly) at scale the winning channel-invariant strategy is to independently encode the channels. We think this is rather striking result given that (a) Xun et al 2024, in their Fig.S1, found that ViTs underperformed CNNs when training on single channels (BoC approach) and (b) Bouriez et al 2024, in their Fig.4, reported that the BoC approach in turn underperforms channel-adaptive ViTs.

We further show that DINO substantially outperforms MAE when combined with ViTs and BoC (this result is now more prominently featured in a new Table-1). Finally, we report new results comparing DINO BoC to Microsnoop on the Cyclops dataset, which is highly out of domain in our case (images of yeast with unseen channels). We find that DINO BoC outperforms Microsnoop by a large margin, especially on rare classes (up to 30 points).

As such, we report with DINO BoC a specific recipe that, unexpectedly, outperforms all other methods, and thus provides an (we think) important challenge to a swath of recent papers reporting the superiority of joint channel-encoding, published at venues like CVPR and ICLR.

Why didn't you use datasets with more channels (as stated in L38, up to 8 channels)?

This is a great prompt: a key differentiator between methods that seek to reconcile the joint modeling of channels (to learn interactions) with varying number and content of input channels, such as Channel-ViT, and our method, is that the former is only able to adapt to varying number of input channels up to some maximum (see L235-237), whereas DINO-BoC is agnostic with respect to the number and kind of input channels. As such, a Channel-ViT trained on the 5 input channels of ExtendedCHAMMI cannot process e.g. the full JUMP-CP 8-channel dataset (which includes 3 extra brightfield channels). In contrast, we now include additional results (Appendix I) that show that in the same setting, DINO-BoC is not only compatible with the additional input channels, but productively leverages them to improve performance on our JUMP-CP benchmark. DINO-BoC thus outperforms Channel-ViT in generalization to unseen channel combinations both quantitatively (where the comparison is possible) and categorically.

The experiments are performed on datasets with 2-5 channels, is there any correlation between the number of channels and the method performances?

We find that using more channels does not always lead to improved performance, as shown in (new) experiments on aerial imaging data (Appendix J), using four NAIP channels alone results in higher performance than when additional channels from Sentinel-2 are also used. On the other hand, adding the 3 additional brightfield images on top of 5 fluorescent channels in JUMP-CP improves performance on evaluation (see above). The value of additional channels likely depends on how well their semantics can be interpreted by the feature extractor, and their relevance to a given downstream task.

Continuation:

Novelty

A primary concern shared by all reviewers is how our proposed method is different from previous works. We have thus substantially revised the narrative, presentation of our results, and provide additional experiments to more clearly delineate our work. Briefly, encoding channels independently, as proposed by Xun et al 2024 initially showed promise. However, rather than sacrificing the ability to learn channel-interactions, a recent set of high-profile works at CVPR (Kraus et al 2024, Bourriez et al 2024), ICLR (Bao et al 2024), and NeurIPs (Pham & Plummer 2024) reported considerable advantages for channel-adaptive methods that reconcile the need to account for variable channel number with the ability to joint encoding of channels (as is established practice for RGB images). Bourriez et al 2024 reported substantial gains of such methods compared to independent channel encoding (BoC approach). Xun et al 2024 further found that convolutional U-Nets outperformed ViTs in the BoC setting. Surprisingly, and contrary to this emerging consensus, we here show that, at scale, the specific combination of BoC with ViTs is the winning recipe. Moreover:

In a revised Table-1, we highlight that DINOv2 substantially outperforms MAE, i.e. the SSL method used in Microsnoop, as a learning objective for BoC, and does so even when handicapped by a ViT-S instead of ViT-L. We note that even with MAE, BoC still outperforms Channel-ViT on 3 of 4 benchmarks. This indicates that both the ViT architecture, and the DINOv2 as the SSL objective (as the key technical differences to Microsnoop), independently synergize with the BoC approach at scale.

We further provide new results in Table-2, demonstrating that DINO BoC outperforms Microsnoop on the challenging Cyclops dataset (highly out of domain) by a large margin, and in particular for rare classes (by up to 30 points).

We further emphasize and expand our results on generalization to unseen channel combinations, as a key metric for the practical utility of foundation models in scientific domains with variable channel combinations. In particular, we show that DINO BoC not only outperforms Channel-ViT quantitatively, but in some instances categorically, since Channel-ViT cannot process datasets with a number of channels that exceeds that of its training dataset (Table-10).

Ideally, we would also compare the Microsnoop implementation (U-Net + MAE) trained on ExtendedCHAMMI directly to DINO BoC, but Xun et al did not release training code. Yet, even absent this comparison, our results provide an important beacon to the field: our findings refute the claimed superiority of joint-channel modeling channel-invariant learning at scale. By identifying surprising synergies between DINO, ViTs and BoC, we publish the most powerful channel-invariant model for microscopy to date, compatible with any number of input channels. We open-source both weights and all code for further improvement by the community.

Novel Results on Aerial Imaging

We also demonstrate that DINO BoC holds promise for imaging domains other than microscopy. Specifically, we train and benchmark the performance of DINO BoC and DINO Channel-ViT on the Meter-ML aerial imagery dataset.

DINO BoC consistently outperforms DINO CHannel-ViT. Furthermore, using the exact same normalization and evaluation protocol as for the microscopy benchmarks, DINO BoC performs on par or surpasses state-of-the-art models for aerial images. This is achieved even though DINO BoC is pre-trained on a much smaller dataset and does not use remote sensing specific architectures. This highlights the generality of our approach across a wider range of imaging domains.

Additional Experiments

We have also included additional results addressing individual concerns raised by the reviewers, such as an ablation study on the feature dimension of DINO BoC and DINO Channel-ViT on Appendix K, as well as results for tasks with 8 channels on Appendix I (previously the tasks were limited to 5 channels).

All of those experiments reinforce our core conclusion: independent channel-encoding is the most robust channel-invariant strategy, outperforming all other methods, even those based on more sophisticated techniques tailored to the problem setting; the pervasive assumption that joint-channel-encoding is necessary in channel-heterogeneous domains does not hold, specially at scale. In summary, through a rigorous analysis of existing methods, we challenge the previously held understanding of channel-invariant learning, which we believe is vital for advancing scientific discourse.