Enhancing Feature Diversity Boosts Channel-Adaptive Vision Transformers

The authors repeatedly stress that each modification in DCS and TDL helps ensure the model's robustness. What ablation study did you apply to endorses it? Did you do any statistical testing on your experiments?

Table 3 of our paper presents a leave-one-out ablation study, highlighting that best performance is obtained by incorporating each component. We have expanded upon that study in the rebuttal PDF Table 1 (including the standard deviation from model variance), where we clearly demonstrate the benefits of each model component. Finally, Tables 1 and 2 of our main paper, Table 1 of our rebuttal PDF, and our response to the second question of Reviewer gjzo also discusses experiments using “partial” sets of channels, i.e., evaluating our robustness to missing channels. Note that we do not report a separate “partial” result for CHAMMI, as the different image sources within this dataset only contain “partial” sets of channels (so no “full” channel results are possible).

The method, as stated, is plug-and-play. Did the authors apply their strategy to other efficient transformer structures with token sampling strategies [1], window-based [2], or linear attentions [3,4]? Because such methods don't have such redundancy exist in the standard Transformer.

We appreciate the reviewer’s suggestion to apply our method to efficient transformer structures. Our method can be easily incorporated as a plug-and-play module into various vision transformer models. Following the reviewer's suggestion, we conducted experiments to test our strategy on Adaptive Token Sampler (ATS) [A]. The results, shown in the table below, indicate that using our method on top of ATS results in a 2% performance boost on So2Sat when testing on all training channels. The improvement is especially significant when only partial channels are available at test time, with a performance boost of up to 38%. In the case of CHAMMI, the average score sees a 3% improvement when incorporating our method.

[A] Fayyaz, Mohsen, et al. "Adaptive token sampling for efficient vision transformers." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

The paper needs proofreading. As instances: In line 37, the authors used “to” two times repeatedly. Omit one. In line 120, it is better to say “as noted in Section 1” rather than “as noted in the Introduction”. When citing a work, cite it after the author's name. For example, in line 98, reformat it as “ Recently, Bao et al. [18] …”.

We thank the reviewer for carefully reading our manuscript and providing suggestions to enhance the clarity of our paper! We will make the necessary revisions to address these points.

Hello Reviewer 4kDP,

As we are halfway through the discussion period, we are hoping you would read through our rebuttal and let us know if you have any additional questions. We would be happy to answer them, thank you!

-Authors of paper 13979

1. It is important to investigate and interpret where the new setup is improving the performance. Additional experiments on interpretations and comparisons of predictions between baseline methods and proposed methodology might help address this weakness.

Thank you for your valuable suggestions! To investigate and interpret where the new setup is improving the performance, we conduct additional analysis in responses 1, 2 and 3 of the Global Response, and Fig. 1&2 in the rebuttal PDF. Additionally, Fig. 7 in the supplementary also gives some insight on the distribution of channel tokens.

2. Given the observation above and performance improvements in Table 2 being quite small with large standard deviation, the data is insufficient to the readers to demonstrate strict performance gains.

Thank you for the question. We believe that this comment stems from a misunderstanding of what the standard deviations pertain to in Tab. 2. Specifically, these refer to variances due to the presence or absence of a channel during inference (not model variance). The relatively large standard deviations show some channels are very informative, and removing them results in large drops for both models, but they do not suggest our gains over ChannelViT are not significant. To illustrate, below we provide individual channel combination results when using 7 channels (of 8), i.e., the “7” column from Table 2, representing 7C8 = 8 channel combinations. However, for each combination we report the mean and standard deviation of the models computed over 3 runs. Our results demonstrate that our approach gets 1-2% better performance for each combination while also providing more stable results (i.e., smaller model variance) than ChannelViT. Other experiments in Tab. 2 reported similar behavior, clearly demonstrating our benefits over prior work.

3. Table 3: It is also important to show performance improvements with inclusion of each of the losses independently to understand the independent effects and contributions of each of the modifications

We thank the reviewer for their suggestions! In Table 1 of the rebuttal PDF, we extended Table 3 in the main paper to have a better understanding of the individual effects and contributions of each of the losses. We observe that adding DCS helps improve the performance (e.g., by ~4% on CHAMMI), and robustness of the model, especially when tested on partial channels (a boost of ~35% on So2Sat Partial). Similarly, TDL and CDL also show improvement across the three datasets. For example, TDL improves the performance by 2.5% on CHAMMI and 2% on So2Sat on full channels.

Can the authors provide a more intuitive explanation for why patch token orthogonality (especially within channel patch token orthogonality) is necessary or might be useful?

Patch token orthogonality encourages the tokens generated from different image patches are distinct from each other, helping in preserving and enhancing the discriminative features of the patches. Patches within the same channel can still contain significant variations due to imaging different image regions, and our model helps encourage that they encode different features, if possible.

An identical patch of pixels would produce an identical set of tokens. Is patch token diversification unreasonable?

In this case the model can not further optimize the diversification loss (it’s at the minimum). That said, like any regularizer, it encourages a bias whose contribution must be carefully tuned.

The results from Fig 4c further indicates that imposing weaker constraints on tokens from same channels compared with tokens from different channels obtains best performance. So, how is the patch token diversification loss on same channels helpful?

Fig. 4C implies that one may want to apply a larger penalty for the similarity between patches from different channels, compared to within channel. That said, patch token diversification loss on same channel is still helpful, as shown in Table 4 of the main paper. Patches within the same channel can still contain significant variations due to different image regions. By enforcing some level of diversification within patches from the same channel, the model can capture finer details and nuances that might be missed.

Limitations. Experiments are conducted with limited datasets with smaller dataset sizes.

We appreciate the reviewer's feedback. We begin by noting that our experiments are already more extensive than prior work in MCI, as we combine the benchmarks used by two different recent MCI papers [14,18]. In addition, these are similar sizes to many widely used benchmarks (e.g., COCO has 330K images while CHAMMI, JUMP-CP, and So2Sat contain 220K, 217K, and 376K images, respectively). While they are not suitable for large-scale pretraining, many (if not most) of the applications in MCI simply cannot do so due to lack of data availability and the expertise required to collect new data. For example, collecting new data and annotations for CHAMMI requires a biologist to perform a time consuming experiment in a lab. As such, we would argue that our experiments are more representative of the real-world applications of MCI.

Hello Reviewer gjzo,

As we are halfway through the discussion period, we are hoping you would read through our rebuttal and let us know if you have any additional questions. We would be happy to answer them, thank you!

-Authors of paper 13979

Can you provide an in-depth analysis on the effect of losses on the distribution and content of channel and patch tokens?

Thank you for your valuable suggestions! Please refer to Fig. 1a for the effect on channel token content and Fig. 7 the effect on channel token distribution. Additionally, the first two questions of the global response we further analyze model performance, and Fig. 1 of the rebuttal PDF reports the effect on patch tokens. In summary, our contributions result in less redundant information being captured, boosting performance across the board. Additional discussion in the global response.

Are the gains in Tab. 2 significant (they have large standard deviations)?

We believe that this comment stems from a misunderstanding of what the standard deviations pertain to in Tab. 2. See Response 2 to Reviewer gjzo.

Are the gains of CDL and TDL significant?

Tab. 1 of the rebuttal PDF shows when including CDL in our full model (exp. 6 & 10) we boost performance by 1-2% with standard deviations from model variance of 0.2-0.4 in most settings, clearly demonstrating significant gains. The gains from TDL are similar (exp. 8 & 10).

The differences in performance for varying hyperparameters are not remarkable.

This is a benefit of our model as it helps show our model does not need extensive tuning when applied to new datasets.

Can you use metrics other than accuracy?

Classification accuracy (So2Sat and Jump-CP) and F1 score (CHAMMI) were selected by prior work due to dataset restrictions [14,18]. E.g., CHAMMI’s tasks reveal differences and similarities among cellular phenotypes with a carefully designed evaluation protocol. Changing this protocol to include other metrics would require collecting a new dataset.

Notation clarifications/suggestions

We thank the reviewer for carefully reading our manuscript and providing useful suggestions! We will revise our manuscript to make these points clear.

1. Regarding CDL: Have you tried to see what is the impact of imposing orthogonal initialization on tokens belonging to very similar channels? Overall, I see these methods as effective only on MC datasets in which channels contain sufficiently diverse information

Our datasets do contain very similar channels, yet models still report improved performance when using orthogonal initialization. E.g., So2Sat (18 channels) contains Red, Green, and Blue channels, which are known to be similar, and even show stronger correlation among Bands B6 - B8a (Fig. 2 of the rebuttal PDF). Yet, we still obtain improvements with CDL (see Fig. 3 main paper, and Tab. 1 of the rebuttal PDF). We find that some of these similar channels do result in similar tokens in a trained model, i.e., if there is important information the model can overcome an orthogonal initialization. Additionally, if two channels are similar, then the likelihood they end up learning the same features is high, especially if they are informative features. Thus, a model may miss learning any unique information contained in only a single channel (resulting in high mutual information in Fig. 1a of the main paper). As such, in the case of similar channels, it is even more important to encourage diversification so that more than just the redundant information is learned.

Regarding CDL: Why did you use the Euclidean distance in the loss?

Eq. 1 and L143 note that we use L2-norm + Euclidean distance following ProxyNCA++ [52]. This combination is equivalent to common alternatives like cosine similarity (e.g., as used in [A]). That said, we are happy to compare to other alternatives the reviewer would suggest, but also argue these implementation details do not significantly affect any conclusions we make. [A] Learning transferable visual models from natural language supervision. Radford et al., ICML, 2021.

How are the anchors initialized?

The trainable channel anchors are also orthogonally initialized, encouraging diversity.

2. Why is cosine similarity between channel tokens used in DCS, whereas in CDL euclidean distance is employed instead?

In CDL, we utilize the Euclidean distance with L2-Norm, as mentioned in Line 143, to be consistent with [52]. However, this is equivalent to using Cosine similarity as noted earlier. We will revise our manuscript to clarify this point.

3. Can you use DiChaViT instead of ChannelViT when analyzing the role of channel tokens in section 4.4?

We conducted the same experiments using DiChaViT and presented the results in Table 2 of the rebuttal PDF. The conclusion remains the same as ChannelViT’s.

4. Have you tried to measure their effectiveness on hyperspectral images with several tens of channels?

We combined the evaluation protocols used by prior work on MCI [14,18], resulting in a more comprehensive study than prior work. That said, we would be happy to explore additional suitable datasets suggested by the reviewer.

Don’t two patches referring to the same spatial location from different channels exhibit similar semantic features?

We found empirically that a larger penalty for patches from different channels compared to patches within the same channel improves performance. However, in Eq. 4 $\lambda_s$ and $\lambda_d$ balance the loss of tokens belonging to the same channels (first term) and different channels (second term). Thus, one can adjust these hyperparameters according to the dataset if needed. That said, our experiments already show that similar channels can still benefit.

It would be interesting and meaningful to compare the learned features at different levels of the model (e.g., by visualizing the attention maps) to explicitly assess the disentangling effect brought by these methods.

We compare the learned features at different layers by visualizing the attention maps in Fig.1 of the rebuttal PDF. We found that DiChaViT has more evenly distributed attention scores across channels, indicating each channel contributes more to the prediction.

Hello Reviewer K1xF,

As we are halfway through the discussion period, we are hoping you would read through our rebuttal and let us know if you have any additional questions. We would be happy to answer them, thank you!

-Authors of paper 13979

Unseen channels during training: One of the key limitations of this work, which is also briefly acknowledged by the authors, is that the method is only focused on dealing with missing channels only at inference. Meaning, all channels have to be seen by the model during training. This might not always be the case. I would like the authors to elaborate more on why this is not considered in this work. I would be happy with an explanation in terms of which of the contributions would breakdown for unseen channels. Just to be clear, I am not seeking additional experiments.

Our contributions should still benefit from a setting requiring inferring unseen channels, but taking advantage of it is not straightforward. A key challenge when generalizing to unseen channels is establishing a connection between existing and new channels. This is not challenging, as simply identifying a similar channel may not be sufficient. For example, using the weights for a similar channel may result in simply extracting the same features as that channel, resulting in redundancy that does not improve performance. Instead, unseen channels require extracting the most informative features, which may not be learned by the most similar channel. This is further complicated in the presence of domain shifts, which makes finding the move informative channel weights even more difficult. Thus, this requires a more focused effort to create this mapping that goes beyond the scope of our paper.

Balancing loss function: The composite loss function has multiple hyperparameters: 𝜆𝐶𝐷𝐿, 𝜆𝑠, 𝜆𝑑, and temperature, 𝑡. The authors describe the tuning of these parameters individually with discussions on the ranges without elaborating on how to obtain these for other datasets. For example, 𝜆𝐶𝐷𝐿 is set to 0.001 for So2Sat and 0.1 for CHAMMI. How were these obtained?

Hyperparameters are set using grid search over a validation set. We shall update our plot to include both the test and validation results in our revised paper.

Fig. 4 (a,b) show the sensitivity of test accuracy wrt these parameters. And do these magnitudes mean anything? Why the log-scale?

The magnitudes show how these parameters affect the model performance. For example, a very high $\lambda_{𝐶𝐷𝐿}$ value could cause the model to excessively prioritize diversifying the channel token features, which may negatively affect the performance on downstream tasks. In contrast, if $\lambda_{𝐶𝐷𝐿}$ is too small, it may not effectively diversify the learned channel token features, leading to underutilization of its benefits.

Similarly, $\lambda_{s}$ and $\lambda_{d}$ control the constraint of tokens belonging to the same channels and different channels in Eq. 4.

Using a log scale allows us to visualize a wide range of parameters to observe the impact of these parameters on the final performance.

Channel anchors: The discussion on using channel anchors is missing some key information. This could be because I was not aware of the ProxyNCA++ loss referenced in [52]. How are the channel anchors initialised and then selected for each channel?

As noted in L132 of our paper, each channel has its own anchor initialized orthogonally to the other channel anchors. There is no need for them to be “selected” as they are trained as their anchor. I.e., in So2Sat there are 18 channels, so we would orthogonally initialize 18 channel anchors, each trained to correspond to one specific channel.

What would happen if instead of learning these vectors, one used a one-hot encoding to specify the channels? Perhaps some clarification here can be useful.

Each channel anchor is the same dimension as the channel token so that we could compute similarities between them. For the 384-D dimension channel tokens used in our experiments, we would have a 384-D channel anchor. A one-hot encoding on the 18 channel So2Sat dataset would effectively ignore 384-18=366 of the channel token features (as their loss contribution would always be 0 in a one-hot encoding of the anchors). Thus, this would, in large part, make our CDL loss completely ineffective in practice.

Clarification on two Temperatures: In L. 148 authors say that results are not sensitive to the choice of 𝑡 in Eq. 1. And, there is another temperature parameter (?) in Sec. 3.3 when describing DCS which is later reported in Table 5. They are different softmax temperatures, as far as I can see. This is very confusing. And which of the temperatures is more important?

Thank you for carefully reading our paper and spotting out the confusion regarding the temperatures! In line 148, the temperature $t$ controls the sharpness of the probability distribution when using CDL. We observed that the model is not sensitive to the value of $t$, and simply fixing it consistently yields good results across datasets.

The second temperature $t$ mentioned in Section 3.3 (Algorithm 1) is used to control the distribution of CDS. This $t$ is the one presented in Table 5 and is more sensitive in our experiments.

We will revise the manuscript by using $t_{CDL}$ and $t_{CDS}$ for these temperatures to make them clear.

However, they set it to a specific (and a peculiar) value of 𝑡=0.07.

In Eq. 1 of our paper the temperature $t$ is represented as the denominator of a fraction. We used grid search centered around the denominator value of $t=1/9$ used by ProxyNCA [52]. None of the results were statistically different, but we used $t=1/14=0.07$ as it had the best average result.

Hello Reviewer q1Xa,

As we are halfway through the discussion period, we are hoping you would read through our rebuttal and let us know if you have any additional questions. We would be happy to answer them, thank you!

-Authors of paper 13979