Revisiting Continuity of Image Tokens for Cross-domain Few-shot Learning

Thank you for your thoughtful feedback and constructive suggestions. Below are our responses to your concerns:

1. More Proofs for Continuity

Our method is consistent with the proofs of the previous works [1-3] in that the small patterns are easier to transfer than larger ones. But we differ in that (1) our method discusses the transferability of small patterns for ViT while [1-3] are originally designed for CNNs; (2) we provide a simple way to enhance the learning of small patterns by perturbing patches with no additional branches or losses.

[1] Cross-domain few-shot learning with task-specific adapters

[2] Task-aware adaptive learning for cross-domain few-shot learning

[3] Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning

2. Frequency Disruption Breaks the Continuity

Qualitatively, images from an anonymous GitHub link ([https://anonymous.4open.science/r/More-visualization/visualizations.png]) illustrate how our method breaks large-scale spatial patterns while preserving fine-grained, domain-agnostic features (e.g., textures, edges), which aligns with our motivation.

Quantitatively, experiments in Fig. 2 and Fig. 4 validate the feasibility of our methods.

3. Source domain performance trade-off

As suggested, we have provided the performance on the source domain (miniImageNet) for reference.

While there is a slight performance trade-off on the source domain, this aligns with our hypothesis: disrupting token continuity prioritizes learning smaller, transferable patterns over domain-specific holistic features. Crucially, the significant gains on target domains (e.g., +6.5% on ISIC) demonstrate the effectiveness of our approach for cross-domain adaptation.

4. Experiments with other pretraining and backbones

(1) Other pretraining

We have conducted experiments with CLIP pretraining. Results confirm our method’s generalizability.

(2) Other backbone

While our method is designed for ViTs, we test an MLP-Mixer backbone (patch-based architecture):

This demonstrates applicability to other patch-based architectures.

5. Role of CDFSL vs. large pretrained models

While large vision-language models (VLMs) have demonstrated remarkable generalization capabilities, their effectiveness heavily relies on the assumption that downstream tasks share similar data domains with their pre-training corpora (e.g., natural images and generic text). However, in vertically specialized scenarios (e.g., medical imaging, remote sensing), where domain gaps are extreme and task-specific patterns diverge significantly from generic priors, directly fine-tuning VLMs often yields suboptimal performance [4], even worse than training domain-specific models (e.g., UNet) from scratch [5]. Pre-trained models like DINO struggle with medical X-rays (22% accuracy on ChestX) due to fundamentally different texture and structural semantics compared to natural images. However, training large vision-language models from scratch is impractical due to limited data in target domains. This highlights the necessity of CDFSL-specific designs to address domain shifts that challenge even powerful pre-trained models.

[4] Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

[5] Lightweight Frequency Masker for Cross-Domain Few-Shot Semantic Segmentation

6. Computation cost

Our method introduces no additional trainable parameters.

During training, the computational overhead (baseline: 126.42s vs. ours: 140.48s per epoch) stems from the clustering-based frequency disruption during training, which is acceptable compared with the gains in cross-domain performance. Moreover, some engineering tricks, such as bipartite matching, can accelerate the clustering process, which we leave for future works.

During inference, since our method introduces no additional parameters, inference time matches the baseline exactly with no computational overhead.

Thank you for your thorough review and constructive feedback. Below are our detailed responses to your concerns:

1. Justification of Clustering Hyperparameters

The clustering threshold in Eq. 16 controls the granularity of patch grouping in the frequency domain. Higher threshold values (e.g., >40%) enforce stricter similarity criteria, resulting in smaller clusters (more fragmented groups). At the extreme (threshold → 100%), each patch forms its own cluster, which degenerates to non-cluster random sampling. Since some image regions occupy larger areas (e.g., background), this will lead patches in the dominant areas to change frequency with other patches in the same area, i.e., their frequencies are not changed, maintaining the original continuity. In contrast, lower thresholds (e.g., <40%) allow looser grouping, creating larger clusters. As shown in Fig. 6b, the optimal threshold (30%) achieves a critical balance.

The amplitude sampling standard deviation (ϵ) in Eq. 21 governs the diversity of style proportions assigned to patch clusters during frequency-domain disruption. Specifically, smaller ϵ results in near-uniform style proportions across clusters. This over-regularizes the disruption, limiting diversity and failing to suppress large patterns effectively. As shown in Fig. 7, Larger ϵ introduces high variability in style mixing ratios, creating heterogeneous disruptions that break global continuity.

2. Comparison with Diversity-Promoting Methods

We add comparisons to state-of-the-art diversity-enhancing techniques:

Our method outperforms both approaches because:

Random-Drop indiscriminately discards patches, losing critical local patterns. Wave-SAN focuses on global frequency bands, overlooking localized high-frequency components critical for fine-grained transfer.

3. Limitations and Non-ViT Applicability

Our method is inherently ViT-dependent due to:

• Reliance on patch-based tokenization for spatial and frequency disruptions.
• Self-attention mechanisms that propagate disrupted token relationships.

Experiments with CNNs (DINO-ResNet50) show limited gains (+0.7% average vs. +2.8% for ViT), as CNNs’ overlapped sliding-window convolutions and local inductive biases hinder explicit token continuity control. Notably, we also apply our methods to MLP-Mixers (see answer 5), and we observe 2.8% improvements in target-domain accuracy, which verifies our adaptability to token-based structures.

4. Sensitivity to Clustering Threshold

As shown in Fig. 6b, performance remains stable for thresholds between 10%-35%, with optimal results at 30%. This "sweet spot" balances local pattern retention and global disruption, indicating our method is not sensitive to the specific choice of clustering threshold.

5. Generalization to Other Structures

While our method is designed for ViTs, we test an MLP-Mixer backbone (patch-based architecture):

This demonstrates applicability to other patch-based architectures. However, traditional CNNs (+0.7% average) remain incompatible due to their lack of explicit tokenization and overlapped sliding-window convolutions.

---

We thank you for highlighting these critical points. Revised sections in the manuscript (marked in blue) address all concerns. Your feedback has significantly strengthened our work!

Thank you for your thorough review and constructive feedback. Below are our detailed responses to your concerns:

1. Classification Values in Figure 1

The numerical values in Fig.1 are summarized below:

2. Source-Domain Performance

Ours achieves 66.76% on target (+2.85%), with 96.33% vs Baseline’s 97.78% on source.

3. Meta-Dataset Results

Due to time and resource constraints, we first pretrain on our datasets (miniImagenet), and then we evaluate on parts of the Meta-Dataset under 5-way 5-shot settings:

4. Computation cost

Our method introduces no additional trainable parameters.

During training, the computational overhead (baseline: 126.42s vs. ours: 140.48s per epoch) stems from the clustering-based frequency disruption during training, which is acceptable compared with the gains in cross-domain performance. Moreover, some engineering trick, such as bipartite matching, can accelerate the clustering process, which we leave for future works.

During inference, since our method introduces no additional parameters, inference time matches the baseline exactly with no computational overhead.

5. Justification of Clustering

Algorithm Overview

Step 1: Patchify & Standardize

Image Segmentation

Split an input image into non-overlapping patches:

$$\mathbf{P} \in \mathbb{R}^{N \times p^2 \times C}$$

Normalization

Standardize each patch to zero mean and unit variance:

$$\mathbf{P}_i = \frac{\mathbf{P}_i - \mu_i}{\sigma_i + \epsilon},$$

Step 2: Similarity Computation

Cosine Similarity Matrix: Calculate cosine similarity between all neighbouring patch pairs:

$$\mathbf{S}_{i,j} = \frac{\mathbf{P}_i \cdot \mathbf{P}_j}{\|\mathbf{P}_i\| \cdot \|\mathbf{P}_j\|}$$

Threshold Clustering: Group patches into clusters beyond the threshold

$$\text{Cluster}(i,j) = \begin{cases} 1 & \text{if } \mathbf{S}_{i,j} > \tau \newline 0 & \text{otherwise} \end{cases}$$

Step 3: Cluster Merging

For each image:

• Initialize each image as a singleton cluster.
• Merge neighbouring patches i and j if beyond threshold
• Repeat until no merges occur.

6. References and Other Revisions

Our method is consistent with [1-3] in that we hold the small patterns are easier to transfer than larger ones, but we differ in that (1) our method discusses the transferability of small patterns for ViT while [1-3] are originally designed for CNNs; (2) we provide a simple way to enhance the learning of small patterns by perturbing patches with no additional branches or losses.

We promise we will add discussions to the [1], [2], [3] and increase font sizes for Fig.5 Readability.

7. Balance Between Information Loss and Generalization

To complement the CKA metric, we also use the MMD distance to measure the domain distance, with larger value indicating larger distance.

Moderate disruptions achieve optimal MMD-CKA-accuracy balance. Extreme disruptions harm both domains (low accuracy, high MMD, low CKA). This analysis confirms our method preserves transferable semantics while suppressing domain-specific structures, implying the default patch size is enough to achieve the balance.

Thank you for your constructive feedback and valuable suggestions. Below are our responses to your comments:

1. Source domain performance

As suggested, we have provided the performance on the source domain (miniImageNet) for reference.

Although there is a slight performance trade-off on the source domain, this aligns with our hypothesis: disrupting token continuity prioritizes learning smaller, transferable patterns over domain-specific holistic features. Crucially, the significant gains on target domains (e.g., +6.5% on ISIC) demonstrate the effectiveness of our approach for cross-domain adaptation.

2. Visualization of disrupted images

An anonymous GitHub link ([https://anonymous.4open.science/r/More-visualization/visualizations.png]) with 16 visualized examples of disrupted images in three types is provided. These visualizations illustrate how our method breaks large-scale spatial patterns while preserving fine-grained, domain-agnostic features (e.g., textures, edges), which aligns with our motivation.

We appreciate your insightful feedback and hope these revisions address your concerns. Thank you again for your time and effort in reviewing our work!