Random Registers for Cross-Domain Few-Shot Learning

Thank you for your suggestion.

1. Claims

[1] studies the effectiveness of prompt learning in in-domain data, while we specifically target extreme cross-domain shifts (e.g., natural images to satellite images), where standard prompt tuning fails. As shown below, REAP outperforms [1] by +30% under 5-shot source domain pretraining settings and by +9% under target domain finetuning. To further validate REAP’s effectiveness, we conducted extensive comparisons with prominent prompt tuning methods (e.g., VPT, CoOp, MaPLe) across extreme cross-domain benchmarks.

We can see existing prompts optimize for in-domain adaptation and harm the performance when facing the huge domain gap, while REAP’s random register perturbation explicitly alleviates the domain gap. Similar results in Fig.1, Fig.3, and Fig.4 also verify that the regular prompts are not enough for handling the CDFSL task.

2. Methodological Novelty

While prior works (e.g., VPT) naively add learnable prompts, we are the first to find that it harms the transferability to target domains, and we theoretically design the random register and prove it suppresses domain-specific attention patterns. Based on it, we propose a novel method, REAP, to enhance perturbations on attention maps. This mechanism is novel, as noted by Reviewer zHqd and UKT5: "The proposed method is innovative... new in the CDFSL generalization viewpoint."

3. Sharpness-aware minimization

(1) Performance Validation The experimental results (Tab. 3) demonstrate significant performance improvements (e.g., +4.6% on ISIC), which empirically confirm that the introduced randomness is well-calibrated and beneficial rather than "excessive" or detrimental.

(2) Sharpness Validation The sharpness analysis in Fig. 3b explicitly validates that our method reduces loss sharpness by 75% compared to vanilla VPT, indicating that the random registers enhance flat minima discovery without violating SAM’s core principles. This demonstrates that our design extends SAM’s hypothesis to cross-domain scenarios rather than conflicting with it.

(3) Perturbation magnitude Random registers are initialized with a near-zero magnitude (σ=0.01) and adaptively scaled during training. Visualization in Fig. 5 further confirms that their impact on attention maps is subtle, primarily acting as "regularized perturbations" to suppress domain-specific biases rather than overwhelming the original features. This design is consistent with SAM’s small-perturbation premise while addressing cross-domain challenges.

In all, our design extends SAM to cross-domain settings without violating its core hypothesis.

4. Experimental Completeness

The requested analyses are fully provided in the paper:

Ablation: Simple Masking vs. Random Registers

• Table 3a: Naive masking (no registers) drops accuracy by 5.8%.

VPT Comparison

• Fig. 1 has already shown that VPT harms performance during pretraining while random registers improve model transferability.

Initialization Analysis

• Supp. Fig. 17: Gaussian initialization of the random registers.

5. Clarification on VPT Similarity

While the reviewer notes "similarity with VPT", we explicitly differentiate:

• Motivation (Fig.1): We are the first to find that VPT fails under extreme domain gaps due to absorbing domain information, and propose random registers to solve it.
• Solution: We come up with REAP to replace fixed learnable prompts with random registers during source training.
• Result: REAP outperforms VPT by 4% on distant domains.

We sincerely appreciate your thorough and insightful review of our submission. Your recognition of our work’s innovative integration of random prompts with SAM principles and its practical value in addressing extreme domain gaps is deeply encouraging.

We will keep on polishing our paper in the final version. Thank you again for your appreciation!

We sincerely appreciate the reviewer’s constructive feedback. Below are detailed responses to your questions:

1. Clarification on Learnable Registers (Fig.5)

(1) Source-domain performance and visualization

Learnable registers achieve slightly higher source-domain accuracy by exploiting patterns (over)fitting the source domain, so they can suffer in cross-domain generalization. For example, in Fig.5, birds are often seen with branches in the source domain, and the learnable register focuses on the bird and the branch (domain-specific contextual cues), which is the pattern that is only useful in the source domain (i.e., overfitting to the source domain) but may be useless in target domains. In contrast, random registers force attention to the bird itself, with less overfitting to the source domain, and thus benefit the target-domain generalization.

Our visualization is designed to verify whether the model captures such domain-specific (object-irrelevant) patterns, the generalization is also quantitatively verified by the domain similarity in Fig.4 and sharpness in Fig.3.

(2) Relation to [1]

[1] focuses on the in-domain training, utilizing registers to capture the global information in the in-domain data, thereby reducing the outlier value in attention maps. In contrast, our method focuses on the generalization to target domains, by resisting the overfitting to in-domain data. Our observation is not contrary to the effect of learnable registers in resisting the outlier value in attention maps and is consistent with [1] in finding learnable registers can absorb in-domain information. We take a step further to identify such in-domain information as domain-specific information, which is further handled by our random registers.

2. Sharpness Evaluation on Attention Maps

We choose the attention map for the sharpness experiments because registers majorly interact with other tokens through the self-attention mechanism. In other parts of ViT, such as FFN, each token is processed separately and, therefore can hardly reflect the influence of registers. To verify it, we use LayerNorm and FFN perturbations to report the sharpness, and we can see no significant influences (Δ < 0.01) compared with the attention map.

(Lower values indicate flatter minima and better generalization)

3. Source-domain performance comparison

REAP balances moderate source-domain accuracy with significant target-domain gains as below. A 1.5% source-domain drop (which is acceptable) enables +2.68% target-domain gain, which is a favorable tradeoff for CDFSL.

4. Generalization to Meta-Dataset

Due to time and resource constraints, we first pretrain on our datasets(miniImagenet), and then validate on parts of the Meta-Dataset under the 5-way 5-shot protocol below.

5. Efficiency analysis

1. Training Time: REAP introduces <7% additional training time compared to the baseline (127.08s vs. 118.98s per epoch), attributable to the lightweight random register sampling.
2. Inference Cost: Inference time remains identical to the baseline with no additional overhead. Architectural parity ensures seamless deployment in real-world applications.
3. Parameter Efficiency: Only one learnable standard deviation is added to control register initialization.

This demonstrates REAP’s lightweight design achieves significant cross-domain gains with near-zero parameter and time penalties.

6. Figure Readability Improvement

We promise we will polish our paper in the final version.