Thank you for your valuable comments and for recognizing our thorough experimental validation across baselines and datasets and for providing useful insights into in-domain parameter-efficient pre-training. You may find responses to your questions below:

Q: Further justification for which blocks to unfreeze for ExPLoRA
We agree that block selection is an important component of our method. In section 7, we systematically analyze the properties of patch embeddings from different transformer blocks:

• Blocks that output patch embeddings with low mean/variance eigenvalues inversely correlate with higher linear probing accuracy on predicting patch position
• Linear probing accuracy for image class peaks in final layers (which also have low mean/variance eigenvalues), while patch position accuracy peaks in middle layers
• The eigenvalue patterns and classification accuracy suggest unfreezing block L or L-1 would have highest impact for extended pre-training, which is confirmed in section 6.1.2
• ExPLoRA results in lower mean/variance eigenvalues in the unfrozen block's output, and higher classification and localization accuracies across all ViT layers
• Linear probing performance in section 6.1.2 follows: block 23 > block 22 > block 9 > block 1, matching exactly the trend in the mean eigenvalue plot (figure 3)

Upon your suggestion, we've included additional baselines with blocks [L-1,L] and [1,L-1,L] unfrozen in the expanded version of table 3 linked here. We find that increasing unfrozen blocks with ExPLoRA indeed improves linear probing accuracy, where our highest performing D-ExPLoRA model with blocks [1, L-1, L] unfrozen and LoRA-r64 on the rest achieves 78.04% linear probing accuracy, a 9% improvement over prior SoTA.

Note that while more unfrozen blocks improve performance, each adds to the parameter count. We aim to demonstrate strong results with a limited parameter budget. Our analysis in section 7 provides guidelines for selecting additional blocks to unfreeze when higher parameter budgets are available.

Similar trends as in figures 3-7 held across different datasets, guiding our design choices in other experiments.

Q: Provide a DinoV2 full pre-training baseline for linear probing for fMoW-RGB
We agree that this comparison would be useful, however, we want to note that no such baseline in prior literature exists. The SatMAE/ScaleMAE/CrossScaleMAE models were pre-trained on fMoW-RGB, with which we provide direct comparisons to demonstrate that our efficient in-domain pre-training vastly outperforms existing SoTA pre-training methods in the literature by 8% while using 8x-10x less compute.

Upon your suggestion, we have trained a full DinoV2 baseline on fMoW-RGB in the revised Table 3 linked above. This required ~1200 GPU hours, more than 12x the compute required for ExPLoRA and failed to match its linear probing performance. This likely suggests that much more compute and time is required to pre-train a DinoV2 on fMoW-RGB.

Q: The paper expands on existing methods
With ExPLoRA, we demonstrate a highly effective alternative to expensive full pre training for new image domains. While it combines existing methods, we would like to cite peer conference NeurIPS guidelines, which mentions that demonstrating the effectiveness of combining existing techniques can provide substantial research value.

Thank you again for your time in reviewing our work. We look forward to addressing any follow-up questions you may have. If your concerns are addressed, we kindly ask that you reconsider your score.

Thank you for your valuable feedback. We appreciate your recognition of our extensive, well-designed experimental validation and for ExPLoRA’s improved performance over baselines. Here are our responses:

Q: Can you provide qualitative analysis that illustrates the difference between pre-trained vs ExPLoRA pre-trained embedding spaces?
Section 7 contains our analysis of patch embeddings output by ViT blocks across different models (ExPLoRA, DinoV2, MAE, SatMAE), revealing important quantitative differences. Please also see our response to reviewer Fy1r linked here.

Figure 9 (appendix B.8) also qualitatively analyzes attention maps across different ViT blocks of these models. ExPLoRA concentrates attention more tightly on central objects, especially in final layers. This correlates with lower mean eigenvalues and higher positional/classification accuracies from linear probing.

Q: Include discussion of test-time adaptation (TTA) methods.
Thank you for mentioning these methods. We have updated our related work section to include these references which will be present in our camera-ready revision. Our work has key differences with TTA methods:

• TTA methods make a critical assumption: the label space Y is shared between the source and target domains. This makes [1,2,3] incompatible with unsupervised pre-trained backbones when domains have different label sets (common in many settings, including ours). Unsupervised pre-training techniques like DinoV2 or ExPLoRA don't assume specific downstream label sets, requiring some supervised adaptation (linear-probing/PEFT) to parameterize $p^{(\tau)}_T(y|x)$ for downstream task $\tau$ in target domain $T$. TTA methods could then be applied on top of ExPLoRA backbones just as with any ViT. This means that ExPLoRA is complementary with TTA methods rather than a competing method.
• [3,4] are tailored for CNNs, not applicable to our work with ViTs, which consistently outperform CNNs on our datasets (eg: fMoW-RGB, fMoW-Sentinel, temporal images etc. as verified in prior work).
• [2] uses visual-prompt tuning (VPT) for ViTs, but lacks a public codebase. We include multiple recent VPT baselines in Table 1 (VPT [5], GVPT [6], SA2VP [7]). ExPLoRA outperforms all by ~2%, including a pre-training baseline with VPT in Table 3.

Q: ExPLoRA requires a non-trivial amount of additional training compared to fine-tuning a natural-image pre-trained model for an improvement in performance (Figure 7)
We would like to refer you to our response to reviewer W3i1, linked here.

To summarize, we realize that our plot in figure 7 paints an incomplete picture. ExPLoRA is a pre-training method, providing an alternative to domain-specific full pre-training. While extended pre-training + fine-tuning requires more GPU hours than directly fine-tuning, the extended pre-training phase is amortized across multiple downstream tasks, since the same ExPLoRA model can be re-used for initialization. Further, it can be used for methods such as feature extraction, linear probing etc. that a task-specific fine-tuned model cannot do. For linear probing, ExPLoRA outperforms DinoV2/prior SoTA models by >8%. This difference is not captured via figure 7, since that only compares with fine-tuning.

Thus, the fairer comparison is with domain-specific full-pretraining. Here, ExPLoRA requires 8x-10x less compute, 16x fewer parameters and achieves similar or higher performance. Our expanded Table 1 demonstrates that ExPLoRA is vastly more efficient than other pre-training methods.

Figure 7 shows only fMoW-RGB performance. On fMoW-Sentinel (larger domain shift to multi-spectral data), full-finetuning a natural-image baseline shows a 9% performance gap with ExPLoRA (row 1, Table 4). Combined, these are significant results since ExPLoRA achieve/surpass fully pre-trained baselines using ~8-10x less compute and 10-16x fewer parameters.

Thank you for your feedback which has strengthened our work. If your concerns are addressed, we kindly ask that you reconsider your score.

References:
[1] Tent: Fully Test-time Adaptation by Entropy Minimization, ICLR 2021.
[2] Visual Prompt Tuning for Test-Time Domain Adaptation, arxiv 2210.04831.
[3] DomainAdaptor: A Novel Approach to Test-time Adaptation, ICCV 2023.
[4] Convolutional Visual Prompt for Robust Visual Perception, NeurIPS 2023.
[5] Visual prompt tuning. ECCV 2022.
[6] Improving visual prompt tuning for self-supervised vision transformers. ICML 2023.
[7] SA²VP: Spatially Aligned-and-Adapted Visual Prompt. AAAI 2024.

Thank you for your feedback and recognition of the empirical performance and novelty of using parameter-efficency for pre-training in ExPLoRA. Here are our responses:

Q: Is PEFT needed in either pre-training or fine-tuning beyond memory savings?
Our primary motivation for parameter-efficient techniques is efficiency:

• Reduced memory footprint provides significant compute savings with larger batch sizes and/or fewer GPUs, and since gradients for <10% of parameters need to be stored and propagated. This makes ExPLoRA much faster than full pre-training (see revised Table 1 and response to reviewer W3i1)
• Memory savings enable using models such as ViT-L or ViT-G, otherwise inaccessible with constrained GPU budgets
• While not our main motivation, constraining the parameter space mitigates catastrophic forgetting from the natural image domain. ExPLoRA outperforms fine-tuning from MAE/DinoV2 or domain-specific SatMAE models by >1% in Table 1, and full continual pre-training methods (eg: GFM) by 5-7%. For linear probing, this gap widens to 8%, suggesting benefits from both original and extended pre-training.

We've included a baseline for fully pre-training a ViT-L from scratch on fMoW-RGB with DinoV2 objective in our revised Table 3. This shows full pre-training cannot match ExPLoRA while requiring >10x compute and >16x parameters.

Lastly, our parameter-efficient pre-training is orthogonal to fine-tuning method choice. ExPLoRA preserves the ViT architecture, enabling any fine-tuning approach. While we primarily use PEFT for efficiency, ExPLoRA also outperforms fully fine-tuned SatMAE and ScaleMAE models by 1.4% (Table 1).

Q: Are there baselines that fully fine-tune natural-image pre-trained models on the domain of interest?
We use PEFT primarily for efficiency. At your request, we've included fully fine-tuning DinoV2 and ExPLoRA models on fMoW-RGB alongside the SatMAE paper's fully fine-tuned MAE result in our expanded Table 1.

Full fine-tuning is viable with sufficient resources. ExPLoRA models benefit from it just as well as other baselines and outperform them. However, LoRA-r8 PEFT is much cheaper and equally effective. Table 4 includes a comparison where MAE is directly fully fine-tuned on fMoW-Sentinel, showing poor performance (>9% gap) due to the large domain shift.

We reiterate that we don't aim to replace full or PEFT fine-tuning, both complementary to ExPLoRA. Instead, we provide an efficient alternative to full, domain-specific pre-training.

Q: Relation to “self-supervised pre-training improves...” (HPT)
We will update our paper to include this reference. Key differences between ExPLoRA and HPT:

• HPT uses two full backbone pre-training phases on different datasets. ExPLoRA uses a single efficient extended pre-training phase that generalizes to multiple downstream tasks.
• While HPT offers tuning only batch norm layers in the second phase, it centers around full-rank pre-training, like GFM. ExPLoRA uses parameter-efficient techniques that reduce compute by >10x and parameters by >16x while achieving similar or higher performance.
• HPT studies MoCo with ResNet-50. ExPLoRA addresses ViTs with different SSL objectives (DinoV2, MAE), demonstrating broader applicability.

ExPLoRA goes beyond "modernizing HPT" by demonstrating parameter-efficient extended pre-training's value via strong performance at reduced costs across diverse datasets, SSL objectives, and tasks.

Q: Relation to “Finetune like you pretrain...” (FLYP)
We agree this reference is also valuable. As above, there are important differences with ExPLoRA:

• FLYP is a fine-tuning technique while ExPLoRA is a pre-training technique
• FLYP focuses on contrastive SSL methods with ResNet. ExPLoRA is compatible with any ViT SSL objective (as we show with DinoV2, MAE).

FLYP does further justify continuing pre-training with the same loss function (eq. 5). We'll update our final manuscript to include this reference.

Q: Show additional evidence in Table 4 ,5 for D -[L}-r32 and M-[L]-r32 in Tables 7-9.
We have now included MAE results in tables 7-9, linked here, which underperforms our SoTA Dino-ExPLoRA.

For tables 4-5, we use MAE SSL from SatMAE. There's no DinoV2 modification for multi-spectral/temporal data in prior work. Creating one would require modifying the ViT and Dino SSL mechanism for non-RGB temporal/multi-spectral sequences, which is out of scope. Our goal is to use the SatMAE SSL architecture with MAE weights, showing ExPLoRA matches/outperforms full pre-training (0.67%) and vastly outperforms full MAE fine-tuning on multi-spectral data by 8.54% (Table 2).

We hope these answers address your questions. If your main concerns are resolved, we kindly request you reconsider your score.

Thank you for your detailed and insightful feedback. We appreciate your recognition of ExPLoRA’s novelty in combining fundamental ideas and its strong empirical performance across datasets.

Q: Discussion of required compute of ExPLoRA vs fine-tuning baselines
We agree that more details on compute requirements is needed, which we have added to the main text. Upon reviewing the feedback, we'd like to clarify:

• ExPLoRA is a pre-training method and is meant to provide an efficient alternative to domain-specific full or continual pre-training (eg: SatMAE, ScaleMAE, GFM, etc.). These methods require at least 8x more compute than ExPLoRA pre-training and achieve lower or similar performance to ExPLoRA across different domain gaps (eg: RGB, multi-spectral, temporal data).
• Directly comparing compute requirements for fine-tuning vs extended pre-training + fine-tuning paints an incomplete picture. The ExPLoRA pre-training phase is functionally different from fine-tuning a natural-image pre-trained model. A pre-trained checkpoint serves multiple purposes- feature extraction, linear-probing, fine-tuning on various downstream tasks etc. In contrast, compute used for supervised fine-tuning yields a model usually suited for just one task (eqs 2, 3). Pre-training compute is amortized across multiple downstream tasks, and would be double counted if included in the fine-tuning budgets for each specific task.
• We realize now that Figure 7 in section B.2 is incomplete. It only compares the compute for fine-tuning a natural image model (eg: DinoV2) vs. pre-training + fine-tuning for ExPLoRA, for the fMoW-RGB task. Crucially, the pre-training + fine-tuning compute required for full domain-specific pre-training (eg: SatMAE, ScaleMAE) is missing. Further, for different domains (eg: multi-spectral satellite images), the gap between fine-tuning a natural-image model (eg: MAE) and domain-specific pre-training is much larger than for fMoW-RGB (i.e. 6%, from Table 4). Lastly, to reiterate, a directly fine-tuned DinoV2 on fMoW RGB is less flexible than an ExPLoRA pre-trained model, as the latter can be used as any pre-trained model- for feature extraction, probing, or further task-specific fine-tuning.

We've expanded Tables 1 and 4 with compute details linked here. Note that:

• We only need to allocate 100 GPU hours to ExPLoRA pre-training to achieve the reported gains. For fairness, we use ~220 GPU hours for fine-tuning across models.
• Providing extra 100 GPU hours to fine-tuning non-ExPLoRA baselines (for an equitable total compute comparison) doesn't improve top-1 accuracy.
• VPT techniques significantly increase fine-tuning compute by extending token sequence length.

Q: The DinoV2 baselines reach close to their peak performance in 30 GPU hours while ExPLoRA requires 420 GPU hours
This is not a direct comparison, since Figure 7 provides different fine-tuning variants. The red curve is DinoV2 fine-tuned with LoRA-r8, which we also use for ExPLoRA models. The orange and purple curves have blocks unfrozen with LoRA-r64, representing a similar parameter budget to ExPLoRA pre-training. The orange curve reaches peak performance at ~260 GPU hours, by which point all ExPLoRA models have surpassed it.

Q: Experimental results on larger backbone sizes beyond ViT-L
Thank you for the suggestion. As you mention, we compare with ViT-B and ViT-G in Table 15 (appendix B.5). ViT-G experiments are more expensive (>3x parameters vs ViT-L), especially on academic compute budgets. Most prior work uses ViT-L as their backbone, so we maintain this architecture for fair comparison in our main experiments.

Q: More detailed descriptions of datasets
Thank you- we'll include further dataset details in the appendix. For fMoW:

• fMoW-RGB: 363k training, 53k test data points
• fMoW-Sentinel: 713k training, 85k test data points

Q: lines 760-761, parameter budget clarification
You're right - it should be 18.7M.

Q: ln 765: Is it 320 GPU hours or 420?
Within 320 total hours (pre-training + fine-tuning), ExPLoRA achieves >1% improvement over the DinoV2 baseline. The 420 hour limit was added as a buffer to demonstrate earlier convergence.

Q: Eq.6 $\Theta(r)$ notation
Thank you- the notation should be $\Theta$ to refer to an unconstrained parameter space (regular full-rank weights).

Q: iWildcam linear probing result
The unfrozen ViT block likely overfit to the training data due to the small domain gap. This effect disappears with PEFT (showing improvement over DinoV2) since all ViT attention layers' Q,V matrices are tuned with LoRA.

Q: Row number clarifications and "blk" notation in figures 3-6
Thank you! We've corrected these mistakes in our manuscript.

Thank you for your thorough feedback which has improved our paper. Please let us know if you have follow up questions. If your main concerns are resolved, we kindly request that you reconsider your score.