ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts

Thank you for your insightful feedback and recognition of ExPLoRA's value in efficient foundation model creation, as well as our comprehensive experimental results. We also appreciate that you found our ablation study to be insightful for parameter-efficient pre-training. Incorporating your feedback has significantly improved our paper. You may find responses to your concerns below:

Q: Is starting with natural-image foundation models necessary given existing geo-spatial models?
This is a fair question. While domain-specific foundation models exist, new domains and datasets continually emerge. ExPLoRA demonstrates that adapting natural-image foundation models from frontier labs can outperform fully pre-trained domain-specific models (e.g., SatMAE, ScaleMAE) while using significantly fewer resources. This is valuable because it enables researchers and practitioners to create effective foundation models for new domains without expensive from-scratch pre-training.

Q: Comparisons against GFM and GDA should be included
Thank you for this valuable suggestion. We agree that GFM [1] is a relevant prior work, and have included results from GFM in our revised Table 1.

GDA [2] was published in June 2024, which is just short of the July 1 2024 cutoff that ICLR considers for concurrent work and after we posted a pre-print of this work in June. Even so, we have worked to include results from GDA in Table 1.

ExPLoRA outperforms these works by ~6%, with several key advantages:

• Parameter Efficiency: Unlike GFM, ExPLoRA doesn't require training the full ViT backbone
• Model Flexibility: ExPLoRA works with and evaluates non-MAE methods (e.g., DinoV2), achieving SoTA on remote sensing benchmarks
• Architectural Preservation: Unlike GDA's non-mergeable adapters that modify architecture (due to the scaling vector) and can increase inference latency with higher ranks, ExPLoRA's LoRA weights merge into Q,V matrices
• Fine-tuning Freedom: ExPLoRA allows varying LoRA ranks between pre-training and fine-tuning, and supports any PEFT method. GDA requires using pre-trained adapters during fine-tuning
• Broader Applicability: We handle larger datasets (fMoW-RGB, fMoW-Sentinel) and diverse domains beyond remote sensing (i.e. WiLDS)
• Systematic Block Selection: Our analysis in Section 6.3 provides clear insights into which transformer blocks encode local vs. global information, offering a principled approach to block selection

We have also summarized and discussed these differences in an expanded related work section in the revised appendix A.1.

Q: This submission combines known approaches with potentially limited ICLR value
We understand this concern– while individually LoRA and unfreezing blocks are existing strategies, ExPLoRA’s novelty is in demonstrating the substantial value of combining them for extended pre-training:

• We demonstrate that selectively combining full-rank tuning of ViT blocks with LoRA is both more parameter-efficient and effective than prior continual pre-training approaches
• We achieve SoTA on fMoW (a key foundation model benchmark) and show significant improvements in linear probing, indicating strong feature extraction capabilities
• Unlike GFM/GDA, our approach generalizes beyond masked-image modeling - our strongest results use DinoV2, challenging the MAE-based paradigm for remote sensing
• We demonstrate generality across multiple domains via the WiLDS benchmark

Q: Distribution questions about $p_{D_T}(x)$ and $p_{d_T}(x, y)$
Good question. Yes, our formulation indicates that a subset of $D_T$ datasets are labeled, optionally allowing unsupervised pre-training on all unlabeled domain images. The distributions $p_{D_T}(x)$ and $p_{d_T}(x, y)$ are indeed similar with respect to x as they share domain $T$.

Q: Inconsistent ScaleMAE comparisons across tables
Thank you for pointing this out. We've added ScaleMAE (0.8M parameters) results to Table 1, showing it underperforms ExPLoRA-DinoV2. ScaleMAE is absent from Tables 4-5 as no pre-trained model exists for fMoW-temporal/Sentinel. We include ScaleMAE baselines in Table 6 as they evaluated on SpaceNet/Resisc-45.

Q: Have you also run experiments showing how MLP adaptation without attention adaptation would perform, for Table 3 (ablation study)?
Thank you for the suggestion. We have included this experiment in our latest revision in Table 3. We find that attention layers are more receptive to low-rank tuning, with MLP-only adaptation showing reduced representation learning capacity.

Q: In Fig. 2, where can I find U in the figure, which is described in the image caption.
Thank you for pointing this out. U corresponds to the unfrozen blocks. We have updated the figure to be clearer.

References:
[1] Towards geospatial foundation models via continual pretraining. ICCV 2023.
[2] Parameter Efficient Self-Supervised Geospatial Domain Adaptation. CVPR 2024.

Thank you for your thoughtful insights and recommendations. We appreciate your acknowledgement of our extensive SoTA empirical results as well as the method’s soundness. As for your concerns, which we have worked to address, please see our responses below:

Q: What are the differences between ExPLoRA and LoRA?
This is a fair question. ExPLoRA extends on LoRA but differs in both purpose and approach. While LoRA is a fine-tuning method for downstream tasks, ExPLoRA introduces parameter-efficient extended unsupervised pre-training. Though we use LoRA-style adapters for Q,V matrices, we combine this with selective block unfreezing - a combination that proves significantly more effective and parameter-efficient than either using LoRA alone or LoRA with higher ranks (Table 3, ablation study).

Q: How does ExPLoRA compare to UDA methods and perform on UDA benchmarks?
Great question! ExPLoRA and traditional UDA methods address different stages of domain adaptation. UDA methods require labeled source domain data while adapting to unlabeled target data, focusing on the downstream supervised task. In contrast, ExPLoRA creates domain-adapted backbones without requiring any labels.

To clarify using notation from section 4 of our paper: UDA methods assume access to a labeled distribution for the source domain $D_S$ given by $p_{D_S}(\mathbf{x}, \mathbf{y})$ and an unlabeled distribution for the target domain $D_T$ given by $p_{D_T}(\mathbf{x})$. Datasets such as VisDA2017 or OfficeHome assume shared label sets $Y$ between $D_S$ and $D_T$, i.e., $Y_{D_S} = Y_{D_T}$. ExPLoRA's setting is different. We only assume access to weights $W_{D_S}$ from a model pre-trained via unsupervised learning on $p_{D_S}(\mathbf{x})$, without requiring direct access to the source distribution $p_{D_S}(\mathbf{x})$ (which may not have labels, unlike in UDA). Further, we don't place any restrictions on the label set $Y$ for the different domains. Thus, ExPLoRA differs from traditional UDA considered in the works you have cited, and so we don't label our method as "UDA". Thank you for prompting this contextualization- we will add a discussion in our paper (appendix A.2) to clarify this.

Rather than competitors, UDA methods can be viewed as complementary to ExPLoRA - they can benefit from initialization with ExPLoRA's pre-trained weights instead of standard natural-image pre-training.

We will add some additional experiments using UDA approaches on top of ExPLoRA backbones to demonstrate their compatibility.

Q: How does ExPLoRA compare to SoTA PEFT Methods?
Thank you for suggesting these additional works. ExPLoRA is designed to complement rather than replace PEFT methods. Since ExPLoRA preserves the ViT architecture, any PEFT method can be used for subsequent fine-tuning. Nonetheless, we are still interested in how ExPLoRA and PEFT methods perform in tandem. CLIP-LoRA applies low-rank matrices on key, value, and query matrices of the text and vision encoders with ranks of 2 [7]. We have already evaluated the analogue of this for vision where we do not have access to text. That is, we fine-tuned with LoRA, using different ranks and applying low-rank matrices to different subsets of key, value, and query matrices of the vision encoder.

Please also see our overall response for a discussion of all recent PEFT methods. While there are many PEFT techniques and comparing with all of them will be infeasible given time constraints, we do our best to select SoTA baselines from a variety of PEFT families (eg: SAVP, Gated-VPT for visual-prompt tuning, Adapter+ for adapters, AdaLoRA for LoRA-based methods, BOFT for multiplicative methods, GDA for scaled-low-rank adapters/side-tuning). If there are crucial works missing in our comparison, please let us know.

Dear reviewer 47Sp,

Thank you very much for your time and consideration in providing us with your review that has improved our work.

As today is the final day to make revisions to the pdf of the paper, please let us know if you have remaining concerns that need to be addressed. If your concerns are resolved, we kindly request that you reconsider your score to reflect that.

-Authors

Thank you again for your time and valuable feedback. While ExPLoRA is not a traditional unsupervised domain adaptation (UDA) method, it can serve as an initialization for ViT-based UDA approaches. As promised in our initial rebuttal response, we demonstrate this compatibility below.

Classification accuracy (%) on VisDA-2017 (validation). Results marked with * use our reproduced results. Using ExPLoRA initialization improves UDA performance compared to standard ImageNet initialization.

For context, The VisDA2017 dataset contains 152,297 training and 55,388 validation images across 12 object classes representing a synthetic-to-real domain shift: training images are synthetically rendered 3D models under various lighting conditions, while validation images come from MS-COCO.

The table above shows ExPLoRA's effectiveness when combined with TVT [1], a state-of-the-art UDA method. Using ExPLoRA D-[12]-r64 (DinoV2-initialized ViT-B with last layer unfrozen and LoRA-r64 elsewhere) pre-trained on both synthetic and real domains, we outperform traditional ImageNet-21k initialization by 1.5-3% while achieving more balanced per-class accuracy. With ExPLoRA initialization, TVT's performance rises to match recent SoTA methods, a significant improvement over its original results. Most notably, we surpass DinoV2 initialization by ↑6%, demonstrating that ExPLoRA's unsupervised initialization matches state-of-the-art UDA methods that rely on supervised ImageNet-21k pre-training.

These results demonstrate the benefits of ExPLoRA as an unsupervised pre-training method for new domains, as well as its wide compatibility not only with PEFT (see Table 1 of our main paper), but also with UDA. We will be including these results in our paper, thanks to your suggestions.

Please let us know if you have any further questions or suggestions. If these are resolved, we kindly request that you reconsider your score.

References:
[1] TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation, WACV 2023.
[2] Safe Self-Refinement for Transformer-based Domain Adaptation, CVPR 2022.
[3] CDTRANS: Cross-Domain Transformer For Unsupervised Domain Adaptation, ICLR 2022.
[4] Patch-Mix Transformer for Unsupervised Domain Adaptation: A Game Perspective, CVPR 2023.

Dear Reviewer 47Sp,

Thank you for your thoughtful review of our paper. As we approach the end of the discussion period, we would greatly appreciate if you could review our rebuttal responses and let us know if we have adequately addressed your concerns.

If you feel that our clarifications and additional experimental results have resolved your initial concerns, we kindly request that you consider updating your evaluation accordingly. Your final assessment is valuable to us and to the broader review process.

We understand you are likely managing many responsibilities, and we truly appreciate your time and attention throughout this process.

Best regards,
Authors

Thank you for your thoughtful feedback. We appreciate your consideration of our rebuttal and would like to clarify several key points.

Value of ExPLoRA's Technical Contribution
While ExPLoRA combines existing strategies (LoRA and selective unfreezing), its novelty lies in demonstrating an effective approach for parameter-efficient pre-training of vision transformers for new domains. This represents a significant finding with immediate practical impact.

Our contributions, with further detail in the full rebuttal response, include:

1. A novel approach combining LoRA with selectively unfreezing ViT blocks for continual pre-training on new visual domains. ExPLoRA works with popular self-supervised learning methods (DinoV2, MAE) and preserves the ViT architecture, further enabling compatibility with any downstream method (PEFT, UDA etc.)
2. Extensively verified state-of-the-art performance across multiple large datasets and challenging domain shifts (eg: satellite, medical, wildlife, agricultural, synthetic imagery) while using <10% trainable parameters.
3. Providing systematic analysis of information encoding in ViT layers, offering clear guidelines for block selection during pre-training

As noted in peer conference NeurIPS guidelines, demonstrating the effectiveness of combining existing techniques can provide substantial research value. Our extensive experiments confirm this - ExPLoRA outperforms both from-scratch pre-training and recent PEFT methods across multiple domains while using significantly fewer parameters. This is particularly valuable given the increasing costs of pre-training foundation models for new domains.

Reviewers 8d6C and ZQZA specifically highlighted these strengths, noting ExPLoRA's value for "parameter-efficient unsupervised pre-training" and its "strong results on multiple domains and benchmark datasets."

Experimental Comparisons
We appreciate your suggestion regarding UDA benchmarks. However, as detailed in our previous response and appendix A.2, ExPLoRA addresses a fundamentally different problem than traditional UDA. While UDA methods require labeled source domain data, ExPLoRA enables unsupervised domain adaptation using only pre-trained weights. Thus, ExPLoRA is not a UDA method and comparison with UDA methods is not the main focus of this paper.

Our experiments focus on demonstrating ExPLoRA's superior performance against pre-training from scratch, continual pre-training and PEFT across challenging real-world scenarios, such as:

• Multiple satellite image modalities: high-res RGB, low-res multi-spectral, and temporal sequences
• Various downstream tasks: classification, segmentation, detection
• Different application domains: medical, wildlife, and agricultural via WiLDS benchmark
• Synthetic domain transfer through VisDA2017

Upon your valuable recommendation and given the tight timeline of the ICLR rebuttal, we also included results on VisDA2017 to further demonstrate ExPLoRA’s compatibility with UDA methods and on synthetic domain data. The VisDA2017 results are noteworthy - ExPLoRA initialization elevates TVT's performance to match SOTA UDA methods, demonstrating its value even in traditional domain adaptation settings. This is a novel finding that further validates ExPLoRA's effectiveness.

These comprehensive experiments across diverse, large-scale datasets provide strong evidence of both ExPLoRA's soundness and its practical utility. The breadth and depth of our experimental validation offers practitioners a high degree of confidence in applying our method to real-world domain adaptation challenges.

Thank you for your valuable feedback, recognition of ExPLoRA’s novelty in parameter-efficient post pre-training for visual data, and for our extensive and logical experimental results. We appreciate your vote of confidence in our method, and have worked to incorporate your suggestions, found below:

Q: Figure 1 seems to express the difference between full fine-tuning and PEFT in a more accurate form.
Could you please clarify which specific aspects of Figure 1 or 2 need improvement? We welcome concrete suggestions to enhance their clarity.

Q: Comparison with more recent PEFT methods.
We include several recent SoTA PEFT methods in Table 1 (BOFT, GVPT, SA^2VP, AdaLoRA). We've expanded our experimental comparisons to include:

• Parameter Efficient Self-Supervised Geospatial Domain Adaptation, CVPR 2024.
• Towards geospatial foundation models via continual pretraining. CVPR 2023
• Adapters Strike Back (CVPR 2024)
• Mona [11], follow-on work to LoRand [10] in CVPR 2023, (to be added to Table 1).

The other cited works are either already outperformed or superceded by prior works we have included experiments from. E.g.,

• SA²VP outperforms Pro-Tuning [8] on CIFAR-100, Oxford flowers.
• BOFT [4] and AdaLoRA [7] outperform SSF [9] on VTAB-1K.
• “Adapters Strike Back” [3] includes comparisons with the most recent adapter-based methods, outperforming [9, 12, 13] on VTAB.
• Mona [11] outperforms LoRand [10] and other adapter methods [12] on COCO and other benchmarks

Q: Analysis of LoRA's limitations and proposing a novel PEFT method
Thank you for your suggestion. We want to clarify that our primary contribution is demonstrating the effectiveness of parameter-efficient unsupervised pre-training for domain adaptation to visual data. We are the first to show that combining LoRA with selective ViT block unfreezing creates strong foundation models for new domains at a fraction of the computational cost. This addresses a significant challenge in foundation model development - while frontier labs and organizations invest substantial resources in developing natural-image foundation models like DinoV2, most practitioners cannot afford to pre-train new models for each domain. ExPLoRA enables direct adaptation of these pre-trained models to new domains without expensive from-scratch pre-training.

Moreover, our analysis in Section 6.3 provides insights into ExPLoRA's effectiveness:

• We evaluate patch embeddings for both local information (patch position prediction) and global information (image classification). We demonstrate ExPLoRA enhances both types of information in patch representations
• Through spectral analysis, we identify a strong correlation between patch feature map eigenvalues and position accuracy, providing a systematic approach for selecting blocks to unfreeze. E.g., for classification tasks, target layers that have low eigenvalues and high global information (i.e. class accuracy).

While a theoretical investigation of LoRA and full-rank tuning interactions would be valuable, our current focus is on empirical validation of ExPLoRA's effectiveness across multiple realistic and challenging domain shifts. We demonstrate SoTA results on several benchmarks while maintaining computational efficiency.

Please let us know if you have any recommendations for experimental analysis that would further strengthen our work, and we will be happy to incorporate them.

References:
[1] Towards geospatial foundation models via continual pretraining. ICCV 2023.
[2] Parameter Efficient Self-Supervised Geospatial Domain Adaptation. CVPR 2024.
[3] Adapters Strike Back. CVPR 2024.
[4] Parameter-efficient orthogonal finetuning via butterfly factorization. arXiv:2311.06243 (2023).
[5] Improving visual prompt tuning for self-supervised vision transformers. ICML 2023.
[6] SA²VP: Spatially Aligned-and-Adapted Visual Prompt. AAAI 2024.
[7] AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv:2303.10512 (2023).
[8] Pro-tuning: Unified prompt tuning for vision tasks. NeurIPS 2022.
[9] Scaling & shifting Your Features: A New Baseline for Efficient Model Tuning. NeurIPS2022. [10] 1% vs 100%: Parameter-efficient low rank adapter for dense predictions. CVPR 2023.
[11] 5%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks. arXiv:2408.08345 (2024).
[12] AdaptFormer: Adapting vision transformers for scalable visual recognition. NeurIPS 2022.
[13] Sensitivity-aware visual parameter-efficient tuning. _ ICCV 2023_ .

Thank you very much for considering our rebuttal and for increasing your score. We have incorporated your suggestions for relevant citations into our revised rebuttal draft. We value your vote of confidence in our work-- your review has been very helpful to improve our paper.

Thank you for your very insightful suggestions and feedback. We appreciate your recognition of ExPLoRA’s value in significantly reducing computational costs via unsupervised extended pre-training, its compatibility with established and future ViT models, and our extensive experiments that demonstrate its SoTA performance. Please find responses to your concerns below:

Q: What is the computational cost trade-off between supervised fine-tuning and extended pre-training?
Thank you for prompting this analysis– this is a great question. We analyze this in detail in new appendix section B.2, (figure 6), examining two key aspects

• Fixed parameter budget: Does equivalent supervised fine-tuning match ExPLoRA + fine-tuning?
• Fixed compute budget: Can supervised fine-tuning alone achieve similar performance with equal GPU-hours?

For both scenarios, the answer is no - ExPLoRA's extended pre-training provides gains that supervised fine-tuning alone cannot match. Even with the same configuration (unfrozen blocks + high-rank LoRA), direct fine-tuning falls short by ≥0.9% in top-1 accuracy. Increasing the parameter budget by unfreezing more blocks doesn't close this gap. Here, the computational budget is measured via total GPU hours allocated to pre-training + fine-tuning.

Moreover, ExPLoRA provides unique benefits beyond methods that only support fine-tuning:

• Can leverage large unlabeled domain-specific datasets (eg: unlabeled satellite, medical etc. imagery)
• Creates strong ViT feature extractors (7%+ improvement in linear probing, over prior SoTA such as SatMAE [1], ScaleMAE [2] etc., Table 2) which can be used for unsupervised image retrieval or compression
• Serves as a “foundation model”. i.e. can be used as an initialization for other downstream tasks (Tables 6, 11 show SoTA on Resisc-45 [3] and EuroSAT [4] without additional pre-training)

Q: How necessary is extended pre-training with the ExPLoRA configuration? Can we use the same configuration (LoRA + unfreezing blocks) directly for fine-tuning?
Thank you for this suggestion. Our experiments in section B2 show that using the same configuration (LoRA + unfrozen blocks) directly for fine-tuning hits a lower accuracy ceiling compared to fine-tuning with ExPLoRA weights. This demonstrates the value of our extended pre-training phase.

Q: Extending the model to multi-modal models like CLIP and zero-shot settings like using natural language for zero-shot understanding in downstream tasks
This is a valuable suggestion. While ExPLoRA is applicable to CLIP, it would require paired image-caption data for new domains, making it a supervised setting. However, in this paper we choose to focus on unsupervised pre-training on image data (eg: DinoV2, MAE etc.). We leave valuable exploration of multi-modal extensions to future work.

Q: Can we change the unsupervised training objectives in the extended pertaining stage?
This is a very interesting question. While it's possible to mix objectives during extended pre-training, it requires careful consideration. Our experiments with using MAE weights for DinoV2 pre-training showed suboptimal results compared to continuing the original objective. While there may be better objective combinations, our current goal is to demonstrate the value of unsupervised extended pre-training for existing visual foundation models (e.g., MAE, DinoV2).

References:
[1] SatMAE: Pre-training transformers for temporal and multi-spectral satellite imagery, NeurIPS 2022.
[2] ScaleMAE: A scale-aware masked autoencoder for multiscale geospatial representation learning, ICCV 2023.
[3] Remote sensing image scene classification: Benchmark and state of the art, CVPR 2017.
[4] Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification, IEEE 2019.

Dear reviewer 7P1n,

Thank you very much for your time and consideration in providing us with valuable suggestions that have improved our work.

As today is the final day to make revisions to the pdf of the paper, please let us know if you have remaining concerns that need to be addressed. If your concerns are resolved, we kindly request that you reconsider your score to reflect that.

-Authors

Dear Reviewer 7P1n,

Thank you for your thoughtful review of our paper. As we approach the end of the discussion period, we would greatly appreciate if you could review our rebuttal responses and let us know if we have adequately addressed your concerns.

If you feel that our clarifications and additional experimental results have resolved your initial concerns, we kindly request that you consider updating your evaluation accordingly. Your final assessment is valuable to us and to the broader review process.

We understand you are likely managing many responsibilities, and we truly appreciate your time and attention throughout this process.

Best regards,
Authors

References:

[1] SatMAE: Pre-training transformers for temporal and multi-spectral satellite imagery, NeurIPS 2022.
[2] Parameter Efficient Self-Supervised Geospatial Domain Adaptation. CVPR 2024.
[3] Adapters Strike Back. CVPR 2024.
[4] Parameter-efficient orthogonal finetuning via butterfly factorization. arXiv:2311.06243 (2023).
[5] Improving visual prompt tuning for self-supervised vision transformers. ICML 2023.
[6] SA²VP: Spatially Aligned-and-Adapted Visual Prompt. AAAI 2024.
[7] AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv:2303.10512 (2023).
[8] Pro-tuning: Unified prompt tuning for vision tasks. NeurIPS 2022.
[9] Scaling & shifting Your Features: A New Baseline for Efficient Model Tuning. NeurIPS2022.
[10] 1% vs 100%: Parameter-efficient low rank adapter for dense predictions. CVPR 2023.
[11] 5%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks. arXiv:2408.08345 (2024).
[12] AdaptFormer: Adapting vision transformers for scalable visual recognition. NeurIPS 2022.
[13] Sensitivity-aware visual parameter-efficient tuning. ICCV 2023.

Thank you for your feedback, recognition of ExPLoRA's novelty as a parameter-efficient pre-training method, and appreciation of our extensive empirical results. You may find responses to your concerns below:

Q: Are the number and index of unfrozen layers sensitive to different datasets and domains?
This is a good question. In section 6.3, we analyze each block's sensitivity to ExPLoRA's extended pre-training through spectral analysis and linear-probing for local vs global information. We found consistent results across datasets - block 23 consistently showed the highest propensity to improve global information in output feature vectors. To emphasize parameter efficiency, we limited unfrozen blocks to 1-2 across all experiments. For most datasets except fMoW-Sentinel, unfreezing 1 block was sufficient to achieve ExPLoRA's benefits while maintaining low parameter cost.

Q: Was LoRA used as a pre-training baseline?
Yes - we already include these results in Table 3, rows 3 and 4. Using only LoRA, even with high ranks, performs worse than ExPLoRA by at least 2% while using 7M more parameters.

Q: Was the pre-trained MAE used for full fine-tuning on the new domain dataset? Is extended pre-training necessary?
The SatMAE paper [1] shows this experiment in their Table 1- direct fine-tuning of pre-trained MAE performs 0.93% worse than pre-training SatMAE. As shown in our Table 1, LoRA fine-tuning with ExPLoRA outperforms LoRA fine-tuning with either pre-trained MAE or SatMAE while using the same objective.

ExPLoRA's impact is even more pronounced with DinoV2, achieving SoTA at 79.2% and outperforming fully fine-tuned models. Beyond fine-tuning performance, Table 2 shows ExPLoRA's strong feature extraction capabilities through significantly improved linear probing accuracy compared to SatMAE, DinoV2, and other SoTA methods (>7%). This enables use in label-free tasks (e.g., image embedding retrieval) and provides strong initialization for downstream tasks (Tables 6, 10, 11).

Please also see our new section B.2 (figure 6) in the appendix which analyzes the impact of extended pre-training over simply fine-tuning in more detail. To summarize, we find that ExPLoRA reaches a higher max top 1 accuracy than simply fine-tuning for longer.

Q: Comparison with related PEFT methods
Thank you for suggesting these works. We included several state-of-the-art PEFT methods in our results in table 1, including BOFT[4], GVPT[5], SA^2VP[6], AdaLoRA[7], which were all published within 1-2 years ago and have shown strong performance for ViT backbones. We have cited all references you have provided, and have included further results from specifically the following:

• Parameter Efficient Self-Supervised Geospatial Domain Adaptation [2], CVPR 2024 (added to Table 1).
• Adapters Strike Back [3], in CVPR2024 (added to Table 1).
• Mona [11], follow-on work to LoRand [10] in CVPR 2023, (added to Table 1).

The other cited works are either already outperformed or superceded by prior works we have included experiments from. E.g.,

• SA²VP outperforms Pro-Tuning [8] on CIFAR-100, Oxford flowers.
• BOFT [4] and AdaLoRA [7] outperform SSF [9] on VTAB-1K.
• “Adapters Strike Back” [3] includes comparisons with the most recent adapter-based methods, outperforming [9, 12, 13] on VTAB.
• Mona [11] outperforms LoRand [10] and other adapter methods [12] on COCO and other benchmarks
• HydraLoRA was published past the July 1 2024 date that ICLR considers concurrent work, and so we omit this.

Our expanded results in Table 1 demonstrate that ExPLoRA unsupervised pre-training (before fine-tuning) outperforms all modern PEFT methods that directly adapt MAE/DinoV2 pre-trained weights.

Further, we would like to emphasize that our work is fully compatible with any SoTA PEFT method, post extended-pretraining. One of ExPLoRA’s advantages is that it doesn’t change the architecture of the ViT, which allows us to plug the final unsupervised weights into any modern or future PEFT method that operates on ViTs. We demonstrate value in the extended unsupervised pre-training phase, which creates new foundation models cheaply to bridge difficult domain gaps.

Q: Results on detection and segmentation tasks
We include results for remote sensing image segmentation (Table 6) and agricultural image detection (Table 9), demonstrating parity or SoTA performance. Please let us know if there are specific datasets representing significant domain shifts that you'd like us to evaluate.

Dear Reviewer ANFL,

Thank you for your thoughtful review of our paper. As we approach the end of the discussion period, we would greatly appreciate if you could review our rebuttal responses and let us know if we have adequately addressed your concerns.

If you feel that our clarifications and additional experimental results have resolved your initial concerns, we kindly request that you consider updating your evaluation accordingly. Your final assessment is valuable to us and to the broader review process.

We understand you are likely managing many responsibilities, and we truly appreciate your time and attention throughout this process.

Best regards,
Authors

References:

[1] Towards geospatial foundation models via continual pretraining. ICCV 2023.
[2] Parameter Efficient Self-Supervised Geospatial Domain Adaptation. CVPR 2024.
[3] Parameter-efficient orthogonal finetuning via butterfly factorization. arXiv:2311.06243 (2023).
[4] Improving visual prompt tuning for self-supervised vision transformers. ICML 2023.
[5] AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv:2303.10512 (2023).
[6] SA²VP: Spatially Aligned-and-Adapted Visual Prompt. AAAI 2024.
[7] Adapters Strike Back. CVPR 2024.
[8] 1% vs 100%: Parameter-efficient low rank adapter for dense predictions. CVPR 2023.
[9] 5%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks. arXiv:2408.08345 (2024).
[10] Pro-tuning: Unified prompt tuning for vision tasks. NeurIPS 2022.
[11] Scaling & shifting Your Features: A New Baseline for Efficient Model Tuning. NeurIPS2022.
[12] AdaptFormer: Adapting vision transformers for scalable visual recognition. NeurIPS 2022.
[13] Sensitivity-aware visual parameter-efficient tuning. ICCV 2023.