Learning De-Biased Representations for Remote-Sensing Imagery

We sincerely appreciate your positive feedback on our paper's clarity, effectiveness, and novelty! Thank you for these encouraging comments. We are also grateful for your insightful suggestions regarding ablation studies and the generalization of our method. We have addressed each point in detail below and will incorporate these additional experimental results in our revised paper. Please let us know if you believe any further investigation is necessary!

Weakness 1: Sensitivity to cluster number $K$

Can't agree more! We conducted an ablation study on the number of clusters ($K$) in Table R6 in the attached PDF. We start from $K=16$, slightly above DOTA's 15 categories, to allow for potential subclusters. We can observe:

1. performance generally improves as $K$ increases, with the most significant gains observed for tail classes. For instance, when $K$ increases from 16 to 32, the F1 score for tail classes improves by 4.7%; and

2. the performance peaks around $K$=32, suggesting a good default value for our method.

Weakness 2: Generalization to more datasets

We appreciate your feedback! To address this generalization concern, we conducted experiments on two natural image domain datasets (Places365-LT and iNaturalist 2018) and one multi-spectrum RS dataset (fMoW-S2). We chose these datasets based on their unique properties:

1. Places365-LT exhibits a substantial domain gap from SD's pre-training data. It is thus proper to evaluate the performance of a domain adaptation model.

2. iNaturalist2018 has a high imbalance ratio of 500 and can evaluate the models under severe imbalance.

3. fMoW-S2 contains multi-spectral data, including visible, near-infrared, and shortwave infrared bands, which complement our existing experiments on optical (i.e., DOTA dataset) and SAR (i.e., FUSRS dataset) imagery.

We conducted adaptation experiments following the same setup used for Table 2 (in the main paper). On Places365-LT and iNaturalist 2018, debLoRA consistently outperforms LoRA, especially for tail classes (gains of 4.3% and 7.2%). On fMoW-S2, debLoRA achieves the best overall (46.8%) and tail class (41.2%) accuracies, surpassing ResLT by 0.3% and 2.6%. Kindly refer to our response to reviewer bDAb (weakness 2 & 8 & 10) for more detailed analysis.

I thank the authors for addressing my concerns.

1. The ablation of K is helpful. However, I think it would have been very informative to do this on multiple datasets. The results would tell us if there exists some value for K that is good for any/all tasks. Currently, it is not clear if the K=32, which is optimal for DOTA would serve as a good number for fMoW or Places365-LT. I think such information is useful because it tells us how much work a user needs to do to find the correct hyperparameters if they want to use the model.
2. The results provided in the rebuttal suggest that the model generalizes to tasks outside of RS, increasing its utility.

The authors have, to a large extent, addressed my concerns. After reading the reviews from other reviewers and the respective rebuttals, I have decided to stick with my initial review score.

Weakness 1 & Question 1: Lack of clarity

We sincerely appreciate the comment! In the revision, we will perform careful proofreading, e.g., 1) by simplifying complex clauses; 2) by reducing starting sentences with ambiguous "It"; and 3) by fixing the issues noted by reviewer: correct modifier placement, replace "what" with "which", and rewrite unclear sentences.

Weakness 2 & 10: "not clearly set the scope for the work" and "tested on non-remote sensing datasets"

Thank you! Our original experiments focus on Remote Sensing (RS) because: 1) RS images present significant domain gaps from natural images, offering a more challenging test case (as detailed in Lines 22-26). 2) The RS domain has heavier problems of data scarcity and class imbalance (as clarified in Lines 36-43).

We supplement the experiments on two natural image datasets: Places365-LT and iNaturalist2018. Both datasets are widely adopted in long-tailed learning [66]. 1) Places365-LT exhibits a substantial domain gap from SD's pre-training data. It is thus suitable to evaluate the performance of a domain adaptation model. 2) iNaturalist2018 has a high imbalance ratio of 500 and can evaluate the models under severe imbalance. Results are shown in Table R2. Our debLoRA consistently outperforms second best, and gains tail improvements of 4.3% and 7.2% for Places365-LT and iNaturalist 2018, respectively. We will clarify our research scope in the revision.

Weakness 3: "why it works"

We apologize for this misleading sentence. The paragraph led by "why it works" describes our experimental observation instead of justifying truly for "why it works". We will remove this sentence to avoid confusion.

Weakness 5: Rewrite Section 4

We'll make the following improvements to Section 4:

1. Emphasize the input/output of our 5-step method:

• Feature extraction: raw images $x$ → biased features $z$
• Clustering: biased features $z$ → cluster centers $\mu_k$
• De-biased center calculation: cluster centers $\mu_k$ → de-biased centers $\hat{\mu}_c$
• Feature calibration: biased features $z$, de-biased centers $\hat{\mu}_c$ → calibrated features $\tilde{z}$
• debLoRA training: original & calibrated features → debLoRA module $g_\phi$

2. Simplify formulas:

• Equation 1: $\min_{\mu_k} \sum_{i=1}^N \min_{k} |z_i - \mu_k|^2, \quad \text{s.t.} n_k \geq \frac{N}{K\rho}, \forall k$.
• Equation 3: Add $\alpha = \min(1, \frac{10}{\gamma})$.

3. Clarify key sentences:

• Line 208: "We calibrate each tail class representation by moving it closer to the de-biased center."
• Line 217: "For tail classes with larger imbalance ratio, a higher $\alpha$ moves features closer to the de-biased center."

Question 2: Clarification of "biased feature space"

We provide a more precise definition of "biased feature space" here. Let $\mathcal{Z}_h$ and $\mathcal{Z}_t$ be the feature spaces of head and tail classes, respectively. We define feature spaces as biased if $\text{Vol}(\mathcal{Z}_h) \gg \text{Vol}(\mathcal{Z}_t)$, and $\exists z_t \in \mathcal{Z}_t : P(z_t \in \mathcal{Z}_h) > P(z_t \in \mathcal{Z}_t)$, where $\text{Vol}(\cdot)$ denotes feature space volume and $P(\cdot)$ denotes the probability predicted by the model.

Weakness 4 & 7: Compare with contrastive methods

Thank you! Our work focuses on adapting pre-trained foundation models to solve RS domain problems. These foundation models have consistently outperformed conventional contrastive methods. For instance, SatMAE [7] achieves 63.84% accuracy on fMoW-S2, outperforming SeCo (51.65%) and GASSL (50.69%).

While these results demonstrate the superiority of adapting foundation models, we still agree that a direct comparison would provide a comprehensive view. However, due to private data [7,43] and GPU constraints, we cannot large-scalely train contrastive models in a fair data scale (as foundation models). However, we will try our best to conduct more experiments during the discussion period if you require.

Weakness 6: Statistical error bars

Agree! We conducted three runs of the SD $\to$ DOTA experiment with random initializations. The results are shown in Table R3.

Regarding "the results aren't significantly better (in absolute terms)", we gracefully disagree. Our debLoRA consistently outperforms the second-best methods by notable margins on tail classes, e.g., 2.5% over SADE in Table 2 and 2.4% over ECM in Table 4. These improvements are regarded as substantial. For reference, Table 3 in [R1] considers an improvement of 1.59% on DOTA as significant.

Weakness 8: More Remote Sensing datasets

Glad to do so! We conducted experiments on the fMoW-S2 dataset [7] because: 1) it contains 13 bands of multi-spectral data, including near-infrared bands, which complement our existing experiments; 2) it exhibits severe class imbalance (imbalance ratio of 130.8); 3) it provides a substantial validation set (84,966 validation samples). We performed experiments on SD $\to$ fMoW-S2. Results are given in Table R4. Our debLoRA achieves the highest overall accuracy (46.8%) and tail class accuracy (41.2%), surpassing the second-best method ResLT by 0.3% and 2.6%, respectively.

Weakness 9: Quantified feature analysis

We treasure your feedback! We provided an actual feature distribution in Figure 3. To further address your concern, we present quantitative experiments on inter-class and intra-class distances in Table R5. Two key findings: 1) debLoRA achieves higher inter-class distances for both head and tail classes, indicating improved feature separability. 2) debLoRA maintains lower and more consistent intra-class distances among tail classes, suggesting more compact tail features.

Weakness 11: Ablation of dataset size

Thank you! The ablation study using 5% and 50% of the DOTA dataset is ongoing. We will report the results during the discussion period.

Reference

[R1] Pu, Y. et al., Adaptive rotated convolution for rotated object detection.

We greatly appreciate your thorough review and insightful comments! Your positive remarks on our paper's structure, historical summary, and method effectiveness are encouraging. We value your critical questions and have addressed them in detail below. Thank you for helping us improve our research.

Weakness 1: Regarding linear space

Thanks for the insightful comments. We address this question from three perspectives:

1. Image features extracted from large-scale pre-trained foundation models already exhibit good linear separability. Recent literature [R1,R2,43] shows that applying simple linear classifiers (e.g., linear probing head and k-NN classifier head) on foundation models (DINO, CLIP and SD) achieves impressive performance for discrimination tasks.

2. Our de-bias method has indeed improved the feature discrimination between classes. To validate this, we report quantitative analysis on inter-class and intra-class distances in Table R5 in the attached PDF. Results show our debLoRA:

   • enlarged inter-class distance between tail and head classes. The average cosine distance increased from 0.702 to 0.719, indicating improved separation;
   • reduced intra-class distance for tail classes. The average cosine distance decreased from 0.182 to 0.146, suggesting tighter clustering of tail samples;
   • increased inter-class distance among tail classes. The average cosine distance rose from 0.607 to 0.632, demonstrating better separation among different tail classes.

3. We acknowledge the potential benefits of exploring non-linear de-biased centers. A simple way to achieve a non-linear transformation of the class centers is to apply a few non-linear layers (e.g., MLP) on them (i.e., taking the centers as input into a learnable MLP). The learning of these layers can be realized by either meta-learning or training together with our debLoRA parameters, where the former requires an additional validation set to be built. Both methods cost time in model designing and training. We are still working on them, and will report the results in OpenReview if we obtain any during the discussion period. We are also happy to try other methods of introducing non-linearity to de-biased centers if we receive any suggestions from the reviewers during the discussion.

Question 1: Clarity of Figure 2(a) caption

We apologize for any confusion in Figure 2(a). We want to highlight that:

1. The blue star represents the center of tail training samples (in blue color), but not the center of all samples.
2. The dashed blue triangles represent tail validation samples misclassified as head classes. They demonstrate the model bias towards head classes.

We will refine the caption of Figure 2 to avoid any confusion. Please refer to the caption of Figure R1 in the attached PDF for the updated caption.

Question 2: Fully supervised FCOS baseline

Thank you for the suggestion! Please note that the methods (except Zero-Shot) are all fully-supervisedly trained on DOTA datasets with bounding boxes as labels. If we understand the question correctly, the reviewer wanted to check the results without any transfer learning, i.e., the results of training a model from scratch as in the original paper of FCOS [50]. Table R1 in the attached PDF provides such results with two backbone networks: ResNet-101 (used in its original paper) and SD U-Net (used in our submission). SD U-Net is used for fair comparison regarding the amount of network parameters.

We can observe from this table that From-Scratch underperforms transfer learning approaches (Fine-Tune, LoRA, and debLoRA), especially on tail classes, and shows a larger performance gap between head and tail classes. For example, comparing rows 2 and 4, we see that it is lower than the basic Fine-Tune method by 0.2, 0.7, and 1.5 percentage points of mAP for head, middle, and tail classes, respectively. Moreover, the head-tail gap for From-Scratch (row 2) is as high as 13.3 percentage points, while for Fine-Tune, LoRA, and debLoRA, the gaps are 12, 11, and 6.2 percentage points, respectively (please note that the lower is better). This phenomenon occurs because transfer learning can leverage the robust representation of a pre-trained model (learned on large-scale datasets), while the From-Scratch model is trained with only the long-tailed training data of a small dataset (please note that here the "small" is a relative concept, compared to the large-scale pre-training datasets such as LAION-5B).

Question 3: Results from [7, 15]

First, for SatMAE [7], we have reported SatMAE results in Table 4 of the main paper on the "SatMAE → FUSRS" columns.

Second, SkySense [15] results are not directly comparable to ours because they did not benchmark their method on long-tailed settings. Besides, we cannot implement their methods in our setting because their pre-trained model is not open-sourced yet. If their model becomes available during the revision period of NeurIPS'24, we will gladly include it in our final paper.

As requested by other reviewers, we evaluated our method on more general long-tailed datasets (see Tables R2 and R4). On Places365-LT and iNaturalist 2018, debLoRA consistently outperforms LoRA, especially for tail classes (gains of 4.3% and 7.2%). On fMoW-S2, debLoRA achieves the best overall (46.8%) and tail class (41.2%) accuracy, surpassing ResLT by 0.3% and 2.6%. Kindly refer to our response to reviewer bDAb (weakness 2 & 8 & 10) for detailed analysis.

References

[R1] M. Raghu et al., "Do vision transformers see like convolutional neural networks?," Advances in neural information processing systems, vol. 34, pp. 12116–12128, 2021.

[R2] M. Caron et al., "Emerging properties in self-supervised vision transformers," in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 9650–9660.