SARDet-100K: Towards Open-Source Benchmark and ToolKit for Large-Scale SAR Object Detection

We appreciate the reviewer's insightful comments and suggestions. We have carefully considered the points raised and provide the following clarifications and additions:

While it is true that merging multiple datasets can improve performance in general object detection tasks where datasets share similar concepts (e.g., optical concepts), this approach has been shown to improve performance due to shared features and representations. However, in our scenario, SAR and optical (RGB) datasets represent different modalities, which present distinct challenges:

SAR and RGB datasets differ significantly in terms of data modality and conceptual representation. Joint training of these datasets can result in a crowded feature and representation space, limiting the model's expressiveness for SAR data.

The distinct nature of SAR and RGB data leads to different learning difficulties, causing unsynchronized optimization rates across various parts of the model. Inconsistent optimization objectives across modalities can cause conflicting optimization directions, further reducing the performance of SAR object detection.

To justify our analysis, we conducted experiments using ConvNext-T backbones pretrained on ImageNet. The results are presented below:

The table demonstrates that joint training on SARDet-100K and DOTA results in a decreased mAP for SARDet-100K (from 53.2 to 51.9). In contrast, our proposed pretraining on DOTA before training on SARDet-100K improves the performance (mAP of 54.8). Additionally, the DOTA dataset's performance also drops significantly with joint training, supporting our claim.

We have incorporated additional recent works for comparison in Table 5 to provide a more comprehensive evaluation. The updated table is shown in Table R1 (in the rebuttal pdf).

Sorry for the lack of clarity in the manuscript. The reported performance metrics are based on the test set of the SARDet-100K dataset. The models are trained on the training set for a total of 12 epochs, and the checkpoint from the 12th epoch are used for testing.

We acknowledge the importance of reporting actual runtimes and memory usage. In response, we will release all training checkpoints and logs as part of our open-source code repository. These logs will include detailed information such as actual runtimes, memory usage, hardware specifications, system environment, package versions, and training loss, ensuring full transparency and reproducibility.

Sorry for any confusion caused by the abbreviations used in Table 1 and S12. Here are the clarifications: Ins.: Instances Img.: Images Cls.: Classification Det.: Detection B.S.: Batch Size L.R.: Learning Rate.

Thank you for pointing out this practical issue. In our dataset standardization process, we handle objects that lie on the slicing border as follows: If at least 50% of the object’s area remains within the image slice, we keep the bounding box; otherwise, we ignore the bounding box and treat it as background. For the kept bounding boxes, we retain the original full box coordinates, meaning that the annotation coordinates for such objects may extend beyond the image boundaries.

We will add all the above details and clarifications to the revised version of the paper.

We appreciate your feedback regarding the introduction and related work on handcrafted feature descriptors. We have indeed provided more extensive related work on these descriptors in the Appendix section of our submission. In the revised version of the paper, we will include comprehensive visualizations of different handcrafted feature descriptors (including HOG, Canny, GRE, Haar and WST) to enhance clarity and accessibility, as Figure R1 (in the rebuttal pdf).

Thank you for pointing out the need for a clearer guide in the supplementary materials. Our code is based on the open-source mmdetection framework, and a detailed guide can be found in the official mmdetection GitHub repository. However, we acknowledge that a more specific guide related to our work would be beneficial. Therefore, we will include a clear, step-by-step guide on training and testing the models directly in our open-sourced code to facilitate easier replication of our results.

The category distribution in Figure S6(b) indeed reveals an imbalance in the SARDet-100K dataset. We intentionally maintained this long-tail distribution to simulate real-world applications of large-scale multi-category SAR object detection, where such imbalances commonly exist. Addressing the long-tail problem is a significant challenge and an active area of research. Our work aims to provide a realistic benchmark for future studies that may focus on developing techniques to address these imbalances effectively.

Yes, the image pairs in Figure 1 are not spatially aligned, except for the second column. This misalignment is due to the difficulty in finding publicly available SAR-RGB paired images. The purpose of Figure 1 is to provide a conceptual overview, and the lack of spatial alignment does not hinder the understanding of the key concepts being presented.

In the context of our study, the term 'model gap' refers to the inconsistency introduced when only the backbone of the detector is pre-trained, while other components, such as the neck and heads, are initialized randomly. For downstream object detection tasks, the model comprises the entire detector, which includes the backbone, neck, and heads. When performing object detection, initializing only the backbone while randomly initializing the other components creates a mismatch or gap between the pre-trained weights and the fine-tuned detector network. This inconsistency, referred to as the 'model gap,' can negatively impact the overall performance of the detector as the synergy between the backbone and other components is disrupted.

We have considered the approach of directly training the SAR detection model on the proposed dataset. However, there are two main reasons for opting for the pretraining method:

1. Versatility of Pretrained Models: Our proposed method involves pretraining the model once, which can then be utilized for object detection across multiple SAR datasets. For instance, in our study, we pre-trained the model using the MSFA framework and demonstrated its applicability on the SARDet-100K, SSDD, and HRSID datasets. This 'one-for-all' pretraining approach saves considerable time during the fine-tuning stage, as the pretrained model serves as a robust starting point for various datasets.

2. Mitigation of Overfitting: Pretraining helps reduce the risk of overfitting. As illustrated in Table S11 of our manuscript, pretraining on the DOTA dataset for 12 epochs followed by fine-tuning for 24 epochs (totaling 16.1K iterations) resulted in an mAP of 54.5. In contrast, directly fine-tuning for 17.7K iterations without pretraining yielded a lower mAP of 52.8. This demonstrates that the pretraining stage not only enhances performance but also provides a more efficient training process.