P2RBOX:A SINGLE POINT IS ALL YOU NEED TRAINING ORIENTED OBJECT DETECTOR

W1: The content of Fig. 2 is not enough to intuitively understand the process of the proposed framework. It is suggested that the author include more details and explanations in Fig.2.

Thank you for your feedback. I appreciate your suggestion to enhance the clarity of our training process overview (Fig. 2). We will incorporate additional visual aids and improve the representation of the point selection in the Point Bag Generation module for better comprehension.

Q1: Oriented object detection is a generalized form of general object detection. Please discuss the effect of the proposed framework on horizontal object detection.

Thank you for your comments. haven't conducted experiments on widely-used detection datasets like COCO. But in my analysis, the final performance mainly depends on the performance of the SAM model as an anchor generator. The method proposed in this article determines a new criterion to select a more matching mask. For a general target detection data set, SAM may perform better than the DOTA data set. At the same time, the SAE module will become useless. When applied to a more diverse dataset like COCO, especially for categories like humans with challenges such as occlusion and varying poses, the boost from semantic scores might not be as pronounced. Since I have not done experiments on COCO, I will be happy to show the experimental results on COCO in the future.

W1.1:(1) Strictly speaking, this method is not a pure point-supervised oriented object detection approach. This method depends on mask proposals from SAM-based mask generator, which has been trained with lots of data.

You're correct in saying that our method doesn't depend only on one specific type of guidance, but it's not strongly supervised either. I want to highlight that I didn't train SAM model's details for the DOTA dataset; I used the settings given by the official website. The comparison baseline is also created using SAM. I truly value your insights on this issue.

W1.2:(2) Oriented object detection is a compromise solution between the hbox-based object detection and fine-grained segmentation tasks. This paper adopts the "point-to-mask-to-rbox" manner. Compared with rbox with orientation information, mask has fine-grained pose information. The approach designs to generate the relatively coarse rbox based on the selected masks, which may put the cart before the horse. The proposed method can solve the high-quality mask selection based on point prompt for general mask generator like SAM for aerial image filed, which is applicable.

Thanks to the reviewers for their feedback. I understand your point of view, that is, if you already have mask-level annotation, you may not need to pay attention to the rotation box-level annotation, because the latter is relatively rough. However, I would like to highlight application-specific aspects of our method for rotating box annotation. In some scenarios, rotating box annotation still has unique advantages, such as in applications that deal with tilted targets, directional detection, or where the target direction needs to be considered. In these cases, rotating box annotations may provide more accurate information rather than just being redundant as mask-level annotations. I hope this will make you reconsider this article.

W1.3:(3)This method follows the assumption that the annotated point of an object is close proximity to the center of the mask. This assumption is also not rigorous enough.

This is to refer to the following two articles to obtain some priors for point annotation[1][2]. Therefore, the article makes an assumption that the labeling points often present an approximate Gaussian distribution at the center of the object.

W2: The proposed Symmetry Axis Estimation (SAE) module is not general for all categories, which is effective for PL and HC for DOTA dataset. For most of categories, the peformance is descreased form the Table 7 in Appendix. For other datasets, how to determine whether to use this module.

Thank you for bringing up important concerns about how well the Symmetry Axis Estimation (SAE) module works in different situations. Let me address a few points: first, it's crucial to mention that a decrease in performance when using SAE doesn't happen a lot across most categories. Also, we've explained in detail why there's a significant drop in performance for category BD in the additional information section. Additionally, the small decrease in mIoU doesn't strongly affect either the training results (measured by mAP). When deciding whether to use the SAE module, it basicly depends on whether the object is symmetric or not, which needs prior knowledge.

W3: Some experimental settings is not clear, such as all models adopts r-50 model with 1x trianing schedul? Some reported results miss the mAP results in some ablation studies, why only report mIoU?

Thank you for giving us feedback on how we set up our experiment. I used the ResNet-50 architecture for all our models and trained them with a 1x schedule. I choose to report mIoU instead of mAP because, in my observation, mAP values for each category can vary a lot due to inaccurate label in the same training condition. Using mIoU seemed like a better way to show the differences accurately in our situation.

• [1] Point-to-Box Network for Accurate Object Detection via Single Point Supervision
• [2] AttentionShift: Iteratively Estimated Part-Based Attention Map for Pointly Supervised Instance Segmentation

W1: Generator (SAM) is a high-performance segmentation method, and its introduction may not be fair to other methods.

Your observation regarding the high-performance segmentation method SAM is valid, and we appreciate your concern. SAM makes this article not a purely weakly supervised work, and it is unreasonable to compare its performance with weakly supervised methods. Thus, our baselines are also built upon SAM and we directly use SAM with given weights without any change.

W2: Table 1 should include some recent SOTA methods (such as GWD[1], KLD[2], CGCDet[3], OSKDet[4], KFIoU[5],). Although the proposed p2rbox has significantly improved compared to baseline, there is still a significant gap in its performance compared to supervised methods. Is its performance persuasive enough?

Q2: By using SAM to segment the target, the quality of the segmentation mask obtained is relatively high. By filtering false positives through the Constrainer Module, higher performance should be achieved. However, compared to supervised methods, the performance difference is still significant. What is the main reason for the performance loss?

Our baseline have shown the performance of SAM with only one point on DOTA. Moreover, we have conducted experiments to explore the upper limit of P2RB. Selecting the highest IoU Rbox from three different masks and training on RetinaNet yields a performance of 58.94 mAP.In contrast, our achieved result of 55.50 (selected based on our criterion) outperforms the result of 47.91 (selected based on the highest score given by SAM), showcasing substantial improvement under the same settings. The gap mainly from performance of SAM, result of not receiving specialized training in DOTA. In the future, I hope to find some inductive approach to breaking through the upper bound.

W3: Compared to supervised methods, the method adopts a segmentation mask generator. Will its speed be significantly affected? Please provide an analysis.

Thank you for your feedback. The training speed may be affected a little. For an image with given annotated points, the “get mask” step(generate masks with given annotated points and image) cost within one second. Moreover, this article will eventually use the results of P2RBox to train strong supervised detectors. In practical applications, only strong supervision detectors are involved.

Q1: The Constrainer Module selects high-quality masks to create the union mask. Is it better to choose an optimal mask, or to generate the optimal mask through the mask generated by SAM?

Yep, through choosing the optimal mask based on the IoU with ground truth, the performance will get to its upper limit. During training, the only information we get is a single point of every object. Our purpose is to select the best performance mask using our design criterion.

After carefully reading the author's response and comments from other reviewers, though I think this paper proposes an interesting attempt on oriented object detection, I agree this paper should be further improved. Thus, I would update my rating from "7: accept" to "6: marginally above the acceptance threshold". Thanks.

W1: As we all know, SAM is a fully supervised visual model, and its training data over 1 billion masks on 11M licensed and privacy respecting images. Therefore, the method proposed in this paper does not seem to be the so-called weakly supervised oriented object detection, more like zero shot object detection. This is extremely unfair to compare with other truly weakly supervised algorithms.

The use of the SAM model makes it no longer a purely weakly supervised task, but the baseline in the article is also designed based on SAM. I hope this will make you reconsider this article.

W2: The authors used the mask generated by SAM to generate the rotated bounding box, which seems to be redundant and unnecessary. In my opinion, if a higher quality mask can be produced, there is no need to study other rough object representations, i.e. horizontalrotated/poly bounding box. Therefore, I'm more curious the performance comparison of instance segmentation models trained by masks before and after refinement on commonly used segmentation data sets (i.e. COCO, LVIS). Relevant experiments are given in SAM.

Thanks to the reviewers for their feedback. I understand your point of view, that is, if you already have mask-level annotation, you may not need to pay attention to the rotation box-level annotation, because the latter is relatively rough. However, I would like to highlight application-specific aspects of our method for rotating box annotation. In some scenarios, rotating box annotation still has unique advantages, such as in applications that deal with tilted targets, directional detection, or where the target direction needs to be considered. In these cases, rotating box annotations may provide more accurate information rather than just being redundant as mask-level annotations. We will further consider how to explicitly highlight the unique features of our approach and adapt accordingly in the paper. Thank you again for your helpful suggestions, we will take them seriously and make improvements accordingly.

W3: As mentioned earlier, the issue of masks generated by SAM found by the authors is interesting, but the proposed solution seems to be engineered and customized. The large number of hyperparameters and thresholds/scores makes me question the generality of the proposed method, and the authors only used one dataset (i.e. DOTA) for verification in the paper, which is not credible.

Thank you for your feedback. The problem of large number of parameters in this article is indeed a shortcoming. As an author, I will add more experiments to prove that my experimental settings are actually relatively general and not specially designed.

W4: The title is too exaggerated (i.e. single point is all you need). According to the author's experimental results, even if a powerful SAM is used, the performance is still far behind the fully supervised algorithm, e.g. RetinaNet vs. P2RBox (RetinaNet): 67.83% vs. 55.50% (-12.33%).

Thank you for your reply. I'm sorry that the title of this article is indeed exaggerated and I will change it. I hope it does not affect the contribution and innovation of this article. I hope this will make you reconsider this article.

Q1: The article lacks details on how point annotations are generated.

The generation of point annotation is not the mean point of this article. In fact, the point annotations are generated through center point with slightly adjustment [1][2] to ensure it always locate on the object even for object of shape L.

Q5: Appropriate failure cases need to be discussed to analyze why the current performance is still far below the performance of fully supervised algorithms.

Thanks for your advice. I will add more analysis experiments in the article, including parameter ablation and results on more data sets. I will also refer to your valuable opinions to add contrast in the visualization part.

• [1] Point-to-Box Network for Accurate Object Detection via Single Point Supervision
• [2] AttentionShift: Iteratively Estimated Part-Based Attention Map for Pointly Supervised Instance Segmentation

After reading the comments of other reviewers, I find that many comments are consistent, and some of the comments are also acknowledged by the authors. Due to the extensive revisions needed to this paper and the lack of substantial feedback from the authors, I think the current version is not suitable for publication. Therefore, I maintain my original rating.

W1: To study target detection methods, it is important to design a novel network structure. However, most of the structures in this paper are borrowed from Oriented-RCNN, and there is no discussion on the performance/computational gain of the structure introduced by Oriented-RCNN, so it is difficult to evaluate the effectiveness of the proposed methods.

P2RBox is the first step of two stage method in this article(generate rotated box from point and train with a fully supervised detector). P2RBox is a point-to-mask network to generate rotated box as pseudo label (1st stage). Then the generated pseudo label is used to train a fully supervised detector. The performance of the supervised detector is used to measure the quality of generated pseudo label. The performance of P2RBox combined with different fully supervised detectors will not be compared together as they have different baseline to compare with. The structure of P2RB have little relationship with Oriented-RCNN, and is totally original.

W2.1: Extracting masks from SAM itself can be a labor-intensive task. Additionally, this approach may not scale well to larger datasets beyond just rotating target datasets like DOTA.

Thank you for your feedback. For an image with given annotated points, the “get mask” step(generate masks with given annotated points and image) cost within one second. Moreover, this article will eventually use the results of P2RBox to train strong supervised detectors. In practical applications, only strong supervision detectors are involved.

W2.2: From the standpoint of SAE, it is unclear if the axis of symmetry problem is statistically characteristic. The explanation regarding whether eigenvalue decomposition represents the statistical characteristic.

Since DOTA is a data set from an overhead perspective, occlusion problems usually do not exist, and the objects are all rigid bodies. Therefore, the problem about the symmetry direction described in the main text only exists in the PL and HC categories, and is prevalent in these two categories.

Q1.1: The part of the SAE discussion is very interesting, but does it ignore the proportion of axisymmetric targets?

In the DOTA dataset, most object categories show symmetry, except for the HARBOR category. DOTA, being a dataset of remote sensing images, is mainly captured from a top-down view, meaning that whether the object is symmetrical or not is only related to its category excluding occlusion situations.

Q1.2: What is the proportion of axisymmetric targets that cause problems?

The ambiguity occurs when the orientation of the minimum bounding rectangle differs from the annotation direction, particularly in the helicopter and aircraft categories. This creates a unique challenge for these categories, unlike the others.

Q1.3: How do you consider other special situations, such as circular targets, playgrounds, baseball fields, etc?

The SAE module is versatile across various object shapes, providing two perpendicular directions. While directions for nearly circular objects may not have clear geometric significance, it doesn't seem to affect detection performance because a rectangle represented by any two vertical direction is suitable. For other categories that is symmetric like baseball fields, ship and palygrounds, SAE module’s outputs are same with minimum bounding boxes, showing its direction.

Q2: The experimental part of this article is actually weak. This work proposes a lot of loss functions, but the experiments didn't perform the ablation study on these loss functions. Besides, it lacks visual comparisons with other methods and lacks the analysis of failure cases.

Thank you for your valuable advice. I will add more experiments to ensure the generalizability of the method.

W1: The MIL is widely used in weakly supervised object detection[1] and point-based object detection[2].

A1: While MIL finds extensive application in the field of weak supervision, in specific situation, how to define and design the Bag to provide a valid supervision, guiding the model to optimize in the right direction is the crucial part. The key contribution of this article lies in the design of point bag construction, which differs from previous weak supervision tasks.

W2: This paper used SAM to generate the mask proposals. However, the training of SAM used many complete mask annotations. Therefore, the proposed method still rely on the complete mask supervision, which is not suitable to be considered as a point-supervised method. It is also not consistent with the tile “A SINGLE POINT IS ALL YOU NEED.”

A2: Thank you for your thoughtful feedback. Regarding my title, it's really inappropriate. The use of the SAM model makes it no longer a purely weakly supervised task, but the baseline in the article is also designed based on SAM. I hope this will make you reconsider this article.

W3: The proposed SAE only significantly outperforms minimum in PL category. In some other cases, it is even lower than minimum.

A3: Thank you for your comment. The purpose of the design of the SAE module is to solve the problem of inconsistent orientations of the minimum circumscribed rectangle of the symmetry target. This problem is currently only observed on aircraft and helicopter categories. For other categories, the applicability of asymmetric targets is not as good as the minimum circumscribed rectangle, such as the SP category (mini 63.5 vs SAE 62.95), but the difference is not significant. For other symmetry categories, such as SH (mini 67.97 vs SAE 68.15), LV (mini 69.22 vs SAE 69.12), BC (mini 44.80 vs SAE 43.80), the overall positive or negative IoU impact is not certain. Through some visualization, when SAM cannot give an accurate mask or the object is occluded, the impact of different decisions (mini or SAE) on IoU becomes unpredictable. To sum up, SAE is only suitable for the generation of directional frames with symmetrical targets as the direction.

W4: Can the authors compare the proposed method with point supervised instance segmention[3], in the same supervision setting?

A4: Thank you for your comment. Adding a point-supervised instance segmentation as a reference experiment is indeed a solution that can be used as a baseline. However, the reference you gave is based on multi-point supervision rather than a segmentation work based on single-point supervision. I think the following work may meet the requirements of single-point supervision: AttentionShift: Iteratively Estimated Part-Based Attention Map for Pointly Supervised Instance Segmentation. Since there is no mask-level annotation on DOTA, training this model is not yet possible.

I have read the responses and the comments of other reviewers. I decided to keep my original rating.