RANK++LETR: Learn to Rank and Optimize Candidates for Line Segment Detection

Thank you for your detailed review and valuable suggestions. We will try our best to clearly explain your concerns point by point.

Q1: Evidence in the theory and experiments for justifying LADA and Line Ranking Loss. Connection/Contribution are not clear. A1: (1) Connection and theory of LADA are in Sec 3.2. We design line-aware deformable attention (LADA) module in which attention positions are distributed in a long narrow area and can align well with the elongated geometry of line segments. Theoretical expression style is similar to classical works such as Deformable-DETR([46]). (2) Connection and theory of Line Ranking Loss are in Sec 3.3. We propose line ranking loss in line segment detection task with the design of contiguous labels according to the detection quality. Theoretical expression style is similar to classical works such as RSLoss ([24]), RANKED ([4]). (3) The effectiveness of our proposed LADA and Line Ranking Loss are shown in our ablation study in Sec 4.3, which bring 1.5 and 1.7 improvement respectively on the $sAP_5$ metric. (4) Visualization of LADA is exhibited in Fig.5. Ranking quality can be reflected in Fig.4.

Q2: Concern about the innovation. A2: Our innovation can be summarized as: (1) propose a pure end-to-end learnable propose-optimize line segment detection framework. Moreover, we consider that introducing and modifying proper module structures appropriately to adapt to a specific task (LSD) for improving performance/ solving problems is also an innovation, just like LETR itself. (2) design line-aware deformable attention (LADA) for line segment detection. (3) line ranking loss. The key innovation here is finding ranking-based loss naturally suitable for the optimizing process in our pipeline, and designing the contiguous labels $l_i$ for the LSD task. (4) Performance: more accurate than previous methods and faster than previous Transformer-based method.

Q3: Concern about the improvement. A3: We argue that our method gets an obvious improvement. Our method get 2.4/2.6 improvement on $sAP_5$, $sAP_{10}$ compared with previous the SOTA method, and get 2.9/2.4 improvement compared with the SOTA Transformer-based method. These two previous SOTA method achieve 1.4/0.8 and 1.7/2.6 improvement on the same dataset, respectively.

Q4: Show the weaknesses of the proposed solution. A4: The main weakness is the speed gap benchmarked against optimized CNN-based implementations, which is mentioned at line221-227 in the original paper.

We are grateful for your clear and thoughtful reviews. We are willing to continue addressing any uncertainties and improving our work if further questions or suggestions are shared.

Thank you for your detailed review and valuable suggestions. We will try our best to clearly explain your concerns point by point.

Q1: Concern about innovation A1: Our innovation can be summarized as: (1) propose a pure end-to-end learnable propose-optimize line segment detection framework. Moreover, we consider that introducing and modifying proper module structures appropriately to adapt to a specific task (LSD) for improving performance/ solving problems is also an innovation, just like LETR itself. (2) design line-aware deformable attention (LADA) for line segment detection. (3) line ranking loss. The key innovation here is finding ranking-based loss naturally suitable for the optimizing process in our pipeline, and designing the contiguous labels $l_i$ for the LSD task. (4) Performance: more accurate than previous methods and faster than previous Transformer-based method.

Q2: Slower than optimized CNNs/FLOPs/parameter A2: We consider it a drawback of our approach, as discussed in our original paper. CNNs are truly fast. Our future work will employ efficient Transformers or even replace some modules with CNNs. We keep trying our best to establish a new milestone in LSD. Comparison on FLOPs and parameter are listed below. Rotation augmentation operation imposes much calculation overhead for Rank-LETR.

Q3: Theoretical justification for selecting $\delta_l$: A3: We choose distance factor $\delta_l$ according to existing metrics AP/sF_x, for x in [5, 10, 15]. Thus $\delta_l$ can be chosen as [1/5, 1/10, 1/15]. We choose it 1/10 in our experiments.

Q4: Performance on unstructured scenes: A4: For non-structured scenes, we directly applied the trained model to the semantic line detection dataset (NKL). We find that the detection of the boundary between linear strip-shaped traces is more in line with human perception (such as sea level, bridge, road, grass), while the detection of the boundary between serrated traces is relatively random (mountain-sky boundary, rugged tree crowns). We guess that the reason for this is the large gap between the line patterns in such scenes and the images in the training set. Since no figures can be supplemented in the rebuttal, we will add some visualization of non-structural scenes in our final version.

Q5: Application in 2D inspection: A5: Yes, it can be directly used in 2D inspection, such as document upright adjustment, graphic vectorization.

Thank you for your detailed review and valuable suggestions. We will try our best to clearly explain your concerns point by point.

Q1: Difference between LADA and the attention in DINO A1: The difference can be noticed in Fig.2(a). For the attention in DINO, the referencing point is the center point of the candidate box, and the attention points are generally distributed around the referencing point. Please refer to Fig.6 in the Deformable DETR article for a similar visualization. This distribution is not conducive to the perception of narrow targets such as line segments. The LADA uniformly samples the candidate line segments as the referencing points, so that the attention points are distributed along the line segments, which is beneficial for the perception of narrow areas. The results of our ablation study also confirm this point. For attention in models such as DINO, the shape of the candidate box is only related to the boundary of reference points. Not to mention that the collapsed box can only approximate horizontal or vertical line segments.

Q2: Compare with DINO+LSD(+line ranking loss) A2: DINO+LSD can be seen as the 2nd row in Table 2 (sAP5:65.5) and the DINO+LSD(+line ranking loss) can be seen as 4th row in Table 2 (sAP5:67.2) in the original paper, which has a performance gap with our proposed method.

Q3: Difference between the proposed method and RANK-LETR A3: (1) The proposed method uses a pure learnable Transformer-based decoder to re-rank (scores) and optimize (locations) the line segment simultaneously, while Rank-LETR uses CNN CNN-based manner to re-rank the score with pre-designed hyperparameters. (2) Design line-aware deformable attention (LADA) for line segment detection. (3) Introduce a powerful line ranking loss. The key insight here is finding ranking-based loss naturally suitable for the optimizing process in our pipeline, and designing the contiguous labels $l_i$ for the LSD task.

Q4: Inference speed/Ablation study A4: Our method is faster than LETR and Rank-LETR in Table 1. The higher the FPS value is, the faster. In Table 2, we conduct the experiment using a module adding pipeline, and the comparison between with and w/o LADA can be obtained by comparing the results in 2/3 rows and 4/5 rows, respectively. sAP5 increase 1.7 when adding the LADA module compared to the baseline, while decrease 0.7 when subtracting the LADA module. Please note that it also reflects the ability of Line Ranking Loss. The change in improvement is caused by the ability overlapping of the two proposed modules.

Q5: Limitation and failure cases A5: The main limitation is the speed gap benchmarked against optimized CNN-based implementations, which is mentioned at line221-227 in the original paper. Most failure cases happen in dense short line segment detecting, e.g., the 3rd column in Fig.3. It is worth noting that this has always been a common problem in learning based methods. Compared to the existing methods, our method has made improvements in this issue.

Q6: Training Iterations A6: Our training iteration is set as 225000.

Q7: Explain FPS A7: Yes, FPS is short for Frame Per Second, which indicates the number of images the model can process in 1 second. A large value indicates high inference speed.

Q8: Row 1 in the ablation study A8: The first row corresponds to a baseline with an encoder and without a decoder.

Q9: Explain the hyperparameter values A9: $\lambda_c$, $\lambda_{ep}$, $\lambda_{dp}$ are set as the same as RANK-LETR, $\lambda_r$, $\lambda_s$ are set according to previous study on ranking based losses ([24] in the paper) and parameter study.

Q10: Explain the blue line from the ground truth to the initial position and score boxes in Fig.1. A10: The initial scores and positions are supervised by the ground truth line segments during training.

We are grateful for your detailed and thoughtful reviews. We are willing to continue addressing any uncertainties and improving our work if further questions or suggestions are shared.

Thank you for your detailed review and valuable suggestions. We will try our best to clearly explain your concerns point by point.

Q1: Multiple components increase implementation complexity. A1: Ranking loss and warm up are used in the training phase. So they will not affect the efficiency of the model in inference. LADA brings a limited complexity increase, but also an obvious performance improvement.

Q2: Benefit from unsupervised or semi-supervised pre-training? A2: It is a very good suggestion. Some studies have tackled the task with unsupervised or semi-supervised approaches (e.g., [25][26] in our original paper). Our future work will focus on unsupervised and semi-supervised methods, such as pre-training the network through bootstrapping or synthetic labeling, as well as possibly performing some appropriate model adaptation. We think that there will be a significant improvement in generalization at least.

Q3: Extend LADA to curved feature detection. A3: We strongly believe that LADA can be used for curved feature detection since they almost share the same design motivation, which is to sample the attention points along the high-confidence line/curve features, allowing the model to learn relative information. On the contrary, a traditional attention machine tends to focus around a point.

Q4: Impact of increasing the number of attention points beyond 8. A4: We conduct an experiment that increases the number of attention points to 16. The experimental results show that there is not much change in accuracy and inferring time. We think that the accuracy has reached saturation, and an excessive increase in the number of attention points may only continue to increase (not much) computational overhead.

Q5: Sensitive to inaccuracies in the continuous label generation. A5: We randomly perturb the positions of all candidate lines by 1-5 pixels, which has some impact on $sAP_5$ and little impact on the $sAP_10$. Moreover, the candidate lines are also learned and predicted by the network. So they are inherently inaccurate compared to the ground truth, but generally do not deviate significantly, as analyzed in line282-289 in the original paper.

Q6: Complex scenes including overlapping/ occluded/ non-structured scenes. A6: (1) Overlapping： Dealing with overlapping is one of the strengths of our method. Our method recalls complete line segments because of its focus on global information, and will not be segmented or truncated due to overlapping. It is reflected in columns 2/5 of Fig.3 and the result visualization in the supplementary materials (e.g., carpets, windows and cabinets). (2) Occluded： For slight foreground occlusion, our method exhibits good anti-occlusion ability, as shown in the 2nd column of Fig.3, and (r1c2), (r1c3), etc., in the result visualization in the supplementary materials (e.g., foreground lights, chairs). For severely occluded scenes, our method considers them as two line segments on both sides of the occlusion, as shown in the supplementary materials (r3c5), which aligns with human intuitive perception. (3) Non-structured scenes： For non-structured scenes, we directly applied the trained model to the semantic line detection dataset (NKL). We find that the detection of the boundary between linear strip-shaped traces is more in line with human perception (such as sea level, bridge, road, grass), while the detection of the boundary between serrated traces is relatively random (mountain-sky boundary, rugged tree crowns). We guess that the reason for this is the large gap between the line patterns in such scenes and the images in the training set. Since no figures can be supplemented in the rebuttal, we will add some visualization of non-structural scenes in our final version.