Symbol as Points: Panoptic Symbol Spotting via Point-based Representation

Dear Reviewer q1T8,

Thank you very much for your comment and support. We address your concerns as follows.

1. Motivation for Symbol Segmentation

While tools like AutoCAD enable the rapid creation of CAD drawing, it typically lack detailed categorization information for each graphical primitive or symbol. The task of panoptic symbol spotting aims to effectively extract and interpret this information in graphic primitive granularity. Accurately spotting symbols is crucial for Building Information Modeling(BIM), as the picture shows here: https://anonymous.4open.science/r/x-BB39/bim.png .

2. Clarification on Methodology and Experiments

• Regarding Equation (2) and Primitive Representation: $l_k$ represents the distance between $v_1$ and $v_2$ for linear primitives. For circular primitives like circles and ellipses, $l_k$ is defined as the circumference. For arcs, $l_k$ means the length of curve, not the distance between $v_1$ and $v_2$. Since all arcs in CAD are quadratic Bezier curves, we can easily compute the length of curves using the python package svgpathtools
• On Defining Neighborhood $M(p_i)$ in Equation (3): In $M(p_i)$, adjacent points means top K nearest primitives which are measured by the distance between the primitive positions (Eq 1) and $p_i$, not locally connected primitives. $A(p_i)$ means a combination of $M(p_i)$ and $C(p_i)$ which means locally connected primitive points (Eq. 8). The reason that we use a small threshold in Eq. 7 to define the connectivity of two primitives is that for most CAD drawings, the interconnected endpoints of two connected lines are not the same point, but two points that are very close to each other. We therefore regard lines whose endpoints are close enough as connected.
• Motivation Behind KNN Interpolation (Section 3.5): This KNN interpolation is used to extract different levels (resolutions) of features in symbol region (i.e, mask region). Although it may lose some information of the symbol in low resolution, but it provides a coarse estimated symbol region at different resolutions, which could be used for pyramid feature extraction as in [R1]. These pyramid of features, including both high-resolution and low-resolution features will be fed into the spotting head for symbol mask/label predicting. Note that we do not only use low-resolution features but different levels of features for final segmentation and label prediction.
• Downsampling and Upsampling in Point Transformer: The downsampling and upsampling processes of point features is shown in https://anonymous.4open.science/r/x-BB39/pool&unpool.png, which is from [R2]. Adjacent point features are downsampled by pooling to get a point feature, which could also be upsampled through unpooling.
• Explanation of Equation (12): $e_i$ is a graphical entities, $L(e_i)$ is the length of the graphical entitity $e_i$
• On Experimental Evaluation: (1) Semantic Symbol Segmentation does not distinguish between objects of the same category, while Panoptic Symbol Segmentation does. (2) We follow previous works such as PanCADNet [R3], and GAT-CADNet [R4] to use F1 as metric for fair comparison; (3) The F1 is the harmonic mean of Precision and Recall, The wF1 is length-weighted F1. Both metrics are well defined in [R3]. We will provide the detail definition in the supplementary material in future.

[R1] Bowen, Cheng, et al. "Masked-attention mask transformer for universal image segmentation" CVPR. 2022.

[R2] Xiaoyang Wu, et al. "Point Transformer V2: Grouped Vector Attention and Partition-based Pooling" CVPR. 2021.

[R3] Zhiwen, Fan, et al. "Floorplancad: A large-scale cad drawing dataset for panoptic symbol spotting" ICCV. 2021.

[R4]Zhaohua, Zheng, et al. "Gat-cadnet: Graph attention network for panoptic symbol spotting in cad drawings" CVPR. 2022.

Dear Reviewer u21w,

We sincerely appreciate your comments and your detailed questions. We address your concerns as follows.

1. Regarding Originality and Technical Contribution:

While SymPoint incorporates well-established methods like Point Transformer, Mask2Former, and InfoNCE, our innovation lies in adapting and optimizing these methods for the specific challenge of CAD vector graphics recognition. It is worth noting that the combination of Point Transformer and Mask2Former is just our baseline.

Specifically, the novelties of our paper lie in several aspects:

• We carefully analyzing the data characteristics of CAD drawings and design novel and effective way of transferring CAD primitive entities into point feature in Section 3.1;
• We propose Attention with Connection Module (ACM) , which is novel in efficiently capturing the local connection information of primitives and enhancing their discriminability when in conjunction with Contrastive Connection Learning (CCL) module in the context of CAD graphics in Section 3.3 and 3.4 ;
• We propose KNN Interpolation to downsample attention mask for masked attention calculation, which shows superior performance compared with the naive bilinear interpolation used in Mask2Former in Section 3.5.

Our ablation study in Table 4 (a) has demonstrated that these modules effectively promoted the performance of the model. These strategies represent a unique approach not previously explored in this field. We achieves 77.3/87.1/ 88.7 (PQ/RQ/SQ) with these novel designs, significantly outperforming the baseline 73.1/83.3/87.7 (PQ/RQ/SQ) . Note that our baseline method has already surpassed many state-of-the-art methods, such as PanCADNet and CADTransformer.

2. On the Potential Application Range:

In response to the concern regarding the application range of SymPoint, we acknowledge that our current focus is primarily on CAD vector graphics. However, we believe there is potential for applying our method to a broader range of fields, like 3D modeling[R1] and circuit design[R2], sketch segmentation[R3] as shown here: https://anonymous.4open.science/r/x-BB39/application.png. We have conduct some preliminary experiments on circuit design in the appendix A.2. and achieve state-of-the-art performance on two datasets as shown in Table 7. Although this beyond the scope of this paper, we see it as a valuable direction for future research.

[R1] Lv, Xiaolei, et al. "Residential floor plan recognition and reconstruction" ICCV. 2021.

[R2] Jiang, Xinyang, et al. "Recognizing vector graphics without rasterization" NeurIPS. 2021.

[R3]Yang, Lumin , et al. "Sketchgnn: Semantic sketch segmentation with graph neural networks" TOG. 2023.

Dear Reviewer AtGi,

We sincerely appreciate your comments and your detailed questions. We address your questions as follows.

1. Reliance on Existing Methodologies

• While our approach to primitive features bears a resemblance to the vertex features defined in GAT-CADNet, our method focuses on leveraging the most straightforward and fundamental features in CAD drawings, such as type, length, and clockwise angle. These features, while simple, are highly effective in capturing the essence of individual primitives. Although more complex features could potentially be utilized, we found that these basic attributes yield impressive results, striking a balance between simplicity and effectiveness.
• Compared to CADTransformer, our method is simple yet effective, marked by three key distinctions:
  • Our process eliminates the need for rasterization of CAD drawings to images, which is a time-intensive step required for many CAD segmentation methods including CADTransformer.
  • We rely on simple and direct primitive features, as opposed to CADTransformer's use of HRNet for feature extraction. This approach not only simplifies the process but also enhances the scalability of our method to complex, high-resolution CAD drawings.
  • Our method employs the Point Transformer for interactions with each graphic primitive, whereas CADTransformer utilizes the Vision Transformer (ViT), which more easily occurs OOM when dealing with high resolution CAD drawings;
  • Our methodology is designed for end-to-end training without the necessity for any post-processing operations. In contrast, CADTransformer requires the use of a clustering algorithm to derive the final instance results, adding additional steps to the process;

2. Design of Baseline Method

Our baseline is a simple combination of Point Transformer and Mask2Former's transformer decoder with input of point-based representation. This serves as a foundation for comparing the enhanced performance of SymPoint.

3. Performance of the ACM Module

Our original intention in introducing ACM was to utilize these connections between each graphic primitive. we conduct experiments in SESYD-floorplans dataset that is smaller than floorplanCAD, ACM can significantly promote performance and accelerate the model convergence. An convergence curve between without/with ACM is shown here: https://anonymous.4open.science/r/x-BB39/sesyd_val_loss.png, https://anonymous.4open.science/r/x-BB39/sesyd_val_PQ.png, https://anonymous.4open.science/r/x-BB39/sesyd_val_RQ.png, https://anonymous.4open.science/r/x-BB39/sesyd_val_SQ.png, The quantitative results are shown below,

But, floorplanCAD is more complex compared with SESYD-floorplans, and more noisy connections could be introduced. As this figure shows : https://anonymous.4open.science/r/x-BB39/noisy-connection.png . Therefore, we have introduced an additional Contrastive Connection Learning to mitigate the impact of noise connections and more effectively utilize connection information with category consistency. Ablation study in Table 4 (a) verifies the effectiveness of these modules.

4. Explanation and Visualization of KNN interpolation Technique

While bilinear interpolation, as utilized in Mask2Former, is tailored for regular data, such as image, but it is unsuitable for irregular sparse primitive points. Here, We provided some visualizations of point masks for KNN interpolation and bilinear interpolation as shown in https://anonymous.4open.science/r/x-BB39/vis2-knn_interp-vs-bilinear_interp.png. Note that these point masks are soft masks (ranging from 0 to 1) predicted by intermediate layers. After downsampling the point mask to 4x and 16x, we can clearly find that KNN interpolation well perverse the original mask information interpolation, while bilinear interpolation causes a significant information loss, which could harm the final performance.

5. Submission of Code for Review

We appreciate the emphasis on the practical application of our framework. we promise to release our source code for result reproduction and promote the development of this field.