EFDTR: Learnable Elliptical Fourier Descriptor Transformer for Instance Segmentation

Response to Reviewer GvJY

Thank you very much for your valuable and constructive feedback. We truly appreciate your time and effort. Below, we provide point-by-point responses to the concerns and suggestions you raised.

Response to More Dataset

We have conducted additional experiments on the SBD and Cityscapes datasets. The results are as follows:

SBD Results:

Cityscapes Results (Val):

Our method outperforms all other polygon-based instance segmentation methods on both datasets, achieving the highest AP. Notably, PolySnake, which is recognized as the best contour prediction model according to Paperwithcode, is surpassed by our approach by 10 AP$_{vol}$

Response to EFD with Higher Order

We evaluated the impact of different EFD orders on SBD and Cityscapes:

We observe that the first-order EFD consistently delivers the best performance. We hypothesize that, within the current model architecture, higher-order EFDs introduce optimization challenges, resulting in unstable point sampling during the second stage. In contrast, the first-order EFD acts as a stable and learnable geometric representation, analogous to rotated bounding boxes. Although higher-order EFDs are capable of capturing more detailed shape distributions, they tend to be unstable, which adversely affects the performance of the downstream polygon decoder. Below, we present samples of the fitting results using first- and fourth-order EFDs.

img

Response to Self-intersection and Degraded IoU

On the COCO dataset, we apply a minimum spanning tree (MST) algorithm to connect multiple polygons into a single contour. Among the COCO instances annotated with multiple polygons (9.71% of the total), we sample up to four polygons per instance, which accounts for 99.67% of all multi-polygon annotations. With a maximum of four polygons, the likelihood of self-intersection is extremely low. Given the small proportion of instances with more than four polygons, we believe the impact on training is minimal.

Response to Holes and EFD Order

In COCO, instance masks are annotated using simple polygons without regard to orientation, and thus do not include holes. However, the SBD dataset does contain instances with holes. As discussed in the Response to EFD with Higher Order, we observed that higher-order EFDs do not lead to better performance. Visualization shows that the 1st-order model captures the outer contour more robustly.

Although some hole-containing instances exist in SBD, they represent a small portion of the dataset. Our analysis suggests that the model’s strength lies in accurate outer contour detection rather than specific handling of holes. This could be a limitation of the current model architecture, which we aim to address in future work.

[1] Zhang T, Wei S, Ji S. E2ec: An end-to-end contour-based method for high-quality high-speed instance segmentation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 4443-4452.

[2] Feng H, Zhou K, Zhou W, et al. Recurrent generic contour-based instance segmentation with progressive learning[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024.

Response to Reviewer jXM9

Thank you for your constructive comments and insightful suggestions, which have helped us improve the quality and clarity of our manuscript. We address your points in detail below.

Response to Hole Case

The COCO dataset primarily uses polygon annotations for instance masks, with only 1.2% of annotations using RLE. Polygon annotations in COCO do not explicitly account for holes during the annotation process, meaning that hole structures are generally not represented in the ground truth masks. Therefore, during training on the original dataset, the model has limited exposure to true hole instances.

To further investigate this, we conducted experiments on the SBD dataset, which includes explicit annotations for hole regions. Upon qualitative analysis, we found that the current 1st-order EFD formulation struggles to accurately capture hole structures. However, using higher-order EFDs—while helpful for capturing fine-grained shapes—led to a drop in overall performance. This trade-off suggests a limitation in the current design, and we consider hole representation as a promising direction for future improvement.

Response to Balance between Sharp and Other Regions

This is an excellent point. Our current approach samples points uniformly from the EFD-predicted contours in the first stage. While this ensures consistent coverage, it may under-sample sharp corners or highly curved regions. A more adaptive sampling strategy—allocating denser points around such regions—could yield more accurate reconstructions. However, this would require an additional module to learn the sampling distribution dynamically. We greatly appreciate this suggestion and plan to explore it in future work.

Response to Reconstruction Performance

To evaluate the balance between reconstruction accuracy and inference speed, we tested different vertex group settings on the SBD dataset. All experiments were conducted at an input resolution of 512×512 using an NVIDIA RTX 3090 GPU, with CUDA 11.8, PyTorch backend, and FP32 precision. The results are as follows:

$^*$ E2EC is tested at an NVIDIA A6000.

These results demonstrate that our method maintains a favorable trade-off between accuracy and inference speed, especially compared to pixel-based methods like MaskDINO.

Response to Reviewer 3kZq

Thanks a lot for the time and effort you invested in providing the detailed reviews. Regarding the current weaknesses you pointed out, we are glad to give our responses.

Summary and Strengths

We are pleased that you found our method well motivated, the design sound, and the paper clearly written. We're also glad that you appreciated our comprehensive ablation studies and performance improvements over existing polygon-based methods.

Weakness: Comparison to Pixel-Based Methods

"The main weakness would be that this method still performs worse than some pixel-based approaches. While these polygon methods might be more efficient, it seems likely that for many applications, performance would be more important and pixel-based methods would be used instead."

Thank you for this valuable observation. We agree that in high-accuracy applications, pixel-based methods often remain the preferred choice. However, our goal is to provide a segmentation model that outputs structured data, which is particularly useful in tasks such as bird's-eye view vector map construction and remote sensing vector map generation. We believe our method can contribute to these areas by offering a novel solution.

Additionally, we provide inference performance comparisons in our submission, which highlight the efficiency of our method. For further context and elaboration, we also encourage referring to our responses to the other reviewers.

Once again, thank you for your valuable comments and support.

Response to Reviewer tj3E

We sincerely appreciate your time and effort in reviewing our paper and are glad to provide detailed responses to your insightful questions and suggestions.

Clarification on Method

While higher-order EFDs can offer finer contour approximations, the concern regarding "same phase but different amplitudes" may stem from a misunderstanding of the concept of phase in the EFD frequency domain versus angular coordinates in polar representations.

In our approach, Elliptic Fourier Descriptors establish a bijective mapping between each contour point $p(\theta)$ and a unique phase value $\theta \in [0, 2\pi)$. This bijection guarantees a one-to-one correspondence between contour points and their associated phase values in the frequency domain.

Our point regression strategy exploits this property to avoid ambiguities present in Cartesian and polar systems. For example, in polar coordinates, a ray from the centroid may intersect the contour at multiple points sharing the same angle but differing in radius. In contrast, under EFD, such points are associated with distinct phase values, as illustrated in the figure below.

img

Response to Vanilla Ellipse

To further validate the effectiveness of our two-stage architecture and the role of the first-stage EFD prediction, we conducted additional experiments on the SBD dataset in response to the reviewer’s concerns:

• Type 1: Identical to the original model, except the first stage predicts a naive inscribed ellipse.
• Type 2: Based on Type 1, the number of layers in the first stage is reduced to 2, while the second-stage polygon decoder is deepened to 6 layers.

The results demonstrate that the EFD-based first-stage prediction significantly outperforms the naive ellipse initialization. Moreover, increasing the depth of the second-stage decoder alone does not compensate for the lack of EFD-based guidance.

We attribute the superior performance of our design to the following:

1. Although both are ellipses, the 1st-order EFD behaves more like a PCA-based initialization, better capturing the object's principal orientation and shape distribution.
2. EFD provides a more principled framework for assigning regression targets than heuristic elliptical approximations.

Response to Larger Backbone

You raised an important point regarding the potential benefit of a stronger backbone in enhancing the learning of high-order EFDs. We conducted experiments with a Swin-L backbone using a 1× training schedule on the COCO val2017 set:

Interestingly, even with a more powerful backbone, the 1st-order EFD still achieves the highest mAP. We believe that, under the current architecture, high-order EFD parameters are difficult to optimize effectively, introducing instability in the second-stage point sampling. Conversely, the 1st-order EFD serves as a robust and easily learnable target, similar to rotated object detection, which helps stabilize the overall training.

Response to Model Design Question

Remove the EFD decoder and just use a polygon decoder with more layers.

This experiment is in the Response to Vanilla Ellipse section.

Additional Remarks

Our method is polygon-based, where both the output and the supervision signals are sequences of points. Within the domain of polygonal contour prediction, our method surpasses all prior work. Although in terms of AP, pixel-based methods outperform ours, we emphasize that our vector-based formulation offers a unified and end-to-end framework that holds practical value in applications such as vectorized HD map construction and remote sensing vectorization. We believe our contribution provides an effective approach for tasks requiring structured, vectorized outputs.

For further comparisons on additional datasets and inference efficiency, please refer to our responses to other reviewers. Once again, we sincerely thank you for your valuable feedback.