Differentiable Polygon Modeling for Object Instance Segmentation

1. It is unclear that what is the superiority of the polygon-modeling-based instance segmentation over the bit-mask modeling method. The performance of the polygon-modeling-based instance segmentation is also inferior to the bit-mask modeling method, e.g., SOLO [r1], SOLOv2 [r2], CondInst [r3]. This makes the significance of this work unclear.

2. The paper did not report the runtime latency. So the efficiency of the proposed method is unclear.

3. It is evidenced by Table 3 that a small $K$ is a better choice. Why the paper spend so many effort in presenting the method with a large $K=250$?

4. It would be better present the results of Sec. 3.1 at tables.

[r1] Wang X, Kong T, Shen C, et al. SOLO: Segmenting objects by locations. ECCV 2020.

[r2] Wang X, Zhang R, Kong T, et al. SOLOv2: Dynamic and fast instance segmentation. NeurIPS 2020.

[r3] Tian Z, Shen C, Chen H. Conditional convolutions for instance segmentation. ECCV 2020.

• The performance improvements seem marginal. For a paper in 2024, 35 AP on COCO is not satisfactory. To showcase the method's effectiveness, the authors should compare with stronger baselines using better backbones/training protocols. Otherwise, it is not convincing why the method is more promising than other approaches.
• The computational complexity and inference time analysis is missing.
• The method still relies on fixed topology (K vertices), limiting its flexibility.
• The authors should verify their method on more datasets such as Cityscapes, LVIS, etc.

Overall, the performance of this paper is not strong enough for this competitive instance segmentation task. If the authors believe polygon-based methods are promising, they should revise their experimental results with better baselines or find a better experimental setting (such as for improving human annotation).

1. The advantage of using polygons instead of bit masks as the prediction target is unclear. It is well-known that polygons are harder to learn and are limited to only connected instances. The authors did not discuss what taking polygons as training targets brings to us. Therefore, from this view, the paper is not well motivated.
2. The performance is moderate. The results in Table 1 do not show the instance segmentation results of the Sparse R-CNN method, which is the base model used by the authors and should be an important baseline. Since previous approaches listed in Table 1 applied different base models, the comparison cannot prove the method's advantage convincingly. Besides, Table 2's results also show that the improvement of the affine transformation is marginal. 
3. The original Sparse R-CNN reports AP 42.8%~46.4% on the COCO val set with different backbones. However, the authors claimed it can only achieve 37.9% (line 369). Therefore, the reviewer suggests aligning the base model with the original Sparse R-CNN for a more convenient comparison.
4. It lacks ablation studies of the refinement strategy.

• The effectiveness of the proposed method is not clear enough. Tables 2 and 3 show that the proposed modifications (i.e., larger K, affine transformations) both have a minor impact on the results. So what led to the performance improvement over previous work? Is it the proposed CLF sampling strategy or simply because it is based on a better detector (i.e., Sparse R-CNN)?
• Besides, ablation experiments on CLF-based vertex re-sampling are lacking.
• Fig. 5 does not show a clear advantage and instead, the contours of the proposed method appear to be over-smooth.
• Lack of inference speed comparison between the proposed method and previous methods.
• There are a lot of typos, for example, PolygonAligh (L240), use (L349), MS-COO (L440), repeated citation (L625 and L629).