Weakness 3

As suggested, we further provided the extra comparison of whether reducing ambiguities during training or not is necessary. (1) The vocabulary dropout is a strategy that implicitly reduces ambiguous vocabulary during training. Alongside the Table 4 presents an ablation study on different dropout rates $p$, reflecting varying levels of ambiguity reduction. We conduct another experiment with the CAT-Seg model as shown below:

Reducing ambiguous vocabularies during training enhances the zero-shot capability of OVS. Additionally, we manually merged certain vocabularies from the top 10 ambiguous terms in COCO (e.g., house and building into house, building), explicitly reducing ambiguity. Training the CAT-Seg model with this refined vocabulary shows a slight improvement, as detailed below:

However, we would like to emphasise that this is not the main contribution of our paper. Our primary focus is to propose a new evaluation method for OVS models that resolves the issue that traditional image segmentation metrics may misclassify visually similar objects as errors. Ambiguous vocabulary graph (matrix) is a tool helpful for understanding the OVS model predictions.

We would be happy to provide further details and experiments if the reviewer have any questions, thanks!

Weakness 1

Thank you for your comment. We clarify that the ambiguous vocabulary graph does not affect model performance or evaluation metrics. It is a tool to analyse overlaps in masks with differing category labels between model predictions and human annotations. While it aids in understanding ambiguous predictions, it is not a factor in performance evaluation, which relies on metrics like $front$, $back$, and $err$. And we provide extra following analysis of the ambiguous vocabulary graph.

We take CAT-Seg model as an example, the top10 ambiguous vocabularies are shown below:

This analysis highlights dataset labelling issues (e.g., ambiguous category definitions) and model limitations (e.g., difficulty in capturing category boundaries), guiding improvements in labelling and model optimisation. For instance, in response to Weakness 3, we enhanced performance by merging ambiguous vocabularies in the training data.

We would be happy to provide further details and experiments if the reviewer have any questions, thanks!

Weakness 2

Thank you for the comments. The updated Algorithm 1 in the revised version illustrates the process of obtaining the confusion matrix (CM), ambiguous vocabulary matrix (AV), and error matrix (EM). The computation of our three metrics is detailed in Section 4, as shown below:

$front_\tau = \frac{1}{|C|}\sum_{c \in C} \frac{CM_\tau[TP_c]}{CM_{\tau[TP_c]} + CM_\tau[FP_c] + CM_\tau[FN_c]}$

$back_\tau = \frac{1}{|C|}\sum_{c \in C} \frac{CM_\tau[TN_c]}{CM_\tau[TN_c] + CM_\tau[FP_c] + CM_\tau[FN_c]}$

$err_\tau = \frac{1}{|C|}\sum_{c \in C} EM_{\tau, c}$

where $\tau$ is the threshold, $c$ is the class.

We would be happy to provide further details and experiments if the reviewer have any questions, thanks!

Weakness 1

How to determine "better" performance? Thank you for the suggestion. In our context, "better" refers to the quantitative superior performance of the OVS model compared to the pixel-wise model within our mask-wise evaluation framework. We attribute this improvement to addressing the limitations of the pixel-wise evaluation framework, where mis-matched categories are forcibly treated as erroneous predictions. We have revised this in the revised version, line 20 - 22. We would be happy to provide further details and experiments if the reviewer have any questions, thanks!

Weakness 2

Thank you for pointing out the writing suggestions. We intended to refer to the capabilities of OVS models and have corrected this in the revised version, line 25. We would be happy to provide further details and experiments if the reviewer have any questions, thanks!

Weakness 3

Thank you for pointing out the writing suggestions. "open-vocabulary segmentation" specifically refers to "open-vocabulary semantic segmentation." We have updated all of them in the revised version. We would be happy to provide further details and experiments if the reviewer have any questions, thanks!

Weakness 4

The best threshold $\tau^\star$ is used to automatically determine the threshold at which the model performs best. It considers simultaneously maximising the value of $front$ and $(1 - err)$. Under our evaluation framework, the theoretical optimal values are $front = 1$ and $err = 0$. This indicates a model that achieves 100% accuracy in identifying target categories while making no incorrect predictions for non-target categories. The best threshold refers to the point closest to the theoretical optimal value in terms of Euclidean distance. We would be happy to provide further details and experiments if the reviewer have any questions, thanks!

Weakness 5

Thank you for pointing out the writing suggestions. We have updated this in the revised version, line 340.

Q1

Thank you for the insightful feedback. We recognise the importance of ensuring a coherent narrative throughout the paper. Specifically, Equation (3) introduces $P(V\mid\Theta)$ showing that the model's parameters need to fit the distribution of the given vocabulary, so ambiguous vocabularies present during training may affect the model's optimisation. This serves as the motivation for the analysis of the ambiguous vocabulary graph and the reducing ambiguous vocabularies during training detailed in Section 6.2. We would be happy to provide further details and experiments if the reviewer have any questions, thanks!

Q2

Thank you for the comments. Apologies for the typo of omitting the definitions in the manuscript and have updated the following in the revised version.

The $\mathbb{A}$ is all the predicted masks, $\mathbb{B}$ is the mask list where the predicted category by the model aligns with the category annotated by human (not all predicted masks). The $\mathbb{\hat{B}}$ is the set of masks obtained by performing bipartite matching between $(\mathbb{A} \setminus \mathbb{B})$ and the GT, where the IoU of the matched pairs exceeds the threshold $\tau_{AV}$. $\mathbb{C}$ is defined as $\mathbb{C}=\mathbb{A} \setminus (\mathbb{\hat{B}} \cup \mathbb{B})$. We would be happy to provide further details and experiments if the reviewer have any questions, thanks!

Q3

(1) Thank you for the feedback. The updated Algorithm 1 in the revised version illustrates the process of obtaining the confusion matrix (CM), ambiguous vocabulary matrix (AV), and error matrix (EM). We have added the contents revised version, Section 4.2 where CM is then used to compute the metrics $front$ and $back$.

(2) The computation of our three metrics is also detailed in Section 4, as shown below:

$front_\tau = \frac{1}{|C|}\sum_{c \in C} \frac{CM_\tau[TP_c]}{CM_{\tau[TP_c]} + CM_\tau[FP_c] + CM_\tau[FN_c]}$

$back_\tau = \frac{1}{|C|}\sum_{c \in C} \frac{CM_\tau[TN_c]}{CM_\tau[TN_c] + CM_\tau[FP_c] + CM_\tau[FN_c]}$

$err_\tau = \frac{1}{|C|}\sum_{c \in C} EM_{\tau, c}$

where $\tau$ is the threshold, $c$ is the class. We have added this in the revised version.

(3) The best threshold is not directly applied in Algorithm 1. When we obtain the $front$, $back$, and $error$ metrics under different thresholds $\tau$, the best threshold is identified as the threshold that model performances the best.

We would be happy to provide further details and experiments if the reviewer have any questions, thanks!

Dear Reviewer eu9j,

Thank you for your contributions to the reviewing work.

As the deadline for reviewer-author discussion is approaching soon, we kindly ask you to take a look at our responses at your convenience, and let us know if your concerns have been well addressed.

We would be more than happy to provide any additional responses or justifications if you have further concern.

Many thanks and looking forward to hearing from you.

Dear reviewers,

We thank you again for your effort in reviewing our paper, and appreciate the constructive comments. We have addressed the concerns and updated the paper accordingly.

The deadline for reviewers/authors to post messages is approaching, and we may not be able to see your comments soon. We understand your busy schedule, but it would be appreciated if you could let us know if our responses have addressed your previous concerns, and any further comments or questions you may have, and we will do our best to address them before the deadline.

Thanks again for your feedback and time!