Open-Vocabulary Part Segmentation via Progressive and Boundary-Aware Strategy

We thank reviewer for the constructive comments. We provide our feedbacks as follows.

W1-W3: Comparison with prior works.

A1: We agree that distinguishing our work from prior methods is essential for clarifying its contributions. In response, we provide the following explanation:

• DiffuMask [45] uses attention maps from Stable Diffusion to obtain category masks for generating semantic segmentation datasets. OVDiff [46] combines diffusion models, segmenters, and feature extraction models to achieve OVSS. RIM [38], building on OVDiff, introduces a stronger feature encoder (DINOv2) and segmenter (SAM), and then conducts relation-aware matching. We highly recognize the contributions of these works, among which the visual prototype construction module provides an important foundation for subsequent research. Our paper follows this technical route, but our innovation does not lie in prototype construction. We focus on the problem of "low segmentation accuracy for structurally connected part boundaries" in OVPS. Guided by HPCGraph, hierarchical segmentation is performed, and the BAR module improves boundary precision via feature optimization and dynamic prototype mechanisms. This is also the key reason why our method outperforms OVDiff, RIM, and other methods on the three datasets (as shown in Table 2).

• The progressive segmentation guided by HPCGraph also differs significantly from the top-down fine-grained segmentation in ViRReq [24] and TAPPS [62]. Firstly, when calculating the matching score between pixel features and visual prototypes (L145-147), we fully utilize object-part and part-part relationships. The matching cost between a pixel feature f and a node $v ∈ V$ is defined as the maximum cosine similarity within its dominance set $D(v)$ = {$v$} $\cup$ { descendant nodes of $v$ }. This combines the current node with all its child nodes into a joint semantic space, thereby fully leveraging hierarchical context to enhance discriminative power. In addition, the adjacency edges in HPCGraph representing the structural connection between two parts (L141-144). These adjacency edges guide BAR to refine the boundaries of structurally connected parts.

• It should be emphasized that we have never intentionally avoided referencing existing works, nor have we claimed visual prototype construction as our innovative contribution. In Section 2 (Related Work), we provided a brief overview of the aforementioned related works. In Table 2, our method was also fairly compared with these works. Meanwhile, to avoid misleading readers, Section 4.1 only briefly describes the prototype generation process and cites relevant works. Due to the space constraints of the main text, we did not elaborate on the differences between our method and existing works. In the subsequent revision of the paper, we will add a separate section to this end, helping readers more intuitively understand the core contributions and research significance of our work.

W4: The statement "propose the first training-free OVPS framework" in the abstract may cause ambiguity.

A2: Thank you for your valuable comment, and we apologize for any confusion caused by the original wording regarding “the first training-free OVPS framework.” We provide the following clarification:

• To avoid any potential ambiguity, we will revise the original abstract statement to: “We propose a novel and effective training-free framework specifically designed for OVPS...” This revision more accurately convey the positioning and contributions of this work.
• Although OVDiff and RIM can indeed be applied to OVPS, they are not inherently designed for part-level segmentation tasks. Specifically, these methods focus on object-level OVSS, and their usage in OVPS represents only an indirect adaptation. They do not address OVPS-specific challenges such as structural dependencies between parts and boundary ambiguity.
• In contrast, our method PBAPS is a novel and effective training-free framework explicitly designed for the OVPS task. It targets the core challenge of “low segmentation accuracy at structurally connected part boundaries.” Under the guidance of HPCGraph, PBAPS performs hierarchical segmentation and refines boundaries via the BAR module. As shown in Table 2, under the same prototype generation setup, PBAPS outperforms the state-of-the-art method RIM by 7.3%, 18.1%, and 8.4% on Pascal-Part-116, ADE20K-Part-234, and PartImageNet, respectively.

W5: Readability of the paper.

A3: Thank you for your suggestions regarding the readability of the paper. We will make the following specific revisions:

• For non-generic terms such as "cross-hierarchy matching" and "deterministic region" that appear early in the abstract and introduction, we will add brief, context-rich explanations when they first appear (e.g., explaining how "deterministic regions" serve as reliable context for refining ambiguous boundaries), ensuring readers grasp their core meaning upfront.
• We will restructure the abstract and introduction to first outline the overall goal of OVPS and highlight the key challenge of structurally connected part boundaries. The PBAPS framework and its core modules (HPCGraph, BAR) will then be introduced progressively in a logical flow, with technical details deferred until the basic concepts are established. Additionally, we will simplify overly technical language to better emphasize the motivation, main contributions, and high-level workflow, thereby improving accessibility for readers without a specialized background.

W6: Some operations in the paper lack sufficiently detailed explanations.

A4: Thank you for pointing out that some operations in this paper lack detailed explanations. In response, we provide the following supplementary notes:

• Regarding the extraction of cross-attention maps in L130–131. To avoid misleading readers into thinking this is an innovation of our work, we only give a brief description of this part in the main text (L130–131) and provide more details in Appendix B.1. We further supplement as follows: we follow the prior works such as DiffuMask [45], OVDiff [46], and RIM [38], obtaining robust localization signals through cross-timestep and cross-layer aggregation. Cross-attention maps are extracted from all 50 diffusion timesteps, and noise from a single step is suppressed through an averaging operation. The attention maps come from the 16×16 resolution cross-attention layers generated during the downsampling process of the U-Net, as well as the 32×32 and 64×64 resolution cross-attention layers generated during the upsampling process. Finally, attention maps of different resolutions are scaled to 512×512 through interpolation and weighted averaged to generate the final semantic localization heatmap.
• Regarding the feature extraction for masked regions in L134. We use the masked image as input to DINOv2 for pixel feature extraction, instead of extracting features from the entire image first and then applying masking. This pre-masking operation enhances the accuracy of prototype construction.
• Regarding the inconsistent descriptions of the number of visual prototypes (L135, L208, L522). Each part category corresponds to multiple subcategory prototypes, which is the configuration of our method. However, in L135, for the sake of narrative simplicity and to help readers understand, we described the visual prototype as "one", and this wording may have caused misunderstandings. We apologize for this.
• Regarding the calculation range of the cost divergence map D (L165–166). Your understanding is correct. The cost divergence map is calculated only within the parent mask region. When constructing the cost divergence map, we only consider pixels that belong to the upper-level part segmentation mask (parent mask) to avoid misjudgment caused by irrelevant regions.
• Regarding the number of synthetic images (L204). For most part categories, we generated 120 prototype images; only for a few categories with poor generation results, an additional 30 images were generated to improve prototype quality. Since we cannot add images, PDFs, or links during this NeurIPS rebuttal stage, we will provide a complete list of image quantity configurations for all categories in subsequent revisions of the paper and make the corresponding image data publicly available.

W7: The scalability of HPCGraph.

A5: Due to the space constraints of the rebuttal, we kindly refer you to our detailed response to Reviewer 76j9’s Weakness 3, where we provide an in-depth explanation of the scalability of HPCGraph, including its construction principles, reuse mechanism across structurally similar categories (group 158 classes into 11 superclasses, topology count drops by 93 % while PBAPS still leads RIM by 8.4 %), and dynamic graph generation mechanism (automatically add adjacency edges for highly relevant parts based on the spatial adjacency and visual feature similarity between part response regions). The comparative results between dynamic and static graphs are as follows:

[24] Visual recognition by request (CVPR 2023)

[38] Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation (CVPR 2024)

[45] Diffumask: Synthesizing391 images with pixel-level annotations for semantic segmentation using diffusion models (ICCV 2023)

[46] OVDiff: Diffusion Models for Open-Vocabulary Segmentation (ECCV 2024)

[62] Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations (CVPR 2024)

(2/3)

Q3: HPCGraph is constructed based on commonsense structural knowledge [..] and does not rely on any manual annotations. It is not at all clear to my why it does not rely on manual annotations. If this has to be done for a truly open vocabulary (i.e., millions of concepts), this would require a lot of manual labor.

A3: We apologize for the ambiguity. By "HPCGraph does not rely on manual annotations," we specifically mean it avoids costly dense visual annotations like pixel-level masks or bounding boxes used in traditional supervised training.

The construction of HPCGraph can be based on existing general knowledge bases (such as WordNet, ConceptNet, etc.), which widely contain semantic relationships between objects and parts like "part-of" and "has-part". Compared with the traditional method that relies on pixel-level annotation, this kind of structural knowledge definition has controllable workload and strong cross-category reusability—with superclass merging and LLM-assisted parsing, manual effort is further reduced.

Regarding the effectiveness of the topology reuse mechanism, it has been verified on PartImageNet in the paper: after dividing 158 categories into 11 superclasses, the number of topological structures is reduced by 93%, while PBAPS improves upon RIM by 8.4% relatively (Table 2), indicating that topological structures have good shareability. We further supplement the judgment method for topology reuse: based on the hierarchical taxonomy provided by WordNet, we first query the hypernym chain of a target category (e.g., “tiger” is a hyponym of “quadruped”) and then combine this with structural features such as limb count and part connectivity to identify semantically close and structurally similar categories (e.g., “tiger,” “leopard,” and “golden retriever” all share a “head-torso-four-limb” configuration). These are grouped into the same superclass (e.g., “quadruped animals”), which is then assigned a unified part topology.

To verify the scalability of this mechanism, we follow prior work[63] and extract 10,000 concepts from the CC3M dataset. For each concept, we obtain its WordNet hypernym chain (e.g., "Persian cat" → "cat" → "mammal" → "quadruped") and cluster concepts with semantic distance ≤2 (e.g., "dog," "wolf," and "fox" into the superclass "canine quadrupeds") based on convergence in their upper-level hierarchy. In this way, the 10,000 concepts are grouped into 40 superclasses, each corresponding to a shared topology. This process not only ensures the validity of topology reuse within each superclass but also significantly reduces the number of topologies required under large-scale settings, validating the feasibility and efficiency of our topology reuse mechanism.

[63] Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation (CVPR 2024)

[64] Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (ACL 2018)

(3/3)

Q4: The rebuttal states that visual prototypes could be reused, e.g., 'horse-leg' could be applied to 'donkey-leg'. Are there empirical results that demonstrate that this can be done? How do you know that visual prototypes can be reused?

A4: Since part prototypes directly participate in the calculation of classification scores, the conditions for reuse are more stringent than those for part-topology reuse. Even within the same superclass, it is not appropriate to naively apply a single set of part prototypes to all categories. To address this, we employ LLMs to analyze local structural and textural similarities between categories and determine whether reuse is appropriate.

We designed experiment on the PartImageNet dataset to verify the feasibility of part prototype reuse. PartImageNet contains 40 object categories, which can be divided into 11 superclasses based on their morphological and structural characteristics. All objects within a superclass share the same part topology. It should be noted that due to the significant differences in overall appearance between different object categories (e.g., "goldfish" and "killer whale"), we still need to generate separate visual prototypes for each object to ensure the accuracy of object-level segmentation. At the part level, we explored the possibility of reusing prototypes across objects.

For each superclass, we selected a most representative category (e.g., "tiger" for the "Quadruped" superclass) and generated complete part visual prototypes for it (including tiger head, tiger torso, etc.). Next, through LLM analysis and screening: objects with significant differences in local structure or texture from the representative category (e.g., "giant panda" vs. "tiger") are excluded from reuse candidates to avoid introducing errors. After screening, the part segmentation of the remaining objects in the superclass directly reuses the part prototypes of the representative category without the need to generate new prototypes. The specific representative categories and target categories are shown in the table below:

We compared the performance of models with and without the part prototype reuse strategy, and the results are shown in the table below. Although the reuse of part prototypes leads to a certain degree of degradation in segmentation performance, the overall performance remains well usable. In scenarios with large-scale categories, this strategy significantly reduces the cost of generating visual prototypes while maintaining good segmentation results. In summary, under the premise of high consistency in local structure and texture within a superclass, the reuse of visual prototypes is feasible, eliminating the need to generate entirely new part prototypes for each object category.

We once again extend our sincere respect to you. Regarding the responses during the Rebuttal stage and the supplementary content provided this time, if you still have any questions or need further clarification on any aspects, please do not hesitate to let us know. We will be happy to provide explanations for you.

(1/3)

Thank you sincerely for your timely feedback. We provide our feedbacks as follows.

Q1: What will the revised abstract look like exactly? Could you please provide a revised version of the text?

A1: Thank you for your suggestions to help refine our paper. Following your advice, only minor revisions are needed to achieve clearer readability. For example, we replaced the original term "matching cost divergence" with "measuring classification uncertainty", and "deterministic-region context" with “high-confidence context"—phrases that more intuitively convey the underlying ideas without assuming prior familiarity with our technical design. The revised abstract is as follows:

Open-vocabulary part segmentation (OVPS) struggles with structurally connected boundaries due to the conflict between continuous image features and discrete classification. To address this, we propose PBAPS, a novel training-free framework specifically designed for OVPS. PBAPS leverages structural knowledge of object-part relationships to guide a progressive segmentation from objects to fine-grained parts. To further improve accuracy at challenging boundaries, we introduce a Boundary-Aware Refinement (BAR) module that identifies ambiguous boundary regions by measuring classification uncertainty (quantified via the difference in classification scores between adjacent parts), enhances their features using high-confidence context (through fusion with features from reliably classified pixels), and adaptively refines part prototypes to better align with the specific image. Experiments on Pascal-Part-116, ADE20K-Part-234, PartImageNet demonstrate that PBAPS significantly outperforms state-of-the-art methods, achieving 46.35% mIoU and 34.46% bIoU on Pascal-Part-116.

Q2: We would have to feed all these millions of concepts through Stable Diffusion, SAM, and DINOv2, generating hundreds of millions of images to obtain the visual prototypes, which is rather inefficient. Even if semantic categories can share part hierarchies in the HPCGraph (as done for PartImageNet), a true open-vocabulary model would require the generation of millions of images to obtain the visual prototypes, which is inefficient.

A2: We fully agree with your point: when scaling to millions of semantic concepts, the process of generating visual prototypes indeed faces efficiency limitations. However, it should be clarified that such efficiency limitations are relative to the extreme scenario where "no reuse mechanism is employed, requiring the generation of hundreds of millions of images." Regarding your statement that "Even if..., a true open-vocabulary model would require the generation of millions of images to obtain the visual prototypes, which is inefficient." , we have verified the feasibility of efficiency through actual generation tests: using 8 A6000 GPUs for parallel generation, we have accumulated 181K images over 21 hours. Based on this, it is estimated that generating one million images would take no more than 5 days, an efficiency acceptable in practical applications. We further supplement from the perspectives of technical background and work positioning:

• First, to the best of our knowledge, no semantic segmentation method can truly and efficiently handle millions of concepts. The core obstacle lies in the requirement for massive densely annotated data to train such models. The annotation costs alone far exceed practical feasibility, and the model also faces huge challenges in representing and generalizing such a large semantic space. Thus, the core goal of open-vocabulary segmentation is not to "cover infinite categories," but to break free from reliance on large-scale pixel-level annotations, exploring universal segmentation solutions for low-resource, weakly supervised, or zero-shot scenarios. In this context, our method uses visual prototype matching for label assignment, leveraging the validated pipeline to obtain visual prototypes, which aligns with the fundamental intent of open-vocabulary tasks.
• Second, the core contribution of our work is not to "expand category coverage," but to address segmentation errors caused by "ambiguous boundaries and structural coupling between parts" under the open-vocabulary setting. It should be emphasized that PBAPS itself is agnostic to the source of prototypes, offering high plug-and-play flexibility and universality. PBAPS does not rely on a specific prototype generation pipeline. We can directly leverage large-scale image-text pair data from the internet, extract candidate regions (via SAM), and then use CLIP to match regions with text, thereby mining highly diverse and naturally distributed part prototypes in the real world.

We thank reviewer for the constructive comments. We provide our feedbacks as follows.

W1: Dependence on foundation models.

A1: Regarding PBAPS's dependence on foundation models, we need to clarify: this dependence is not a flaw but rather a best practice in cutting-edge research. The value of this paper lies in how to leverage the capabilities of foundation models to solve the core challenge of OVPS, namely "low segmentation accuracy for structurally connected part boundaries."

Foundation models such as Stable Diffusion, SAM, and DINOv2, having been trained on massive amounts of data, possess strong general visual understanding capabilities, which serve as the premise for PBAPS's training-free design. The outputs of foundation models (such as SAM's masks and DINOv2's features) are essentially general visual signals, whereas OVPS requires structured cognitive understanding of part relationships: How to locate part boundaries from diffuse attention maps? How to adapt global prototypes to instance-specific differences in specific images? Our HPCGraph, through structured knowledge constraints, guides the model in hierarchical reasoning; the dynamic prototype mechanism within the BAR addresses the misalignment between foundation model outputs and the actual distribution of parts. These designs are our innovation—not relying on the foundation models, but constructing the paradigm for applying their capabilities. When more powerful foundation models emerge, our method can directly reuse their capabilities without any modifications.

W2: Sensitivity to hyperparameters.

A2: Due to the space constraints of the rebuttal, we kindly refer you to our detailed response to Reviewer Sy1A’s Weakness 2, where we thoroughly address the selection rationale, robustness, and generalizability of our hyperparameter settings across diverse datasets.

W3: Manual construction of HPCGraph.

A3: Regarding the scalability of HPCGraph, we provide the following supplementary explanations:

• HPCGraph is constructed based on commonsense structural knowledge and does not rely on any manual annotations. As such, it incurs low construction cost and exhibits strong transferability and scalability. In large-scale object scenarios, both the topological structure and visual prototypes can be shared and reused. For novel object categories with similar structures to existing ones (e.g., “zebra” as a novel category relative to the existing category “horse”), we can directly reuse existing topologies (e.g., “zebra head ↔ zebra neck” correspond to the structure of "horse"). For PartImageNet (L198–200), we divided 158 object categories into 11 superclasses, where each superclass shares a unified topological graph. This strategy reduces the number of topology by 93% (from 158 to 11) while maintaining high performance—PBAPS outperforms RIM by 8.4% on PartImageNet (Table 2). When the number of object categories expands to tens of thousands, a large number of them are highly similar in form and structure (such as horses, donkeys, mules, zebras, etc.). We can use LLMs to analyze inter-category structural similarities and achieve cluster-level topology reuse (i.e., categories with similar structures share a set of topologies). Furthermore, part visual prototypes can also be reused (e.g., the prototype for “horse leg” can be applied to “donkey leg”).
• In contrast to existing methods such as PartCLIPSeg [27] and PartCATSeg [53], which require extensive part-level annotations and retraining when introducing new object categories, our method is training-free and supervision-independent. It supports open-world scalability by simply extending the HPCGraph and adding a small number of visual prototypes.
• As noted in Appendix A.1, when dealing with objects with irregular structures (e.g., modular furniture), the static graph structure has insufficient adaptability. To address this, a dynamic graph generation mechanism can be considered. The core logic is to automatically add adjacency edges for highly relevant parts based on the spatial adjacency and visual feature similarity between part response regions. The following table compares performance using static and dynamic graphs. While static graphs yield better performance in well-structured domains, the dynamic version provides competitive results and significantly enhances flexibility for deployment in unconstrained real-world scenarios. | HPCGraph | Pascal-Part-116 | ADE20K-Part-234 | PartImageNet| |:-|:-:|:-:|:-:| | | mIoU  bIoU | mIoU  bIoU| mIoU  bIoU | | Dynamic | 44.63  33.12 |23.87  15.63| 41.66  28.72 | | Static | 46.35  34.46 |24.70  16.41 | 42.61  29.31 |

W4: Incomplete ablation study.

A4: In response to your comment, we supplement the ablation experiment results of PBAPS on the ADE20K-Part-234 and PartImageNet datasets. As shown in the table below, on three datasets with significant differences (natural objects, indoor scenes, and cross-domain complex objects), each component significantly improves the model performance (mIoU). This not only verifies the effectiveness of each component design but also demonstrates that our method has good generalizability and robustness in diverse data environments.

Q1: What is the likelihood that the resulting visual prototype actually represents the region corresponding to the prompt?

A5: Thank you for your valuable comment, we provide the following supplementary explanation:

• It should be emphasized that the approach of constructing visual prototypes using Stable Diffusion + SAM + DINO has been adopted and validated as effective in multiple existing works (e.g., OVDiff [46], RIM [38]). Our contribution does not lie in the prototype construction, but rather in addressing the challenge of low segmentation accuracy at structurally connected part boundaries in OVPS. Guided by HPCGraph, hierarchical segmentation is performed, and the BAR module improves boundary precision via feature optimization and dynamic prototype mechanisms. This design is the key reason why our method outperforms OVDiff, RIM, and other methods on the three datasets (Table 2).
• Regarding whether Stable Diffusion can accurately localize prompt regions, we conducted a quantitative verification using CLIP cross-modal similarity. For the synthetic images of each part category, SAM generates target masks based on the attention maps of Stable Diffusion. Subsequently, we used CLIP to calculate the similarity between the masked regions, non-masked regions, and the entire image with the target text prompts, respectively. The results are shown in the table below. The similarity score of the target masked regions is significantly higher than that of the entire image, verifying that Stable Diffusion can accurately focus on the target parts. In addition, the low similarity score of non-masked regions also indirectly confirms the accuracy of the masked regions.

• Furthermore, even if there is a certain deviation in the target masks (e.g., including a small number of non-target regions), leading to ambiguity in the generated part visual prototypes, the "dynamic prototype adaptation" mechanism of the BAR module can effectively alleviate this issue. By weighted fusion of global prototypes with the features of deterministic regions in the current image, the prototypes are dynamically adapted to the real target parts in the image, thereby improving segmentation accuracy. The ablation experiment in Table 3 also confirms the effectiveness of the "dynamic prototype adaptation" mechanism.

Q2: From which timestep and which specific layer of the U-Net do you extract the attention maps for localization purposes?

A6: As shown in L129-131 and L456-467, we follow the prior works such as DiffuMask [45], OVDiff [46], and RIM [38], obtaining robust localization signals through cross-timestep and cross-layer aggregation. Cross-attention maps are extracted from all 50 diffusion timesteps, and noise from a single step is suppressed through an averaging operation. The attention maps come from the 16×16 resolution cross-attention layers generated during the downsampling process of the U-Net, as well as the 32×32 and 64×64 resolution cross-attention layers generated during the upsampling process. Finally, attention maps of different resolutions are scaled to 512×512 through interpolation and weighted averaged to generate the final semantic localization heatmap.

[27] PartCLIPSeg: Understanding Multi-Granularityfor Open-Vocabulary Part Segmentation (NeurIPS 2024)

[38] Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation (CVPR 2024)

[45] Diffumask: Synthesizing391 images with pixel-level annotations for semantic segmentation using diffusion models (ICCV 2023)

[46] OVDiff: Diffusion Models for Open-Vocabulary Segmentation (ECCV 2024)

[53] PartCATSeg: Fine-Grained Image-Text Correspondence withCost Aggregation for Open-Vocabulary Part Segmentation (CVPR 2025)

Dear Reviewers,

We sincerely thank you for your time and effort in reviewing our paper and offering valuable suggestions. As the author-reviewer discussion deadline is approaching, we would like to kindly confirm whether our previous responses have effectively addressed your concerns. We submitted detailed replies to your comments a few days ago and hope they have clarified the issues you raised. If any questions remain or if further clarification is needed, please do not hesitate to let us know. We are more than happy to continue the discussion and provide any additional information you may require.

Best regards,

The authors of Paper ID 8092

Thank you very much for your prompt and thoughtful feedback. Regarding the phenomenon in the W4 ablation study where "the combined gain is less than the sum of individual gains," we provide the following explanation:

First, in the ablation study, "w/ HPCGraph" refers to the introduction of hierarchical segmentation with cross-hierarchy matching (L227-229), and "w/ BAR" refers to the introduction of the BAR module to optimize the boundaries of structurally connected parts (L231-236). HPCGraph provides prior constraints on the overall structure, primarily addressing global localization errors caused by structural inconsistencies, whereas BAR focuses on locally connected regions, improving boundary segmentation accuracy through uncertainty modeling and feature refinement.

Although their design objectives differ, they may correct the same segmentation errors during actual inference. For example, in the case of the "cat leg-torso boundary": HPCGraph uses structural priors to guide the model to first locate the overall region of the cat, then divide parts such as legs and torso within it, reducing localization errors in the boundary region from a global perspective; BAR optimizes boundary precision from a local perspective by enhancing the feature discriminability of the boundary region. When combined, some errors are already corrected in advance by HPCGraph, leading to compressed incremental space for the BAR module. Thus, the performance improvement shows diminishing marginal returns—which is not an "interference effect" between modules but a natural result of functional complementarity. This cross-correction capability precisely confirms the synergy of the two modules: they have clear divisions of labor in general directions (structural modeling vs. boundary optimization) and complementarily cover specific error cases, ultimately achieving overall performance improvement through the progressive logic of "structural constraints + detail optimization."

To further verify the complementarity of the two modules, we designed a cross-region ablation experiment to clarify the scope of action and collaborative mechanism of each module. Based on classification uncertainty (L164-170), we divided the image prediction masks into:

• Core Region: High-confidence, structurally clear regions with concentrated target distribution, such as body trunks and heads;
• Boundary Region: Structurally ambiguous, texture-complex regions or junctions of multiple parts, such as limb joints and component seams.

Based on the above region division, we evaluated the segmentation performance of different model configurations on the ADE20K-Part-234. As shown in the table below, HPCGraph achieves the most significant improvement in the segmentation accuracy of core regions, verifying its ability to model the overall structure; BAR shows more obvious improvements in boundary region segmentation, confirming its refinement effect on locally ambiguous regions; the combined model achieves the best results in both regions, further confirming the spatial complementarity of the two modules rather than redundancy or mutual interference.

In summary, although the combined gain does not numerically equal the sum of the gains from the two modules, this is a reasonable phenomenon of diminishing marginal effects and does not indicate "negative interference." In fact, the complete PBAPS framework consistently achieves the optimal performance across all tested datasets (W4 Table). HPCGraph and BAR synergistically optimize at different granularities, collectively improving segmentation results through structural modeling and boundary perception, respectively.

Thank you again for your valuable feedback! If you have any further questions, we are happy to provide detailed explanations.

Thank you for your time and thoughtful feedback. We appreciate your consideration and are grateful for the opportunity to address your concerns.

We thank reviewer for the constructive comments. We provide our feedbacks as follows.

W1: The method relies on generating hundreds of prototype images per part class, which may introduce a risk of overfitting or inadvertently aligning too closely with the test set, especially on relatively small or simple datasets. It would be helpful to demonstrate the method on more complex or diverse scenarios.

A1: Thank you for pointing out this important concern regarding the potential proximity between generated prototypes and the test set. We provide the following clarifications:

• The prototype images are entirely synthesized by Stable Diffusion without using any images from the target datasets. All part prototype images are generated using the publicly available pre-trained text-to-image model (Stable Diffusion v1.4). These synthesized prototypes differ significantly from real-world test images in style, background, and composition, thereby minimizing the risk of overfitting or unintended alignment with the test set.

• To quantitatively assess the distinctness between the synthesized prototypes and the test set images, we conducted a feature-level similarity analysis. First, we input the prototype set and test set images into DINOv2 ViT-B/16 to extract global image features. Then, we performed L2 normalization on these image features and calculated the mean values of the features of the prototype set and test set respectively as their central vectors. The cosine similarity between these two central vectors was used to quantify the similarity between the prototype set and the test set. For comparison, we also calculated the similarity within the test set. We randomly split the test set into two halves: one half was used to generate a central vector $\mathbf{V}_1$, whose similarity with the prototype set’s central vector $\mathbf{V}$ was calculated. The other half was used to generate a central vector $\mathbf{V}_2$, whose similarity with $\mathbf{V}_1$ was calculated as a reference for the consistency of the test set. The results are shown in the table below. The similarity between the prototype set and the test set is significantly lower than the internal similarity of the test set, indicating that the synthesized prototypes are significantly different from the test set images and there is no risk of "excessive proximity". | Similarity Type | Pascal-Part-116 | ADE20K-Part-234 | PartImageNet| |:-|:-:|:-:|:-:| | Prototype set vs. Test set |0.63 ± 0.01|0.59 ± 0.01|0.62 ± 0.01| | Within Test set |0.87 ± 0.01|0.86 ± 0.02|0.89 ± 0.01|

• In terms of generalizability, PBAPS has been comprehensively evaluated on three highly diverse datasets: Pascal-Part-116 (natural objects such as animals), ADE20K-Part-234 (indoor scenes with furniture and appliances), and PartImageNet (cross-domain categories like vehicles and instruments). These datasets vary significantly in part types, scene complexity, and domain coverage. As shown in Table 2, using a single prototype generation process and fixed hyperparameters, PBAPS achieves state-of-the-art results across all three datasets, validating its strong cross-domain generalization ability.

In summary, the prototype generation process in PBAPS is entirely independent of the target datasets and does not introduce any data leakage risk. Our quantitative analysis confirms that the synthesized prototypes are significantly distinct from the test images, alleviating concerns of overfitting or excessive similarity. Furthermore, the consistent performance across diverse datasets demonstrates the robustness and generalizability of our method, highlighting its suitability for large-scale and cross-domain open-vocabulary part segmentation tasks.

W2: The paper lacks an analysis of how the similarity between generated prototypes and dataset images affects performance, which would be important to clarify whether the method generalizes or exploits dataset biases.

A2: Thank you for highlighting the potential impact of prototype-test image similarity on model generalization. We provide the following clarifications and additional analysis:

• It should be emphasized that the approach of constructing visual prototypes using Stable Diffusion + SAM + DINO has been adopted and validated as effective in multiple existing works (e.g., OVDiff [46], RIM [38]). Our contribution does not lie in the prototype construction, but rather in addressing the challenge of low segmentation accuracy at structurally connected part boundaries in OVPS. Guided by HPCGraph, hierarchical segmentation is performed, and the BAR module improves boundary precision via feature optimization and dynamic prototype mechanisms. This design is the key reason why our method PBAPS outperforms the state-of-the-art method RIM by 7.3%, 18.1%, and 8.4% on Pascal-Part-116, ADE20K-Part-234, and PartImageNet, respectively (as shown in Table 2).
• Although the prototype generation mechanism is not the contribution of this paper, we agree that this mechanism may affect model performance. Following your suggestion, we designed experiments to analyze the relationship between "prototype image-test set similarity" and model performance (mIoU). First, we synthesized prototype images with "low similarity" (<0.3) to real test set images by modifying the style prompts of Stable Diffusion (e.g., "cartoon"). Then, we slightly augmented and perturbed the training set images of the target datasets to generate prototype images with "high similarity" (>0.8). We evaluated the performance of PBAPS using prototype features generated from these two types of prototype images. The results are shown in the table below. Segmentation performance is positively correlated with prototype similarity, which is reasonable and expected. This is because prototypes with high similarity can more easily provide discriminative visual features, enabling more accurate segmentation. However, using prototypes with high similarity to the target dataset may weaken the model's cross-domain generalizability and make it less robust to style or distribution shifts. | Prototype-Test Similarity | Pascal-Part-116 | ADE20K-Part-234 | PartImageNet| |:-|:-:|:-:|:-:| | Low (<0.3)|43.20|21.87|38.62| | Our method (0.55~0.65) |46.35|24.70|42.61| | High (>0.8)| 50.83|29.04|47.06|
• To address the discrepancies between the prototype set and the test set in terms of style, distribution, or visual details, relying solely on static visual prototypes may lead to inaccurate matching or misaligned responses, ultimately affecting the precision of part localization. To mitigate this issue, the BAR module introduces a Dynamic Prototype Adaptation mechanism, which effectively enhances the adaptability of prototypes to test images and improves segmentation performance. As described in Lines 182–189, this mechanism fuses the global prototype with high-confidence region features extracted from the current image to generate an adaptive prototype that better aligns with the actual content of the image. Compared to using only pre-generated static prototypes, this fusion strategy enables the model to capture instance-specific variations and contextual cues, making the prototype more consistent with the spatial distribution and structural characteristics of the target part. As shown in the ablation study (Table 3), removing the dynamic prototype adaptation mechanism leads to a significant drop in mIoU and bIoU across all three datasets, further confirming its effectiveness in improving segmentation accuracy and mitigating the impact of cross-image discrepancies.

In summary, while prototype similarity does influence segmentation performance, our method is carefully designed to maintain a balance between effectiveness and generalizability. The use of synthesized prototypes with moderate similarity, combined with the BAR module's dynamic prototype adaptation mechanism, ensures robust performance without overfitting to specific datasets. These design choices allow PBAPS to generalize well across diverse domains while preserving high segmentation accuracy.

We once again sincerely thank the reviewer for the insightful and constructive feedback. If there are any further concerns or points requiring clarification, we would be glad to provide additional details.

[38] Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation (CVPR 2024)

[46] OVDiff: Diffusion Models for Open-Vocabulary Segmentation (ECCV 2024)

Dear Reviewers,

We sincerely thank you for your time and effort in reviewing our paper and offering valuable suggestions. As the author-reviewer discussion deadline is approaching, we would like to kindly confirm whether our previous responses have effectively addressed your concerns. We submitted detailed replies to your comments a few days ago and hope they have clarified the issues you raised. If any questions remain or if further clarification is needed, please do not hesitate to let us know. We are more than happy to continue the discussion and provide any additional information you may require.

Best regards,

The authors of Paper ID 8092

We thank reviewer for the constructive comments. We provide our feedbacks as follows.

W1＆Q1: Contamination from blended-part features in the “deterministic” pixels of BAR.

A1: Regarding the question of whether BAR weakens the original discriminative features, we provide the following supplementary explanation:

• In terms of design, BAR does not directly perform average context fusion on ambiguous regions. The attention weights in Equation 4 only aggregate features from deterministic regions that are most relevant to the current ambiguous pixel, avoiding the introduction of irrelevant or ambiguous contexts. Subsequently, a weighted fusion strategy of "original features ($\gamma$=0.8) + context (1-$\gamma$=0.2)" is adopted as specified in Equation 5, ensuring that the discriminative features of the pixels remain dominant. This design not only leverages context to resolve ambiguity but also prevents the loss of discriminative features.
• In Appendix B.4, we evaluated the model performance under different $\gamma$ values. As shown in Figure 8(a), using only the original ambiguous region features (without context fusion, $\gamma$=1.0) results in 44.83% mIoU. Using only context features (without original features, $\gamma$=0), the mIoU drops to 43.31% (not included in the figure), indicating the loss of discriminative features and excessive boundary smoothing. With balanced fusion ($\gamma$=0.8), the mIoU increases to 46.35%, which retains discriminative features while eliminating ambiguity through context.
• Additionally, the ablation experiment in Table 3 effectively verifies the contribution of the BAR module. Figure 4 provides a visual comparison of the segmentation at the "cat neck-torso boundary" before and after refinement. Due to NeurIPS rebuttal constraints, we are unable to include images, PDFs, or external links at this stage. We will supplement additional qualitative visualization results in subsequent revisions of the paper to fully demonstrate the effectiveness of the BAR module.

W2＆Q2: Extensibility of the HPCGraph.

A2: Regarding the Extensibility of HPCGraph, we provide the following supplementary explanations:

• HPCGraph is constructed based on commonsense structural knowledge and does not rely on any manual annotations. As such, it incurs low construction cost and exhibits strong transferability and scalability. In large-scale object scenarios, both the topological structure and visual prototypes can be shared and reused. For novel object categories with similar structures to existing ones (e.g., “zebra” as a novel category relative to the existing category “horse”), we can directly reuse existing topologies (e.g., “zebra head ↔ zebra neck” correspond to the structure of "horse"). For PartImageNet (L198–200), we divided 158 object categories into 11 superclasses, where each superclass shares a unified topological graph. This strategy reduces the number of topology by 93% (from 158 to 11) while maintaining high performance—PBAPS outperforms RIM by 8.4% on PartImageNet (Table 2). When the number of object categories expands to tens of thousands, a large number of them are highly similar in form and structure (such as horses, donkeys, mules, zebras, etc.). We can use LLMs to analyze inter-category structural similarities and achieve cluster-level topology reuse (i.e., categories with similar structures share a set of topologies). Furthermore, part visual prototypes can also be reused (e.g., the prototype for “horse leg” can be applied to “donkey leg”).
• In contrast to existing methods such as PartCLIPSeg [27] and PartCATSeg [53], which require extensive part-level annotations and retraining when introducing new object categories, our method is training-free and supervision-independent. It supports open-world scalability by simply extending the HPCGraph and adding a small number of visual prototypes.
• As noted in Appendix A.1, when dealing with objects with irregular structures (e.g., modular furniture), the static graph structure has insufficient adaptability. To address this, a dynamic graph generation mechanism can be considered. The core logic is to automatically add adjacency edges for highly relevant parts based on the spatial adjacency and visual feature similarity between part response regions. The following table compares performance using static and dynamic graphs. While static graphs yield better performance in well-structured domains, the dynamic version provides competitive results and significantly enhances flexibility for deployment in unconstrained real-world scenarios. | HPCGraph | Pascal-Part-116 | ADE20K-Part-234 | PartImageNet| |:-|:-:|:-:|:-:| | | mIoU  bIoU | mIoU  bIoU| mIoU  bIoU | | Dynamic | 44.63  33.12 |23.87  15.63| 41.66  28.72 | | Static | 46.35  34.46 |24.70  16.41 | 42.61  29.31 |

W3＆Q3: Additional supervised open-vocabulary part-detection baseline.

A3: Regarding the comparison with supervised open-vocabulary segmentation methods, as shown in Table 2, our method (PBAPS) has already compared with ZSSeg (ECCV 2022) [15] and CAT-Seg (CVPR 2024) [19], and outperformed these methods on all three datasets, effectively verifying the superiority of our training-free approach. Following your suggestion, we have further compared with two latest supervised open-vocabulary segmentation methods, SED (CVPR 2024) [60] and DPSeg (CVPR 2025) [61]. Both methods introduce a multi-scale feature fusion strategy based on CAT-Seg, achieving significant improvements in the segmentation of small objects and boundaries. We adjusted their decoders to support part-level prediction and fine-tuned them on the target datasets. As shown in the table below, PBAPS still maintains superior performance, further validating its advantages in OVPS.

W4＆Q4: Ablation decoupling hierarchical reasoning from boundary refinement.

A4: As shown in the table below, we directly applied BAR to optimize all structurally connected part boundaries based on the baseline (the 1st row, which excludes all components and directly matches pixel-wise features with visual prototypes). Furthermore, to further demonstrate the robustness and generalizability of our method, we supplement the same ablation experiment on ADE20K-Part-234 and PartImageNet. The results indicate that without introducing hierarchical structural constraints, BAR can still effectively model local contextual consistency and improve the segmentation accuracy of boundaries between structurally connected parts. Moreover, combining BAR with hierarchical reasoning yields further improvements, demonstrating their complementarity.

We once again sincerely thank the reviewer for the insightful and constructive feedback. If there are any further concerns or points requiring clarification, we would be glad to provide additional details.

[15] ZSSeg: A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model (ECCV 2022)

[19] CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024)

[60] SED: ASimple Encoder-Decoder for Open-Vocabulary Semantic Segmentation (CVPR2024)

[61] DPSeg: Dual-Prompt Cost Volume Learning for Open-Vocabulary Semantic Segmentation (CVPR2025)

Dear Reviewers,

We sincerely thank you for your time and effort in reviewing our paper and offering valuable suggestions. As the author-reviewer discussion deadline is approaching, we would like to kindly confirm whether our previous responses have effectively addressed your concerns. We submitted detailed replies to your comments a few days ago and hope they have clarified the issues you raised. If any questions remain or if further clarification is needed, please do not hesitate to let us know. We are more than happy to continue the discussion and provide any additional information you may require.

Best regards,

The authors of Paper ID 8092

We thank reviewer for the constructive comments. We provide our feedbacks as follows.

W1: The paper only showcases successful examples. A discussion of failure cases would provide a more balanced perspective on the method's limitations and offer crucial insights for future work.

A1: We fully agree with your comment. In the main paper, we primarily showcased representative successful cases to clearly demonstrate the advantages of our method. Although appendix A.1 includes a discussion on the limitations of our approach—such as difficulties in distinguishing multiple instances and handling complex or irregular object structures—it lacks visual examples that intuitively illustrate failure cases. Due to NeurIPS rebuttal constraints, we are unable to include images, PDFs, or external links at this stage. We briefly describe the failure cases here: when there are multiple animals of the same category in an image (e.g., three cats) with intertwined limbs, PBAPS fails to distinguish "identical parts belonging to different individuals" (e.g., the tail of a black cat vs. the tail of a white cat), resulting in cross-instance confusion. For modular furniture or mechanical parts, whose part hierarchies are ambiguous or topologies are variable, the static HPCGraph struggles to cover such scenarios, leading to part missed detections or over-merging (e.g., misclassifying folded table legs as part of the tabletop). We will include visualizations of typical failure cases in the revised version of the paper and provide an in-depth analysis of the underlying causes.

W2: The paper lacks a clear justification for how key hyperparameters were chosen without access to the test set. Furthermore, their robustness and generalizability across different datasets are not discussed.

A2: Thank you for your valuable comment. Regarding hyperparameter selection, we strictly adhered to the "no access to test set" principle. Specifically, we evaluated model performance (mIoU) under different hyperparameter configurations on the training sets of the three target datasets. As shown in the table below, we selected a single set of hyperparameters ($\gamma$ = 0.8, $\alpha$ = 0.7, $K$ = 4) that consistently performed well across all three datasets. This selection was made without any feedback from the test sets, thus avoiding dataset-specific tuning. As for robustness, Table 2 demonstrates that PBAPS (ours), using the same hyperparameter configuration, achieves state-of-the-art performance on Pascal-Part-116, ADE20K-Part-234, and PartImageNet—datasets with significant differences in visual style and structural granularity—validating the robustness and generalizability of our hyperparameter choices across diverse domains.

W3: The method's reliance on a manually constructed Hierarchical Part Connected Graph (HPCGraph) is a major bottleneck, raising significant concerns about its scalability and applicability to novel object categories not defined in advance.

A3: Regarding the scalability of HPCGraph, we provide the following supplementary explanations:

• HPCGraph is constructed based on commonsense structural knowledge (e.g., compositional relationships, anatomical structures) and does not rely on any manual annotations. As such, it incurs low construction cost and exhibits strong transferability and scalability.
• In large-scale object scenarios, both the topological structure and visual prototypes can be shared and reused. For novel object categories with similar structures to existing ones (e.g., “zebra” as a novel category relative to the existing category “horse”), we can directly reuse existing topologies (e.g., “zebra head → zebra eye”, “zebra head ↔ zebra neck” correspond to the structure of "horse"). For PartImageNet (L198–200), we divided 158 object categories into 11 superclasses, where each superclass shares a unified topological graph. This strategy reduces the number of topology by 93% (from 158 to 11) while maintaining high performance—PBAPS outperforms RIM by 8.4% on PartImageNet (Table 2). When the number of object categories expands to tens of thousands, a large number of them are highly similar in form and structure (such as horses, donkeys, mules, zebras, etc.). We can use Large Language Models (LLMs) to analyze inter-category structural similarities and achieve cluster-level topology reuse (i.e., categories with similar structures share a set of topologies). Furthermore, part visual prototypes can also be reused (e.g., the prototype for “horse leg” can be applied to “donkey leg”).
• In contrast to existing methods such as PartCLIPSeg [27] and PartCATSeg [53], which require extensive part-level annotations and retraining when introducing new object categories, our method is training-free and supervision-independent. It supports open-world scalability by simply extending the HPCGraph and adding a small number of visual prototypes, making it more suitable for large-scale and dynamic settings.
• As noted in Appendix A.1, when dealing with objects with irregular structures (e.g., modular furniture), the static graph structure has insufficient adaptability. To address this, a dynamic graph generation mechanism can be considered. The core logic is to automatically add adjacency edges for highly relevant parts based on the spatial adjacency and visual feature similarity between part response regions. The following table compares performance using static and dynamic graphs. While static graphs yield better performance in well-structured domains, the dynamic version provides competitive results and significantly enhances flexibility for deployment in unconstrained real-world scenarios. | HPCGraph | Pascal-Part-116 | ADE20K-Part-234 | PartImageNet| |:-|:-:|:-:|:-:| | | mIoU  bIoU | mIoU  bIoU| mIoU  bIoU | | Dynamic | 44.63  33.12 |23.87  15.63| 41.66  28.72 | | Static | 46.35  34.46 |24.70  16.41 | 42.61  29.31 |

We once again sincerely thank the reviewer for the insightful and constructive feedback. If there are any further concerns or points requiring clarification, we would be glad to provide additional details.

[27] PartCLIPSeg: Understanding Multi-Granularityfor Open-Vocabulary Part Segmentation (NeurIPS 2024)

[53] PartCATSeg: Fine-Grained Image-Text Correspondence withCost Aggregation for Open-Vocabulary Part Segmentation (CVPR 2025)

Dear Reviewers,

We sincerely thank you for your time and effort in reviewing our paper and offering valuable suggestions. As the author-reviewer discussion deadline is approaching, we would like to kindly confirm whether our previous responses have effectively addressed your concerns. We submitted detailed replies to your comments a few days ago and hope they have clarified the issues you raised. If any questions remain or if further clarification is needed, please do not hesitate to let us know. We are more than happy to continue the discussion and provide any additional information you may require.

Best regards,

The authors of Paper ID 8092