LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentation

We appreciate the reviewer’s positive feedback on our work, especially the highlighting of novelty by integrating MLLMs deeply into the OVPIS task and the handling of varying part granularity based on this. Furthermore, our core contribution grounding object-part hierarchies in the language space and performance over prior baselines is well acknowledged.

We address the reviewer's concerns in the following sections.

1. Granularity Variation

Part granularity is guided by text and can be flexibly specified by the user or by the dataset annotations used for training and evaluation in our approach.

• For example, during evaluation, if the ground truth annotation specifies “face” as the part of interest for a “human” object, our model constructs the corresponding part query by concatenating the text embedding of “face” with that of “human”, resulting in a semantically aligned object-part query (e.g., “human-face”).
• This design allows the model to be explicitly guided to the appropriate level of granularity, depending on the dataset’s supervision or user input. Moreover, our cross-dataset evaluation (Tables 1 and 2 main paper) is specifically intended to test this capability: different datasets may annotate parts at different levels of detail, and our model’s language-grounded hierarchy allows it to adapt accordingly.
• This granularity awareness is a key reason why our method consistently outperforms baseline models in the cross-dataset setting.

We will include more visualization of the model’s predictions on the same image but under different specified part granularities to better illustrate the model’s flexibility in handling varying granularity levels.

2. Notation

We appreciate the reviewer’s feedback and agree that the mathematical notations should be clear and consistent. We have carefully revised all equations and these corrections will be reflected in the updated version.

Dear Reviewer,

Thank you for your dedicated efforts in reviewing this paper. We are currently in the reviewer-author discussion phase, but we have not yet seen your engagement.

This year's Responsible Reviewing initiative requires all reviewers to communicate with authors during this period, emphasizing that ghosting is not acceptable. We kindly ask that you reply and engage with the authors. Please note that participation in discussions with authors is mandatory before submitting "Mandatory Acknowledgement," as submitting it without any engagement is not permitted in this review cycle.

Best, AC

Dear Reviewer,

We would like to express again our sincere appreciation for your positive review of our work. We hope that we have addressed your concerns and remain committed to further improving the paper as promised. Please feel free to reach out with any additional questions or suggestions.

We appreciate the reviewer highlighting the originality of our work by introducing language-grounded object-part hierarchies and MLLMs into the object-part instance segmentation task, as well as, in line with other reviewers, our experiments verifying our approach and showing the superior performance.

1. Novelty Clarification

Our work is not an extension of PartGlee with the integration of an MLLM, but rather proposes novel intermediate representations to enable language-grounded reasoning and the corresponding training strategy. In detail, our additional contributions are as follows:

• Our method introduces language-grounded modeling throughout the whole architecture, closely coupling object and part decoding as well as query generation in the same space. Furthermore, the language-grounded object-part hierarchy modeling replaces the brute-force hierarchical representation using a fixed set of parts in the Q-Former of PartGLEE's architecture. Overall, this design enables adaptive part hierarchies in our framework and OVS adaptation to varying part granularity beyond the training data.
• We explore the recipes of both one-stage and two-stage training strategies. We provide an empirical study (Table 7 in the submission) showing that the one-stage training strategy leads to better in-domain performance while the two-stage one leads to better cross-dataset performance. This empirical finding suggests that decoupling coarse-to-fine supervision helps improve generalization in tasks involving hierarchical modeling, offering practical insights for similar structured perception problems.
• Architectural Verification: We conducted experiments to verify and derive our architecture against a direct extension of PartGLEE, PSALM, and PartCatSeg. Most importantly, our experiments (Table 4 main paper) show that our design improves performance far beyond only utilizing the additional capacity and pretraining of the MLLM, but rather builds a framework around it to effectively utilize its generalist capabilities. We add these verification experiments to the final version of our paper.

Together, these contributions show that our method is not merely an MLLM extension of PartGLEE, but introduces important innovations in representation design and training methodology that contribute to its effectiveness.

2. Parameters and Inference Cost

We now report the footprint of GPU hours, carbon cost, inference cost, and model size of PartGLEE, PSAL,M and LangHOPS in Table R-9. The GPU hours and inference time are reported for a NVIDIA H200 GPU. The spec. power (700W) of H200 and world average carbon intensity of electricity (~0.475 kg CO₂e / kWh) are used for calculating the footprint.

Table R-9: Specs of LangHOPS and the baselines. PPS116 + INS + PART -> PartImageNet.

The Table shows that LangHOPS has the largest model size, mainly due to the usage of MLLM (Paligemma2-3B). PSALM† has the longest training time and carbon footprint since it trains the LLM instead of using LoRA, and needs to process all candidate category names, which leads to long input prompts to the LLM. LangHOPS achieves the best performance with reasonable training and inference cost compared to the baselines.

3. Terminology Inconsistency

We appreciate the reviewer's correction. The second "Object Segmentation" in L133 is indeed redundant, and that paragraph is part of object-part parsing. We will address them in the revised version of the paper. "Language-Space Object-Part Representation" and "MLLM-based Object-Part Parsing" are actually both part of "Object-Part Parsing" in Figure 2. We understand that it can cause confusion, and we will clarify this.

4. Error Bars

We have now included mean ± standard deviation results computed over three random seeds (20, 43, and 4337) for both LangHOPS and PartGlee, the strongest baseline.

These results are reported in Tables R-5 and R-6. As shown, the averaged AP values closely align with the originally reported numbers, slightly increasing in most experiments for our method. The standard deviations have a low magnitude, indicating that our results are stable and show a significant improvement over the PartGlee baseline.

Table R-5: PPS116→PartImageNet with standard deviation (±) of the proposed method LangHOPS± and the strongest baseline PartGlee±.

Table R-6: PartImageNet→PPS116 with standard deviation..

5. Code Release

We will make the code public upon the paper’s release. Due to the IP limitations of the work, it was not possible to release the code during review.

6. Typos

We thank the reviewer for reporting the typos. We will address them and go through the paper carefully, correcting them.

7. Instance-level Annotation

The reviewer is correct that the original Pascal-Part-116 [42] dataset only provides semantic part segmentation annotations. However, as detailed in Appendix B of PartGLEE [20] and its official git repository (assets/DATA.md), PartGLEE further processes these annotations to produce instance-level part segmentation labels, which support the OVPIS task. Our experiments in Table 1 and Table 2 use this PartGLEE-provided version of the PPS-116 dataset, which includes part-instance-level annotations. We will clarify this in the revised version of the paper to avoid confusion.

8. Visualization Clarification

The visualizations in Figure 1 are part instance segmentation outputs. For readability, we display each part belonging to the same type in the same color, allowing the reader to reidentify objects and show only a single label for each class of parts to avoid clutter. We thank the reviewer for pointing out this potentially distracting visualization and will provide a more common visualization showing the separation of instances in the final paper.

9. MLLM Output

The MLLM outputs are embeddings of MLLM-processed prompt tokens, and we extract part queries from those embeddings. Specifically, we feed a structured language prompt—containing object-part hierarchical information—into the MLLM. Unlike typical generative usage, the MLLM here returns token-level embeddings for the entire input sequence, with the output length matching the input. We then extract the embeddings at the positions corresponding to the initial part query tokens. These embeddings are used as the refined, language-informed part queries, which are passed to the visual part decoder.

Dear Reviewer,

Thank you for your dedicated efforts in reviewing this paper. We are currently in the reviewer-author discussion phase, but we have not yet seen your engagement.

This year's Responsible Reviewing initiative requires all reviewers to communicate with authors during this period, emphasizing that ghosting is not acceptable. We kindly ask that you reply and engage with the authors. Please note that participation in discussions with authors is mandatory before submitting "Mandatory Acknowledgement," as submitting it without any engagement is not permitted in this review cycle.

Best, AC

We appreciate the reviewer's further constructive comments and suggestions. We agree that clarifying the raised questions is important for improving the quality and clarity of our work. Below, we provide our detailed responses.

1. Novelty Clarification

Architecture Novelty Clarification

In our model, the embeddings of bus1’s wheel1 and bus2’s wheel1 are distinct in the language space E, while the embeddings of bus1’s wheel1 and bus1’s wheel2 are identical. Specifically, the part embedding is constructed by combining the object query $O_{bus1}$ and the CLIP text embedding of the part name “wheel”. Importantly, $O_{bus1}$ is not just a category-level embedding; it encodes instance-specific visual features, as it is used to generate the segmentation mask for that specific bus1 instance. Thus,

• $O_{bus1} \neq O_{bus2}$
• Consequently, E(bus1’s wheel1) $\neq$ E(bus2’s wheel1)

This differs fundamentally from the alternative approach of simply concatenating the CLIP embeddings of the object and part names (e.g., CLIP(“bus”) + CLIP(“wheel”)), which would not capture instance-level visual features and variation and would lead to ambiguity, especially in scenes with multiple instances of the same object type.

Following the reviewer’s helpful suggestion, we have clarified this point explicitly in Section 3.4 of the revised paper.

Training Strategy

We agree that the training strategy analysis provides useful insight. In response to the reviewer’s comment, we have moved the relevant analysis to Section 4.3 in the main paper from the supplementary and explicitly discussed its impact on performance. While it may not be the central novelty of our method, we believe it adds valuable context and guidance for future work, and we are grateful for the reviewer’s encouragement to better highlight it.

3.Clarity and Consistency of Presentation

We appreciate the reviewer's suggestion for improving the presentation of the paper. In the revised paper, we have

• corrected the duplicate “Object Segmentation” subsection title,
• ensured consistency and clarity of the method components by
  • (a) replacing "object-level segmentation" with "object segmentation" in sections 3 and 4 for consistency;
  • (b) unifying the name of section 3.4 in figure 2 to "Language-Space Object-Part Representation";
  • (c) clarifying the hierarchy between "Object-Part Parsing" and Section 3.4 and 3.5 in the method overview
  • (d) added tags of the subsection numbers to the corresponding modules in Figure 2.
• ensured consistency of the mathematical notations and the terminology

6. Additional Typos and Inconsistencies Noted

We have addressed all the typos and inconsistencies noted by the reviewers. In addition, we have carefully proofread the paper to ensure its clarity and coherence.

---

We sincerely appreciate the reviewer’s meticulous suggestions and thoughtful feedback, which have significantly improved the quality and presentation of the paper. In the revised version, we have made significant efforts to address these issues with

• clarified model design and analysis on training strategies in main paper
• refined structure of the method section and clear figure–text alignment;
• standardized notation and consistent terminology;
• corrected typos and coherent phrases

We hope these changes effectively address the reviewer’s concerns and make the contributions of the paper more accessible and impactful. We deeply appreciate the reviewer’s acknowledgment of the strengths and novelty of our approach, and we respectfully ask for reconsideration of the overall rating in light of the improvements.

We thank the reviewer for the highly constructive comments and appreciate the recognition of significance and originality, together with highlighting solid empirical evidence in different settings and clear implementation details. We answer to the reviewer's comments as follows.

1. Error Bars

We report mean ± standard deviation results over three random seeds (20, 43, and 4337) for both LangHOPS and PartGlee, the strongest baseline.

As shown in Tables R-5 and R-6, the averaged AP closely align with the originally reported numbers, slightly increasing in most experiments for our method. The standard deviations have a low magnitude, indicating that our results are stable and show a significant improvement over the PartGlee baseline.

Table R-5: PPS116→PartImageNet with standard deviation (±) of the proposed method LangHOPS± and the strongest baseline PartGlee±.

Table R-6: PartImageNet→PPS116 with standard deviation.

2. Prompt Sensitivity

We conducted two ablation studies on the ordering and wording of the structured input prompts to assess the robustness of our method to prompt formulation.

Robustness to prompt ordering: We randomly shuffled (i) the order of object queries, and (ii) the order of part queries within each object, multiple times during inference. For instance, object 3 may appear before object 1, or part queries within an object may be permuted (e.g., "part 9, part 4, part 6"). As shown in Table R-7, our method remains highly stable across these permutations, with minimal performance degradation, demonstrating robustness to input ordering.

Table R-7: Ablations on the ordering of the object and part queries - PPS116->PartImageNet.

Robustness to wording: We further test the model's robustness to unseen part names by replacing the subset (from $0\%$ to 100%) of the original part category names with GPT-4o-generated synonyms (e.g., "foot" → "leg"). As shown in Table R-8, LangHOPS significantly outperforms PartGlee under increasing synonym replacement ratios, indicating strong generalization to semantically similar but unseen phrasing. Note that synonym substitutions may introduce granularity mismatches with the dataset’s ground-truth annotations (e.g., "leg" may exclude "paw" in the ground truth for “foot”), which partially explains the observed performance drop.

Table R-8: Ablation on the robustness to input part category names. PPS116 + INS + PART -> PartImageNet. Different percentages of part category names replaced with GPT-4o generated synonyms.

Together, these studies demonstrate that LangHOPS is robust to both the structure and vocabulary of the input prompt.

3. Compute Footprint

We now report the footprint of GPU hours, carbon cost, inference cost and model size of PartGLEE, PSALM and LangHOPS in Table R-9. The gpu hours and inference time are reported with Nvidia H200 GPU(s). The spec. power (700W) of H200 and world average carbon intensity of electricity (0.475 kg CO₂e / kWh) are used for calculating the footprint.

Table R-9: Specs of LangHOPS and the baselines. PPS116 + INS + PART -> PartImageNet.

The Table shows that LangHOPS has the largest model size, mainly due to the usage of MLLM (Paligemma2-3B). PSALM† has the longest training time and carbon footprint since it trains the LLM instead of using LoRA, and needs to process all candidate category names, which leads to long input prompts to the LLM. LangHOPS achieves the best performance with reasonable training and inference cost compared to the baselines.

4. Robustness to Noisy Hierarchy

We test on the common OVS setting using clean object-part hierarchies, but believe in the value of closing the gap towards a noisy real-world deployment. To evaluate the robustness of LangHOPS to noisy or automatically mined hierarchies, we replace a portion of the clean object-part taxonomy with GPT-4o-generated object-part hierarchies. These auto-mined hierarchies are constructed solely from the object category names and may introduce ambiguity, inconsistency, or irrelevant parts. In Table R-10, we report performance under varying noise levels: for example, at $25\%$, $25\%$ of the object categories use noisy hierarchies, while the remaining $75\%$ use the clean dataset annotations. We observe that:

1. LangHOPS consistently outperforms PartGlee across all noise levels;
2. LangHOPS degrades more gracefully as noise increases, maintaining reasonable AP even when $100\%$ of hierarchies are noisy;
3. The performance gap widens especially at high noise levels, demonstrating LangHOPS's stronger resilience to imperfect or automatically mined hierarchies.

Please note that the auto-generated hierarchies are often inconsistent with the ground truth annotations in the dataset, leading to lower evaluation metrics. Overall, we agree with the reviewer that developing evaluation protocols for adaptive, task-specific hierarchies remains an open problem and a promising direction for future benchmark design.

Table R-10: Ablations on the noisy hierarchy construction. Different percentages of obj-part hierarchies from the dataset are replaced with GPT-4o generated ones.

5. Synergy Mechanism - Attention Score

We further prove the object-part synergy mechanism by reporting the average attention score, since the attention maps cannot be provided in the rebuttal format. The average attention score is calculated by summing attention scores of true positive predictions inside the ground truth masks, divided by the area of the masks. The attention is the normalized cos similarity between object queries and the dense features of the final layer of the object/part decoder. The score shows the amount of attention correctly assigned by the model to the ground truth area, and is mapped to the range [0, 1].

In the setting of PPS116+INS+PART -> PartImageNet, as shown in Table R-11, compared to the "detached object-part seg.", the synergized object-part segmentation leads to higher attention scores for both object and part segmentation, proving strong evidence of the synergy between both segmentation tasks.

Table R-11: Attention score of the object and part segmentation.

We will further show the attention maps and the attention scores in more experimental settings in the revision.

6. Comparison to PartCatSeg

We agree that the improvement over PartCatSeg in OVPS is moderate in a specific dataset (ADE20K). However, it is noticeable that our method is developed and trained with object-part-level instance segmentation. Directly evaluating our method in the semantic segmentation (Table 3 in the main paper) still leads to superior performance (hIOU) in PPS-116 and PartImageNet datasets, showing the great potential of the proposed method.

We appreciate the reviewer's comments and recognition of performance gains achieved by our method on different benchmarks, especially on open vocabulary cross-dataset tasks, which are a central target of our method.

1. Motivation Clarification

We do not perform object-specific part segmentation, but rather segment parts of all requested objects simultaneously, following the broadly accepted open-vocabulary part segmentation (OVPS) framework.

For the relationship between part- and instance-segmentation, in fact, prior work such as PartGLEE follows the approach mentioned by the reviewer — simply combining object and part segmentation modules via a Q-Former.

In contrast, our method models the object-part hierarchy in language space, leading to significant performance gains over PartGLEE. It is significant to address the challenge:

While object-level instance segmentation is well-defined, part-instance segmentation is task-dependent and inherently requires open-vocabulary capabilities.

For example:

• Using a laptop may require segmenting the lid to open it.
• Repairing it may require finer segmentation of screws or hinges.

We solve it by:

• Incorporating open-vocabulary prompts into object segmentation;
• Modeling hierarchical object-part relationships with structured prompts;
• Enforcing a one-to-one mapping between object parts and part segmentation queries;
• Using language-grounded queries in the part decoder that leverage the reasoning capabilities of an MLLM.

We will clarify this in the introduction.

2. Connection Between Modules

We believe this is a misunderstanding.

Our object and part segmentation modules are tightly coupled via the hierarchical object-part representation (mentioned in Line 140, 174, and Figure 2) and aim at utilizing the joint information between parts and objects through the following information flow:

• For each segmented object, a set of part queries is constructed by a concatenation of the object embedding $\mathbf{O^H}$ generated in the object decoder and the language-encoded part description.
• These queries are processed by the MLLM, generating queries that are capable of open-vocabulary part segmentation.
• The processed queries are used in the part decoder to segment each requested part.

Our ablation study in l269ff and Table 5 shows a comparison with detached object-part segmentation that demonstrates that the joint training with information flow from the part decoder to the object decoder is efficient and improves performance.

We will add this paragraph to the architecture overview.

3. Language Grounding

We respectfully disagree and refer to our method as being language-grounded since we design the whole object-part representation and processing to remain in a language-aligned space. Our motivation to follow this approach is to utilize the generalist language representations learned by the MLLM.

In detail the language-grounding is enabled in the following way:

• Both object $\mathbf{O^H}$ and part queries $\mathbf{P}$ generated in the segmentation modules are aligned to the language space and compared against text embeddings of category labels for classification.
• More critically, language processing in our method goes beyond the frozen text encoder. We combine the hierarchical part structure in language space with semantic object embeddings produced by the object segmentation, explicitly modeling object-part hierarchies in language space.
• We leverage these text-aligned embeddings in a pretrained MLLM, which enriches the queries with contextual and semantic knowledge from the vision-language domain. Similarly, the output of the MLLM resides in a text-informed feature space, allowing the part decoder to extract visual features in a way that is conditioned on open-vocabulary semantics.

This design enables the open-set generalization capability and allows the model to process novel categories and part-object relationships.

4. Limitation of Previous Methods

We clarify the limitations of the existing methods. It is recognized that CLIP’s embeddings have limited capacity for compositional understanding and modeling object-part hierarchies[1, 38, 48]. As a result, CLIP-based methods such as OV-Part [42] and PartCLIPSeg[7] tend to show suboptimal performance in fine-grained part segmentation. To address this, PartCatSeg augments the model with DINOv2 for structural guidance . However, the approach still lacks an explicit mechanism for modeling object-part hierarchies grounded in language, which is critical for compositional generalization.

For another existing work, PartGLEE [20], it has inherent limitation by design in handling part granularity variation across datasets. It parses objects into parts through a learning-from-scratch Q-Former, which is incapable of handling object-part granularity unseen during training. For example, trained with fine-grained annotations like "eye", "nose", "ear" for the cat on Pascal-Part-116, PartGLEE performs poorly when evaluated on PartImageNet, where the cat is annotated with the only coarser parts ("head", "body", "foot", and "tail").

We will revise the part mentioned by the reviewer to make it clear.

5. Training of Visual Backbones and Pixel Decoder

It is indeed necessary to train the visual backbone and pixel decoder. The visual backbone is from Mask2Former CVPR'22, pretrained only in the object-level segmentation task and thus, is limited in segmenting fine-grained, part-level instances. Therefore, we train the backbone on part-level datasets to grant it part-aware segmentation ability. This practice is actually not uncommon in the literature on part-level segmentation (e.g., [20, 40]).

We further conducted an ablation study on the freezing/training of the visual backbone and the pixel decoder. As shown in Table R-1 and R-2, training the visual backbone and the pixel decoders leads to better performance in part-level segmentation and higher overall AP.

Table R-1: Ablations on freezing backbone (Bk) and pixel decoder (Pd). Cross-dataset and in-domain evaluation, PPS-116 -> PartImageNet (with Random seed 4337).

Table R-2: Ablations on freezing backbone and pixel decoder. PartImageNet -> PPS-116.

6. Differentiation from PartGLEE

Language-Space Object-Part Representation, a core innovation of our method, distinguishes ours from PartGLEE. Unlike PartGLEE, which uses a fixed set of learnable part queries processed by a Q-Former, our method dynamically constructs part queries in language space, conditioned on both the object context and a user-defined list of candidate part categories.

For each object query from the object segmentor, we iterate over candidate part labels and concatenate each text embedding with the object query to form context-aware initial part queries. These are then further refined by a multimodal LLM in Section 3.5. To address the reviewer's concern, we conduct an ablation study by replacing the language-grounded part query initialization with fixed learnable tokens as in PartGLEE and use them as the input to the MLLM, which drops the cross-dataset performance significantly compared to the proposed method:

Table R-3: Ablations on Learnable Query. Cross-dataset and in-domain evaluation, PPS-116 -> PartImageNet (with Random seed 4337). LangHOPS w. LQ means that the "Language-Space Object-Part Representation" is replaced by the PartGLEE-like learnable queries concatenated with object queries.

Table R-4: Ablations on Learnable Query. PartImageNet -> PPS-116.

Dear Reviewer,

Thank you for your dedicated efforts in reviewing this paper. We are currently in the reviewer-author discussion phase, but we have not yet seen your engagement.

This year's Responsible Reviewing initiative requires all reviewers to communicate with authors during this period, emphasizing that ghosting is not acceptable. We kindly ask that you reply and engage with the authors. Please note that participation in discussions with authors is mandatory before submitting "Mandatory Acknowledgement," as submitting it without any engagement is not permitted in this review cycle.

Best, AC