Object-Centric Concept-Bottlenecks

We thank the reviewer for their time and suggestions. We provide clarifications for these points below.

Clarification object-level vs image-level concepts in CBMs

Traditionally, CBMs compute a single concept representation for the entire image, often relying on supervised concept labels [1] or CLIP-based methods that match global image features to textual descriptions [e.g., [2]]. The image-level concepts reflect properties of the overall scene but do not explicitly distinguish between individual entities within.

In contrast, object-level concepts, as introduced in OCB, are computed separately for each detected object in the image. While both are based on the same backbone encoder, object-level concepts are grounded in specific, localized entities, enabling a structured representation of the scene. This difference is not merely about granularity; it provides explicit grouping of concepts by object, which is not available in global image-level approaches.

This structure has important implications:

• It enables reasoning over multiple entities in complex, multi-object scenes.
• It allows models to distinguish which concepts belong together, tied to specific objects.
• It facilitates compositional and interpretable reasoning, aligning more closely with how humans understand scenes.

Our results support this distinction. For example, our sparsity experiments show that object-centric representations encode concepts more efficiently and faithfully than image-level representations. Moreover, object-centric models perform better in multi-label settings, where distinct object-level reasoning is essential.

While object-level concepts could be forced into an image-level framework, this ignores the inductive bias that gives object-centric representations their strength. Our contribution goes beyond hierarchical grouping, it offers a practical, interpretable path from flat concept reasoning to structured, entity-aware representations, essential for scaling CBMs to real-world vision tasks.

We have updated the manuscript to make sure that these points are presented more clearly in the paper.

2. What is the novelty of this work?

We acknowledge that the individual components of our Object-Centric Bottleneck (OCB) framework, such as object proposal generation and concept classification, are not novel in isolation. However, the novelty of our work does not lie in proposing a new object detection mechanism, but in how we integrate object-centric representations into the Concept Bottleneck Model (CBM) framework, making it scalable and applicable to complex, real-world visual data.

While this design may appear simple, bridging object-centric representations with CBMs in a real-world setting is non-trivial and has not been demonstrated before. Prior work exploring object-centric inductive biases has largely been limited to synthetic and single-class environments. Our contribution lies in extending these insights to realistic, diverse, and noisy data, removing the need for manual concept annotations, improving cross-dataset generalization, and demonstrating that object-level concept reasoning is viable and beneficial in practical scenarios.

In summary, the strength and novelty of our approach lie in making CBMs more structured, scalable, and interpretable by rethinking their core representation from flat, image-level concepts to grounded, object-level ones without requiring custom supervision or architectures.

Small improvements on tasks that do not require object information.

The primary goal of OCB is to extend the applicability of CBMs to more complex visual tasks, such as multi-label classification or visual reasoning, where reasoning over distinct objects is essential. We had included results on SUN, which is commonly used in CBM evaluations, to demonstrate that adding object-centric representations does not harm performance on tasks where object structure is less relevant.

Evaluation against common benchmarks in non-object domains.

As discussed in the previous point, the goal of OCB is not to improve performance on non-object-centric tasks, but to enable concept-based models to operate effectively on object-centric, multi-entity domains, a setting where traditional CBMs struggle. For this reason, our evaluation primarily focuses on datasets with explicit object structure, such as Pascal VOC and COCO. That said, we agree that broader benchmarking is valuable and present additional experiments on CIFAR-100, to expand the evaluation beyond object-segmented scenes. We observe that OCB and CBM perform comparably, indicating that OCB does not incur any performance loss on non-object-centric benchmarks. We have added the results to the paper.

Questions:

Re: Qualitative concept examples:

1. Does CBM view the object-concepts as less important? A: Yes, that is a valid interpretation. The base (image-level) CBM lacks the spatial focus to attend to smaller or less prominent objects like the fork or wine glass. As a result, these concepts may not be activated or may be deemed less relevant, even if they are present and potentially informative. In contrast, OCB explicitly examines individual regions, allowing it to surface concepts tied to localized entities that might be overlooked at the global level.
2. Why same concept module/vocabulary? A: We used a shared, general-purpose concept vocabulary to keep the approach task-agnostic and free of additional supervision. This ensures broad applicability without requiring domain-specific concept sets. That said, we agree that using different vocabularies for image-level and object-level representations when task-specific information is available is a promising direction for future work.

Re: Examples for concepts for SUN

1. Example concepts for SUN: For SUN, many images from natural scenes do not have any recognized objects. However, there are also scenes like "computer room", "stable" or "dining room" which have detectable objects like computers, horses, furniture or food that help identifying the global scene category. This explains the slight performance increase over the base model for this dataset.

2. Relations, Actions, Hierarchies? A: We see object-level representations as a foundational step toward more structured reasoning, including relations, actions, and hierarchical object decomposition. Our goal in this work was to establish object-centric concept representations within CBMs. While we stop at the object level, we agree that further decomposition into sub-parts, relationships, or action-level concepts which often emerge from interactions between objects or between objects and the global scene is a natural and valuable next step. Our current design provides a flexible base for such extensions, and we view this as a promising direction for future work.

Re: What kind of control and evaluation for concepts?

A: OCB enables more fine-grained control and evaluation by explicitly structuring the concept space around individual objects. In particular, when using the concat aggregation, the model retains which concepts are tied to which object proposals. This allows for object-level explanations, e.g., identifying which specific object triggered a certain prediction, and makes it possible to evaluate concept activation per object, rather than just at the global image level. This structured representation also allows for targeted interventions, such as modifying or masking concepts for a specific object.

Additional baseline results

We additionally wish to highlight novel baseline and ablation evaluations resulting from other reviews and strengthen the findings of our work.

1. Opaque Upper-Bound Model

We introduce a non-interpretable MLP classifier on top of frozen CLIP embeddings as an upper bound. As expected, it outperforms interpretable CBMs but lacks transparency. OCB narrows the gap significantly, offering strong accuracy with interpretable object-centric reasoning.

2. Coarse-to-Fine CBM (C2F)

We compare OCB to Coarse-to-Fine CBM [4], using the same LAION concept vocabulary. Evaluated on SUN and COCOLogic (single-label tasks), OCB outperforms C2F, highlighting the benefits of object-centric over patch-based representations.

3. Sparsity Analysis

We assess both CBM and OCB under varying sparsity constraints (via $\ell_1$ penalty). OCB consistently outperforms CBM at all sparsity levels, confirming that its gains stem from object-centric structure, not capacity.

We trust that these clarifications and additional results strengthen the clarity and completeness of our work, and we respectfully encourage the reviewer to reconsider their evaluation in light of these updates.

[1] Koh et al. "Concept bottleneck models." ICML, 2020.

[2] Yang et al. "Language in a bottle: Language model guided concept bottlenecks for interpretable image classification." CVPR, 2023.

[3] Marconato et al. "Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts." NeurIPS, 2023.

[4] Panousis et al. "Coarse-to-Fine Concept Bottleneck Models." NeurIPS, 2024.

We thank the reviewer for their helpful suggestions and the appreciation of our results. We provide clarifications for the raised points below, starting with a general response regarding the novelty of our method followed by additional baselines.

Novelty

We acknowledge that the individual components of our Object-Centric Bottleneck (OCB) framework, such as object proposal generation and concept classification, are not novel in isolation. However, the novelty of our work does not lie in proposing a new object detection mechanism, but in how we integrate object-centric representations into the Concept Bottleneck Model (CBM) framework, making it scalable and applicable to complex, real-world visual data.

To the best of our knowledge, this is the first work that operationalizes object-centric concept representations within CBMs and applies them to real-world, multi-label classification tasks. Traditional CBMs rely on global, image-level features and often suffer from spurious or entangled concept associations [1], especially in cluttered scenes. In contrast, our object-centric bottleneck grounds concepts in specific, localized regions via pretrained object detectors, enabling finer-grained, interpretable, and more faithful concept-level reasoning.

While this design may appear simple, bridging object-centric representations with CBMs in a real-world setting is non-trivial and has not been demonstrated before. Prior work exploring object-centric inductive biases has largely been limited to synthetic and single-class environments. Our contribution lies in extending these insights to realistic, diverse, and noisy data, removing the need for manual concept annotations, improving cross-dataset generalization, and demonstrating that object-level concept reasoning is viable and beneficial in practical scenarios.

In summary, the strength and novelty of our approach lie in making CBMs more structured, scalable, and interpretable by rethinking their core representation—from flat, image-level concepts to grounded, object-level ones without requiring custom supervision or architectures.

Additional baselines.

We agree that adding additional baselines strengthens the contextualization of our results which we highlight below:

1. Opaque Upper-Bound Model

To quantify the performance potential of our concept encoder backbone (CLIP), we introduce a non-interpretable model that directly maps CLIP image embeddings to class predictions using a non-linear multi-layer perceptron (MLP) (2 layers). It serves as an upper bound on the performance that can be achieved using CLIP features alone. As expected, it outperforms interpretable CBM variants slightly, but comes at the cost of complete opacity, offering no meaningful explanations. OCB closes the performance gap between CBM and the opaque model, underscoring its value for transparent, high-performing concept reasoning.

Caption: Upper bound with frozen CLIP encodings and MLP classifier compared to CBM and OCB.

2. Coarse-to-Fine CBM (C2F)

We further evaluate against the recent Coarse-to-Fine CBM [2], which introduces hierarchical, patch-based concept processing. Unlike many CBM variants that differ primarily in vocabulary, C2F contributes a distinct architectural inductive bias. To ensure a fair comparison focused on architecture rather than vocabulary, we ran C2F with the same LAION-based concept vocabulary of our OCB setup and compare to OCB without sparsity constraints. Since C2F only supports single-label classification, we evaluate it on the SUN397 and COCOLogic datasets. We observe that our OCB model consistently outperforms C2F under these conditions, highlighting the benefit of object-centric representations over patch-based hierarchies.

Caption: Comparison to Coarse-to-Fine CBM (with LAION vocabulary)

All results and discussions have been integrated into the main paper, and we thank the reviewers for this valuable suggestion.

Sparsity vs. Performance curves

We next analyze the impact of the sparsity constraint on the base CBM and OCB (Pascal-VOC). We evaluate both models (Base CBM and OCB) with varying $\ell_1$-penalties, which controls the number of active concepts for image and concept-level encodigns. As reference, we also provide the result where both models are not constrained and generate activations for every concept.

The results below show that both models improve as sparsity is relaxed. However, OCB consistently outperforms the Base CBM across all sparsity levels, indicating that object-centric representations yield more informative and efficient concepts.

These trends confirm that the performance gain of OCB is not due to increased capacity, but rather to the stronger inductive bias of object-centric representations. We have added this experiment and discussion to the main paper.

COCOLogic relevance

While we agree that the COCOLogic dataset includes constructed labels, this is by design: COCOLogic is intended as a proxy task for structured visual reasoning within single-class image classification, i.e., to capture aspects such as multi-object composition, negation, and hierarchical categories while still using real-world visual data. For example, distinguishing a "personal transport scene" involving a person on a bicycle versus an image of a bicycle alone without a person is the kind of entity-based and compositional reasoning that object-centric models are meant to handle. We have added a discussion of this point in Section 4 of the revised manuscript to clarify its motivation and role in benchmarking progress toward real-world reasoning tasks, such as visual question answering and scene understanding.

Ablation: only object encodings

We further observe that removing the image-level encodings from OCB leads to a substantial performance drop (Pascal VOC), confirming that object-centric and global representations provide complementary information. These results support a key message of our work: both object-centric and global features are necessary for achieving high predictive performance. We thank reviewer YQGn for this insightful suggestion.

Limitations discussion.

We agree that one limitation of our approach is the reduced computational efficiency, as multiple CLIP inference runs are necessary. However, as it is possible to batch the runs of multiple objects together, this should not be a major hinderance in practice. Nevertheless, we added this limitation to the discussion section of the paper.

Ablation: Impact object size.

We appreciate the reviewer YQGn's observation and agree that very small object crops may fall outside the typical CLIP training distribution. We therefore evaluated OCB on Pascal VOC using different minimum object size thresholds (as a fraction of the image area). The results below show that performance gradually degrades as smaller objects are excluded, suggesting that including smaller object proposals contributes positively, despite the potential domain gap:

Aggregation methods object mapping?

While four out of the five aggregation methods do lose the explicit mapping of concepts to individual objects, the concat method retains this information and offers a good trade-off—provided memory usage and a fixed object limit are not an issue. We discuss this in Section 5.3. Additionally, gradient-based techniques or attribution methods could potentially help disentangle which objects contribute to a decision, even when using pooled representations.

We trust that these clarifications and additional results strengthen the clarity and completeness of our work, and we respectfully encourage the reviewer to reconsider their evaluation in light of these updates.

[1] Marconato et al. "Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts." NeurIPS, 2023.

[2] Panousis et al. "Coarse-to-Fine Concept Bottleneck Models." NeurIPS, 2024.

We thank the reviewer for their time and questions. We provide clarifications for the points that were unclear below.

Novelty is a bit unclear.

We acknowledge that the individual components of our Object-Centric Bottleneck (OCB) framework, such as object proposal generation and concept classification, are not novel in isolation. However, the novelty of our work does not lie in proposing a new object detection mechanism, but in how we integrate object-centric representations into the Concept Bottleneck Model (CBM) framework, making it scalable and applicable to complex, real-world visual data.

To the best of our knowledge, this is the first work that operationalizes object-centric concept representations within CBMs and applies them to real-world, multi-label classification tasks. Traditional CBMs rely on global, image-level features and often suffer from spurious or entangled concept associations [1], especially in cluttered scenes. In contrast, our object-centric bottleneck grounds concepts in specific, localized regions via pretrained object detectors, enabling finer-grained, interpretable, and more faithful concept-level reasoning.

While this design may appear simple, bridging object-centric representations with CBMs in a real-world setting is non-trivial and has not been demonstrated before. Prior work exploring object-centric inductive biases has largely been limited to synthetic and single-class environments. Our contribution lies in extending these insights to realistic, diverse, and noisy data, removing the need for manual concept annotations, improving cross-dataset generalization, and demonstrating that object-level concept reasoning is viable and beneficial in practical scenarios.

In summary, the strength and novelty of our approach lie in making CBMs more structured, scalable, and interpretable by rethinking their core representation—from flat, image-level concepts to grounded, object-level ones without requiring custom supervision or architectures.

"The aggregation methods show no clear winner accross datasets"

We are somewhat puzzled the reviewer considers this as a weakness of the paper as it answers a practically relevant question, namely that the specific choice of aggregation method is often not critical. As discussed in Section 5.3, in the absence of dataset-specific knowledge or validation data, simple methods like sum or max generally perform best. More complex options like concat or sum+count can help, e.g., in terms of retaining object-mapping. Importantly, we further observe that on all object-based tasks, OCB outperforms the non object-based baseline independent of the aggregation method used.

Reliance on pre-trained models

We agree that the reliance on pretrained models such as Mask-RCNN and SpLiCE introduces potential failure points, especially when their outputs are of low quality (cf. Section 5.4). However, their usage also brings a major advantage: it enables concept extraction at the object level for real-world data without the need for task-specific, large-scale annotation or supervision. This addresses a well-known limitation of prior CBM approaches, which often require carefully curated concept annotations or extensive domain-specific training [2, 3].

More comprehensive analysis

We agree a failure-case analysis is beneficial and have added such a section to the paper. As we cannot add images and plots here, we describe some of the failure cases of the proposal generation that we encountered.

Overlapping proposals: Sometimes it can happen that the object proposal model generates multiple proposals from the same object, despite filtering them for a low overlapping IOU. In this case, the same object appears in the concept space multiple times, which can lead to incorrect results if the exact number of concepts is relevant.

Detecting irrelevant objects: If some objects are inherently not relevant for the task, detecting them and adding them to the concept space can introduce spurious correlations. For instance, in the "plane interior" class of the SUN dataset, most samples also contain humans, which are detected as objects. This can lead the model to incorrectly associate the presence of humans with the "plane interior" label.

Additional baseline results

We additionally wish to highlight novel baseline and ablation evaluations resulting from other reviews and strengthen the findings of our work.

1. Opaque Upper-Bound Model

To quantify the performance potential of our concept encoder backbone (CLIP), we introduce a non-interpretable model that directly maps CLIP image embeddings to class predictions using a non-linear multi-layer perceptron (MLP) (2 layers). It serves as an upper bound on the performance that can be achieved using CLIP features alone. As expected, it outperforms interpretable CBM variants slightly, but comes at the cost of complete opacity, offering no meaningful explanations. OCB closes the performance gap between CBM and the opaque model, underscoring its value for transparent, high-performing concept reasoning.

Caption: Upper bound with frozen CLIP encodings and MLP classifier compared to CBM and OCB.

2. Coarse-to-Fine CBM (C2F)

We further evaluate against the recent Coarse-to-Fine CBM [4], which introduces hierarchical, patch-based concept processing. Unlike many CBM variants that differ primarily in vocabulary, C2F contributes a distinct architectural inductive bias. To ensure a fair comparison focused on architecture rather than vocabulary, we ran C2F with the same LAION-based concept vocabulary of our OCB setup and compare to OCB without sparsity constraints. Since C2F only supports single-label classification, we evaluate it on the SUN397 and COCOLogic datasets. We observe that our OCB model consistently outperforms C2F under these conditions, highlighting the benefit of object-centric representations over patch-based hierarchies.

Caption: Comparison to Coarse-to-Fine CBM (with LAION vocabulary).

Sparsity Analysis

We next analyze the impact of the sparsity constraint on the base CBM and OCB (Pascal-VOC). We evaluate both models (Base CBM and OCB) with varying $\ell_1$-penalties, which controls the number of active concepts for image and concept-level encodigns. As reference, we also provide the result where both models are not constrained and generate activations for every concept.

The results below show that both models improve as sparsity is relaxed. However, OCB consistently outperforms the Base CBM across all sparsity levels, indicating that object-centric representations yield more informative and efficient concepts.

These trends confirm that the performance gain of OCB is not due to increased capacity, but rather to the stronger inductive bias of object-centric representations. We have added this experiment and discussion to the main paper.

OCB: Contribution of Object vs. Image-Level Encodings

Finally, we observe that removing the image-level encodings from OCB leads to a substantial performance drop (Pascal VOC), confirming that object-centric and global representations provide complementary information. These results support a key message of our work: both object-centric and global features are necessary for achieving high predictive performance. We thank reviewer YQGn for this insightful suggestion.

All novel results and discussions have been integrated into the main paper.

We hope that these clarifications and additional results strengthen the clarity and completeness of our work, and we respectfully encourage the reviewer to reconsider their evaluation in light of these updates.

[1] Marconato et al. "Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts." NeurIPS, 2023.

[2] Koh, Pang Wei, et al. "Concept bottleneck models." International conference on machine learning. PMLR, 2020.

[3] Yang, Yue, et al. "Language in a bottle: Language model guided concept bottlenecks for interpretable image classification." CVPR, 2023.

[4] Panousis et al. "Coarse-to-Fine Concept Bottleneck Models." NeurIPS, 2024.