Explainable Concept Generation through Vision-Language Preference Learning for Understanding Neural Networks' Internal Representations

Thank you for your thoughtful review and for recognizing the technical soundness and novelty of our framework. We appreciate your critical insights and provide detailed clarifications below.

Please refer to this Anonymized GitHub Link where we have compiled detailed explanations for better understanding.

Q1: Concerns about seed prompt acquisition.

We agree with the reviewer’s observation that our current setup is primarily focused on low-level visual concepts. But our current setup can easily be modified to capture higher-level semantic concepts (e.g., gender, age) by adding task specific questions like, “What is the gender of the person in the image?” or “Which age category does the person in the image belong to (young/old)?”.

To verify the usability of the proposed methodology with higher-level semantic concepts like gender, we trained a Resnet18 classifier on CelebA dataset to classify images as “Blonde” and “Not Blonde”. This dataset is known for having a spurious correlation between class “Blonde” and females. With our method, we were able to find the same correlation. Concepts generated for the female face were more important than male face. As a side note, when we train RLPO to capture higher-level semantic concepts, it starts combining one or more than one low-level features. As shown in examples on anonymized github, the generated samples start developing long and blonde hairs for both male and female concepts.

Q2: Concerns regarding limited evaluation.

We would like to clarify that, while the main text includes examples like “zebra” and “tiger” for clarity and interpretability, RLPO was evaluated on a broader range of randomly selected samples from ImageNet (see Section 4.3, Appendix D.3, and Table 4). RLPO has also been tested on multiple pretrained models such as GoogleNet, InceptionV3, indicating RLPO is not tied to a specific architecture.

Additionally, as mentioned in Appendix D.9, rather than predominantly featuring specific cases, while conducting the human survey we considered 10 unique classes and randomly selected examples. Apart from that, we also demonstrate how RLPO generalized to non-visual domains like sentiment analysis (see Section 4.6 and Fig. 8).

Q3: Justification for the assumption that $\text{C}_H \subseteq \text{C}_G$ and $\text{C}_R \subseteq \text{C}_G$.

Intiutively, the generative models like Stable Diffusion are trained to learn the distribution of real-world data (which contains human defined and retrieved concepts). By leveraging this learned distribution, they can create existing or new data which by design can represent human defined or retrieved concepts (or neither). That being said, to solidify this assumption we plotted the clip embeddings of generated, retrieved and human defined stripes concept. For retrieval based concepts we used the stripes collected by CRAFT for the zebra class, for human defined concepts we used the concepts collected by TCAV authors for the zebra class, and for generated concepts we used pre-trained stable diffusion 1.5 to generate random stripes images. As shown in the plot on anonymized github, the generated stripes encapsulates both human-defined and retrieved concepts hence proving the assumption $\text{C}_H \subseteq \text{C}_G$ and $\text{C}_R \subseteq \text{C}_G$.

Please refer to this Anonymized GitHub Link where we have compiled detailed explanations for better understanding.

We are thankful to the reviewer for the feedback. While we appreciate the reviewer's recognition of the proposed framework, we believe there are some key misunderstandings regarding the claims and the experiments conducted in the paper.

In this work, we propose a novel approach to “generate” concepts that truly matter to the neural network by utilizing reinforcement learning-based preference optimization (RLPO) on diffusion models. And, we show qualitatively and quantitatively that the generated concepts matter to the neural network. This has been supported by Reviewer 1, 2 and 4. With regards to experiments, please see our detailed answers below.

Q1: Generalizability beyond visual concepts.

RLPO does not solely depend on direct linguistic grounding. Instead it uses reinforcement learning to evolve and refine the generative model via XAI feedback (from the model under test) - indirectly capturing internal representation that may not have exact language equivalents. The use of language (seed prompts), serves primarily to narrow the search space and provide a reasonable initialization for the concept generation. While the generated concepts may initially align with the seed prompts, RLPO iteratively optimizes the generation process via TCAV-based rewards, allowing for drift and refinement toward more model-relevant abstractions—even beyond the original prompt scope. Additionally, to highlight the generalizability of our approach beyond visual concepts, in Section 4.6 (Fig. 8), we demonstrate how RLPO generalized to non-visual domains like sentiment analysis, using textual input and preference.

Q2: Clarification on C-deletion.

We agree with the reviewer that the original C-deletion is performed in the pixel space. But we would like to clarify that, the C-deletion we show in the paper is also performed in the pixel space and not in the textual space. As highlighted in Q1, the use of language is to narrow down the search space. Once the training is completed, we generate concepts via the trained diffusion model and map those generated concepts back to the input space using CLIPSeg (as shown in Figure 5).

The C-deletion graphs shown in paper are obtained from deleting most relevant to least relevant target concepts from the input images. We have provided more elaborate examples in Fig. 21 and 22 of Appendix.

Q3: Potential semantic leakage in diffusion alignment since the vision models are already trained on ImageNet? We need more details on the concept vocabulary, how many semantic names are out of ImageNet vocabulary? How many classes can it generate? Can it generate novel categories/objects?

We understand the reviewer’s concern on potential semantic leakage in diffusion alignment since these models have already been trained on ImageNet data, but we would disagree with the reviewer because the concepts we generate don't come from class data. As shown in Table 4, the generated concepts by RLPO are farthest from the class data. On a contrary, this semantic leakage occurs in retrieval based methods where the concepts are collected from class data. This is the exact problem we are trying to resolve with our method.

Q4: The action space is not scaled, only a few words. Besides, most prompts are single phrases, and thus cannot scale up to diverse compositions?

Our current action space consists of 20 seed prompts, preprocessed and extracted using VQA (see Appendix C.3). We would like to emphasize that our approach does not rely on a direct mapping from seed prompts to the final generated outputs (concepts). While the generated concepts may initially align with the seed prompts, RLPO iteratively optimizes the generation process via TCAV-based rewards, allowing for drift and refinement toward more model-relevant abstractions—even beyond the original prompt scope. Consequently, the final generated concepts capture more diverse and model-relevant abstractions that extend beyond the limitations of the initial single phrases.

Q5: What if generated images in both groups are both garbage? Will it still update the SD?

If both image sets yield low TCAV scores (i.e., do not activate the model meaningfully), our method does not update the diffusion model. Only when a concept has the potential to move toward explainable states do we update the diffusion model. We will clarify this further in the revised manuscript.

We appreciate the reviewer’s constructive comments and recognition of the importance of concept generation in our work. Our method improves upon traditional automatic concept retrieval approaches. While conventional methods extract concepts directly from the dataset—risking semantic information leakage—our approach generates novel concepts that are independent of the dataset.

We have addressed the reviewers' concerns as follows.

Please refer to this Anonymized GitHub Link where we have compiled detailed explanations for better understanding.

Q1: Reliability on text prompts.

Our use of seed prompts serves primarily to narrow the search space and provide a reasonable initialization for the concept generation. Given that Stable Diffusion is conditioned on text prompts, starting with semantically meaningful phrases helps the RL agent converge faster toward relevant concept regions.

While the generated concepts may initially align with the seed prompts, RLPO iteratively optimizes the generation process via TCAV-based rewards, allowing for drift and refinement toward more model-relevant abstractions—even beyond the original prompt scope. We demonstrate this evolution through multiple RL steps (e.g., Figure 2, Appendix D.4), and also show in ablation (Appendix C.4) that random prompts perform significantly worse, indicating the importance of a good starting point rather than dependence.

Q2: Hyperparameters used for TCAV score calculation.

TCAV can be sensitive to choices like the layer of activation and the classifier used. To address this, as stated in Appendix C.1, we replaced the default SGD classifier with Logistic Regression, which we found provided more stable CAVs with lower variance. Regarding the choice of layers and target classes, we provide details in Appendix C.3.

Q3: Bias introduced by pre-trained models.

We acknowledge that the diversity of generated outputs depend on the generative model's capabilities. Issues such as insufficient representation of certain patterns could limit the range of explanations. However, such limitations are not unique to generative approaches—they are also inherent to retrieval-based methods, which are similarly constrained by available data and even human collected concepts, which are influenced by cognitive bias. However, we agree that this is a good point and we will include a discussion of this limitation in the revised manuscript. Specifically, we will highlight the dependency of the explanations on the generative model's capability to produce high-quality and diverse outputs. Thank you for pointing it out.

Q4: In Figure 6, it is unclear what x-axis “Steps” refers to.

“Steps” in figure 6 refers to a step in c-deletion. At each step we delete a part of the image representing the target concept. We have provided more elaborate examples in Fig 21 and 22 of Appendix. We will clarify this in the figure caption of the revised manuscript.

Suggestions on related works

Thank you for the suggested references. We will include a discussion of Feature Visualization (Olah et al., Distill) and the more recent diffusion-based concept learning works (e.g., https://arxiv.org/abs/2312.02974, https://arxiv.org/abs/2410.05217). While our method relies on classifier feedback via TCAV and thus differs in motivation, your point about manual curation versus automated discovery is well-taken and worth addressing more directly in Section 2.

Thank you for identifying the novelty and appreciating the experiments. We hope the following explanations will clarify the queries for the reviewer.

Q1: It seems that the users need to have a good understanding of the target concept in order to generate good concept seeds (i.e, proper VQA design and possibly selective prompts). Is it possible to extract concepts from a large paragraph of description texts without prior human knowledge?

We agree with the reviewer’s observation that in our current setup, in order to generate good concept seeds proper VQA design and selective prompts are needed. But because of the modularity of our approach, the generation of seed prompts can be replaced by any other concept generation method. As described in [1], the end user can extract concepts from text descriptions without prior human knowledge and directly use it in our proposed method.

[1] Zang, Yuan, et al. "Pre-trained vision-language models learn discoverable visual concepts." arXiv preprint arXiv:2404.12652 (2024).

Q2: After the concept pictures are generated by SD+LORA, it is unsure how you can align them with the original input picture?

We employ CLIPSeg, a transformer-based segmentation model, to establish visual correspondence between generated concepts and regions in the input images (See Section 4.4 Figure 5). By feeding the generated concept images as prompts into CLIPSeg, we produce heat maps highlighting areas in the input image that resemble the generated concept. This allows us to localize abstract concepts (e.g., “stripes”, “mud”) within the original images, enabling interpretable alignment between concept space and class-specific features.

We hope these responses clarify your concerns. We appreciate your recognition of the work's broader implications and suggestions for future extensions, which we plan to incorporate.