Explainable Concept Generation through Vision-Language Preference Learning

We are thankful to the reviewer for the detailed feedback. While we appreciate the reviewer's recognition of the interesting questions posed and the novel directions explored in our work, we believe there are some key misunderstandings regarding the goals and contributions of the proposed method.

Since there are various types of contributions such as frameworks, datasets, algorithms, insights, applications, etc. at ICLR, we would like to clarify that our paper is an algorithmic contributions paper. Please see our detailed answers below.

Please refer to this Anonymized GitHub Link where we have compiled detailed explanation and images/plots for better understanding.

W1.

Are our goals misaligned with XAI? We would like to clarify that the stated goals of our work are not misaligned with the broader goals of explainable AI (XAI). Our primary objective, consistent with XAI and what the reviewer states, is to improve the user’s understanding of the model. However, there is more nuance to this broad goal. The focus on "novelty," that most existing methods cannot provide, is not a deviation from this goal but rather an added dimension to further this goal. While we appreciate the reviewer's perspective, we probably do not want to fix the goals of XAI as it’s an evolving field.

Unlike in the most classical setting of XAI where we want to come up with explanations of a particular decision that helps a non-ML-expert user (e.g., say, a medical doctor), our goal is to provide insights to expert machine learning engineers or data scientists to identify what the neural network has learned (so that if there are any vulnerabilities they can fix before deployment). As highlighted in recent discussions [1] and our paper (Figure 2), humans cannot understand everything a neural network has learned because they do not learn using the same concepts that we learn–their learning manifold is different to ours. If a human were to manually come up with a concept set, they are going to miss important concepts because they do not know about the neural network’s learning manifold and therefore the engineers cannot fix the vulnerabilities of the network. That is why we need a method to automatically probe millions of concept configurations and see what really matters. Since trying millions of configurations is not computationally feasible, we formulate a deep RL solution that generates concepts using diffusion. By doing so, we do not deviate from the original goal.

We believe the confusion happened because “human” was used as an overloaded term, resulting in a perspective clash. In the classical TCAV setting, two groups of humans are involved: those who create concept sets offline (“creator humans”) and those who utilize these concepts online (“user humans”). When we state “humans" we are specifically referring to the creator humans, as our contribution is developing an algorithm for concept set creation/generation rather than the downstream application of an existing method. We argue that the concepts that the model indeed uses can be divided into two groups: concepts that creator humans can think of/retrieval methods can create and those they cannot (i.e., novel concepts). Our method can generate concepts from both groups. The latter group is more useful to debug models as shown in our concluding experiment (Figure 9).

We also believe that "abstractness" is an interesting byproduct of RLPO, as correctly identified by R1, R3, and R4. They provide a layered understanding of the model’s reasoning, revealing both high-level and low-level concepts that contribute to its decisions—something that, for the best of our knowledge, the XAI community has never seen before.

While we like the utility metric introduced in [2], it does not help measure novelty. If we crop part of an image as a concept, the utility score will be high, though this concept is not novel at all. Our argument is that there are other patterns/concepts that trigger the neural network (the goal of this work) and identifying them is important to fix the issues. Please see Table 2 where we explain this gap between “what the human thinks” vs. “what the neural network has actually learned.” One motivation for closing this gap stems from our attempt to understand why neural networks perform badly for certain complex decision-making tasks.

1. Schut, Lisa, et al. "Bridging the human-ai knowledge gap: Concept discovery and transfer in alphazero." arXiv preprint arXiv:2310.16410 (2023).
2. Colin, Julien, et al. "What i cannot predict, i do not understand: A human-centered evaluation framework for explainability methods." Advances in neural information processing systems 35 (2022): 2832-2845.

I'm not as familiar with RL and I find your responses adequate for Q1, Q2.

Training the RLPO framework takes approximately 6–8 hours for a particular class...

This is a lot of time spent to analyze a single class of a model. If you were to analyze all of imagenet this would result in 6000 hours (250 days). Once again, this strikes me as not particularly useful for most users unless the method can provide extremely valuable insights.

As highlighted in recent discussions [1] and our paper (Figure 2), humans cannot understand everything a neural network has learned because they do not learn using the same concepts that we learn–their learning manifold is different to ours. If a human were to manually come up with a concept set, they are going to miss important concepts because they do not know about the neural network’s learning manifold and therefore the engineers cannot fix the vulnerabilities of the network.

I have no problem with the goal of generating concepts that may be different than what humans come up with, this is one of the strengths of the work as I indicated. My concern is that the results in Figure 4 do not immediately strike me as useful for human understanding. If it was clear that they were useful, experiments showing that the explanations help human understanding would not have been as necessary. However, the results are, as you indicate, abstract. Thus, you are making a strong claim that this abstractness is a positive and not a negative for human understanding of the model. Unfortunately, there are no experiments that back this claim up. I still believe this work needs a clear demonstration of usefulness for explainability. Also, it's a reach to claim that this method surfaces vulnerabilities in networks without providing experiments that either exploit or defend against these vulnerabilities.

We believe there is a misunderstanding about terminology. We mean abstractions not as in “abstract arts” but as abstraction levels as in Computer Science—a representation of an idea or concept, often removing specific instances or details to focus on the broader principle. In our case, we go from zoomed-out to zoomed-in the neural network’s activation space (to clarify, not physically zooming): zoo->animals->animals with stripes->stripes, etc.

We do not think we make a strong claim about abstractions as what we state is “we observe the progression of output concepts generated by the SD” and, we visualize it to help understand the progression theoretically (Figure 1 and Appendix B). Also, please note that abstraction levels are a byproduct—the main contribution is generating novel concepts, for which we have provided plenty of experiments (the main results figure is Figure 6, not Figure 4.). Since it is not a must for the user to use all levels of abstractions (just using the final abstraction level is totally fine for some applications as in most XAI methods), we do not see how it is a negative aspect. For applications such as long-horizon planning, abstractions are important, though not mandatory for every use case. Instead, they serve as an additional layer of insight.

We do not claim “method surfaces vulnerabilities,” what we claim is that "if there are any vulnerabilities, they can be fixed before deployment.” The first sentence is a definite claim and the second one is an ability/opportunity. If it was misunderstood, we can further tone it down. The typical workflow of a methods paper with 9 pages is providing the method and experimentally validating that the method is correct ideally with some theoretical insights. We went a step further by illuminating the future potential for model debugging as depicted in Figure 9. We believe it is not fair to ask to run adversarial examples and how to fix them as it is an application which is work worthy for a completely different contribution/paper.

Thank you for the response! The reviewer’s new understanding about the claims is correct. Thank you for acknowledging that the paper has successfully demonstrated its claims.

Since we have clarified all of previous concerns and new concerns below through new experiments, we sincerely hope the reviewer can reconsider the score. As the reviewer can also see, we honestly put a lot of extra effort into rebutting 6 reviewers, mostly because this is not a classical XAI-style paper. We are sure that the reviewer also thinks that the contribution of the paper, the amount of experiments we have run, and the rebuttal is worth updating the score. Please see our answers below.

The results presented in Figure 9 of the paper highlights a potential use case of the proposed method. It shows how concepts shift with fine-tuning and the proposed method’s ability to detect these new concepts.

To further showcase the usefulness, we conducted an additional experiment. In this experiment, we choose a pre-trained Googlenet classifier for the Tiger class whose important seed prompts were ‘orange black and white’, ‘orange and black’, and ‘blurry image’ with TCAV scores of 0.66, 0.66, and 0.62, respectively. Out of these seed prompts, ‘orange black and white’ and ‘orange and black’ highlight the tiger pixels while ‘blurry image’ seed prompt highlights the background pixels (see sample explanations in anonymous github). What that means is, in order to classify a tiger, Googlenet looks at both the foreground and background. Now the engineers want the classifier to classify the tiger based on tiger pixels, not its background (note: from the classical Wolfe-Husky example in LIME [1], we know the spurious correlation of background).

To this end, we generated 100 tiger images based on concepts related to ‘orange black and white’ and ‘orange and black’ using a separate generative model and fine-tuned our Googlenet model. Running RLPO on this fine-tuned model revealed that the model learned some new concepts such as ‘whiskers’ and also revealed that previous concepts such as ‘orange black and white’ and ‘orange and black’ are now more important with TCAV scores of 1.0 and 1.0, respectively. This means that the classifier is now only looking at tiger pixels, not the background. (see dataset samples and shift plot in anonymous github). This experiment clearly shows how the proposed method can be used to improve a neural network’s undesirable behavior.

1. Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. "" Why should i trust you?" Explaining the predictions of any classifier." Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 2016.

We thank the reviewer for the constructive comments and identifying the method as interesting. We have addressed the concerns in weakness and questions as follows.

Please refer to this Anonymized GitHub Link where we have compiled detailed explanation and images/plots for better understanding.

W.

To summarize:

Our claim: When considering automated concept set creation techniques (that are retrieval based), the proposed generative method can come up with novel concepts that trigger the neural network (unveiling such new patterns can help engineers fix models).

Experimental evidence to support the claim:

1. Quantitative: Table 4 shows that our method can generate new concepts.
2. Qualitative: Figure 4 illustrates some results of Table 4.
3. Human evaluation: Most humans cannot imagine certain patterns will even trigger the neural network.

Let us elaborate on this.

To evaluate the concept's relevance to the model, we evaluate the method’s success through multiple metrics, as reflected in the experiments section. Table 4 provides quantitative evidence for the relevance of the generated concepts by measuring TCAV scores. Higher TCAV scores indicate that the concepts discovered are aligned with the model’s internal representations, confirming their significance for decision-making. This shows that the generated concepts "matter" to the model. We use metrics such as cosine similarity and euclidean distance to show that the generated concepts are novel—generated concepts are farther away from the class images, indicating that concepts generated by our method are not a subset of class data as in retrieval methods.

We also conducted a human experiment (results shown in Table 2), where when asked for identifying relevant concepts within generated, retrieved or both, most volunteers irrespective of laymen or experts mostly picked retrieved concepts, even though both were equally important to the network. This experiment indicates that it is not easy for humans to imagine concepts by themselves as the neural network’s learning process is different to ours.

Furthermore, to help humans understand what each relevant concept represents, we made use of ClipSeg to identify the intersection between generated concept images and test images. Figure 6 highlights the regions each relevant concept represents in the test image.

Q1.

For the experiments presented in the paper, the RLPO framework with DQN+Diffusion typically requires approximately 8 hours per class on a machine equipped with an NVIDIA RTX 4090 GPU to train, with the most computationally intensive step being the iterative fine-tuning of the generative model. This concept set creation is a one-time investment for an application of interest. Evaluating TCAV is pretty much real-time once we have the concept set. We will include this discussion in the revised version of the paper.

Q2.

The images in Figure 6 are final outputs of the RLPO framework–this is what most people want as explanations. They are the lowest level of abstraction in explanations. However, if someone wants higher levels of abstractions in explanations, as shown in Figure 5, they can also obtain them. In Figure 5, the explanation of the tiger class progresses through levels of abstraction from high to low: the importance of zoos, followed by animals in zoos, then striped animals in zoos, and finally orange-and-black striped animals, and so on.

Do concepts explain the model?

We believe the philosophical question of whether the concepts really explain the model is valid for all concept-based techniques. If a human collects a concept set, is it guaranteed that another human will perceive the same pattern? Not necessarily, as cognitive biases can influence interpretation. If a retrieval method crops and collects some parts of images, is it always understandable to humans? We believe the same argument is true for generated concepts as well. The concepts our method generates are explainable by design since RLPO iteratively fine-tunes the diffusion model to generate images with high TCAV scores. Additionally, to further verify whether these generated concepts are indeed explaining the neural network we performed the classical c-deletion experiment. As shown in Fig. 8, we see that gradually removing these concepts from the input leads to drop in average-class accuracy. If they were some irrelevant concepts, we would not have seen such a drop.

The user human would see something similar to Figure 6 in manuscript (see Anonymized GitHub Link). For a zebra test point, the user will see three sets of concept images shown on the right, with the most important at the top. The red set (highest TCAV) highlights stripes, and on the left, it shows which parts of the test image these concepts are most relevant to. The other two sets provide supplementary explanations. The green set illustrates a green wooded background (note that Stable Diffusion’s seed for this was running, but RLPO fine-tuned it to generate such backgrounds—seeds are used only for our analysis and are not important for the user human). The blue concept set depicts a brown background with some green on the horizon. These supplementary explanations primarily describe the background. Consolidating everything, while the most important concept for the network to classify a zebra is black-and-white stripes, its habitat, including a brown background and green trees, also contributes to the classification.

“To this end, we leverage the state-of-the-art text-to-image generative models to generate high quality explainable concepts.” What we meant by high quality is the quality of images generated by the model. We can rephrase and tone it down. We acknowledge that our perspective was rooted in the creator human’s mindset as we were on a quest to automate the concept set creation process. As highlighted in the concluding experiments of section 4.5, we truly hope our work would benefit 1) engineers debug issues in neural networks (Figure 9) and 2) make the use of TCAV easier in downstream applications (Figure 10).

We thank the reviewer for their prompt response and giving us the opportunity to clarify the queries.

Please refer to this updated Anonymized GitHub Link where we have compiled detailed explanation and images for better understanding.

We believe the confusion happened because “human” was used as an overloaded term. In the classical TCAV setting, two groups of humans are involved: those who create concept sets offline (“creator humans”) and those who utilize these concepts online (“user humans”). When we state “humans cannot,” we are specifically referring to the creator humans, as our contribution is developing an algorithm for concept set creation/generation rather than the downstream application of an existing method. To clarify, we do not claim that the generated concepts are not understandable to the user humans. Rather, we argue that the concepts that the model indeed uses can be divided into two groups: concepts that creator humans can think of and those they cannot (i.e., novel concepts). Our method generates explanations from both groups, and both types of concepts are understandable to user humans.

Understandable should be corrected as guessed. For instance, in the reviewer’s summary “concepts are not likely to be understood guessed by creator humans.” Vanilla TCAV requires creator humans to come up with (i.e., guess) a set of concepts to test ahead of time. In other words, it assumes the creator humans know a set of explanations ahead of time as TCAV only picks from the set which concept/explanation is correct. When we tried to apply TCAV in some real-world applications, we found out that it is impossible for creator humans to guess all such concepts/explanations ahead of time. That motivated us to generate concepts.

Interpretation of results in Table 2:

Hypothesis: Creator humans struggle to guess all important concepts that truly matter to the network.

Experimental setup: Rather than asking creator humans to come up with large concept sets, we present them with two concepts and ask them which one they would include in the concept set. To this end, as detailed in Appendix D6 (and the following image), we show creator humans a test image along with two explanations—one obtained from a retrieval-based method and the other generated by our method (without revealing the experimental setup to them). Those creator humans were asked to determine which of the two explanations could have influenced the network's decision that the test image, for example, represents a zebra. They could choose the first explanation, the second, or both.

Metrics: All generated concepts matter to the network as they have a high TCAV score. Out of them, how many of them are identifiable by creator humans?

Results: The results in Table 2 shows that creator humans often recognized explanations from retrieval-based methods, as these align with cropped elements of the test images. However, they were less likely to guess generated explanations, even though these are valid concepts that influence the neural network's decision and have high TCAV scores. This confirms that our method successfully discovers valid explanations that are not immediately apparent to creator humans.

Thank you for recognizing the innovation and clarity of RLPO, as well as its contribution to improving the efficiency and generalizability of concept-based explanation methods. We also appreciate your valuable feedback on areas requiring clarification, which we will address comprehensively in the revised version.

W1.

We acknowledge that some sections, such as the algorithm explanation and mathematical proofs, may appear overly lengthy. To address this, we plan to streamline these sections in the final version by:

• Providing concise summaries alongside detailed explanations to improve accessibility for readers.
• Moving some of the more intricate details (e.g., mathematical derivations) to the appendix, ensuring the main text remains reader-friendly while preserving the rigor.

We believe this restructuring will make the algorithm and proofs more digestible without compromising their completeness.

W2.

Given the page limit we were not able to include all the details on the sentiment analysis experiment in the main paper. We have added additional explanation for our experiment on sentiment analysis tasks in the Appendix D.4. We will highlight it in the main paper.

W3.

We appreciate your feedback regarding the inconsistent use of terms like “concept generation” and “concept extraction.” We will fix these in the revised paper.

Thank you for your detailed and insightful review! We appreciate the reviewer’s novelty of ideas. Please see our answers below.

W1.

To track the evolution of concepts over iterations, it is necessary to retrain the RL algorithm iteratively. This ensures that we can account for changes in the network as it learns new features and potentially forgets old ones during fine-tuning. However, we can maintain a concept set and keep adding newly discovered concepts to the set. Conditioning on a prior set, how to efficiently find a novel set would be a very new and interesting research direction for generative models in general.

W2.

Thanks for pointing this out! The x-axis represents the concept identifiers, and we will include appropriate annotations in the revised manuscript to make this clear.

W3.

In terms of comparisons, we extensively compared the novelty of concepts generated by our method and retrieval methods using a variety of metrics (Table 4) as well as through human participant evaluation (Table 2). The challenge with retrieval methods is that they leak information about the test set into the explanation, as in its simplest form, they are cropped parts of class images. Human created or generated methods do not have this limitation. However, we find it difficult to design a fair and solid experiment to show this aspect numerically. Not just in our method, even if we compare human created concepts and retrieval methods, they are hard to compare as they offer different perspectives.

W4.

We acknowledge that the quality and diversity of generated outputs depend on the generative model's capabilities. Issues such as mode collapse or insufficient representation of certain patterns could limit the range of explanations. However, such limitations are not unique to generative approaches—they are also inherent to retrieval-based methods, which are similarly constrained by available data and even human collected concepts, which are influenced by cognitive bias. Though mode collapse was an issue in GANs, we find it hard to find solid literature on mode collapse in SOTA diffusion models. However, we agree that this is a good point and we will include a discussion of this limitation in the revised manuscript. Specifically, we will highlight the dependency of the explanations on the generative model's capability to produce high-quality and diverse outputs. Thank you for pointing it out.

W5.

Although the experiment the reviewer suggested is really interesting, it would be difficult to replicate RLPO with GANs, since we can not generate arbitrary images based on any random text prompt with GANs. As explored by past research [1], it is possible to develop a GAN to generate random images based on textual description, but they have to be trained specifically for a particular dataset. That is in a way one of the major limitations of using GANs, it is hard to use them beyond the task they are trained to do. Instead, to test the reviewer’s hypothesis, we tested the RLPO experiment with an older version of Stable Diffusion (SD v1.1), which is known to produce biased (arguably, it is a result of mode collapse) and suboptimal images.

1. Reed, Scott, et al. "Generative adversarial text to image synthesis." International conference on machine learning. PMLR, 2016.

Area under curve obtained after applying c-deletion on top 3 concepts generated by SD-1.1 and SD-1.5 for “zebra” class:

We see that we get the lowest area under the curve for the SD v1.5, indicating that concepts generated by SD v1.5 (the good generator) are more related to the class features learned by the neural network. Therefore, the quality of the generator matters.

We appreciate your thoughtful review, which acknowledges the originality, experiments, and clarity of our approach. Please see our response below. Please refer to this Anonymized GitHub Link where we have compiled detailed explanation and images/plots for better understanding.

W1.

Are generated images not representative of the seed prompt? We would like to clarify a potential misunderstanding that the generated images should look like the seed prompt. With the example described in the paper (Figure 5), we wanted to show that there are millions of very diverse images that can be generated using the seed prompt (in this case, “zoo”). However, because of RLPO, the diffusion model learns to only generate images that explain the network’s decisions. Figure 5 demonstrates this process. Please observe that the seed prompt “zoo” becomes more animal-ish at t=10, then animals get more stripes at t=20, then colors appear at t=30, and so on. We have now added this description in the paper.

Are images unrealistic? Neural network classifiers can provide the same output for different input patterns/concepts. Some of those concepts might be expected whereas some others are unexpected or unrealistic. As we argue with reference to Figure 2, we want RLPO to find such concepts that the human might think of. Therefore, the ability to reveal unrealistic concepts that give a high TCAV score is what we indeed wanted (note that the image quality does not degrade in this case as the TCAV score is higher and the Aesthetics score>3.). Having said that, for the specific case in Figure 5, please note that t=30 is not the final image—we just showed timesteps until the class label flips to tiger.

Aesthetics scorer: https://github.com/discus0434/aesthetic-predictor-v2-5?tab=readme-ov-file

W2.

What’s the necessity of RL? Using only GPT was indeed our first attempt, which proved to be highly inefficient. Note that the action space is not 20 seed prompts but the combination of the 20 seed prompts. Assuming 30 mins per run, it will take 2^20 * 30 = 182 years to do this if we brute-force. Since RL intelligently and dynamically picks which prompt combinations to use (and not use), RLPO takes only ~8 hours. Therefore, unlike a static ranking approach, our RL-based framework is much more pragmatic to handle unbounded generative models. The high epsilon case in Table 1 is somewhat similar (yet better) to brute forcing through the seed prompts. To see the quality of generated images with and without RL (seed prompt “stripes” for the zebra class after 300 steps), please see the images in this Anonymized GitHub Link.

W3.

Is computational complexity an issue? We agree that we should have clarified this point in the paper. We can still obtain explanations in real-time because TCAV can run in real-time. However, we agree that RLPO cannot create concept sets in real-time, mainly because of the diffusion fine-tuning step (RL is very fast). However, we do not think there is a need to create concept sets in real time. For instance, if we apply TCAV for identifying a disease from an X-ray, we can create the concept set using a test set ahead of time before deployment, which will take a few hours, and then run TCAV in real-time. Hence, concept set creation is a one-time investment. In case of a long-term distribution shift in a particular application, we can keep adding concepts to the dictionary, if RLPO discovers anything new. Please also note that the traditional method of manually creating a concept set can not only be slow and labor intensive but also can miss the important concepts.

Q1.

Are explanations faithful? Good point. Compared to a random search algorithm, RL dynamically rewards updates that give a high TCAV score. Therefore, conceptually, we do not expect RLPO to generate totally different concept sets every run. We now ran the RLPO algorithm 3 times (i.e. 3 trials) for the same seed prompt set. During inference, we calculated the CLIP embedding similarity between trials for the top three seed prompts (stripes, running, and mud for the zebra class - see Figure 6). The low Wasserstein and high cosine similarities indicate that the generations are similar. A low Hotelling's T-squared score (a generalization of the t test for multiple dimensions) also indicates that the generated images are from the same distribution (for reference, this score is 7488.86 for stripes-running).

Inter-trial Concept Comparisons for “zebra” class across three trials:

Q2.

In our approach, SD is fine-tuned iteratively: generating a set of images, assessing their relevance, and then refining the model to improve alignment with explainability objectives. This sequential framework allows the model to adaptively optimize towards explanations by building upon the outcomes of previous steps. Each step in the sequence informs the next, ensuring that the fine-tuning process converges towards increasingly meaningful and interpretable concept representations. This dynamic adjustment is essential for steering SD toward generating high-quality explanations that progressively align with the target class, rather than relying on static, one-shot methods that lack adaptability. Please see the animation we posted in the Anonymized Github.

Q3.

Yes, fine-tuning one RLPO framework with DQN+Diffusion per class is necessary. This is because each class may have unique features and abstractions that require tailored exploration to generate meaningful and interpretable concept representations. The RLPO framework ensures that the generated concepts align closely with the specific nuances of the class as learned by the model, which a generic approach cannot achieve.

“An untuned diffusion model and prompt LLMs” is exactly what is behind most VLMs such as GPT4 (it calls DALL.E2 which is based on a diffusion). In other words, the question is more akin to why don’t we use GPT4 to generate images directly. Finding a single long, descriptive prompt that effectively encapsulates the target class's abstractions is highly challenging, if not impossible, except for something obvious such as “stripes.” Therefore, even in that case we have to develop an automated prompt engineer using RL. From our experience, it is much more efficient and controllable to fine-tune stable-diffusion compared to developing an automated prompt engineer.

Q4.

Thank you for catching this oversight in the appendix. We have rectified this in the revised paper. $t_{\eta}$ refers to the time at which the system reaches an explainable state. After this point, though we can fine-tune SD, TCAV does not change significantly.

Thank you for identifying the novelty and appreciating the experiments. We hope the following explanations will clarify the queries for the reviewer.

Please refer to this Anonymized GitHub Link where we have compiled detailed explanation and images/plots for better understanding.

Wa.

To compute TCAV scores, we need two concept image groups. The reason for random grouping is that, initially, we do not have TCAV scores for individual concept images. Random grouping is not problematic–it is the unbiased sampling method. Why this works is, because images in the two groups are different, the TCAV score in one group is slightly higher. Then we fine-tune stable diffusion to generate images similar to that group with higher TCAV. We regenerate two groups of images from the fine-tuned model and iterate this process. Although the images in initial groups are highly variable, overtime they learn how to generate images of a particular type/concept.

We can also think of a different analogy. This sampling step is somewhat analogous to rejection sampling and Metropolis-Hasting (MH). In rejection sampling, we pick a sample from a proposal distribution, which is typically a uniform distribution, and reject it based on some criteria. Similarly, we generate two sets of images randomly and evaluate which one to reject. In M-H also we compare two points (though they are old and new points). Through such iterative rejections and model updates, we can converge to the target distribution.

Wb.

We would like to clarify that, the action space is not 20 just seed prompts but the combination of the 20 seed prompts. Assuming 30 mins per run, it will take $2^{20} * 30 = 182$ years to do this if we brute-force. Since RL intelligently and dynamically picks which prompt combinations to use (and not use), RLPO takes only ~8 hours. Therefore, unlike a static ranking approach, our RL-based framework is much more pragmatic to handle unbounded generative models. The high epsilon case in Table 1 is somewhat similar (yet better) to brute forcing through the seed prompts.

To see the quality of generated images with and without RL (seed prompt “stripes” for the zebra class after 300 steps), please see the images in this Anonymous Github.

Wc.

We agree that we should have clarified this point in the paper. We can still obtain explanations in real-time because TCAV can run in real-time. However, we agree that RLPO cannot create concept sets in real-time, mainly because of the diffusion fine-tuning step (RL is very fast). However, we do not think there is a need to create concept sets in real time. For instance, if we apply TCAV for identifying a disease from an X-ray, we can create the concept set using a test set ahead of time before deployment, which will take a few hours, and then run TCAV in real-time. Hence, concept set creation is a one-time investment. In case of a long-term distribution shift in a particular application, we can keep adding concepts to the dictionary, if RLPO discovers anything new. Please also note that the traditional method of manually creating a concept set can not only be slow and labor intensive but also can miss the important concepts.

Wd.

Let us explain with an analogy. Why does it snow on Mount Denali in Alaska? It could be due to its high elevation, its location in the Arctic, or the orographic effect—all valid explanations. Similarly, if an autonomous vehicle hits a pedestrian, why did it happen? Perhaps the pedestrian was occluded, the AV struggled to identify pedestrians wearing pants, or a reflection might have confused its sensors.

If only engineers can obtain the range of reasons why a neural network triggers for a particular output, they can assess the vulnerabilities of the neural network and fix them. Humans cannot think of all these reasons because they do not understand the neural network’s learning process. That’s where our method shines.

While generating multiple concepts is not mandatory, it provides an added advantage. In critical applications, relying on a single explanation can be risky, especially if it fails to capture the full scope of the model's behavior. A diverse set of concepts ensures that users and domain experts can explore multiple dimensions of the model's reasoning. These explanations could also potentially allow experts to learn from the model itself [1].

1. Schut, Lisa, et al. "Bridging the human-ai knowledge gap: Concept discovery and transfer in alphazero." arXiv preprint arXiv:2310.16410 (2023).

Once again, thank you for taking the time to provide us with your feedback on our submission. We understand that the reviewer may not have had the time to thoroughly go through the rebuttal yet. If there are any further clarifications or additional details that we can provide to address your queries, please let us know. We are more than happy to provide any additional information or explanations to ensure our approach is clearly conveyed.