LICORICE: Label-Efficient Concept-Based Interpretable Reinforcement Learning

We thank the reviewer for acknowledging that our work identifies an important limitation of concept-based models in reinforcement learning and the method is novel. We’ll address your comments below:

I think what I am missing the most in this paper is more explanations on the interpretability of the policy. Given that the novelty of the method is limited in a way that it combines two existing methods, it would be good to demonstrate the importance of the proposed solution more. As the goal is to produce interpretable policies, it would be good to see some examples of interpretable policies, and maybe even verify that users of the policies find them convincing and useful. Also, the paper talks about the "concepts" in a very abstract way and I would like to see some more examples of the concepts to make it more concrete. For example, table 1 shows the number of concepts and their type, but does not communicate that each concept can take more than 100 values in some cases.

We respectfully disagree that the proposed solution is simply combining two existing methods. Although, at a high level, we leverage tools from the existing literature, these tools do not solve the problem on their own. For example, just using active learning (Disagreement-Q) does not enable us to learn performant RL policies. More generally, CPM assumes unlimited access to an expert, which is not feasible. Understanding and characterizing these challenges is a research contribution on its own. Additionally, our method goes beyond simply applying existing active learning techniques by introducing components to resolve key challenges for interpretable RL training. The results of our ablation study (Table 3) demonstrate that each component is beneficial for achieving high reward and concept accuracy on all four environments.

Regarding interpretability, we appreciate the suggestion to further showcase the interpretability of our approach! To do so, we visualize two examples of how test-time concept intervention can help facilitate proper decision-making. Specifically, we show two examples in DoorKey. We show how the agent incorrectly predicting the door as being closed results in it attempting to open the door with the key (toggle). In contrast, incorrectly predicting the door as being open leads the agent to incorrectly try to move through the doorway. Intervening on this concept enables the agent to correctly choose to open the door or not. We include this visualization in Figure 5 of the updated paper, with the discussion highlighted in blue in Section 4.2, under RQ4.

Good point regarding Table 1. We updated Table 1 to also showcase the number of binary concepts if we binarize the categorical concepts. In our work, concepts are typically properties of the objects present in the environment (e.g., the agent's position and whether the agent is movable along each direction). We are happy to provide additional visualizations in the revised paper. In the meantime, the precise concept definitions can be found in the prompts for the VLM in Appendix A.3.

Additionally, while the experimental studies are good, I think it could be further improved with some real world experiments (as the original concept-based RL paper presented the experiments in the real world) to increase the impact of the paper.

Thank you for the suggestion! We would like to highlight that the original concept-based RL paper was published at a robotics conference. Although we do not have the ability to run real-world experiments (we would love to be able to!), we have run experiments on another complex Atari environment, Pong, to further investigate LICORICE’s performance. Pong includes 1848 binary concepts in total. Of note: this environment includes flickering effects that cause the paddle to disappear for tens of timesteps after each score increase, increasing the concept prediction difficulty. Using $B=5000$ budgets, the results show LICORICE enjoys a much lower concept error than baselines while achieving almost perfect reward.

How do you manage to obtain the diverse predictions in the ensemble given that they are trained in the same way?

We use different random seeds for initializations and stochasticity of SGD for each model in the ensemble. This difference is sufficient to obtain diverse predictions.

More details on the architecture of the policy and concept predictor would be useful

Thanks for your interest in the model architecture! Please check the detailed description of the architecture in Appendix A.2.

Thank you for taking the time to review our paper! We appreciate the detailed comments on our use of VLMs. We’d like to point out the following additional contributions:

1. To the best of our knowledge, we are the first to investigate the problem of a limited concept annotation budget for interpretable reinforcement learning. This problem is important because (online) RL is interactive by definition, requiring human-in-the-loop labels during training. As we have shown with the Sequential-Q baseline, we cannot simply deploy a random policy to sample states from the environment and ask a person to label them: this leads to an inability to learn a performant policy.

2. We introduce LICORICE, a novel training scheme that enables label-efficient learning of concept-based RL policies. LICORICE is designed to address key challenges that arise from the sequential nature of the RL setting. Compared to the state-of-the-art, LICORICE requires 2,000-13,000x fewer queries to achieve the same performance.

3. We present the first study of VLMs-as-annotators in the context of interpretable reinforcement learning, finding that there is some promise for these methods to help alleviate human annotation burden.

We’ll address your comments below:

This method is not general enough. Because environments like cart pole are very simple, it’s possible to use some textual “words” to describe the state. But what about in the real environment with much more states? Also, some of the states are not describable by language.

We conduct experiments on four diverse environments, from CartPole to Atari. Although CartPole is seemingly simple, we would like to call your attention to Figure 2, which shows that budget-constrained baselines fail on CartPole, whereas LICORICE does not. The takeaway is that seemingly simple environments may also require a more carefully-designed algorithm to address key challenges, such as human annotation efficiency.

Furthermore, we would like to point out that Atari is the most common benchmark in reinforcement learning, motivating our decision to select Boxing as one of our environments. In fact, Atari is used more in scientific research than the next 8 benchmarks combined [1]. In addition to its popularity, we choose Boxing due to the size of its state space: Boxing has eight ordinal concepts, meaning that the number of states is of the order quintillion (please refer to Table 4 for the concepts and possible values for this environment). If we were to define these concepts as binary variables as in previous work [2], then Boxing would have k=1480 binary concepts. Using the same process for the other two environments with discrete concepts, we would have k=30 binary concepts for DynamicObstacles and k=46 binary concepts for DoorKey. In contrast, the maximum number of concepts studied in the CPM paper is k=28 binary concepts [2], which is fewer than the minimum number of binary concepts studied in our work.

Additionally, we have run experiments on another complex Atari environment, Pong, to further show the generalizability of our method to different complex environments. Pong includes 1848 binary concepts in total. Of note: this environment includes flickering effects that cause the paddle to disappear after each score increase, increasing the concept prediction difficulty. Using $B=5000$ budgets, the results show LICORICE enjoys a much lower concept error than baselines while achieving almost perfect reward.

Could you please provide a reference for an environment with states not describable by language?

The current experiments are mainly conducted on limited concepts. Can LICORICE be validated on a larger concept space? I suggest the authors validate their approach on more complex tasks. The chosen active learning baseline is relatively simple, but considering this is a first exploration for label-efficient concept-based reinforcement learning, it is acceptable.

Thanks for the comment! First, we want to call attention to the concept space for the two more challenging environments: CartPole and Boxing. In CartPole, the four concepts are continuous-valued, meaning that there is a potentially infinite number of states assuming infinite precision. In Boxing, there are eight ordinal concepts, meaning that the number of possible states is in the order of at least trillions (please refer to Table 4 for the concepts and possible values for this environment). If we were to follow previous work and define these concepts as binary variables, then we would transform each concept value into a separate binary concept, resulting in k=1480 concepts for Boxing. Using the same process for the other two environments, we would have k=30 binary concepts for DynamicObstacles and k=46 binary concepts for DoorKey. In contrast, the maximum number of concepts studied in the CPM paper is k=28 binary concepts [2], which is fewer than the minimum number of binary concepts studied in our work.

Second, we have run experiments on another complex Atari environment, Pong, to further show LICORICE’s performance. The concepts for Pong are borrowed from the VIPER paper [3], including position $(x, y)$ and velocity $(v_x, v_y)$ of the ball, and the position $y_p$, velocity $v_p$, acceleration $a_p$, and jerk $j_p$ of the player's paddle (in total 1848 binary concepts). Of note: this environment includes flickering effects that cause the paddle to disappear after each score increase, increasing the concept prediction difficulty. Using $B=5000$ budgets, the results show LICORICE enjoys a much lower concept error than baselines while achieving almost perfect reward.

Finally, while it may deviate from the topic of interpretable RL, one of the greatest contributions of concept learning to RL is its ability to help RL achieve compositional generalization. I suggest including a related discussion in the section where the concept model is applied to RL.

Thank you for the suggestion! We added these papers to the updated draft and updated the related work section accordingly. Compositional generalization in the context of concepts and, more generally, concept representation learning for RL is a very interesting direction to explore in future work.

Once again, thank you for your valuable feedback! We hope we have addressed your comments. Please consider raising your review score if you feel this process has improved the quality of our paper.

[1] Koh, Pang Wei, et al. "Concept bottleneck models." International conference on machine learning. PMLR, 2020.

[2] Zabounidis, Renos, et al. "Concept learning for interpretable multi-agent reinforcement learning." Conference on Robot Learning. PMLR, 2023.

[3] Bastani, Osbert, Yewen Pu, and Armando Solar-Lezama. "Verifiable reinforcement learning via policy extraction." Advances in neural information processing systems 31 (2018).

Thank you for your thoughtful review! We’re thrilled that you think that addressing the data challenge in interpretable RL is an interesting and important problem. We’ll address your comments below:

This work emphasizes interpretable reinforcement learning, but the authors seem to have directly substituted concept learning for interpretability. I understand that concept learning can, to some extent, help improve interpretability, but the authors should demonstrate more "interpretable" results beyond just performance. I noticed that Table 4 provides definitions of concepts for each task, but how do these concepts influence the actions of different policy models? For me, the relationship between concepts and actions is what truly reflects the interpretability of decisions. For example, when the door is locked and the key is to the agent's right, the agent will move to the right; when the door is open, the agent will move towards the door. The authors should provide similar analyses to demonstrate the interpretability of decisions, rather than stopping at the accuracy of concept learning.

Thank you for the suggestion! We completely agree that concept accuracy is not the same as interpretability; however, it is an important condition for bottleneck models to be utilized in practice since erroneous concepts make it harder to interpret the model’s decision.

One benefit of concept bottleneck models is the ability of a person to intervene on the concepts, as originally demonstrated in section 6 of the CBM paper [1]. We show the results of a test-time concept intervention experiment in Figure 4. Specifically, we find that intervening one by one on the concepts of a concept model with additional noise introduced results in increased reward to optimal or near-optimal performance. This experiment also shows that some concepts may not be necessary to learn completely accurately to obtain a high-performing policy (e.g., in CartPole, we can achieve 100% optimal reward by only intervening on 3 of the 4 total concepts).

We appreciate the suggestion to further showcase the interpretability of our approach! To do so, we visualize two examples of how test-time concept intervention can help facilitate proper decision-making. Specifically, we show two examples in DoorKey. We show how the agent incorrectly predicting the door as being closed results in it attempting to open the door with the key (toggle). In contrast, incorrectly predicting the door as being open leads the agent to incorrectly try to move through the doorway. Intervening on this concept enables the agent to correctly choose to open the door or not. We include this visualization in Figure 5 of the updated paper, with the discussion highlighted in blue in Section 4.2, under RQ4.

Thanks for your response and additional results. Please see my specific response below.

Concept learning and interpretability
If I understand correctly, test-time concept intervention operates during the testing phase. Due to the proposed method, or rather the concept model's ability to provide concept-based annotations for human understanding, humans/VLMs/LLMs can intervene to help improve decision-making performance. This is reasonable and demonstrates the advantages of interpretable models.

However, this does not directly address my question about how well or poorly the method performs in terms of interpretability—similar to how we evaluate a method's performance using metrics. I have further read some interpretable RL methods proposed in recent years, and it seems there isn't yet a good evaluation method for interpretability. Does the author have any thoughts on how to evaluate the interpretability of decisions beyond concept learning performance, or perhaps I haven't properly understood the role of test-time concept intervention in this aspect?

Validation on a larger concept space [solved]

​Literatures
A recently released paper [1] follows previous research on RL with concept learning (PDSketch, NeurIPS'22). Although it would be unreasonable to require method comparisons now, considering that it also focuses on the data efficiency problem in concept learning, I suggest providing a discussion in the literature part.
[1] Learning for Long-Horizon Planning via Neuro-Symbolic Abductive Imitation. arXiv preprint arXiv:2411.18201, 2024.

Overall, the author's detailed responses have addressed most of my concerns. Considering other reviewers' positive reception of this work, I do not oppose the paper's acceptance.

Yes, this is exactly right. Test-time concept interventions enable users to not only improve the performance but also interrogate the policy. For example, a user could ask the counterfactual question: what would happen in this state if the door was locked? By intervening on the door locked concept, the user can directly understand and interpret the impact of this change on the policy.

More broadly, we agree that assessing interpretability is challenging. We could utilize proxies like how rapidly the model improves due to test-time intervention to partly indicate interpretability, yet the most fundamental evaluation should require humans' reception in the context. Given the novelty of concept-based explanations for RL, we believe that conducting a large-scale user study to investigate whether people can accurately predict the outcomes of concept changes, policy behavior, and more would be a promising future direction for assessing interpretability in this context.

Thanks for the reference! One key difference is this work studies sample efficiency in terms of human task demonstrations, whereas we study sample efficiency in terms of human concept labels. However, this paper is very interesting, and we are happy to include it in the related work section.

We appreciate your detailed and constructive review! Regarding significance, we agree that current benchmarks enable easy extraction of concepts by effectively opening the black box of the simulator. However, real-world applications like robotics or multi-agent settings with emergent concepts like high-level strategies would require human annotation. More broadly, as you have noted, the problem of label efficiency is underexplored for concept models in general. By focusing on the neglected area of data efficiency in interpretable and interactive systems, we hope that this work can open up a new avenue of research on human-centered challenges in interpretable machine learning. We would now like to take this opportunity to clarify some of the questions you raise.

In P4L120, Line18 of the algorithm states "Continue training the concept network g on Dtrain, using Dval for early stopping". Does Dtrain refer to only the new data added in this iteration, or the entire dataset collected thus far?

Dtrain refers to the entire dataset collected, since we perform data aggregation (please see L17 of the pseudocode).

In P6L285, the authors state "In Disagreement-Q, the agent similarly spends its budget at the beginning of its learning process". What does the beginning of its learning process refer exactly?

Thanks for the question! The Disagreement-Q agent spends all the budget on the initial policy rollout, with an additional active learning function $\alpha(\cdot)$ to choose samples. In contrast, Sequential-Q does not include this active learning function, so it queries the first B states encountered.

In P9L449, Figure 3 only shows the performance of LICORICE and its variants. Including the 3 naive baselines (particularly random) would help readers understand how LICORICE relatively scale with respect to budget.

Great suggestion. We revised Figure 3 to additionally include the 3 baselines. Across all environments, LICORICE is the most query efficient, overall achieving higher reward and lower concept error than the baselines across all budget levels. This study reveals another interesting finding: LICORICE and Random-Q achieve similar concept error for DynamicObstacles for all budgets. However, LICORICE achieves higher reward for lower budgets (100, 200) and comparable reward when the budget is 300. This finding suggests that LICORICE can learn more efficiently through strategic query selection compared to random sampling.

Once again, thank you for your valuable feedback! We hope we have addressed your comments.

Thank you for your thoughtful review! We’ll address your comments below:

During concept learning, g is trained using D_{train}. However, as the agent's behavior changes, it may encounter new concepts during behavior learning. This can lead to a distribution mismatch between the data used for concept learning and the actual states visited during behavior learning.

We agree! We mention this challenge in Section 3, starting from line 153. To alleviate the issue of concept distribution shift, we propose iterative training so that new collected concepts are on-policy and helpful for concept training. In our ablation study (Section 4, starting from line 460), we find that iterative training primarily leads to higher reward in the two environments requiring more precise control. We also find that it leads to reduced concept error for three of the four tested environments, indicating that it can indeed help with the challenge of concept distribution shift.

The authors mention that LICORICE reduces the need for extensive annotations. I agree with this point. However, I think right now people care more about the computation efficiency instead of the cost of labeling data. It would be better to show the training efficiency with less budget.

Thank you for this thoughtful point. We want to clarify that computational efficiency was carefully considered in our work. As shown in Figure 9, all methods (including LICORICE and baselines) use the same number of environment timesteps and have comparable computational costs. All experiments across the 4 environments were completed within 12 hours on a single A6000 GPU, which demonstrates LICORICE's practical efficiency for real-world deployment.

In the budget reduction session, the authors initially use a large budget and then reduce it to a smaller one. I suggest starting with a small budget and gradually increasing it to identify the minimum human labeling effort required to maintain an acceptable reward.

Thanks for the suggestion. We have revised section 4, RQ 3 accordingly. Changes are marked in blue and the results for this experiment are in Figure 3. We define acceptable to mean 99% of the optimal reward. To study how efficiently the various algorithms can leverage limited labeling budgets, we have also included the baselines in this figure. We find that LICORICE is at least as label-efficient than the baselines, typically yielding much higher performance for the same label budget.

In Algorithm 1, please clarify the definitions of s and c. In line 187, there is an extra “the.”

In Algorithm 1, s means the state and c refers to the queried concept label. We removed the additional “the” in the revised paper.

Once again, thank you for your valuable feedback! We hope we have addressed your concerns. Please consider raising your review score if you feel this process has improved the quality of our paper.