Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents

We first want to thank the reviewer for their work and their appreciation of our manuscript, notably for highlighting the novelty of our method and the clarity of our visualizations. We are also adding to our manuscript new ablation studies. The reviewer can find them in the general answer and in the attached PDF. Let us address the opportunities for improvement provided by the reviewer:

How to adapt to new objects types that could require new relations to consider?

To extend our method to continual RL settings, i.e. if novel objects are detected, the object detector would provide a new object instance. We could incorporate a mechanism to automatically reprompt the LLM/expert to ask them if this novel object type might necessitate novel relations, and train our SCoBots with these novel objects and relations.

Is the concept learned through supervised learning? What if there is no concept label?

Our work indeed assumes access to the objects' properties and labels, and/or to the task context. Interpreting the policies would be difficult without human understandable concepts. However, there exists work in the field of predicate discovery (or predicate invention) [1,2,3,4,5]. Such methods can unsupervisely identify useful relations without access to any label. This represents a valuable future work avenue, so we are adding this discussion to our manuscript.

What if the LLMs generated relation function keeps having errors?

In our experimental settings, this has not been observed. A simple first solution here would be to use code checkers, and reprompt the LLM if the code checker returns an error. There also exists research works on more complex approaches to ensure that LLMs provide working code [6,7,8]. We believe that these could also be incorporated in SCoBots.

[1] Clarke, Edmund, et al. "Counterexample-guided abstraction refinement." Computer Aided Verification: 12th International Conference, 2000.

[2] Athakravi, Duangtida, Krysia Broda, and Alessandra Russo. "Predicate invention in inductive logic programming." 2012 Imperial College Computing Student Workshop.

[3] Hocquette, Céline, and Stephen H. Muggleton. "Complete bottom-up predicate invention in meta-interpretive learning." Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 2021.

[4] Silver, Tom, et al. "Predicate invention for bilevel planning." Proceedings of the AAAI Conference on Artificial Intelligence. 2023.

[5] Sha, Jingyuan, et al. "EXPIL: Explanatory Predicate Invention for Learning in Games." arXiv preprint (2024).

[6] Liu, Jiawei, et al. "Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation." Advances in Neural Information Processing Systems (2024).

[7] Ni, Ansong, et al. "Lever: Learning to verify language-to-code generation with execution." International Conference on Machine Learning. PMLR, 2023.

[8] Kwon, Minae, and Sang Michael. "Reward Design with Language Models." International Conference on Learning Representations (ICLR). 2023.

We thank the reviewer, again, for their valuable time and feedback. We are glad that they found our paper very well written, our method quite interesting and that our experiments do a good job of demonstrating advantages of [our] approach. Let us now address the reviewer's concerns.

Add missing ablation studies.

Before adding these ablations, we wanted to assess their soundness by training agents with the CNN/VAE based object extractors from [1]. We conducted prechecking experiments training neural SCoBots on Pong and Boxing using their object extraction methods and obtained comparable results to the one with our simulated noise.

Due to time constraints and limited writing space, we excluded these experiments from the initial submission. We have summarized our results in the one-page PDF attachment. We are of course incorporating these ablation studies into the main body of the manuscript, along with the following new questions, which are addressed in greater details in our manuscript:

Q4) Can object-centric agents learn with imperfect object extractors ?

Yes, SCoBots policies can compensate for noisy object extraction, as the added noise only leads to slight performance drops or to no performance issue in our experimental evaluation. We believe that the use of robustness techniques such as Kalman filters can help to stabilize further the performances of such agents.

Q5) Can SCoBots learn without the relational concept bottleneck ?

Yes, the neural SCoBots are mostly able to recompute the relations provided to the SCoBots, and obtain comparable performances to the neural SCoBots that use relations. However, the fully interpretable decision tree based SCoBots already suffer some performance loss, and most importantly, agents that do not use relations are more difficult to interpret, thus leading to more difficult misalignment detection and corrections.

[1] Delfosse, et al. "Boosting object representation learning via motion and object continuity." Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Cham: Springer Nature Switzerland, 2023.

Missing related work.

We apologize for this. We apparently have forgotten this related work, in the submitted version of the manuscript. We added it to another paragraph on interpretable/transparent RL methods, placed in our related work section together with further recent publications about interpretable RL. Do not hesitate to provide us with any other reference that we might have missed, we will gladly add them.

We thank the reviewer for their dedicated effort to help us improve our manuscript. Let us address the raised concerns.

1. Limited evaluation (Atari): ALE environments are light, incorporate a diverse set of challenges, and are still heavily used by RL practitioners. E.g., searching "reinforcement learning" together with "robogen" on Scholar, since 2023 leads to 51 results, 6 with "robocasa", and 9330 with "atari". Moreover, our evaluations highlight that even these simple tasks already lead to agents that learn spurious correlations, as acknowledged by reviewer Zj4A ("the experiments do a good job of demonstrating advantages of this approach"). Specifically SCoBots interpretability allowed us to detect and correct these, as well as other RL specific caveats, (e.g. sparse reward, often encountered in robotic environments). Additionally, we evaluate relational reasoning, and ALE games overall include many interacting objects, thus making it a valid choice for evaluations. Robotic environments are indeed more applicable, but they are typically used for learning accurate controls (e.g. the tasks of RoboGen, and Pick and Place, Turning levers, Twisting knobs of RoboCasa). Control policies are complementary to our work, and can be used as high level actions together with our SCoBots higher level reasoning system. We are indeed planning to extend SCoBots to robotic environments for future work, e.g., for tasks involving relational thinking such as the Composite Tasks from RoboCasa. But this will require much adaptive work. We are adding this discussion to the paper and thank the reviewer for the hint.

2. Improve the relation extractor definition. We are now extending the definitions of Diuk et al.: Using their definitions of classes, $C_i$, explicitly reintroduced in our manuscript. The relation extractor augments the object centric space, by applying interpretable expert/LLM provided python functions to every detected object. We extend Diuk et al.'s relation definition from boolean relations (e.g. touch) to real ones (e.g. distance). Formally, $r: C_i \times C_j \rightarrow \mathbb{R}$ is a function, defined over the attributes of objects of classes $C_i$ and $C_j$. The relation extractor is meant to provide richer interpretable states, from which an action selection is easy to derive (allowing for compact interpretable policies). The functions are here not optimized through an explicit loss, but are pruned at tree extraction time. We added these precisions to our manuscript. Thank you helping us improving the clarity of our method.

3. Which object extractor is employed ? As said in the evaluation (l.205): "We focus our evaluations on the agent’s reasoning process, rather than the object identification". We thus used the imperfect light-weight object extraction of OCAtari. For a fairer comparison to deep RL agents, we have conducted additional evaluations, with noisy object extraction (detailed in the PDF and global response).

4. Incomplete MDP definition. Thank you for pointing this out, we have completed this definition (l. 69).

5. Object-centric states are limited. We agree with this remark. We discuss it in the limitation section (cf. l. 320). To address these cases, we can either find suitable neuro-symbolic representations, or combine the symbolic module with a deep one for cases where symbolic reasoning falls short. We updated our limitation section, thank you.

6. Fixed set of relations. Indeed, we have not touched upon continual RL. However, our setup is interactive (human the loop RL) so this set can be updated by the expert/LLM if the current one is insufficient. E.g. if an interpretable agent performs worse than the deep baseline, or new object classes are observed, the expert/LLM can be asked to extend the set of available functions. This represents an interesting future work opportunity. We have added this discussion.

7. Additional reward signals can affect the optimal policy. Modifying the MDP may indeed change the optimal policy, but we report the scores using the original reward signal, thus show that guidance allows for better performing policies in the original MDP. Concerning misinterpretation, our SCoBots policies are completely transparent: one can follow the decision tree (that contains relations and properties of each detected uniquely identified object). Contrary to e.g. importance maps, our method does not imply interpretation hypotheses, but each decision at every time step is fully transparent, independently of the training MDP. We are completing our manuscript to avoid confusion.

Questions

1. Study more realistic tasks like vision-based manipulation? See our detailed answer above (1).

2. What are the model structure and objective functions? See our detailed answer above (2).

3. How to deal with the state space that can not be represented by objects? We actually have a broader definition of objects, which could also be referred to as entity. In this manner, the wind can be represented as an entity (with the characteristic direction and speed). Now if the wind is explicitly measurable, e.g., displayed on the rendered state, the object/concept extractor could extract it as well. For interactive properties that are not explicitly rendered (e.g., wind or invisible wall), we can extend our object detector to deduce properties based on the difference between the expected state and the observed state. We have added this discussion to our limitations.

4. How to resolve the fidelity issues coming from the added rewards? SCoBots have a transparent pipeline, from the detected objects, to each evaluated condition in the decision tree. For example, in the Skiing state depicted in Fig. 4, the agent decides to select RIGHT because the distance between the player and the flag (on the x-axis) is bigger than 15. The interpretable decision is thus readable, independent of the MDP.

We appreciate the reviewer's feedback and agree that the practical application of RL methods in industrial settings is an important direction for future research, and for research in general. However, it is important to recognize that the scope of our paper is to introduce and validate a novel interpretable reinforcement learning method using the most commonly used ALE RL benchmark, that is already enough to support our claims, to demonstrate that deep agents face many caveats, notably learn misalignments, that are easily detectable and fixable using our concept bottlenecks. Contrary to our work, most transparent RL methods are evaluated on even simpler environments (e.g. Cartpole, GridWorld) [1,2,3,4,5,6,7] and are thus not comparable with most RL research and many post hoc explanation methods use Atari or simpler environments [8,9,10,11]. While our method is indeed applicable to other environments, including industrial ones, demonstrating this would require a separate and more extensive study, as we discussed above, with the development and inclusion of options for e.g. robotic tasks. We believe it is somewhat unfair to critic our work based on this, as the gap between research on benchmark datasets and real-world application is a broader challenge facing the entire RL community. Our aim here is to contribute to the foundational science that will ultimately support such real-world applications.

---

[1] Verma, A., Murali, V., Singh, R., Kohli, P., & Chaudhuri, S. (2018). Programmatically interpretable reinforcement learning. In International Conference on Machine Learning ICML.

[2] Bastani, O., Inala, J. P., & Solar-Lezama, A. (2020). Interpretable, verifiable, and robust reinforcement learning via program synthesis. In International Workshop on Extending Explainable AI Beyond Deep Models and Classifiers.

[3] Shi, W., Huang, G., Song, S., Wang, Z., Lin, T., & Wu, C. (2020). Self-supervised discovering of interpretable features for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4] Hein, D., Udluft, S., & Runkler, T. A. (2018). Interpretable policies for reinforcement learning by genetic programming. Engineering Applications of Artificial Intelligence.

[5] Silva, A., Gombolay, M., Killian, T., Jimenez, I., & Son, S. H. (2020). Optimization methods for interpretable differentiable decision trees applied to reinforcement learning. In International conference on artificial intelligence and statistics.

[6] Roth, A. M., Topin, N., Jamshidi, P., & Veloso, M. (2019). Conservative q-improvement: Reinforcement learning for an interpretable decision-tree policy.

[7] Jiang, Z., & Luo, S. (2019, May). Neural logic reinforcement learning. In International conference on machine learning (pp. 3110-3119). PMLR.

[8] Mott, A., Zoran, D., Chrzanowski, M., Wierstra, D., & Jimenez Rezende, D. (2019). Towards interpretable reinforcement learning using attention augmented agents. Advances in neural information processing systems.

[9] Shi, W., Huang, G., Song, S., Wang, Z., Lin, T., & Wu, C. (2020). Self-supervised discovering of interpretable features for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10] Lyu, D., Yang, F., Liu, B., & Gustafson, S. (2019). SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In Proceedings of the AAAI Conference on Artificial Intelligence.

[11] Liu, G., Schulte, O., Zhu, W., & Li, Q. (2019). Toward interpretable deep reinforcement learning with linear model u-trees. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD.

We thank the reviewer for their high appreciation of our paper and valuable feedback. We are glad that they found our method well motivated and it's strong empirical performances. We address the reviewers' raised points hereafter.

Add an experiment with inaccurate object detectors:

We first want to stress that the object detectors included in OCAtari are imperfect (in terms of both F-scores and IoU). Nevertheless, we have conducted a novel experiment, with additional noise (of 5% probability of misdetection and gaussian noise on the position of standard deviation of 3 pixels).

The results are in the PDF, together with another ablation study, using agents with no relational concept bottleneck. \

While the results show that the policy is still mainly able to maintain correct performances besides the added noise, Kalman Filters could further help stabilize the policies. We are interested in robotic domains, we plan on expanding our work to integrate detection methods such as Segment Anything methods (maybe with Kalman Filters). We think that this will require adaptations like the use of control policies as options (i.e. high level actions), as we have added in our future work paragraph.

This suggestion experiment indeed helped us to show that concept bottleneck agents can be used in more complicated environments.

Use Calgary Aytekin's NN to DT conversion algorithm:

We have not been able to reproduce this algorithm to directly extract decision trees. We, however, believe that the decision tree that it will provide will be larger than those of VIPER, which reduces the sampling space. As shown by the authors, the decision tree obtained on the simple two moons dataset already contains many nodes. If the reviewer knows of any other implementation (than the one on this github page), we would be happy to try another algorithm again.

We have also tried the REMIX algorithm for rule extraction, but they both led to bad performing agents.

We can add this comment in the paper.

Questions:

1. How would you handle more complex relationships like those of cooperative behaviors in multi-agent games?
   We have asked an LLM to provide such functions, simply prompting it with:
   "What about more complex relationships like those of cooperative behaviors in multi-agent games ? Can you think of any function that might be required for these cases ?"
   Its answer provided python functions to evaluate and perform:

• Communication Signal,
• Collective Goal Achievement,
• Formation Maintenance,
• Role Assignment,
• Tactical Retreat or Advance,
• Resource Allocation Among Agents,
• Cooperative Targeting,
  which showcase abilities to use LLM in Multi Agents RL as well. We believe that when provided with more context about the task's objectives, the relational functions will be even more accurate. We can as well add this to the appendix.

2. In POMDP setting, would you allow the relational functions to reveal information on previously unobserved properties? If not, how would you plan to handle them?

   Many approaches can be taken. In POMDP, one could indeed maintain a state of beliefs at the relational level as well, from which one could deduce partially occluded properties. For example, an occlusion relation could be utilized, allowing to deduce if an object is likely to be situated behind another one. We are now discussing this advantage in our manuscript. Thank you for the hint!