Incorporating Human Preferences into Interpretable Reinforcement Learning with Tree Policies

Thank you for your thoughtful review! We appreciate that you found the paper to be clear and the algorithm framework to be well designed. We’ll address your comments below:

The authors acknowledge the limitation of using synthetic data and assuming static user preferences, suggesting future work to address this using real-world data and models of evolving preferences. Still, I find it significantly limits the generalizability of the results. Particularly, the authors propose an extremely complicated algorithm, yet only demonstrate it on synthetic simple environments. While the motivation of their method is clear, the complexity of their approach calls for much more involved experiments, to test the scalability of PASTEL on larger and more complex RL environments with higher dimensional feature spaces. I would also strongly recommend adding user studies with actual human participants to validate the effectiveness and user experience of PASTEL.

Thank you for your thoughtful feedback about generalizability. While we chose synthetic data to establish clear ground truth for preferences, we recognize this creates limitations. The synthetic environments allow us to test PASTEL's various components. However, we agree that testing on more complex environments and with real users would provide valuable additional validation. We appreciate this perspective and will consider how to address these limitations in future work.

Real user preferences are likely far more complex, potentially involving non-linear relationships, inconsistencies, and individual variations that are not captured by the simple linear utility model employed with synthetic data. The authors should incorporate user studies with real human participants, evaluating the algorithm's ability to generalize to their utility functions.

While non-linear preferences and inconsistencies can certainly exist, the effectiveness of linear utility models in many real-world recommendation systems [1, 2] suggests they can sometimes capture meaningful preference structures. Our synthetic framework allowed us to validate PASTEL's core mechanisms with known ground truth, which was the primary focus of this study. We will consider how to incorporate various functional forms of preferences in future versions of the work.

The experiments use a relatively small and pre-defined feature set for representing DT policies. A broader or automatically discovered feature set could better capture user preferences.

We would like to clarify that our approach is feature-agnostic: PASTEL can work with any feature generation method, whether pre-defined or automatically discovered. The key is that our framework optimizes over whatever features are provided, without making assumptions about what they represent. One could even leverage language models to generate potential features [3], though that would need validation in these new settings.

The use of an RL agent for initializing the DT policy population has a potential to introduce bias in the initial policy.

We agree that leveraging a pretrained RL agent could bias the initial policy space. We intentionally center the policy space around reward-relevant regions, based on the practical assumption that users typically have some preference for task success, even if the exact tradeoffs are unknown. This initialization approach helps guide exploration toward policies that balance reward optimization with other user preferences, rather than exploring the entire policy space (which could include many impractical solutions).

To clarify: if maximizing reward isn't a relevant feature for a particular application, we can easily modify the initialization process. The VIPER initialization is a design choice, not a requirement of our framework. Our goal is to start with sensible policies and then refine them based on user preferences, rather than exploring policies that might completely disregard task performance.

[1] Jin, Ruoming, et al. "Towards a better understanding of linear models for recommendation." Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021.

[2] Li, Lihong, et al. "A contextual-bandit approach to personalized news article recommendation." Proceedings of the 19th International Conference on World Wide Web. 2010.

[3] Jeong, Daniel P., Zachary C. Lipton, and Pradeep Ravikumar. "Llm-select: Feature selection with large language models." arXiv preprint arXiv:2407.02694 (2024).

Thank you for your thoughtful review! We appreciate that you found the approach to be practical for adapting interpretable structures based on user feedback and the strategies to improve sample efficiency to be clever. We’ll address your comments below:

The paper fails to connect with key frameworks and research in preference optimization, especially in the context of reinforcement learning from human feedback and large language models.

We had initially thought that including a lengthy discussion section on this might be confusing, as it could lead a reader to believe that the techniques and motivation are similar. However, as you point out, more explicitly contrasting with this body of work actually can improve clarity. We are happy to update the draft to include a discussion of LLM fine-tuning. In particular, we will highlight that these methods do not immediately give us preferences over attributes of the policies themselves, which is necessary for our setting.

This narrow scope raises serious questions about the method’s scalability and effectiveness on more complex, real-world problems, and this was not even mentioned in the limitations section. I think a real-world application example where DTs could be effectively used to solve the issue is an important missing point.

We would like to clarify that our work’s novelty is in learning an interpretable policy that adheres to the user’s preference, both the way we do it and the goal as a whole. Using DTs as the policy representation is common in XRL literature [1-3]. We agree they have limitations, and we note that our preference learning elements do not rely on the policy’s form.

That said, we agree that instantiating at least one concrete domain and use case would strengthen the paper. One practical use case is inventory management [4]. As a simple example, an inventory manager might prefer trees where the order of splits more closely aligns with their own mental model of inventory management priorities. Given two trees with similar performance, they would then prefer the policy that more closely aligns with human decision-making.

Mention of Imitation Learning / Inverse Reinforcement Learning... I think it would have also been interesting to compare or have a comment on preference elicitation vs recovering rewards using Inverse Reinforcement Learning.

Yes, this is also something that we had considered as a baseline! The challenge here is as follows. Suppose we leverage VIPER. Then, we can use the estimated preference model to update Q to guide VIPER training. If we have preferences over transitions, then this is easy to do, but if we have preferences over policy attributes, then having this Q would not enable us to e.g., obtain a tree of depth 2 that still achieves 80% optimal reward.

To your point, we can essentially view our framework from the perspective of recovering a preference model to learn a policy (similar-ish to inverse RL). We are happy to add a discussion comparing these approaches to clarify the positioning of the work.

Why an evolutionary algorithm?

We use an evolutionary algorithm due to the non-differentiable structure of decision trees. We considered relaxing the tree structure and using gradient-based approaches, but the discretization step to improve interpretability often incurs a large performance decrease [1].

Decision trees have known limitation in complex / high dimensional data settings, would this apply to complex user feedback too?

To clarify, the decision tree doesn’t model the user preference itself. Instead, the policy is represented as a tree. For the user preference model, we assume linearity in features that are functions of a policy decision tree; however, if the preferences are complex we can use a polynomial basis for the features. Exploring what types of complex preferences users could have in an even more practical setting would be an interesting future direction for an HCI study.

[1] Silva, Andrew, et al. "Optimization methods for interpretable differentiable decision trees applied to reinforcement learning." International conference on artificial intelligence and statistics. PMLR, 2020.

[2] Koirala, Prajwal, and Cody Fleming. "Solving offline reinforcement learning with decision tree regression." 8th Annual Conference on Robot Learning. 2024.

[3] Roth, Aaron M., et al. "Conservative q-improvement: Reinforcement learning for an interpretable decision-tree policy." arXiv preprint arXiv:1907.01180 (2019).

[4] Madeka, Dhruv, et al. "Deep inventory management." arXiv preprint arXiv:2210.03137 (2022).

Thank you for your detailed review! We appreciate that you found the problem setting to be original and clear. We’ll address your comments below:

Motivation Although it seems reasonable to want to have preferences over decision trees policies, I am not convinced that the framing as an active learning problem setting is important and realistic (a strong concrete motivating example is not present), and the synthetic nature of the experiments do not help. Also, see questions. The work seems inspired by RL with human feedback, and in that literature, methods are typically developed using preference datasets that are collected offline in a batch, rather than online.

We agree that methods for RL with human feedback indeed typically leverage offline preference elicitation methods for developing datasets that capture general human preferences. In contrast, we envision that users may have very specific, in-context use cases that would necessitate online and few queries. In this case, it would be impractical to query a user so many times for their preferences. We appreciate the opportunity to clarify how our motivation differs from the more standard RLHF setup, and we will revise the next draft with a more detailed comparison.

Method is entirely heuristic (and evaluations are not realistic) The proposed method is entirely heuristic and a lot of choices are not really justified (see questions section). This, combined with limited experiments, makes it difficult to understand under what settings it could perform well vs not. Also see questions on hyperparameter choice. I think this would be fine if either the problem setting and experiments were very convincing, or if the method is principled, or you can prove its performance.

Thank you for the feedback. We simplified the algorithm significantly to begin studying it theoretically. Specifically, we assume 1) trees are compared using random selection, 2) we maintain only a single preference estimate (rather than an ensemble), 3) we leverage the more standard PL model without the temperature parameter $\beta$, and 4) we only run the EA at the end of the $T$ comparisons to obtain the tree that maximizes $\hat{\theta}$.

In this case, we can use the result that the feedback can be modeled as a GLM feedback model on the decision space of pairwise differences [2], meaning that the results from [3] directly apply under suitable conditions. If we obtain sufficient samples to make our estimation error small, then if we assume that the EA can return the tree that is a maximizer, we will obtain a near-optimal solution.

Of note: different from these two prior works, we only care about the simple regret at time $T$, whereas their algorithms are designed to minimize cumulative regret, so their algorithms are not directly applicable.

Experiments are not realistic The experiments assume that a user's utility is linear in {reward, depth, num leaves, state feature used}, which seems unrealistic. Is there a realistic setting where someone would want a linear tradeoff between reward vs e.g. depth, rather than e.g. put a limit on depth and then maximize reward (which you could do very simply)?

Although linear utility functions are a simplifying assumption, we believe they serve an important purpose. First, linear utility functions are well-established in the contextual bandit literature [2, 4, 5], from which we derive some of our framework. Second, although the function itself is linear, we could transform the feature vector by applying any arbitrary function to the raw input, such as a polynomial basis. Third, the suggestion of using hard constraints (like depth limits) is possible. However, selecting a poor value for the constraint leads to a worse outcome. For example, if a greater depth than necessary is permitted, then the policy will still grow to this greater depth. Likewise, if the depth is set too low, then the policy will perform poorly, and since a deeper policy is not learned, the degree of performance loss will be unknown. Our approach can be seen as learning to select the proper constraints. Fourth, we use the tradeoff between two features for exposition, but we envision even more attributes that users may have preferences over (e.g., the state features tested within the tree policy itself).

Presentation Although the method is reasonably clear from reading the text, there are still many issues with presentation: the notation is inconsistent, and I found Figure 1 difficult to understand, even though it is visually pleasing.

We really appreciate these detailed suggestions! We will update the paper accordingly.

(1/n)

Thank you for your review! We appreciate that you found the framework clear and the motivation persuasive. We’ll address your comments below:

It is assumed that humans can compare two tree policies and give reliable feedback. While I agree that, tree policies are interpretable in the sense that decision rules can be read from visualizations, it is questionable if humans can compare two trees reliably. For example, this requires humans to precisely predict the consequences of any change in single or multiple leafs. On the contrary, in recent literature of PbRL, humans are only require to compare trajectories, which is much less demanding than comparing policies directly.

We agree that these UX/UI concerns are important. We envision that users could either be presented with two trees or only the entries captured by our $d$-dimensional feature vector. The latter option circumvents the concern that users may not be able to detect small changes in the tree, as the downstream impact of this change would be captured in a more succinct manner. Furthermore, we would like to highlight that we assume a Bradley-Terry preference model that already captures the concern of reliability. Specifically, the Bradley-Terry preference model introduces additional preference noise when the items being compared are closer together. This is a common assumption even in recent work in preference learning literature [1, 2]. Any specific suggestions for more realistic theoretical assumptions underpinning the stochastic feedback model would be appreciated.

It is assumed that a d-dimensional feature vector can not only summarize the structure of a tree but also all information required for decision-making. This assumption significantly restricts the applicability of this method.

We would like to emphasize that we are the first to study the problem of aligning interpretable structures with human feedback. As a result, previous work has only assumed that users care about reward under depth constraints [3] or performed human evaluations of interpretability without incorporating this feedback back into the interpretable structure [4]. Both directions neglect capturing potentially complex preference trade-offs that real users may have over various attributes.

We would also like to mention that the assumption that there exists a set of shared attributes that describe sets of items is standard and adopted from literature on contextual bandits for potentially infinite decision spaces [5].​​ The benefit of this approach is that converging to the best item in a set of size $K$ traditionally requires the explicit maintenance of a $K^2$ matrix, which is cumbersome and unnecessary if there is some structured context that is shared among all items.

That said, we appreciate the comment and will consider how to demonstrate the potential cost of misspecification in future work.

In addition, though interpretability is considered as the motivation of this paper, there is no qualitative results on the extent of interpretability of learned policies.

Thanks for the suggestion! We are happy to provide tree visualizations in future drafts.

Can this method be applied to more complex domains, such as the Mujoco tasks (Ant, HalfCheetah, or Humanoid)?

Yes, this method could be applied to more complex domains. To handle the continuous action spaces of these domains, the policy trees would be regression trees (rather than classification trees). The other elements are compatible. These environments naturally admit safety-based preferences & trade-offs that could be directly incorporated into our method (e.g., joints should not bend more than X amount to avoid damaging the robot).

(1/2)