Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneity

Thank you for your thoughtful and in-depth feedback. We are encouraged that you find the problem setting important, and the proposed method sensible with theoretical and empirical justification. We address your questions below.

The empirical evaluation is limited to simple domains and it is not discussed why that is the case...

Please see the General Response for our detailed answer.

Comparison to ExPLORe: if there is a way to use the same actor-critic algorithm for the proposed method ...

Thank you for mentioning this interesting point. As we have discussed in the paper, we agree with you that some of the results might be due to the difference between the base actor-critic method in ExPLORe and the bootstrapped DQN method used in our approach. However, note that in Table 1, on the left side (Fixed # Hazard = 9), under $\beta = 0.1$, the ExPLORe method achieves near-optimal results while our approaches suffer large regrets. This particular example shows that, at least in the Frozen Lake environment, the baseline actor-critic method is superior to the baseline bootstrapped DQN since the expert data is relatively uninformative for $\beta = 0.1$, i.e., the taken actions are almost random. As we increase the value of $\beta$ and the expert data gets more informative, our proposed approaches outperform the naive bootstrapped DQN, showing that the main performance comes from the proposed method of learning the priors and not the base bootstrapped DQN algorithm. We leave the variant of ExPerior that relies on actor-critic methods to future work.

ADVISOR [1] targets the same problem setting. Differences to that algorithm should be discussed ...

The main difference to ADVISOR is that they assume having access to the expert policy $\pi^{\text{teach}}$, i.e., they have a human-in-the-loop setting during training and can query the expert for new states. On the other hand, our approach only assumes a finite offline data from the expert trajectories. In other words, there is no expert demonstration for new states that the learner can encounter. We will elaborate on the difference in the revised version. Please see our General Response for the high-dimensional domains.

Figure 4: (1) Explain the pane meanings explicitly in the caption. (2) Explain the axes explicitly in the caption. I.e., are these learning curves?

Thank you for the suggestions. We will include the meaning of each panel (low-entropy to high-entropy setting) in the caption. The y-axis refers to the expected reward for some fixed number of evaluation episodes at each training episode $t \leq T$. We will include that in the caption as well.

Figure 4: How come the naive baseline does worse in the low entropy setting than in the high entropy one?

This is an astute observation. The reason is that as the entropy of the tasks gets higher, i.e., the goal location gets uniformly distributed over all the columns, the task will intrinsically get easier. Since the agent always starts from the top-left corner, it is much harder for it to explore and reach the bottom-right corner compared to bottom-middle or bottom-left cells. This is why, on average, the naive baseline gets higher rewards in the high-entropy case compared to the low-entropy setting (the goal column always being in the right corner).

Q. Why is human-in-the-loop highlighted as a limitation?

By human-in-the-loop, we mean the requirement of the algorithm to query experts during training or inference. Our approach, on the other hand, only requires an offline dataset of expert demonstrations, which is a strictly simpler requirement. It might be difficult to access an online expert (e.g., a clinical doctor) during training. This is what we mean by a limitation. Having said that, there are benefits in human-centric AI and looping humans in decision-making that we haven’t discussed in our paper. We will elaborate on this point in the paper’s Conclusion.

Q. Can you give a more fleshed out example of how the problem of unobserved contextual information might appear in a realistic problem?

In clinical applications, the objective descriptions of a patient are often documented through electronic health records (EHR). However, the subjective opinion of a doctor after having conversations with the patient and observing their visual clues or other information, such as their lifestyle, might lead to potentially different diagnoses. Even by adding clinical notes to the EHR data, it is fundamentally challenging to capture all the observations made by the doctors that influence their decisions. Another example is large language models, which are trained on datasets that do not capture contextual information. For example, suppose we ask an expert about the location of a historical building. If the expert remembers that as part of her internal knowledge, she will answer immediately. Training LLMs on such datasets might result in hallucination as the model does not capture the contextual information – When asked about the location of another historical building, the model might think like an expert and generate some response based on their internal knowledge, which can be wrong. On the other hand, what you might expect from such a model is to search on the internet to respond. Please See [1] and [2] (Figure 7) for more detail.

Q. Can you lay out an idea on how to scale the method?

Please see the General Response for our detailed answer.

We hope our response addressed your comments. If you have any additional questions that could help you improve your evaluation of the work, please feel free to let us know.

===================================

[1] Johnson, Daniel D., et al. "Experts Don't Cheat: Learning What You Don't Know By Predicting Pairs." arXiv preprint arXiv:2402.08733 (2024).

[2] Wang, Kuan, et al. "Adapting LLM Agents with Universal Feedback in Communication." ICML 2024 Workshop on Foundation Models in the Wild.

Thank you for your insightful feedback. We are pleased that you consider the paper well-motivated, relevant, and with a conceptually easy-to-understand approach. We have addressed your questions below.

Notation: even with the notation paragraph, I find it quite difficult to follow...?

We assume the states are partitioned by the horizon $[H]$, i.e., each state $s \in \mathcal{S}$ is only reachable in one specific horizon $h \in [H]$. This can be achieved by re-defining the state space as $\mathcal{S} \times H$. This way, none of the value functions or Q-functions need to depend on the horizon. The superscript $t$, however, corresponds to the episode number and not the horizon. The value/Q-functions are defined independently of the episode. The only variable that gets updated over episodes is the learned policy $\pi^t$. Also, we agree that $\mathcal{C}$ should be part of the definition of the MDP. We will include these comments in the notation page of the revised manuscript, as there will be more space (10 pages instead of 9).

Terminology: I'm a little confused with the terminology "parametric" vs "non-parametric"...

Looking at the max-entropy expert prior in Proposition 1 (line 189), you can see that. $\mu\_{\text{ME}}(c) = \lim\_{k \to \infty} \frac{\exp \left( \mathbf{m}(c)^\top \boldsymbol{\alpha}_{k} \right)}{\mathbb{E}\_{c'\sim \mu\_{0}} \left[ \exp \left( \mathbf{m}(c')^\top \boldsymbol{\alpha}\_{k} \right) \right]}$ where each $\boldsymbol{\alpha}\_k$ is an $N$-dimensional parameter with $N$ being the number of demonstrations. In other words, the length of parameters required to model the max-entropy prior increases by the number of samples. This is what we mean by a "non-parametric" approach, in contrast to "parametric" models with a fixed set of learnable parameters. Moreover, we do not assume any specific form for the family of distributions in $\mathcal{P}(\epsilon)$, except belonging to $L^1(\mathcal{C}, \mu\_0)$ defined in lines 577-583.

Typo: in figure 1, annotation below "Step 3" is repeated, I guess one should be \mu_{\theta^\star}.

This is a mistake, and we will fix it in the revised version. Thank you for pointing this out.

Figure 1: clarity can be improved if you write down what is being learnt and what is being updated.

$\mu\_{\theta^\star}$, $\mu\_{\text{ME}}$ are learned from data. The posterior distributions $\mu\_{\theta^\star}(\cdot | \text{history})$, $\mu\_{\text{ME}}(\cdot | \text{history})$ are being updated, and the policy $\pi\_c(\cdot | s)$ is learned/calculated given the sampled Q-functions. We will differentiate between those with different colours in Figure 1.

I found the algorithm box quite useful, will be great to have it in the main paper (e.g. next to Figure 1).

We moved algorithm boxes 1 and 2 to the appendix solely for the lack of space. If the paper gets accepted, we will include them in the main paper. (The camera-ready version allows for one more page.)

Clearly, if $\mu\_0 \in \mathcal{P}(\epsilon)$, then we get a trivial solution, $\mu\_{\text{ME}} = \mu\_0$ (usually Uniform) ...?

Your assessment is correct. If the uniform distribution is already in the set of feasible prior functions $\mathcal{P}(\epsilon)$, then $\mu\_{\text{ME}} = \mu\_0$. However, this is not a trivial solution since the uniform distribution will maximize the marginal likelihood of the data and is a plausible distribution. This scenario can happen if the effect of unobserved confounding is extreme, i.e., the unobserved factors can change the optimal policies arbitrarily and provide no information on the task distribution. For example, consider a multi-armed bandit setting with $K$ actions, where there are $K$ task types, each with a different arm as the optimal action. The expert demonstration data will contain each of the arms with equal probability. In other words, the expert data provides no information, and the best thing we can do is to have a non-informative prior like the uniform prior distribution.

If $\mathcal{C}$ is unbounded, then you can't really define a uniform distribution on it ...? Can you handle improper priors?

Note that our result does not force $\mu\_0$ to be a uniform distribution or $\mathcal{C}$ to be bounded. It only requires $\mathcal{C}$ to be a measurable set with an existing measure $\mu\_0$. In the case of unbounded $\mathcal{C}$, one can choose $\mu\_0$ to be a Gaussian distribution. However, our current results, like Lemma 5 on page 14, require the priors to be in $L^1\left(\mathcal{C}, \mu\_0\right)$, i.e., $\mathcal{C}$ should have a finite measure under the feasible priors. For that reason, the current theoretical results do not handle improper priors.

Q. Do you have an intuition on how badly "misspecified" the learnt expert prior can get ...?

This is the exact reason we have provided the non-parametric max-entropy approach. Table 3 (Page 22) shows that misspecified priors result in large regrets. In cases where the practitioner is not quite sure about the parametric form of the prior, we suggest using the non-parametric max-entropy approach.

Q. For the "parametric" ExPrior - why score gradients? Have you thought about or tried continuous relaxation when the problem is discrete?

In the parametric ExPerior, we do not assume the parameter set $\boldsymbol{\Theta}$ is necessarily continuous, as long as we can solve the optimization in line 175 (potentially without gradient-based methods).

Q. On line 112 "Note that since the task variable is unobservable ...

We meant explicit dependence here. The learner policy only depends on the history of interactions and the current state. As you mentioned, one can use the history to infer/integrate the task distribution. We will clarify more on this.

We hope our comment addresses your questions. Please let us know if we can answer any further questions that might improve your assessment of the work.

Thank you for your thoughtful feedback. We are encouraged that you find the paper to have a realistic and significant setting and sufficient theoretical and experimental results. We address your questions below.

Just as stated at the end of the paper, the experiments conducted in the paper are limited, and the experiments under MDPs even made the assumption that the transition functions are invariant to the task variables, which makes the experimental environment overly simple. In more difficult experimental benchmarks such as MuJoCo, it is questionable whether the proposed method can show the expected effect.

We agree that the current experiments do not consider complex high-dimensional settings. Please see the General Response we provided for our detailed plan to scale our approach and include such experiments in the revised version.

Due to the deficiency mentioned in the first point just now, the paper also cannot further compare with modern offline meta RL algorithms (such as PEARL, FOCAL, CORRO) under complex MDPs, making the baselines seem a bit scarce.

Thank you for mentioning the works FOCAL [1] and CORRO [2]. We will cite these in the related work section of the revised paper. However, independent of the complexity of the MDPs, the mentioned baselines are not immediately applicable for our problem setup. PEARL, FOCAL, and CORRO all require reward signals to be available in the offline data. Moreover, FOCAL and CORRO assume the offline data is diverse enough to learn the optimal policies solely based on the offline data and do not continue the learning during the online phase. On the other hand, our setting only assumes offline expert demonstration data (with no reward) and requires using the reward signals during the online phase. This is why we did not include those baselines in our experiments, rather than the complexity of the MDPs.

The introduction of related concepts in the paper is slightly insufficient. Besides, there is a clerical error in the upper right part of Fig. 1: the first \mu_{\rm{ME}} should be \mu_{\theta^*}.

Could you please elaborate on which parts of the introduced concepts are slightly insufficient so we can explain them in more detail? Also, thank you for finding the typo in $\mu_{\rm{ME}}$. We will fix it in the updated version.

Q: What does "Heterogeneity" mentioned in the paper represent? Does it refer to the differences between the transition functions and reward functions in MDPs? The paper should explain this.

Yes, your understanding is accurate. We will elaborate more on what we mean by "Heterogeneity" in the introduction.

Q: The paper mentions that the off-policy meta RL method PEARL requires task labels. Does the task label here refer to the reward label? In fact, the trajectories required by PEARL only contain (s, a, r, s') information.

Thank you for pointing this out. This confusion is probably due to the paper's phrasing. Quoting the paper (lines 85-86): "Similarly, Zhou et al. [40] and Rakelly et al. [41] require the task label and reward labels.". This should have instead been, "Similarly, Zhou et al. [40] and Rakelly et al. [41] require the task label and reward labels, respectively." PEARL requires offline reward information, which is not accessible in our setting. We will fix this confusion in the revised version.

We hope our response addressed your comments. Should you have any additional questions that could assist in improving your evaluation of the work, please feel free to let us know.

===================================

[1] Li, Lanqing, Rui Yang, and Dijun Luo. "Focal: Efficient fully-offline meta-reinforcement learning via distance metric learning and behavior regularization." arXiv preprint arXiv:2010.01112 (2020).

[2] Yuan, Haoqi, and Zongqing Lu. "Robust task representations for offline meta-reinforcement learning via contrastive learning." International Conference on Machine Learning. PMLR, 2022.

Thank you so much for your helpful comments. We are pleased that you find the paper clearly written, well-organized, and with comprehensive simulations. We address your questions in the following.

The proposed method requires the expert to follow a specific form of policy to generate the demonstration data. This parametric restriction is rather strong and may not necessarily hold in many applications.

We agree with you that Assumption 1, about the expert policy being noisily rational, may not hold in practice. Appendix C.1, Table 2 on Page 22 considers misspecified expert models. In particular, we looked into “optimal” experts, “noisily rational” experts with misspecified values of $\beta$, and a new type of experts we called “random-optimal,” who take completely random actions with some fixed probability and optimal actions otherwise. In all instances, our methods, which rely on the noisily rational experts, achieve near-optimal results. This observation justifies the use of Assumption 1, even in instances where it does not necessarily hold. We would appreciate it if the reviewer has any suggestions for including other types of misspecification in Table 2.

It is unclear to see from Theorem 2 under which condition the expert's demonstration could accelerate the online learning process. It would be appreciated if the authors could further elaborate on this point.

As discussed in lines 302-312, as well as Figure 3 (b), the extra term in the regret bound of Theorem 2, i.e., $\sum\_{a, a’ \in \mathcal{A}, a \neq a’} \sqrt{\frac{p(a)}{p(a) + p(a’)} \left(1 - \frac{p(a)}{p(a) + p(a’)}\right)} \left[\sqrt{p(a)} + \sqrt{p(a’)}\right]$ closely resembles the variance (or the entropy in the case of $K = 2$) of the optimal actions under the distribution of unobserved factors. In other words, if the variance of the optimal actions is small, i.e., the effect of unobserved factors is negligible, the regret will be close to zero. We will elaborate more on this point in the revised version of the paper.

Some related references are missing. Extrapolating knowledge from confounded data is one of the main problems in causal inference. There is a growing line of work studying learning from confounded demonstration data in canonical RL tasks ...

Thank you for providing the missing related references. We will include those papers in the revised manuscript. We emphasize that the first work, Kallus et al. (2018), only considers the offline setting, while the latter assumes the availability of rewards in the offline observational data and does not assume the demonstrations are generated by experts. Our work, on the other hand, integrates offline expert demonstration data with online reinforcement learning.

Q: Could the regret bound in Theorem 2 outperform standard bandit regret when the expert's demonstration is benign? Could the authors elaborate on this?

Yes, as discussed above and in lines 302-312 of the manuscript, the standard bandit regret is bounded by $\tilde{\mathcal{O}}\left(KT\right)$ for $K$ number of actions and $T$ episodes. The bound in Theorem 2, however, is $\tilde{\mathcal{O}}\left(\sqrt{T} \cdot \underbrace{\sum\_{a, a’ \in \mathcal{A}, a \neq a’} \sqrt{\frac{p(a)}{p(a) + p(a’)} \left(1 - \frac{p(a)}{p(a) + p(a’)}\right)} \left[\sqrt{p(a)} + \sqrt{p(a’)}\right]}\_{\text{term} I}\right)$ When the expert’s demonstration is benign, e.g., the variance of the optimal actions under unobserved confounding is small, the second term (term I) will be smaller than $\sqrt{K}$. As an example, consider a $K$-armed case, where a given fixed arm $a_1$ is optimal regardless of the unobserved heterogeneity. Then, for all $a \neq a_1$, we have $p(a) = 0$, resulting in term I to be zero. This is a case when the expert data is extremely informative and hence gives us zero regret.

We hope our comment addresses your questions. Please let us know if we can answer any further questions that might improve your assessment of the work.

1. The big-O notation $\tilde{\mathcal{O}}$ in this comment ignores the log terms.