ExID: Offline RL with Intuitive Expert Insights in Limited-Data Settings

Thank you for your valuable feedback. Please find our response below:

W1: Discrete Action Space Limitation

• We conducted experiments on discrete action space domains as many important real world problems use discrete action policies. For example navigation with actions forward or turning, finance trading with actions buy, hold or sell etc. We have also mentioned the use of discrete policies in our limitations.
• However, we would like to highlight the proposed methodology can be extended to any continuous domain problem by using the regularization in Eq 4 during critic $(Q_s^\theta)$ training for continuous domain and using actions from actor network $(\pi_s)$ for cross entropy loss in Eq 7.
• We empirically demonstrate this by conducting additional experiment by training policy on continuous version of sales promotion task (action range coupon (0 - 5), discount (0.6 - 1) ) and continuous Type-1 Diabetes basal bolus control task (action range basal and bolus (-1 to 1) (results in Q3).

W2 and Q4: Hyperparameter Tuning Challenge

• We agree hyperparameter tuning in offline setting is challenging. In practice this can be done by using tuning methods like Bayesian Optimization and observing the behavior of Q values as proposed by [2].
• $k$ and $\lambda$ should be selected with peak Q value observed during training. We discuss the effect of different $k$ and $\lambda$ in Fig 11 and Fig 12 (supplement) and empirically show $\lambda$ = 0.5 and $k$ = 30 works well in most setting. We also show the proposed methodology is better than $\lambda$ = 0 and $k = 0$ demonstrating robustness to hyperparameters.
• The Q based tuning does not require additional computational resource.

[2] Kumar, A., Singh, A., Tian, S., Finn, C., & Levine, S. (2021). A workflow for offline model-free robotic reinforcement learning. arXiv preprint arXiv:2109.10813.

W3: Strong experiment comparisons

• The paper compares ExID to SOTA discrete offline RL methods. However continuous control algorithms like SCQ, ReDS, A2PR, CPED also suffer from performance degradation for OOD state. We empirically compare EXID to SCQ for SalesPromotion continuous task. Please find the results in table of W1.
• The suggested offline RL methods use action constraints to correct OOD actions for states in dataset but do not have any mechanism for states unseen in dataset. It has been recently established generalization is the main bottleneck of offline RL [3].
• ExID addresses this generalization gap by using knowledge distillation and teacher network updates to correct actions for states not seen during training. We will add this discussion in our revised manuscript.

[3] Park, Seohong, et al. "Is Value Learning Really the Main Bottleneck in Offline RL?." arXiv preprint arXiv:2406.09329 (2024).

Q1: Deviation and Recovery from incorrect Domain Knowledge

• In practice it is never possible to obtain perfect domain knowledge. Thus the algorithm updates the initial teacher network ($\pi_t^\omega$) obtained from domain knowledge using Eq 5, 6 and 7 during training.
• Thus ExID uses a self correction methodology to find the optimal policy even when there is significant deviation from the domain knowledge. Please refer to Pg 5 for details of the teacher update.
• The domain knowledge used for the experiments is also not optimal as reported in performance of behavior cloned teacher under the column $\mathcal{D}$ of Table 1 and 2.

Q2: Robustness to errors and biases in domain knowledge.

• The self correction methodology provides robustness to the errors and biases in domain knowledge. We discuss the Effect of varying $\mathcal{D}$ quality in Sec 5.6. Fig 6 a shows using imperfect knowledge (Rule 3) and knowledge with high error (Rule 2) with ExID can bring substantial performance improvement over baseline.
• We agree using absolutely incorrect domain knowledge will cause performance degradation as shown in Fig 6 a and mentioned in limitations. However, domain knowledge is available for practical problems in domains like business and healthcare.

Q3: Other real world scenarios

We conducted experiments on 6 benchmark and open source datasets for reproducibility. The method can be applied to many real problems in business, healthcare where domain knowledge is available. However such datasets are often proprietary. If the reviewer could kindly suggest any other opensource real world dataset we would be happy to conduct experiments. Additionally we tested ExID for Type 1 diabetes control task by administrating basal and bolus insulin level [2]. This is a continuous control task

• The following basic basal bolus insulin control is known for diabetic patients

Domain knowledge :

1. basal = u2ss * BW / 6000 where u2ss: The patient's or default steady-state insulin. BW: The patient's or default body weight.
2. meal > 0 $\implies$ bolus = (carbohydrate / carbohydrate_ratio) + (current_glucose - target_glucose) / correction_factor)/ sample_time

The offline data is obtained from open access NeoRL2 github repository.

[2]. Jinyu Xie. Simglucose v0.2.1 (2018) [Online].

Q4: Hyperparameter selection

Please refer to our response of weakness 1.

Limitation : Dependence on high quality domain knowledge

We would like to highlight the domain knowledge used in this work is suboptimal (obtained from heuristic) and incomplete (covering only part of state space). This can be observed in Table 1 and 2 under $\mathcal{D}$ where performance of the initial domain knowledge is not optimal. Heuristic based domain knowledge is available for many real world use cases for business, healthcare etc.

Thank you for your valuable feedback. Please find our response below:

W1: Limited novelty and comparison with traditional knowledge distillation

• The domain knowledge considered in our setting is imperfect and is updated using expected improvement of RL policy in a completely offline manner using Eq 6 and 7 which is different from traditional knowledge distillation methods. This update is essential as shown by the ablation study in Fig 5 c.
• The study of performance degradation of offline RL due to limited data is unexplored in literature as agreed upon by Reviewer Cy8u. To the best of our knowledge, a knowledge distillation based regularization approach has not been studied for Out of Distribution (OOD) action correction for unseen states in dataset (OOD states) in offline RL prior to this work.
• Please refer to discussion in Pg 18 Section E "Related Work: Knowledge Distillation" for comparison with prevalent knowledge distillation methods in RL.

W2 : Offline data from domain knowledge, empirical studies on more real-world datasets or rigorous theoretical analysis

• In practice offline data generally comes from human demonstrations. Since it is costly to explore all possible states during data collection, the offline data distribution is often narrow. The policy learned on such demonstrations fails on encountering OOD states. Please refer to Pg 16 Fig 8 for pictorial representation of the complementarity of domain knowledge to the dataset.
• The policy performance is improved over these OOD states via incorporation of domain knowledge. Proposition 4.2 formalizes this notion with respect state coverage and $\mathcal{D}$ quality theoretically establishing performance improvement is possible as highlighted by reviewer Cy8u. Please refer to pg 13 App A for the full proof.
• We conducted experiments on 6 standard datasets. The results of Minigrid environment are in Table 3 pg 19 due to page limitation in the main manuscript. Additionally we tested ExID for diabetes management task by administrating basal and bolus insulin level [2]. This is a continuous control task.
• The following basic basal bolus insulin control is known for diabetic patients

Domain knowledge :

1. The basal insulin is based on the insulin amount to keep the blood glucose in the steady state when there is no (meal) disturbance.

basal = u2ss (pmol/(L*kg)) * body_weight (kg) / 6000 (U/min)

2. The bolus amount is computed based on the current glucose level, the target glucose level, the patient's correction factor and the patient’s carbohydrate ratio.

bolus = ((carbohydrate / carbohydrate_ratio) + (current_glucose - target_glucose) / correction_factor)/ sample_time

The offline data is obtained from open access NeoRL2 github repository.

• It has been recently established generalization is the main bottleneck of offline RL [1]. ExID addresses this generalization gap by using knowledge distillation and teacher network updates to correct actions for states not seen during training as shown in Fig 4 Pg 8.

[1] Park, Seohong, et al. "Is Value Learning Really the Main Bottleneck in Offline RL?." arXiv preprint arXiv:2406.09329 (2024). [2]. Jinyu Xie. Simglucose v0.2.1 (2018) [Online].

W3: Visualization on Mountain Car dataset

As the state space in mountain car is only two dimensional and the entire dataset is available it is easier to visualize the performance degradation for OOD states. The state space of sales promotion is 40000 (4 features for 10,000 users). Hence visualization of SP task is difficult. We show the performance improvement for Sales Promotion over baseline in Fig 3c Pg 7.

Definition 4.1

We define $\mathcal{B}_r$ as with respect to full dataset and not just states space because $\mathcal{B}_r$ also contains actions, rewards and next states. A performance drop is a result of both the conditions in Def 4.1 please refer to our analysis in Appendix B Pg 16.

Definition of $\eta$

Due to page limitations we provide the definition $\eta$ in Appendix A Pg 13. For any deterministic policy $\pi$ the performance return is formulated as $\eta(\pi) = E_{\tau \sim \pi}[\sum_{t=0}^{\infty} \gamma^t r(s_t, a_t)]$ following [3]. Where $\tau$ is trajectory, $\gamma$ is the discount factor and $r$ is the reward function. We will move the definition to the main text in revised manuscript.

[3] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In 347 Proceedings of the Nineteenth International Conference on Machine Learning, pages 267–274, 2002

Definition of domain knowledge

Domain knowledge $\mathcal{D}$ is formally defined as hierarchical decision nodes capturing $S \to A$ where $A$ is non optimal as represented by Eq. 2 Pg 3. This can be generalized to any source of knowledge capturing $S \to A$ mapping for example dataset without reward or next state labels. General guidelines are typically available in practical domains like business, healthcare and autonomous driving which can be distilled in teacher network. ExID will work with any type of domain knowledge that can be represented as S → A mapping.

Difference between the proposal and the traditional model distillation

Please refer to our response for W1.

Dear reviewer, Can you please confirm that you've read the author's responses? Given that the other review is now both vote for acceptance, it's important for you to voice any remaining concerns if you still believe the paper should not be accepted.

Thank you!

Thank you for your appreciative and constructive feedback. Please find our response below:

W1 Generalization to Continuous Domain

• The proposed methodology can be extended to any continuous domain problem by using the regularization in Eq 4 : $\mathcal{L}(\theta) = \mathcal{L}cql(\theta) + \lambda E_{s \sim \mathcal{B}r \land s \models \mathcal{D}} [Q_s^\theta(s, a_s)-Q_s^\theta(s,a_t)]^2$ during critic $(Q_s^\theta)$ training for continuous domain and using actions from actor network $(\pi_s)$ for cross entropy loss in Eq 7 : $\mathcal{L}(\omega) = -\sum_{s \models D} (\pi_t^\omega(s)log(\pi_s(s)))$.
• We empirically demonstrate this by conducting additional experiment by training policy on continuous version of sales promotion task (action range coupon (0 - 5), discount (0.6 - 1) ) and continuous Type-1 Diabetes basal bolus control task (action range basal and bolus (-1 to 1) for SimGlucose dataset. Please refer to the results in table below:

W2 Scalability

EXID can be scaled for complex domain knowledge trees if the teacher neural network is able to capture the $S \to A$ mapping of the tree. However the effect of complexity on the teacher network is currently beyond the scope of this paper.

Q1 Comparative Analysis

• DKQ uses Q operator guided by domain knowledge using Eq: $\mathcal{T}{\mathcal{F}} Q(s, a) := r(s, a) + \gamma E{s' \sim P(s'|s, a)} \left[ \sum_{i=1}^{K} \alpha_i \max_{a' \in \text{supp}(f_i)} Q(s', a') \right]$ which requires the importance of all actions $a' \in \text{supp}(f_i)$ to be labelled from domain knowledge. The domain knowledge is not updated during the training process. This operator also only works on the states observed in the dataset. Contrary to this our method does not require action support labels for each action and incorporates the domain knowledge for unseen states through knowledge distillation and teacher update.
• CQL SE uses an uncertainty-weighted regularization of OOD actions using safety experts represented by $Q(s, a) = r + \gamma \ast \max_{a'} Q(s', a') - \underbrace{(1 - conf(s)) \ast (a - \pi_T(s))^2}_{\text{uncertainty weighted learning from the safety expert}}$. The safety expert is considered to be optimal and not updated during training. conf(s) is calculated based on states observed in the dataset and this method does not account for unseen states.

Q2 Examples of Domain Knowledge Improvement

Since the domain knowledge is partial, direct improvement of $\pi_t^\omega$ is not always observed. We show domain knowledge improvement using ablation study in Fig 5c which depicts no teacher update leads to a suboptimal policy. This is also depicted using baseline CQL+D where simply using domain knowledge without improving it does not lead to an optimal performance.