ExID: Offline RL with Intuitive Expert Insights in Limited-Data Settings

W1: Literature review is insufficient

• We have made efforts to cover the most well-known methods in offline RL, as detailed in Section 2 of the main text. Additionally, we have provided comparisons with recent offline RL works, such as SCQ, A2PR, and CPED, in Sections D and F of the Appendix.
• If the reviewer could kindly point out any relevant references that we may have missed, we would be happy to include them in the revised manuscript.

W2: Stringent requirements for the domain knowledge input

• As the reviewer correctly points out, "in many real-world tasks, we might only be able to write very few heuristic rules for a task, which can only sparsely cover the state-action space." This is precisely the problem setup addressed in this manuscript.
• For example in the sales promotion task in line 398 the domain knowledge only covers states with order_number > 60 and avg_fee > 0.8. Also the rules considered in experiments are imperfect and updated during the offline policy training which extends applicability to many domains where only partial heuristic rules are available.
• Beyond the six standard discrete Gym environment tasks, the mythology is also validated on human generated data from business domain (Sales Promotion) and healthcare task (Simglucose). These tasks are not simple and exhibit intricacies of real world problems. Please refer to [1] and [2] for the description of these tasks.

[1] Simglucose: Jinyu Xie. Simglucose v0.2.1 (2018) [Online]. Available: https://github.com/jxx123/simglucose.

[2] Qin, Rong-Jun, et al. "NeoRL: A near real-world benchmark for offline reinforcement learning." Advances in Neural Information Processing Systems 35 (2022): 24753-24765.

W3: Problem induced by random actions for states not covered by domain knowledge

This is not problematic as Q values are only regularized when the states match the domain knowledge as depicted in Eq 3 $\mathcal{L}_r(\theta) = \underbrace{\mathbb{E}_{s \sim \mathcal{B}_r \land s \models \mathcal{D}}}_{\text{states matching domain rule}} \underbrace{[Q_s^\theta(s, a_s)-Q_s^\theta(s,a_t)]^2}_{\text{Q regularizer}}$.Random actions are merely placeholders in the behavior cloning (BC) policy during training from domain rules and do not affect the Q-value regularization process.

W4: The algorithm is essentially a combination of CQL and UWAC

• We respectfully disagree with this point. In practice, it is rarely possible to obtain perfect domain knowledge. Therefore, our algorithm updates the initial teacher network ($\pi_t^\omega$) domain knowledge using Eq 5, 6 and 7 during training. This mechanism of knowledge distillation has not been studied before for offline RL.
• The proposed algorithm deviates from standard CQL or UWAC, which primarily promote conservatism for out-of-distribution (OOD) actions in states observed within the dataset. The key contribution of ExID lies in distilling knowledge for OOD actions in states that rarely or never appear in the dataset, addressing a critical gap that existing methods do not tackle.

W5: Propositions 4.2 and 4.3

• We acknowledge that combining CQL with uncertainty-based methods complicates theoretical guarantees, and we agree that certain assumptions in Propositions 4.2 lacks direct support from the algorithm. These assumptions were intended as idealizations to simplify the analysis and provide preliminary insights into how reasonable domain knowledge regularization can lead to performance improvements. They are not intended to be the major contribution of the paper.
• In response to your concerns, we have moved proposition 4.2 completely to the appendix in the revised manuscript and have explicitly labeled the assumptions as simplifying in line 273. We have also removed proposition 4.3. We hope this adjustment provides a more transparent and grounded presentation of our results.

W6: There are also some inconsistencies in the symbols used in the paper

Thank you for pointing these out. We have corrected in revision.

Q1: How can the method be used if only limited domain knowledge rules are provided?

Please refer to our response for W2.

Q2: Is it possible to not use random action for cases that are uncovered by domain knowledge rules?

Please refer to our response for W3.

Thank you for your valuable feedback. Please find our response below:

W1: Unconvincing motivation and contributions:

• The motivation for this work is rooted in real-world use cases where heuristic-based domain knowledge is readily available but offline data is limited. This scenario applies to many practical problems, such as resource allocation in business, healthcare, and autonomous driving, where domain-specific heuristics are common. The limited attention given to this problem setup provides a strong motivation for research in this direction.

• To demonstrate the practical effectiveness of our method, we validated it not only on six standard discrete Gym environment tasks but also human generated data from business domain (Sales Promotion) and healthcare task (Simglucose). These tasks are not simple and exhibit intricacies of real world problems. Please refer to [1] and [2] for the description of these tasks.

• We have shown comparison with existing model free RL baselines that use domain knowledge like CQL with safey expert (CQL-SE) and ensemble of offline RL methods and domain knowledge.

• For continuous domain tasks we have also compared with recent offline RL baselines SCQ, A2PR, CPED (Section D). We will be happy to compare with any other baseline that the reviewer might suggest.

[1] Simglucose: Jinyu Xie. Simglucose v0.2.1 (2018) [Online]. Available: https://github.com/jxx123/simglucose.

[2] Qin, Rong-Jun, et al. “NeoRL: A near real-world benchmark for offline reinforcement learning.” Advances in Neural Information Processing Systems 35 (2022): 24753-24765.

W2 : Other presentation issues

• We have restructured parts of the manuscript to make the presentation better. Please find the summary in the global comment. We are happy to revise for any additional suggestion the reviewer might have.
• Thank you for suggesting citep we have used this for the updated manuscript.
• In Def 4.1 The first condition discusses about missing states. This means that there are states $s$ in $B$ for which no transitions $(s,a,s')$ exists in $B_r$.
• The second conditions denotes reduced sampled for some transition. This means that while the transition $(s,a,s')$ exists in both $B$ and $B_r$, there are fewer instances of it in $B_r$,compared to $B$. This is not guaranteed to hold given the first condition. For example let B contain 10 sample of t1: (s_1,a_1,s_2) and 5 samples of t2: (s_1,a_2,s_3) just removing s_2 from $B_r$ does not denote reducing samples for t2.

W3 : Inadequate treatment of model-based RL

• The proposed method is specifically designed to address the generalization gap of offline model-free RL methods in limited data settings. In this context, a direct comparison with different model-based RL methods may not be entirely fair to the proposed approach.
• Both model-free and model-based RL have their respective caveats. As discussed in the original MOPO paper, dynamics inaccuracies in poorly covered regions of the state space can still lead to performance degradation, even when uncertainty penalties are applied. The paper highlights that substantial model inaccuracies, particularly in out-of-distribution (OOD) regions, can result in performance losses due to compounding errors in hypothetical rollouts. Given the limited data setting we address, learning an accurate dynamics model is inherently challenging. The paper does not make claims beyond this limitation. The validity of this claim is empirically supported by results on the human-generated Sales Promotion (SP) dataset.
• Regarding tuning of baselines, all baselines have been taken from standard implementations associated with the respective papers. We cited the sources in the manuscript, and further details can be found in our code, which is provided in the supplementary material.

Q1. Are there more modern offline RL methods which ought to be considered?

We have compared the discrete tasks with existing discrete offline RL algorithms and the continuous tasks with recent algorithms, such as SCQ, A2PR, CPED. If the reviewer has suggestions for additional relevant baselines, we would be happy to include a comparative analysis in future revisions.

Q2. Can your approach be viewed as a form of actor-critic? Would an actor-critic method be more performant?

• Only inclusion of an actor critic network does not improve performance in limited data setting. The continuous domain tasks in our study utilize actor-critic networks, and Figure 3c illustrates the performance gap.

Thank you for your valuable feedback. Please find our response below:

W1: The presentation and Eq. 2.

• The buffer B contains tuples (s, a, s', r) representing state, action, next state, and reward, as is standard in offline RL buffers. The first point states that some states s' the exist in B are not present in reduced buffer $B_r$. We have revised the explanation of of Eq 2 in the manuscript providing clarity for Def 4.1.

W2: Use of simple environments

• We would like to emphasize the proposal is validate on two real world task sales promotion (business domain) and Simglucose (healthcare) tasks. These tasks are not simple and exhibit intricacies of real world problems. Please refer to [1] and [2] for the description of these tasks.

• While we acknowledge that our methodology relies on domain knowledge—which may be challenging to design for high-dimensional robotics tasks such as those in d4rl (e.g., HalfCheetah)—domain knowledge is often more accessible in business and healthcare contexts. We believe that this type of problem setup deserves research attention, as offline RL is not solely intended for robotics. Its application to real-world business domains is a promising and motivating area of study.

• If the reviewer has any suggestions of datasets relevant to the proposed problem domain we shall be happy to conduct experiments.

[1] Simglucose: Jinyu Xie. Simglucose v0.2.1 (2018) [Online]. Available: https://github.com/jxx123/simglucose.

[2] Qin, Rong-Jun, et al. “NeoRL: A near real-world benchmark for offline reinforcement learning.” Advances in Neural Information Processing Systems 35 (2022): 24753-24765.

Q. How many feasible s of Proposition 4.3?

We apologies for not understanding this question. If the reviewer could clarify we would be happy to answer.

Thank you for clarifying the question. As you mentioned the teacher network updates are driven by uncertainty, meaning the frequency of updates is not fixed and depends on the dataset. We tracked the number of teacher network updates for each environment using one seed and following the same training protocol. The results are summarized below:

Thank you for your valuable feedback. Please find our response below:

W1: The method ‘Training Teacher’ is somehow naive, and Eq. (2) is not clear.

• Our proposal is not centered on training the teacher network itself but rather on demonstrating how domain knowledge effectively contributes to out-of-distribution (OOD) action regularization through critic regularization and teacher updates. We employ behavior cloning to train the teacher network, which is a well-established practice in knowledge distillation. If the reviewer has any suggestions for a better training paradigm, we would be happy to explore them.
• For Eq 2 we add the following explanation in the revised manuscript: We assume that a set of common-sense rules in the form of domain knowledge, denoted as $\mathcal{D}$, is available. This domain knowledge defines a hierarchical mapping from states to actions $(S \to A)$, structured as decision nodes. Each decision node $T_{\eta_i}$ has constraint $\phi_{\eta_i}$ that determines its branching, a Boolean indicator $\mu_{\eta_i}$ selects the branch $(\swarrow or \searrow)$ to follow based on whether the constraint $\phi_{\eta_i}$ is satisfied.

W2: Limited domain

This method is extended to continuous domain by using the regularization in Eq 4 : $\mathcal{L}(\theta) = \mathcal{L}cql(\theta) + \lambda E_{s \sim \mathcal{B}r \land s \models \mathcal{D}} [Q_s^\theta(s, a_s)-Q_s^\theta(s,a_t)]^2$ during critic $(Q_s^\theta)$ training for continuous domain and using actions from actor network $(\pi_s)$ for cross entropy loss in Eq 7 : $\mathcal{L}(\omega) = -\sum_{s \models D} (\pi_t^\omega(s)log(\pi_s(s)))$. We have added this explanation to the revised manuscript (lines 273–275) and also demonstrate it through experiments. The Sales Promotion and Simglucose tasks reported in Section 5.3 operate on continuous state-action domains.

W3: The hyperparameter

• We agree that hyperparameter tuning in offline settings can be challenging. In practice, this can be addressed using methods such as Bayesian Optimization and by observing the behavior of Q-values, as proposed in [2].
• $\lambda$ should be selected with peak Q value observed during training. We discuss the effect of different $\lambda$ in Fig 12 (supplement) and empirically show $\lambda$ = 0.5 works well in most setting. We also show the proposed methodology is better than $\lambda$ = 0 in most environments demonstrating robustness to hyperparameters.

[2] Kumar, A., Singh, A., Tian, S., Finn, C., & Levine, S. (2021). A workflow for offline model-free robotic reinforcement learning. arXiv preprint arXiv:2109.10813.

W4 : Updating teacher

We restructured the updating teacher section in the revised manuscript providing more clarity as follows:

• Given a reasonable warm start, the critic is expected to give higher Q values for optimal actions for $s \in \mathcal{D} \cap \mathcal{B}_r$ as it learns from data. We aim to leverage this knowledge to enhance the initial teacher policy $\pi_t^\omega$ trained on heuristic domain knowledge. For $s \sim \mathcal{B}$ and $s \models \mathcal{D}$, we calculate the average Q-values over actions suggested by the critic and the teacher, and compare them.
• If $\mathbb{E}(Q_s^\theta(s,a_s)) > \mathbb{E}(Q_s^\theta(s,a_t))$ , this indicates that the critic expects a higher average return from its action than from the teacher's action. In such cases, we can use the critic’s action to update $\pi_t^\omega$, thereby improving the teacher policy over the domain $\mathcal{D}$. However, solely relying on the critic's Q-values can be misleading, as high Q-values may appear for out-of-distribution (OOD) actions. To prevent the teacher from being updated by OOD actions, we measure the average uncertainty of the Q-values for both the critic and teacher actions.
• The uncertainty is measured using predictive variance by adding dropout to the Q network over T forward passes. If $\mathbb{E}(Var^T Q_s^\theta(s_r,a_s)) < \mathbb{E}(Var^T Q_s^\theta(s_r,a_t))$ , it suggests that the critic’s actions are learned from expert data in the buffer and are not OOD samples.

Q4 : detach() and target Q

Yes we use .detach() for $Q_{s}^{\theta}(s,a_t)$ treating it as stable target and only $Q_{s}^{\theta}(s,a_s)$ is adjusted. Thank you for pointing this out we have added the definition in revision in line 128.

Thank you for your valuable feedback. Please find our response below:

W1. Training the teacher network highly requires domain knowledge.

• Yes, we have explicitly acknowledged the dependency on domain knowledge in our problem setting and discussed it as a limitation in the manuscript. However, we would like to emphasize that the domain knowledge used is not perfect; it is heuristic-based and derived from human understanding, which reflects practical constraints in real-world scenarios.

W2. The method is heuristic and plug-and-play in different algorithms.

• We respectfully disagree with this point. The method requires careful regularization of the critic network and an uncertainty-based update of the initial teacher network. We have detailed the effects of each hyperparameter introduced in our approach in Figures 11 and 12 of the appendix (supplementary material).
• While the initial knowledge is distilled from heuristics, it is further updated during offline training, which is a critical step. This update process is essential for achieving the performance improvements demonstrated in our experiments and ablation studies. To the best of our knowledge this mechanism of knowledge distillation has not been previously studies.

W3. The simulation tasks are simple

• The datasets used in this paper are recognized and utilized by the research community. Beyond the six standard Gym environment tasks, the mythology is also validated on human generated data from business domain (Sales Promotion) and healthcare task (Simglucose).

• These tasks are not simple and exhibit intricacies of real world problems. Please refer to [1] and [2] for the description of these tasks.

[1] Simglucose: Jinyu Xie. Simglucose v0.2.1 (2018) [Online]. Available: https://github.com/jxx123/simglucose.

[2] Qin, Rong-Jun, et al. "NeoRL: A near real-world benchmark for offline reinforcement learning." Advances in Neural Information Processing Systems 35 (2022): 24753-24765.

Q1 Can this method be applied to a continuous environment? If yes, should the authors try difficult simulated tasks, such as d4rl?

• Yes this method is applicable to continuous domain. Both sales promotion and sim glucose tasks are from continuous domain. However, this method is designed for use cases where reasonable domain knowledge is available as stated in our problem setup and limitation.
• While we acknowledge that our methodology relies on domain knowledge—which may be challenging to design for high-dimensional robotics tasks such as those in d4rl (e.g., HalfCheetah)—such domain knowledge is more accessible in business and healthcare contexts. We believe that this type of problem setup deserves research attention, as offline RL is not solely intended for robotics. Its application to real-world business domains is a promising and motivating area of study.
• If the reviewer has any suggestions of datasets relevant to the proposed problem domain we shall be happy to conduct experiments.

Q 2. Can this method be applied to different base algorithms, e.g., BEAR, IQL.

Yes, the method is applicable to other base algorithms with an actor-critic structure, as it primarily adds a critic regularization term.