Offline Reward Inference on Graph: A New Thinking

I am truly appreciative of your dedication in reviewing my work. Your attention to detail and the suggestions you provided have immensely contributed to its improvement.

[I]. Explanation of weaknesses

[1/2] W1: The paper doesn't present complete details of the experiment setup.

For most environments, the network architecture of the function $f_\Theta$ is a two-layer fully connected network with ReLU activation, and the number of hidden layer neurons is 64.

The network structures for policy training are as follows:

For Meta-World environments, it is a two-layer fully connected network with ReLU activation and the number of hidden layer neurons is 64.

For DeepMind Control Suite environments, it is a three-layer neural network with ReLU activation and the number of hidden layer neurons is 64.

[2/2] W2: While the paper does sufficient comparison over several existing methods.

As shown in Table 3 of Appendix A.5, our data has annotations for only a small fraction (mostly 10%-15%) of the environments. Therefore, for deep learning fully supervised methods, we found them to be ineffective in our testing. We believe that the KNN-based approach is a promising idea and may yield good results for certain problems. However, applying the KNN method in this scenario requires complementary measurement methods and an appropriate way of propagating rewards. We will give special consideration to the suggestions you have provided in our future work.

[II]. Explanation of questions

[1/2] Q1: Can the author give some comments on the computation cost?

Thanks for the insightful question. For this question, we choose to evaluate the computational costs of our method in several experimental environments, including training time and memory consumption. We also include the results as a section (A.7.1) in the paper's Appendix.

Most of our datasets consist of $1 \times 10^6$ state-action pairs. For computational convenience, we segment the dataset. We conducted tests on a computer with the following specifications: CPU: Intel i9-9900KF 3.6GHz, GPU: RTX 2070 SUPER (8GB VRAM). The training time and memory consumption for several environments are presented below:

Dataset slices containing 10000 state-action pairs:

Dataset slices containing 20000 state-action pairs:

We also conducted tests on a server with the following specifications for larger dataset slices: CPU: Intel Xeon Gold 6230 2.10GHz, GPU: RTX 3090 (24GB VRAM).

Dataset slices containing 30000 state-action pairs:

Dataset slices containing 40000 state-action pairs:

[2/2] Q2: What's the advantage of the proposed method, compared with this simpler formulation?

Thanks for this insightful question. We have carefully considered it and modified the corresponding section in the paper (Section 4.3). Additionally, we have included the detailed derivation process in the Appendix (A.2).

The inference of rewards for unlabelled nodes requires considering the information transfer between labelled and unlabelled nodes, we use $W_{UL}R_L$ for this calculation while also taking into account the relationships between unlabeled nodes themselves, computed using $W_{UU}R_U$. Since unknown rewards $R_U$ are a variable to be learned, we provide an iterative computation formula by:

$

R_U \longleftarrow W_{UU}R_U + W_{UL}R_L.

$

---

We once again sincerely thank Reviewer EMLT from the bottom of our for reviewing our paper. We hope our answers have addressed all your concerns. If so, we would greatly appreciate it if Reviewer EMLT could consider raising their score. Please let us know if there are more questions.

Thanks, Reviewer jJQM, for generously sharing your time and knowledge in reviewing our work. Your professionalism and attention to detail have played a crucial role in enhancing the quality of our paper.

[I]. Explanation of weaknesses

[1/4] W1: The TGR and UDS are presented too succinctly.

Thanks for pointing out this crucial issue. We have provided further details in the description of TGR and UDS.

[2/4] W2: Reward Propagation Using Graph Convolutional Networks would be an important baseline.

$\Phi_{GCN}$ is an outstanding and impactful work. We have cited the mentioned article in our paper, and the citation has been updated in the PDF. Upon this baseline, we conducted experimental validations on selected environments. In the experiments, we utilized the same data annotation ratio as our approach for $\Phi_{GCN}$. We performed five random seed experiments for each environment, calculating the average and standard deviation of the results. The experimental results are presented in the following table:

$\Phi_{GCN}$ considers information derived from the state and performs well in environments where the state contributes significantly to the reward. For the environments where the action substantially impacts the reward, the performance of $\Phi_{GCN}$ is slightly inferior.

[3/4] W3: The empirical evaluation is not adequate from a scientific rigour point-of-view.

1. The goal of RL can be described by maximizing the expected cumulative reward. In this setting, environments exhibit randomness, and different initializations of these environments result in varying learned policies, which affects the std value of Meta-World tasks. For statistical convenience, the average of returns in those tasks presents an intuitive observation of the policies' performance. It is convinced that the std values that assist in showing the stability of the policies in random environments are additional statistics.

2. In the reinforcement learning community, adopting 5 seeds is a feasible setting. In our experimental setup, conducting experiments with 10 seeds would take approximately 4 to 6 months, which is a substantial time investment.

3. Based on your suggestion, we have highlighted the statistically significant highest average returns for each task with bold numbers, in which the average return is the highest, and there is not a clear overlap for the standard deviations among the baselines. For the other results, we only mark the highest average returns for each task with italic numbers, which avoids confusion caused by intersecting standard deviations. We have modified Table 1 and Table 2 in the draft, including updating the captions for both tables.

[4/4] W4: Very little work is done to understand the impact of different components in the method.

The information in our ablation study in Appendix A.2.1 could address your concern. We evaluated the performance of methods employing four different input modes. We endeavored to utilize deep learning (supervised learning) for reward learning, but deep learning requires substantial amounts of annotated data, and our dataset comprises at most 10%-20% annotated data. This quantity is inadequate for deep learning, making the approach based on deep learning (supervised learning) practically ineffective.

[II]. Explanation of questions

[1/4] Q1: Section 4.4 "we" typo -> it should be capitalized

Thank you very much, we have made the necessary corrections.

[2/4] Q2: Doesn't this add more prior knowledge into the problem?

Thanks for the insightful question. We believe that if the factors' information is known, it achieves a fine-tuning of the input. If uncertain, one can directly learn along the dimensions of state and action.

[3/4] Q3: Figure 2 "prpagation" typo

Thank you very much, we have updated the content of the images accordingly.

[4/4] Q4-5: some rephrasing

The two modifications you suggested are extremely valuable. Your suggestions prompted us to reconsider related content carefully. We have modified the paragraph and updated it accordingly.

---

We once again sincerely thank Reviewer jJQM for reviewing our paper.

I am genuinely appreciative of your dedication to reviewing my work. Your valuable questions and insights have significantly enriched the quality of our work. We briefly summarize the questions into the following three questions:

[1/3] Q1: How diverse is enough for the proposed method to learn a decent function $f_\Theta$?

Thanks for the very valuable and profound question. The commonly used practice in existing work involves datasets generated by training classic reinforcement learning algorithms (such as SAC) policies online[1][2]. During the online training process, the policy evolves from weak to strong, interacting with the environment and generating diverse data. The question raised by the reviewer is very valuable; it represents a specific case but is indeed a scenario that could exist. We ensure the diversity of our dataset in two ways. Firstly, our dataset also adopts the training process data of classic reinforcement learning algorithms (such as SAC). Secondly, we use uniform sampling on the dataset to label state-action pairs, ensuring the uniform presence of annotated state-action pairs with rewards in various "regions" of the dataset. In our future work, we will consider how to address the specific case raised by the reviewer.

[1] How to Leverage Unlabeled Data in Offline Reinforcement Learning. Yu, Tianhe, et al. ICML 2022.

[2] Conservative Data Sharing for Multi-task Offline Reinforcement Learning. Yu, Tianhe, et al. NeurIPS 2021.

[2/3] Q2: Are the rewards calculated by the graph bounded by the annotated data, which means $max(R_U) \leq max(R_L)$?

Thanks for the deepful and valuable question. For this question, we added a section in the paper's Appendix A.7.2 to examine the changes in the maximum value of $R_U$ and verify whether the learned reward values are bounded by the annotated data. We design an illustration to demonstrate this issue concerning the formula, which is also shown in Appendix A.7.2. We can observe that within certain ranges of $(I-W_{UU})$ and $W_{UL}$ values, the maximum value in $R_U$ can exceed the maximum value in $R_L$, that is the learned rewards are not bounded by the annotated data.

[3/3] Q3: What is the computation cost requirement for the method in terms of computation time for the reward inference to converge and memory consumption?

Thanks for the insightful question. For this question, we choose to evaluate the computational costs of our method in several experimental environments, including both training time and memory consumption. We also include the results as a section (A.7.1) in the Appendix of the paper.

The majority of our datasets comprise $1 \times 10^6$ state-action pairs. To facilitate computation, we segment the dataset. We conducted tests on a computer with the following specifications: CPU: Intel i9-9900KF 3.6GHz, GPU: RTX 2070s (8GB VRAM). The training time and memory consumption for several environments are presented below:

Dataset slices containing 10000 state-action pairs:

Dataset slices containing 20000 state-action pairs:

We also conducted tests on a server with the following specifications for larger dataset slices: CPU: Intel Xeon Gold 6230 2.10GHz, GPU: RTX 3090 (24GB VRAM).

Dataset slices containing 30000 state-action pairs:

Dataset slices containing 40000 state-action pairs:

---

We once again sincerely thank Reviewer kgGb from the bottom of our for reviewing our paper and giving suggestions. We hope our answers have addressed all the your concerns. If so, we would greatly appreciate it if Reviewer kgGb could consider raising their score. Please let us know if there are more questions.