Thanks for your appreciation and useful suggestions to our work. Please see our responses below.

Q1: How does the concept of mental reward relate to other sciences?

A1: Thanks for raising this interesting question. To be honest, we don't have a clear answer for now. But we would like to share some of our thoughts on this. First, our study is definitely related to psychology, and perhaps biology. In psychology, if we want to evaluate a person's preference on an item, we usually ask a set of questions from multiple perspectives (e.g., Likert Scale) and build some models (e.g., regression) to analyze the person's feedback. However, in the recommendation process, we don't have the chance to ask user many questions about their feelings on an item, therefore the aforementioned psychology models may not work here. But perhaps there are other psychology models that focus on estimating user's preferences from very limited feedback. We will look into this in future works.

Q2: Adding standard deviation to Figure 3A.

A2: This is a very helpful suggestion! Unfortunately, we can't upload figures in the rebuttal phase according to NeurIPS's policy. We will update the figure in the next version of our paper.

Q3: Adding more citations on user behavior modeling in recommender systems.

A3: Sure, we will add those citations and check other parts of the paper. Thanks for your advice!

Thanks again for your time spent on our paper. We would be very grateful if you could re-evaluate our work based on our rebuttal.

Dea Reviewer nSej,

Thank you for your insightful feedback and appreciation to our work. Your kind suggestions are really helpful and shed light on the future directions of our work. As the deadline of the author-reviewer discussion phase approaching, could you please let us know if there are any remaining concerns or anything we can do to further improve our work?

Warm regards,

The authors.

Thanks for your valuable suggestions to our work. Please see our responses below.

Q1: The model’s direct interpretation on the mental reward is limited.

A1: Thanks for raising this question. In the third paragraph of the Introduction, we define the user's mental reward as "the summary of her private feelings about how she is satisfied". We also use multiple examples to illustrate this concept. However, we agree with you that we did not provide a very specific definition of the mental reward, because it is essentially a highly abstract concept and can be interpreted in different ways depending on the scenario contexts. For example, a user might click on a news article because of its attractive headline, but end up feeling uncomfortable after reading the content. Here, the mental reward is a reflection and quantification of how the user feels uncomfortable.

On the other hand, despite its definition, the mental reward does exist, and it lies between user's implicit feedback (e.g., click) and her real preferences. Our work aims to narrow this gap by using the mental reward as an effective tool (and the experimental results demonstrate this). In summary, our work makes an important step towards understanding the gap between users' implicit feedback and their real preference. We contribute a novel and theoretically solid algorithm to uncover the mental rewards and demonstrate its effectiveness by extensive experimental results.

In addition, we agree with the reviewer that evaluation about how accurately the model captures true preferences is an open question. In our paper, we compute the correlation between the mental rewards and the lengths of user-RS interaction trajectories. We also visualize the distribution of the mental rewards conditioned on the users’ real actions (please refer to Section 5.3). We believe that these results are instructive in answering the above question.

Q2: Oversimplified Modeling of User Feedback.

A2: Thanks for mentioning this issue. Real-world recommender systems may involve many types of user feedback, depending on specific application scenarios. For research works, we have to make appropriate simplification. In fact, we follow existing literature (see related works and Section 2 and 5) and classify user feedback as positive and negative, in order to cover as many scenarios as possible. However, we aggree that extending MTRec to include more diverse user feedback is an interesting future direction.

In addition, the reviewer mentioned that the uncertainty in user feedback may influence the accuracy of the mental reward model. This is true. Actually, such uncertainty affects almost all the recommendation models, because user feedback is random variable by its nature. To mitigate this issue, we model the mental reward also as random variable, which motivates the design of our algorithm QR-IQL (see Secion 4.2). QR-IQL addresses the uncertainty in the mental rewards and achieves better performace than conventional IRL algorithm (please refer to Section 5.1).

Q3: More ablation experiments.

A3: We agree that including more experimental results would strengthen our claims. Therefore, we add three new set of experiments to test different choices of hyper-parameters. The results are shown below.

Exp1: Ablation on different choices of N (number of quantiles). Intuitively, a larger N (more quantiles) provides more flexibility to model the underlying distribution of $Q_\lambda$, however, it may also bring overfitting issues. We agree that searching for the optimal N is necessary. Therefore, we add a new set of experiments on the Electronic dataset over three models, as shown below.

From the above table, we can see that different choices of N do not result in significant changes in terms of AUC and NCIS. In addition, N=5 performs the worst, perhaps due to limited flexibility to represent the underlying quantile distribution.

We also test different choices of N on Virtual Taobal over two RL-based methods. The results are reported below.

Surprisingly, we obverve a clear trend of improving performance, with respect to N, of both PPO and SAC. These results show that a larger number of quantiles helps to learn more accurate mental reward models. Note that this trend is not that obvious in supervised learning models (refer to the first table). We hypothesis that it is because RL-based methods are more sensitive to rewards.

Exp2: Ablation on different choices alignment loss weights $\kappa$. As RL-based methods are more sensitive to rewards, we test 7 different choices of $\kappa$ under the PPO and SAC experiments. The results are shown below.

We can see that the optimal choices of $\kappa$ are 0.4 and 0.3 respectively. We can see that increasing $\kappa$ does not always bring improvement. This indicates that the mental reward is better used as regularizer, instead of the main source of the reward. We hypothesis that the mental rewards, while provide useful supervision information, may also bring inductive bias.

Exp3: Computational costs. Our experiments are run on a server with 2×AMD EPYC 7542 32-Core Processor CPU and 2×NVIDIA RTX 3090 graphics. For the offline experiments on Amazon datasets, it takes about 3 hours for 50,000 iterations of training with a 4000 batch size. For online experiments on Virtual Taobao, it takes about 4 hours for 50,000 RL training steps.

Furthermore, although training MTRec's additional mental reward model takes several hours, this cost is worthwhile since recommender systems are primarily concerned with inference latency, and MTRec adds no computation during inference.

Thanks again for your time spent on our paper. We would be very grateful if you could re-evaluate our work based on our rebuttal.

Dear Reviewer bFhU,

We are so sorry that our rebuttal does not fully address your concerns. Could you please slightly elaborate what are the questionable rationales? On the existence of the mental reward or the inappropriate modeling of it? Or perhaps the lack of evidence to support it? It would be very helpful for us to respond to your concerns.

Meanwhile, we totally respect your decision because the "mental reward" is indeed a new concept, and it might require a progress to fully accepting it. Anyway, we thank for your constructive suggestions that help to improve our work. We hope that you can give us a chance to explain for your remaining concerns. Thank you!

Best wishes,

The authors.

Dear Reviewer bFhU,

Thanks for your thorough reviews and helpful suggestions. Your instructions really inspired us and help to make this work more solid. We have tried our best to address the issues you mentioned in the review. We will definitely follow your suggestions and integrate the new results in the next version of our paper. Could you please let us know if there are any remaining concerns, or anything we can do to further improve our work?

Warm regards,

The authors.

Thanks for your appreciation and helpful suggestions to our work. Please see our responses below.

Q1: Why the number of quantiles N=10 is chosen instead of 5 or 20?

A1: Thanks for raising this question. Intuitively, a larger N (more quantiles) provides more flexibility to model the underlying distribution of $Q_\lambda$, however, it may also bring overfitting issues. We agree that searching for the optimal N is necessary. Therefore, we add a new set of experiments on the Electronic dataset over three models, as shown below.

From the above table, we can see that different choices of N do not result in significant changes in terms of AUC and NCIS. In addition, N=5 performs the worst, perhaps due to limited flexibility to represent the underlying quantile distribution.

We also test different choices of N on Virtual Taobal over two RL-based methods. The results are reported below.

Surprisingly, we obverve a clear trend of improving performance, with respect to N, of both PPO and SAC. These results show that a larger number of quantiles helps to learn more accurate mental reward models. Note that this trend is not that obvious in supervised learning models (refer to the first table). We hypothesis that it is because RL-based methods are more sensitive to rewards.

Q2: Whether the calculation of mental rewards affects the recommendation response time?

A2: In fact, according to Algorithm 2 in Appendix A.3, we only calculate the mental rewards during training. There is no extra computation that might increase the response time during testing. We are sorry for the misunderstanding caused and will state more clearly in the paper.

Q3: Whether the calculation of mental rewards affects the recommendation response time?

A3: Thanks for your insightful suggestion! Since there is no direct way to evaluate the mental reward model, computing correlations is an effective approach. In the first figure of Figure 3, we report the correlation between users' mental rewards and the lengths of their interaction with the recommender system. We can see that users' mental rewards gradually decrease with the number of interaction steps increases. This aligns with the intuition that people might lose interest and get tired after consuming enough recommendation contents.

Thanks again for reviewing our paper, we hope that the above rebuttal can address your concerns.

Dear Reviewer 7bFs,

Thank you for your appreciation and valuable support to our work. Your thorough review and insightful suggestions really make our work better. As the deadline for the rebuttal phase approaching, could you please let us know if there are any remaining concerns or further suggestions?

Warm regards,

The authors.

Thanks for your helpful suggestions to our work. Please see our responses below.

Q1: On the definition of the mental reward.

A1: Thanks for raising this question. In the third paragraph of the Introduction, we define the user's mental reward as "the summary of her private feelings about how she is satisfied". We also use multiple examples to illustrate this concept. However, we agree with you that we did not provide a very specific definition of the mental reward, because it is essentially a highly abstract concept and can be interpreted in different ways depending on the scenario contexts. For example, a user might click on a news article because of its attractive headline, but end up feeling uncomfortable after reading the content. Here, the mental reward is a reflection and quantification of how the user feels uncomfortable.

On the other hand, despite its definition, the mental reward does exist, and it lies between user's implicit feedback (e.g., click) and her real preferences. Our work aims to narrow this gap by using the mental reward as an effective tool (and the experimental results demonstrate this). In summary, our work makes an important step towards understanding the gap between users' implicit feedback and their real preference. We contribute a novel and theoretically solid algorithm to uncover the mental rewards and demonstrate its effectiveness by extensive experimental results.

Q2: How sensitive is the proposed method to the assumption that users are maximizing their cumulative mental rewards?

A2: This is an insightful question. Actually, this assumption is the basic motivation of using inverse RL, where we aim to seek a reward function that rationalize users' behaviors. If the users are not maximizing their mental rewards, which means that learned mental rewards are noises, they should not bring improvements to the recommendation models. However, our experiments (including offline, online, real-world deployment) demonstrate the effectiveness of the learned mental rewards, which in turn justifies the correctness of this assumption.

Q3: How does MTRec conceptually and practically differ from existing reinforcement learning-based recommendation frameworks?

A3: MTRec is actually a complementary to RL-based methods. RL-based methods heavily rely on accurate rewards to learn good policies, yet existing approaches simply use user implicit feedback (e.g., click) as rewards. Due to the gap between the implicit feedback and their true preferences, the implicit feedback may not be the optimal reward signal. MTRec makes a non-trivial correction to the reward signals, which significantly improves the performance of RL-based methods. Please refer to Section 5.2 for more details.

Q4: Have the authors considered any methods to analyze or visualize what the model has learned?

A4: If we understand corretly, this question is related to Q1. Perhaps answer A1 could clarify some of the concerns. In a word, the mental reward model learns to summarize and differentiate user's feelings after she takes an action. For example, if a user clicks and reads a piece of news, how much does she like it actually? In Section 5.3, we visualize some analysis on the mental reward model. We can see that the mental rewards actually correlated with users' behaviors, which also aligns with our intuitions. Although we don't know exactly how the learned mental reward correspond to users' feelings (e.g., like, neutral, dislike, angry, favorite), it indeed provides useful information to the recommender systems and helps to better understand the users.

Q5: Comparisons to Decision Transformer based methods.

A5: Thanks for raising this question. In the paper we consider online RL setting and test MTRec on PPO and SAC. As the reviewer mentioned, it is also interesting to test MTRec on offline RL methods such as Decision Transformers. We use the dataset containing 100,000 user-RS interaction trajectories, as described in Section 5.2, to train a DT model (including 2 Transformer layers, 8 heads, and an embedding size of 128) for 5 epochs, and test it the same way with PPO and SAC in the online Virtual Taobao environment. The comparative results are shown below.

It can be observed that the original DT performs worst across all models, since DT in trained in offline setting while PPO and SAC are trained in online setting. Therefore, it is not a really fair comparison. However, we can see that DT-MTRec still improves DT, which demonstrates that MTRec indeed brings more useful information than simply treating click as reward signal.

Note that all of the four references mentioned by the reviewer are based on DT, therefore we only test the original DT and combine it with MTRec. In the next version of our paper, we will discuss more about the differences between our work and existing RL-based methods, including but not limit to the references mentioned by the reivewer.

Thanks again for your time spent on our paper. We would be very grateful if you could re-evaluate our work based on our rebuttal.

Dear Reviewer pp7H,

Thanks again for your constructive suggestions and precious time spent on our paper. As the deadline of the author reviewer approaching, we want to briefly summarize the follow up actions and try to win your support on our work.

Action 1:

Based on your comments on the mental reward, we will state more clearly on the definition of the mental reward and explain how this new concept relates to existing concept such as user preference modeling, user satisfaction, explicit and implicit preference. However, we may still argue that the definition of the mental reward needs some flexibility. Because we want to use it to approximiate user's real preference, which itself is a vague concept.

Action 2:

We will discuss about the sensitivity of our assumptions and continue to explore in this direction. For example, if there are only part of the users are rational, how should our model respond to improve robustness. We thank you for raising this interesting point.

Action 3:

We will discuss the related works you mentioned in the reviews and integrate new comparative experimental results in the Experiment section. This would definitely strengthen our claims. Thanks for reminding us of these works.

Action 4:

As we have mentioned in the paper, our model brings more improvements on user's long-term engagement (measured by NCIS), instead of AUC. However, we will follow the reviewer's suggestions and state more clearly on the potential risk of losses. Thanks for raining this important point.

We hope that our responses help to clarify the misunderstandings of the original paper. We would be so grateful if you consider to re-evaluate our paper based on the responses and follow up actions.

Thank you sincerely,

The authors.