DIAR: Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation

Q1. DIAR is different from simply adding value function learning to LDCQ. Our goal was to make Q-function learning more precise, so we introduced a value function to ensure balanced training of the Q-function. By incorporating constraints from the value function during training, we can predict Q-values with greater accuracy. Additionally, the diffusion model, as a generative model, can produce a wide variety of meaningful latent vectors. We aimed to leverage this capability to enhance both Q-function and value-function learning. In the Bellman equation for Q-function learning in LDCQ, $Q(s_t,z) ← (r_{t:t+H}+\gamma^H Q(s_{t+H}, argmax(Q(s_{t+H},z_i)))$, the reward term $r_{t:{t+H}}$ is undefined if $z$ is not sampled from the offline dataset. However, by introducing the value function, we can indirectly evaluate the value of latent vectors generated by the diffusion model. As a result, DIAR uses not only the offline dataset but also latent vectors sampled from the diffusion model in the training process for both Q-functions and value functions. This approach led to significant performance improvements on datasets such as Maze2D, AntMaze, and Kitchen.

Q2. Our goal was to address offline reinforcement learning problems using a diffusion model. To this end, we compared DIAR with algorithms like Diffuser and DD, as well as skill-based algorithms. While the tasks in our experiments focused on goal based tasks, we did not explicitly involve providing the goal as input or solving tasks where the objective is to directly reach a specified destination. However, since DIAR can also function as a goal-conditioned RL algorithm, it would be possible to include the goal as part of the input and conduct additional experiments for comparison under goal-conditioned settings. Evaluating the ability to find an optimal path from the current state to a specified goal is as important as the experiments we have conducted so far and could offer further insights into DIAR’s capabilities.

Q3. The value function is used to evaluate whether the current decision is appropriate, and if it is not, a new decision is generated. However, it is challenging to train the value function perfectly. If the value function fails to accurately assess the value of the current state, the new decision generated may not be more optimal. Additionally, since we do not have access to unlimited data, this limitation can pose challenges for the DIAR model. Currently, AR is designed based on a few assumptions, but we plan to refine it into a more general algorithm in the future, making it applicable to a wider range of tasks and domains.

Q4. The purpose of introducing AR was to address the possibility of finding a more optimal choice when selecting a long action sequence all at once. Therefore, if there is a method to evaluate the current decision, AR can be applied even without relying on the value function. Additionally, AR can be used in environments with dense rewards. However, to enable more general applicability, the algorithm needs to be further expanded and refined.

Q5. As the reviewer mentioned, a tighter bound for the value is indeed $\gamma^{-H}V(s_t)$. In Section 4.3, the bound is calculated using $\gamma^{-H}V(s_t)$. However, when applying AR, we did not explicitly use $\gamma$. Instead, we evaluated the latent vectors' values based on the value function $V(s_t)$. To account for the possibility of a noisy value function, we incorporated a slight margin, allowing AR to operate effectively when there is a significant difference in value. When the value differences are small, the algorithm tends to follow the original decision more closely, maintaining a balance between exploration and adhering to the initial choice.

Q1.

Thank you for clarifying the distinction between DIAR and LDCQ. I now have a clearer understanding of the motivation behind incorporating the value function and the differences between the two approaches.

However, I still perceive DIAR as an "IQL version of LDCQ" combined with "adaptive revaluation." While the authors argue that DIAR is distinct from IQL + LDCQ, the critic updates closely mirror those of IQL, with the primary difference being the skill-based approach. Additionally, the diffusion latent extraction appears to be derived from LDCQ. Specifically, lines 251–264 seem to follow the IQL approach with $z\_t$ replacing $a\_t$ (I would strongly recommend citing IQL in relation to the expectile loss here).

While combining existing methods is a valid and valuable contribution, it typically requires strong and broad empirical support to compensate for limited novelty.

 

Q2.

I appreciate the clarification that DIAR is not a goal-conditioned RL algorithm. However, its applicability appears largely restricted to goal-conditioned tasks. Without demonstrating its performance in more diverse environments, comparisons with general-purpose algorithms may be unfair, as DIAR seems particularly tailored to goal-conditioned tasks.

 

Q3, Q4, Q5.

Thank you for addressing the practical considerations surrounding adaptive revaluation (AR). I agree that refining adaptive revaluation into a more generalized framework is a promising direction for future work. However, applicability of AR still remains limited, and effectiveness of AR is not strongly demonstrated in the current version.

 

Despite the clarifications, I still have two remaining concerns:

• Novelty: The critic updates largely follow IQL, and the diffusion latent extraction process is derived from LDCQ.
• Effectiveness of Adaptive Revaluation: The adaptive revaluation, which is the key novelty of the paper, demonstrates limited applicability, and its empirical effectiveness remains unclear.

Due to these unresolved concerns, I will maintain my current score.

Weakness 1. In this study, we focused on long-horizon sparse reward problems, such as those in Maze2D, AntMaze, and Kitchen. Accordingly, the methods proposed in DIAR were designed to perform well in these environments, achieving significant performance improvements. However, we aim to expand our approach in the future to work effectively in other environments, such as half-cheetah, walker2d and hopper, by developing more generalizable ideas. We are always open to new ideas or discussions about extending the DIAR algorithm to a broader range of tasks.

Weakness 2. It is challenging to train a value function without any errors. A noisy value function struggles to accurately assess the value of a state, which can hinder the ability to make optimal decisions. In fact, experiments analyzing the impact of AR show that it improves performance in some cases but causes slight decreases in others. Additionally, since we do not have access to unlimited data, this limitation can make it difficult for the DIAR model to perform well in certain scenarios. Currently, AR is designed based on several assumptions, but we plan to refine it into a more general algorithm in the future, enabling its application across a wider range of domains.

Weakness 3 & Q1. DIAR is not simply an extension of LDCQ with a value function added. Our goal was to make Q-function learning more precise, and to achieve this, we introduced a value function to ensure balanced training of the Q-function. By incorporating constraints derived from the value function, we can predict Q-values with greater accuracy. Additionally, the diffusion model, as a generative model, can produce a diverse range of meaningful latent vectors. We sought to leverage this capability for both Q-function and value-function learning. In the Bellman equation used for Q-function learning in LDCQ, $Q(s_t,z) ← (r_{t:t+H}+\gamma^H Q(s_{t+H}, argmax(Q(s_{t+H},z_i)))$, the reward term $r_{t:{t+H}}$ is undefined if $z$ is not sampled from the offline dataset. However, by introducing the value function, we can indirectly evaluate the value of latent vectors sampled from the diffusion model. As a result, DIAR incorporates not only the offline dataset but also latent vectors generated by the diffusion model in the training process for Q-functions and value functions. Using this approach, we demonstrated significant performance improvements on datasets such as Maze2D, AntMaze, and Kitchen.

Weakness 4. According to the LDCQ paper, a fixed value for $\beta$ is used, highlighting its advantage of being less sensitive to various hyperparameters. In our current experiments, we also use a fixed value of 0.9 for $\tau$. However, as the reviewer suggested, analyzing the sensitivity of parameters like $\tau$ could be a valuable direction for further investigation.

Weakness 1. We reviewed and revised sentences from the paper that were difficult to understand. The revised content is highlighted in blue for clarity.

Weakness 2. As the reviewer mentioned, LDM and IQL are existing methods in their original forms. In our model, DIAR, LDM serves two purposes: (1) generating candidate latent vectors for decision-making and (2) aiding in Q-function training. However, our model is not simply a combination of IQL and LDM. We aimed to make Q-function learning more precise by introducing a value function, which ensures balanced training of the Q-function. By training the Q-function with constraints derived from the value function, we can achieve more accurate Q-value predictions. Additionally, the diffusion model, as a generative model, can produce a diverse range of meaningful latent vectors. We sought to leverage this capability for both Q-function and value-function training. In the Bellman equation used for Q-function learning in LDCQ, $Q(s_t,z) ← (r_{t:t+H}+\gamma^H Q(s_{t+H}, argmax(Q(s_{t+H},z_i)))$, the reward term $r_{t:{t+H}}$ is unknown if $z$ is not sampled from the offline dataset. However, by incorporating the value function, we can indirectly evaluate the value of latent vectors sampled from the diffusion model. Thus, DIAR utilizes both the offline dataset and latent vectors generated by the diffusion model during Q-function and value-function training. With this approach, we demonstrated performance improvements on datasets such as Maze2D, AntMaze, and Kitchen.

Weakness 3. The need to incorporate a latent diffusion model (LDM) is also highlighted in the LDCQ paper for several reasons:

• Flexible decoder design: Since the latent diffusion model operates in the latent space, even discrete action spaces can be effectively represented. This allows for more flexibility in designing the decoder.
• Temporal abstraction: With the help of LDM, a powerful generative model, it is possible to create a temporally abstract and information-dense latent space.
• Faster training and inference: Generating latent vectors is much more efficient than directly generating action-state sequences, as done in traditional methods, leading to faster training and inference processes.

Weakness 4. In Maze2D and AntMaze, rewards are given only when the agent reaches the goal, and all other states yield a reward of 0. Under the assumption of an expert policy, the value increases steadily as the agent gets closer to the goal. However, if additional rewards are provided for states outside the goal region, this condition may no longer hold, potentially leading to performance degradation when using AR. In the case of Kitchen, there are several intermediate sub-goals, and rewards are given when these sub-goals are achieved. In such scenarios, it can be challenging to make optimal decisions using AR based solely on the value function. We would greatly appreciate any ideas, suggestions, or feedback on AR, as they could significantly help us improve our algorithm.

Weakness 1,5. We have reviewed the sentences that were difficult to understand in the paper and revised them to make the content easier to understand. The revised content is marked in blue text.

Weakness 2,3 & Q1. Our approach is different from simply introducing value function learning into LDCQ. We aimed to make Q-function learning more precise by incorporating a value function, which helps ensure balanced training of the Q-function. By providing constraints from the value function during training, the Q-function can predict Q-values with greater accuracy. Additionally, the diffusion model, as a generative model, is capable of producing diverse and meaningful latent vectors. We wanted to leverage the diffusion model's ability to generate a variety of samples for both Q-function and value function learning. In the Bellman equation for Q-function learning in LDCQ, $Q(s_t,z) ← (r_{t:t+H}+\gamma^H Q(s_{t+H}, argmax(Q(s_{t+H},z_i)))$, the reward term $r_{t:{t+H}}$ cannot be determined if $z$ is not sampled from the offline dataset. However, by introducing the value function, we can indirectly evaluate the value of latent vectors sampled from the diffusion model. As a result, DIAR uses not only the offline dataset but also latent vectors sampled from the diffusion model in the training process for Q-functions and value functions. With this approach, we demonstrated performance improvements on datasets such as Maze2D, AntMaze, and Kitchen.

Weakness 4. We agree with the reviewer’s observation that many optimal algorithms have been proposed for D4RL tasks. However, as shown in our comparative analysis (e.g., Maze2D), there is still significant room for performance improvement. Nevertheless, we believe it is important to further validate the algorithm's performance across various environments. In the future, we plan to evaluate our model on a wider range of datasets to better assess its effectiveness.

Q2. In this study, we focused on long-horizon sparse reward problems, such as those in Maze2D, AntMaze, and Kitchen. Accordingly, the methods proposed in DIAR were specifically designed to perform well in these environments, and they demonstrated significant performance improvements. However, we aim to expand our approach in the future to incorporate more general ideas, enabling strong performance in other environments, such as half-cheetah, walker2d and hopper.