Augmenting Offline Reinforcement Learning with State-only Interactions

We thank the reviewer for the constructive review.

Q1: Questions regarding experimental results analysis.

A1: We appreciate the reviewer’s suggestion to include standard deviations for the baseline results to enable more accurate comparisons. We will incorporate this in our future work.

Q2: Incomplete related works.

A2: We thank the reviewer for pointing out the additional related works. We will address this in our future revision.

Q3: DITS seems to outperform baselines in only half of the considered tasks.

A3: In RL experiments, the results are generally far more mixed than supervised learning, typically with high variance and flipped superiority on different environments. So outperforming in half of the tasks while not losing much in most of the rest is already quite a good result.

We thank the reviewer for the constructive review.

Q1: In Figure 2, the trajectories from DITS trajectory generator are annotated as "trajectories from good policies". However, they are actually obtained by interacting with the real environment using a planning-like method. DITS uses the DD-based trajectory generator to plan the following states, and predicts the action by an IDM. I think this annotation is somehow misleading, and pointing out they are from the real environment might be better.

A1: We appreciate the reviewer's feedback and acknowledge that the statement could be misleading. We will revise this annotation into "trajectories from state-only interactions".

Q2: In Line 16, Algo 2, as you train various offline RL methods besides BC on $\mathcal{D}_{k+1}$, it is imprecise to say "for behavioural cloning training".

A2: We agree with the reviewer that this statement is misleading. We will revise it to: "for downstream RL method training."

Q3: In Algo 4, how do you train the state-value function $V$? Is it trained from scratch for each iteration $k$, or only fine-tuned?

A3: At each iteration $k$, we train the state-value function $V$ from scratch. As the value network is a lightweight three-layer MLP, the computational overhead of training remains low.

Q4: For experiments, some of the ablation studies takes BC as the base RL algorithm. However, as DITS generates rewards for new transitions, and Appendix E proves the accuracy of the generated rewards, training BC is a little wasting because it does not utilize reward information. TD3+BC, a simple modification of BC, seems more likely to take full advantages of DITS. Could you please conduct ablation studies that comparing DITS against DITS w/o sitcher and DITS w/o generator, as well as the ablation on the number of generated trajectories, using TD3+BC as the base RL policy? If they cannot be completed due to the time limit, I will also appriciate it.

A4: We agree with the reviewer that using TD3+BC as the baseline for the ablation studies is more appropriate. We will incorporate it in future work.

We thank the reviewer for the constructive review.

Q1: In Line 205, What is the meaning of $m$?

A1: $m$ denotes the dimensionality of the state space $\mathcal{S}$; please see line 140.

Q2: What is the meaning of Equation 8?

A2: This equation defines the criterion for establishing a new stitching connection. Intuitively, it compares the probability of transitioning to $\hat s'$ and to $s'$. But the former is based on the minimal value over the ensemble, while the latter uses its mean, creating a conservative and stringent condition of preferring the proposed $\hat s'$ to the original $s'$.

Q3: How does the iteration rounds K affect the performance?

A3: As the number of rounds $K$ increases, the performance of the downstream RL method improves but eventually saturates beyond a certain $K$. Empirically, we find that $K = 2$ is sufficient for most cases, as discussed in Appendix B.

Q4: The second aspect is to blend the high-returned trajectories with the original trajectories. The current paper lacks an analysis of the importance of the trajectory stitcher.

A4: We have done ablation study on it. Please refer to Section 5.3 and Figure 4. We also give a conceptual illustration in Figure 1(d).

Q5: The main results in Table 1 only contain baselines BC and TD3+BC, which ignore many state-of-the-art offline RL baselines including CQL, IQL, DT, etc.

A5: We have indeed compared with CQL, IQL, and CPED in Table 3, focusing on Walker2d and Hopper. We will extend the comparisons to other environments (Halfcheetah, Kitchen, Antmaze).

Q6: The data augmentation baseline only includes SynthER as the diffusion-based data augmentation methods, while currently there are many latest diffusion-based data augmentation methods like MTDiff. Given the fact that SynthER is a transition-level generation method, the comparison is unfair and incomplete.

A6: We will compare with MTDiff in the future. However, we are not sure how it can take advantage of state-only interactions. So we will only compare in its vanilla form.

Q7: The main contribution of this paper can be divided into two aspects. One is to generate more accurate transitions based on the state-only observable environment. However, this part is more like directly using diffusion-based planning methods to do data augmentation, while remaining the rewards as predicted. It is doubtful that this scheme is useful enough as it requires too much interaction with the environment like online reinforcement learning although it doesn't need the reward.

Also, an additional ablation study is needed to show the effectiveness of this generation pipeline like using Diffuser, DD, or other trajectory-level data augmentation methods to generate trajectories given that the methods can access the dynamics of the environment.

A7: Our experiment used state-only interactions to generate only 300 trajectories (line 410). As a result, the number of state-only interactions is at most 30% of the sample size in Mujoco and AntMaze, and 22% in Kitchen. We also kept the same number of generated samples/transitions between modified SynthER and DITS (line 405).

In the applications we motivated in Section 1, the state-only interactions can be obtained with ease, typically through a simulator. But the reward can be much harder to get. We understand many applications do not fall into this setting, but it is indeed a realistic application scenario.

We will add the suggested ablation studies in the future revision.

We thank the reviewer for the constructive review.

Q1-a: In section 3.2, the authors claim the last C steps are needed to make the trajectory consistent. However the tasks in D4RL that are evaluated here are all Markovian.

A1-a: While we agree that the underlying physics are generally first-order Markovian in D4RL (we will omit "first-order" henceforth), the behavior policy used to generate the offline data does not necessarily only look at $s_t$, i.e., not Markovian. To quote the original D4RL paper:

AntMaze: "the controllers for this task are non-Markovian as they rely on tracking visited waypoints."

Kitchen: "we use 3 datasets of human demonstrations". Humans are generally not Markovian.

As a result, it is unclear whether the offline RL algorithm should assume the state trajectory $\{s_t\}$ is Markovian. In Mujoco, the policy does depend only on $s_t$, but to quote the D4RL paper:

The “medium-replay” dataset consists of recording all samples in the replay buffer observed during training until the policy reaches the “medium” level of performance.

we further introduce a “medium-expert” dataset by mixing equal amounts of expert demonstrations and suboptimal data, generated via a partially trained policy or by unrolling a uniform-at-random policy.

So the offline data is generated by a mixture of (many) policies, which may leave the function space of our policy, even though the mixture remains Markovian. Using a function space with overly high capacity may suffer from overfitting.

Even in online RL with a Markovian dynamics, looking further into the history can be beneficial for several reasons:

1. Approximation Errors in Function Approximation Most RL algorithms, particularly in large or continuous state spaces, rely on function approximators like neural networks. These approximators can struggle to perfectly capture the dynamics or the value function, hence benefitting from additional context, such as past states, to improve prediction accuracy and to learn richer representations of the environment.

2. Exploration and Strategy In some environments, the history can inform an agent about its past actions, helping it to avoid revisiting unproductive regions of the state space, and to identify exploration strategies (e.g., avoid loops or backtracking). For example, in sparse-reward environments, remembering past states can help the agent understand how far it has explored and where it should focus.

3. Policy Regularization Using history can act as a form of implicit regularization, smoothing the policy and making it less sensitive to small variations in the current state. This can stabilize learning, particularly in environments with high noise or chaotic dynamics.

4. Long-Term Dependencies in Policy For some tasks, even if the environment is Markovian, achieving the optimal behavior might require strategies that depend on recognizing long-term patterns. For instance: a robot navigating a maze might benefit from recalling previously visited dead ends. A trading algorithm may perform better if it considers historical trends in market behavior.

5. Empirical Success In many benchmarks, adding historical context (e.g., via recurrent architectures or stacking observations) has shown to empirically improve performance, even in tasks that are theoretically Markovian. This suggests that the additional capacity to model temporal dependencies compensates for other limitations in the agent's design or training process. For example, the Decision Diffuser [1] does exactly this.