Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learning

We thank you for the encouraging and insightful comments. All of them are invaluable for further improving our manuscript. Please refer to our response below.

Q1: In my opinion, moving too much crucial algorithm components to the appendix is a main weakness of the paper...

R1: While we agree that the appendix provides valuable details on the implementation of the algorithm, we do not want to make it appear as though key implementation details were buried in the appendix. To clarify, we used the action graph implementation of HInt in all the domains except for: Sprite Obstacles, Air Obstacles, and Robo Obstacles. In these domains, we found that using the control method resulted in more stable performance. We modify the main paper to include additional clarification in the methods section of the distinction between the control-target filtering strategy, and in the experiments section to clarify which domains utilized the which strategy.

In practice, the control method could be applied interchangeably with the action graph formulation in other environments, since it is essentially an action graph formulation where graphs with more than two state factors in the control chain are filtered out. As a result, we do not consider the ``control'' method to be a heuristic injecting significant amounts of knowledge about the environment since it still relies on interaction identification to 1) identify the control object through correlation with actions 2) identify interactions with the target object using the null assumption. The other alternative filtering schemes are simply strategies which we employed and we believe would be informative to a researcher building on this method.

Interaction identification is a different problem from just action influence detection, as it relies on identifying the counterfactual possibilities from relations that are often quite sparse. Inductive biases about interactions often have to be encoded through physical assumptions about the simulation, as seen in [1]. Furthermore, the problem of causal interactions is similar to that of actual cause [2,3], especially functional actual cause [4], and remains an open and challenging problem for reasoning. NCII, while it makes some assumptions, is more general than just a contact heuristic because it does not rely on special knowledge about the physical domain such as contact or physics, except for the fact that state factors can be not present and thus "nulled" out. To clarify this point, however, we are currently running CAI on several of our domains, with upcoming results to be added to this thread, to demonstrate that the benefit of the algorithm is not purely heuristic.

Related revised parts: Section 4.2, Appendix D

Q2: The content of Appendix C reads like quite a few tricks were needed to get HInt to work well.

R2: With respect to the heuristic elements in Appendix C, while this passive reweighting strategy is meaningful for empirical results, it is a common practice in interaction-based methods because in many domains, especially when dealing with random actions, interactions are exceedingly sparse. In particular, we note that other work involving interactions such as [4, 5] also employ similar strategies when learning the interaction models. While we entirely agree that these are important implementation details, they are 1) specific to domains with sparse interactions 2) Not a core contribution of this particular work. For clarity, we added the following to the methods section:

In practice, learning to identify interactions is challenging when interactions are rare events, such as a robotic manipulation domain where the robot rarely touches the object it should manipulate. These challenges have been approached in prior work [4,5] by reweighting and augmenting the dataset using statistical error. We include a description of how those methods are adapted and used in this algorithm in Appendix C.

We do not convey that the work in the main paper is entirely divorced from the details in the appendix, or that those details are entirely unimportant, but rather just that those techniques have been employed in similar forms in prior work.

Related revised parts: Section 4.1, Appendix C

Thank you for acknowledging that we have addressed your concerns and raising the score.

To address your further concerns on Q1/R1, we have updated Appendix D accordingly. Specifically, we use action-graph interactions in all domains except Spriteworld Obstacles, Robosuite, Pushing Obstacles, and Air Hockey Obstacles. In these domains, we use control-target interactions for experimental inference.

We would also like to update you with the additional experimental results below.

CAI Results

Reulsts are given in Figure 12, Appendix K. We evaluated CAI across four domains: Sprites Obstacles, Robo Default, Air Default, and Franka Default. In these domains, CAI performs comparably to ELDEN, as both methods incorporate an inductive bias towards interactions, with CAI specifically emphasizing an inductive bias towards actions.

Wall-clock time comparison

We provided the wall-clock time comparison in Table 8 in the revised appendix, showing that HInt does not require significantly more computational time compared to the baselines. While HInt-learned takes more time, it still outperforms CAI, which also infers causal structures among objects. Note that the wall-clock time reported for ELDEN excludes the dependency inference component, making it appear more efficient than HInt-learned. Overall, our approach does not bring substantial computational overhead compared to other interaction-based methods.

Ablations with different filtering criteria

We have compared HInt and Hindsight on different sampling schemes ("final", "future", "episode"). Results are given in the revised Figure 10 and Appendix I, which suggest that hindsight filtering is applicable in any sampling scheme.

Please do not hesitate to let us know if you have any additional comments or concerns. Thank you again for your valuable feedback and effort!

Thank you for fixing the formulation in appendix D, for the CAI results, and wall clock times. I appreciate the effort you put into improving the paper during the rebuttal!

I agree that the wall clock time of HInt is comparable to the baselines and therefore reasonable. The comparison to CAI indeed shows a significant benefit of filtering based on interactions instead of action influence alone.

I just noticed that in Figure 4, NCII is sometimes referred to as NII or NCI. It would be better to be consistent here.

Thank you for the new results in Appendix I. From the caption of Figure 10, it is not entirely clear to me, which sampling scheme was used for HER. Maybe this could be clarified with an additional sentence.

As my main concerns have been addressed and the remaining small issues can be addressed easily, I have raised my score.

We thank the reviewer for the insightful and encouraging comments. Please see our response as follows.

Q1: The null operation in Equation 3 depends on the threshold ( \epsilon_{\text{null}} )...

R1: We appreciate that the reviewer's identification of $\epsilon_\text{null}$ as a key parameter. In practice, we used the same null epsilon parameter of $\epsilon_\text{null} = 1$ for all environments and all experiments, even across domains such as random vectors, where the dynamics differ significantly from those in the physical interaction domains such as air hockey and SpriteWorld. This is because when the state inputs and deltas are normalized the effect of interactions as a result of an interaction is fairly significant. For example, the average change in velocity as a result of an interaction such as a ball hitting another ball or a robot manipulating an object in most physical domains is significant compared with the effect of drag. We are also working on an ablation illustrating the effect of changing the $\epsilon_\text{null}$ value in both the random sprite domain and the random vectors domain (to show the difference across two significantly different dynamics), to illustrate the insensitivity of this hyperparameter, and will include that in this thread when those runs are completed.

This being said a poor choice of $\epsilon_\text{null}$ can result in the method failing. We suggest the following strategy for selecting $\epsilon_\text{null}$: We can take the null model $f(\mathbf s, \mathbf a, \mathbb B(\mathbf v))$ and observe that the interaction states will be those where the difference between the nulled and non-nulled model outputs will be larger (meaning the non-nulled model will have higher likelihood.) On the other hand, in non-interaction states, the error should be small. Thus, we can take the differences, identify two clusters, and then take the midpoint between the higher cluster center (corresponding to interaction states) and the lower cluster center (corresponding to non-interaction states. As $\epsilon_\text{null}$. We add a formalization of this hyperparameter selection strategy in the appendix to strengthen the overall generalizability of the paper and add some analysis illustrating the threshold selected by applying these operations.

Related Revised Parts: Appendix E

Q2: Hindsight Experience Replay (HER) is an important baseline here

R2: HER sampling is a vital element on which we ran early experiments. In the work, we use the "final" variant of HER, and thus compare directly against this variant. However, we also implemented the final and future version of HER, and are presently running an ablative comparison of each of these methods for both HInt and baselines on the random Sprite environment. As these experiments are currently running, we will update this thread with the results of these experiments.

We again thank the reviewer for their response and constructive feedback, we have provided a general response, and also include additional details on the updated analysis of $\epsilon_\text{null}$ here.

See General response, Table 2 in Appendix E

We ablated on the $\epsilon_\text{null}$ hyperparameter with 3 seeds for each setting of $\epsilon_\text{null}$ to empirically analyze the dependence on the threshold for identifying null interactions. Our experiments evidence a limited dependence on this parameter except at the upper and lower extremes, as values within 0.3 to 2.5 show performance within variance for both random DAG and Box2D domains, which have significantly different dynamics. Since epsilon-null compares the difference in normalized log-likelihood of the predictions, this is an invariance across two orders of magnitude. This result corroborates with the evidence that across all of the domains that we tested, air hockey, Spriteworld, Robot pushing, Franka Kitchen and random DAG, and Random DAG with nonlinear relationships, the same $\epsilon_\text{null} = 1$. We believe that this suggests the efficacy of NCII across a variety of domains, though future work can investigate strategies for automatically setting the $\epsilon_\text{null}$ parameter.

Please let us know if you have any other questions. We are more than happy to discuss and address them. Thank you again for your positive feedback!

Thank you for your constructive comments. We have responded below.

Q1: How is the state space defined across all environments? Assuming the entire environment has a low-dimensional state space, I’m curious how it computes the difference between states (Eq. 3) and infers the null state (Eq. 4) in a high-dimensional case (e.g., image).

R1: Thank you for pointing this out. For high-dimensional cases, we can address two scenarios:

• Case 1: The object states are accessible but are high-dimensional. We believe our framework can still handle this scenario. To validate this, we are conducting evaluations on simulated high-dimensional random vectors and will provide results soon.

• Case 2: The object states are not directly accessible, and only images/videos are available. In this case, we propose leveraging object- or slot-centric encoders to model the generative process from low-dimensional states to high-dimensional observations. The encoder's outputs can then serve as the state representations, allowing our framework to operate seamlessly. To test this, we are currently evaluating a combination of a VAE and object-state representation in the Box2D environment. We will share these results within this rebuttal period once they are available.

Q2: From my understanding, inferring the null state should have a complexity of $O(n^2)$ based on state dimensionality, which may limit scalability in high-dimensional state spaces. However, L263 mentions a time complexity of O(1). Could the authors clarify this?

R2: This is because, during inference, we can directly use $h$, the learned inference model designed to align with the counterfactual test, to infer the relationship. Thus the process operates in O(1) time.

Q3: [Dependence on hyperparameters] The method distinguishes null states based on prediction error (Eq. 3), but setting this hyperparameter could vary depending on environments and tasks. Moreover, certain states, even within an environment or task, may have more complex dynamics than others. In such cases, how does the method define a single $\epsilon_\text{null}$?

R3: We appreciate that the reviewer's identification of $\epsilon_\text{null}$ as a key parameter. In practice, we used the same null epsilon parameter of $\epsilon_\text{null} = 1$ for all environments and all experiments, even across domains such as random vectors, where the dynamics differ significantly from those in the physical interaction domains such as air hockey and SpriteWorld. This is because when the state inputs and deltas are normalized the effect of interactions as a result of an interaction is fairly significant. For example, the average change in velocity as a result of an interaction such as a ball hitting another ball or a robot manipulating an object in most physical domains is significant compared with the effect of drag. We are also working on an ablation illustrating the effect of changing the $\epsilon_\text{null}$ value in both the random sprite domain and the random vectors domain (to show the difference across two significantly different dynamics), to illustrate the insensitivity of this hyperparameter, and will include that in this thread when those runs are completed.

We suggest the following strategy for selecting $\epsilon_\text{null}$: We can take the null model $f(\mathbf s, \mathbf a, \mathbb B(\mathbf v))$ and observe that the interaction states will be those where the difference between the nulled and non-nulled model outputs will be larger (meaning the non-nulled model will have higher likelihood.) On the other hand, on non-interaction states the error should be small. Thus, we can take the differences, identify two clusters, and then take the midpoint between the higher cluster center (corresponding to interaction states) and the lower cluster center (corresponding to non-interaction states. As $\epsilon_\text{null}$. We add a formalization of this hyperparameter selection strategy in the appendix to strengthen the overall generalizability of the paper, and add some analysis illustrating the threshold selected by applying these operations.

Related Revised Sections: Appendix E

Thank you for acknowledging our response and raising the rating. We appreciate your continued discussion on high-dimensional cases. Here, we would like to provide an update with our new results on high-dimensional scenarios and discussions, along with ablation studies on $\epsilon_\text{null}$.

High-dimensional cases

• NCII using VAE encodings table:

See General Response, Appendix J

• HInt using VAE：

See General Response, Appendix J

We provide some preliminary experimental results that verify that NCII and HInt have the potential to scale to higher-dimensional domains. In these experiments, we use state-segmented 80x80 object masks for each object. The input state is then pixel-based statistics on the objects (position, delta position, etc.), as well as the latent state of a 128-latent dimension variational autoencoder trained to encode all of the segmented objects. The pixel-based statistics are primarily to encode dynamic information like velocity, which is often a small (if any) portion of the variational autoencoder. To match the encoding dimension of the latent space, the pixel statistics are tiled to 128 dimensions. Together, each factored state is a 256-dimensional input and is passed into NCI, and used as the observation for RL using hindsight filtering. Additional details can be found in Appendix J (Figure 11, Table 10).

The performance of all existing interaction-based methods, and the performance of GCRL, decline in the image-based domain and noise increase. This is unfortunately also the case with NCII and HInt. Nonetheless, by comparison to the baseline methods, both methods show improved performance. Furthermore, a VAE-based encoding may not be ideal for NCII-based methods. Future work might investigate object-based representation [1-3], and image-based goal-conditioned methods [4,5]. Investigations of this sort, while important, are tangential to the research objective of this work: to demonstrate that the null hypothesis can be a valuable inductive bias for interaction inference and that using interactions for hindsight filtering can significantly improve the performance of goal-conditioned RL. We suggest that the existing experiments in the paper demonstrate this clearly and that the additional experiments exploring higher dimensional states validate this claim. Incorporating the elements investigated through NCII and HInt would undoubtedly be invaluable for future work in scaling GCRL and interaction inference, but both of these general research areas remain open problems when scaling to more complex domains.

[1] Yoon, Jaesik, et al. "An investigation into pre-training object-centric representations for reinforcement learning." ICML 2023.

[2] Zadaianchuk, Andrii, Maximilian Seitzer, and Georg Martius. "Object-centric learning for real-world videos by predicting temporal feature similarities." Advances in Neural Information Processing Systems 36 (2023).

[3] Li, Yulong, and Deepak Pathak. "Object-Aware Gaussian Splatting for Robotic Manipulation." ICRA 2024 Workshop on 3D Visual Representations for Robot Manipulation.

[4] Chane-Sane, Elliot, Cordelia Schmid, and Ivan Laptev. "Goal-conditioned reinforcement learning with imagined subgoals." International conference on machine learning. PMLR, 2021.

[5] Zhang, Zichen, et al. "Universal visual decomposer: Long-horizon manipulation made easy." 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024.

Update on $\epsilon_\text{null}$

See General response, Table 2 in Appendix E

We ablated on the $\epsilon_\text{null}$ hyperparameter with 3 seeds for each setting of $\epsilon_\text{null}$ to empirically analyze the dependence on the threshold for identifying null interactions. Our experiments evidence a limited dependence on this parameter except at the upper and lower extremes, as values within 0.3 to 2.5 show performance within variance for both random DAG and Box2D domains, which have significantly different dynamics. Since epsilon-null compares the difference in normalized log-likelihood of the predictions, this is an invariance across two orders of magnitude. This result corroborates with the evidence that across all of the domains that we tested, air hockey, Spriteworld, Robot pushing, Franka Kitchen and random DAG, and Random DAG with nonlinear relationships, the same $\epsilon_\text{null} = 1$. We believe that this suggests the efficacy of NCII across a variety of domains, though future work can investigate strategies for automatically setting the $\epsilon_\text{null}$ parameter.

Please feel free to let us know if you have any further questions. Thank you again for your valuable comments and efforts!

We thank the reviewer for the insightful feedback, please see the following for our response.

Q1:This method seems relatively limited to object-centric domains, where the dynamics between objects is relatively simple.

R1: Yes, in this work, we assume the framework operates in object-centric domains. However, the null counterfactual reasoning framework can also be applied to general factored domains (i.e..., Factored MDPs), as the learning and inference processes using Equations (3) and (4) do not require the states to be explicitly factorized as objects.

Additionally, we believe that learning object interactions is non-trivial, particularly in highly interactive, object-rich environments where such interactions are critical for decision-making. Hence, we believe that robotics and embodied AI that require compositional and controllable representations and decision-making systems could highly benefit from object-centric RL [1-3].

[1] Hong, Yining, et al. "Multiply: A multisensory object-centric embodied large language model in 3d world." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Shi, Junyao, et al. "Composing Pre-Trained Object-Centric Representations for Robotics From" What" and" Where" Foundation Models." arXiv preprint arXiv:2404.13474 (2024).

[3] Zheng, Ying, et al. "A Survey of Embodied Learning for Object-Centric Robotic Manipulation." arXiv preprint arXiv:2408.11537 (2024).

Q2: Certain set-based architecture (such as PointNet and some version of GNN) might not work in general domains to model dynamics.

R2: Thank you for pointing this out. We would like to emphasize that our framework is architecture-agnostic, meaning any architecture can be used to model dynamics within it. PointNet and GNN are the architectures we selected for the simulated dynamics systems and RL environments in this work. Additionally, we include results for Transformers in Appendix E.3; however, these did not perform as well empirically as the other two architectures.

We also argue that PointNet and GNN hold potential for scalable applications, especially when object-factored representations are available. These representations can be learned through object-centric video encoders such as Video-DINOSAUR [4] or object-aware 3D perception models like Object-Aware Gaussian Splatting [5] and other robotic foundation models. Once the representations are obtained from these encoders, GNNs or PointNet can still be used as effective interaction models.

[4] Zadaianchuk, Andrii, Maximilian Seitzer, and Georg Martius. "Object-centric learning for real-world videos by predicting temporal feature similarities." Advances in Neural Information Processing Systems 36 (2023).

[5] Li, Yulong, and Deepak Pathak. "Object-Aware Gaussian Splatting for Robotic Manipulation." ICRA 2024 Workshop on 3D Visual Representations for Robot Manipulation.

Q3: The simulated nulling procedure and the filter criterion feel very heuristic and specific to the considered domains.

R3: Yes, this actually indeed our goal - providing a flexible and heuristic method applicable to various physical and planning domains involving numerous objects and interactions, in which object interactions are critical for effective planning and policy learning. Since the framework does not rely on specific domain physical priors or detailed domain information, it only requires object states, which can be obtained either from observations or off-the-shelf object-centric encoders. So the flexibility makes the framework can be applicable across many domains, rather than being limited to specific ones.

Additionally, learning and leveraging interactions in physical domains is challenging, as many foundational models still struggle to accurately capture and understand physical interactions (see one recent evaluation [6]). The problem of identifying interactions from a causal perspective is related to the ongoing research problem of actual causality [7], with ongoing recent work demonstrating that this problem requires a computationally intractable search over the set of counterfactuals [8,9]. Therefore, we see the potential in this direction and believe in the scalability and utility of the null counterfactual framework.

[6] Kang, Bingyi, et al. "How Far is Video Generation from World Model: A Physical Law Perspective." arXiv preprint arXiv:2411.02385 (2024).

[7] Halpern, Joseph Y. Actual causality. MiT Press, 2016.

[8] Chuck, Caleb, et al. "Automated Discovery of Functional Actual Causes in Complex Environments." arXiv preprint arXiv:2404.10883 (2024).

[9] Beckers, Sander. "Causal explanations and XAI." Conference on causal learning and reasoning. PMLR, 2022.

We sincerely thank you for your feedback. We have provided a general response for all reviewers. Here, we would like to discuss your specific comments.

First, we would like to clarify that the tasks discussed in our paper are not niche cases. Rather, learning and leveraging interactions are foundational and crucial research areas in general dynamic systems and probabilistic inference [1-2], and actual causality [3-5]. In RL, identifying and inferring object-centric dynamics and interactions is particularly important, as it helps agents better understand the structured dynamics and enhances decision-making efficiency [6-10]. As demonstrated in our paper, along with related works, incorporating interaction inference has shown positive effects on (GC)RL tasks in general.

To further clarify our claim, our approach performs well in domains where counterfactual nulling can be applied, and where the distribution of goals is mismatched with the hindsight distribution. While these assumptions may not hold for every domain, they are relevant to several tasks such as robotic manipulation, logistics, and video games. In these domains, the states observed by an agent are significantly influenced by the policy the agent follows and the state factors (e.g., objects, facilities, game elements) it interacts with. Our empirical evidence shows that filtering the hindsight distribution using HInt provides clear benefits in sample efficiency.

Additionally, we would like to emphasize that our model is not limited to physical interactions. Given the general nature of null counterfactual inference, our approach can be extended to interactions involving any state variables. Please refer to our previous responses for further clarification on this point.

We hope this addresses your concerns. If you have any additional questions or points for discussion, we would be happy to further clarify. We truly appreciate your time, effort, and valuable suggestions to help improve our work.

[1] Kipf, Thomas, et al. "Neural relational inference for interacting systems." ICML 2018.

[2] Bleistein, Linus, et al. "Learning the dynamics of sparsely observed interacting systems." ICML 2023.

[3] Halpern, Joseph Y. Actual causality. MiT Press, 2016.

[4] Chuck, Caleb, et al. "Automated Discovery of Functional Actual Causes in Complex Environments." arXiv preprint arXiv:2404.10883 (2024).

[5] Beckers, Sander. "Causal explanations and XAI." Conference on causal learning and reasoning. PMLR, 2022.

[6] Wang, Zizhao, et al. "ELDEN: exploration via local dependencies." NeurIPS 2023.

[7] Seitzer, Maximilian, Bernhard Schölkopf, and Georg Martius. "Causal influence detection for improving efficiency in reinforcement learning." NeurIPS 2021.

[8] Yoon, Jaesik, et al. "An investigation into pre-training object-centric representations for reinforcement learning." ICML 2023.

[9] Hwang, Inwoo, et al. "On discovery of local independence over continuous variables via neural contextual decomposition." ICML 2023.

[10] Qiu, Yiwen, Yujia Zheng, and Kun Zhang. "Identifying Selections for Unsupervised Subtask Discovery." NeurIPS 2024.

More Evaluations and Major Modifications in the Revision

[Regarding evaluating scalability in high-dimensional cases]

• NCII on nonlinear and high dimensional state compared to baselines (Table 9)
• NCII on the image-encoded state (Appendix J, Table 10)
• HInt with Image encoded state as input (Appendix J, Figure 11)

[More ablation studies]

• NCII with different $\epsilon_\text{null}$ (Appendix E, Table 2)
• comparing different kinds of hindsight (final, episode, and future). (Appendix I, Figure 10)
• wall clock time comparison (Table 8)

[More baselines]

• CAI baseline (Figure 12)

Other clarifications are highlighted in red throughout the revision. We hope that our responses have addressed the reviewer’s concerns and remain available for any follow-up questions. Thank you again for your time and effort!