Towards Empowerment Gain through Causal Structure Learning in Model-Based Reinforcement Learning

We thank you for your constructive suggestions and detailed comments. We have provided the response and updated the revision.

Q1: Notations are not clearly defined. For instance, the causal mask M is introduced (i) without clear mathematical definition and (ii) it's not M that is used later but Ms\rightarows′ in Equation 5 (see lines 204 to 215). This makes the methodology unclear.

R1: The causal mask M provides a general representation of causal structures. In ECL, we learn two specific causal masks - $M^{s \to s}$ and $M^{a \to s}$ based on the MDP framework to optimize both dynamics and policy learning. We have revised Section 3.1 (lines 201-203) to provide a clearer mathematical definition and description of the causal mask and its relationship in the revised Sections 2.1 and 3.1.

Furthermore, we have provided a notation table in Appendix Table 2, where the essential variables and parameters in the objective functions are given (together with details and remarks). We also illustrate the trainable parameters in each objective function (marked in red in revised Appendix C.2).

Related Revised Section: Section 2.1 (lines 138-142) and 3.1 (lines 201-203), Appendix Table 2, Appendix C.2.

Q2: The vocabulary does not relate to clearly defined concepts, e.g. "causal understanding", "causal reasoning" in lines 153-155.

R2: We have revised the ambiguous terms, replacing "causal understanding" and "causal reasoning" with the more precise statement: "We aim to enhance the empowerment gain under the causal structure of the environment for improving controllability."

Related Revised Section: Section 2.2 (lines 155-156)

Q3: Key elements are described in the appendix instead of the main text, e.g. end of page 4: the causal loss that represents "the objective term associated with learning the causal structure" are given in Appendix D.2.

R3: We have moved the important content about the causal loss objective from Appendix D.2 into the main text of Section 3.1 for better accessibility.

Related Revised Section: Section 3.1 (lines 211-213)

Q4: How can a curiosity-based reward prevent "overfitting during task learning" (lines 270-272)?

R4: Overfitting may occur during steps 1 and 2 of causal discovery. To mitigate this, the curiosity-based reward is designed to encourage the agent to explore states that are challenging for the causal dynamics model to predict but are captured by the dense dynamics of the true environment. This incentivizes broader exploration, preventing the policy from becoming overly conservative and stuck in suboptimal behaviors.

Q5: Figure 5: is it the result of only one seed? Why is there only one seed used?

R5: For statistical robustness, each experiment is conducted using 4 random seeds. We have clarified this result in Section 5.2.1 of the revision.

Related Revised Section: Figure 5, Section 5.1

We hope that these comments have addressed the reviewer’s concern. We are happy to answer any follow-up concerns. We thank the reviewer's comments, this helps us a lot to improve the paper!

We thank the reviewer for the insightful and encouraging comments, please see the following for our response.

Q1: The framework aims at learning a consistent causal structure, thus cannot deal with scenarios when the causal dynamics change (change of number of objects, etc.) that might correspond to different behavior components. Maybe consider discussing potential extensions or modifications to their framework that could handle changing causal dynamics.

R1: Thank you for raising the important consideration about changing causal dynamics. In ECL, we have explored this through out-of-distribution settings by changing object positions, as detailed in Section 5.2.2. For the cases with changing dynamic functions or changing numbers of objects, we can extend ECL to handle these by incorporating local causal discovery [1] for dynamics learning or models for handling heterogeneous or nonstationary factors in causal discovery [2]. These can be added to the current framework, we will explore them in the future extension. We have added discussions in the future work for these points.

[1] Hwang, Inwoo, et al. "Quantized Local Independence Discovery for Fine-Grained Causal Dynamics Learning in Reinforcement Learning." ICML 2024.

[2] Huang, Biwei, et al. "Causal discovery from heterogeneous/nonstationary data." Journal of Machine Learning Research 21.89 (2020): 1-53.

Related Revised Sections: Section 7 (lines 531-539)

Q2: In the motivating example, why do we merely focus on improving controllability (row 2 and 3) and do not care about whether the agent finds the true target? (L79-l87) And further, how to detect the "controllable" trajectories? Could you provide more specific details on how controllable trajectories are identified and measured?

R2: Thank you for your suggestion. We focus on the true target through the reward feedback (by policy learning) while simultaneously improving controllability.

We use the causal empowerment term $I(s_{t+1};a_t|M)$ (Eq.(3)) to measure the controllability. Higher values of this mutual information can somehow indicate greater controllability. The policy $\pi_e$ (Eq. (7)) is trained to maximize the causal empowerment and collects controllable trajectories which are then used to optimize both causal dynamics and reward models. Hence, by optimizing this target, the agent can identify the controllable representation and increase controllability over environments. We have clarified this statement in the revised Section 3.2.

Related Revised Section: Section 3.2 (lines 262-265)

Q3: While maximizing mutual information I(s;a), would it be helpful to also take into account the causal structure, i.e. maximizing the state and action dimensions that are dependent on the dynamics graph? Potential benefits and challenges of incorporating the related s,a dimension into the mutual information objective could be made clearer.

R3: We sincerely appreciate your insightful suggestion. The causal empowerment proposed in ECL actually aligns with your thinking. We intentionally integrate the causal mask $M^{a\to s}$ into the mutual information where the state is processed through this mask, which enhances empowerment gain under the learned causal structure.

Moreover, to validate this approach, we also conducted experiments that directly maximize empowerment under the causal dynamics model (ECL w/ Emp) by maximizing $I(s_{t+1};a_t|M)$ without maximizing the difference between causal and dense models as used in Eq. (7). Our results of average return below and learning curves in Figure 30 of Appendix D.8 in the revision show that ECL w/ Emp still achieves better learning performance than baseline IFactor while not quite matching the original objective Eq. (7) used in ECL.

Related Revised Section: Figure 30, Appendix D.8 (line 1823-1827)

Q4: The result in Fig. 24 in DMC environment CHeetah and Walker seem to have not converged yet. Could the author compare ECL and IFactor when they both converge? I agree that the learning curve of ECL is already going up faster and steadier than that of IFactor, and I think it could also be helpful to see the convergence point of the policies.

R4: Thank you for your valuable suggestion. Following your suggestion, we extended our experiments with more environmental steps, reflected in the revised Figure 24 of Appendix D.6 and the results of the average return shown below. The updated results show that even at convergence, ECL maintains its better performance and sample efficiency than IFactor.

Related Revised Section: Figure 24, Appendix D.6

We hope that these comments have addressed the reviewer’s concern. We are happy to answer any follow-up concerns. We thank the reviewer's comments, this helps us a lot to improve the paper!

We thank you for the encouraging and insightful comments. All of them are invaluable for further improving our manuscript. Please refer to our response below.

Q1: Curiosity-driven exploration in RL is often sensitive and can be challenging to implement effectively. Are there different settings for setting curiosity in different experimental environments? Whether ECL is sensitive to curiosity rewards?

R1: Thank you for your valuable feedback about curiosity-driven exploration. We follow the same curiosity setting across different experimental environments. For sensitivity analysis, we examined the impact of the hyperparameter $\lambda$ in the curiosity reward function across four threshold settings (0, 1, 2, 3). The experimental results of episodic reward across different environments are shown below. From these results, we observe that ECL maintains robust performance for $\lambda$ values under 3, with optimal policy learning achieved when $\lambda$ = 1.

Furthermore, regarding curiosity design, we investigated a causality motivation-driven reward (ECL w/ Cau) that inverts the learning objective in step 3. Results in Figure 29 of Appendix D.8 in the revision demonstrate that ECL significantly outperforms this alternative approach. We also conducted a thorough examination of causal empowerment exploration, comparing directly maximizing the empowerment (Eq. (8)) given the causal dynamics model (ECL w/ Emp) versus empowerment distance maximization (Eq. (10)) used in the paper (ECL w/ Dis). Our findings in Figure 30 of Appendix D.8 in the revision reveal that while ECL w/ Emp demonstrates robust learning capabilities, ECL w/i Dis achieves superior performance.

Related Revised Section: Appendix D.7 (line 1752-1758) and Figure 29, Figure 30, D.8 (line 1823-1827)

Q2: Given the complexity of the ECL structure, the ablation studies should not be omitted, i.e., ablations on reward design, basic model and other related parts should be added.

R2: Thank you for your valuable suggestion. We have significantly expanded our ablation studies in Appendix D.8 of the revision. We examined different reward design variations for causal empowerment, including

• (1). cases without maximizing the difference between causal and dense model (Eq. (10)) instead of maximizing causal empowerment (noted as w/ Emp in Appendix D.8) of Eq. (8) ,
• (2). cases with the causality-driven reward $r\_{\mathrm{cau}}=\mathbb{E}\_{(s_t, a_t, s_{t+1} \sim \mathcal{D})}\left[{\mathbb{KL}}\left({P}\_{\rm{env}}||{P}\_{{\phi_c},M}\right)-{\mathbb{KL}}\left({P}\_{\rm{env}}||{P}\_{\phi_c}\right)\right],$ (noted as w/ Cau in Appendix D.8),
• (3). cases without curiosity-driven reward $r_{\mathrm{cur}}$ (noted as w/ Sha in Appendix D.8);
• (4). cases without the first-stage model learning, simultaneously conducting causal model and task learning (noted as w/ Sim in Appendix D.8) to verify the effectiveness of the proposed three-stage optimization framework.

The detailed comparative results for these variations are presented in Figures 29 and 30 of Appendix D.8. We plan to further explore additional basic model combinations with ECL to enhance policy learning capabilities.

Related Revised Section: Figures 29, 30, Appendix D.8

We thank the reviewer for the insightful and constructive feedback. We provide the response below.

Q1: How the causal models are used for downstream tasks is not specified anywhere except a single line of the Appendix “the CEM planning”

R1: We appreciate your important question. The causal models are used to execute dynamic state transitions defined in Eq. (2). The causal models are utilized in conjunction with reward models for downstream tasks through CEM planning. Specifically, The causal model is a combined dynamic model with the causal structure that maximizes state transitions likelihood (Eq. (5)). The reward model evaluates these transitions and provides feedback in the form of rewards. The CEM handles the planning process by leveraging the predictions from the causal and reward models to optimize the task's objectives effectively. The causal masks have been used in all steps (See the visualized masks in Figure 2). We have clarified this process in Section 4 of the revision.

Related Revised Section: Section 4 (lines 302-306)

Q2: The dynamics encoder predicts an independent probability for each feature of the next state given the current state and action- how realistic is this? In a similar vein, this method seems dependent on a well-defined feature space for the states and actions.

R2: Thank you for the insightful question. It is not necessary to have well-defined features for our approach. In MDPs, where well-defined observable states are available, we can directly learn the graph based on those observations. For POMDPs, where raw states are not accessible and only high-dimensional, entangled observations (e.g., videos) are provided, we can still utilize encoders to extract low-dimensional states and learn the graph on these representations. Prior works, particularly in causal representation learning [1], have demonstrated that these low-dimensional states are identifiable, with a mapping from the estimated states to the true states existing under mild assumptions. While this aspect is not the focus of our work, we build on such findings (e.g., in our experiments on pixel-based benchmarks).

Specifically, in our experiments: for chemical, manipulation, and physical environments, we utilize well-defined feature spaces for states and actions, which are explicitly designed for causal structure learning. For pixel-based environments such as DMC, Cartpole, and RoboDesk, ECL operates on latent states extracted by visual encoders. These encoders are supported by the identifiability theory proposed in IFactor [2], which ensures that these latent states can effectively map to the true states. While establishing identifiability is not the primary focus of our work, we leverage IFactor's encoders and include comparisons with IFactor in our experiments. Importantly, even in these settings, we can learn meaningful causal graphs. We have clarified this point in the revised Appendix D.2.

[1] Schölkopf, Bernhard, et al. "Toward causal representation learning." Proceedings of the IEEE 109.5 (2021): 612-634.

[2] Liu, Yuren, et al. "Learning world models with identifiable factorization." Advances in Neural Information Processing Systems 36 (2023): 31831-31864.

Related Revised Section: Appendix D.2 (lines 1110-1117)

Q3: The choice of “standard MBRL” baselines to complement the causal baselines do not seem very standard- why not compare against state of the art MBRL algorithms such as [1]?

R3: We're grateful for your suggestion about baseline comparisons. Following your feedback, we expanded our evaluation to include DreamerV3 [1], a state-of-the-art MBRL algorithm, testing it across three DMC tasks under noiseless settings. The results of average return are shown below and the detailed learning curves are shown in Figure 25 of the Appendix D.6 in the revision. These results demonstrate that ECL can achieve better performance and sample efficiency than DreamerV3.

[1] Hafner, Danijar, et al. "Mastering diverse domains through world models." arXiv preprint arXiv:2301.04104 (2023).

Related Revised Section: Figure 25, Appendix D.6

Q4: The use of Dense dynamics model/dynamics encoder interchangeably to describe the same thing is confusing.

R4: Thank you for your valuable suggestion. We have unified these two statements in the revision.

Q5: Section 4 is quite repetitive of section 3

R5: Thank you for your comment. The relationship between Sections 3 and 4 is intentional: Section 4 provides the practical implementation details of the theoretical formulation presented in Section 3. We have further simplified the practical implementation in Section 4.

Related Revised Section: Section 4

Q6: The causally-factored MDP is difficult to understand and the explanations are very brief; it would be helpful to show the causal masks and adjacency matrices for the example environment in figure 1, and provide more explanation/intuition for what each term in the equations represents. See the questions section.

R6: Thank you for the suggestion. We have added a more detailed explanation of the causal factored structure using Figure 1 as an example (see the revised Section 1). For formal definitions, we recommend referring to Section 2.1. To ensure simplicity and intuition, Figure 1 represents a special case where the state dimension of each object is set to 1. Additionally, we have clarified the explanation of the equation in Section 2.

Related Revised Section: Section 1 (paragraph in red, on page 2), Section 2

Q7: In the dynamics function f in equation (2), it’s not clear how exactly the adjacency matrices indicate the influence of current states and actions on the next state, what the output of each dot product represents and how they can be combined to get the ith dimension of the next state?

R7: Regarding the dynamics function in equation (2), each dot product represents the causal effect between dimensional factors. For instance, in the adjacency matrices $M^{a\to s}$, each point (i,j) represents the causal effect from the i-th dimension of the action to the j-th dimension of the state. These causal effects are combined to determine each dimension of the next state.

Q8: What is the relationship between the causal mask M and the adjacency matrices?

R8: They represent the same meaning in the causal structure. We have unified them in the revision.

Q9: How can the mask be updated without also having to update the dynamics encoder? If the mask was very bad at Step 1, wouldn’t the dynamics encoder also be very suboptimal and not appropriate to continue using as the mask is improved in step 2?

R9: The causal mask M is optimized by maximizing $\mathcal{L}_{causal}$ (Eq.(5))​, while keeping the parameters of $\phi_c$ kept fixed during this learning step. If the mask is suboptimal in Step 1, the empowerment policy can implicitly refine the causal structure by iteratively maximizing the controllability of the environment while updating the causal mask (Eq. (7) and Step 2 in Figure 2). We have clarified this explanation in Section 3.2 of the revised manuscript.

Related Revised Section: Section 3.2 (lines 230-231)

Q10: Why does it not work to simply maximize the empowerment given the causal dynamics model, rather than the difference between that and the empowerment under the dense model?

R10: Thank you for raising this insightful point! We explore the difference between simply maximizing empowerment under the causal dynamics model (Eq. (8)) versus maximizing the difference between causal and dense model empowerment (Eq. (7)). By doing so, we aim to implicitly make the model with causal structures can provide better controllability than the dense model.

To validate this, we also conducted experiments that directly maximize empowerment under the causal dynamics model (ECL w/ Emp) by maximizing $I(s_{t+1};a_t|M)$ without maximizing the difference between causal and dense models as used in Eq. (7). Our results of average return below and learning curves in Figure 30 of Appendix D.8 in the revision show that ECL w/ Emp still achieves better learning performance than baseline IFactor while not quite matching the original objective Eq. (7) used in ECL.

Related Revised Section: Figure 30, Appendix D.8

Q11: It is not explained in section 3 where the reward from step 1 and 2 comes from.

R11: The reward is derived from the defined environmental feedback during online exploration when collecting trajectories.

Thank you so much for your response and for raising these insightful questions. We have provided our discussions below.

It will be difficult to update the causal mask effectively if it is based on an incorrect dynamics encoder, which never gets updated.

Yes, a good dynamics encoder is generally necessary for learning causal masks. The problem of learning dynamics has been extensively studied in the MBRL literature [1-3]. Recent works in causal MBRL have also integrated structure learning into dynamics models [4-5].

In our work, we jointly learn the dynamics model and causal masks (eq. 5), following a similar approach to [4-5]. Our empirical evaluation demonstrates the ability to learn meaningful causal structures with different causal discovery approaches (see Figures 12-16 in the appendix). Therefore, we believe that learning dynamics is not the primary challenge in this context.

For future work, such as applying ECL to specific domains like real-world robotic systems, task-specific dynamics learning could be incorporated. Potential solutions would be (1) incorporating domain/task/physical knowledge [6]; and (2) pretrain dynamics ensembles with offline data [7]. However, these would be orthogonal to our current work, as the overall framework remains the same—only the method for learning dynamics models would differ.

It looks like adding the curiosity reward (lambda=1) is not significantly better than no curiosity reward (lambda=0) would it not be better to remove it, if it adds complexity with no performance gains (and even harms performance if not scaled carefully)?

This curiosity reward is used as a regularization term to prevent policy overfitting on causal dynamics, and our results demonstrate performance improvement across tasks. However, you are right that due to the relatively simple causal relationships in these chemical tasks, the performance differences from shaped rewards for preventing overfitting appear modest.

We are actually now conducting additional experiments exploring the impact of this term under complex environments (Manipulation tasks) and also more $\lambda$ values including {0.2; 0.4; 0.6; 0.8}. We will update our results with these findings in this thread once the experiments are complete.

In downstream task learning, the state and action inputs are fed into the reward model to predict rewards But the reward function is often different for downstream tasks, how could it predict them correctly？

Yes, we learn task-specific reward models, similar to approaches in model-based and causal reinforcement learning [1,3,4,5]. This is necessary as each task has its own uniquely defined reward model. Even within the same environment, the reward structure can vary significantly depending on the specific task objectives. Hence, for downstream tasks, we follow the approaches in MBRL that involve learning a predictor using state-action pairs to predict rewards from collected data, as described in Section 3.1.

Thank you once again for your response. Please do not hesitate to let us know if you have any further questions!

References

[1] Moerland, Thomas M., et al. "Model-based reinforcement learning: A survey." Foundations and Trends® in Machine Learning 16.1 (2023): 1-118.

[2] Deisenroth, Marc, and Carl E. Rasmussen. "PILCO: A model-based and data-efficient approach to policy search." ICML 2011.

[3] Hafner, Danijar, et al. "Mastering atari with discrete world models." ICLR 2021.

[4] Wang, Zizhao, et al. "Causal dynamics learning for task-independent state abstraction." ICML 2023

[5] Huang, Biwei, et al. "Action-sufficient state representation learning for control with structural constraints." ICML 2022.

[6] Lutter, Michael, et al. "Differentiable physics models for real-world offline model-based reinforcement learning." ICRA 2021.

[7] Rafailov, Rafael, et al. "Moto: Offline pre-training to online fine-tuning for model-based robot learning." CoRL 2023.

As mentioned in the above response, we have conducted additional ablation studies on this curiosity reward in the manipulation reach task. The learning curves for episodic reward and success rate, shown in Figure 31 of the revision and results of episodic reward shown below, indicate that the curiosity reward plays a crucial role in the complex environment.

To summarize, the curiosity reward plays a significant role in the framework. However, you are correct that for simpler tasks (e.g., chemical environments), the improvement may not be very significant—though there is still empirical evidence of gains. However, we would keep this since we are targeting to the complex environments, where causal dynamics might overfit. We appreciate you bringing this up and hope that the new empirical validations address your concerns.

Thank you for raising the score—we sincerely appreciate your time and effort!

As a side note, besides the validation shown in the above message, we have also conducted experiments under different $\lambda$ in the manipulation task. The learning curves of success rate are shown in Figure 31 of the updated revision. The results also indicate the importance of the curiosity reward.

Thank you once again for your valuable suggestions. Your suggestions and comments have been invaluable in improving our work.

We thank the reviewer for the insightful and useful feedback, please see the following for our response.

Q1: Minor Contribution: The current framework appears more like a combination of existing approaches rather than a novel advancement. Causal structure learning in model-based RL has been extensively studied in prior work, such as [1-2], as has empowerment in RL [3-4]. This may limit the perceived originality of the contribution, as it builds on established methodologies without significantly advancing them.

R1: Thank you for raising this concern. Striking a balance between explicit causal discovery and prediction performance, as in [1, 2], is indeed challenging, and the resulting policies often exhibit limited controllability over the system. Similarly, while empowerment encourages efficient exploration, it does not take into account the causal relationships within the environment [3, 4].

Our work introduces the causal empowerment framework, which uniquely integrates considerations of both controllability and causality. Importantly, this is not a simple combination of causal structure learning and empowerment. Instead, our framework addresses critical gaps in existing causal RL approaches, which often fail to fully exploit learned structures for enabling agents to actively explore, enhance system controllability, and utilize collected data to refine causal structures.

Empowerment gain is a natural choice to enhance controllability, and our framework builds upon this by employing an iterative process between empowerment and causal discovery. Furthermore, our approach is method-agnostic, meaning it can seamlessly integrate with existing models, such as those in [1, 2]. This flexibility makes our framework broadly applicable across various scenarios and use cases. We sincerely appreciate your perspective and hope this distinction clarifies the novelty and significance of our contribution!

[1] Huang B, Feng F, Lu C, et al. Adarl: What, where, and how to adapt in transfer reinforcement learning[J]. arXiv preprint arXiv:2107.02729, 2021.

[2] Huang B, Lu C, Leqi L, et al. Action-sufficient state representation learning for control with structural constraints[C]//International Conference on Machine Learning. PMLR, 2022: 9260-9279.

[3] Zhang J, Wang J, Hu H, et al. Metacure: Meta reinforcement learning with empowerment-driven exploration[C]//International Conference on Machine Learning. PMLR, 2021: 12600-12610.

[4] de Abril I M, Kanai R. A unified strategy for implementing curiosity and empowerment driven reinforcement learning[J]. arXiv preprint arXiv:1806.06505, 2018.

Q2: High Computational Cost: The framework’s iterative process of empowerment maximization and causal model updating may result in substantial computational requirements, potentially limiting its scalability in large or dynamic environments.

R2: We conducted a detailed analysis of the computation time (h) across two environments (Chain and collider), results are shown below. We can find that ECL achieves its performance improvements with minimal additional computational burden - specifically less than a 10% increase compared to CDL and REG. These findings demonstrate that ECL’s enhanced performance does not come at the expense of significant computational overhead. All experiments were conducted on the same computing platform with the same computational resources. For the detailed analysis, please refer to Appendix D.7 of the revision.

Furthermore, for large-scale environments and downstream tasks, such as real-world robotic applications, we can incorporate pre-trained visual dynamics functions or object-centric representations [5–7] as surrogates. This approach can significantly reduce the computational burden of training the representation and dynamics model. As these are not the focus of our current work, we plan to explore this direction in future work (see the revised future work section).

[5] Shi, Junyao, et al. "Composing Pre-Trained Object-Centric Representations for Robotics From" What" and" Where" Foundation Models." arXiv preprint arXiv:2404.13474 (2024).

[6] Wang, Jianren, et al. "Manipulate by seeing: Creating manipulation controllers from pre-trained representations." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[7] Luo, Hao, Bohan Zhou, and Zongqing Lu. "Pre-trained Visual Dynamics Representations for Efficient Policy Learning." European Conference on Computer Vision. Springer, Cham, 2024.

Related Revised Section: Appendix D.7, Section 7 (lines 531-539)

Thank you for your feedback!

To clarify, the iteration cycles in our framework do not require fine-tuning across different environments or tasks. Instead, we use fixed numbers of the iteration across diverse experiments for each domain, including different experiments in chemical, physical, and manipulation tasks. This demonstrates that our framework is not overly sensitive to this hyperparameter.

And thanks for raising the point of real-world scenarios. For those real-world deployment, the number of iteration steps can be determined based on task-specific factors such as task complexity, # steps used for standard policy learning. Additionally, monitoring metrics like loss magnitudes or rewards can be also useful for setting how many steps of iterations we need to use, similar to the generic and typical deployment process for RL in real-world cases like robot learning. We would like to point out that addressing real-world, large-scale scenarios remains the focus of our future work. And, in the current paper, we have tested our approach on well-established RL benchmarks without significantly tuning this parameter, further confirming that it does not pose a specific challenge (compared with those baselines).

Thank you again for your response! If you have any further questions, please feel free to let us know. We are more than happy to provide further discussions.