Learning Fused State Representations for Control from Multi-View Observations

Thanks for your comments, and we will address your concerns in the following.

---

Potential Issues: hyperparameter tuning

Thank you for your valuable feedback. For all baseline methods with publicly available implementations, we directly used their officially released code and default configurations to ensure they achieve the performance reported in the original papers. We will include a clarification of this in the experimental details section.

---

Potential Issues: limited metrics in CARLA

Thank you for your suggestion. Our choice of evaluation metrics in CARLA is consistent with previous works. In addition, we would like to clarify that our evaluation metrics not only focus on driving performance (e.g., distance traveled, success rate of reaching 100 meters), but also take driving safety into account, including steering amplitude, braking intensity, and collision severity.

---

Potential Issues: reward normalization

We acknowledge the reviewer’s point that reward normalization may mask certain nuances in the reward signal. In our experiments, whether to apply reward normalization under different benchmarks depends on the implementation of the methods we compare against. We also conducted ablation studies on MetaWorld tasks, where reward normalization is required, to analyze the impact of reward normalization. Detailed results can be found at this link (Fig.2). The results show that without reward normalization, the performance of our method degrades significantly. This is because the large scale of raw rewards causes large gradients during Q-value updates, leading to oscillations in Q-value loss and overall instability in training. Additionally, reward normalization constrains the reward signal to a feasible range, which helps stabilize the optimization of the bisimulation objective, further improving the quality of the learned representations.

---

Essential References Not Discussed

We appreciate the additional references provided and are pleased to include Multi-view Dreaming and MOSER in our manuscript, as they significantly enhance the completeness and relevance of our work.

---

W1 and Q1: the lack of comparison with model-based MVRL

We appreciate the reviewer’s suggestion to compare our method with MB-MVRL algorithms. Since MV-MWM relies on additional expert demonstrations and Multiview Dreaming does not release its implementation code, we chose to compare our method with the MB-MVRL algorithm MOSER. MOSER seeks the optimal viewpoint for learning task representations under multiple views. Its implementation code is available at https://github.com/yixiaoshenghua/MOSER, and the corresponding experimental results can be found at this link (Fig.3). Our experimental results show that our approach outperforms the MOSER algorithm in both tasks, demonstrating the effectiveness of our method. We believe these additional comparisons help validate the superiority of our method.

---

W2: quantitative analysis on bisimulation metric

Thanks for your insightful comment. To provide a more quantitative analysis of the bisimulation metric, we include the training curve of the bisimulation loss and additionally measure the mutual $I\left( z_{L}^{0};s \right)$ information between the final fused embeddings $z_{L}^{0}$ and the ground-truth states $s$ using the MINE [1] method. The results can be found at this link (Fig.4). As training progresses and the bisimulation loss converges, we observe a consistent increase in mutual information. This indicates that the model not only optimizes the bisimulation criterion but also gradually constructs a task-relevant representation space aligned with the true environment states.

[1]Belghazi, et al. Mutual information neural estimation.

---

Q2: Will conducting reinforcement learning from multiple perspectives lead to increased time and storage costs, and how can this issue be mitigated?

While multi-view observations provide a more comprehensive understanding of the environment, they may lead to increased computational time, storage cost. To mitigate this, we can leverage pretraining techniques to reduce training costs. Additionally, inspired by MOSER, we can train a view selector to dynamically choose the most relevant views for decision-making, thus may potentially minimize unnecessary computational and storage demands.

---

Please do not hesitate to let us know if you have any additional comments.

We sincerely thank the reviewer for thoughtful and constructive feedback. We are pleased that our proposed framework for Multi-View Reinforcement Learning (MVRL), including the integration of bisimulation metric learning and masking-based latent reconstruction, was well-received. We appreciate the recognition of our theoretical formulation, experimental design, and the robustness of our method across diverse benchmarks. We truly appreciate your insightful feedback and constructive suggestions, and we will address your concerns in the following.

---

W1: experimental setup of Deep Mind Control Suite

We would like to clarify that our primary goal is to explore how multi-view information can be leveraged to enhance decision-making efficiency. While we focus on spatial multi-view in the paper, as demonstrated in our experiments with environments like Metaworld, PyBullet, and CARLA, we also recognize that "multi-view'' can manifest in both temporal and even multi-modal forms. Hence we further explore the temporal information fusion in DeepMind Control environments. To address the potential confusion, we will clarify the distinction between these two experimental setups in the next version of the paper.

---

Other Comments Or Suggestions 1: ablation study on varying view counts

We appreciate the reviewer’s insightful suggestion. In response, we have added an ablation study that varies the number of views, and the results are presented in Fig. 1 at the following link link. It shows that using any single view out of the three performs worse than the full multi-view setup. In particular, View 3 — a top-down perspective — suffers from severe occlusion, resulting in the lowest performance among the three. This highlights the limitations of relying on individual views and underscores the effectiveness of our approach in extracting and fusing complementary information across views. Overall, these results validate the necessity and benefit of multi-view learning in achieving more robust and informative state representations.

---

Other Comments Or Suggestions 2: In Fig.1, the symbol was mistakenly written.

Thanks for your careful review and for pointing out the mistake. We appreciate your feedback and will correct the symbol in the next version of the paper.

---

Q: model design details

Thanks for your comment, and we apologize for any confusion caused. We clarify this design as follows: we introduce an additional learnable [state] embedding that interacts with the view-specific embeddings through a self-attention mechanism. The output of this embedding, denoted as $z_L^{0}$, is updated through the bisimulation loss. The theory of bisimulation guarantees that the learned representations retain all task-relevant information from multiple views, enabling $z_L^{0}$ to effectively aggregate this information into a unified representation. Importantly, MFSC optimizes the bisimulation objective exclusively with respect to $z_L^{0}$, not the view-specific embeddings $z_L^{1-3}$. This design choice follows a similar paradigm to BERT and ViT, where supervision is applied exclusively to a designated token (e.g., the [class] token). We also recognize that the current depiction of gradient flow in Fig.1 may be misleading, and we will revise it in future versions to accurately reflect the optimization process.

---

Please do not hesitate to let us know if you have any additional comments.

We sincerely thank all the reviewers for their thoughtful evaluation, constructive feedback, and valuable suggestions, which have greatly contributed to improving the clarity, depth, and overall quality of our work. We will address your concerns in the following.

---

Q: Could you share the differences between MFSC and DBC when using the bisimulation metric?

We sincerely appreciate the reviewer’s insightful question and are pleased to provide a more detailed clarification of the differences between MFSC and DBC in their use of the bisimulation metric. Specifically, MFSC differs from DBC in two key aspects:

1. Avoiding Wasserstein distance for behavioral similarity. The bisimulation metric generally requires calculating the Wasserstein distance between distributions, which can be computationally expensive. To mitigate this complexity, DBC models latent dynamics transitions as Gaussian distributions, leveraging Euclidean distances to compute a closed-form Wasserstein distance, optimizing an $\ell_1$ distance between representations. However, this approach assumes Gaussian dynamics and introduces an inconsistency between $\ell_1$ and $\ell_2$ (Euclidean) distances, potentially leading to inaccurate approximations. MFSC adopts sampling-based computation approach via independent coupling strategy, encoding current observations into latent representations and explicitly learning latent dynamics to generate representations of subsequent states.
2. Cosine distance as the base metric for latent state representation. Inspired by SimSR, MFSC adopts cosine distance as the base metric for bisimulation computation. Please refer to our response W1 to Reviewer JyV5 for more details regarding the benefit of using cosine distance as the base metric. Besides, both $\ell_1$ distance (used for reward differences) and cosine distance (used for state differences) are conveniently scalable to matrix operations. Consequently, MFSC can efficiently compute a correlation matrix of state representations within a batch and optimize it to accelerate representation learning. This strategy significantly enhances efficiency compared to DBC’s pairwise comparisons via permutation, resulting in faster and more effective representation updates.

We sincerely thank the reviewer for the valuable comment and will provide a detailed clarification in the appendix of the revised manuscript.

---

Please do not hesitate to let us know if you have any additional comments.

Thanks for your insightful comments, and we will address your concerns in the following.

---

W1:

We noticed that the citation in Definition 3.1 was incorrect; the correct reference should be the $\pi$-bisimulation metric proposed by Castro et al.(2020)[1], rather than Castro et al.(2021)[2]. We apologize for the confusion caused. To clarify, we provide a brief clarification on the connection between these works:

Conventional bisimulation metric needs to compute the Wasserstein distance over the transition distributions across all actions, which is computationally expensive. Instead, Castro et al.(2020)[1] developed $\pi$-bisimulation which removes the requirement of considering all actions and only needs to consider the actions induced by a policy $\pi$. Castro et al.(2021)[2] introduces independent coupling and proposes a sampling-based metric that does not rely on the Wasserstein distance; however, this comes at the cost of violating the “zero self-distance”, potentially leading to representational collapse. In contrast, our use of the cosine distance is both efficient and guarantees this key property by design, as also demonstrated in SimSR [3]. This theoretically safeguards against collapse in learned representations. We will elaborate on this point more comprehensively in the next revision.

[1] Castro, et al.(2020). Scalable Methods for Computing State Similarity in Deterministic Markov Decision Processes.

[2] Castro, et al.(2021). MICo: Improved representations via sampling-based state similarity for Markov decision processes.

[3] Zang, H, et al.(2022). SimSR: Simple Distance-Based State Representations for Deep Reinforcement Learning.

---

W2:

We fully agree that both bisimulation metrics and self-attention mechanisms are well-established techniques, yet each has limitations when applied in isolation to MVRL. Specifically, self-attention effectively aggregates information from multiple views but typically lacks an explicit mechanism to focus on task-relevant features, potentially introducing irrelevant or redundant information into the final representations. On the other hand, bisimulation alone does not explicitly utilize the correlations across multiple views. To the best of our knowledge, such an integration has not been explored in the context of MVRL. Following your suggestions, we will carefully revise the manuscript to more clearly emphasize the distinct advantages of this integration.

---

The paper title and method name

Thanks for your point out. We agree that emphasizing core elements such as bisimulation and self-attention would better highlight the novelty of our method. We will consider revising the paper title and method name to more clearly reflect these components by incorporating terms like “Bisimulation-Constrained Attentive”.

---

Q1: necessity of the reconstruction loss

The reconstruction objective is designed to exploit the inherent cross-view dependencies in multi-view observations, thereby enhancing the model’s representation capacity. The effectiveness of this self-supervised objective has been demonstrated in our ablation study (Fig.7b).

We would like to offer some clarifications regarding your concerns. Crucially, the reconstruction objective is applied in the latent space, distinct from pixel-level reconstruction. This design encourages the model to reconstruct information that is shared across multiple views, rather than relying on superficial pixel-level redundancy. Moreover, in our downstream RL tasks, we exclusively utilize $z_{L}^{0}$, which is optimized via the bisimulation objective. The theoretical foundation of bisimulation guarantees that $z_{L}^{0}$ preserves task-relevant information aggregated from multi-view observations. For this representation, the reconstruction objective serves to recover task-relevant information. We therefore contend that the representations used for downstream control primarily capture task-relevant features rather than redundant information, as demonstrated by the visualizations in Fig.4&6. We hope this explanation clarifies our motivations and design choices.

---

Q2: results of F2C

We conducted a careful review of recent MVRL baselines and considered that F2C demonstrates competitive performance on the MetaWorld benchmark according to its performance report. However, the official implementation of F2C for MetaWorld has not been released, and we were unable to reproduce its reported results, despite reaching out to the authors multiple times without receiving any response yet. To enable a comprehensive comparison and study in MVRL, we report the convergence performance of F2C as provided in the original paper. We appreciate your suggestion and will provide a detailed explanation of this decision in the revised version of our paper.

---

Please do not hesitate to let us know if you have any additional comments.