Seek Commonality but Preserve Differences: Dissected Dynamics Modeling for Multi-modal Visual RL

We are truly grateful for your thoughtful remarks and experimental suggestions. These remarks shed light on what we can improve and are crucial for refining our work. We address your main concerns as follows:

The method is only tested on visual modalities, limiting its generalizability and contribution.

We appreciate the comment and recognize the importance of generalizing to different modality types. We think our method’s contributions are twofold: the fundamental idea and the method itself. While our method mainly focuses on visual modalities, the core idea—that modality-correlated and distinct features are both crucial—might also provide initial spark of inspiration in non-visual fields. For instance, consider a photo of a sea and a corresponding text description like “Vibrant sea under a clear blue sky, with fluffy white clouds.” The photo may show details that the text misses, such as distant ships. Similarly, the text may also capture high-level concepts that lack in the image feature, such as the sensory concept of “fluffy”. These distinct features between modalities can be valuable for the task at hand, which is worth exploring and might motivate further research on image-text learning.

We hope the potential inspiration brought by our central idea can benefit the non-visual fields and help address some of the concerns. We are thankful for your reflective remark, which suggests a promising extension of our work to fit other modalities (e.g., audio, text) and benefit the entire multi-modal research community. We sincerely appreciate this insight and are committed to work on these extensions.

Experiments for visual inputs from multiple camera positions?

Thanks for the experiment advice. We have further verified the ability of our model on multiple camera positions. First, we switch the camera view of RGB modality in DMControl. Second, we test on CARLA with RGB and LiDAR BEV as input modalities. LiDAR BEV is a bird view map, whose perspective is very different from RGB. Due to the limited time budget, we only compare our method with the most competitive methods on these two environments.

The results and illustrations of different camera views are presented in Table 1,2 and Fig. 1(e),(f) in the rebuttal PDF. These results show that our method also works on multiple camera positions.

How is the masking done? Do the authors have results with other masking ratio?

Our masking operation is performed independently at each timestamp. Therefore, both the masked modality type and the masking locations vary randomly across different timestamps, which simulates a challenging occlusion scenario.

For other masking ratios, we further test the ratio of 0%, 40%, 60%, and compare our method with SPR, the most competitive baseline on DMControl. The new results are in Table 2 of the rebuttal PDF. The results show that as the masking ratio increases, both SPR and our DDM experience performance drops. However, DDM still outperforms SPR at different ratios.

Some places are not written accurately, such as Line 46 - 48 and Line 50-51.

We apologize for the confusion caused. For the first issue in Line 46 – 48, we confirm it was a misuse of the word "complemented," which inaccurately described modality consistencies. We meant to convey that the modality-consistent features contain shared and common information and create a unified description for the environment. We appreciate the detailed attention to this error and will correct it.

For the second issue in Line 50-51, our use of the phrase “filter out” may have been overly definitive. We intended to convey that modality alignment aims to encourage consistency and, consequently, mitigate inconsistencies. To clarify this, we provide evidence both in the literature and with experiments. Specifically, we discuss several alignment methods as follows:

Imposing consistency constraints is a straightforward method for modality alignment, typically achieved by minimizing cross-modality feature distances [1,2]. Ideally, if two modality features have zero distance, it means perfect consistency and the absence of inconsistencies. We reference multiple works as evidence to support this design goal, as quoted below:

“... the states expressed by different modalities can be consistent at the same time... to achieve this, we use a similarity measurement...” (Sec. IV.A in [1])

“...to ensure the consistency of the latent embeddings of different modalities in the shared latent space, we develop two parallel cross-alignment schemes...” (Sec.3.2 in [2])

Mutual information optimization [3] is also designed to enhance feature consistency (and thereby decrease inconsistencies), we quote [3] as follows:

“The key innovation is a specially-designed density ratio estimator that encourages consistency between the latent codes of each modality.” (Abstract in [3])

For experimental evidence, we calculate the CKA index [4] between modality features. CKA is a metric to quantify similarity between network features for the same input. A higher CKA indicates greater feature correlation. Specifically, for RGB modality on CARLA, we extract features $z$ from a baseline SAC model without modality alignment, and the modality-consistent feature $\overline{z}$ and inconsistent feature $\hat{z}$ from our DDM. We Then compare the CKA between these features across 1K input samples. The results are as follows:

① CKA($z$,$\overline{z}$)=0.865

② CKA($z$,$\hat{z}$) = 0.897

③ CKA($\overline{z}$,$\hat{z}$) = 0.561

It can be seen the both ① and ② are relatively high, showing that without modality alignment, $z$ retains both common and unique modality information. Further, ③ is notably lower than ②, indicating that compared with the non-aligned feature $z$, the inconsistent information in the aligned $\overline{z}$ is indeed reduced.

References:

[1,2,3] correspond to [32,27,4] in our paper

[4] Similarity of neural network representations revisited, ICML 2019

I appreciate the authors' responses, which addressed my concerns well. After reading other reviews and responses, I decided to update my rating from Borderline Accept to Weak Accept.

Please make necessary modification and add the relevant clarification, discussion, and experiment results to your final paper.

Thank you so much for the time and effort invested in reviewing our work. The positive remarks are truly appreciated, and we feel encouraged by the feedback. We address the points raised in the comments as follows:

The authors report the episode to return and driving distance but do not report DS/RC/IP, is there any reason why the authors do not report those metrics?

Thanks for the thoughtful question on the metrics. Our evaluation protocol on CARLA follows the commonly adopted one in many existing RL works [1,2,3,4]. Different from the setting in TransFuser [5] (which reports DS/RC/IP), the RL agent's goal in this protocol is to drive as far as possible in limited timesteps without colliding into other moving vehicles or barriers. Each episode (i.e., evaluation trial) immediately ends when the agent vehicle collides or reaches the timestep limit. There is no predefined route during evaluation, so RC cannot be calculated. Because each episode immediately ends after a collision, the calculation of IP also becomes impossible. Since RC and IP are both impractical to be obtained, DS cannot be provided. Instead, the RL agent’s performance is primarily evaluated by Episode Reward (ER), which accounts for driving distance and driving stability, such as less abrupt steering.

Note that although we compare with TransFuser in our paper, we evaluate it under the RL protocol rather than its original protocol. As we explained in the paper, this involves integrating the modality fusion and alignment modules of TransFuser with a baseline RL algorithm (SAC). The original setting in TransFuser and the RL setting in our work represent distinct approaches to training autonomous driving agents. The evaluation protocols and metrics for these two settings are also different, which is why we did not report DS/RC/IP scores.

[1] SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies, ICML 2021

[2] Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning, NeurIPS 2023

[3] Model-Based Reinforcement Learning with Isolated Imaginations, TPAMI 2023

[4] Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning, NeurIPS 2023

[5] TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving, TPAMI 2023

L265-267 states that $\mathcal{L}\_{fp}$ and $\mathcal{L}\_r$ do not benefit much to the existing DeepMDP and SPR. Are there any supporting references/results for this claim?

Thanks for identifying this issue. There might be a misunderstanding here. Specifically, we are not trying to use $\mathcal{L}\_{fp}$ and $\mathcal{L}\_r$ to benefit DeepMDP and SPR. In L265-267, we state “However, this enhancement does not bring a significant advantage over conventional methods such as DeepMDP and SPR”. We mean that although $\mathcal{L}\_{fp}$ and $\mathcal{L}\_r$ can improve the performance of the baseline model, the improved results are still not significantly better than the results of DeepMDP and SPR (reported in Table 1 of our paper). So, we are describing a cross-table comparison, by comparing the $+\mathcal{L}\_{fp}$ and $+\mathcal{L}\_r$ rows in Table 3 of our paper and the DeepMDP and SPR rows in Table 1 of our paper. We did not mention Table 1 in L264-267, which may cause confusion. We appreciate your detailed examination and will correct this issue in the revised paper.

Do the authors think that incorporating a component of their modeling, e.g., $\mathcal{L}\_{tp}$, into existing models could result in a boost?

We are confident that our modeling such as cross-modality transition prediction $\mathcal{L}\_{tp}$ can boost existing methods. This is because our method is flexible and is not limited to any particular RL model or network architecture. To verify the effectiveness of our method on other models, we further apply $\mathcal{L}\_{tp}$ to DrQ [1] and evaluate its performance. The results are given in Table 2 of the rebuttal PDF, which show that $\mathcal{L}\_{tp}$ can also boost the performance of existing RL models.

[1] Image augmentation is all you need: Regularizing deep reinforcement learning from pixels, ICLR 2020

In Section 3.2 [L166], the authors suggest replacing a stronger objective (Eq. 6) with Eq. 7. Did you observe in your experiments that, indeed, a model trained with Eq.7 is better than Eq.6?

Yes, we have compared Eq.6 and Eq.7 in Sec.4.4 (L304-L313) of our paper. The results are presented in Fig.7 of our paper, which show that models trained with Eq.7 consistently perform better than models trained with Eq.6.

We would like to express our sincere gratitude for the insightful comments. We also deeply appreciate the suggestions regarding presentation and experimentation. Below are our responses to your concerns:

[W1] Some implementation and experiment details are not clear.

Please refer to our responses to [Q1-Q5] for detailed clarification.

[W2] Connections to related work need to be emphasized.

Thank you for the suggestion. Here we discuss the connections in detail:

(i) Connections with dynamics modeling methods [a-d]

The primary similarity between [a-d] and our DDM is the state and reward prediction strategy commonly used in RL dynamics modeling. However, DDM differs as it is tailored for multi-modal RL, introducing novel approaches in both modality decomposition and dynamics modeling.

Specifically, both DeepMDP [a] and DDM predicts latent states and rewards. However, DDM does not address modality relationships or learn state transitions in a modality-aware manner. SPR [b] uses multi-step state prediction but it overlooks modality interactions. Dreaming [c] mainly focuses on learning over different timesteps and samples. Differently, DDM aims at dissecting features across different modalities. DBC [d] employs a bisimulation metric to model dynamics, which differs from our dissected modeling strategy.

(ii) Relation to existing consistency extraction methods

Although the goal of drawing modality features closer for consistency extraction is similar between DDM and existing methods, the techniques vary significantly. MAIE [32] directly minimizes modality features at the same timestep. In contrast, DDM predicts cross-modality transition across adjacent timesteps. TransFuser [5] and EFNet [40] utilize attention, MUMMI [4] optimizes mutual information, and HAVE [20] employs attention and hypernetworks. Like MAIE, these methods do not create a synergy between dynamics modeling and modality alignment, which is a unique feature of the cross-modality transition prediction in our DDM.

(Reference numbers are the same as in our paper)

[W3] No theoretical support is provided.

Thank you for emphasizing the need for a theoretical analysis. We have conducted initial research to find theoretical insights that support our approach.

Our investigation suggests that information theory [1] might provide a suitable theoretical framework for our study. For instance, research in [2] shows how multiple sources provide structured multivariate information about a target variable. Analogously, in multi-modal RL, each modality can be treated as a source and the agent's action as the target. Based on this, we aim to establish a lower bound on the information gap between using only modality-consistent features versus all available information. Deriving this lower bound will underscore the importance of both modality consistencies and inconsistencies in decision-making, thus supporting our method design and empirical findings.

However, despite our earnest efforts on this idea, establishing a definitive result remains challenging for us within the limited rebuttal timeline. We sincerely value your feedback and will deepen this theoretical analysis in future work.

[1] The Mathematical Theory of Communication, Bell System Technical Journal, 1948

[2] Nonnegative Decomposition of Multivariate Information, arXiv, 2010

[Q1] Are gradients of $z$ stopped in (13)?

Sorry for not clarifying this. We stop the gradient of all prediction target features in (4), (5), (8), and (11). That is, for each loss term of the form $||X(z_a)-z_b||^2$, where $X$ represents prediction head and $z_a$, $z_b$ are modality features, we stop the gradient of $z_b$.

[Q2] 1. Are $z_{t+1}$ in (5) and (11) used the same encoder as $z_t$? 2. Are gradients taken through $z_{t+1}$?

1. Yes, they are. We do not utilize a moving average encoder like in SPR [b] but rather use the same online encoder while stopping the gradient of the target values like in DBC [d].

2. The gradients are stopped as explained in Q1. This avoids potential model collapse, which is observed in SPR paper when no stop gradient operation is applied to the target values.

([b,d] are the same as in [W2])

[Q3] What are single modality baseline results? Any results using only RGB inputs?

The results consider a combined input of all modalities. Following the advice, we have tested the case where only RGB inputs are used. The results are in Table 1 in the rebuttal PDF, which shows that the baseline with multi-modal input indeed outperforms RGB.

[Q4] Why use DeepMDP, SPR and DrQ as baselines instead of DreamerV3 and DrQ-v2?

For single-modality baselines, we would like a direct comparison of the dynamics modeling technique itself and reduce the impact of other factors (e.g., the RL algorithm and the use of planning mechanism or not). Therefore, we choose DeepMDP, SPR and DrQ, all use SAC as RL algorithm without planning, just like our method. We appreciate the recommendation of DreamerV3 and DrQ-v2 and have conducted experiments with them. As shown in Table 2 of the rebuttal PDF, DrQ-v2 obtains stable performance, while DreamerV3 gets inferior results on the Cartpole swingup_sparse task. This is probably caused by the sparse reward and modality inconsistencies, which hampers the learning of its transition model.

[Q5] Has the performance converged?

Our setting of training steps follows DBC [d] and SECANT [1]. To verify the convergence, we further train different methods with double steps (i.e., 200K/1M for CARLA/DMControl). The results are given in Fig.1 (a)-(d) in the rebuttal PDF. The figure shows that the improvement beyond 100K/500k steps becomes exceedingly marginal. Therefore, our current training can achieve a satisfactory level of performance.

[d] is the same as in [W2]

[1] SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies, ICML 2021

Minor: Inon.-> Incon.

Thanks for spotting this, we will correct it.