Rethinking Latent Redundancy in Behavior Cloning: An Information Bottleneck Approach for Robot Manipulation

Please open link to anonymous material for easy reference.

Comment 1 on Novelty: Limited novelty beyond adding a loss term to BC.

Our contributions go beyond simply adding a loss term to BC. Specifically, we highlight the following:

1. Method Novelty

(1) As shown in Figure 1 (a) of the anonymous material, traditional IB methods [1-3] typically apply the bottleneck to a single modality (e.g., images) and its latent features, with the information flow being O→Z→A. (2) In contrast, BC for robot manipulation is more complex than earlier control or single-modal tasks, requiring models to handle diverse, multimodal data and inevitably necessitating feature fusion. If strictly following the previous IB paradigm (as shown in Figure 2 of the anonymous material) would require constraining each input modality and its features individually, resulting in a pipeline that lacks elegance and scalability, is highly complex, and lacks the ability to capture cross-modal associations. (3) Thus, we propose redefining the "input" at the representation level and treating the features before the policy head as latent features, with the information flow being O→X→Z→A. The advantages of this approach are: 1) It allows us to apply redundancy reduction in a unified way across all modalities. 2) In practice, encoders are often frozen, making this approach more effective in such cases. 3) It better scales to a wider range of robotic algorithms.

2. Theoretical Novelty

As shown in Figure 1 of the anonymous material, we adapt Theorems 4.1 and 4.2 to support the robot task with information flow of O→Z→A. To demonstrate that our paradigm remains compatible with these theorems, we present Theorem 4.3: Even if we optimize I(X;Z) by applying the bottleneck at an intermediate feature level X, as long as X preserves the essential structure of the original input o, we are effectively controlling I(O;Z), with the difference bounded by a small constant δ.

3. To support our claims, we conduct extensive experiments across various models and settings.

Comment 2 on Experiment Performance: The improvements are marginal in most cases.

The marginal improvements are not due to limitations of the IB theory, but rather due to:

1. Model capacity and task complexity: Our models, such as BC-ViLT+IB, have limited capacity (e.g., 10M parameters). They perform well on simpler tasks like LIBERO-Goal (80% success, 8% improvement), but show limited gains (2%) on more complex tasks like LIBERO-Long, due to the performance ceiling of the baselines. We illustrate this issue with Difusion Policy (The MLP head (1.14M parameters) of BC-Transformer is replaced with the DP head(90M)) experiments on LIBERO-Long.

2. Dataset and task complexity: In benchmarks like Cortexbench, particularly Trifinger (Figure 7 in the appendix), the visual input contains only the robot arm and objects, with minimal redundant information, so applying IB yields minimal gains. In contrast, LIBERO environments feature more distractors, complex backgrounds, and diverse tasks, where IB leads to more noticeable improvements.

Comment 3 on Experiments: No testing of out-of-distribution generalization.

1. Object position. We modify the positions of manipulated objects in MetaWorld of Cortexbench and perform 25 rollouts during testing.

2. Generalization on unseen tasks with more distractors. Refer to Comment 4.

Comment 4 on Experiments: Limited/unconvincing real-world evaluations.

Due to word limit, please refer to our response to Comment 8 from reviewer xKwq. The videos are in anonymous material.

Comment 5 on Method: In Figure 2.c, why does only the [CLS] embedding go into the policy head?

We follow the common practice in transformer-based architectures, where the [CLS] token serves as a global summary of the input sequence, as seen in prior works [4-6]. The [CLS] embedding aggregates information through attention layers, making it ideal for downstream tasks. It is then passed to the policy head to represent the fused multimodal information. This design is computationally efficient and empirically effective, as shown in prior vision-and-language modeling works.

Comment 6 on Writing: Explaining the transition from I(Z;A) to the regression loss in Equation (6) more clearly would improve the presentation.

We will revise the manuscript to explain this connection more clearly.

Thank you for your time and effort in reviewing our work. We will revise the paper based on the rebuttal to better present our work.

Reference

Please refer to Reviewer a6Qr's reference list.

Please open link to anonymous material for easy reference.

Comment 1 on Novelty of Method: simply apply the IB in BC and other works had done it [1-3].

1. Differences with [1-3]

(1) Papers [1] and [2] differ significantly from our idea of reducing mutual information to eliminate redundancy. Paper [1] loosely references the IB concept but lacks theoretical grounding, simply adding noise to control signals to shift reliance toward local perception. Paper [2] uses IB to identify critical states based on goal dependence, aiming to improve exploration rather than reduce redundancy.

(2) Paper [3] uses the IB to obtain task-relevant information for control, which is similar to our idea. We explain the differences in Point 2.

2. Our contribution

(1) As shown in Figure 1 (a) of the anonymous material, traditional IB methods [3-6] typically apply the bottleneck to a single modality (e.g., images) and its latent features, with the information flow being O→Z→A. (2) In contrast, BC for robot manipulation is more complex than earlier control or single-modal tasks, requiring models to handle diverse, multimodal data and inevitably necessitating feature fusion. If strictly following the previous IB paradigm (as shown in Figure 2 of the anonymous material) would require constraining each input modality and its features individually, resulting in a pipeline that lacks elegance and scalability, is highly complex, and lacks the ability to capture cross-modal associations. (3) Thus, we propose redefining the "input" at the representation level and treating the features before the policy head as latent features, with the information flow being O→X→Z→A. The advantages of this approach are: 1) It allows us to apply redundancy reduction in a unified way across all modalities. 2) In practice, encoders are often frozen, making this approach more effective in such cases. 3) It better scales to a wider range of robotic algorithms.

Comment 2 on Novelty of Theory: Simply applying theoretical claims to robotics does not constitute much novelty.

As shown in Figure 1 of the anonymous material, we adapt Theorems 4.1 and 4.2 to support the robot task with information flow of O→Z→A. To demonstrate that our paradigm remains compatible with these theorems, we present Theorem 4.3: Even if we optimize I(X;Z) by applying the bottleneck at an intermediate feature level X, as long as X preserves the essential structure of the original input o, we are effectively controlling I(O;Z), with the difference bounded by a small constant δ.

Comment 3: "Rethinking Latent Representations" is confusing.

Thank you for the feedback. We agree that the original title "Rethinking Latent Representations" may be confusing, and we will revise it to "Rethinking Latent Redundancy" in the revised version.

Comment 4: Slight improvements in success rate. For example, in Table 1 (if we remove the MetaWorld tasks), the improvements in all the other tasks are very small, often less than 1%.

Due to word limit, please refer to our response to Comment 2 from reviewer PrSW.

Comment 5: Slight improvements in success rate do not really show that the representations are more compact and more robust. And this paper does not really tell us much about the redundancy.

As discussed in Comment 4, Trifinger contains minimal redundancy, limiting IB’s impact (less than 1% gain). In contrast, LIBERO-Goal includes more clutter and complex backgrounds, where IB yields a larger improvement (~8%). This observation suggests that the representations in certain environments, especially those with more visual distractors and complex backgrounds, tend to contain higher redundancy, allowing IB to be more effective.

As shown in the attention map (Figure 3 in anonymous material), IB helps the model focus on task-relevant regions (e.g., the arm and target object) by suppressing attention to redundant background features—an effect less evident without redundancy reduction, highlighting IB’s distinct role beyond standard regularization.

Comment 6: LIBERO benchmark is not really a challenging multi-task benchmark since it has little task instruction diversity.

While LIBERO may have limited instruction diversity, it is a widely used benchmark in goal-conditioned manipulation and is sufficiently challenging for our study. The perceptual redundancy present in LIBERO allows IB to demonstrate clear benefits, as reflected in the significant performance gains observed in our experiments.

Furthermore, We conduct more challenging real-world experiments. Due to word limit, please refer to our response to Comment 8 from reviewer xKwq. The videos are in anonymous material.

Thank you for your time and effort in reviewing our work. We will revise the paper based on the rebuttal to better present our work.

Reference

Please refer to Reviewer a6Qr's reference list.

Thanks for your detailed review of this paper! We hereby address your concerns as follows.

Comment 1 on Writing: The claimed improvements from incorporating IB are overstated.

Thanks for the feedback. We will emphasize the consistency of improvements rather than their magnitude in the revised manuscript.

Comment 2 on Experiment Performance: Why is the performance on LIBERO so low, and why does IB underperform in complex, long-horizon tasks?

The relatively low performance on LIBERO, especially on the LIBERO-Long, is attributed to the following factors:

1. Differences in data processing:

In OpenVLA, the image observations are saved at a resolution of 256×256 (instead of 128×128) with additional filtering (e.g., removing "no-op" (zero) actions and unsuccessful demos). We use the raw LIBERO data with lower-resolution images without filtering.

2. Differences in model capacity:

OpenVLA has around 7B parameters, and diffusion policies may have around 100M parameters. Our models such as BC-ViLT+IB contain only around 10M parameters.

As a result, our models are less effective in complex, long-horizon tasks like LIBERO-Long.

Comment 3 on Method: The performance gains on LIBERO are uneven, raising concerns about the effectiveness of the approach.

This limitation arises not from the IB theory itself, but from the mismatch between model capacity and task complexity. Our lightweight baseline models perform well on simpler tasks like LIBERO-Goal, but their limited capacity constrains performance on more complex tasks like LIBERO-Long. As a result, the potential gain from incorporating IB is limited. We illustrate the issue with experiments on LIBERO-Long in Comment 5. (Due to word limit, please refer to our response to Comment 2 from reviewer PrSW for further discussion.)

Comment 4 on Essential References: This paper does not adequately contextualize IB in relation to recent advances in large-scale pretrained vision-language-action models like RT-2[1] and OpenX-Embodiment.

First, we clarify that VLAs like RT-2[1] and OpenVLA[2] are mentioned in the introduction to highlight our motivation, and we also discuss them in the limitations section. Second, our work focuses on improving generalization within a given model architecture by optimizing the training paradigm through information compression, rather than designing a new large model. The IB method can also be incorporated into VLA-based models.

Comment 5 on Baselines: Include more established baselines rather than design baselines all by themselves.

We include two more baselines. OpenVLA[3] (4 A100 GPU, 80K steps, 32 batch size) and Difusion Policy (The MLP head (1.14M parameters) of BC-Transformer is replaced with the DP head(90M)). Each task is rolled out 20 times during testing.

Rationale for baselines: Our most baselines are consistent with previous works like[4,5]. Our custom baseline, BC-MLP, complements the experimental setup by introducing a spatial fusion approach, as the first three baselines rely on temporal fusion.

Comment 6 on Experiments: The manuscript lacks sufficient empirical evidence to support the claim that redundancy is the primary limitation.

The attention map in Figure 3 of anonymous material shows that without redundancy reduction, the model focuses on areas beyond the robotic arm and target object.

Comment 7 on Experiments: Providing a systematic sensitivity analysis of key hyperparameters.

The complete experimental hyperparameters are in Appendix Tables 3 to 5. All IB-related hyperparameters, except for the Lagrange multiplier β, remain consistent across experiments. The range of β is reported in Finding 8, with exact values provided in Appendix Tables 6 to 9.

Comment 8 on Experiments: More diverse real-world examples.

We conduct more challenging real-world experiments focusing on:

1. Increased distractor objects and random positions.
2. Training and evaluating a multi-task policy.
3. Investigating generalization ability (*denotes unseen, which refers to instances not observed during the SFT stage).

We use 8 A100 GPUs to train both CogAct[3] and CogAct+IB for 8K steps on our newly collected dataset. The batch size is 128. Each task is tested 10 times. Videos are in anonymous material.

Thank you again for your time and effort in reviewing our work. We will revise the paper based on the rebuttal to better present our work.

Reference

Please refer to Reviewer a6Qr's reference list.

Thanks for your affirmative review of this paper! We address your questions as follows.

Comment 1 on Writing and Organization: Some of the less important findings (e.g. finding 9) are unnecessarily highlighted.

We sincerely appreciate the reviewer’s feedback on refining the presentation of our findings.

Our intention in structuring the findings as they are was to provide a comprehensive and systematic analysis of both the impact and potential applications of the IB in BC. For Finding 9, in robot manipulation, it is common practice to fine-tune a generalist model on domain-specific data to better adapt to new tasks. Our intention was to emphasize the practical value of our method in this context, showing that it can be effectively integrated into such fine-tuning processes commonly encountered in real-world scenarios.

That said, we acknowledge the reviewer’s concern and will revise the manuscript accordingly by highlighting only the most essential findings and integrating more general observations into the discussion section to improve focus and readability.

Comment 2 on Theoretical Clarification: Finding 2 implies that the latent representation Z derived from input X is redundant, making information compression essential for performance improvement. Can it be understood that feature learning is key for action decoding, and that redundancy between X and Z can be addressed through optimized feature learning?

1. Yes, feature learning itself is crucial for action decoding.

Most existing approaches aim to learn a generalizable representation and typically focus on scaling up data and model capacity. For example, many recent methods leverage larger datasets to train larger models, such as vision-language-action (VLA) models [1]. Fundamentally, these methods aim to learn more generalizable features Z by ensuring that the input space X covers a sufficiently broad distribution in the mapping X→Z.

Others incorporate additional sources of information to guide the learning process—for instance, CoTPC [2] introduces richer textual context, and ATM [3] integrates visual information from robot trajectories to guide robotic actions. These methods essentially introduce additional information to guide or constrain the learning of Z.

In contrast, our approach takes a different perspective. Instead of adding more information, we focus on reducing redundancy in the input to learn more compact and task-relevant representations.

2. Yes, the redundancy between the input X and the latent representation Z can be mitigated through optimized feature learning.

For instance, we use I(Z;X) to measure the amount of information transferred from X to Z, and aim to minimize I(Z;X) while ensuring that I(Z;A) increases, in order to ensure that the information being reduced is redundant. This approach is essentially based on the IB framework. It encourages the model to retain only task-relevant information, while discarding irrelevant or redundant information, thereby reducing overfitting and improving generalization (as illustrated in Figure 3 in Link).

Finally, thank you again for your time and effort in reviewing our work. We will revise the paper based on the rebuttal to better present our work.

Reference

Reviewer a6Qr

[1]Rt-2: Vision-language-action models transfer web knowledge to robotic control. [2]Chain-of-thought predictive control. [3]Any-point trajectory modeling for policy learning.

Reviewer xKwq

[1]RT-2: Vision-language-action models transfer web knowledge to robotic control. [2]Openvla: An open-source vision-language-action model. [3]Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation. [4]Spa: 3d spatial-awareness enables effective embodied representation. [5]Learning manipulation by predicting interaction.

Reviewer H7Dt

[1]MeMo: Meaningful, modular controllers via unformation bottlenecks. [2]Infobot: Transfer and exploration via the information bottleneck [3]Learning task-driven control policies via information bottlenecks [4]Learning representations for neural network-based classification using the information bottleneck principle. [5]Multi-view information-bottleneck representation learning. [6]Protecting your llms with information bottleneck.

Reviewer PrSW

[1]Learning representations for neural network-based classification using the information bottleneck principle. [2]Multi-view information-bottleneck representation learning. [3]Protecting your llms with information bottleneck. [4]Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation. [5]CALAMARI: Contact-aware and language conditioned spatial action MApping for contact-RIch manipulation. [6]Predictive inverse dynamics models are scalable learners for robotic manipulation.