Blindfolded Experts Generalize Better: Insights from Robotic Manipulation and Videogames

We thank the reviewer for the insightful comments. We are pleased that our work is found to be "intuitive and powerful" and recognized for its "immediate practical implications for robotics and other fields". In the following, we provide detailed replies to the questions and concerns.

How to quantify the mutual information I(Z;T)?

This is a very good point. We can still lower bound the information between observations and tasks using a variational approach -- a classifier that predicts task identity from the observations [1]. We believe that this bound will be lower when we add the mask, and may be used in the future to devise better masks. Of course, whatever information the human gleans from the observation is not as easy to quantify.

How to design the blindfold?

The study of principled methods for automatic design of the blindfold is an interesting direction for future work.

Here we can provide some insights on how we designed the blindfold in practice. Take the peg insertion case, in which the blindfold is a mask on the image observation. We tweaked the mask to hide most information that still allows the human to solve the task. At that point, removing even more information would not have produced meaningful demonstrations. This heuristic can be replicated by practitioners in other domains.

Question 1.

We expect the sensitivity to the degree of blindfolding to follow the theoretical result in Theorem 3.1: More blindfolding leads to more generalization. However, if the blindfold is so severe that the expert's success rate drops, we do not expect the cloned policy to work well. Recall that we are doing behavioral cloning (and not offline RL), so the performance of the policy is limited by the performance of the expert.

Question 2.

We tested a single implementation of the mask, so we do not have a sensitivity analysis. The idea of adding noise is interesting for this matter: We can add experiments with different noise added to the observations (proprioceptive noise is less relevant as the demonstrator can see the robot in the image, but we can add image noise instead of masking the hole).

Address limitation: Beyond monolithic goals.

We acknowledge that future studies are required to determine the full potential of blindfolding. That being said, our method is not a silver bullet for all generalization problems.

Nevertheless, we provide an additional experiment with some glimpses of the challenges the reviewer is suggesting. In Procgen Heist, the agent has to solve a sequence of sub-goals in order to solve the task. At each level, the agent needs to collect a gem hidden behind a network of color-coded locks accessible through keys (of corresponding colors) that are scattered throughout the level (See [1] for more details on the environment setting). This task requires multi-stage planning in the form of lock-waypoints and to overcome variations in sub-task ordering, making the task more complex and challenging than the regular maze. For training the BC policies, we collected human demonstrations as in the Maze experiment, applying a similar blindfold to the blindfolded experts. In Table 1 below, we compare the results of the agent trained with the fully informed expert's data ($\pi_{BC}$) to the results of the agent trained on the blindfolded experts' data ($\pi_{BF-BC}$). While the training performance is above 90% for both agents, $\pi_{BF-BC}$ significantly outperforms $\pi_{BC}$ on unseen test levels (43% vs. 7%), which indicates better generalization performance (results obtained over $4K$ trajectories from $200$ different seeds).

Table 1: Performance comparison on the Procgen Heist experiment. The mean and std are computed over 10 seeds. Top performer in bold.

References

[1] Cobbe, Karl, et al. "Leveraging procedural generation to benchmark reinforcement learning." International conference on machine learning. PMLR, 2020.

We thank the reviewer for the insightful comments and for considering our approach to be effective and well-founded. We are reporting below detailed replies to the questions and concerns.

Task diversity.

We include additional results on Procgen Heist to increase the task diversity in the paper. In Heist, the agent needs to collect a gem hidden behind a network of color-coded locks accessible through keys (of corresponding colors) that are scattered throughout the level (See [1] for more details on the environment setting). This task requires multi-stage planning in the form of lock-waypoints and to overcome variations in sub-task ordering, making it significantly more complex than the regular maze. For training the BC policies, we collected human demonstrations as in the Maze experiment, applying a similar blindfold to the blindfolded experts. In Table 1 below, we compare the results of the agent trained with the fully informed expert's data ($\pi_{BC}$) to the results of the agent trained on the blindfolded experts' data ($\pi_{BF-BC}$). While the training performance is above 90% for both agents, $\pi_{BF-BC}$ significantly outperforms $\pi_{BC}$ on unseen test levels (43% vs. 7%), which indicates better generalization performance (results obtained over $4K$ trajectories from $200$ different seeds).

Table 1: Performance comparison on the Procgen Heist experiment. The mean and std are computed over 10 seeds. Top performer in bold.

How the approach generalizes to more complex or diverse task families.

For robotic applications, a simple "rule of thumb" we found is -- can you solve it in the dark? Many important tasks fall under this criterion, including grasping, assembly, and disassembly. This is not exhaustive, and additional blindfold techniques may be found in future studies to extend the applicability of the idea.

On this line, we mention an interesting (yet speculative) application to LLM reasoning. When answering a complex question (e.g., "what is the tallest tower in Europe"), we must balance between recalling from memory, and performing some reasoning computation ("what are the countries in Europe, what is the tallest tower in each European country, etc."). For supervised fine tuning, if we manage to induce the humans to provide more reasoning traces, they may lead to better generalization. We may "blindfold" the humans by asking them not to rely on memory, for instance, but more on tool use and reasoning.

Increased demonstration effort offsetting data efficiency gains

This is a valid point, for which we dedicated an ablation study in the manuscript (Figure 3). Since the blindfolded experts' trajectories are longer, we collected more trajectories with fully-informed experts ($\pi_{BC-ext}$) to match the total number of samples of the blindfolded expert. The results are mostly the same: This means that the blindfolded approach generalizes better with the same number of samples and therefore the same time for collecting the data.

Q1. How might you automatically determine the optimal "blindfolding" strategy for a new domain without domain-specific engineering?

This is an interesting direction that would require substantial additional study, which we leave as future work.

We can provide some insights on how we designed the blindfold in practice. Take the peg insertion case, in which the blindfold is a mask on the image observation. We tweaked the mask to hide most information that still allows the human to solve the task. At that point, removing even more information would not have produced meaningful demonstrations. This heuristic can be replicated by practitioners in other domains.

The latter requires some engineering, but note that any BC experiment requires engineering anyway (of the data collection setup, the human demonstrator instruction, etc.), and we believe blindfolds will become another factor in this list.

Q2. How does this approach compare to simply collecting more diverse demonstrations from standard experts? Is there a point where additional task diversity would make the blindfolding unnecessary?

If we understand the reviewer's intention correctly, "more diverse demonstrations" refers to "demonstrations of more tasks". The intuition is correct: When demonstrating more and more tasks, the gap between blindfolded and standard experts shrinks. This is supported both by the theory and the experiments.

The theory shows that the generalization error scales with $O (\sqrt{I_{Z;T} / m})$, where $m$ is the number of demonstrated tasks. When $m$ is large enough, the effect of a smaller $I_{Z;T}$ is reduced.

The experiment on peg insertion shows that the generalization error of the policy cloned from regular demonstrations shrinks slightly when increasing the training tasks from 2 (Figure 5 left) to 5 (Figure 5 right). The gap with the blindfolded strategy remains significant, but we expect it to reduce with even more demonstrated tasks.

References

[1] Cobbe, Karl, et al. "Leveraging procedural generation to benchmark reinforcement learning." International conference on machine learning. PMLR, 2020.

We appreciate the reviewer's constructive comments. We discuss the raised points below.

It is difficult to quantify the mutual information since we don't know the expert's internal representation Z.

This is a very good point. We can still lower bound the information between observations and tasks using a variational approach -- a classifier that predicts task identity from the observations [1]. We believe that this bound will be lower when we add the mask, and may be used in the future to devise better masks. Of course, whatever information the human gleans from the observation is not as easy to quantify.

Task diversities and more variation of tasks.

As we discuss in the paper, not all task variability can be improved by our method. Following Theorem 3.1, we need the blindfold to reduce task information, but still allow the human to solve the task. In robotics, a simple "rule of thumb" for such tasks is -- can you solve it in the dark? Many important tasks fall under this criterion, including grasping, assembly, and disassembly. The tasks in Meta-World and Libero do not -- the observation contains the task description itself -- what to do and where to move, and they are impossible to solve without a very detailed observation. How to reduce task information in such domains is still an open problem.

In the meantime, we provide an additional experiment on Procgen Heist game to have more variation of tasks in the paper. In Heist, the agent needs to collect a gem hidden behind a network of color-coded locks accessible through keys (of corresponding colors) that are scattered throughout the level (See [2] for more details on the environment setting). This requires multi-stage planning in the form of lock-waypoints and to overcome variations in sub-task ordering, making it significantly more complex than the regular maze. For training the BC policies, we collected human demonstrations as in the Maze experiment, applying a similar blindfold to the blindfolded experts. In Table 1 below, we compare the results of the agent trained with the fully informed expert's data ($\pi_{BC}$) to the results of the agent trained on the blindfolded experts' data ($\pi_{BF-BC}$). While the training performance is above 90% for both agents, $\pi_{BF-BC}$ significantly outperforms $\pi_{BC}$ on unseen test levels (43% vs. 7%), which indicates better generalization performance (results obtained over $4K$ trajectories from $200$ different seeds).

Table 1: Performance comparison on the Procgen Heist experiment. The mean and std are computed over 10 seeds. Top performer in bold.

Q1. Elaboration on the transition from (5) to (6) in Theorem 3.1.

The derivation in Eq. 6 is $\mathcal{E}\_{gen} (\hat{\pi}) \leq \mathcal{E}\_{gen} (\pi^E)+ \mathbb{E}\_{T \sim P\_0} \mathbb{E}\_{XA \sim \mathbb{P}^{ \pi^E}\_{T}} [1 (\hat\pi (X) \neq A)].$ This is obtained by noting that the suboptimality of the cloned policy $\hat \pi$ cannot be larger than the suboptimality of the expert's policy $\pi^E$ plus a term that accounts for every time $\hat\pi$ is different from $\pi^E$. Then, Assumption 2 allows us to write the suboptimality of the expert's policy as $\mathcal{E}_{gen} (\pi^E)$.

References

[1] Chen et al., Infogan: Interpretable representation learning by information maximizing generative adversarial nets. NeurIPS 2016.

[2] Cobbe, Karl, et al. "Leveraging procedural generation to benchmark reinforcement learning." International conference on machine learning. PMLR, 2020.

I sincerely thank the authors for their thorough response. It has addressed most of my major concerns and questions that I raised in my original review.

We can still lower bound the information between observations and tasks using a variational approach -- a classifier that predicts task identity from the observations [1]. We believe that this bound will be lower when we add the mask, and may be used in the future to devise better masks.

This variational approach would be a very interesting idea for future work.

Regarding the authors' explanation of equation (6), my interpretation is as follows: $\mathbb{1}(\hat{\pi}(X) \neq A) \leq \mathbb{1}(\pi^E(X) \neq A) + \mathbb{1}(\hat{\pi}(X) \neq \pi^E(X))$ for any $X$ and $A$. I verfied myself that it is true, but I want to make sure my interpretation is correct.

While the authors admit how to reduce task information in language-conditioned settings is still an open problem, such an acknowledgement of limitation should be rewarded rather than punished since it improves clarity of the paper. (Please discuss this point in the final version.) The new Procgen Heist experiment demonstrates strong generalizability of blindfolded experts in tasks that require long-horizon and complex interaction with the environment. Thus, I have raised the clarity and the significance scores.

We thank the reviewer for the thoughtful comments and for recognizing the key strengths of our work in its novelty, theoretical analysis, and experiments. Below we provide detailed answers to the questions and comments.

"Experimental environments are relatively simple".

Our work tackles generalization across tasks, and therefore, we focus on domains in which generalization is the main challenge, rather than other factors that may confound the analysis of generalization (such as control or the need to plan for long horizons). In our experiments, most of the complexity stems from the limited variability of training tasks, which produce a cloned policy that memorizes the data. The resulting model lacks the ability to generalize to unseen test tasks - which we show can be resolved with the introduction of blindfolds (Figures 3 and 5).

Nevertheless, note that the peg insertion task is not as easy as it seems: It is a contact-rich manipulation task -- not trivial, and much harder than common pick-and-place manipulation tasks.

Regarding the maze experiment, we provide an additional experiment of Procgen Heist game, in which the levels are more challenging compared to a standard maze. At each level, the agent needs to collect a gem hidden behind a network of color-coded locks accessible through keys (of corresponding colors) that are scattered throughout the level (See [1] for more details on the environment setting). This requires multi-stage planning in the form of lock-waypoints and overcoming variation in sub-tasks (key color) ordering. We apply the blindfold similarly to the Procgen maze and follow the same experimental setting as in the paper. In Table 1 below, we compare the results of the agent trained with the fully informed expert's data ($\pi_{BC}$) to the results of the agent trained on the blindfolded experts' data ($\pi_{BF-BC}$). While the training performance is above 90% for both agents, $\pi_{BF-BC}$ significantly outperforms $\pi_{BC}$ on unseen test levels (43% vs. 7%), which indicates better generalization performance (results obtained over $4K$ trajectories from $200$ different seeds).

Table 1: Performance comparison on the Procgen Heist experiment. The mean and std are computed over 10 seeds. Top performer in bold.

"It would be beneficial to provide more qualitative results".

Thanks for the valuable suggestion. We computed two additional metrics: Map coverage score (Table 2) and the entropy of state visitation (Table 3).

Map coverage score is the ratio $C=N_v/N_{total}$ given by the number of visited states $N_v$ divided by all accessible states $N_{total}$, averaged over all episodes. For the peg insertion experiment, we consider the rotation of the robotic arm around the Z-axis as the crucial component of the state space for obtaining the correct articulation for insertion. We compute the ratio of the rotation performed (in radians) divided by $2\pi$ in each trajectory, averaged over all trajectories.

Entropy of the state visitation is defined by $H= -\sum_s p(s)\log p(s)$. We calculate $p(s)$ using a histogram (with 20 bins) of rotation angles along the trajectory, averaged over all trajectories.

The results confirm that blindfolded experts explore a larger portion of the state space to compensate for the redacted information in the observations.

Table 2: Map coverage score of the trajectories demonstrated by fully-informed experts (expert) and blindfolded experts (BF-expert).

Table 3: Entropy of the state visitation of the trajectories demonstrated by fully-informed experts (expert) and blindfolded experts (BF-expert).

Extension of the theory to continuous action space.

Robotic manipulation tasks typically allow for modeling with discrete action spaces due to the inherent smoothness of the physical world. Notably, even advanced state-of-the-art models (e.g., OpenVLA [2]) are trained by mapping continuous actions into discrete tokens. This makes our theory applicable to the experimental setting in the paper and many other settings. We do, however, see the value of the reviewer's suggestion to extend the analysis to a continuous action space. That said, the theory of imitation learning with continuous actions is known to be highly non-trivial, and hardness-analysis has been derived recently [3]. We leave settling this open question to future work.

Q1. Optimal trade-off between generalization and sub-optimal behavior. Principled method or insight for finding this optimal trade-off?

Any BC experiment requires some engineering (of the data collection setup, the human demonstrator instruction, etc.), and we believe blindfolds will become another factor in this list.

We can provide some insights on how we designed the blindfold in practice. Take the peg insertion case, in which the blindfold is a mask on the image observation. We tweaked the mask to hide most information that still allows the human to solve the task. At that point, removing even more information would not have produced meaningful demonstrations.

Further studies on principled methods for the automatic design of the blindfold are a good direction for future work.

References

[1] Cobbe, Karl, et al. "Leveraging procedural generation to benchmark reinforcement learning." International conference on machine learning. PMLR, 2020.

[2] Moo Jin Kim et al. OpenVLA: An Open-Source Vision-Language-Action Model. 2024

[3] Thomas T. Zhang et al. Imitation Learning in Continuous Action Spaces: Mitigating Compounding Error without Interaction. 2025