CLIP-Guided Reinforcement Learning for Open-Vocabulary Tasks

Thank you for your efforts and valuable comments. We would like to address the weaknesses and questions you raised in the review.

Why wasn't EmbCLIP evaluated on unseen instructions in the hunt domain?

We do not evaluate EmbCLIP on unseen instructions in the hunt domain due to its limited performance on the training instructions. Since the model does not perform well on tasks it is trained for, we determine that it would not provide meaningful insights to test it on unseen instructions.

Why is "shear a sheep" considered an unseen instruction when it's part of the instructions seen during training?

We would like to clarify that 'shear a sheep' is not considered as an unseen instruction here. As shown in Figure 8, we do not highlight 'wool' in green. The inclusion of 'shear a sheep' in the generalization test is used to calculate the precision. As defined in the same paragraph, precision is calculated as the number of times correctly harvesting the specified target divided by the number of times harvesting any target declared in the group’s instructions. Therefore, 'shear a sheep' is grouped with 'shear a mushroom cow' since both tasks require the tool 'shears'. This grouping allows us to assess the precision of the model in correctly identifying the unseen target object 'mushroom cow', ensuring that the agent does not simply shear any object indiscriminately. Similarly, another training instruction 'milk a cow' is also included in a group along with an unseen instruction 'harvest water' for the same reason.

Comparison to imitation learning methods.

In our study, the comparison between COPL and the imitation learning methods, STEVE-1 and Cai et al., is due to the lack of alternative Minecraft foundation models with goal- or instruction-conditioned policies trained via reinforcement learning. We select STEVE-1 and Cai et al. for their capabilities in mastering multiple basic skills in Minecraft, similar to the objectives of our open-vocabulary evaluation.

We acknowledge that comparing an online RL method with imitation learning methods is not a perfect match since the training data is quite different. Therefore, our intention is not to determine which method performs best in learning basic skills in Minecraft. Instead, we just use STEVE-1 and Cai et al. as a reference to understand how COPL performs in this specific domain.

Smaller comments.

Thank you for your detailed suggestions. We will make improvements based on them in the revision.

If you have any additional questions or comments, please do not hesitate to let us know. We are more than happy to provide any further clarifications or information that may be helpful. If you find our responses satisfactory, we would be grateful if you could consider raising the score.

Dear reviewer, we are happy to receive your response and appreciate the opportunity to further discuss and address your concern.

While it's fine to propose a method with more assumptions and more limited applicability, the paper should make those assumptions clear. The paper currently claims that the proposed method solves open-vocabulary tasks, implying the method is applicable to all tasks that can be specified with language.

In the original version of our paper, we claimed "By open-vocabulary task, we mean that the agent is instructed to interact with diverse objects beyond the training scope." at the beginning of Section 2, by which we limit the scope of tasks that our method is designed to solve. We acknowledge, however, that this description may not be precise enough. Therefore, based on the comments from reviewers, we append an additional statement to provide a more restrictive claim on our assumption and application: "More specifically, we focus on object-centric tasks and the open-vocabulary ability over target objects.", in the latest revision. Thank you for bringing this to our attention, and we hope this clarification aligns with the paper's objectives more accurately.

Additionally, the proposed method only outperforms much simpler baselines (one-hot and CLIP embedding conditioning) in the hunting domain.

Firstly, we would like to highlight that our proposed method outperforms baselines not only in the hunt domain but also in the harvest domain. While the learning curves of the three models on training instructions appear comparable, as shown in Figure 8(a), the gap between our method and the baseline is significant on unseen instructions during test, as illustrated in Figure 8(c). This result demonstrates our method's superior ability to generalize to unseen instructions than EmbCLIP, suggesting that our proposed approach of taking 2D confidence map as a policy conditioning representation offers better open-vocabulary ability over target objects compared to relying on language descriptions.

Secondly, we think that using a one-hot vector as the task indicator (One-Hot) and conditioning policy on language (EmbCLIP) are not simple, but are essential instead. COPL, EmbCLIP, and One-Hot share the same network architecture and training procedures, thus enabling a clear and direct comparison of generalization ability between policy conditioning representations from different modalities (vision, language, and vector), excluding the influence of other factors.

... this proposed reward is only valid for tasks that involve moving towards a target object while the prior methods work for more general Minecraft tasks (MineClip, STEVE-1, Cai et. al).

We acknowledge the limitation of COPL in the aspect of generality, as we newly added description in Section 2. At last, we would like to offer a comprehensive perspective that combines both theoretical and empirical considerations here.

MineCLIP. Although theoretically, MineCLIP can align video clips with a wide range of language descriptions, besides object-centric tasks, previous work [1,2] and our experiments demonstrate its practical limitations. We find that in practice, MineCLIP still tends to align entities in video clips with those in language descriptions like the original CLIP, instead of utilizing temporal information (otherwise the MineCLIP reward would effectively guide the agent to approach targets in object-centric tasks).

Imitation learning. STEVE-1 struggles with accurately differentiating targets, as evidenced by its low performance in the hunt domain. Cai et al. (2023) show limited generalization ability over target objects, as illustrated in Figure 8(b), due to its condition on language input. This aligns with our observations on EmbCLIP. Theoretically, we can attribute these deficiencies to the limitation of the imitation learning dataset, instead of the methods themselves. However, from another perspective, reinforcement learning shows advantages here of being independent of the quality and distribution of the demonstration dataset.

In summary, no method can perfectly handle all tasks and perform zero-shot generalization. Our method could be regarded as a practical compromise that sacrifices some of MineCLIP's theoretical capabilities of generality but enhances the practical performance for object-centric tasks, which compose a large portion of tasks in Minecraft.

[1] Cai et al., 2023, Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction.

[2] Yuan et al., 2023, Plan4MC: Skill Reinforcement Learning and Planning for Open-World Minecraft Tasks.

For the “harvesting” domain, the benefits of the technique already appear smaller or not present.

A detailed explanation of the open-vocabulary evaluation in the harvest domain is in the answer to the first question.

... what stops us from directly using traditional reward shaping in the state space of the Minecraft world (not general purpose, but strong)?

If the goal is only to learn tasks in Minecraft, we agree that using traditional reward shaping in the state space of the Minecraft world, such as the 'manual reward' designed in the original MineCLIP paper, is a direct approach. But from our perspective, we wish our proposed methodology applicable in diverse environments, rather than only in Minecraft, for the problem defined in Section 2. As you succinctly summarized, "Essentially, the 'COPL' technique consists of a method for ... getting close to them", the whole framework is not tailored for Minecraft and can also be applied to other similar first-person view and object-centric environments such as robotic navigation. We use Minecraft as a convenient and suitable testbed to validate our method. Therefore, we avoid the use of environment-specific information to guide reinforcement learning.

Comparison with MineCLIP in terms of generality.

As mentioned above, our method could be viewed as a framework applicable to diverse environments, not limited to the specific use case of Minecraft. In Minecraft, we use MineCLIP to realize the first component of our method, "turning a suitable open-vocabulary image segmentation model into a 2d confidence map". We do not intend to build a new fundamental model or method that theoretically surpasses MineCLIP in terms of multimodal ability and generality in Minecraft. Instead, it just serves as a key module specific to Minecraft within our framework to solve the problem we defined in Section 2. In other first-person view and object-centric environments, our framework can adapt by replacing the modified MineCLIP with general models like Grounding DINO + SAM or domain-specific models.

If you have any additional questions or comments, please do not hesitate to let us know. We are more than happy to provide any further clarifications or information that may be helpful. If you find our responses satisfactory, we would be grateful if you could raise the score.

Thank you for your efforts and valuable comments. We appreciate such a detailed and insightful discussion on our method. We would like to address the weaknesses and questions you raised in the review.

Why are learning curves so similar for all methods in panel (a), but not in the panels (c) and (d), where MineCLIP seems to perform worse than COPL?

There might be a typo in the question, as 'MineCLIP' is not mentioned in Figure 8; we assume 'EmbCLIP' is intended.

In Figure 8(a), the learning curves show the success rates of three models on training instructions throughout the training process. The similarity in these curves indicates that all methods have comparable performance on the training tasks. The results demonstrated in Figure 8(c) and (d) include training instructions and unseen instructions (highlighted in green). The gap between COPL and EmbCLIP on training instructions ('milk a cow' and 'shear a sheep') is narrow and approximately aligns with the gap observed at the end of the training in Figure 8(a). However, the gap widens on unseen instructions ('harvest water', 'shear a mushroom cow', and 'harvest sand'), suggesting COPL's superior generalization ability.

Specifically, for EmbCLIP, while the text encoder can produce a language embedding for the novel target object, this embedding is essentially out-of-distribution for the policy network. This is because the language embedding space is huge but the policy network is only trained on a limited set, i.e., embeddings of four training target objects. In contrast, COPL's policy network takes a 2D confidence map as input. After our modified MineCLIP converts the unseen target name into a confidence map, the policy network can process it without a generalization gap.

We would like to highlight this novel use of a 2D confidence map as a policy conditioning representation is another contribution of our work, besides the intrinsic reward. Multi-task RL with the original MineCLIP model cannot acquire such a unified representation but only takes as input the language instruction, as implemented in the original MineCLIP paper.

The elaborated experiment settings are available in Appendix B.3.3.

Why is there no 'one hot' in panels (b), (c) and (d)?

No One-Hot in Figure 8(b): Figure 8(b) focuses on training instructions. Since COPL, EmbCLIP, and One-Hot exhibit similar performance on these training instructions, we omit EmbCLIP and One-Hot here for brevity. We provide success rates of these methods on each training instruction in Figure 17.

No One-Hot in Figure 8(c) and 8(d): Figure 8(c) and 8(d) are intended to present generalization performance on unseen instructions. However, One-Hot is only a multi-task baseline and cannot perform meaningful generalization test. In detail, it uses a one-hot vector as the task indicator, which is set to the same length as the number of training tasks. This design means that One-Hot lacks the flexibility to handle unseen instruction, as there is no appropriate task indicator.

From a cursory look, it seems that the MineCLIP baseline agent for tasks such as “hunt a cow” seems to severely underperform relative to the one from the original MineCLIP paper. Can you comment on this?

We appreciate your observation of this noticeable phenomenon that we did not mention in the paper but is worthy of analysis.

In the original MineCLIP paper, four tasks, 'milk a cow', 'hunt a cow', 'shear a sheep', and 'hunt a sheep', are trained together under a multi-task reinforcement learning framework. Among these four tasks, 'milk a cow' and 'shear a sheep' are relatively easier to learn than the others. Therefore, the MineCLIP reward likely guides the agent to successfully learn these easier tasks first, and then a behavior bias, moving towards a cow or sheep, is introduced into the policy network, facilitating the learning of more challenging hunt tasks. However, in our hunt task learning experiments, there are no such easy tasks to introduce behavior bias, resulting in the underperformance of MineCLIP reward. Other minor reasons include the fact that we do not use action smoothing and self-imitation learning and we employ a different PPO codebase.

Another noticeable point is that, as shown in its Table 1, multi-task RL in the original MineCLIP paper seems to overfit to cow-related tasks, possibly due to the use of language instructions as input. This challenge of correctly grounding language into visual targets is also observed in our experiment (EmbCLIP in Figure 7(a)). This demonstrates the advantage of our method using a unified 2D confidence map as the policy condition.

Additionally, Figure 13 demonstrates that our implementation of the MineCLIP reward successfully guides the agent to master harvest tasks, including 'milk a cow' and 'shear a sheep', confirming the correctness of our implementation.

Thank you for your efforts and valuable comments. We would like to address the weaknesses and questions you raised in the review.

Another example is the negative word list; while effective for denoising, it is determined in a domain-specific fashion, so it is unclear if the benefit of this procedure would be helpful for other domains.

We would like to clarify that constructing a negative word list is inherently domain-specific. For instance, in another domain, indoor navigation tasks, the list should be constructed to contain indoor objects rather than outdoor ones. Although the approach may vary with each domain, the concept of building a task-relevant negative word list to denoise and enhance segmentation effectiveness is universally applicable.

How sensitive is $\lambda$ across multiple task families? Is the same value of $\lambda$ optimal across multiple task families?

In our experiments, we observe that the average focal reward per step consistently falls into a range of 0.1 to 0.3 across all tasks due to the similar state spaces (all tasks are evaluated in Minecraft). The focal rewards' uniformity in order of magnitude across task families indicates the robustness of our $\lambda$ selection, although it is derived from 'hunt a cow' task and 'hunt a sheep' task. To validate this, we conduct additional experiments on two tasks from the harvest task family, 'milk a cow' and 'shear a sheep', and add the results to Appendix B.4. As shown in Figure 20(a) and (b), $\lambda=5$ still outperforms other values on these two tasks.

While we acknowledge that $\lambda=5$ may not be the exact optimal value for every individual task, it is a reasonably near-optimal choice (e.g. on the same order of magnitude as the optimal value), serving as a broadly effective parameter for our multi-task training.

How was the Gaussian kernel constructed? How sensitive is the performance of the method to the specific choice of kernel?

The Gaussian kernel in our method is constructed based on the simple principle that values should be higher near the center and decrease with distance. $\sigma=(H/3,W/3)$ is an intuitive choice, aiming to maintain high contrast between the center and the edge, as well as between the edge and areas outside the field of view.

To quantitatively evaluate this intuition and answer the second question, we conduct additional experiments to test the sensitivity of our methods to Gaussian kernels with different variances and add the results to Appendix B.4. The experimental settings are the same as those in Figure 6(c) and (d). As illustrated in Figure 21, $\sigma=(H/3,W/3)$ outperforms the other two choices. We suppose that the Gaussian kernel with $\sigma=(H/2,W/2)$ is too wide to provide sufficient contrast between the center and the edge, while the Gaussian kernel with $\sigma=(H/5,W/5)$ is too narrow to provide sufficient contrast between the edge and areas outside the field of view.

Why were natural language instructions not included as an input to the policy?

We do not include the natural language instruction as input to the policy primarily because the instruction cannot offer additional useful information beyond what is already provided by the 2D confidence map in our Minecraft experiments. The information in the language instruction, besides the target object, often consists of verbs like 'hunt', 'harvest', or 'collect'. However, such actionable information does not significantly aid in task completion. On the one hand, the action patterns required to complete tasks in Minecraft are relatively simple, i.e., move towards the target object and take 'attack' or 'use' action. On the other hand, the mapping between verbs and action patterns is not one-to-one. For example, an instruction with the verb 'collect' may require the agent to either 'attack' or 'use' depending on the type of the target object. Therefore, we believe that language instruction cannot provide additional useful information for the agent's policy.

We acknowledge that in other domains like robotic manipulation, language instructions often contain essential information, such as verbs 'pick' and 'push' indicating totally different behavior patterns. Under this circumstance, taking as input the language instruction will be necessary.

Given that the language instructions are fairly simple, what is the rationale for using an LLM to find the target object?

The use of an LLM is intended to address the challenge where the name of the target object is not explicitly mentioned in the language instruction. For example, consider the instruction 'obtain wool with shears'. Here, 'wool' is the target item to acquire, but the actual target object for the agent to approach and interact with is 'sheep', which does not appear in the instruction. It is hard to design a programmatical method to extract such an implicit target object from the instruction. Consequently, we resort to LLMs, which understand the underlying relationship between the target item (wool) and the target object (sheep). More examples are available in Appendix A.1.

If you have any additional questions or comments, please do not hesitate to let us know. We are more than happy to provide any further clarifications or information that may be helpful. If you find our responses satisfactory, we would be grateful if you could consider raising the score.

Thank you for your efforts and valuable comments. We would like to address the weaknesses and questions you raised in the review.

I am not exactly sure whether the authors claim the architecture in figure 2 to be a main contribution.

The architecture illustrated in Figure 2 is not the main contribution of our study but rather a key module within our overall framework. As introduced in Section 3.1, it is adapted from MaskCLIP. The differences between our modified MineCLIP and MaskCLIP are twofold: firstly, MineCLIP includes an additional temporal transformer in the vision pathway; secondly, our modified MineCLIP outputs a confidence map for the target object instead of a semantic segmentation result.

From my understanding, for this reward to work properly, the object has to be already in FOV, correct?

Getting a non-zero focal reward indeed requires the target object to be in the agent's field of view. When the target object is not present in the FOV, resulting in a confidence map with no high-probability patches, the focal reward is zero. This characteristic would prompt the agent to look around immediately after spawning in the Minecraft world to find the target object. In our experiments, target objects often appear in the agent's observable range, which means that the agent can see the target object when facing the direction of the target object.

If we consider a more challenging setting where the target object is located out of the agent's observable range at the spawning point, an additional exploration algorithm should be integrated to encourage the agent to move around. For example, incorporating novelty-induced exploration.

The core contribution of the paper is an intrinsics reward for minecraft with a lot of limitations ...

We acknowledge that our method is primarily designed for first-person view and object-centric tasks. But we want to clarify that this setting is not exclusive to Minecraft, but contains a wide range of realistic and general applications. For example, embodied AI research domains, such as autonomous driving, navigation, and robotic manipulation, often employ first-person views and/or focus on object-centric interactions. This setting better mirrors the complexity of real-world challenges compared to canonical RL environments, where our method may not be suitable.

Additionally, we would like to emphasize that our contribution also includes a novel policy conditioning representation, namely the 2D confidence map. Using a 2D confidence map to replace language input substantially improves the generalization ability of learned policy, as demonstrated in Section 4.2. Although currently restricted to object-centric tasks, it offers some insights about grounding language into vision to facilitate multimodal policy learning.

We hope this elaboration clarifies the generality and contributions of our work, while also acknowledging and appreciating the limitations you have pointed out.

... which breaks the purpose of MineCLIP.

As mentioned above, our method is primarily designed for first-person view and object-centric tasks, which are not restricted in Minecraft. We use Minecraft as a convenient and suitable testbed to validate our method and adapt MineCLIP to provide the segmentation functionality that our method requires. Therefore, we do not intend to develop a new fundamental model or method that follows the purpose of MineCLIP and theoretically surpasses it in terms of multimodal ability and generality in Minecraft. Instead, it just serves as a key module specific to Minecraft within our framework to solve the problem we defined in Section 2. In other first-person view and object-centric environments, our framework can adapt by replacing the modified MineCLIP with general models like Grounding DINO + SAM or domain-specific models.

If you have any additional questions or comments, please do not hesitate to let us know. We are more than happy to provide any further clarifications or information that may be helpful. If you find our responses satisfactory, we would be grateful if you could raise the score.

Dear authors,

However, as detailed in our newly appended Appendix A.4, due to the significant domain gap between visuals in Minecraft and the real world, direct application of these models to Minecraft results in inaccurate detection.

This is not True, because open-vocabulary detection easily generalizes in many video games just as well as CLIP (in fact many of them use CLIP). A while ago I tried grounded-SAM on minecraft. Its open-vocabulary detection worked perfectly. I just tried a few images from you paper's Figure 3, and it worked perfectly with a fast speed. The model successfully detected sheep, flower, cow, even the sword.

In addition, I personally tried every single figure in your Appendix figure 9, following one negative word of "grass" which you also did in method. Grounded DINO was able to detect the object 6/6. In your added figure 11, grounded dino got 4/6 correct from your result, but the original prompt I tried is "minecraft sheep" for (d) and it succeeded, so I'd consider open-vocabulary detection to work in 11/12 cases. I believe you can potentially get better results with higher resolution screenshots. I think such result proves they are strong replacements even without minecraft specific thing.

To addresses my concerns, I believe results with grounded SAM for example, or results in other FPV domains are necessary, which can be use to defend the merit of your method.

Dear reviewer, we appreciate the opportunity to further discuss and respond to your point about the use of open-vocabulary detection models in Minecraft or other first-person-view (FPV) games.

Minecraft. Firstly, we would like to clarify that currently, there is not an existing open-vocabulary detection model tailored for Minecraft. An alternative approach is to use some off-the-shelf open-vocabulary object detection models directly. However, as detailed in our newly appended Appendix A.4, due to the significant domain gap between visuals in Minecraft and the real world, direct application of these models to Minecraft results in inaccurate detection. In other words, they require adaptation to the Minecraft domain, implying additional costs associated with labeled data collection and fine-tuning. In contrast, our modification on MineCLIP does not demand further fine-tuning, allowing for immediate and effective implementation.

Other FPV games. Similar to Minecraft, other FPV games also suffer from the domain gap between their visual styles and the real world more or less. As mentioned previously, if we want to adapt an off-the-shelf open-vocabulary detection model to a specific game domain, a labeled game dataset is required. Annotating object bounding boxes and names demands extensive human labor. However, the wealth of game videos on the Internet, such as those on YouTube, presents an alternative approach. These videos often come with subtitles, providing natural labels that can be utilized for training. Therefore, compared to adapting an object detection model, training a CLIP-like vision-language model (VLM), such as CLIP [1] and CLIP4Clip [2], on this massive data would be much more labor-saving and automated. Most importantly, MineCLIP has already demonstrated this route is feasible. Once a domain-specific CLIP-like model is obtained for a game, our method could be then implemented based on this VLM to guide the agent to learn basic skills in this game.

In summary, compared to tailored open-vocabulary detection models, our method could be nested in a general pipeline with VLM learning, which requires less labor and is more automated, for FPV games. We hope the discussion above addresses your concern. If you still have any additional questions or comments, please do not hesitate to let us know.

[1] Radford et al., 2021, Learning Transferable Visual Models From Natural Language Supervision.

[2] Luo et al., 2021, CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval.

Here is my final review after taking a look at other reviews' comments. I've edited response, score and confidence to reflect my final opinion.

I appreciate the authors response to my questions about open-vocabulary detector. The additional ablations contribute to my score increase. However, as all other reviewers pointed out, the paper's method has a lot of limitation that contradicts open-world promise of the claim.

While the promise of this paper / minecraft itself is about open-vocabulary, the presented method is limited to approaching objects in fpv setting. This is also reflected in the evaluation, where the task covered are no way near open-world. This also has been mentioned by multiple other reviewers. The authors should tune down their claim about open-world.

After the rebuttal, I decide to lift my score from 3 to 5 for added result and also raise my confidence from 4 to 5. This confidence score is due to the fact that I had personal experience trying many components the paper used before ICLR (Dense CLIP, MineCLIP, inverse bbox size for distance, LLM extract objects from task, minecraft, open-vocab detection on minecraft), and have tried one of them on the figures the authors provided during rebuttal period. Such confidence score raise has no conflict of interest, as I don't have any in-submission or on-going paper that's close enough to the author's.

I acknowledge current approach's merit, but would believe it has to belong to a more system/experiment heavy paper where such a reward only plays part of the role. Many many more diverse tasks, or a whole end-to-end decision making system would make this paper much stronger. At its current state, I reiterate my belief that such a reward design alone, under the broken promise of open-worldness, doesn't constitute the technical contribution a full ICLR paper needs. On the other hand, I think the paper is above a score of 3 once the authors would incorporate all our feedbacks. If my confidence score between 4 and 5 is an absolute tie breaker for the AC, I recommend accept this paper.

Thank you for your efforts and valuable comments. We appreciate such a detailed and insightful discussion on our method. We would like to address the weaknesses and questions you raised in the review.

By limiting the CLIP model to a set of pre-determined negative words, isn't the paper limiting the scope and moving away from the actual aim of being open vocabulary?

These pre-determined negative words are used to instruct the model on what to ignore, thereby enhancing the accuracy of the 2D confidence map by reducing noise. Concretely, we compute similarities between these negative words and image patches, in addition to similarities between the target object name and image patches. By applying a softmax function to these similarities on each patch, we suppress the activation of the target object on non-target patches, especially where objects from the negative word list are likely present. This results in a cleaner, low-noise confidence map.

Given the principle mentioned above, we construct the negative word list containing objects that frequently appear in Minecraft for effective denoising. Note that while we limit the negative words, we do not restrict the range of target objects. So in principle, our modified MineCLIP can generate the confidence map for any target object and does not move away from the aim of being open-vocabulary.

To extend the previous question, is it possible to perform a similar analysis, but instead of negative words, use the entire vocabulary?

If 'the entire vocabulary' refers to all object names in Minecraft, we believe it will work as well because our negative is equivalent to the most frequent part of these objects and is proven to be effective. If 'the entire vocabulary' means the general vocabulary that CLIP can understand, we suppose that it would be less effective. There are multiple reasons. For example, synonyms for the target object will suppress its activation on the confidence map because the synonyms also have high similarities on the patches where the target object presents. Additionally, words with different levels of granularity will complicate the segmentation results. For instance, a human might be identified as 'human' as a whole but also as 'head', 'arms', etc., separately.

Therefore, a pre-determined, domain-specific word list is essential for open-vocabulary tasks. This is also evidenced in open-vocabulary learning studies, such as open-vocabulary segmentation [1,2,3], where benchmark datasets would provide a pre-determined class set for segmentation evaluation. We hope this could also address your concerns about negative words raised in Weaknesses.

[1] Ding et al., 2022, Open-Vocabulary Panoptic Segmentation with MaskCLIP.

[2] Zhou et al., 2022, Extract Free Dense Labels from CLIP.

[3] Liang et al., 2023, Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP.

Would it make sense to keep the objects fixed and change the intended behavior to a similar but nuanced variant of the behavior?

We appreciate this insightful suggestion and conduct relevant tests to explore our model's ability to adapt to similar but nuanced behavior patterns. We test the hunt domain model on the 'milk a cow' task, where the object 'cow' is familiar, but the required behavior pattern is slightly different. In detail, hunt tasks require 'attack' action to interact but 'milk a cow' requires 'use' action. The hunt model achieves a 43(±16)% success rate, lower than that of the harvest model trained on this task but significantly higher than zero, suggesting that COPL can partially generalize to a nuanced variant of the training behavior pattern.

Another relevant example is the test task 'harvest sand' in the harvest domain, where both object and behavior are out of the training distribution. As we introduced at the end of Appendix B.3.3, 'harvest sand' requires a slightly different behavior from the training task, and at the same time, 'sand' is a novel target object. As demonstrated in Figure 8(c), COPL outperforms the baseline EmbCLIP on this task.

However, it is important to note that this generalization may stem from the stochastic or noisy nature of the policy output and does not represent that COPL has a true open-vocabulary ability over verbs.

If you have any additional questions or comments, please do not hesitate to let us know. We are more than happy to provide any further clarifications or information that may be helpful.