Action as a Modality: Turning Multi-Modal LLMs to General Action Planners

• The proposed action encoding method, while innovative, might encounter limitations when applied to tasks with large or continuous action spaces (e.g., in manipulation tasks, the action spaces are commonly continuous, like the 6D pose of end effectors). The action space in the experiments presented in this paper is limited to selecting from a set of camera IDs, which is significantly simpler than the action spaces encountered in typical robotics tasks. If the authors could validate their proposed method in a more complex action space, it would enhance the quality and impact of the paper.

• The writing quality could be improved, as there are expressions that lack precision and clarity. For instance, in lines 203-215, both sequence $S$ and observation $\mathcal{O}$ are ordered sets, so curly braces should not be used. Additionally, the angle brackets used in line 208 are not well-defined, which could lead to confusion. Clarifying these notations would improve readability and technical accuracy.

In my assessment, the major novelty of this work is representing actions as inputs for an LLM rather than having the LLM output actions. Specifically, ActionVerse encodes actions via new tokens and encodes image inputs using features from a pretrained visual encoder like DINOv2. The LLM then selects which action to choose from all possible encoded actions. This contrasts with prior approaches that output actions directly from the LLM, for example, by tokenizing the action and then outputting the tokens corresponding to that action, as with RT-2. Encoding actions as input in the ActionVerse system poses an interesting alternative to such tokenized outputs. However, the paper does not evaluate if this different way for an LLM to output actions is empirically better than the direct tokenized output strategy of works like RT-2. To properly assess this key contribution, this work should have compared to the exact same system, using the same conversational data format, added disturbances, and all other aspects of ActionVerse, but then selected actions based on the LLM outputting tokens corresponding to the action as RT-2 does.

Instead, the paper conflates the need to maintain complex prompts for zero-shot LLMs with finetuning LLMs for actions. It is still possible to finetune the LLM to take actions by having the actions be output tokens as RT-2 does, and this does not require extensive system prompts or expensive calls to proprietary LLMs.

Overall, to properly assess the key contribution of ActionVerse in finetuning LLMs to encode actions as input, the paper must compare finetuning the LLM in the same setup to output actions instead.

Other weaknesses:

1. How can ActionVerse, which requires encoding all possible actions, extend to continuous action spaces or large discrete action spaces? In continuous action spaces, such a ranking scheme is infeasible. In large discrete action spaces, the input length would grow too large.

2. Given the performance of ActionVerse relative to the expert models, what is the value of using LLMs as agents in these problem specifications? In the VLN in Table 1, there is a large gap between ActonVerse and the expert models. The pretraining knowledge of LLMs should provide some benefit in terms of generalization.

• In the sense of an MDP or POMDP, $s_i$ is typically a state and $a_j$ is an action. The use of the $\mathcal{A}$ for the set of actions but $s$ for individual actions is a confusing mismatch.

• Figure 2 does not sufficiently explain how the projection and action tokenization works. The figure could benefit from a legend and more annotations (e.g., what does (1), (2), and (3) refer to, where did they come from (as labels, for example, as an output of DINOv2 -- though the figure makes it look like an input), and where do they go?

• This example, "For example, the visual observations...robotic arms, etc." is insufficient to convince this reviewer that the framework is generalizable. The framework may work for simple move-to-area actions based upon DINOv2, but more work would need to be done to actually show generalizability to arbitrary embodied AI or robotic tasks. For example, how would this framework support these domains?

https://github.com/Healthcare-Robotics/assistive-gym https://mujoco.org/

• Figure 3 does not clearly describe how the multi-round conversation plays out. Either this figure and/or the text needs to be expanded.

• The disturbance procedure for enhancing robustness needs to be further described.

• LLMs are not planners (see below). It would be helpful if the authors could better describe how they overcame the issues discussed in Kamghampati et al.

Kambhampati, S., Valmeekam, K., Guan, L., Verma, M., Stechly, K., Bhambri, S., Saldyt, L. and Murthy, A., 2024. LLMs can't plan, but can help planning in LLM-modulo frameworks. arXiv preprint arXiv:2402.01817.

• The approach seems relatively simple and has mixed results (not outperforming expert models on one of the two benchmarking tasks). The paper is also making strong claims about generalizability outside of simplistic VLN tasks. As such, the contribution seems limited and that the paper seems to be overclaiming.

• A running example for action encoding would have been more helpful, as that is the crux of the paper, and there is insufficient technical detail to precisely determine how this happens in general (e.g., a description of Figures 2 and 3 with sufficient technical detail to afford reproducibility).

1. Some sections would benefit from reorganization, and certain techniques require further explanation. For example, projection layers are mentioned in lines 38, 74, and 142, but no explanation of what projection layers are until appendix. Additionally, it’s unclear why Algorithm 1 is located in the appendix. It would be helpful to have a clearly defined training and algorithm framework in the methods section to facilitate understanding.
2. The related work section describes prior studies in detail, but it lacks a clear positioning of this paper within that landscape and its relationship to those studies.
3. The techniques proposed require additional tokens or direct operations on embedding spaces, which might not be compatible with advanced LLMs, such as GPT-4. While these models offer strong reasoning capabilities, they often have constrained APIs.