Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning

Thank you very much for your thoughtful review. We sincerely appreciate you pointing out our method's core insight, theoretical soundness, and the credibility of our experimental evaluation. Below, we provide our detailed responses to your remaining questions.

Q1. What specifically is the distance metric? Examples? What function is used to measure the distance?

• (Distance metric) As defined in Eq. (6), the distance metric is the absolute difference in action likelihood between the original input and a modified input, where a specific token is replaced by a placeholder (e.g., "pad_token").
• (Example) Take the Game Card as an example.
  • The original input $y$:   {"cards": [2, 6], "formula": "2*", "thoughts": "'2*' is an incomplete formula. Since '2*6=12', I should append '6' to the current formula", "action": "6"}
    We first feed it into the SCM and let SCM output the baseline action likelihood $\mathbb{P}(a|y)$. Then, we remove each token by replacing it with a placeholder. For example:
  1. Remove 1st token $y^{-1}\cup y^1_{\text{null}}$:
     <pad>"cards": [2, 6], "formula": "2*", "thoughts": "'2*' is an incomplete formula. Since '2*6=12', I should append '6' to the current formula", "action": "6"}
  2. Remove 2nd token $y^{-2}\cup y^2_{\text{null}}$:
     {<pad>cards": [2, 6], "formula": "2*", "thoughts": "'2*' is an incomplete formula. Since '2*6=12', I should append '6' to the current formula", "action": "6"}
  3. Remove 3rd token $y^{-3}\cup y^3_{\text{null}}$:
     {"<pad>": [2, 6], "formula": "2*", "thoughts": "'2*' is an incomplete formula. Since '2*6=12', I should append '6' to the current formula", "action": "6"}
  4. ...
     Each modified sequence is then fed to SCM to obtain a new action likelihood $\mathbb{P}(a|y^{-i}\cup y^i_{\text{null}})$ for $i=1,2,...,n$.
• (Function) The distance function is given by $D=|\mathbb{P}(a|y)-\mathbb{P}(a|y^{-i}\cup y^i_{\text{null}})|$. This measures the absolute change in action likelihood caused by nullifying a specific token $y^i$. By observing how each token's removal changes (or does not change) the final action likelihood, we can measure each token's causal importance.

Q2. Why not randomly select a similar token instead? Causing parse_action to fail and return a large negative reward?

Thank you for raising this point. If we understand correctly, your concern is that using a nullified value might cause the parse_action function to fail, leading to invalid actions and large negative rewards during the interaction between the agent and the environment.

We would like to clarify that our method does not have this issue because the entire causal computation process takes place after the agent-environment interaction phase. Specifically:

• As shown in Algorithm 1 (Lines 6–11), the rollout phase remains exactly the same as in a standard RL loop. In this step, the agent interacts normally with the environment and collects trajectory data (without involving the SCM or any token nullification).
• After that (Lines 12-18 of Algorithm 1), token nullification and the computation of causal weights $B_{y_t\rightarrow a_t}$ are conducted offline, using the stored transitions $(s_t, a_t, r_t, y_t)$ in the replay buffer. Since both the action $a_t$ and reward $r_t$ have already been determined and logged at that point, the SCM's token nullification analysis does not affect action parsing or the reward signal in any way.

Q3. GPU memory usage and training time

We have reported the computational budget of causal weights in Appendix C. It adds only 0.01 B parameters (<0.2%), 0.7 GB of GPU memory (<2%), and 0.5 H100 GPU hours (<4%), which are modest and introduce small overhead.

Thank you again for your valuable feedback. We will incorporate clarifications on the above three points in our revised version.

Thank you very much for your detailed and thoughtful review. We truly appreciate you highlighting the strengths of our method and thinking it timely, well-discussed, easily applicable, and supported by insightful experiments. Regarding your questions, we provide our responses as follows.

Q1. Why a smaller model?

We chose a smaller model mainly to reduce computational cost and training time. As shown in Appendix C and D, we found that the lightweight model already works well for SCM while introducing small overhead.

Q2. How is the offline data collected?

The offline data used to train the SCM comes from the SFT dataset in the original paper (DigiRL and RL4VLM), which includes the agent's "observations" and its "text responses". For SCM's training, we only use the "text responses" as inputs and their corresponding action labels as ground truth.

Q3. The ground truth in the CrossEntropy loss.

In our implementation, the SCM is trained as an action classifier. Thus, the ground truth is the label of the parsed action corresponding to the agent's text output. For example, SCM input: "Action Plan: [PRESS_HOME,DUAL_POINT, DUAL_POINT,DUAL_POINT] ; Action Decision: "action_type": ...", SCM ground truth: 2 (assuming PRESS_HOME maps to index 2)

Q4. What is the input of SCM?

• As explained above, the input is the agent's text output $y$ (e.g., Action Plan: [PRESS_HOME,DUAL_POINT, DUAL_POINT,DUAL_POINT] ; Action Decision: "action_type": ...), then SCM predicts its correct environment action category.
• Moreover, when computing causal weights, we also nullify a specific token (e.g., the first token ($y^{-1}\cup y^1_{\text{null}}$)) and feed the modified sentence "<pad> Plan: [PRESS_HOME,DUAL_POINT, DUAL_POINT,DUAL_POINT] ; Action Decision: "action_type": ..." into the SCM to get token's causal influence.

Q5. How accurate was SCM? Could a smaller model be used?

• During the training, the SCM achieves ~100% accuracy in AitW, ~90–100% in Gym Cards, and ~70–80% in ALFWorld.
• Yes, the smaller model can be used, especially in high-accuracy tasks. However, there is a trade-off between size and the quality of causal weights.

Q6. What does "nullify" a token mean?

To "nullify" a token means replacing it with a placeholder token (e.g., pad_token or unk_token) to simulate its absence.

Q7. Is it necessary to "nullify" each token? Does it require extra environment interactions?

Yes, we need to nullify each token to compute its causal importance. However, this process requires no extra environment interactions.

• As shown in Alg. 1 (Lines 6–11), the 'Rollout phase' remains exactly the same as in a standard RL loop. In this step, the agent interacts normally with the environment and collects trajectory data (without the SCM or token nullification).
• After that, we evaluate the token's importance using the stored transitions $(s_t, a_t, r_t, y_t)$ in the replay buffer $\mathcal{U}$. Specifically:
  1. Feed the raw $y_t$ into the SCM and let SCM output the baseline action likelihood $\mathbb{P}(a_t|y_t)$.
  2. Nullify a specific token and feed $y_t^{-i}\cup y^i_{\text{null}}$ obtain the action likelihood $\mathbb{P}(a_t|y_t^{-i}\cup y^i_{\text{null}})$.
  3. Compute token's importance via $\mathcal{B}^i_{y_t\rightarrow a_t}=|\mathbb{P}(a_t|y_t)-\mathbb{P}(a_t|y_t^{-i}\cup y^i_{\text{null}})|$.
• The above process relies only on the replay buffer $\mathcal{U}$ and the SCM model (without environment interactions). Thus, environment interaction cost stays the same.

Q8. Extension to LLM agents

Thank you for your valuable suggestion. We totally agree that CoSo is not limited to VLM agents and it applies naturally to LLM agents operating in purely textual environments. To show this, we include an experiment on AlfredTWEnv, a purely text-based benchmark in the ALFWorld where several LLM agents (e.g., ReAct, Reflexion) have been evaluated. We achieve RL4VLM and CoSo on Qwen2.5-7B-Instruct and train the LLM agents from scratch (without SFT) over 12,000 environment steps. Here's the result:

Reference

Thank you for the suggestion. We'll cite and discuss token-level credit assignment works like POAD in the revised Related Work section.

Writing and presentation

Thank you for your helpful editorial suggestions. We'll fix the typos in the title of Appendix C, clarify the use of "trial-and-error", unify the policy definition over tokens in the second and third paragraphs in Section 3, and refine the beginning of Section 5 by adding a pointer to Appendix C.

Thank you again for all your constructive feedback. We will include these clarifications, especially the counterfactual reasoning, and updates in our revision.

We sincerely appreciate your in-depth and constructive feedback. We are also grateful for kindly pointing out that our method is well-motivated and convincing, compelling in principle, and implies high expandability and flexibility. Regarding your concerns, we provide our responses below:

Q1: The results of using the likelihood difference of all actions.

We tried both KL divergence and L2 distance on Gym Cards to take all actions into account. Both produced similar results in terms of causal weights and overall performance. KL divergence is asymmetric and more sensitive to small distributional differences, so it may require more careful smoothing or regularization.

Q2: Ablation study for the learning rate and entropy coefficient.

We provide ablations about the learning rate and the entropy coefficient on the Gym Cards NL task below. We found that CoSo needs an appropriate learning rate and entropy coeff to work best. For example, if entropy coeff is too large, it excessively perturbs the output distribution, reducing the agent's stability; if too small, it approximates a setting without exploration incentives.

Q3: Causal differences across intermediate reasoning components.

Actions like DUAL_POINT and TYPE typically involve more complex token patterns than simpler actions such as PRESS_HOME or PRESS_ENTER. Beyond the main action token, they often include additional elements (e.g., coordinate values or type-specific content) that vary across instances. These extra tokens can introduce nuanced information, making the SCM more sensitive to intermediate components.

Q4: Rationale for using causal weights to weigh entropy values?

Yes, the theory of weighted entropy has been formally studied in [1]: $h_\varphi(p)=-\sum_i\varphi(x_i)p(x_i)\log p(x_i)$ (see Eq.(1.2) in [1]). Here $\varphi(x_i)$ is a non-negative weight function, presenting a value/utility of the outcome $x_i$. This formulation makes entropy context-dependent, assigning different weights based on the value of each outcome. In our case, we adopt causal weights as the utility function, assigning higher weights to more important tokens within an action.

[1] Kelbert, Mark, Izabella Stuhl, and Yuri Suhov. "Weighted entropy: basic inequalities." Modern Stochastics: Theory and Applications 4.3 (2017): 233-252.

Q5 & Q6: Attention weights and Shapley value.

Thank you for your valuable suggestions. We evaluated the attention weight and the Shapley value and presented the results below.

• (Attention weights) We found that attention weights perform suboptimally. Since attention weights dynamically evolve during training, especially early stage of the training, they often lack stability and fail to consistently reflect token importance.
• (Shapley value) The computation of Shapley value scales exponentially with token count ($2^n$) which is computationally infeasible. Thus we use Monte Carlo sampling with 10 subsets. It produces similar results as ours. This similarity may be attributed to the fact that Eq. (6) in our paper can be interpreted as a subset-based approximation of the Shapley value. Nevertheless, the Shapley value is significantly more computationally expensive.

Reference

Thank you for pointing them out. We will include citations for SCM and the BERT paper in the updated version.

Notations, typos and suggestions

We appreciate the detailed feedback. We will improve the clear definitions of all notations in (A.6) and (A.11), fix the typos in (A.7), and update Eq.(6) to use a single vertical line for denoting absolute value, as suggested.

Thank you again for your valuable and encouraging feedback. We will incorporate all these clarifications and improvements into our revision.

We sincerely appreciate your valuable comments and the time you took to review our work. We also appreciate your positive remarks about the potential applicability of our method to other challenging control tasks. Regarding your questions, we provide our responses below:

Q1. The weighted term and the entropy term

• Thanks for the question. $\mathcal{B}^{i}_{y \rightarrow a}$ denotes the causal weight of token $y^i$, while $\mathcal{H}(y^i | y^{1:i-1})$ denotes the conditional entropy of the same token. They are therefore aligned in the objective.

• Moreover, the causal weight $\mathcal{B}_{y\rightarrow a}^i$ is used only as a scalar factor (without gradient) for the entropy term $\mathcal{H}(y^i | y^{1:i-1})$. As such, the exact computation of the causal weight is flexible, and in practice, it can be implemented in different ways—globally, locally, or even based on random subsets of $y^{1:n}$ like Shapley Value (as mentioned by Reviewer igVU).

Reference

Thanks for the helpful suggestions. We will add the recommended references and include a discussion of them in the updated Related Work section.