Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement

Thank you for your valuable feedback. In response, we offer the following clarifications and hope our explanations address your concerns and strengthen our work.

---

(A) Theoretical Claims: 1. What's the definition of $\breve{Q} _\mathsf{h}$ in Equation 2.

$\mathsf{h}$ denotes the random variable of the truncated horizon, and $\mathsf{G} _\mathsf{h}(t) := \sum _{t'=t}^{t+\mathsf{h}-1} \gamma^{t'-t} \mathsf{r} _{t'}$ denotes the random variable of the horizon-truncated Return-To-Go (RTG). In Eq. (2), $\breve{Q} _\mathsf{h}$ denotes the random variable corresponding to the Truncated Q (i.e., the neural network we aim to learn), and its learning objective is to approximate $\mathbb{E}\mathsf{G} _\mathsf{h}$.

(B) Theoretical Claims: 2. The explanations of the notations in Theorem 3.1 should be included in the main paper Weakness: 1. Each symbol used in the paper should be clearly defined …

We create a notation table in this link to provide detailed explanations of all the notations used in Thm. 3.1. These definitions will be incorporated into the main text in our revision.

(C) Weakness: 2. The pseudocode of the algorithm is necessary in the main part.

We provide the pseudocode for the TIPR algorithm framework at this link. We appreciate your suggestion and will incorporate it into the main text in our revision.

(D) Weakness: 3. More intuitive explanation is needed as to why truncating Q-values is better.

For this question, please refer to our response to reviewer wBej's comment "(B) Methods And Evaluation Criteria: … However, it is still unclear". You can use Ctrl + F to search for the sentence in quotation marks to quickly jump to there.

(E) Questions For Authors: 1. Can you give a detailed explanation about why the error in the optimization objective of $\bar{Q}$ accumulates …

In Eq. (1), the original Q-value $\bar{Q}$ regresses onto the RTG label $G_t^1$. As the horizon grows longer, estimation errors for individual rewards accumulate due to the cumulative summation of discounted rewards. This compounding of errors directly reduces the empirical accuracy of the learned $\bar{Q}$, especially when datasets are suboptimal.

From a theoretical view (Thm. 3.1), this accumulation manifests explicitly in the term $\Delta(\mathsf{h})$, whose complexity increases with the $\mathsf{h}$ (specifically $\Delta(\mathsf{h})=O(\mathsf{h}^{2})$). Consequently, as the $\mathsf{h}$ extends, the Empirical Risk NMB Probability (the probability of correctly fitting truncated returns) decreases significantly due to this error accumulation.

The longer horizon in $\bar{Q}$ reduces its reliability in identifying actions with the highest true returns, lowering the Overall NMB Probability. Truncating the horizon limits the error accumulation, improving action selection accuracy and policy refinement—especially in challenging suboptimal dataset settings.

(F) Questions For Authors: 2. How to find the optimal $H^{\ast}$? …

Our primary focus in this work is to demonstrate, both theoretically and empirically, that just truncating the horizon for Q-value estimation, without careful optimization of $H$, can already provide significant advantages over not truncating at all. Theoretical insights from Prop. 3.2 explicitly show the existence of an optimal $H^{\ast}$, validating the conceptual soundness of truncation.

Identifying the precise $H^{\ast}$ is indeed an important direction for future research. Here are some possible methods:

1. Cross-Validation on the Dataset: Empirically selecting $H^{\ast}$ by performing cross-validation using subsets of the dataset to minimize prediction errors or maximize validation performance.
2. Adaptive Methods: This approach adaptively adjusts $H$ during training or testing based on real-time confidence or uncertainty estimates. For example, an adaptive strategy could start with a small horizon and increase it if confidence stays high, or reduce it when uncertainty rises—balancing accuracy and computation dynamically.

(G) Questions For Authors: 3. Intuitively, h should not be too large or too small, and It should have robust performance within a certain range …

In fact, the results in Fig. 5 (Question 5 in Section 4.2) clearly demonstrate that setting $H$ too large (e.g., $H = T$, which degenerates into the Original Q) or too small (e.g., $H = 1$) is suboptimal. A moderate value of $h$ tends to yield better performance.

We did not perform an exhaustive search for the optimal $H^{\ast}$ in each environment. In Fig. 3, we used a moderate, untuned $H$. Still, Truncated Q effectively improves suboptimal original OOM policies over the Original Q.

---

Your comments have greatly helped improve our manuscript. If you have further feedback, we'll address it promptly. If your concerns are resolved, we hope you'll reconsider your rating.

Thank you for your valuable feedback. We'd like to offer the following clarifications and hope our responses address your concerns and strengthen our work.

---

(A) Weaknesses: 1. The authors use a MEP approach to generate 4 opponents … Questions For Authors: 4. … How are 20 policies sampled …

MEP uses Population-Based Training (PBT), which maintains a fixed population of 4 policies and evolves them through cross-play and natural selection. During training, we regularly saved checkpoints of all policies in the population, resulting in many policies. We then selected 20 diverse, representative ones (from weak to strong) to form the opponent policy pool.

(B) Weaknesses: 2 … the scalability of TIPR in complex environments (e.g., Google Research Football (GRF)) remains unverified.

Regarding scalability, we argue that TIPR framework is fundamentally built upon highly scalable in-context learning methods and the Transformer architecture, which perfectly aligns with the principles outlined in Richard S. Sutton’s "The Bitter Lesson" [1], emphasizing the inherent scalability of learning-based approaches.

Regarding the choice of environments, although they are not highly complex, existing OOM algorithms still exhibit significant performance drops when the datasets are suboptimal. Moreover, these environments are widely recognized as benchmarks in the OM community and have been extensively used in prior works such as DRON, LIAM, TAO, OMIS [2], etc. While environments like GRF and SMACv2 are indeed more complex, they are typically used in MARL, focusing on agent communication and coordination, not the OM setting.

To further validate TIPR, we also ran preliminary experiments on the more challenging OverCooked (OC) environment [2,3], which has high-dimensional image inputs and requires intensive cooperation to complete a sequence of subtasks for serving dishes. The results can be found at this link. Due to time limits, we currently report results only on the representative OOM algorithm TAO, and we will include the full results in the revision. TIPR consistently improves the original OOM policy in OC, effectively addressing the suboptimality of datasets.

[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html?ref=blog.heim.xyz, Rich Sutton.

[2] Opponent modeling with in-context search, NeurIPS2024.

[3] On the Utility of Learning about Humans for Human-AI Coordination, NeurIPS2019.

(C) Other Comments Or Suggestions: 2. The MSE comparisons between $\bar{Q}$ and $\bar{Q}_V$ may be exacerbated …

Our main concern is the absolute Q-value error during Policy Refinement (PR) in unknown test environments. PR relies on the ranking of Q-values across all legal actions, since actions are chosen based on these rankings. Normalization doesn’t affect this ranking—it only scales values uniformly. Therefore, the reported MSE directly reflects confidence in action rankings for each method. Normalization wouldn't change qualitative conclusion that truncated Q-values improve ranking accuracy and thus enhance PR effectiveness.

(D) Questions For Authors: 1. How to read the colored and shaded areas of Fig.3 in MS and PP? …

We use Black Error Bars (BEB) for the original OOM policy's Standard Deviation (SD) and Grey Error Bars (GEB) for the SD after applying TIPR. In both MS and PP, GEB consistently appears above BEB, with the shaded area showing performance gains. This shows TIPR consistently improves OOM performance. We'll refine Fig. 3 in the revision to address any clarity issues.

(E) Questions For Authors: 2. Why are the refine condition (RC) threshold and … set as 0?

This design is intuitive: (1) Indicator Condition: If a legal action is predicted to yield non-zero reward within $H$ steps, its value estimate is considered high-confidence. (2) RC: If any action meets this, we deem it promising to use Truncated Q to refine the policy at that timestep. This helps in sparse-reward environments and doesn't negatively impact performance in dense-reward ones.

(F) Questions For Authors: 3 … So I wonder how the differences between the trained opponents are.

Regarding the diversity of the opponent policies, please refer to our response to reviewer Chog's comment "(D) Experimental Designs Or Analyses: … However, if the offline dataset". You can use Ctrl + F to search for the sentence in quotation marks to quickly jump to there.

(G) Questions For Authors: 5. Does TIPR introduce significant additional computational overhead …?

For this question, please refer to our response to reviewer Chog's comment "(G) Other Strengths And Weaknesses: 2. The computational efficiency".

---

Your comments have greatly improved our paper. If you have further feedback, we'll address it promptly. If your concerns are resolved, we hope you'll reconsider your rating.

Thank you very much for your valuable feedback. In response to your comments, we would like to make the following clarifications and feedback. We hope our explanations and analyses can eliminate concerns and make you find our work stronger.

---

(A) Other Comments Or Suggestions: The paper should define the concept "policy refinement" in a more concrete way …

In Offline Opponent Modeling (OOM), existing algorithms typically assume that the offline dataset is optimal. We refer to the policy trained using these existing algorithms directly on the dataset as the original policy. When the dataset is suboptimal, we find that the performance of the original policy drops significantly. Our work aims to design a plug-and-play algorithmic framework to improve the performance of the original policy obtained through OOM algorithms—this is the definition of "Policy Refinement" (PR). We will clarify this in our revision.

Notably, our work introduces the novel idea of performing PR instantly at test time, referred to as Instant Policy Refinement (IPR), rather than refining the original policy during the offline stage, as done in traditional Offline Conservative Learning (OCL) methods. As pointed out by Reviewer Chog, this helps mitigate distribution shift, and the advantage of IPR over OCL is also demonstrated in Fig. 4 (Question 2 in Section 4.2).

(B) Methods And Evaluation Criteria: … However, it is still unclear to me why and how truncated Q value would address the sub-optimality of the dataset …

Inspired by OCL methods, we aim to learn a Q-function to refine the original policy obtained through OOM algorithms. However, learning a workable Q-function in multi-agent environments is highly challenging, as the error in return prediction accumulates significantly when learning an Original Q (see Fig. 1). As you noted, this issue becomes more severe with a larger number of opponents—a claim that is also supported by our Theorem 3.1. In contrast, our Truncated Q can effectively reduce this accumulation, making it possible to distinguish between optimal and suboptimal actions.

Our in-depth theoretical analysis (Theorem 3.1 and Proposition 3.2) demonstrates that, under the OOM problem setting, an optimal truncated horizon exists that maximizes the probability of correctly identifying the best actions—formally, the No Maximization Bias probability. This theoretical result supports the fact that using Truncated Q for PR is more sound and effective than using the Original Q.

In addition, our proposed IPR method can automatically decide whether to perform PR based on the confidence of action-value estimates from the Truncated Q. It effectively addresses the distribution shift problem commonly found in OCL methods: when uncertain, it falls back on the original policy (conservative), and when confident, it refines the policy (greedy). This trade-off mechanism is validated in Fig. 4 (Question 4 in Section 4.2) to outperform both always being greedy and always being conservative.

(C) Other Strengths And Weaknesses: The results show that the value of fixed horizon doesn't impact the performance …

In Fig. 5 (Question 5 in Section 4.2), we analyze how the choice of different truncated horizons $H$ for the Truncated Q affects the improvement results. In fact, the results in Fig. 5 show that different values of $H$ lead to varying levels of improvement, and this effect is environment-dependent.

When $H$ is set to a small value, the improvement from PR is limited. When $H$ equals $T$ (i.e., degenerating to the Original Q), PR generally degrades the performance of the original policy. However, when $H$ takes on a moderate value, PR is able to effectively improve the original policy. This observation supports our Proposition 3.2: there exists an optimal $H^{\ast}$, and in general, $H^{\ast}$ is not equal to $T$.

(D) Questions For Authors: Should we consider the method as few-shot adaptation method? …

You've provided a very interesting insight—indeed, our method can be viewed as a form of few-shot adaptation. This is because our Truncated Q is built upon in-context learning and a Transformer-based architecture, which enables it to collect a small set of contextual samples at test time. Through the IPR method, we can adaptively decide whether to perform PR using the Truncated Q—based on the collected samples—without relying on gradient descent. This allows us to generate a refined policy in an adaptive and efficient manner.

---

All your questions and feedback have greatly contributed to improving our manuscript. With the valuable input from you and all other reviewers, the quality of our work can be significantly enhanced. We welcome further comments from you and will seriously consider your suggestions for revisions. If you feel that we have addressed your concerns, we hope you will reconsider your rating.

Thank you for your feedback. We'd like to clarify a few points in response, hoping our explanations address your concerns and strengthen our work.

---

(A) Theoretical Claims: … the empirical risk bound in eq-4 assumes a specific distribution of Q-learning errors, which depends on well-behaved rewards and opponent policies. If the environment dynamics cause non-monotonic error growth …

Our theory (Thm. 3.1 and Prop. 3.2) fundamentally proves that there exists a trade-off between the truncated horizon $h$ and the No Maximization Bias probability. Specifically, the theorem reveals a structured relationship showing that extending the $h$ increases the complexity of accurately estimating returns, reducing the accuracy of Q-values in selecting optimal actions. Thus, using Truncated Q is more sound and effective than using the Original Q for policy refinement.

Importantly, we clarify that our theory does not explicitly assume or depend on any particular structure or specific distributions of Q-learning errors, nor does it require particular properties or assumptions regarding opponent policies. The derived bounds in Eq. (4) are general and are based purely on standard statistical inequalities and complexity terms that arise naturally from the estimation problem itself.

We acknowledge that some environment dynamics may cause non-monotonic error growth. However, our theory doesn't specifically assume monotonicity or well-behaved rewards or opponents. Future work could explore more detailed analyses of environments with complex long-term strategic dependencies.

(B) Theoretical Claims: Further, the assumption that opponent modeling via in-context data is always reliable could be a strong assumption …

We clarify that assuming reliable Opponent Modeling (OM) via In-Context Data (ICD) is not overly strong. TAO (Jing et al., 2024a) has theoretically and empirically proved that using ICD for OM is both feasible and effective, even with non-stationary opponents. Building on this evidence, our Truncated Q uses ICD to enhance OM reliability and performance.

(C) Claims And Evidence: The claim would be stronger if supported by an adaptive mechanism for choosing $h$ rather than manually testing different values (as in Figure 5).

For possible automated mechanisms of choosing $h$, please refer to our response to reviewer RP82's comment "(F) Questions For Authors: 2. How to find". You can use Ctrl + F to search for the sentence in quotation marks to quickly jump to there.

(D) Experimental Designs Or Analyses: … However, if the offline dataset lacks sufficient opponent diversity …

We use MEP algorithm to generate opponent policies. From the MEP population, 12 policies were selected as training opponents (Seen Set), and 8 policies were selected as test opponents in the Unseen Set. A quantitative analysis of the diversity among the 12 training opponent policies is provided at this link.

To measure their distinctions, we use the Pair-wise Expected KL Divergence (PEKLD). If we take 1.0 as a threshold, over 80% of the PEKLD values in all environments exceed this threshold. This indicates that the training opponent policies are generally well-distinguished from one another.

(E) Essential References Not Discussed: … the problem formulation in [1] focuses on a similar setting from the perspective of principal-agent …

[1] is an interesting related work. We will cite [1] and discuss its connections with our work in our revision.

(F) Other Strengths And Weaknesses: 1 … Testing TIPR on more complex strategic settings …

For experiment results on more challenging settings, please refer to our response to reviewer LtaJ's comment "(B) Weaknesses: 2 … the scalability of TIPR".

(G) Other Strengths And Weaknesses: 2. The computational efficiency of TIPR is not discussed …

Compared to the original OOM algorithm, TIPR only requires one additional forward pass using the Truncated Q at each timestep, which can be completed very quickly (on the order of milliseconds to seconds).

Many other policy refinement methods, such as decision-time planning (e.g., Monte Carlo Tree Search), perform a large number of forward passes to rollout numerous simulated trajectories to refine the policy. In contrast to them, the computational cost of TIPR is negligible.

(H) Other Comments Or Suggestions: … introduces the variable $M$ but has it been defined somewhere in the paper?

$M$ denotes the number of $(o^{-1}, a^{-1})$ tuples in the ICD $D$. This value determines how much information is used to characterize the opponent's policy and is typically set as a predefined constant. We will include this in our revision.

---

Your comments have greatly helped improve our paper. If you have further feedback, we're happy to address it. If your concerns are resolved, we hope you'll reconsider your rating.