Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol

We thank the reviewer for the helpful comments and feedback. We note that you rated “3: good” in all subcategories but gave “borderline reject” in overall rating. We hope you can reconsider the rating after seeing our response and are glad that you are open to it (“I will reconsider…”).

(A) The combination/selections of contributions is to my understanding rather unconventional. (Quality-related)

We agree that the combination of contributions is unconventional (theory + experiment protocol + empirical), but respectfully disagree that this hurts the quality of the paper. These contributions emerge naturally when we investigate the topic: after developing the methods and their theories, we want to empirically test them, but the protocols in the prior works have been hard to use and caused many troubles in our replication (see e.g., Footnote 1 on page 2). So we come up with a new protocol that is much better in producing stable candidates and enable interesting studies (Sections 6.2 and 6.3) that aren’t possible before.

(1) Restructure the paper to include Figure 5 in main text

Thanks for the great suggestion. We totally agree that Figure 5 can be very helpful in understanding the overall problem and protocols, especially given that the complications in offline evaluation and selection are not something that most of the RL researchers have thought about carefully. We had to defer the figure to the appendix due to space limit, but if the paper is accepted, we fully intend to include it in the main text given the 1 extra content page.

(2) Is “Model selection in OPE” an established term? especially with the multitude of meanings of the phrase "model"

We agree that the word “model” is heavily overloaded in OPE and RL in general. As we mentioned, model selection in OPE is a heavily under-investigated problem, so it’s hard to say it’s “well-established” given the scarcity of research in this area in the first place, but prior work like [62] did also use “offline model selection”. In addition, despite the multitude of meanings of “model”, “model selection” as a whole is a well-established technical term in statistics, and we are following this convention. In machine learning, model selection is closely related to concepts like cross validation and hyperparameter tuning (which we mention in the first sentence of abstract to help resolve the ambiguity of “model selection”), but the latter two refer to something more specific (e.g., cross validation refers to procedures like k-fold, and hyperparameter tuning often applies to settings where candidate functions/predictors are generated by the same algorithm with different hyperparameters) that do not necessarily apply to our problem setting, and the term “model selection” is the more general and abstract term which we think fit better.

(3) surprising to me that [model-based predictors] do not hold your expectations.

Indeed! Our rationale was that model-based predictors enjoy strong theoretical motivations: they apply to narrower settings (can only select from models, not Q-functions; Line 105), but leverage more information from the model (the Bellman operators); in particular, not having access to Bellman operators is the biggest difficulty in model-free selection (Line 207), but the operators can be learned quite straightforwardly in the model-based setting (Line 208). As a general wisdom, methods that properly leverage additional side information often perform better. Moreover, model-based selectors enjoy more standard and interpretable guarantees found in the offline RL theory literature (Line 360). So our anticipation was that model-based predictors would outperform model-free ones, which is a trade-off between performance (model-based wins) and applicability (model-free wins). Therefore, the fact that model-based performs worse is definitely surprising, and we view this a great opportunity to report interesting negative results, which we hope is more appreciated and encouraged by the community, and the surprisingly poor performance of model-based selectors further highlights the effectiveness of LSTD-tournament.

As for why model-based selectors fail, we have had some preliminary investigation into this, which we can add to the appendix; one possible reason is that these estimators are pretty sensitive to small errors (e.g., the Monte-Carlo estimates still has small nontrivial errors despite we use $l=128$ rollouts (Line 288) which is quite computationally intensive (Appendix D.3)), and the model-based estimator suffers more due to its multi-stage process. In contrast, LSTD-tournament has an interesting property that it is largely immune to the Monte-Carlo errors (Appendix E.3), since the noise in each Q-estimate will be averaged out over the entire dataset due to the lack of absolute value or square inside the loss function (Eq.(10)).

(4) Can you improve the clarity in the abstract and in the introduction (especially the first paragraph) to better reflect the topics of the paper? (B, D)

We are definitely happy to but it is unclear which part causes confusion from your review. In particular, your summary of the paper seems spot on and we don’t see any obvious misunderstanding in your text. If you could elaborate more on where it needs clarification, we can improve the clarity of the text accordingly.

(5) Can you include a small conclusion, maybe would fit the start of section 7, that wraps up the paper? (D, E)

Definitely happy to and we plan to do so if the paper is accepted and given one extra content page.

Dear reviewer ZupL,

This is gentle reminder that the time for author-reviewer discussion is running out. We'd very much appreciate your consideration of rebuttal and further comments (e.g., which aspects of the abstracts that you want to see more clarifications) so that we have an opportunity to better address your concerns.

thanks,

Authors

We thank the reviewer for the positive evaluation, and respond to your questions below.

Maybe I'm unfamiliar with relevant literature, but can you obtain lower bounds for model-free and model-based error rates? How tight are your proposed methods?

If by rates you mean the dependence on n, then $n^{-1/2}$ is the best you can get for simple reasons: OPE is clearly more general than estimating the mean of a bounded random variable and subsumes the latter as a special case. For mean estimation, without additional structures and assumptions, the $n^{-1/2}$ rate is the minimax rate and well-established in the statistics literature, so it is simply impossible to do better than that in general. That said, our bounds not only depend on n but also crucially depend on the coverage parameters. The problem is that different methods depend on different forms of coverage parameters (e.g., the $\sigma\_{\min}$ term in Theorem 2, $C^\pi$ in Theorem 4, and $C\_\infty^\pi$ in Theorem 5), which are often not comparable. We have a nuanced discussion on this topic in Appendix C.5 for RL theorists readers familiar with the literature.

In figure 2, can you explain why mb_naive performs well in the top row but has disastrous error in the last two in bottom row?

As mentioned on Line 203, this is because mb_naive strongly biases towards deterministic models when the candidate models have different levels of stochasticity. In the top row of figure 2 all models have similar levels of stochasticity, so this problem is not exposed. In the bottom row, however, columns 2 and 3 correspond to cases where the groundtruth model has relatively high stochasticity, so the bias is fully exposed. In contrast, in column 1, the groundtruth model is the most deterministic one in the family, so mb_naive’s bias acts in its favor.

Your question (Reviewer Robe also asked the same) made us realize that the bias of mb_naive is much less known than we thought, so here is a more detailed explanation of the underlying math (which we can add to the appendix): The key is that in the loss we have $\\\| s’ - \tilde{s}\\\|$ (Line 120), where $s’$ is from the data and $\tilde{s}$ from the candidate model. When the candidate is the true model, $s’$ and $\tilde{s}$ are iid, but their randomnesses are independent and “out of sync”, which causes the loss to be non-zero and scale with the inherent randomness of the distribution. As a minimal example, suppose the state space is $\mathbb{R}^2$, and we focus on the transition distribution from a particular $(s,a)$ pair where groundtruth is uniform over 4 points $(\pm 1, \pm 1)$. If we calculate the expected loss of groundtruth model itself (on data generated from the same model), we get $1+\sqrt{2}/2$. However, a wrong model that always predicts $(0,0)$ deterministically will get a lower loss of $\sqrt{2}$.

Authors should summarize the theorems briefly in the main text.

We will surely try to!

Thank you for your detailed and helpful feedback! We will refer to your listed weaknesses as W1-4 and questions as Q1-4.

W2: Misleading "Hyperparameter-Free" Claim and Unmotivated Design

We first address a potential technical misunderstanding before responding to your comments.

”the weighting function $Q\_k(s,a)$ in Eq.10 can be any function of $(s,a)$, effectively introducing infinitely many degrees of freedom.”

This is incorrect. In particular, choosing any function, such as a purely random function of $(s,a)$ (more on this below), could lose the guarantees of Theorem 2. As we mentioned on L180, only “non-degenerate linear transformations” of $[Q\_i, Q\_j]$ enjoy the same kind of guarantees. Choosing an arbitrary weighting function unrelated to $Q\_k$ simply cannot be called a variant of LSTD-tournament.

We hope this clarification eases your concerns. Below we respond to your other related comments, assuming you still stand by them after this clarification. As you have noted, we acknowledge this degree of freedom (the linear transformation) and provide a recommended choice (normalized diff). That said, there is a big difference between this kind of “small” hyperparameters vs. the “big” ones we mention in the introduction (e.g., neural architecture of FQE; L35). To put it simply,

Are we comfortable recommending neural nets of 2 hidden RELU layers with 100 nodes per layer whenever someone applies FQE in any OPE problem? Of course not!

Are we comfortable recommending the normalized diff version whenever someone applies LSTD-tournament to any OPE model-selection problem? Yes! (see reasons below)

lacks motivation for the design, and ablation in App E.2 demonstrates [sensitivity]

There is good motivation for normalization. In particular, imagine if a wrong candidate $Q\_k$ has enormous magnitude. When such a $Q\_k$ serves as the weighting function, it can artificially blow up the loss of other $Q\_i$, including the true Q-function, and the proof of our theoretical guarantee relies on true Q having a reasonably small loss. Therefore, controlling the magnitude of weighting functions via normalization is well-motivated. Taking difference is, indeed, more heuristic, but its use in RL to help with numerical stability is also well-documented [25,10].

The 3 variants do have different performance profiles in App E.2. However, compared to other baselines, the qualitative nature of their performances are not that different, and none of them strictly dominate the others. Given that their performance comparison is largely a tie, and there is good reason to prefer normalization (to protect against large Q) and difference (for numerical stability), we feel comfortable recommending the normalized diff version as a universal choice. Fixing such a choice, in our opinion, is exactly eliminating the degree of freedom rather than “hiding it”. For example, BVFT also has a discretization error as a hyperparameter, and the authors provide a heuristic to pick it in [61], which we adopt in our implementation.

Q4: why are $Q\_k$ good weighting functions intuitively?

Using $Q\_k$ as weighting functions purely emerges from our theory. We don’t have more intuitive explanations other than the theory, which we don’t view as a negative; in fact, BVFT is just as counterintuitive as (if not much more than) LSTD-tournament.

Popping up a level, most RL research follows an “empirical-first” methodology of “design intuitive algorithms -> test empirically -> if works, find theoretical justifications”, which is perhaps why the reviewer asks this question. In contrast, we start from derivations with strong theoretical guarantees (Theorem 2) as the basis of research. This methodology has the unique advantage of tapping into those “counterintuitive” ideas in the algorithmic space. If not for it, algorithms like BVFT will perhaps never be found.

W2: How about other weighting functions, such as “random functions of (s,a)”?

Perhaps surprisingly, if you use “random functions” as weights, the loss becomes degenerate. In particular, we can write Eq.10 as

$\max_{w \in \mathcal{W}} |\mathbb{E}\_{\mu}[w(s,a) (Q\_i-\mathcal{T}^\pi Q\_i)]|,$

As you pointed out, LSTD-tournament uses $Q\_k$ as the weighting function $w$. The most random way to generate $w$ is to generate an i.i.d. weight scalar for each $(s,a)$ in the dataset. Suppose this i.i.d. distribution has mean $C$. Now, because the randomness in weight is independent of everything else, the outside expectation will effectively first marginalize out such randomness, which means the expected loss is simply equal to

$|\mathbb{E}\_{\mu}[C (Q\_i-\mathcal{T}^\pi Q\_i)]|.$

When $C \ne 0$, this coincides with “average Bellman error” (L290) which we include in the experiments. It has poor theoretical properties: even if a candidate $Q$ has the wrong shape, you can always add a constant to it–which does not change its shape–to make its average Bellman error $0$.

If we want make the weights “nontrivially random”, we need to leverage information in $(s,a)$, e.g., generate a random function that is smooth over $S \times A$ w.r.t. some metric. However, the choice of the metric and how exactly the function should be generated are “big hyperparameters” that are left as design choices to the user.

Moreover, the sign-flip selector (App B.2) also fits this form, where $sgn(Q\_i - g)$ serves as weighting functions. So that can also be viewed as a comparison where the weighting functions are chosen differently.

Q2 & W1: explain the failure of naive_mb

As mentioned on L203, naive_mb has a strong bias towards more deterministic systems. That is, when the candidate models have varying degrees of stochasticity, naive_mb tends to choose the least stochastic one. You observe that naive_mb performs well in the top row of Fig 2, and that’s because all candidate models in MF.G have the same degree of stochasticity. (There is a potential confusion: “MF.G” and “MF.N” are specified on L271; the value of $g$ and $\sigma$ after “:” describes $M^\star$, not “MF.G” or “MF.N”.) It’s the relative difference in stochasticity among candidate models that matter, not the absolute level. This also explains why naive_mb performs particularly well in the left plot of bottom role, since the groundtruth model has the least stochasticity and naive_mb’s bias happens to play in favor.

We believe this particular bias is well known to the theory community (since the underlying issue is a strong motivation for widely used tools such as Wasserstein distance) and thus omitted a detailed explanation. Given your feedback, however, we will add more discussion. The key is that for $\\\| s’ - \tilde{s}\\\|$ in the loss (L120), $s’$ is from the data and $\tilde{s}$ from the candidate model. When the candidate is the true model, $s’$ and $\tilde{s}$ are iid, but their randomnesses are independent and “out of sync”, which causes the loss to be non-zero and scale with the inherent randomness of the distributions. As a minimal example, suppose the state space is $\mathbb{R}^2$, and we focus on the transition distribution from a particular $(s,a)$ pair where groundtruth is uniform over 4 points $(\pm 1, \pm 1)$. Loss of groundtruth model itself is $1+\sqrt{2}/2$, which is higher than a wrong model that always predicts $(0,0)$ deterministically ($\sqrt{2})$.

Q1 & W3: Motivation for model-based selectors

We understand the sentiment that the model-based selectors feel “extraneous”. In fact, we received similar comments in a previous round of review and decided to defer the contents to the appendix, only leaving a short summary in the main text.

That said, we argue that there is strong theoretical motivations for the model-based selectors. In particular, the selectors can leverage the information about Bellman operators (L207), which is accessible in the model-free setting, and obtain standard and elegant theoretical guarantees (Theorems 3 and 4) familiar to the RL theory literature. Our intuition is that methods that properly leverage additional side information often performs better, and the “standardness” and “elegance” of the guarantees are, in our opinion, enough motivation to consider the model-based selectors.

In fact, before we implement the model-based selectors, we were fully anticipating a trade-off between performance (that model-based should outperform model-free) and applicability/computational cost (model-free applies to more settings and is cheaper to run). The results come out as a big surprise, and we think reporting such negative results is invaluable to the research community. This gets back to our comment on the research methodology: you said that the model-based methods lack motivation because it performs poorly, which reflects the “empirical-first” methodology. In our theory-first approach, we develop the methods in theory and decide to include it before we do any experiments: if the experiments confirm our hypothesis, great; if not (as we saw), the surprisingly poor performance of model-based selectors further highlights the effectiveness of LSTD-tournament. As for why model-based selectors fail, we have some speculations (see response to ZupL’s (3)).

Q3: Generalizability beyond Hopper and DDPG policies

This is a valid point, and as usual, the more experiments on more environments/settings, the always better. That said, unlike most of RL research where training algorithms have many small design choices that can be tweaked to overfit to a setting, our loss is closed-form and optimization-free, leaving very little room for overfitting.

Q1: Sec 2 pointer to Fig 5 describes Sec 5 protocol

Fig 5 left describes the practical setup, which directly corresponds to the text in Sec 2.

We thank the reviewer for your detailed explanation. We now understand your concern about "any weighting function $w(s,a)$ suffices". Indeed we are aware of what you call the "generalized LSTDQ" equation, where the "test function", or sometimes referred to as the "instrument" (which stems from an instrumental-variable (IV) view of LSTD), can be chosen to be other functions. It seems that both of your points 1 and 2 are rooted on this point.

A. Not a well-studied method. This is a valid technical point, but as far as we know, this generalized LSTDQ is a niche idea that floats around anecdotally among a small part of the community. Based on our understanding of literature, this method is never sufficiently investigated in the LSTD literature. The only reason we are aware of this is due to some conversation with researchers in the community, where "LSTD can be generalized in this way" comes up in casual conversations as a side comment. We won't give names here but our impression is that these authors may have mentioned this generalization lightly in perhaps a handful papers as a pass-by comment. We are pretty surprised that the reviewer brings this variant up and uses it as the foundation for most of your technical criticisms, but perhaps we are wrong about its role in the literature; if this variant has been systematically studied by the RL theory community, we would appreciate pointers to the thread of works on this matter.

B. Additional hyperparameters and the lack of basic properties. As we mentioned in the rebuttal, using arbitrary weighting functions may generally lose the guarantee of Theorem 2 (e.g., iid weights). When they don't, there is also the question of additional hyperparameters, e.g., how to choose $\psi$? Note that $\psi=\phi$ is not an arbitrary choice. Not only is this the most dominant choice in the literature, but it also enjoys an "on-policy boundedness" property. That is, we know that $\sigma_{\min}(A)$ serves as the coverage parameter of LSTDQ, and the quantity is lower-bounded away from $0$ when the data is sampled from an invariant distribution of the target policy. Despite the many mysteries around coverage in LSTDQ (discussed in Appendix C.5), this is a basic property that OPE estimators are expected to have (see [21]) and standard LSTDQ enjoys, whereas choosing arbitrary $\psi$ may not.

C. Our research is self-contained without examining other choices of $\psi$. We agree with the reviewer that there might be other choices of $\psi$ that make things work better, but our work is still self-contained, justified, and novel without considering them. In particular, another choice of $\psi$ will be worth considering theoretically if it satisfies two conditions

1. It can be constructed without requiring further information. In our case, the only thing we can rely on is $\\\{Q\_i\\\}$ in the model-free case. (You mentioned the sign-flip estimator as another choice, but that uses the Bellman operator through the $\mathcal{G}$ class and is not applicable to the model-free setting.)

2. It has theoretical properties better than (or at least as good as) $\psi=\phi$, including the "on-policy boundedness" property.

We simply aren't aware of any choice of $\psi$ that satisfies both. Of course exploring alternative choice opens an interesting venue of future research, which we are happy to discuss at the end of the paper. But the potential existence of them does not affect our work: our LSTD-tournament is a framework where any improvement in the "base algorithm" (i.e., LSTDQ) can be directly plugged in to produce a better model-selection algorithm. When theoretically more motivated choices are not obvious---which we believe is the case now---it is only natural to start investigation with the most standard choice widely considered in the literature.

The reviewer also asks us to "eliminate the vast space" to make it "a theoretical necessity". We don't understand how this is doable or even necessary for a publication. As we mentioned above, we believe that better choices can possibly exist, just that we do not know for now. When you design an algorithm, can you only publish it once you prove that no better variants exist? If you were reviewer for all the LSTD papers in the literature (to which your criticisms also apply to), would you reject most of them because they don't study the $\psi$ variant? Perhaps your did not know the "on-policy boundedness" property, but even without it, we feel that the mere fact that $\psi=\phi$ is the standard choice in the literature & no obvious better choice exists should suffice.

normalized diff is NOT using generalized LSTDQ: Upon responding to your comments, we realize a subtle technical point. Your main technical concern is about the vast space of $\psi\ne\phi$, and you mentioned that variants like normalized diff belong to it. That is not correct; for all variants considered in our paper, we are always using the same standard LSTDQ algorithm, only with different choices of $\phi$ (up to linear transformations). $\psi=\phi$ always holds. (In fact, if you look at Appendix E.2, we only specify $\phi$ and never need to introduce $\psi$.) It is indeed true that the final loss will look like what you are imagining: that the TD error part remains the same and only the "test function" changes. This is because when we change the definition of $\phi$ by linear transformation, the candidate $\theta$ will also change accordingly (they are no longer [0, 1], [1, 0] in these variants), and in the loss derivations in Eq.7, the $\phi^\top$ and $\theta$ will always recombine into the familiar TD error.

1. Model-based explanation: We are glad you find the explanation clear. As we mentioned, this is a widely known issue (for which the double-sampling problem can be viewed as a manifestation) and one of the main motivations for studying distribution distances such as Wasserstein's. We will include it in the appendix for readers unfamiliar with the issue. If you have other concrete concerns, we are happy to address in a similar manner.

2. Forward pointer and organization: We ran out of space in rebuttal and unfortunately could not respond to these non-technical issues in detail, but we did respond to the forward pointer issue of Figure 5. Note that the left panel of Figure 5 illustrates a practical setting, which directly corresponds to the text in Section 2. If given more space, we plan to bring Figure 5 to the main text, likely where the Section pointer 2 is. We don't think the existence of the right panel (protocol in Section 5) is a big issue, since the two panels are best viewed side by side as a comparison.

foward pointer to $\mathcal{G}$ in Section 3.1: this is in the parenthesis and we believe the sentence outside is self-contained ("Common approaches to debiasing this objective involve additional “helper” classes"). The parenthetical comment serves as supporting evidence which readers can skip if they believe the leading sentence, and we could very well cite a paper (e.g., Antos et al. [6]). Since we already discussed this later in the paper ourselves, we thought that a pointer to the later contents saves the readers the effort of looking up the literature.

Section 5.2 (Computational Efficiency) [refers to] Section 6.2: again this is a parenthetical comment, which readers can skip without losing the main picture. The computational efficiency is a side comment anyway since most of our paper (both theoretically and empirically) consider the statistical aspects of the algorithms, so a brief discussion of the computational aspects at the end of the protocol section, before the empirical results, seems a natural choice to us.

3. Single-environment is still a limitation despite less overfitting: We completely agree and are happy to include it in the discussion of limitations in the main paper. This is also why Sec 6 is called "Exemplification of the protocol" (not "Experiments") and the introduction also highlights the "preliminary" (L69) nature of our empirical results. Given that the theoretical developments and the design of protocol are also significant contributions, we feel that a preliminary experiment section would be an reasonable supplement.

We thank the reviewer for the consideration of our response and further elaborations. We understand the limited time and efforts the reviewer has on our paper and the intention to make this discussion "final", and we would still like to provide response to your comments for you and other reviewers.

We believe the reviewer's main criticisms (related to revision suggestions 1 and 3) are due to your unfamiliarity of the RL theory literature.

the relevance of this [on-policy boundedness] property in the challenging off-policy setting is not as clear

First, we believe we have successfully convinced the reviewer that,

if this property is relevant, it should largely ease your concern on the hyperparameter-free claim, as it provides a unique theoretical advantage for standard LSTDQ compared to the infinitely many variants the reviewer is suggesting.

We will greatly appreciate it if you can acknowledge this, which will make it easier to explain the core issue to other reviewers and the AC and allow them to make independent judgements.

There is wide consensus of the significance of the property in the off-policy setting in the offline RL theory community. Basically, while it's insufficient (which is aligned with your description of the off-policy setting as "challenging"), it serves as an important sanity-check and necessary condition: in offline RL, the coverage parameter characterizes how the data distribution covers that induced by the target policy. In general, we expect the parameter to be small when the two distributions are close (i.e., on-policy), and gradually increase when the data distribution deviates from the target distribution. $C^\pi$ and $C\_\infty^\pi$ in the model-based guarantees are standard coverage parameters (they also appear in the guarantees of FQI, MWL/MQL, etc) that satisfy this property. If a guarantee comes with a coverage parameter that doesn't have this property, it will be considered highly problematic and of little relevance. So our developments of the algorithms follow the rationale of "obtaining this kind of guarantees is very nontrivial, and we provide 3 ways of doing so".

Disconnected Theory and Model-Based Selectors; more integrated theory

We agree with the technical point that the coverage parameters between model-free and model-based selectors are incomparable. In fact we explicitly acknowledged this and offered a nuance discussion in Appendix C.5. That said, the on-policy boundedness property is a framework that can unify both, though admittedly it is a very loose characterization. While it would be certainly nice to provide a tighter unification, this is a long-standing open problem in offline RL theory, and it is very unreasonable to expect us to solve it in a paper where this is not the focus. In fact, this problem is not specific to our work: if you compare the guarantees of LSTDQ in the literature (which is analogous to our Theorem 1) with those of other algorithms (e.g., FQI, MWL/MQL, which are analogous to our Theorems 3 and 4), you'd reach the same conclusion that they lack a unified framework and are incomparable. This open problem is explicitly acknowledged through a comment on the unusual nature of coverage parameters in LSTDQ and abstractions (compared to those of FQI, MWL/MQL, etc), in a survey that summarizes the recent advances in offline RL theory [21]:

This is a general theme with algorithms that only require realizability [such as LSTDQ and state abstractions], that error propagation and coverage are conceptually much more complicated and hard to interpret.

[21] Offline reinforcement learning in large state spaces: Algorithms and guarantees. Jiang and Xie. 2024.

[weight functions] in minimax approaches to OPE [1]

Indeed, LSTD-tournament looks very similar to MQL in [1] you cited. We are very familiar with the work and intended to comment on it (and did not due to space limit). In MQL, the weight function needs to capture the importance weight function, which requires an additional realizability assumption and thus hyperparameter choices. This is exactly why algorithms like MQL cannot be considered hyperparameter-free for model-selection and we have to base our theory on a different analysis. We also acknowledged that better choice of $\psi$ could exist, which is an interesting future direction and shouldn't be treated as a major flaw of our current work.

To conclude, we appreciate your thoughtful comments and believe we fully understand your technical points, and agree that there is room for improving clarity and structure of the paper. However, we stand firmly by our position and reject the reviewer's criticisms on the "hyperparameter-free" issue and the lack of motivation for model-based selectors (which already have minimized presence in the main text), which form the basis of the reviewer's technical concerns.

We thank the reviewer for the comments. Unfortunately the reviewer misunderstood a major technical point of the paper regarding linear function approximation, which we explain below. We then respond to your other comments.

We do NOT assume Q functions follow linear function approximation.

Line 141 is explaining the background knowledge of LSTDQ, which is indeed a method for learning Q-functions with linear function approximation (LFA). However, when we apply it to the model selection problem, we do not make any assumptions on the Q-functions we select from. They can be obtained by training neural networks using FQE on offline data (Figure 5 Left), or in our protocol, implicitly defined as the Q-function of the simulator MDP (Figure 5 Right). In any of these cases and in our theoretical development, we do not assume these candidate Q-functions to be generated/learned via LFA. The reason LFA is mentioned is because, when we select among the candidates, we use LSTDQ as a subroutine. LSTDQ indeed requires linear features, and these features are constructed from the candidate Q-functions themselves (Eq.6). Essentially we reduce the problem of selecting from general candidate Q-functions (which is difficult and very few solutions exist) to the subproblem of learning with linear features (which is well studied and has algorithms with strong guarantees), and how to do this reduction (without relying on additional hyperparameters or side knowledge) is the key innovation of our method. Also, if you skip the derivations, treat it as a blackbox, and jump right to the final loss (Eq.(10)), you will see that the equation is simple, closed-form, and has nothing to do with LFA. You asked whether the “proofs [are] based on this assumption” – as you can tell from the paper, no we don’t have this assumption in our main theoretical results (see Theorem 2 for example).

Background of BVFT and double-sampling

BVFT is a complicated method and counterintuitive in many ways, and it takes very nontrivial efforts to grasp its main idea even for experts in offline RL theory if they have not seen the work before. So unfortunately we don’t think we can do justice to the method in the limited space. Our paper is written in a way that is meant to be self-contained, and theorists should be able to follow our derivations without needing to know BVFT in the first place, even if we did take the high-level spirit of BVFT (as acknowledged on Line 132) when designing our method. Also, we have provided theoretical and empirical comparisons to BVFT even if the readers don’t know the latter; for example, Line 53 and 173 mention that we have improved rates compared to BVFT, which can be appreciated without knowing the details of the method.

For double sampling, this is a well-documented issue first observed in 1995 [7] and also discussed in Sutton & Barto’s textbook [50] (see Chapter 11.6 in 2nd ed). For a concise explanation, see Section 3.1 of [9]. Again, for the purpose of our paper, the readers just need to know that TD-sq is biased, which we mentioned.

Experiments in real-life applications

We agree that this should be what the OPE community strives for. Unfortunately, 99% of the OPE works (in fact we don’t know any work that does it) don’t do that for understandable reasons: unlike most of ML where realistic experiments only require realistic data (e.g., ImageNet), RL experiments on real-life applications require engaging with the actual system. For OPE specifically, if we only have off-policy data, how do we know the groundtruth performance of the policy? The only way to get reliable groundtruth is to run the target policy in the real-system, and the difficulties in doing so is usually why OPE is needed in the first place. Even if technically possible, this requires closely working with domain experts and obtain access to real-life systems where decisions can cause financial and societal consequences (e.g., in online recommendation, you need to work closely with the company and get their permissions, which often requires an official affiliation with the company), which is a resource not many researchers have access to. All that said, we agree with the sentiment and believe the community as a whole should make efforts to get closer to real-life systems, though this is perhaps a luxurious request at the moment.

Thank you for the further explanation. I can further lower my confidence score. I think perhaps whether the concerns of Reviewer Robe and other reviewers are well addressed matters for the acceptance of this paper.

Thank you for the response. I am OK with your point 3. In your point 1, you do not assume the linear function approximation, so Theorem 1 in the paper is an analysis of the previous algorithm?

About point 2, I have no further comments as I am not very familiar with BVFT. Regarding the paper writing, I think a clearer statement or organization of the paper would be helpful. Perhaps describing the method you proposed first, and then giving the theorem. (No further changes are required for the paper writing in this discussion, as the reviewer cannot fully know the detailed changes. Take this as a suggestion.)