Author response to reviewer jGpc

We sincerely thank Reviewer jGpc for the recognition of our work and for providing constructive comments.

Q1: Notational Clarity.

Thank you for your careful reading and thoughtful comments. To avoid confusion, we have revised the notation in Algorithm 1: the subsample size, previously denoted by $m$ in the submitted version, is now written as $l$ in the revised manuscript.

Q2: Clarify Methodology for Off-Policy Evaluation (Line 171-178).

Thank you for your valuable comments. Lines 171–178 aim to explain how the QTD algorithm of [3] can be adapted to the off-policy setting. The QTD algorithm presented in Algorithm 1 of [3] is designed for the on-policy setting. Extending QTD to the off-policy setting requires careful modifications to account for the distribution shifts between the behavior policy used to collect data and the target policy.

• In the on-policy case, QTD estimates the return distribution conditioned on the initial state, $\widehat{\eta}^{\pi}(s)$, using the following iterative update:

  $$\theta(s,i) \leftarrow \theta(s,i)+\alpha\frac{1}{m}\sum_{j=1}^{m}\left[\tau_i - I(r+\gamma\theta(s^\prime,j)-\theta(s,i)<0) \right],$$

  where $\theta(s, i)$ denotes the $i$-th quantile of $\widehat{\eta}^{\pi}(s)$ and $(s, a, r, s')$ is sampled under the behavior policy $\pi$, which in the on-policy case coincides with the target policy.

• In contrast, the off-policy setting requires estimating the return distribution conditional on both the state and the action, $\widehat{\eta}^{\pi}(s, a)$, using the modified update:

  $$\theta(s,a,i) \leftarrow \theta(s,a,i)+\alpha\frac{1}{m}\sum_{j=1}^{m}\left[\tau_i - I(r+\gamma\theta(s^\prime,a^\prime,j)-\theta(s,a,i)<0) \right],$$

  where $\theta(s, a, i)$ is the $i$-th quantile of $\widehat{\eta}^{\pi}(s, a)$, and $(s, a, r, s')$ is sampled from the behavior policy $\pi_b$. Crucially, $a'$ is drawn from the target policy $\pi$, and the result is marginalized over the action space according to $\pi$. This added complexity arises from the need to correct for the action distribution mismatch between the behavior and target policies.

• This distinction is analogous to the difference between the TD algorithm for estimating the state-value function $V^{\pi}(s)$ and the SARSA algorithm for estimating the state-action value function $Q^{\pi}(s,a)$. We will clarify this point in the revised paper and refer readers to relevant literature, including [1] and [2] for more details on distributional off-policy evaluation.

Q3: Clarify Methodology used in Eqn. (4).

Thank you very much for the insightful comments. We estimate $P(\delta \mid s)$ using logistic regression in our implementation, although other models such as neural networks can also be employed. As for the ratio $P(\delta = 0)/P(\delta = 1)$, it is a constant and does not need to be explicitly estimated, since we normalize the resulting weights into probabilities in the subsequent weighted subsampling procedure. We will clarify this point in the revised paper.

Q4: In Algorithm 1, conformal predictive region can be replaced with interval. And the impact of $\xi$ is not discussed in the paper (although it is discussed in the Appendix).

Thank you for your thoughtful comment. We will replace the conformal prediction region in result line of Algorithm 1 by conformal prediction interval. In addition, following your suggestion, we conducted experiments with $\xi=0,1,0.2,\ldots,0.9$ in Example 1, as shown in Rebuttal Table C. The results indicate that smaller $\xi$ and larger $k$ tend to produce over-coverage, while overall, settings with $\xi\ge0.5$ and $k=2,3$ yield satisfactory performance. We will incorporate these findings into the paper.

Rebuttal Table C. We conduct the experiments for Example 1 across $xi$ for $k=2,3,4$. Each experiment is repeated 100 times, and we report the mean and standard deviation of coverage probability (cov) and interval length (len) at the nominal coverage rate 90%.

Q5: A brief discussion or experiment on bounded Lebesgue density would strengthen the practical relevance.

We appreciate your constructive comments. Our response is structured in three parts:

• The assumption of a bounded Lebesgue density applies only to continuous return distributions, since discrete random variables are not absolutely continuous with respect to the Lebesgue measure. Accordingly, this condition is not required or applicable when the return is discrete, as in Example 3 of our experiments. We acknowledge this and will clarify it in the revised paper.

• For continuous return distributions, the bounded density assumption is mild and satisfied by many commonly used distributions such as Gaussian, exponential, and Gamma with shape parameter no less than 1. For instance, in Examples 1 and 2 of our experiments, the return is continuous and satisfies the bounded density condition.

• To illustrate, consider Example 1 in our paper. In this example, the state space of the environment is discrete with two possible values: $S_t=1$ and $S_t=2$. The agent transfers from a current state to a different state with a certain probability determined by the policy and the discount factor is $\gamma = 0.8$. The reward obtained when transitioning from state $1$ is distributed as $N(2, 1)$, and the reward obtained when transitioning from state $2$ is distributed as N (1, 1), i.e., $R_t|S_t\sim N(-3S_t+5,1)$. Given a history of states $(S_0,S_1,\ldots)$, the $R_t$'s are independent Gaussian variables, so the return $G=\sum_{t=0}^\infty\gamma^tR_t$ also follows a Gaussian distribution. Please note that the Gaussian density is upper-bounded by $1/\sqrt{2\pi\sigma^2}$ where $\sigma^2$ is its variance. The variance of $G$ conditional on $(S_0,S_1,\ldots)$ is $\sum_{t=0}^\infty \gamma^{2t} = 1/(1-\gamma^2)$, so the conditional density is upper-bounded by $\sqrt{\frac{1-\gamma^2}{2\pi}}$. Since this upper bound is irrelevant with $(S_1,S_2,\ldots)$, the conditional density of $G$ given $S_0$ is also bounded by $\sqrt{\frac{1-\gamma^2}{2\pi}}$. Similarly, we can verify that Example 2 also satisfies this bounded density condition.

Reference

[1] Distributional Off-policy Evaluation with Bellman Residual Minimization.

[2] Distributional Off-Policy Evaluation in Reinforcement Learning.

[3] An analysis of quantile temporal-difference learning.

Author response to reviewer SpYL

We thank Reviewer SpYL for your careful reading and valuable comments.

Q1: Problematic claims

• Q1-1: We agree that in certain classes of MDPs the stationary distribution $\mu^\pi(s,a)$ can be well estimated. However, our method depends critically on a more challenging quantity: the long-term return distribution $\eta^{\pi}(s) = \mathrm{Law} \left( \sum_{t=0}^{\infty} \gamma^t r_t \middle| s_0 = s \right).$ Estimating this distribution (e.g., quantiles) requires knowledge not only of $\mu^{\pi}$ but also of the conditional distribution of cumulative future rewards. For a detailed background, please see DRL literature such as [1].

  In infinite-horizon settings, the return is inherently unobservable from finite trajectories, especially under off-policy data, which motivates our work.

  While long-horizon trajectories enable Monte Carlo estimation, it is more common to exploit time-homogeneity of MDP to use all immediate transitions, as in temporal difference (TD) methods, which is our focus in this work.

• Q1-2: The term “model misspecification” refers to the difficulty of constructing valid PIs using estimated return distributions. For example, quantiles estimated by DRL models may be inaccurate, leading to poor coverage.

  While we agree that CP is model-free with guaranteed marginal coverage, applying CP to infinite-horizon and continuous RL remains challenging. Our work addresses this gap, and we will clarify this point in the paper.

• Q1-3: While quantile-based intervals via empirical CDFs are standard, in infinite-horizon RL, full return trajectories are rarely observed. Thus, direct estimation is often infeasible. Though long-horizon Monte Carlo rollouts can approximate return distributions, they are often sample-inefficient, especially in off-policy or high-variance settings. Instead, like much of RL, our work adopts the TD-based approach, where return quantiles are learned from immediate transitions. “DRL-QR” refers to intervals built from these TD-based quantile estimates.

• Q1-4: WCP typically assumes access to full test-time covariates, while our setting only observes the initial state $S_{\text{test},0}$ at test time. Estimating the WCP weight from Foffano et al. (2023, Eq.12) requires marginalizing over full trajectories, which are unobserved and intractable to approximate without strong assumptions.

  We may estimate the WCP weight by solving the objective function in their Eq.14, this requires additional modeling assumptions and may suffer from high variance, particularly in long-horizon settings. The resulting weights can be unstable and sensitive to model choice, limiting practical use in our setting.

  Finally, simulating trajectories with learned transition model $p(s' \mid s,a)$ introduces approximation error and places the burden of accuracy on the model class. More critically, it doesn’t address a key requirement of conformal prediction: distributional alignment between calibration and test data.

• Q1-5: We initially considered a TV-based coverage bound following [5], but it fails to capture how rollout length $k$ affects distributional mismatch. Inspired by [10] which shows Wasserstein distance better reflects conformal coverage under distribution shift (see their Section B for a detailed comparison), we adopt a Wasserstein-based bound to characterize the effect of $k$.

Q2: Missing assumptions and so on

• Q2-1: We assume a standard time-homogeneous Markov Decision Process with data collected from i.i.d. trajectories under a fixed behavior policy. We originally stated these assumptions in the Supplement and will move them to the main paper.

• Q2-2: In our notation, $\mathcal{P}$ denotes a probability distribution. For example, $\mathcal{P}_0$ represents the distribution of the initial state. Since we only observe the initial state of the test trajectory, we refer to this as the distribution of the test state. We use $P$ to denote the probability measure induced by a distribution.

• Q2-3: Lines 171–178 explain how the QTD algorithm from [7], originally developed for the on-policy case, is adapted to the off-policy setting in our work. The key change is that we estimate the return distribution $\widehat{\eta}^\pi(s,a)$ instead of $\widehat{\eta}^\pi(s)$, and account for the mismatch between the behavior policy $\pi_b$ and the target policy $\pi$ by sampling $a'$ from $\pi$ and marginalizing accordingly. This is conceptually similar to moving from TD (estimating $V^\pi(s)$) to SARSA (estimating $Q^\pi(s,a)$). For full update rules, please see our response to Q2 of Reviewer #4. We will clarify this point and cite [4,8] for related work on off-policy distributional evaluation.

• Q2-4: Our responses are organized point-by-point below.

  1. $\mathcal{P}_0$ and assumptions. Please refer to Q2-1 and Q2-2.

  2. On $n$. Please refer to Q4-4.

  3. Boundedness of density. The assumption that the Lebesgue density is bounded holds for any continuous random variable that has a continuous density function that does not tend to infinity at the boundary of its support. Examples include Gaussian distribution, exponential distribution, etc. It can be verified that both Example 1 and Example 2 satisfy this assumption. Please refer to our response to Q5 of Reviewer #4 for details.

  4. Assumptions of IS. We interpret the comment as referring to the weighting function $w$. Our analysis only requires a bounded first moment of $w$, which is a common and mild assumption in the CP literature, including [5,11].

  5. Asymptotic coverage. The theorem ensures asymptotic coverage as the number of calibration samples $n$ tends to infinity, not the number of trajectories. Please refer to Q4-4.

• Q2-5: To address the issue of temporal dependence among transitions, we follow a common technique in the RL literature known as experience replay ([6,7]). Specifically, we apply weighted subsampling from a large replay buffer to construct a subset of transitions that are approximately i.i.d. This approach is widely used in RL to break temporal correlations and is supported both empirically and theoretically in prior work such as [2].

Q3: Weekness 3-5

• Q3-1: Weekness 3. Following your suggestion, we add a comparison with Foffano et al. (2023) in Rebuttal Table A, where our method demonstrates superior performance across coverage probability and average length.

  We highlight that Foffano et al.(2023) address finite-horizon MDPs and mitigate distributional shift via importance weighting (see their Eq.12), but their method is subject to the curse of horizon. In contrast, our approach targets the infinite-horizon setting and reduces the challenge to a $k$-step distributional shift problem. Please refer to our response to Q1 of Reviewer #1 for further discussion.

  Rebuttal Table A. We compare the performance on Example 1 with a fixed horizon of 20. For Foffano’s method, we use their gradient-based linear regression to train $w(x, y)$ and apply WCP. Each experiment is repeated 100 times, and we report the mean and standard deviation of coverage probability (cov) and interval length (len) at the nominal coverage rate 90%. |||| |-|-|-| ||cov|len| |Foffano|0.85(0.01)|6.51(0.11)| |our proposal||| |k=2,$\xi$=0.5|0.91(0.01)|7.67(0.15)| |k=3,$\xi$=0.5|0.91(0.01)|7.77(0.13)| |k=2,$\xi$=0.6|0.92(0.01)|8.00(0.16)| |k=3,$\xi$=0.6|0.92(0.01)|8.02(0.15)|

• Q3-2: Weekness 4. We agree that off-policy evaluation poses major challenges for applying CP to MDPs. Addressing this is a central contribution of our work. To mitigate severe distributional shift in the infinite-horizon setting, we estimate the return distribution using DRL and apply CP to correct model misspecification. We also propose $k$-step pseudo returns to reduce the infinite-horizon shift to a more tractable $k$-step shift.

• Q3-3: Weekness 5. Thanks for pointing out this interesting paper. We will cite it.

Q4: Questions 1-5

• Q4-1: A confidence interval is an interval estimate for a parameter (fixed), whereas a prediction interval is an interval estimate for a random variable (random) with a large probability. Given an initial state of a trajectory, the return is unobserved and can be seen as a random variable. This is why we use “Prediction interval” instead of “Confidence interval”. “PI” is defined at Line 23 and follows standard terminology in CP.
• Q4-2: The event that $k$ exceeds the trajectory length will never happen as we are focusing on the infinite-horizon settings.
• Q4-3: We apologize for the typo in $\Lambda$ in Theorems 1/2. It should be the sum of two estimation errors and we will correct it.
• Q4-4: As defined in Section 2, $N$ is the number of trajectories and $T$ the horizon. In line 242, $n$ denotes the number of calibration samples, not trajectories. Since we extract overlapping $k$-step transitions, $n \approx NT/2$ in split CP.
• Q4-5: In Algorithm 1, $\mathcal{I}$ refers to the index of the transition $(S_{i,t}, A_{i,t}, R_{i,t}, S_{i,t+1})$ in the training dataset. We will add this definition.

Q5: Possible typos

We will correct these typos in our paper.

[1] A distributional perspective on reinforcement learning.

[2] A theoretical analysis of deep Q-learning.

[3] Conformal off-policy evaluation in markov decision processes.

[4] Distributional Off-policy Evaluation with Bellman Residual Minimization.

[5] Conformal inference of counterfactuals and individual treatment effects.

[6] Self-improving reactive agents based on reinforcement learning, planning and teaching.

[7] Human-level control through deep reinforcement learning.

[8] Distributional Off-Policy Evaluation in Reinforcement Learning.

[9] An analysis of quantile temporal-difference learning.

[10] Wasserstein-regularized conformal prediction under general distribution shift.

[11] Conformal off-policy prediction.

I thank the authors for their response. Just a last note, the samples from the buffer are conditionally iid, and for completeness, it should be sampling with replacement.

I will update my evaluation in the coming days after re-reading the paper and the rebuttal. However, I strongly suggest the authors to incorporate the feedback, provide a proof strategy in the main text, and better clarify some of the points that were discussed. Also better structure the proofs in the appendix would greatly help.

I believe most of the theoretical weaknesses have been addressed, including the comparison with Foffano. However, it is still unclear the scalability to high dimensional problems, and large values of $k$, especially in the off-policy setting. While the authors promised to include high-dimensional settings in one of their comments, it is unclear at the moment if that is feasible. Therefore that remains a weaknesses to me.

Thank you for your follow-up comments.

• Regarding your first note, in our paper, the samples from the replay buffer are indeed drawn with replacement.
• We appreciate your suggestions on improving the paper. As promised in our rebuttal, we will incorporate the feedback and improve the structure and clarity of both the main paper and the Supp.
• Regarding your last point on the scalability to large values of $k$, in practice we do not adopt excessively large values of $k$. Please refer to our response to Q3 of Reviewer #1 and Rebuttal Table B for details in both the on-policy and off-policy settings.
• Regarding the scalability to high-dimensional problems, for sure our method is readily applicable to this setting, as long as the underlying DRL algorithm can handle high dimensional settings. This is because our method is a CP framework, and CP can automatically incorporate any statistical and machine learning models or algorithms.
• In addition, we have been actively conducting experiments in high-dimensional scenarios, with preliminary results reported in Rebuttal Table D.

Rebuttal Table D. We extend Example 1 to a 50-state setting, where each feature is binary. The action space is {0,1}, affecting transitions of only one state. Rewards follow the same distribution as in Example 1. We apply quantile temporal difference (QTD) learning with linear regression and a ridge penalty to mitigate overfitting. The behaviour policy specifies the probabilities of transferring from $x_1=1$ to $x_1=2$ and $x_1=2$ to $x_1=1$ are 0.4 and 0.8, respectively. The target policy is identical to the behavior policy. Experiments are run for $k = 1, \dots, 5$, each repeated 50 times, reporting the mean and standard deviation of coverage probability (cov) and interval length (len).

Author response to reviewer dgcF

We sincerely thank Reviewer dgcF for the recognition of our work and for providing constructive comments.

Q1: Can you detail briefly how this article improves on the current literature ?

We thank the reviewer for raising this important question regarding the positioning and contribution of our method within the conformal prediction literature for RL. We agree that this is an important point and have revised the Introduction section accordingly.

• Positioning in Conformal RL Literature. Our work is the first to tackle infinite-horizon off-policy prediction in general reinforcement learning settings for any state and action spaces using conformal prediction. Most existing conformal RL methods concentrate on finite-horizon scenarios or finite state or action spaces, as noted in [1] and [5]. Our approach extends conformal prediction to the infinite-horizon setting while preserving asymptotic coverage guarantees, making it both methodologically novel and technically challenging.

• Differences in Modeling Assumptions. As mentioned above, the primary distinction lies in the infinite-horizon framework and the consideration of more general state and action spaces. Furthermore, while most existing methods require access to complete trajectories, our approach remains effective even when only partial trajectory fragments are available, provided that a $k$-step pseudo return can be constructed.

• Scalability. Prior conformal RL methods typically address distribution shifts between behavior and target policies through trajectory-level importance weighting, which exhibits computational inefficiency scaling for trajectories with increasing horizon. In contrast, our approach utilizes $k$-step pseudo returns and address distribution shift only through a $k$-step importance weighting, therefore it can be implemented efficiently in infinite-horizon settings and continuous tasks, as demonstrated by our experiments on the MountainCar environment.

We will add a more detailed discussion of these points in the revised paper.

Q2: Since algorithms like QTD are proven to converge with guarantees on the limiting distribution, isn't that sufficient?

We appreciate this valuable comment. Our response is structured in three parts:

1. Limiting distribution. While QTD is proven to converge to some limiting distribution, this limiting distribution is not guaranteed to be the true return distribution. Please see the last paragraph in Section 4 of [4]. Consequently, this convergence does not provide any guarantee on the asymptotically validity of QTD-based prediction intervals.

2. Tabular settings. [4] proved the convergence of QTD only in the contex of finite state and action spaces. To the best of our knowledge, no existing distributional RL algorithm has been proven to consistently recover the true return distribution in general settings with continuous state and action spaces.

3. Function approximation. In continuous state and action spaces, distributional RL methods necessarily employ function approximation for return distribution estimation. The theoretical validity of such approaches is inherently contingent upon the accuracy of the modeling assumptions, rendering them susceptible to potential model misspecification. Conformal prediction helps address this by providing model-agnostic prediction intervals, thereby improving the robustness and reliability of uncertainty quantification in practice.

We will incorporate the above discussions in the ''Limitations of DRL" paragraph.

Q3: Clarification on Finite-Sample and Asymptotic Coverage Guarantees

Thank you for pointing this out. We apologize for the confusion. You are absolutely right that our current theoretical results establish asymptotic coverage guarantees. We will revise the wording in lines 37 and 75 in the Introduction section, and in line 323 in the Conclusion section to clarify this point.

Q4: Minor Remarks

Thank you very much for your careful reading. We will incorporate the following changes in the revised paper:

• We will rewrite the code headers in English to ensure better anonymity and clarity.

• In Algorithm 1, $\mathcal{I}$ refers to the index of the transition $(S_{i,t}, A_{i,t}, R_{i,t}, S_{i,t+1})$ in the training dataset. We will explicitly add this definition in our paper for clarity.

• The QTD algorithm presented in Algorithm 1 of [4] is designed for the on-policy setting. Extending QTD to the off-policy setting requires careful modifications to account for the distribution shifts between the behavior policy used to collect data and the target policy.

  Specifically, in the on-policy case, QTD estimates the return distribution conditioned on the initial state, $\widehat{\eta}^{\pi}(s)$, using the following iterative update:

  $$\theta(s,i) \leftarrow \theta(s,i)+\alpha\frac{1}{m}\sum_{j=1}^{m}\left[\tau_i - I(r+\gamma\theta(s^\prime,j)-\theta(s,i)<0) \right],$$

  where $\theta(s, i)$ denotes the $i$-th quantile of $\widehat{\eta}^{\pi}(s)$ and $(s, a, r, s')$ is sampled under the behavior policy $\pi$, which in the on-policy case coincides with the target policy.

  In contrast, the off-policy setting requires estimating the return distribution conditional on both the state and the action, $\widehat{\eta}^{\pi}(s, a)$, using the modified update:

  $$\theta(s,a,i) \leftarrow \theta(s,a,i)+\alpha\frac{1}{m}\sum_{j=1}^{m}\left[\tau_i - I(r+\gamma\theta(s^\prime,a^\prime,j)-\theta(s,a,i)<0) \right],$$

  where $\theta(s, a, i)$ is the $i$-th quantile of $\widehat{\eta}^{\pi}(s, a)$, and $(s, a, r, s')$ is sampled from the behavior policy $\pi_b$. Crucially, $a'$ is drawn from the target policy $\pi$, and the result is marginalized over the action space according to $\pi$. This added complexity arises from the need to correct for the action distribution mismatch between the behavior and target policies.

  This distinction is analogous to the difference between the TD algorithm for estimating the state-value function $V^{\pi}(s)$ and the SARSA algorithm for estimating the state-action value function $Q^{\pi}(s,a)$. We will clarify this point in the revised paper and refer readers to relevant literature, including [2] and [3] for more details on distributional off-policy evaluation.

Reference

[1] Conformal off-policy evaluation in markov decision processes.

[2] Distributional Off-policy Evaluation with Bellman Residual Minimization.

[3] Distributional Off-Policy Evaluation in Reinforcement Learning.

[4] An analysis of quantile temporal-difference learning.

[5] Conformal off-policy prediction.

Author Response to Reviewer g7gT

We sincerely thank Reviewer g7gT for the recognition of our work and for providing constructive comments.

Q1: Add comparison with Foffano et al. (2023), Zhang et al. (2023) and risk-aware RL.

We appreciate the constructive feedback provided. Following the suggestion, we add a comparison with Foffano et al. (2023) in Rebuttal Table A, where our method demonstrates superior performance in terms of both coverage probability and average length.

We would like to highlight that Foffano et al. (2023) focused on finite-horizon Markov Decision Processes (MDPs) and addressed the distributional shift problem using estimates of importance sampling weights. The importance sampling weight function (see Equation (12) of Foffano et al. (2023)) is

$$w(x,y) = E_{\tau\sim P_{\tau|X=x,Y=y}^{\pi_b}} \left[ \frac{\prod_{t=1}^H\pi(a_t|x_t)}{\prod_{t=1}^H\pi^b(a_t|x_t)} \right],$$

where $H$ is the time horizon, $\tau = (x_1, a_1, r_1, x_2, a_2, r_2, \ldots, x_H, a_H, r_H)$ is a trajectory of length $H$, $x$ is the initial state, $y$ denotes the $H$-step cumulative return under the behavior policy $\pi^b$, and $\pi$ is the target policy. Foffano et al. (2023) repeatedly emphasized the difficulty caused by the curse of horizon in the concluding paragraphs of their Sections V-B.1, V-B.2, and VI-D.2. In contrast, our approach is specifically designed for the infinite-horizon setting, where we construct pseudo-returns based on truncated rollouts. This leads to a fundamentally different framework that avoids the compounding challenges associated with long-horizon importance weighting.

Rebuttal Table A. We compare the performance on Example 1 with a fixed horizon of 20. For Foffano’s method, we follow their gradient-based approach to train the likelihood ratio model $w(x, y)$ via linear regression, and apply WCP to construct prediction intervals. For our method, we replace the nonconformity score with the double-quantile score from Foffano et al.(2023), with $\xi$ set to 0.5 and 0.6. Each experiment is repeated 100 times, and we report the mean and standard deviation of coverage probability (cov) and interval length (len) when the nominal coverage level is 90%.

As for Zhang et al. (2023), their COPP algorithm is specifically designed for contextual bandits and can only be extended to short-horizon decision-making scenarios (e.g., $H=5$). In such short-horizon settings, employing truncated rollouts, as done in our method for infinite-horizon problems, is generally unnecessary. Moreover, the resampling-and-matching strategy proposed in COPP is limited to finite discrete action spaces, whereas our method is applicable to both discrete and continuous action spaces. Therefore, we consider the two methods to be fundamentally different in scope: COPP targets short-horizon, discrete-action problems, while our approach is designed for more general and challenging infinite-horizon settings. As such, a direct comparison is not applicable.

Finally, we thank the reviewer for pointing us to the literature on risk-aware RL. We acknowledge that risk-aware RL constitutes an important line of work aiming to optimize risk-sensitive objectives, such as CVaR or variance-penalized returns (e.g., [1,2]). These methods focus on modeling risk preferences rather than constructing valid uncertainty quantification for prediction intervals. We will incorporate a discussion on this literature in the revised ``Related Work'' section.

Q2: Apart from line 228 about Zhang et al 2023, why not Foffano et al. (2023)?

We appreciate the reviewer's insightful question, which prompted deeper reflection on the relationship between our method and that of Foffano et al. (2023). According to our understanding, the method proposed by Foffano et al. (2023) consists of two key components:

• A new construction of nonconformity scores, such as the double-quantile score;
• The estimation of importance sampling weights, as described in our response to Q1.

These components are combined within the standard Weighted Conformal Prediction (WCP) framework. As such, implementing Foffano et al. (2023)'s method in our setting on line 228 essentially amounts to implementing WCP. In response, we will add this point in line 228. While this is feasible, we discuss the limitations of WCP in lines 224--230 and provide a rationale for adopting the Weighted Subsampling approach instead, as detailed in lines 215--218.

Moreover, we find their construction of nonconformity scores particularly insightful. Given that our framework permits flexible selection of nonconformity scores (Step 3), their double-quantile score can be seamlessly integrated. We have presented supporting simulations in Rebuttal Table A. In response, we will add this point in line 153.

Q3: Add exploration of $k$, adaptively choosing $k$, and aggregation of PIs across $k$

Q3-1: Add exploration of the step size $k$

Following your suggestion, we conducted additional experiments with $k = 6, 7, 8$ in Example 1. The results presented in Rebuttal Table B demonstrate that larger $k$ values consistently lead to overcoverage and consequently wider prediction intervals of our method. This empirical finding is consistent with our theoretical analysis in Section 4 (Theorems 1 and 2), which establishes that the coverage probability incorporates an additional slack or an error term $\Lambda(\widehat{w},\widehat\eta^{\pi})$. The error component originates from two distinct sources:
(i) the estimation error of the weight function $\widehat{w}$, and
(ii) the approximation error of the return distribution $\widehat\eta^{\pi}$.

This creates a trade-off: a larger $k$ reduces the approximation error in estimating $\widehat\eta^{\pi}$ but increases the difficulty of accurately estimating the off-policy weights, particularly under significant distributional shift. Based on our empirical findings, we recommend using $k = 2$ or $3$ to balance these competing factors. We will add the above discussion on the role of $k$ in the Theorem section.

Rebuttal Table B. We conduct the experiments for Example 1 across $k=1,2,\ldots,8$. Each experiment is repeated 100 times, and we report the mean and standard deviation of coverage probability (cov) and interval length (len) when the nominal coverage level is 90%.

Q3-2: Adaptive or cross-validated strategies for choosing $k$

We appreciate the reviewer’s suggestion on exploring adaptive or cross-validated strategies for selecting $k$. While these approaches are conceptually straightforward in on-policy settings where target policy validation data are readily available, they present significant challenges in off-policy settings due to the absence of independent validation data sampled from the target policy. This limitation makes performance-based tuning of $k$ difficult under distribution shift.

Q3-3: Aggregating PIs across $k$

We also thank the reviewer for raising the interesting question of aggregating prediction intervals across different $k$ values. Since these intervals are correlated, aggregation is nontrivial. A promising direction is to construct a unified prediction region by combining the corresponding p-values, leveraging the connection between prediction intervals and hypothesis testing. Methods such as the Cauchy Combination Test (e.g., [4,5,6]), which are robust to arbitrary dependencies, offer a viable approach. We will include this discussion in the Conclusion section.

Q4: The settings considered in the experiments are mostly low-dimensional.

Our experiments primarily focus on low-dimensional settings to clearly demonstrate the method’s performance. Due to time constraints during the rebuttal period, we will incorporate high-dimensional experiments in the revised version.

References

[1] Risk-sensitive and robust decision-making: a cvar optimization approach.

[2] Risk-constrained reinforcement learning with percentile risk criteria.

[3] Conformal off-policy evaluation in markov decision processes.

[4] ACAT: a fast and powerful p value combination method for rare-variant analysis in sequencing studies.

[5] Cauchy combination test: a powerful test with analytic p-value calculation under arbitrary dependency structures.

[6] Multi‐split conformal prediction via Cauchy aggregation.

[7] Conformal off-policy prediction.