Beyond Average Value Function in Precision Medicine: Maximum Probability-Driven Reinforcement Learning for Survival Analysis

Response to Reviewer Comments

We sincerely appreciate your valuable feedback. Below are our responses and revisions addressing the questions you raised.

---

1. Figure 1 Clarification

Reviewer Comment:
"Figure 1 is confusing and is not self-explanatory; the given explanation is not sufficient."
Response:
Thank you for your suggestion. We will incorporate the descriptions from lines 115–138 into the figure to improve clarity.
Revision:

• $C \in \mathbb{R}^+$: Censoring time.
• $Y_k$: Observed duration at stage $k$.
• $\Delta_k$: Censoring indicator at stage $k$.
• $\bar{K} \in \mathbb{N}^+$: Number of treatment stages a subject received.
• $T_k$: Actual duration of stage $k$.

---

2. Threshold Selection

Reviewer Comment:
"How should the threshold be chosen? If it's not clear, then isn't this more or less the same as optimizing duration?"
Response:
Thank you for the question. The selection of $\alpha$ is tailored to the specific research question being addressed. For example, in Jiang, et al. (2017)'s study on HIV data, they chose $\alpha = 0.5$, focusing on the optimal strategy for maximizing the probability of survival exceeding the median. In our study, we selected $\alpha = 1/14$ to maximize the probability of an intern's positive mood exceeding one day per bi-weekly period. We generally advise using an $\alpha$ of 0.5 or less, mainly because: (1) $\alpha$ values near 1 rely heavily on tail probability estimation, which can be unreliable; (2) to address your concern, we have conducted an additional simulation experiment to demonstrate the sensitivity of our algorithm to different $\alpha$ values. The results reveal that our method performs better when $\alpha$ is 0.5 or lower.

Revision:
We will add a new appendix section before A.2 titled:
"A.2 $\alpha$ Selection

Table 1: Mean (Standard Deviation) of AARED under different $\alpha_k$ (rounded to two decimal places)

Table 2: AAEV values (Mean ± Standard Deviation) under different $\alpha_k$ (rounded to two decimal places)

To clarify our recommendation for alpha selection, we will include the following remark on page 4, line 153: ”The selection of $\alpha$ is tailored to the specific research question being addressed. For example, in Jiang et al. (2017)‘s study on HIV data, they chose $\alpha = 0.5$, focusing on the optimal strategy for maximizing the probability of survival exceeding the median. In our study, we selected $\alpha = 1/14$ to maximize the probability of an intern’s positive mood exceeding one day per bi-weekly period. We generally advise using an $\alpha$ of 0.5 or less, mainly because $\alpha$ values near 1 rely heavily on tail probability estimation, which can be unreliable. These results are consistent with the numerical results we obtained from our experiment results (see Appendix A.2 for details).“

---

3. Maximizing Duration vs. Survival Probability

Reviewer Comment:
"Why not maximize the durations of being in the good state directly? Since this problem maximizes the sum of log Survival functions. Why not just maximize the expected durations per stage?"

Response:

1. Challenges with Censored Data: Estimating durations in the “good state” becomes significantly more challenging when dealing with censored data, as complete information about the duration is not always available.
2. Limited Reward Identifiability: In discrete settings, the binary nature of the rewards (1 or 0) lacks the discriminative power needed to effectively differentiate between various actions and their impact on stage duration.
3. Focus on Cumulative Benefit: Our primary objective is to maximize the cumulative benefit over the entire process, rather than focusing solely on stage-specific rewards. This requires a framework that considers the long-term impact of actions.
4. Robustness of the Method: We chose to use the survival probabilities as our objective precisely because of its robustness. Existing methods (e.g., Liu et al. (2023)) maximize expected durations using Kaplan-Meier estimates, but expectations are sensitive to outliers. Extreme survival times can skew averages, leading to a misrepresentation of actual patient benefits.

Revision:
We will add after line 51:
"The challenges in relevant research include the difficulty of estimating durations in the "good state" due to censored data, the limited discriminative power of binary rewards in discrete settings for differentiating action impacts, and the need to focus on cumulative benefits. Additionally, maximizing expectations is sensitive to outliers—extreme survival times can inflate averages, misrepresenting actual patient benefits (Bai et al., 2013; Jiang et al., 2017a, 2017b). Thus, we explore maximizing survival probabilities, which better reflect clinical outcomes."

---

4. Relevance of Threshold-Based Probability in Survival Analysis

Reviewer Comment:
"In survival analysis, we focus on the probability that the recurrent-event duration exceeds a threshold. Can you explain why this is decision-theoretically relevant? It’s unclear why this quantity matters to decision-makers."

Response:
Thank you for your question. The importance of this quantity can be explained from both theoretical and practical perspectives. Theoretically, compared to traditional methods that focus on the mean survival time, this quantity is more robust and less sensitive to outliers, providing a more stable representation of the patients outcome. Furthermore, by studying the survival curve, our method can characterize the impact of treatment at different time points. From a practical application perspective, the study of this quantity is relevant to many medical problems and has been investigated by some existing literature. We provide some specific examples below.

1. Bai et al. (2013): In the ASCERT study, CABG and PCI showed crossing survival curves—PCI had lower short-term risk, but CABG outperformed after 4 years. Here, 4-year survival probability precisely captured long-term differences, whereas average survival time obscured this.
2. Jiang et al. (2017a): In ACTG 175, young patients on zidovudine + zalcitabine had significantly higher 400-day survival probabilities. Maximizing $t$-year survival probability aligned better with short-term clinical needs than averaging.
3. Jiang et al. (2017b): For HIV patients, median survival time (50% probability) was more representative of typical outcomes than restricted mean survival time, as it is less influenced by extreme values.

Revision:
We will add after line 35:
"In survival analysis, expectations (e.g., mean survival time) are sensitive to outliers and may obscure time-specific treatment effects. Threshold-based probabilities (e.g., median survival) better align with clinical needs. For example:

• Bai (2013) found CABG’s 4-year survival probability outperformed PCI’s, despite crossing survival curves.
• Jiang (2017a) showed 400-day survival probability was critical for evaluating short-term HIV treatment efficacy.
• Jiang (2017b) demonstrated median survival time’s robustness for typical patient outcomes in HIV studies.
  These studies validate that probability-based metrics overcome the limitations of expectations."

---

References

1. Bai, X., Tsiatis, A. A., and O’Brien, S. M. (2013). "Doubly-Robust Estimators of Treatment-Specific Survival Distributions in Observational Studies With Stratified Sampling." Biometrics, 69, 830–839.
2. Jiang, R., Lu, W., Song, R., and Davidian, M. (2017a). "On Estimation of Optimal Treatment Regimes for Maximizing t-Year Survival Probability." Journal of the Royal Statistical Society, Series B, 79, 1165–1185.
3. Jiang, R., Lu, W., Song, R., Hudgens, M. G., and Naprvavnik, S. (2017b). "Doubly Robust Estimation of Optimal Treatment Regimes for Survival Data—With Application to an HIV/AIDS Study." The Annals of Applied Statistics, 11, 1763–1786.

---

Acknowledgments

We have addressed all of the reviewers' comments in this revision, offering clear explanations, empirical support, and specific revisions to the manuscript. Please advise if further clarifications are needed.

Dear Reviewer fiaA,

We submitted our response to your initial comments some time ago, and we wanted to check in briefly.

We deeply appreciate the time and effort you’ve dedicated to evaluating our work, especially given your busy schedule. Your feedback is invaluable as we work to refine our manuscript, and we’re eager to receive your further thoughts to guide our revisions and move the submission forward.

If you’ve had an opportunity to review our response, we would greatly appreciate your input whenever you have a moment; if not, we hope you might be able to take a look when your schedule allows.

Thank you again for your time and attention. We look forward to hearing from you.

Sincerely,

Authors

Response to Reviewer Comments

We sincerely appreciate your valuable feedback. Below are our responses and revisions addressing the questions you raised.

---

1. Continuous Action Space

Reviewer Comment:It seems that some of the theoretical analysis assumes a discrete action space, such as Theorem 3.2 and the assumption in lines 213-214, which is hard to be met in a continuous space. Do these assumptions still apply in a continuous action space, as used in DDPG?

Response: Thank you for pointing this out. Although the action space in the paper was formulated as discrete, the objective itself is applicable to both discrete and continuous action spaces. Following your advice, we have revised all relevant content in the paper (shown in the revision) to discuss discrete and continuous action spaces separately.

Revision:

• We will change "Theorem 3.2" to "Theorem 3.2 (Discrete action space)", and then add a new theorem after line 170:" Theorem 3.3 (Continuous action space)
  Let the state transition kernel $P_0(x'|x,a)$ be deterministic (i.e., for any $x,a$, there exists a unique $x'$ such that $P_0(x'|x,a)=1$). Let the policy $\pi$ be represented by a deterministic function $\mu$ (i.e., $a_k = \mu(x_k)$), and for any initial state $x_t$, the supremum of the value function $V^{\mu}(x_t)$ is attainable. Define the policy sets: $\Pi^v =${$\mu^v \mid V_0^{\mu^v}(x_t) = \text{max}({\mu}) V_0^{\mu}(x_t)$} , $\Pi^p =${$\mu^p \mid P^{\mu^p}(x_t) = \text{max}({\mu}) P^{\mu}(x_t)$} , then $\Pi^v = \Pi^p$."
  Proof: Given a deterministic transition kernel $P_0$ and a deterministic policy $\mu$, starting from any state $x_t$, there exists a unique trajectory $\gamma_t^\mu = (x_t, \mu(x_t), T_t, x_{t+1}^\mu, \mu(x_{t+1}^\mu), T_{t+1}, \dots, x_{\bar{K}}^\mu, \mu(x_{\bar{K}}^\mu), T_{\bar{K}})$, where $x_{k+1}^\mu$ is uniquely determined by the transition equation. Based on the above unique trajectory, the reward of trajectory $\gamma_t^\mu$ can be expressed as $R(\gamma_t^\mu) = \prod_{k=t}^{\bar{K}} S_{T_k}(\alpha_k G_k \mid x_k^\mu, \mu(x_k^\mu))$. Correspondingly, the objective functions can be simplified to $P^\mu(x_t) = R(\gamma_t^\mu)$ and $V_0^\mu(x_t) = \log R(\gamma_t^\mu)$. Define the set of optimal trajectories $\Gamma_t^\* =$ { $\gamma_t^\* \mid R(\gamma_t^\*) = \text{max}({\gamma_t \in \Gamma_t}) R(\gamma_t)$}, where $\Gamma_t$ is the set of all possible trajectories. Since $\log(\cdot)$ is a strictly monotonically increasing function, $\text{max}(\mu) R(\gamma_t^\mu)$ is equivalent to $\text{max}(\mu) \log R(\gamma_t^\mu)$. It follows that $\mu \in \Pi^p \iff R(\gamma_t^\mu) = \text{max}({\gamma_t \in \Gamma_t}) R(\gamma_t) \iff \log R(\gamma_t^\mu) = \text{max}({\gamma_t \in \Gamma_t}) \log R(\gamma_t) \iff \mu \in \Pi^v$, i.e., $\Pi^v = \Pi^p =${$\mu \mid \gamma_t^\mu \in \Gamma_t^\*$} $, which proves the equivalence of optimal policies.

• Line 211 will be changed to "4.2 Optimization in discrete action space", followed by adding "4.3 Optimization in continuous action space According to DDPG (Lillicrap et al., 2015), we define the Policy Function: $\mu(\theta): \mathcal{S} \to \mathcal{A}$, a parameterized function (e.g., neural network) that maps states to actions. For a fixed policy $\mu(\theta)$, the Q-function follows the Bellman evaluation equation:
  $Q^{\mu(\theta)}(\tilde{X}_t, \tilde{A}_t) = \mathbb{E}^{P_0} [(\tilde{X}_t, \tilde{A}_t) + \gamma Q^{\mu(\theta)} (\tilde{X} _{t+1}, \mu(\theta)(\tilde{X} _{t+1})) ]$ To optimize $\mu(\theta)$, we maximize the expected return $J(\theta) = \mathbb{E}[Q^{\mu(\theta)}(\tilde{X}_t, \tilde{A}_t)]$. Then, parameter updates are performed using the chain rule in DDPG. “

---

2. Core contribution

Reviewer Comment: Thank you for the question. Despite the rigorous theoretical analysis, the code implementation is relatively simple. Does the core contribution lie in treating the logarithm of the survival probability given a specific threshold as the immediate reward?

Response: Our core contribution lies in constructing a novel framework that integrates offline RL with recurrent event data. Furthermore, we propose a new recurrent data-driven Q-function and prove that the proposed practical algorithm is consistent with classical Q-learning, rather than using the logarithm of the survival probability given a specific threshold as the immediate reward.

Revision: For improved clarity regarding our core contribution, we have included a sentence on line 67:"Our core contribution is developing a novel framework that integrates offline RL with recurrent data, proposing a new recurrent data-driven Q-function, and proving our algorithm aligns with classical Q-learning."

---

3. Figure 4 Performence

Reviewer Comment: Could you explain the changes in the relative log probability shown in Figure 4? Specifically, why does it increase from 2/14 to 8/14 and then decrease? Does the absolute log probability follow the same trend?

Response:Thank you for your question. In Figure 4, we previously presented the relative difference in log probabilities (calculated as $\frac{\log \hat{\text{P}}^{\pi^A} - \log \hat{\text{P}}^{\pi^B}}{\log \hat{\text{P}}^{\pi^B}}$ for the Aah model and $\frac{\log \hat{\text{P}}^{\pi^C} - \log \hat{\text{P}}^{\pi^B}}{\log \hat{\text{P}}^{\pi^B}}$ for the Cox model). To better address your question, we have now presented the absolute difference in log probabilities in the revision below. What is shown here is the relative magnitude, which does not have a clear pattern. The absolute probability values do not change in this way. As can be seen below, in fact, as $\alpha$ increases, the absolute difference ($\log \hat{\text{P}}^{\pi^A} - \log \hat{\text{P}}^{\pi^B}$ for the Aah model and $\log \hat{\text{P}}^{\pi^C} - \log \hat{\text{P}}^{\pi^B}$ for the Cox model) becomes larger and larger, indicating that our performance is getting better.

Revision: We will revise lines 327-329 in the paper to "We then presented the log probability differences $\log \hat{\text{P}}^{\pi^A} - \log \hat{\text{P}}^{\pi^B}$ for the Aah model and $\log \hat{\text{P}}^{\pi^C} - \log \hat{\text{P}}^{\pi^B}$ across varying $\alpha_k$ settings." and then replace the figure with the data shown in the table.

---

References

Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.

---

Acknowledgments

All comments from the reviewers have been thoroughly addressed in this revision, with explicit explanations, empirical support, and precise modifications made to the manuscript. Do not hesitate to inform us if additional clarifications are needed.

Dear Reviewer 2BRd,

We submitted our response to your initial comments some time ago, and we wanted to gently follow up.

We greatly appreciate the time and effort you’ve dedicated to reviewing our work, especially given your busy schedule. Your insights are invaluable for refining our manuscript, and we’re eager to receive your further feedback to advance revisions and move the submission forward.

If you’ve had a chance to review our response, we would be grateful for your thoughts at your convenience; if not, we hope you might have a moment to take a look when time permits.

Thank you again for your consideration. We look forward to your reply.

Sincerely,

Authors

Response to Reviewer Comments

We sincerely appreciate your valuable feedback. Below are our responses and revisions addressing the questions you raised.

---

1. Theoretical Justification of Theorem 2

Reviewer Comment: The review is not convinced by the key theoretical theorem 2. How does adding an extra log in the objective still give the same optimal solution? The review is not convinced by the proof in Appendix 9, it seems to state that a greedy action simply solve the MDP. Plus, the reviewer believe that the result in Figure 2 also proves that adding an extra log will give different optimal solutions. The authors are encouraged to make this part more clear as the equivalence claims are build upon it.

Response:
We greatly appreciate your comments. Following your suggestion, we carefully examined the theoretical section and identified a missing condition in Theorem 3.2: the deterministic property of $P_0$. We emphasize that, because our work focuses on offline RL, this condition is naturally met, and therefore the correction to Theorem 3.2 does not affect the overall properties of our proposed framework. In the revised manuscript, we will update Theorem 3.2 and provide its complete proof as below.

Revision:
We will revise the theorem in line 169 of the paper to:

"Theorem 3.2. If for any $\boldsymbol{x}_t$, the supremum of $V^{\pi}(\boldsymbol{x}_t)$ can be attained, and when $P_0(\boldsymbol{x}'|\boldsymbol{x},a)$ is a deterministic transition kernel, with $\Pi^v =${$\pi^v | V_0^{\pi^v}(\boldsymbol{x}_t) = \text{max}(\pi) V_0^{\pi}(\boldsymbol{x}_t)$} and $\Pi^p =$ {$\pi^p | \text{P}^{\pi^p}(\boldsymbol{x}_t) = \text{max}(\pi)\text{P}^{\pi}(\boldsymbol{x}_t)$}, then $\Pi^v = \Pi^p$.

（$P_0(\boldsymbol{x}'|\boldsymbol{x},a)$ is a deterministic transition kernel if and only if $\forall \boldsymbol{x},a$, there exists a unique $\boldsymbol{x}'$ such that $P_0(\boldsymbol{x}'|\boldsymbol{x},a)=1$ ）"

Furthermore, we will revise the proof of Theorem 3.2 in lines 548 to 556 in Appendix to:

"Proof: For any deterministic policy $P_0$, by definition, there exists a unique $x_{t+1}^{P_0}$ such that:

$$P_0(x_{t+1}^{P_0} \mid x_t, a_t) = 1$$

We denote $x_t^{P_0} = x_t$ and define

$$R(\gamma_t) = \prod_{k=t}^K S_{T_k}(\alpha_k G_k \mid x_k^{P_0}, a_k)$$

where $\gamma_t$ is the path:

$$\gamma_t = (x_t^{P_0}, a_t,T_t, x_{t+1}^{P_0}, a_{t+1},T_{t+1} \dots, x_{\bar{K}}^{P_0}, a_{\bar{K}},T_{\bar{K}})$$

We define the set of paths as $\Gamma_t^{P_0}$, then we can obtain:

$$P^\pi(x_t) = \sum_{\gamma_t \in \Gamma_t^{P_0}} \mathcal{P}(\gamma_t) \cdot R(\gamma_t), \quad V_0^\pi(x_t) = \sum_{\gamma_t \in \Gamma_t^{P_0}} \mathcal{P}(\gamma_t) \cdot \log R(\gamma_t)$$

where:

$$\mathcal{P}(\gamma_t) = \pi(a_t \mid x_t^{P_0}) \prod_{k=t}^{\bar{K}-1} P_0(x_{k+1}^{P_0} \mid x_k^{P_0}, a_k) \pi(a_{k+1} \mid x_{k+1}^{P_0})\\ = \prod_{k=t}^{\bar{K}} \pi(a_k \mid x_k^{P_0})$$

Then we can know that

$$P^\pi(x_t) = \sum_{\gamma_t \in \Gamma_t^{P_0}} \prod_{k=t}^K \pi(a_k \mid x_k^{P_0}) \cdot R(\gamma_t) \leq \text{max} ({\gamma_t \in \Gamma_t^{P_0}}) R(\gamma_t)$$ $$V_0^\pi(x_t) = \sum_{\gamma_t \in \Gamma_t^{P_0}} \prod_{k=t}^K \pi(a_k \mid x_k^{P_0}) \cdot \log R(\gamma_t) \leq \text{max}({\gamma_t \in \Gamma_t^{P_0}}) \log R(\gamma_t)$$

where $\sum_{\gamma_t \in \Gamma_t^{P_0}} \prod_{k=t}^K \pi(a_k \mid x_k^{P_0}) = 1$. If we define: $\Gamma_t^{p,\*} =$ {$\gamma_t^\* \mid R(\gamma_t^\*) = \text{max}({\gamma_t \in \Gamma_t^{P_0}}) R(\gamma_t)$} and $\Gamma_t^{v,\*} =$ {$\gamma_t^o \mid \log R(\gamma_t^\*) = \text{max}({\gamma_t \in \Gamma_t^{P_0}}) \log R(\gamma_t)$} Here, since $a \geq b \iff \log a \geq \log b \ (\text{where } a,b > 0)$, we have $\Gamma_t^{v,\*} = \Gamma_t^{p,\*}$, which we denote as $\Gamma_t^{\*}$. Then:

$$P^\pi(x_t) \leq R(\gamma_t^\*), \quad V_0^\pi(x_t) \leq \log R(\gamma_t^\*), \quad \gamma_t^\* \in \Gamma_t^\*.$$

Define the policy set: $\Pi^\* =${$\pi \mid \sum_{\gamma_t \in \Gamma_t^\*} \prod_{k=t}^{\bar{K}} \pi(a_k \mid x_k^{P_0}) = 1$} Then we can know that the equality in the inequalities holds if and only if $\pi \in \Pi^\*$ (see the following proof), so it can be concluded that $\Pi^v = \Pi^p = \Pi^\*$

1. $P^\pi(x_t) = R(\gamma_t^\*) \Rightarrow \pi \in \Pi^\*$: For $P^\pi(x_t)$, we denote $R^\* = \text{max}({\gamma_t \in \Gamma_t^{P_0}}) R(\gamma_t)$. Then, for $\gamma_t \notin \Gamma_t^\*$, we have $R(\gamma_t) < R^*$. Thus:

$$P^\pi(x_t) = \sum_{\gamma_t \notin \Gamma_t^\*} \mathcal{P}(\gamma_t) \cdot R(\gamma_t) + \sum_{\gamma_t \in \Gamma_t^\*} \mathcal{P}(\gamma_t) \cdot R^\* = R^\*$$

So, $\sum_{\gamma_t \in \Gamma_t^*} \mathcal{P}(\gamma_t) = 1$, that is, $\pi \in \Pi^\*$. 2. $P^\pi(x_t) = R(\gamma_t^\*) \Leftarrow \pi \in \Pi^\*$: Prove by contradiction: Assume there exists $\pi \notin \Pi^\*$, then $\sum_{\gamma_t \in \Gamma_t^\*} \prod_{k=t}^{\bar{K}} \pi(a_k \mid x_k^{P_0}) < 1$. We denote $R^\* = \text{max}({\gamma_t \in \Gamma_t^{P_0}}) R(\gamma_t)$. At this time:

$$P^\pi(x_t) = \sum_{\gamma_t \in \Gamma_t^\*} \mathcal{P}(\gamma_t) \cdot R^\* + \sum_{\gamma_t \notin \Gamma_t^\*} \mathcal{P}(\gamma_t) \cdot R(\gamma_t).$$

Since in the second term, $R(\gamma_t) < R^\*$ and $\mathcal{P}(\gamma_t) > 0$, we have:

$$P^\pi(x_t) < \sum_{\gamma_t \in \Gamma_t^\*} \mathcal{P}(\gamma_t) \cdot R^\* + \sum_{\gamma_t \notin \Gamma_t^\*} \mathcal{P}(\gamma_t) \cdot R^\* = R^\* \cdot \sum_{\gamma_t} \mathcal{P}(\gamma_t) = R^\*,$$

which contradicts "$P^\pi(x_t) = R(\gamma_t^\*)$."

---

2. Potential Evaluation Bias Using Cox Model

Reviewer Comment: As the reviewer understand, the offline evaluation part is based on the simulation using the Cox. Will such evaluation cause biased results towards the proposed method based on Cox model?

Response:
Thank you for your suggestion. Following your advice, we explored using a log-normal AFT model for data generation. This alternative approach mitigates the risk of estimation bias associated with the Cox model, as it offers a distinct modeling framework. The results presented in the revision demonstrate that the findings remain consistent with those obtained using the Cox model, even with this new data generation process.

Revision:
We will change in line 332 with "Below, we use the Log Normal AFT Model to estimate $\pi$ in (1). It can be seen that the fitting effect of this model is better than that of Cox and Aah. Under this method, the calculated results of $\frac{\log \hat{\text{P}}^{\pi^{A \text{ or } C}} - \log \hat{\text{P}}^{\pi^B}}{\log \hat{\text{P}}^{\pi^B}}$ are as follows:"

then redraw the figure using the data in the table , and add the following content in the appendix:

" Log Normal AFT Model assumes that the natural logarithm of the survival time $T_k$ follows a normal distribution: $\log(T_k) \sim \mathcal{N}(\mu(x_k), \sigma^2)$, where: The location parameter $\mu(x)$ is a linear combination of the covariates $x = [x_1, x_2, \dots, x_n]$: $\mu(x) = a_0 + a_1 x_1 + a_2 x_2 + \dots + a_n x_n$ Here, $a_0, a_1, \dots, a_n$ are the regression coefficients to be estimated, reflecting the impact of the covariates $x$ on the mean of the logarithm of the survival time. The scale parameter $\sigma$ is a fixed constant (independent of covariates), describing the standard deviation of $\log(T)$ and controlling the degree of dispersion of the distribution. The model estimates parameters by maximizing the log-likelihood function. The concordance index under the Model is as follows:

---

Acknowledgments

This revision addresses all reviewer comments with clear explanations, empirical support, and targeted edits to the manuscript. Let us know if further clarifications are needed.

Dear Reviewer VWkH,​

We submitted our response to your initial questions some time ago.​ We appreciate your busy schedule and the effort you’ve put into reviewing. Your insights are vital for refining our work, and we eagerly await your further feedback to proceed with revisions and move the submission forward.​

If you’ve reviewed our response, please share your thoughts at your convenience; if not, we hope you might take a look when possible.​ Thank you again for your time. We look forward to hearing from you.​

Sincerely,

Authors

Response to Reviewer Comments

We sincerely appreciate your valuable feedback. Below are our responses and revisions addressing the questions you raised.

---

1. Code Availability

Reviewer Comment:
"Why not release code to improve reproducibility?"

Response:
To ensure reproducibility, our detailed source code and its explanation are available in the submission materials.

Revision:
We will add the following statement at the end of line 255 in Section 5 (Simulation Studies):
"Our code is available in the submission materials."

---

2. Selection of $\alpha$ in Practice

Reviewer Comment:
"How would the authors recommend selecting $\alpha$ in practice?"

Response:
The selection of $\alpha$ depends on the research question. For instance, Jiang, et al. (2017) used $\alpha = 0.5$ in their HIV study to maximize the probability of survival exceeding the median. In our research, $\alpha = 1/14$ was chosen to maximize the probability of an intern's positive mood exceeding one day per bi-weekly period. We generally recommend $\alpha$ of 0.5 or less, mainly because: (1) $\alpha$ values near 1 rely too much on tail probability estimation, which is unreliable; (2) Additional simulation experiments show our algorithm performs better when $\alpha$ is 0.5 or lower, addressing related concerns.

Revision:
We will add a new appendix section before A.2 titled:
"A.2 $\alpha$ Selection Mean (Standard Deviation) of AARED under different $\alpha_k$

Mean (Standard Deviation) of AAEV under different $\alpha_k$

To clarify our recommendation for alpha selection, we will include the following remark on page 4, line 153: ”The selection of $\alpha$ is tailored to the specific research question being addressed. For example, in Jiang et al. (2017)‘s study on HIV data, they chose $\alpha = 0.5$, focusing on the optimal strategy for maximizing the probability of survival exceeding the median. In our study, we selected $\alpha = 1/14$ to maximize the probability of an intern’s positive mood exceeding one day per bi-weekly period. We generally advise using an $\alpha$ of 0.5 or less, mainly because $\alpha$ values near 1 rely heavily on tail probability estimation, which can be unreliable. These results are consistent with the numerical results we obtained from our experiment results (see Appendix A.2 for details).“

Furthermore, to better demonstrate the sensitivity of our method to different $\alpha$ values, we conducted a new simulation experiment. The results will be presented in a new section, A.2, in the Appendix. The specific results are shown in the tables below, representing the results for Average Recurrent-event Duration of last 40 epoches (AARED) and Average Estimated Variance of last 40 epoches (AAEV), respectively. The results indicate that our method exhibits a more pronounced advantage when alpha is no greater than 0.5. This provides numerical simulation-based support for our recommendation regarding $\alpha$."

---

3. Variance Benefits Under Different Censoring Rates

Reviewer Comment:
"What happens to variance benefits on data sets with lower/higher censoring?"

Response:
Thank you for the valuable suggestion. For improved clarity, we will add a new table to the Appendix A.3 to show the censoring rates at different stages. As can be seen from this table, the censoring rates vary across stages in the simulation experiments (ranging from approximately 15% to approximately 70%). This table, together with Table 3 and 4 in the original manuscript, demonstrates that the magnitude of the censoring rate has a limited impact on our method, and that our method can achieve good variance benefits under different censoring rates.

Revision:
We will add the following table in line 310:
"Censoring rates differ across stages. Below are the mean (variance) of censoring rates over 50 simulations:

Tables 3 and 4 in Appendix A.3(origanol pdf) confirm that our method maintains strong performance under varying censoring rates."

---

4. Benchmark Comparison

Reviewer Comment:
"Why not include a more detailed benchmark (e.g. Hargrave et al. NeurIPS 2024) or baselines optimizing tail probabilities?"

Response:
Thank you for your suggestion. While Hargrave et al. (2024) present a reinforcement learning framework for longitudinal data, their work is limited to the complete observation setting, unlike our research which addresses censored data. Their approach becomes a special case of our framework when full observability is assumed. This is why we did not use it as a benchmark, as our previous simulations focused on methods capable of handling censoring. However, in response to your suggestion, we will conduct a new simulation study to directly compare our method to that of Hargrave et al. (2024).

Revision:
We will add the following to Related Work (line 81):
"Hargrave et al. (2024) proposed a RL framework for longitudinal medical settings but did not address censored data."

Additionally, we will add a new appendix section before A.4 titled:
"A.4 Comparison with EpiCare (including supplementary experimental results which includes the detailed simulation results that compare our method to that of Hargrave et al. (2024))."

---

5. Computational Cost

Reviewer Comment:
"What is the computational cost of training per epoch?"

Response:
We appreciate your question. Experiments were conducted on an NVIDIA GTX 1650. The results is shown in revision.

Revision:
In Appendix A.5, the following Table 5 includes the computation time of our proposed method and the baseline method. "A.5 Computational Time

Table 5: Training time per epoch (seconds)

---

Acknowledgments

We are particularly grateful for your insightful feedback, which has significantly contributed to the enhancement of our work. We have thoroughly addressed the issues you raised and have revised the manuscript accordingly.

Reference
Jiang, R., Lu, W., Song, R., Hudgens, M. G., & Naprvavnik, S. (2017). Doubly Robust Estimation of Optimal Treatment Regimes for Survival Data—With Application to an HIV/AIDS Study. The Annals of Applied Statistics, 11(4), 1763-1786.

Hargrave, M., Spaeth, A., & Grosenick, L. (2024). EpiCare: A Reinforcement Learning Benchmark for Dynamic Treatment Regimes. Advances in neural information processing systems, 37, 130536–130568.