Treatment Rule Optimization Under Counterfactual Temporal Point Processes with Latent States

We sincerely appreciate the time and effort you’ve taken to provide such thoughtful and detailed feedback on our work.

Weakness 1:

• in Eq. (7) you use the do operator but never define an appropriate SCM or potential outcomes to work with.

Thank you for your thoughtful feedback. In the updated version, we provided details of the SCM for augmenting the thinning process of MTPP with latent states, and explained the corresponding meaning for our intervention.

• treatment times and events are first denoted with $t, m_a$ and later with $x, \tau$

We use the notation $t, m_a$ to represent the treatment events, representing its time and event type. Later we define another notation $x, \tau$ in the example meta rule, in which $x$ is a potential continuous value of dosage defined in a specific rule, and $\tau$ represent the time-lag for administering this drug once the condition is satisfied. We use this notation since it actually represents the different meaning compared with observed treatment events.

Weakness 3: the latent variables $z$ are not confounders (see Fig. 1).

We appreciate a lot for you concern. We suggest the latent variable $z$ could be regarded as a special kind of latent confounder since we assume the treatment and outcome trajectories are jointly generated by the Ogata’s thinning process for MTPP. Therefore, the latent state $\boldsymbol{z}(t)$ at time $t$ would affect treatment and outcome intensities simultaneously in the acceptance and rejection process for a event, regardless of only one type of the event (roughly belong to treatment or outcome, as shown in fig 1) would be generated.

Weakness 5: In Eq. (7), the authors only condition on the latent states z but not on observed variables.

Thanks a lot for noting this, this was a typo and we've corrected it in the updated version.

Weakneess 7: (observed) time-varying confounders.

We appreciate your valuable suggestion. We suppose when observed time-varying confounders exists, they might be incorporated into the multivariate Hawkes model by assuming them as components of the network. This allows the estimation of their causal relationships, which can then be leveraged during the counterfactual sampling process. However, if time-varying confounders are unobserved, handling them presents a separate challenge, which we plan to address in future work.

Weakness 8&9: The method relies on various assumptions that are not spelled out explicitly.

We are thankful for your comments, in the updated version we provided the detailed assumptions we need in the appendix. Combined with the proof of counterfactual identifiability of the constructed SCM, we are able to guarantee that the counterfactual effects could be identified. In theorem 1, we provide the conditions and proof that the model parameters are identifiable, the unique parameter also plays a crucial role since it guarantees the counterfactual intensity we are interested in is also unique under each revision for the treatment plan.

Q: Why does the proposed method work with "meta-rules" instead of learning an optimal policy in a fully data-driven manner? It seems like the aim is to "hard-code" certain aspects of the policy.

Relying solely on a data-driven policy can be challenging when data is limited, especially in high-stakes areas like healthcare where patient variability and sparse critical events make learning difficult. Meta-rules offer a structured way to embed expert knowledge directly into the policy, which is especially valuable in clinical settings where treatment decisions must comply with established guidelines and safety protocols, and where substantial prior knowledge is available. This idea aligns with the principles of constrained RL for safe exploration,

• Le, H. M., et al. (2019). Constrained Policy Optimization.

We sincerely thank you for your insightful comments and valuable suggestions, which would greatly help us improve our work. In the following responses, equation numbers and line numbers refer to the updated version of the document, while line numbers mentioned in the questions correspond to the original version.

Q1&2: Why did you only model the intensity process for the timing of treatment and outcome but not the mark?

We do take markers into account in our model, we model treatment and outcome jointly as a multivariate Hawkes model. A multivariate TPP is a special case of marked TPP, in which markers are discrete and a marker would correspond to a dimension. To illustrate, in Eq. (10) and Eq. (11) the intensity would depends on the marker, $\lambda_{m_j}(t_j|\cdot)$, in which $m_j$ represent the dimension.

Q3&limitation: Can you clarify what the baseline means in Figure 3?

Here we refer to 'Baseline' as the average reward in the baseline date. From line 438 to line 443, we illustrate the design of our baseline policy for simulating our baseline population data. Our main idea is to design some naive rules so that it would perform relatively not good so that it could be optimized.

Q4: In Step 1: Initialization in Section 6, how to initialize treatment decision rule parameters?

In our experiments, $m_{d, k}$ was initialized to a constant value of 5 and is subsequently transformed using the softmax function. $\tau_{d, k}$ was initialized within a range of random values between 0.5 and 1.

Q5: In Section 3.2, you used $x$ to represent dosage, did you switch to $m_{d, k}$ in Section 6?

Yes. This is mainly due to that in the example rule, we use $x$ to represent a potential continuous value of dosage. However, in our treatment and outcome model, we discretize this continuous value into several markers, e.g., two markers corresponding to low and high dosage. Therefore, in Section 6, we use $m_{d,k}$ instead of the original continuous parameter $x$.

Q6: Page 3 line 151, do you mean z(t) = [z_k]_{k <= t}?

Here $\boldsymbol{z}(t)$ represents the current state at time $t$, which is a $K \times 1$ vector.

Weakness1: The parametric and highly linear modeling of intensity process

We are grateful for your detailed and thoughtful suggestion. The identifiability of the counterfactual effect we aim to evaluate is crucial for our optimization process, thus we employ the parametric model for ensuring this. In the revised version, we also added some references about applications of linear Hawkes in healthcare setting. Thank you for your suggestion about non-parametric setting, which we suppose would be really helpful and feasible for improving our work.

Weakness2: How to choose the number of latent states

Appreciated a lot for you constructive feedback, indeed the assumption for the number of latent states $K$ sometimes might not hold in the real application. We suppose some methods could be employed to address this, for example one could compare the model performance with different state number via cross validation. Also, as in our real experiment, we take the latent state interpretability and domain expertise into consideration. We regard developing models that adapt the number of latent states dynamically as a future work topic.

Weakness3: Proof for Theorem 1

Many thanks for you suggestion, in the updated version we present the assumptions for Hawkes process clearly.

We sincerely thank you again for your valuable suggestions.

Explanation for Assumption 1

Thank you for you advice, we have updated Section 5 with detailed explanation provided.

Potential advantage of parametric assumption and choice for the number of latent states

While ICU data is often rich and extensive, healthcare scenarios frequently involve limited datasets. This limitation can arise when studying rare diseases, where the patient population is inherently small, or in situations where data access is restricted due to privacy regulations. For instance, detailed data on specific subpopulations or localized cases might not be readily available or shareable. In such contexts, parametric models, which rely on predefined structures and assumptions, are advantageous. These models are better suited for scenarios with limited data because they can leverage domain knowledge and structural constraints to produce reliable and interpretable results even when data is relatively sparse. Therefore, we still believe our model would be applicable in many practical setting. Also, the ability to provide interpretable parameters for professionals is crucial and desirable under many cases, and allowing professionals to incorporate domain knowledge easily, such as assumptions about biological distributions or event processes, aligns closely with established medical expertise, improving model reliability in sparse data environments.

Also, given the abundance of prior knowledge in healthcare, we believe that in many practical cases determining the number of latent states can be straightforward — for instance, by leveraging well-defined stages or progressions of a disease, or by summarizing static indicators to represent patient conditions effectively. Therefore, we believe our methods will remain highly beneficial and practical across many potentially concerned healthcare scenarios by doctors.

We appreciate a lot for your constructive feedback.

Q1: What if the number of latent factors K is unknown, or the latent states are not categorical - will the identifiability result still establish under such relaxed assumptions?

Our current proof technique requires knowledge of $K$, but this proof might be adaptable to allow for latent variables that are not categorical, which would, however, require further careful consideration.

Q2: How can the framework handle scenarios where meta-rules are not well-defined or unavailable?

Since our framework operates within a counterfactual scenario, our focus is on revising or adding treatment events to the observed treatment sequence based on the predefined meta-rules. Typically, the observed treatment sequence reflects some rational plans established by doctors. Therefore, if our meta-rules are poorly defined, the counterfactual outcome may worsen, signaling that our meta-rules might need adjustment.

While an initial guess about certain meta-rules may be necessary, feedback from the framework as we mentioned above can indicate whether these self-defined rules are appropriate, allowing for subsequent adjustments. Additionally, one might consult professionals or use AI assistance to receive guidance on suggested rules in their setting.

Weakness1: notation of the do-operator.

In the updated version, we provided detailed construction of the SCM and a brief further explanation about do-operator.

Weakenss2

Our approach introduces a novel adaptation of a counterfactual framework to multivariate TPPs, with a unique focus on optimizing meta-rules under this counterfactual setting. To the best of our knowledge, we are the only one address this combination. We thus suppose performing a fair comparison requires methods that also operate within a counterfactual framework tailored for TPPs. However, most off-policy evaluation or optimization techniques do not incorporate counterfactual reasoning. This highlights both the distinctiveness of our work and the challenges of benchmarking against existing methods.

Weakness3

Due to the page length limitation, we add this plot in the appendix.

Thank you so much for your questions and helpful suggestions. In the following responses, equation numbers and line numbers refer to the updated version of the document, while line numbers mentioned in the questions correspond to the original version.

Q1: why are $H_0$ and $H_1$ two distinct nodes

$H_0$ is part of $H_1$, thus there is a rightarrow from $H_0$ to $H_1$.

Q2: What mixture distribution is meant in line 152?

Thank you for your detailed feedback, this was a typo and we have corrected that.

Q3: What is the intuition of the counterfactual interventions in the past from the practical point of view? Should regular future interventions be sufficient for developing optimal treatment regimes?

While future interventions are essential for developing optimal treatment regimes, counterfactual interventions in the past add a crucial layer of insight. To illustrate, in high-stakes settings like healthcare, we cannot go back in time to test whether an alternative treatment plan would have benefited a patient who did not survive. Counterfactual analysis addresses this by focusing on posterior noise given observed information, rather than the prior noise used in interventional analysis in a SCM. One could imagine that this approach allows us to simulate conditions that closely resemble the environment at the time the data was observed, enabling more rigorous assessments of alternative treatment plans, e.g., as mentioned in the following two references,

• Oberst, M., & Sontag, D. (2019). Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models;
• Buesing, L., Weber, T., Zwols, Y., Racaniere, S., Guez, A., Lespiau, J. B., & Heess, N. (2018). Woulda, coulda, shoulda: Counterfactually-guided policy search.

Weakness 1 & 2

Many thanks for your valuable feedback. We model treatment and outcome events jointly via a multivariate temporal point process. We aim to evaluate and optimize a counterfactual quantity by revising the treatment trajectory based on meta-rules. These revisions would influence the intensity of outcome dimensions, enabling us to define a counterfactual intensity of interest. To analyze this, we construct a SCM to augment the thinning process for MTPP. Using this framework, we could sample counterfactual outcomes, and the identifiability of these counterfactuals is ensured by the properties of the constructed SCM. We've uploaded a revised version, provided the causal assumptions we need and the counterfactual identifiability proof, so that our target counterfactual quantity is identified.

Weakness 4

We suppose we provided the detailed definition for each notation, for example,

• $\phi (t-s)$ in Eq. (3) -- explained in line 173.

• triggering functions -- defined in line 180 and Eq. (4).

• $x$ in line 208 (original ver) - defined in example meta-rule , line 250.