Amortized Network Intervention to Steer the Excitatory Point Processes

[Summary] We first want to appreciate reviewer ya7t a lot for your acknowledgment of the innovation and contributions of our work including 1) “the problem formulation of networked point processes is original”, 2) “the scalability issue is addressed well by the proposed amortized policy”, 3) “the presentation is clear”, and 4) “the proposed method is associated with application scenarios of high significance”. We then provide point-wise responses below to address the concern.

[Q1]: Alternative methods.

As the suggestion to add more comparisons, we implemented two common RL baselines: SAC and PPO on the COVID data. The empirical results have been incorporated into Appendix I.2 (Figure 13). By using NJODE as our simulator, we observe that the proposed method has successfully reduced the number of infested people (with an average reduced intensity: 0.49) in six counties among the chosen nine counties while SAC and PPO struggle to have a positive intervention effect to control the pandemic.

One interesting alternative method would be to combine causal inference with RL. In fact, in our experiment, we have introduced a pre-learned Boolean mask to determine the support of the action space, which can be interpreted as leveraging the Granger causal structure to guide policy optimization. Conducting online causal inference given the data collected by RL provides an interesting direction, yet is still open in our context. Particularly, ensuring the identifiability of causal effect estimation requires careful investigation, especially considering our dynamics are modeled as neural ODEs with jumps.

A most related reference we found is [1], which asks the counterfactual question given the observed temporal point processes by behavior policy and is related to off-policy evaluation for target policy.

References: [1] Noorbakhsh, K., & Rodriguez, M. (2022). Counterfactual temporal point processes. Advances in Neural Information Processing Systems, 35, 24810-24823.

[Q2]: Intensity cost as the evaluation metric.

Here the intensity cost refers to the derived average intensity throughout the whole intervention process for each node. Since our model is based on the Counting process, instead of simulating the process itself, this average intensity will give us a fair and efficient indicator of how fast the counting process will grow during a given period. Moreover, the economic effect caused by the growing number of infected people and the carbon emissions because of the traffic congestion are all highly related to the intensity of newly infected people and newly arrived vehicles. Thus, in this paper, we think it is fair to use intensity cost to evaluate t.

[Q3]: Potential limitation due to the use of mean-field approximation for reward modeling.

For the limitation, we have a detailed error-bound analysis for the presented mean-field approximation which is covered in the updated manuscript Appendix E.2. This analysis is built upon a Lipschitz assumption of the given dynamic. The theoretical analysis further provides the insight that, as long as the spectral norm of the ODE neural layer is smaller than one, the mean-field approximation error can be well bounded.

[Q4]: Experiments to demonstrate the incorporation of fairness constraints.

We added experiments by considering two soft fairness-related constraints and provided an empirical study in Appendix Section I.2. Specifically, two fairness augmentation terms were added to the objective function: 1) intervention budget cost; and 2) policy smoothing cost. The first cost penalizes implementing the intervention policy that exceeds the predefined budget and the second one trades off the policy-changing frequency. In addition to the two terms under consideration, more realistic constraints relevant to specific scenarios can also be included in practice.

[Q5]: Experiment to show the balance of health concerns with economic implications and public sentiment.

During the intervention, we have three constraints related to the balance of health concerns and economic implications:

1. We have set up a hard constraint and implemented a dynamic mask to explicitly exclude actions that fall outside the feasible space, ruling out the action that consecutively locks down one county. This maximum length of consecutive lockdown for one city can give us a trade-off between enforcing long-time continuous lockdowns to control pandemics and allowing temporary ‘freedom’ to recover the economy.
2. We introduce a policy smoothing cost to penalize the abrupt difference between consecutive policies. This means a larger weight will heavily discourage the agent from frequently changing the policy (which will incur more financial burden) while a smaller weight will less discourage a frequent policy change.
3. We also introduce an intervention budget cost and penalize the intervention policy that exceeds the predefined budget. Intuitively, a large weight of the intervention cost will discourage the agent from intervening more edges and thus have limited control over the pandemic while a small weight of the intervention cost allows more connections to be controlled but will generate a higher expense.

To understand this trade-off, we have performed experiments of choosing different weights of intervention budget cost and policy smoothing cost in Appendix I.2.

Dear Reviewer ya7t,

Thank you very much for your commitment and effort in evaluating our paper. We understand that this is a hectic time, and we are writing to gently remind you about providing your insights on our rebuttal. As the discussion phase is drawing to a close, your feedback would be invaluable to us.

Should you have any further thoughts or recommendations about our work, we are eager to discuss them with you.

We eagerly await your reply.

Thank you once again,

The Authors

[Summary] Many thanks to reviewer hvRi for your positive comments and recognition of our contribution including 1) “The paper is well-motivated and technically sound”; 2) “The proposed method can notably handle large-scale systems”; 3) “The overall writing is easy to follow” and 4) “The experiment section is comprehensive”. We would like to address your concerns one by one.

[Q1]: NHPI Baselines and Additional Baselines.

As the suggestion to add more baselines, we implemented two common RL baselines: SAC and PPO on the COVID data. The empirical results have been incorporated into Appendix I.2 (Figure 13). By using NJODE as our simulator, we observe that the proposed method has successfully reduced the number of infested people (with an average reduced intensity: 0.49) in six counties among the chosen nine counties while SAC and PPO struggle to have a positive intervention effect to control the pandemic.

As for the constant intensity cost of the NHPI baseline, it was originally proposed on less dense point processes (i.e., tweet data) and used an event-driven RL to steer the point process. We conjecture the less satisfied performance is mainly caused by two aspects:

1. NHPI used a transformer-based sequential model on historical events to learn the dynamic model. This indeed is not easy to capture the inherent connections between different types of events and also not the best model representation for event sequences generated from a dense graph. On the contrary, the proposed NJODE is built on explicit graph embeddings and has a strong potential to model the mutual interaction between different types of events.
2. NHPI learns a stationary value function through the Bellman function on an infinite horizon semi-MDP. We think the single-step TD learning for semi-MDP presented in NHPI could cause bias to the learned value function, especially in the non-stationary environment. Instead, our policy is learned from a receding horizon window without learning additional value functions and can be adapted to a new model by iteratively updating the parameters of neural ODE.

[Q2]: Causal-inference-based baseline method

It is a great idea to combine causal inference with RL. In fact, in our experiment, we have introduced a pre-learned Boolean mask to determine the support of the action space, which can be interpreted as leveraging the Granger causal structure to guide policy optimization. Conducting online causal inference given the data collected by RL provides an interesting direction, yet is still open in our context. Particularly, ensuring the identifiability of causal effect estimation requires careful investigation, especially considering our dynamics are modeled as neural ODEs with jumps.
A most related reference we found is [1], which asks the counterfactual question given the observed temporal point processes by behavior policy and is related to off-policy evaluation for target policy.

References: [1] Noorbakhsh, K., & Rodriguez, M. (2022). Counterfactual temporal point processes. Advances in Neural Information Processing Systems, 35, 24810-24823.

Dear Reviewer hvRi,

Thank you very much for your commitment and effort in evaluating our paper. We understand that this is a hectic time, and we are writing to gently remind you about providing your insights on our rebuttal. As the discussion phase is drawing to a close, your feedback would be invaluable to us.

Should you have any further thoughts or recommendations about our work, we are eager to discuss them with you.

We eagerly await your reply.

Thank you once again,

The Authors

Dear Reviewer YqcV,

Thank you very much for your commitment and effort in evaluating our paper. We understand that this is a hectic time, and we are writing to gently remind you about providing your insights on our rebuttal. As the discussion phase is drawing to a close, your feedback would be invaluable to us.

Should you have any further thoughts or recommendations about our work, we are eager to discuss them with you.

We eagerly await your reply.

Thank you once again,

The Authors

[Summary]: We appreciate that reviewer YqcV has a positive impression of the “sound idea” and “well written” of our paper. To address your questions about the details of our method, we provide point-wise responses as follows.

[Q1]: Related work of networked excitatory point processes.

We have carefully reviewed and incorporated the listed publications in Appendix Section A (related work) in our updated version.

[Q2-Q4]: Definition clarifications of the model details.

Here we make the following clarifications.

1. Edge: For pandemic control cases, the node is defined as county/city, and the edges are defined as the accessibility from one city to another city (see Figure 1). For traffic control cases, the node is defined as a road lane, and edges are defined as the connections between different lanes.
2. Stationary Dynamic: Here $X_{t}$ refers to the number of a sequence of spike counts within a time step, so $X_{t}$ can be stationary in a long-term period. Moreover, under a careful choice of the initial parameters for the neural network (as the contraction condition derived in Appendix E), the dynamic system is also stationary.
3. Influence matrix $W$: In our cases, only the support of $W$ matters, and it can be obtained from prior knowledge. The reason is if we jointly learn the influence matrix $W$ and the discrete kernel function $\phi$ in the proposed system (Eq. (4) in the manuscript), W will become not identifiable and lose its original physical meaning. Therefore, we choose to fix the support of $W$ (i.e., $w_{ij} \in [0,1]$) and the specific value can be inferred from the data-driven method. For instance, in the COVID case, the influence matrix is restricted to $[0,1]$ and is proportional to the inverse distance between different counties.

We have also updated the above clarifications according to the updated manuscript.

[Q5]: Mean-field approximation justification

Here, we provide a brief justification for the mean-field approximation. For supplementary numerical and theoretical justification, please refer to our modified draft (Appendix E).

To evaluate the policy gradient at each iteration, one needs to get sufficient roll-out trajectories. Numerically, obtaining one roll-out trajectory requires solving ODEs repeatedly within each time interval, and sampling random counts at the end of each interval, which will be fed back to the system to update the state. The mean-field approximation, however, only needs to compute the roll-out once – it uses a deterministic trajectory to approximate the mean of the stochastic trajectories, by replacing the random counts sampled at the end of each interval by their means.

Mean-field approximation dramatically reduces the computational burden in policy optimization. We added numerical results (shown in Fig. 11, Appendix E.2) and theoretical analysis (in Appendix E.1) to demonstrate the approximation accuracy.

The theoretical analysis further provides the insight that, as long as the spectral norm of the ODE neural layer is smaller than one, the mean-field approximation error can be well bounded.

[Q6]: Amortized policy construction

Mathematically, the proposed amortized policy $\pi_{ani}$ is learned over a distribution of random tasks from $\mathcal{M}$, i.e., $\pi_{ani} = E_{M_{i}}[\pi_{i}]$ where $M_{i} \sim \mathcal{M}$. The meta-policy gradient over different tasks is then updated in a similar fashion with Reptile [1]. We have clarified the point in the updated version.

[1] Nichol, A., Achiam, J., & Schulman, J. (2018). On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999.

[Q7]: Figure label clarification

For brevity, here intensity cost means the average intensity throughout a fixed period and over all the nodes within a local region. We use this indicator to reflect how fast the pandemic/traffic will grow within a multivariate system. We have clarified this point in the updated version.

Dear Reviewer 9miN,

Thank you very much for your commitment and effort in evaluating our paper. We understand that this is a hectic time, and we are writing to gently remind you about providing your insights on our rebuttal. As the discussion phase is drawing to a close, your feedback would be invaluable to us.

Should you have any further thoughts or recommendations about our work, we are eager to discuss them with you.

We eagerly await your reply.

Thank you once again,

The Authors

[Summary]: We first thank reviewer 9miN for the insightful comments, especially for the questions about our high-level contributions and experiment designs, which benefit us to further clarify our paper. We would like to address the concerns one by one.

[Q1]: Contributions of the paper and the difference between our proposed loss and SimCLR.

We highlight the contributions part in the Introduction of the revised manuscript. Our contributions include three aspects: 1) a technically sound method to tackle the challenge of large-scale network intervention, 2) the proposal of a permutation-equivalent intervention paradigm, and 3) comprehensive experiments on traffic congestion and COVID data. Most importantly, our key contribution lies in introducing the permutation-equivalent intervention paradigm within a dynamically evolving large network topology, facilitated by the utilization of the proposed bi-contrastive loss function.

As for the difference between the proposed Bi-contrastive loss and the previous SimCLR (Chen et al., 2020b):

• Firstly, unlike SimCLR which directly performs augmentations on visual representations, we augment our network on its latent embedding space by utilizing graph augmentation techniques.
• Secondly, our model does not require sampling additional data as negative examples and projecting the augmented sample to one embedding. Instead, we only use one anchor sample (here is the embedded network) project the sample to two different embeddings (p and m), and augment the anchor sample by permuting its node orders or adding noise to the node embeddings such that it can form two negative pairs naturally (as depicted in Figure 1). More details will be provided in Appendix Section F in the updated version.

[Q2]: Global and local pattern for network work intervention.

In our model, we have the permutation equivalent property to extract the global invariant patterns. Regarding the local pattern, we use the node embeddings $H_{t}$ learned by the Networked Jump ODE model to store the information. For instance, the elements of node embedding can have the meaning of a local region’s population and economy. When performing policy optimization, the node embeddings will be used to learn the policy and global invariant patterns. Therefore, our policy can capture both local patterns and global patterns elegantly.

[Q3]: Dealing with soft and hard constraints.

For hard constraints like strictly controlling the maximum number of consecutive interventions, we use a ‘’dynamic mask” in our policy learning. For soft constraints, we can treat the constraints as ‘’regularization/augmentation terms” and add these terms to the objective function.

We added experiments by considering two soft fairness-related constraints and provided an empirical study in Appendix Section I.2. Specifically, two fairness augmentation terms were added to the objective function: 1) intervention budget cost; and 2) policy smoothing cost. The first cost penalizes implementing the intervention policy that exceeds the predefined budget and the second one trades off the policy-changing frequency. In addition to the two terms under consideration, more realistic constraints relevant to specific scenarios can also be included in practice.