Structural Inference of Dynamical Systems with Conjoined State Space Models

We would like to thank Reviewer uuWM for the motivating review! Here are our answers to the concerns:

The implementation of SICSM is computationally intensive, requiring significant resources and expertise. This complexity may limit its accessibility and widespread adoption.

Many thanks! We acknowledge that the complexity of SICSM, with its dual components of SSSM and GFN, may initially appear daunting. To enhance understanding and facilitate easier adoption, we will include a comprehensive tutorial in the code repository that outlines each step and its purpose within the framework.

How does SICSM handle situations where the interaction structures of the dynamical systems change over time? Are there plans to extend the framework to support dynamic graphs?

Many thanks for the good question! Addressing changing interaction structures in dynamical systems is indeed challenging, particularly in distinguishing between existing and emerging connections which could blur the reconstructed adjacency matrix. We are exploring the potential use of latent variables to capture these dynamics more accurately. This approach is in its early stages and will require further research and validation.

Can the authors provide more details on the computational resources required for training? Are there any strategies to optimize resource usage?

We included the training time of each method in Table 4 in the appendix. We trained all of the methods with a single NVIDIA Ampere 40GB HBM graphics card, paired with 2 AMD Rome CPUs (32 cores@2.35 GHz), as mentioned in Section 5.1 of our submission. As we were trying to bridge the Generative Flow Networks with the SSSM, yet only JAX implementations were available, but it comes without the GPU acceleration of SSSM, which was included in its PyTorch implementation. We will work on the way of either implementing the GPU acceleration of SSSM with JAX, or trying to rebuild the GFN with PyTorch.

What specific real-world applications do the authors envision for SICSM? Are there particular domains where it has shown exceptional promise?

Yes, SICSM is particularly suited for complex domains such as single-cell biology, where it can infer gene regulatory networks, and other scientific fields requiring discovery of latent connectivity among variables. Its ability to handle irregularly sampled data and partial observations makes it a valuable tool for scientific discovery across various disciplines.

We hope these clarifications address your concerns, and we are committed to further refining SICSM to enhance its accessibility and applicability in real-world scenarios.

We would like to thank Reviewer 3YzK for the detailed and thoughtful comments. Here are our answers to the questions:

My main concerns for the paper are its novelty and its low number of ablations, which make it hard to understand how specific pieces contribute to the performance of the method.

Many thanks for the comment. The core innovation of our work, SICSM, addresses structural inference with irregularly sampled trajectories and partial observation, a novel approach not previously explored in structural or relational inference fields. SICSM effectively integrates SSM and GFN to tackle these challenges, significantly enhancing accuracy in complex dynamical systems.

Additional ablation studies are detailed in the revised sections of our manuscript. These studies critically evaluate the contributions of individual components within SICSM, underscoring their collective impact on performance enhancement. We show in the other answers on the ablation studies.

[...] It's not always clear what's a novel contribution and what's not. I encourage the authors to be clear and exercise an abundance of caution. [...]

Many thanks! We have clarified the modifications and novel contributions of the GFN in our system, particularly concerning the reward function adaptations detailed in Appendix B.5, now moved to the main text for greater visibility. Actually, JSP-GFN learns the parameters $\lambda$ for each node and therefore restore individuity for each edges. This setup is highly align with the real-world: the connections may be isomorphic and we found it to be really helpful for the reconstruction of graph structure for dynamical systems. We have also revised our paper entirely to have clearer reference to DAG-GFN [14] and JSP-GFN [15] to omit misunderstanding of the contribution of this work. Such as on the objective mentioned in section 4.3, and so on.

I'm not sure what an $𝛼$-distance is, is it meant to be a placeholder for any norm?

Yes. The $\alpha$-distance is indeed a placeholder for any norm, adaptable based on the dynamics of the specific system under study.

Section 3.3 introduces the flow-matching condition, but more modern conditions exist, and this work in particular seems to be using DB. Why is it only introduced in the appendix if it is the chosen objective? Why not just directly present the DB condition used?

Many thanks! We revised Section 3.3 to better articulate the use of the DB condition, which we initially included in the appendix. It is now prominently discussed in the main text, aligning with its significance in our methodology.

[...] This describes a standard residual model, [...]

Many thanks for the comment. We just wanted to have more detailed discussion on the advantages of using residual setup here in the text. We revised them to be more precise.

What the authors as "intricate multi-hop relationships" is simply a natural and normal consequence of depth in deep neural networks [...]

We sincerely disagree with this comment. We tried to handle this with regards to the multi-hop relationships illustrated in the Fig. 4 in the submission. As showcase here, on the third column, the red link from node 1 to 4 is a multi-hop connection, while the direct connections would be 1 -> 3 -> 4. So we are trying to state that a finer temporal resolution on the trajectories can help to decrease the possibility of getting multi-hop connections. So with more residual blocks, the learned dynamics will be finer in the temporal aspect.

The trick presented in (9) is neat, but it does imply spending $𝐵$ times more compute. Are baselines also allowed to use this trick?

Yes, the computational trick in Eq. 9 is also employed by baselines. This technique has been verified across all implementations, confirming its efficacy without disproportionately increasing computational demands.

Section 5.4 poses an interesting hypothesis, but it's unfortunate that it is only qualitatively evaluated. Why not run proper experiments and measure the effect of residual depth?

The observations in Section 5.4 are backed by rigorous experiments, with detailed quantitative results now included in Table 2 of the attached PDF. As shown in the table, the negative multi-hop edges are commonly observed among setups with just one residual block, and decreases to 6.5 if we use just one first block, but at this moment, the true positives decrease as well. As shown in the table, the best setup is the concatenation the outputs of all blocks, which on one hand decrease the count of negative multi-hop edges and on the other hand increase the count of true positives.

Another issue more generally is that the design choice of using a residual SSSM model doesn't seem compared to alternatives [..] i.e. ablations?

We have expanded our ablation studies to include comparisons with Transformer, LSTM, and GRU models on both irregularly sampled trajectories with 30 time steps and partial observation with 12 nodes of VN_SP_15 dataset. We show the average results below of 10 runs. The parameters of all of the modules are set to match the length of the trajectories and we performed hyperparamter search with Bayesian optimization. As shown in the table, transformers perform slightly better then LSTM and GRU, but all of them are inferior to SSSM, as they cannot deal with multi-hops effectively. Moreover, they fell short on irregulaly sampled trajectories, as they can not learn $\Delta$ adaptively. We included these results in revision.

We hope these clarifications and additions address your concerns effectively. We are grateful for your insights, which have significantly contributed to enhancing the rigor and clarity of our work.

We would like to thank Reviewer hoUX for the thoughtful comments. Here are our answers to the questions:

The method has 3 key components: [...]. It is not entirely clear how these individual components interact and the explicit need for the GFN.

The state space model in our approach handles the dynamics between sampled features at each time step and adapts to irregular sampling intervals through SSSMs. Residual blocks help extract multi-resolution dynamics, enabling the model to learn node dynamics across varying time resolutions. The GFN then creates a space of hidden states based on these dynamics to reconstruct the underlying structure. This layered approach ensures comprehensive learning and reconstruction capabilities.

The authors consider a comprehensive set of datasets and baselines, but only one evaluation metrics (AUROC) [...]

Many thanks! We have expanded our evaluation metrics beyond AUROC, as detailed in the PDF attached to the general rebuttal. Figures 4-6 include results for AUPRC, SHD, and F1 score, where our method, SICSM, consistently outperforms baselines across most datasets, highlighting its robustness even in complex structures like those found in the TF dataset.

Including a method like CUTS (Cheng et al. 2023) in this evaluation may be important to create a fairer comparison of SICSM.

Many thanks for the suggestion. We investigated into the CUTS (Cheng et al. 2023) and figured out that they suppose having regular time steps, but some of them are marked out by ZOH placeholders. So actually CUTS work on the regularly sampled trajetories with equal time intervals, but some of them were masked out, which is different from the problem setting of SICSM. Moreover, CUTS can only work on trajectories with one-feature for every node at each time step. In contrast, SICSM is supposed to have irregularly sampled trajectories without any ZOH placeholders, and can work on multi-dimensional features. We performed experiments of CUTS on all of the one-dimenstional features trajectories mentioned in this paper, and the results are shown in Fig. 1, 2, and 4 -6 in the attached PDF. As shown in the figures, CUTS has the ability to match with the best baselines, but still outperformed by SICSM. We included the new results in the revision.

For the reward defined in Equation 8, what is the explicit form of $𝑅(<𝐺,𝜆>)$? The authors state that $𝑃(𝑈_{𝑎𝑙𝑙}|𝜆,Adj)$ represents the likelihood model implemented via a neural network. Is this model trained beforehand?

The reward function $R(<G, \lambda>)$ is transformed logarithmically (see Eq. 37 in Appendix B.5) for implementation purposes. The model $P(U_{all}|\lambda,Adj)$, implemented via a neural network, is learned simultaneously with the GFN, ensuring integrated optimization and learning.

A central advantage to using a GFN to model structure is the ability so approximate the distribution over this structure (and in this case also over the parameters). [...] Why not consider a distributional metric to evaluate how well $𝑃(Adj,𝜆|𝑈_{𝑎𝑙𝑙})$ is approximated?

Many thanks for the comment! We evaluate the approximation returned by SICSM by comparing with the exact joint posterior distribution $P(Adj, \lambda | U_{all})$. Similar to the experiments in JSP-GFN, we consider here models over d = 5 variables, with linear Gaussian CPDs. We generate 20 different datasets of N = 100 observations from randomly generated Bayesian Networks. The quality of the joint posterior approximations is evaluated separately for $Adj$ and $\lambda$. For $Adj$, we compare the approximation and the exact posterior on different marginals of interest, also called features in JSP-GFN (Deleu et al, 2023), e.g., the edge feature corresponds to the marginal probability of a specific edge being in the graph. Fig. 3 in the attached PDF shows a comparison between the edge features computed with the exact posterior and with SICSM, proving that it can accurately approximate the edge features of the exact posterior. To evaluate the performance of the different methods as an approximation of the posterior over $\lambda$, we also estimate the cross-entropy between the sampling distribution of $\lambda$ given G and the exact posterior $P(\lambda | Adj, U_{all})$. The results are shown in Table 1 in the PDF. We observe that again SICSM samples parameters $\lambda$ that are significantly more probable under the exact posterior compared to other methods.

What is the motivation of also learning the parameters $𝜆$ if the primary objective is to learn $Adj$?

The diverse nature of real-world graph connections, such as those differing significantly in physical properties (e.g., springs vs. electric forces), necessitates modeling each connection with unique parameters $\lambda$. This individualized approach enhances the accuracy and applicability of our model in complex scenarios.

Is $Adj$ embedded in the SSM of each residual block? Is the approximated graph structure being used by the SSM or is this an independent output?

The graph structure $Adj$ is not explicitly integrated into the SSM of each residual block; it is an independent output of our model. We are exploring methodologies to potentially link $Adj$ more closely with SSM operations in future work.

We hope these clarifications meet your concerns and illustrate the depth and rigor of our study. We are committed to further enhancing our model based on the feedback received and appreciate your guidance in this process.

We would like to thank the reviewer for the inspiring review. And here are our answers to your concerns.

To improve the writing, a running example might help bridge the abstractions (node, edge, state...) to physical reality [...]

We would like to sincerely thank the reviewer for this advice. We added the following paragraph to our revision in Section 3.1 and it will appear with the camera-ready as we cannot upload the revision during rebuttal:

We may image a dynamical system of balls and strings, in which the balls are randomly connected with the springs. Then set initial positions and velocities of each ball and then let them go. Because of the existence of the spring forces, arising from the structural connections between the balls, the balls will change their positions and velocities in the observational period. Then suppose we have no idea on which balls are connected, so the task of structural inference would be, the inference of the connectivity between the balls based on the observational trajcetories of them.

However, all datasets are synthetic. The only real dataset, PEMS, presented in Annex C.5 [...]

Many thanks! As acknowledges by the research in this field, the acquisition of trajectories with reliable understanding of the underlying structure from real-world, is of huge cost and time-expensive. We do acknowledge this urgent call of the reliable real-world data, and we are now working on it.

There seem to be a duplication in the presentation of datasets [...]. Also, sec3.3 seems to be internally redundant with duplicated points (e.g. l.151 vs eq3, and l.148 vs l.157, which again is confusing.

Many thanks for the comments, we revised our paper to reduce the duplications and tries our best to keep most relevant information to understanding of our paper in the main body.

Eq.2: since s' is terminal, isn't any $s'' = s_f$?

Actually, no. $s''$ here in Eq. 2 is used to mark the child of the state $s'$. So only if $s'$ pointing to a terminal, then $s'' = s_f$.

Eq.5: in your contemplated application scenario, the interval between sampling times, or equivalently the timesamp of each sample, allowing to calculate Δ, is not given with the samples, correct? I've worked on mobile accelerometer/GPS data where sampling is irregular but the timestamps are given, which is why I'd like to make sure. Can you clarify whether a posterior over Δ can be pratically recovered?

No, $\Delta$ is not given with the samples. Many thanks for the inspiring question. Technically, we think the posterior over $\Delta$ can be partially recovered, but we have to change the modeling of $\Delta$ in the SSSM module to be with two vectors. The two vectors work similar to those in variational autoencoders, and we can recover the posterior over $\Delta$ partially from them.

Annex 1 fig5: shouldn't all variables be indexed with $i$, the node id? [...]

No, the nodes share $A$ and $h^t$.

fig6: do I have it right that GFlowNet only adds edges, but doesn't remove any, moving from start to end? Does that have as a consequence that any prior knowledge can only formulated as known-to-be-present edges, but not known-to-be-absent edges?

Yes, it does not remove any edge. Yes, currently we can only incorporate known-to-be-present edges to be the prior knowledge, as it fullfills the scenarios in some fields like the GRN inference, where several connections are validated by experiments, so we are more sure about the existence of some edges than the absence of certain edges.

Annex C.5: what is the physical model? What is a node, an edge of the model?

The model in Annex C.5 is that: many sensors are used to count the traffic in a roadmap, and these sensors may be connected by the road they align with. In this case, suppose we have no idea on the map, but try to reconstruct the map from the traffic-counting from the sensors, as in most cases, continuous traffic flows happen between adjacent sensors.

Annex l.872: how is the % of prior knowledge defined?

It is defined as the proportion to the count of all positive edges in the groundtruth graph.

Annex l.863: link is referred to but missing AND Checklist point 4 and 5: implementation link is claimed to be provided here and in Annex l.863, but I don't see it [...]

Many thanks! We revised our paper with the link. As you may interested in our implementation, we included them in our supplementary materials.

Limitations: a few more assumptions on the usage scenario should be spelt out

Many thanks! We will include the assumption on prior knowledge: sure-to-to-be existed in the limitations among other suggested. And also the experiments in this work mainly covers synthetic data.

Checklist point 7: It is certainly possible to report error bars on plots through shading, provided they are not as tiny as you make them here. [...]

Many thanks for the advice! We revised the figures to make them with shadings. Please refer to the images in the pdf file attached in the general rebuttal.

Checklist point 6: experimental settings are not as detailed [...]

Many thanks! We revised this section and include the batch size: 32, data splits into training, validation and test: 8: 2: 2 for all of the datasets. We also include more such as number of epochs and so on.

Checklist point 12: the claim, and requirement of the checklist that assets have their license mentioned is not complied with regarding either datasets or existing code for competing methods. [...]

Many thanks! We will include the license in our repository. As current we have to obey the double-blinded rule, we removed them from the repo.

Checklist point 13: does this imply the code is not intended to be released as an asset?

Sorry for our mistake, We correct it to be: We will release the code in our github repo upon acceptance. But we do not include the data, as they are from other work and are public available.

We hope our answers addressed you concerns correctly.

Thanks to all. I have read all reviews and rebuttals, which are all very informative. Overall, I seem to disagree with fellow reviewer uuWM; I'm glad that hoUX and 3YzK could shed light on links to related work and technical choices.

Reviewers' clarification questions and misunderstandings, including my own, are unmistakeable symptoms of obscurity, since they cannot be dispelled even after careful reading by typical professionals. Some improvements were achieved through revisions, and I very well know how difficult it is to implement these in a short time. Nonetheless, writing is still unclear in many places, and I find that concerns by reviewer 3YzK on distinguishing from competitor methods have not been fully addressed. Extra experiments and ablations proposed in your answer to this reviewer are very useful.

We may image a dynamical system of balls and strings, ... Despite broken English, the example helps. The text would further benefit from introducing the terminology (edges, nodes, observations, sampling) and notation on the example.

The model in Annex C.5 is that: ... I now understand that the node variables are "volume" units (vehicle counts) as opposed to flow units (volume per time, i.e. vehicles per second). Therefore in this example, increasing $\Delta$ will increase the value of the variable. In springs and balls, this also applies to displacement (an integrated form of velocity). Most examples I know, however, have variables in flow ("intensity") units. Is this correct? Does this affect performance of SISCM or some competitor methods? Therefore, for SICSM to work on datasets where variables are "volumes", is it an important hypothesis that throughout observations, the same partial set of node variables is observed at each time point?

I understand your response to reviewer hoUX on taking a pointwise estimate of graph instead of taking advantage of the distribution. I believe that the issue goes somewhat wider than your response hints at: your evaluation is carried out against known ground truth distributions (e.g. via cross-entropy) only because all your datasets are synthetic and you have access to the ground truth. It is still a major weakness that only synthetic datasets are used.

Extra experiments, ablations and clarifications offered during the rebuttal definitely helped; since the reviewing procedure does not support paper revisions, one must hope that the paper can benefit from them, which is not trivial considering that owing to length limitations, every addition of text or figures requires a deletion.

On the whole, many weaknesses pointed out by myself and fellow reviewers during the review process have been confirmed. I maintain my assessment.