SiBBlInGS: Similarity-driven Building Block Inference using Graphs across States

We appreciate the reviewer's thoughtful feedback on our manuscript and the referral to additional sources and references.

Additional Baselines:

We appreciate the feedback from the reviewer. In response, we have provided a brief description of additional models in the ''BACKGROUND AND RELATED WORK" (see in blue font) to underscore the distinctions from SiBBlInGS.

Furthermore, we added two more experiments, comparing our method also to: 1) NONFAT [1], and 2) NNDTN (discrete-time NN decomposition with nonlinear dynamics). The modifications include updates to Figure 2 in the paper to incorporate these comparisons, along with the addition of Figures 14 and 15, illustrating the identified components for all methods and the correlation with the ground truth per BB and per state. Please consult the updated paper for details on how we extracted these components, as described in the supplementary material.

Broadly, SiBBlInGS serves a distinct purpose compared to methods mentioned in [1, 2, 3] or other Gaussian Process (GP)-based approaches. While these methods leverage temporal dependencies for robust decompositions with predictive power for future time points, SiBBlInGS aims to identify interpretable functional groups based on co-activity with subtle controlled variability along different axes—the trial and state axes, which are orthogonal to the time axis. In particular, SiBBlInGS focuses on discovering interpretable components with their traces rather than predicting future time points (as presented in [1] for instance) or solely reconstructing test data.

Furthermore, the identified components in some of these other methods are not naturally sparse, and they do not aim to group sparse subsets of channels based on per-state temporal similarity. Hence, their interpretation does not align well with the discovery of functional groups that, in various fields, are assumed to represent co-active components, such as neural ensembles. Moreover, some of these methods cannot naturally work with data of varied duration or rates due to the inherent tensor structure.

Although integrating GP ideas and dynamical processes into the analysis, as seen in [1, 2], is promising and could be very interesting for future work to better model within-trial non-stationary temporal conditions, it is not the primary goal of the current framework.

[1] Wang, Z., & Zhe, S. (2022, June). Nonparametric Factor Trajectory Learning for Dynamic Tensor Decomposition. In International Conference on Machine Learning (pp. 23459-23469). PMLR.

[2] Tillinghast, C., Fang, S., Zhang, K., & Zhe, S. (2020, November). Probabilistic neural-kernel tensor decomposition. In 2020 IEEE International Conference on Data Mining (ICDM) (pp. 531-540). IEEE.

[3] Ahn, D., Jang, J. G., & Kang, U. (2021, October). Time-aware tensor decomposition for sparse tensors. In 2021 IEEE 8th International Conference on Data Science and Advanced Analytics (DSAA) (pp. 1-2). IEEE.

I thank the authors for the responses to my questions.

We thank the reviewer for the valuable feedback on our paper and would like to address the concerns below:

Novelty:

The novelty of our work goes far beyond the application of a concept to neuroscience. It lies in the development of a versatile framework that identifies functional building blocks through re-weighted temporal graphs, fostering sparsity in multi-way data while considering within-state and between-state information.

Our method tackles issues of real world data that other models typically overlook, such as varying trial durations, different trial counts per state, and handling of missing data or sampling rates, as well as enabling the discrimination between background and state-specific building blocks.

The potential applications of this method extend well beyond neuroscience, with potential applications in genomics, consumer behavior, financial markets, environmental sciences, urban planning, and the study of social dynamics (For additional details, please refer to the examples provided in our second response to reviewer T8G6). We believe our approach is a standalone method with universal applicability, including contributions to neuroscience, as well as to other fields.

Empirical results and theoretical guarantees:

Our paper presents a comprehensive empirical exploration of SiBBlInGS, highlighting its adaptability and broad applicability through various synthetic and real-world examples. The approach is grounded in the theoretical principles of dictionary learning, drawing from seminal works in the field (e.g., [1], [2], [3], [4]). However, we acknowledge that a comprehensive theoretical discourse on dictionary learning is beyond the scope of our 9-page, application-focused paper.

In particular, in [1], the authors provide a geometric characterization of the optimization landscape in dictionary learning, eliminating concerns over spurious local minimizers. [2] shows theoretical guarantees that detail how searching within an over-realized model space can foster the recovery of dictionaries, conceptually linking this process with generalization bounds and introducing an efficient strategy for extracting accurate model components. [3] establishes the theoretical framework necessary for confidently recovering dictionaries under sparsity constraints, and [4] marks a deep dive into sample complexity, stating how much data is needed to reliably engage in dictionary learning and parallel matrix factorization efforts.

We will include these references with a brief explanation in the introduction of the updated paper to emphasize the above.

[1] Sun, J., Qu, Q., & Wright, J. (2016). Complete dictionary recovery over the sphere I: Overview and the geometric picture. IEEE Transactions on Information Theory, 63(2), 853-884.

[2] Sulam, J., You, C., & Zhu, Z. (2022). Recovery and generalization in over-realized dictionary learning. The Journal of Machine Learning Research, 23(1), 6060-6082.

[3] Gribonval, R., & Schnass, K. (2010). Dictionary Identification—Sparse Matrix-Factorization via $\ell_1$-Minimization. IEEE Transactions on Information Theory, 56(7), 3523-3539.

[4] Gribonval, R., Jenatton, R., Bach, F., Kleinsteuber, M., & Seibert, M. (2015). Sample complexity of dictionary learning and other matrix factorizations. IEEE Transactions on Information Theory, 61(6), 3469-3486.

We thank the reviewer for raising concerns about potential similarity to SSMs and VAEs.

Comparison to SSMs and VAEs:

SSMs:

In State-Space Models (SSMs), latent states evolve according to a transition matrix that dictates how the state evolves, accompanied by an emission matrix that determines how the latent state translates to the observations. The resemblance of SSMs to SiBBlInGS may not be immediately clear—whether it pertains to the structure of the emission matrix or the potential dynamics operators in latent space.

If the concern was about potential similarities between SiBBlInGS and the emission matrix in SSMs, our approach incorporates re-weighted sparsity as a guiding principle, unlike SSMs' emission. Specifically, we utilize temporal-similarity-driven re-weighted $\ell_1$ regularization to find cross-state BBs, a feature distinct to SiBBlInGS. Furthermore, SSMs do not focus on studying cross-trial variability or on pinpointing the formation of emergent BBs under specific conditions.

If the reference is to the potential analogy between the BBs and the transition operators of SSMs or other dynamical systems models (e.g., as seen in the identification of dynamical operators such as those in rSLDS [1] or dLDS [2]), SiBBlINGS find BBs that are not a function of time, and the temporal operators are not evolving by dynamical operators. SiBBlInGS’ BBs act as sparse interpretable functional clusters of channels found based on co-activity with cross-state constraints, while dynamical operators in [1] or [2] are neither sparse nor integrating temporal-similarity regularization.

VAEs:

In the context of Variational Autoencoders, if the comparison is being drawn between SiBBlInGS' BBs and VAE's latent layer, it's essential to note that VAEs' latent layers are not inherently sparse and do not represent clusters from the original input (channels) space. They are found through sequential non-linear projections, resulting in a loss of interpretability and representing something different from SiBBlInGS' BBs. Even in the context of sparse VAEs (e.g., [3]), it's important to note that there is no co-learning of the components alongside their temporal traces. Additionally, with respect to dynamical VAEs (e.g., [4]), these models not align with SiBBlInGS's aim of understanding multi-state variability.

[1] Glaser, J., Whiteway, M., Cunningham, J. P., Paninski, L., & Linderman, S. (2020). Recurrent switching dynamical systems models for multiple interacting neural populations. Advances in neural information processing systems, 33, 14867-14878.

[2] Mudrik, N., Chen, Y., Yezerets, E., Rozell, C. J., & Charles, A. S. (2022). Decomposed Linear Dynamical Systems (dLDS) for learning the latent components of neural dynamics. arXiv preprint arXiv:2206.02972

[3] Ashman, M., So, J., Tebbutt, W., Fortuin, V., Pearce, M., & Turner, R. E. (2020). Sparse Gaussian process variational autoencoders. arXiv preprint arXiv:2010.10177.

[4].Girin, L., Leglaive, S., Bie, X., Diard, J., Hueber, T., & Alameda-Pineda, X. (2020). Dynamical variational autoencoders: A comprehensive review. arXiv preprint arXiv:2008.12595.

Distinguishing Within and Between State Variability:

SiBBlInGS adeptly integrates analysis both within and across states, maintaining consistency of the structure of within state BBs and accommodating nuanced controlled variations of between-state BBs. The degree of resemblance between corresponding cross-state BBs is methodically adjusted to reflect the corresponding levels of similarity between the states (as seen in the last term of Equation 1).

I thank the author for the detailed response with insights.

We appreciate the strengths highlighted by the reviewer, as well as the questions and feedback provided. We will now address the concerns and questions raised below:

Implementation Instructions and Code: As presented in Algorithm 1 (page 14) as well as given the code that was attached in the supplementary materials, we believe the algorithm can be used by future researchers as well. Once accepted and dis-anonymized, a GitHub repository with the code will be shared and a Python PyPi package for simple pip installation will be created for simple usage of the algorithm and to enable users to develop custom implementations and modifications.

Examples for Real World Applications: In addition to the real-world examples mentioned in the ''CONCLUSIONS AND DISCUSSION" section (section 6), please find below some more details about potential applications of SiBBlInGS:

• Consumer Products: SiBBlInGS could identify groups of products (channels) with similar purchasing patterns over time (time) across countries (states), and explore how these patterns vary by location to inform personalized customer recommendations.
• Economic Analysis: SiBBlInGS could be applied to cluster stock volume data (channels) over time (time) and across countries (states), to uncover patterns reflecting cross-country market trends and differences in investor activity.
• Climate Science: SiBBlInGS could be used to cluster carbon dioxide emission patterns of various countries (channels) across daily intervals (time) and yearly changes (states). This analysis aids in discerning global emission trends and could inform more targeted climate policy initiatives.
• Social Dynamics Analysis: SiBBlInGS could be utilized to analyze and understand the formation and evolution of online user groups (channels’ BBs) based on shared temporal patterns of user activity (time) across different states (e.g. before/ during / after the COVID-19 pandemic). This approach could enable the observation of shifts in social interactions and group dynamics in response to global events.
• Genetics Research: SiBBlInGS could analyze multi-tissue gene expression data (e.g., as the data described in [1]) to identify functional gene groups (channels) and assess how their expression across tissues (~time) differs between individuals (states).

[1] Hore, V., Vinuela, et al. (2016). Tensor decomposition for multiple-tissue gene expression experiments. Nature genetics, 48(9), 1094-1100.

Potential areas for improvement: As outlined in the final paragraph of the paper, certain limitations, beyond the current scope of the paper, will be addressed in future work. In particular, currently SiBBlInGS presumes normal noise distribution (e.g., as seen in the $\ell_2$ norms of the data fidelity terms in equations 1,2) and Euclidean distance between state labels in the supervised approach (definition of $P$), which may not align well with some data (e.g., neural data may be modeled with a Poisson distribution in low spike count rates). Additionally, while the identified BBs capture channel membership and magnitude, they currently do not account for potential directed connectivity within the BBs, presenting an opportunity for future extensions. For future work, we also plan to showcase SiBBlInGS's versatility by integrating diverse data modalities with the same structure (e.g., fMRI vs ultrasound of the same brain area). Additionally, we plan to extend its usability to handle a continuous, infinite number of non-discrete labels, as well as incorporating extra axes for state changes, allowing a trial to belong to several states within a pool of states.

Potential limitations and improvements of SiBBlInGS in terms of scalability and computational efficiency: As shown in Section E on model complexity, the current implementation's speed scales with the dataset's dimensionality, encompassing the number of channels, states, time points, and the rank of each trial. To enhance computational speed, the implementation already incorporates random subsampling of time periods and trials within each state during iterations (as mentioned in the paper after equation 1). This allows the model to process only a portion of the data at each update step, eventually exposing the model to all trials and time points. Additionally, the current implementation offers users to choose between different lasso solvers that vary in speed. Future improvements for efficiency may involve implementing the model in languages other than Python, such as C++ or Julia, and applying parallel computing for faster and more efficient performance. With respect to real-time applications, SiBBlInGS is not intended for real-time inference. However, if BBs are found previously based on some training data for a state, they can be easily used in real-time applications to find their respective traces when new observations arrive (i.e., by solving equation 9).

We appreciate the reviewer's comments on SiBBlInGS compared to other approaches and its effectiveness.

Comparing SiBBlInGS with Other Approaches:

As demonstrated in the synthetic data results (Figure 2), EEG experiments (Figure 13), and trends experiments (Figure 8), along with additional comparisons detailed in Table 2, we assessed our method against alternative tensor and matrix factorization techniques, through both theoretical analysis and practical experiments.

SiBBlInGS stands apart from classic tensor factorization methods, as it finds similarity-driven sparse BBs with subtle cross-state variations, accommodates variable trial durations and inherently incorporates state information and similarities into the analysis, effectively differentiating intra- and inter-state variability— aspects not addressed by other approaches. Additionally, certain methods presume orthogonality among components, a condition that may not hold true for some real-world cases (e.g., neural ensembles). These alternative approaches often tend to find components based on maximum variance or energy rather than co-activity, and they are not inherently tailored to differentiate state-specific from state-invariant building blocks (as $\nu$ does in SiBBlInGS).

Significance of Temporal Activity, Inter/Intra-State Relationships, and Non-Orthogonality:

Shared Temporal Activity:

We appreciate the reviewer’s question about the contributions of different features of SiBBlInGS to the framework. As detailed following Equation 1 in the paper (starting with 'The weighted evolving regularization terms for each state...'), the channel similarity graph (${H}$) is built based on temporal activity similarities between channels per trial. This graph is integrated into the denominator of the regularization weight ($\lambda$) in the re-weighted lasso equation (2nd part of Equation 1) and thus fosters the aggregation of channels with co-active patterns into the same BBs.

In particular, the term $H_{n:}A_{:j}$ that appears in the denominator for updating $\lambda_{n,j}^d$ (2nd formula in Equation 1) reflects the correlation between the activity of the $j$-th BB (as defined by $A_{:j}$) with the similarity levels of all channels to the $n$-th channel (as defined by $H_{n:}$). Since that term appears in the denominator of $\lambda_{n,j}^d$—a higher correlation between $A_{:j}$ and $H_{n:}$ (which implies that the temporal neighbors of the $n$-th channel might be in the $j^{th}$ BB), results in a smaller sparsity weight ($\lambda_{n,j}^d$ decreases)—thus promoting that $n^{th}$ channel's inclusion in the $j^{th}$ BB and effectively ‘’pushing” the channel to the BB that includes its temporal neighbors.

Conversely, if the temporal neighbors of channel $n$ mainly do not belong to that $j$-th BB, the term $H_{n:}A_{:j}$ will be smaller, hence $\lambda_{n,j}^d$ is larger, resulting in an increased sparsity regularization that effectively pushing that $n$-th channel out of the $j^{th}$ BB.

Another effect of the shared temporal activity is expressed in the data-driven approach, as the state graph is a function of the temporal similarities between states, and the regularization on the distance between cross-state BBs is proportional to the temporal similarities between these states (stored by $P$, and as seen in Equation 1), thereby encapsulating state-temporal similarities within the analysis.

Specifically, last term in Equation 1 ($P_{dd'} \|\|a_{i:} - a_{j:}\|\|_2^2$) modulates the $\ell_2$ norm on the distance between corresponding BBs across different states in accordance with the states' similarities, ensuring that more similar states are associated with more similar BB patterns.

Non-orthogonal components:

Our method's ability to identify components without enforcing orthogonality constraints, as opposed to techniques like HOSVD or PCA, aligns better with real-world data where such constraints may not naturally exist. For example, in the context of neural ensembles, it is plausible to encounter functional groups that are non-orthogonal, both structurally and temporally.

Intra vs inter state relationships:

As written just before Equation 1 (the paragraph starting with “Assuming consistent underlying groups between same-state trials,....”), the BBs are kept constant between trials of the same state while supporting subtle controlled structural variations between states, thus addressing both resolutions. Unlike the regularized BBs, SiBBlInGS allows for flexible variations in temporal traces within and between states, capturing the complete spectrum of activity variability while maintaining a shared basis of BBs for interpretability. Additionally, as shown in Figure 4E, the within-state temporal trace correlations are larger than between-states temporal correlations of corresponding ensembles, indicating reduced within state compared to between states variability.