Graphical Multioutput Gaussian Process with Attention

Q7. Given the proposed model builds upon a direct effect among nodes, how can one deal with the missing data case?

For low missing rates, standard imputation or interpolation methods suffice. In the case of unbalanced data, where some outputs have significant missing entries, we discard the missing data since the GMOGP models each output separately with shared kernel hyperparameters.

The vanilla GMOGP, grounded in Equation (13), obtains predictions through inference, diverging from the simple elementwise weighted summation as in attention networks. Proposition 3.1 asserts that the GP prior for each output depends solely on its own data, facilitating the effective handling of unbalanced data per output.

Dealing with unbalanced data requires an additional weight matrix for transforming attention inputs to a uniform dimension, a common practice in attention networks. The updated version includes more concise descriptions, clarifications, and insights into the model's limitations.

Q8. There are relevant works in the literature that are not mentioned in this paper. A model that is particularly similar is the Semiparametric latent factor model SLFM, which builds an MOGP by linearly combining a set of single-output GPs (rather than relating them through a graph as done in this paper). Also, there have been a number of expressive kernels proposed in the last decade, but this paper only considers LMC (one of the first approaches) and GPRN.

The SLFM is a instance of the GPRN and the LMC (see (Bonilla et al., 2007), (Wilson et al., 2011), (Liu et al., 2018), etc.). However, the GPRN already outperforms SLFM in the Jura dataset (Wilson et al., 2011). Advanced MOGP models, such as LMC, SLFM, GPRN, convolutional LMC, etc., necessitate the construction of large covariance matrices, each characterized by distinct matrix-valued kernel functions.

In contrast, the proposed GMOGP departs from the need for matrix-valued kernel design, presenting a unique paradigm in MOGPs. It provides an efficient framework with enhanced flexibility and representations, accommodating more expressive kernels.

In recent experiments, we introduced convolutional MOGP (CMOGP [3]), a non-separable model unlike LMC variants. See the table for predictive results. However, CMOGP encounters memory limitations when applied to the full SARCOS dataset.

• [1] Mauricio A Alvarez, Lorenzo Rosasco, and Neil D Lawrence. "Kernels for vector-valued functions: A review." Foundations and Trends in Machine Learning, 4(3):195–266, 2012.
• [2] Bruinsma, Wessel, et al. "Scalable exact inference in multi-output Gaussian processes." International Conference on Machine Learning. PMLR, 2020
• [3] M.A. Álvarez and N.D. Lawrence, "Sparse Convolved Multiple Output Gaussian Processes", Advances in Neural Information Processing Systems 21, 2009

Thank you for your review! We would like to underscore that our work introduces a novel MOGP framework, offering extra flexibility and graphical representations. We address your key concerns below.

Q1. "The proposed dependence structure between an output node and its parents is Gaussian (eqs 12 and 13), which in turn results in an oveall (MO)GP. Therefore, the proposed model could be understood as a (standard) MOGP with a specific kernel. In addition to the conceptual impact of this fact, there is also a practical consequence: It is unclear if the exhibited experimental superiority of the proposal comes from the introduced graph idea or the fact the there is an implicit, more expressive, kernel being used. From the paper is not possible to clarify this because the proposal is only compared empirically to classic kernels for MOGP (e.g., LMC), leaving behind the MO spectral mixture, deep kernels, cross spectral mixture, etc."

The proposed model cannot be understood as a standard MOGP with a specific kernel due to a crucial reason:

• We interpret the correlation with asymmetric measure. Any specific MOGP kernel must be symmetric. In our approach, the interaction (correlation) between different outputs is quantified by the attention mechanism (denoted as $\alpha_{ij}$). Consequently, our GMOGP introduces an asymmetric interaction measure ($\alpha_{ij}\neq \alpha_{ji}$), offering enhanced flexibility.

As a result, there is no necessity to construct the large kernel matrix (Eq. (5)), thereby alleviating computational and storage burdens. In this paper, we explored an alternative graphical MOGP model structure and learning framework.

By introducing the graphical model, we can derive a more flexible GP prior (even non-Gaussian), flexible interaction measurements, an efficient learning framework, and graphical representations. This heightened model flexibility and improved representation enable us to achieve experimental superiority.

In our experiments, we aim to discern the prediction performance between the proposed graphical MOGP and the classic MOGP framework using the same base kernels. Additionally, our GMOGP accommodates other expressive kernels (spectral mixture, deep kernel) for further exploration.

Q2. The proposed model introduced the (hyper) parameters \alpha as weights learned via attention. These hypers have a direct effect on the linear combination of the mean of the marginal GP which models each node. Though the intuition of attention is that it represents cross relationships among variable of interest, there is no reason to assume that they linearly affect the output. In general, attention networks have an MLP stage after the attention stage, where the MLP "discovers" the role of the attention weights.

Starting from simpler concepts and progressing to more complex ones, we initially demonstrate how a multivariate Gaussian can be represented as a directed graph, corresponding to a linear Gaussian model (linear combination). Subsequently, in Section 3.4, we address "linearly affect" by introducing the marginal flow. Specifically, the target node in the TGMOGP undergoes transformation via the non-linear Sinh-Archsinh (SAL) function (denoted by ${\mathbb{G}}_{\phi_k}(\cdot)$ ) in our experiments. See Eq.(52) in Appendix A.

In the realm of graph attention networks, the MLP stage is indispensable for achieving the final prediction. In contrast, the prediction in the GMOGP is derived through GP inference, not following an 'end-to-end' approach. Furthermore, we can apply K layers of non-linear transformations ('deeper MLP stage'), leading to a more adaptable GP prior/posterior. In our experiments, we implement 3-layer SAL transformations. Detailed information about the TGMOGP model and other non-linear transformation functions is provided in Section 3.4 and Appendix B. We emphasized this insight in our revised paper.

Q4. To mitigate computational costs, the model introduces DGP, which can also be applied to conventional MOGP. However, the paper falls short in discussing why this solution is faster than the other baselines. Both the attention mechanism and non-Gaussian transformation inherently raise computational costs, making it essential to have a comprehensive and detailed discussion on this issue within the paper. This is particularly important because it is one of the claims made in the paper. Regrettably, the authors only touch upon this matter in the experiments section, which is not sufficient to provide a thorough understanding of the computational efficiency of their approach.

Directly applying the DGP to the conventional MOGP ($\mathcal{O}(N^3S^3)$) results in a complete loss of interaction information among other outputs. Consequently, the corresponding distributed MOGP degrades to isolated SOGPs, rendering it meaningless.

In our experiments, we have already compared isolated SOGPs. The key challenge lies in designing a distributed MOGP framework without sacrificing interaction information, and the proposed GMOGP provides a viable solution.

Q5. A similar issue pertains to the prediction quality. The proposed solution integrates a combination of existing methods within a unified framework. However, the question arises: why does this method enhance prediction quality and surpass other baselines? While the experiments demonstrate a significant improvement, the underlying reasons have not been sufficiently elucidated within the paper.

The proposed model outperforms others due to the following advantages:

1. Enhanced Flexibility in Interaction Measures: Unlike traditional approaches that measure dependence between outputs through covariance (kernel function), our model employs an attention mechanism. This introduces a more flexible and asymmetric dependence measurement, allowing for $\alpha_{ij}\neq \alpha_{ji}$.
2. Separable Model Structure: Our model eliminates the need to construct the large gram matrix $K_M\in\mathbb{R}^{SN\times SN}$ used in standard MOGPs. This streamlines computational and storage requirements.
3. Efficient Learning Framework: Diverging from the $S$-dimensional loss function typical in standard exact MOGPs (Equation 6), our model adopts a $1$-dimensional weighted sum objective (also the ELBO). This formulation facilitates the derivation of Pareto optimal solutions for kernel hyperparameters and distributed learning schemes.
4. Additional representation ability: The model can generate non-Gaussian prior/likelihood, alleviating the model misspecification on ubiquitous non-Gaussian data and providing graphic representations.

See more details in Section 3; we added more description in the updated version.

Thank you for your feedback. We wish to highlight that our GMOGP can seamlessly extend to a distributed MOGP framework without compromising interaction information. We trust that the responses effectively address your concerns.

Q1. The current manuscript still exhibits several weaknesses. Notably, in certain sections, the paper's coherence is lacking, making it challenging for readers to follow. Mathematical concepts, in particular, suffer from inadequate definitions. For instance, the paper introduces $f_X^{(i)}$ as a latent variable representing the GP functions for a specific target variable i. However, it later employs $f_X^{(i)}$ to denote the distribution of a GP conditioned on its parents (as seen in Eq. 11 and Eq. 13). Given that prior and conditional distributions hold paramount importance in this work, it is imperative that related variables are precisely defined, and vague expressions are avoided.

In Eq. (11), $f_X^{(i)}$ represents the target node in the specific directed graph. Preceding Eq. (11), each node, including the target node, is explicitly defined as a SOGP evaluated at finite inputs X. Hence, $f_X^{(i)} \in \mathbb{R}^N$ and follows a multivariate Gaussian distribution (see Eq. (3) and (4)).

In Eq. (13), we introduce a linear Gaussian model for the target node, providing an explicit expression for the conditional distribution $p(f_X^{(i)}|pa_i)$ in Eq. (11). The corresponding multivariate Gaussian distribution, $p(f_X^{(i)})$, is derived in Proposition 3.1.

We welcome the identification of any other unclear expressions that may pose challenges for understanding.

Q2. Additionally, a considerable portion of the explanations provided in Section 2 concerning directed graphs appears to be redundant. The authors delve into the intricacies of directed graphs and the definition of a Bayesian network, yet the second paragraph on page 4 reveals that the graph relating the GPs in this paper is a fully connected undirected graph (as indicated by the definition of for all output functions). It begs the question of why the authors did not emphasize this type of graph from the outset. There are several undirected graph models that can be pertinent when dealing with Gaussian variables, including Gaussian graphical models, pararnormal/nonparametric graphical models, and functional graphs.

The definition of $pa_i$ does not influence the graph type; it solely designates the nodes with arrows pointing directly to the target node.

In our proposed model, the graph is directed for the following reasons:

1. Two links exist between any two nodes due to the asymmetry of the interaction measured by the attention mechanism (with attention coefficients $\alpha_{ij} \neq \alpha_{ij}$).
2. Equation (11) defines a specific directed graph and establishes the fundamental basis for expressing a multivariate Gaussian as a directed graph corresponding to a linear Gaussian model (Equations (11), (12), and (13)).

Furthermore, a directed graphical representation is more fitting, providing more flexible dependence measures. Additional elucidations have been incorporated in the updated version.

Q3. Indeed, the description of the distributed Gaussian process (DGP) in the paper is lacking. The conventional DGP model, as proposed by Deisenroth and Ng in 2015, operates under two fundamental assumptions. Firstly, it assumes that all GP experts share the same hyperparameters during the training phase, which serves as a mechanism to address overfitting. Secondly, it assumes perfect diversity between the target variable GPs. However, this second assumption does not align with the scope of MOGP, as interactions between GPs are a crucial aspect of the model. The paper also falls short of providing a clear explanation of how the distributed framework depicted in Figure 4 functions

The proposed GMOGP framework can seamlessly extend to distributed learning without resorting to extra approximations and assumptions, as seen in the DGP. Figure 3 illustrates the fundamental framework distinctions between the GMOGP and standard MOGPs.

In the GMOGP, each output has an independent workflow, leveraging shared interaction information from other outputs captured by kernel hyperparameters. Consequently, each output contributes specific information and amalgamates the information of dependent outputs. We have provided additional clarifications on this point and enhanced the descriptions in Figure 4 in the updated version.

I appreciate the authors for their rebuttal and revised version of the paper. They addressed my concerns regarding the clarity of the paper, in particular about GMOGP workflow and directed graph structure. Since my points have been answered, I increased my score to 8.

We sincerely appreciate the time and effort you invested in reviewing our work. Your invaluable feedback has played a pivotal role in refining and improving the quality of our paper. We address your key concerns below.

Weaknesses

W1. The authors have claimed that their framework ensures pareto optimality but there is no numerical evidence. I suggest considering an example with only 2 outputs and adding plots allowing them to validate their claim. They can consider a well-known MOO toy example where the Pareto front has been already studied.

The objective function of the proposed GMOGP is $\sum_{i=1}^S w_i\mathcal{L}^{(i)}(\Theta)$. Given that kernel hyperparameters are shared across all nodes, optimizing for the kernel hyperparameters $\Theta$ corresponds to solving a Multi-Objective Optimization (MOO) problem. Utilizing weights greater than zero in the weighted sum method satisfies the conditions for a Pareto optimal solution of $\Theta$, as detailed in the paper (Marler & Arora, 2010). The inequality constraint in the primal MOO problem serves as a valid kernel constraint.

However, it's crucial to emphasize that our weighted sum loss function still relies on parameters within the attention mechanism and noise variance. These learning parameters are not shared among outputs but are tailored to simultaneously minimize all objectives to zero. This clarification is detailed in the updated version.

W2. Limitations of the proposed framework are not discussed throughout the paper. I believe that scaling the model to large datasets and/or to systems with tens/hundreds of outputs remain a challenge. Can the authors provide further insights about the limitations of the proposed framework and potential solutions?

For large datasets, we refer to using the TGMOGP, which is learned by sparse variational inference. For tens/hundreds of outputs, one can use the distributed (T)GMOGP, but the limitation goes with the hardware.

Key limitations of the GMOGP can be mitigated through:

1. Interpretation of Attention Coefficients: Explore insights into attention coefficients learned by TGMOGP and leading graph attention networks.
2. Dynamic Dependence Measures: Adapt to changing interactions by exploring dynamic dependence measures.
3. Extensions for Online Learning and Graph Construction: Enhance real-time adaptability with online learning and dynamic graph construction extensions.
4. Tailored Attention Mechanisms/Scoring Functions: Optimize performance by tailoring attention mechanisms, scoring functions, or dependence measures.
5. Alternative Aggregation Models (Eq. 13): Explore diverse aggregation models for capturing intricate relationships.
6. Custom Objective Functions (Weights Setting): Tailor objectives and weights to diverse applications for enhanced flexibility.

These strategies present opportunities for refining and expanding the proposed model.

Questions

Q1. Although the diversity of the examples, it is not completely clear that the model properly estimated the dependencies between the outputs.

We simulated 200 samples of $f_1$ and $f_2$ from a GMOGP with $\alpha_{12} = 0.5$ and $\alpha_{21} = 0.8$, and $f_3$ from a SOGP. The kernels are SE with length scales $\theta_1 = 3$, $\theta_2 = 5$, and $\theta_3 = 0.5$ respectively. Using Adam with a learning rate of 0.08, after 500 iterations, the recovered interaction attention coefficients are $\alpha_{12} = 0.8811$, $\alpha_{21} = 0.9810$, $\alpha_{13} = 0.0086$, $\alpha_{23} = 0.0086$, $\alpha_{31} = 0.2311$, $\alpha_{32} = 0.2327$. The estimated kernel hyperparameters are $\theta_1 = 3.044$, $\theta_2 = 3.245$, and $\theta_3 = 0.295$. The results imply that output$_1$ and output$_2$ are highly correlated. However, identical estimates of the underlying parameters occur only when the kernel hyperparameters have the same values. The problem is analogous to a linear system with three variables (undefined coefficients) and only one equation, providing non-unique solutions.

We sincerely appreciate your valuable feedback! Your insights greatly contribute to enhancing the quality of our work. Additional experiments and clarification are carefully refined. We address your key concerns below.

Weaknesses

W1. Nonetheless, the work lacked of discussion, citation and experimentation of the Gaussian process models based on convolution processes, a more general and powerful approach than the Linear Model of Coregionalisation (LMC).

We included experiments for the convolutional MOGP (CMOGP (M. A. Alvarez and N. D. Lawrence, 2011)), a non-separable model, which outperforms most baseline MOGPs. The results show that the CMOGP outperforms most baseline MOGPs.

In the revised version, we have added experiments, limitations, and additional descriptions. Typos and errors have been rectified.

W2. There is not discussion in the paper about the limitations of the work.

The main limitations of the GMOGP can be mitigated through various avenues:

1. Interpretation of Attention Coefficients: The attention coefficients signify dependencies among nodes and graph structures in this paper. However, interpretations of the coefficients learned by the TGMOGP and state-of-the-art (SOTA) graph attention networks are anticipated in Explainable Artificial Intelligence (XAI).
2. Dynamic Dependence Measure: The interaction or dependence dynamically changes with inputs.
3. Extensions of Online Learning Frameworks and Graph Construction Schemes: Opportunities exist for extending the GMOGP to online learning frameworks and enhancing graph construction schemes in an online setting.
4. Adaptation of Attention Mechanism/Scoring Function/Dependence Measure for Different Tasks: Consideration of diverse attention mechanisms, scoring functions, and dependence measures tailored to specific tasks.
5. Exploration of Different Aggregation Models (Eq. (13)): Investigating alternative aggregation models, as defined in Equation (13).
6. Exploration of Different Objective Functions (Weight Settings) for Varied Applications: Exploring diverse objective functions and weight settings tailored to different applications.

Questions

Q1&Q2. In the GMOGP I only noticed a derivation for isotopic data, but I did not see any mention that the method would allow the use of heterotopic data, would the method allow its application to heterotopic data? In line with the last question, how is the behaviour of the GMOGP in the context of notable unbalanced data per output?

The proposed GMOGP can handle heterotopic data ($X^{(1)}\neq… \neq X^{(S)}$). For simplicity, we use the same notation $X$. This point has been clarified in the updated version.

In the case of unbalanced data, where different sample sizes exist, an additional weight matrix is required in the attention mechanism to map the inputs of Equation (15) to the same dimensions. This weight matrix is commonly applied in graph attention networks. Instead of the inner product measure in the scoring function (Equation (15)), other types of scoring functions such as concatenation measure can be considered for unbalanced data.

The GMOGP prior remains consistent in the context of unbalanced data, as it depends only on its own training data samples, and the kernel hyperparameters are shared to exchange information among outputs. However, it's noted that the specific information in the output with a small sample size may be inadequately captured. Descriptions addressing this point have been added in the updated version.

Q3. It is mentioned that the weights $w_i$ are set equal since there is not priority among the outputs, is this assumption still valid in the context of markable unbalanced data along the outputs?

If we aim to predict all outputs simultaneously, it is preferable to assign equal weights to the loss function for each output, regardless of the number of samples provided. In cases where certain outputs are associated with low-quality data (biased labels), consideration should be given to assigning a lower weight.

Q4. From the practitioner perspective, what would it be an appropriate way to initialise the weights $w_i$?

In our experiments, we set all weights to 1 since there is no preference for the outputs. Practical guidelines for setting weights are outlined in the paper by (Marler & Arora (2010))

Q5 ... the performance comparison with a MOGP with Convolution processes ...

Refer to W1.

Q6 ...limitations...

Refer to W2.