Multi-View Oriented GPLVM: Expressiveness and Efficiency

We thank the reviewer for the valuable feedback. We have addressed the raised questions and suggestions, with most of the changes made in the experimental section. Below, we summarize the key additions to the experiments:

1. New Datasets:
   Added experiments on more challenging multimodal datasets, including MNIST–SVHN, MOVIES, and CORA.

2. New Metrics:
   Used neural network-based classifiers (3-layer MLP) for evaluating classification accuracy.

3. Image Reconstruction Experiments:
   Reported MSE-based reconstruction results and added cross-view reconstruction tasks.

Below, we provide point-by-point responses to each of your comments.

C1: While the approach advocates for learning a model using multimodal data, the chosen experiments do not include sufficiently challenging multimodal datasets. In the (image, label) experiments, the use of multi-view techniques primarily serves as dataset augmentation since the new views are just rotations of the original images. Additionally, it is unclear to the reviewer whether the model is supposed to reconstruct the class labels for the images. Examples of more challenging multimodal datasets can be found in reference [1].

C4: Can the authors utilize neural network-based classifiers for evaluating classification accuracy results?

We thank the reviewer for the constructive feedback. To address the concerns about multimodal challenges and evaluation, we have:

• Evaluated on more challenging datasets (MNIST-SVHN [1], MOVIES, and CORA), where our method maintains strong performance (see table below);
• Added neural network-based classifiers (3-layer MLP) as an additional evaluation metric, further validating our approach (see table below);
• Included cross-view reconstruction tasks (e.g., MNIST → SVHN) to better assess representation learning. Due to image upload limitations, the visual results cannot be shown here but will be included in the revised version.

Remark: We also provide a quantitative analysis of reconstruction quality on MNIST-SVHN in our response to Comment C3.

The new datasets used here are:

• MNIST–SVHN: Two views: Paired MNIST (784-dim) and SVHN (3072-dim) digits (10 classes).

• MOVIES: Extracted from IMDb, each movie is represented by a keyword vector (1878-dim) and an actor vector (1398-dim). The dataset contains 617 movies categorized into 17 genre classes.

• CORA: Each document has two views: a bag-of-words content vector (1433-dim) and a citation structure vector (2708-dim). The dataset includes 2708 documents across 7 research topic classes.

Clarification:

• Section 4.2 evaluates classification on learned latent variables, not label reconstruction. These additions will be incorporated into the revision.

C2: Clarity. The reviewer believes that including diagrams illustrating the dependencies of the random variables, both overall and for each experimental setup, would enhance the understanding of your work. Although the authors provide experimental details in the supplementary materials, it remains unclear to the reviewer what Y represents for each dataset.

We appreciate the reviewer’s constructive suggestion. To improve clarity, we will add a probabilistic graphical model of NG-MVLVM in the revision to explicitly illustrate the dependencies between random variables.

Clarify the role of $Y$: Across all experiments, $Y$ represents the observed multi-view data (e.g., images, text, or graphs), while our model infers a unified latent representation from these inputs.

C3: For the image-based datasets, a quantitative analysis of reconstruction quality is missing. How do the different approaches compare concerning data reconstruction?

We appreciate the reviewer’s suggestions. We now include a quantitative comparison of reconstruction quality on image-based datasets, using MSE as the reconstruction metric.

From the table, our method consistently achieves lower MSE values compared to other baselines on all image-based datasets, demonstrating better reconstruction quality.

C5: Why did the authors decrease the image size of the original datasets? How does the approach adjust to larger images?

We appreciate the reviewer's question regarding image size adjustments.

To clarify, the size reduction was applied only in the single-view experiments (Appendix 4.1) when comparing with RFLVM. This was necessary because RFLVM's MCMC sampling becomes computationally prohibitive for full-size images, as noted in our paper.

However, in the main experiments (Section 4.2), we used full datasets, and all benchmarks reported in Section 4.2 can handle the full-size data without reduction.

C6: How do the latent space sizes of the different models compare? Do they have similar dimensions?

All models are evaluated with the same latent space dimensionality, which is set to 2 by default to ensure a fair comparison.

C7: For the wireless communication data, what do the ten different classes represent?

The ten classes correspond to ten different user equipments (UEs), each following a distinct trajectory. Within each class, the samples are the channel matrices observed at different time steps as the UE moves.

We acknowledge and appreciate the reviewer’s detailed feedback. Please see our responses addressing each comment below.

C1: It's hard to identify the contributions over previous work. The authors are suggested to summarize how each component differs from them.

We have summarized the comparison with previous work in Table 1 (page 2 of the paper). To highlight our contributions more clearly, we extract the key content of Table 1 below:

From this table, our contributions can be summarized as:

1. Expressiveness: We propose a Next-Gen Spectral Mixture (NG-SM) kernel that models the spectral density as a Gaussian mixture, making it a highly expressive kernel capable of capturing complex data structures. Building on this, we design NG-MVLVM, which learns a more informative shared latent representation across views.

2. Efficiency: We develop an unbiased RFF-based approximation for the NG-SM kernel that is differentiable with respect to kernel parameters, allowing us to use variational inference while reducing training cost. Combined with a two-step reparameterization trick, our approach further lowers computational complexity and enables efficient, scalable learning for multi-view data.

C2: Despite the huge amount of experiments, the paper does not fully analyze the individual contributions of the NG-SM kernel and the random Fourier feature approximation through ablation studies. The terms "kernel expressiveness", "time-varying correlations" are abstract and hard to quantify.

We appreciate the reviewer’s comment. First, we note that the NG-SM kernel and the RFF approximation target different aspects of our model: NG-SM focuses on enhancing expressiveness, aiming to learn a more informative shared latent representation across views, while RFF improves efficiency by reducing computational complexity.

Furthermore, we demonstrate the individual contributions of each component in our model through our experiments. In Section 4.2, we compare our model against baselines using various kernels (e.g., SM kernel in MV-ARFLVM, Gibbs kernel in MV-NGPLVM), where our model with NG-SM kernel consistently achieves better results, demonstrating its stronger expressiveness and improved manifold learning. Figure 2 in Section 4.2 illustrates the model fitting wall-time on the MNIST dataset as the dataset size increases, highlighting the efficiency gains achieved by the RFF-based approximation.

Finally, we will provide clearer definitions of key terms in the revision. For example:

• Kernel expressiveness: Refers to how flexible a kernel function is in modeling various data patterns. A highly expressive kernel can approximate complex, nonlinear relationships, periodic behaviors, or multi-scale structures in the data.

• Time-varying correlations: Means the similarity between two points is influenced not only by their distance, e.g., $\|x_1 - x_2\|$, but also by their specific locations $x_1$ and $x_2$ in the input domain. For example, two points with the same distance apart may have high correlation in one region but low correlation in another, depending on the local data characteristics.

C3: The claimed efficiency is not empirically validated with runtime comparisons against other MVC methods.

Thank you for your suggestion. We note that Figure 2 in Section 4.2 have presented the runtime on the MNIST dataset, where we compare our method against other multi-view baselines (e.g., MV-DGPLVM, MVAE). The results clearly demonstrate the efficiency of our model relative to the other Gaussian process-based models. And we assume that "MVC" here refers to multi-view learning methods, which include the baselines we have compared against in our experiments.

C4: Is the bound of Theorem 4 tight and does it play an important role?

Tightness: The bound in Theorem 4 is considered reasonably tight because it correctly captures the convergence behavior of the RFF approximation.

• Specifically, the term $L$ appears in the numerator of the exponent, $\exp \left(- \frac{3c^2 L}{\cdots}\right),$ which implies that the probability of the approximation error exceeding $\epsilon$ decays exponentially as the number of random features $L$ increases.
• This aligns with the result that the expected RFF approximation error decreases at a rate of $O(1/\sqrt{L})$ [1], because the exponential tail bound guarantees that if we set $\epsilon \propto 1/\sqrt{L}$, the probability of the error exceeding $\epsilon$ decreases rapidly with L, which is consistent with the same convergence rate.

Importance: The bound provides a crucial theoretical guarantee for the unbiased and differentiable RFF estimator, ensuring that the error between the RFF-based kernel and the true NG-SM kernel matrix diminishes as $L$ increases.

[1] Rahimi, A., and Recht, B. (2008). Random Features for Large-Scale Kernel Machines. In Advances in Neural Information Processing Systems (NeurIPS), 1177–1184.

C5: Are there guidences or requirements when choosing the kernel?

Our NG-SM kernel is a universal kernel, meaning it can theoretically approximate any continuous kernel function. Unlike traditional GP models, our approach does not require selecting a predefined kernel, as the kernel is learned through inference of the model parameters. This is particularly advantageous since, as suggested by the question, the ``correct'' kernel is often unknown a priori.

By learning the kernel directly from the data, our method can adaptively capture the underlying structure and dependencies without manual specification. As demonstrated in our experiments, this flexibility allows the NG-SM kernel to model diverse and complex data patterns effectively. Thus, we recommend the NG-SM kernel as a default choice in most applications where kernel flexibility is desired.

C6: The full phrase of the abbreviation RFF should appear at its first use on line 48

Thank you for pointing this out. We have revised the manuscript to include the full phrase “Random Fourier Features (RFF)” at its first occurrence on line 48.

We thank the reviewer for the invaluable feedback and comments. Below are our detailed responses.

C1: While this paper claims that the proposed NG-SM kernel is capable of approximating any continuous kernel via Bochner duality and Gaussian mixture spectral density modeling, the theoretical guarantee is only loosely discussed.

Thank you for raising this point and we agree that a rigorous proof is necessary. Accordingly, we have included a complete proof in the revised paper and have provided a brief sketch below for your reference.

Step 1:

As established by our universal Bochner’s theorem, any continuous kernel $k_{t}(x_1,x_2)$ can be represented as:

$$k_{t}(x_1, x_2) = \frac{1}{4} \int \Gamma(x_1, x_2; w_1, w_2) \, p_{t}(w_1, w_2) \, dw_1 dw_2,$$

where $p_{t}(w_1,w_2)$ is the spectral density of the target kernel and $\Gamma$ is a bounded sum of exponential terms.

Step 2:

It is known that Gaussian mixtures are dense in $L_1(\mathbb{R}^{2d})$ [1]. Thus, for any $\varepsilon > 0$, there exists a $Q$-component Gaussian mixture $p_{ngsm}$ such that

$$\| p_{ngsm} - p_{t} \|_{L_1} < \varepsilon,$$

provided $Q$ is sufficiently large.

Step 3:

We define the approximate kernel:

$$k_{ngsm}(x_1, x_2) = \frac{1}{4} \int \Gamma(x_1, x_2; w_1, w_2) \, p_{ngsm}(w_1, w_2) \, dw_1 dw_2.$$

The difference between the target and approximated kernels is:

$$k_{t}(x_1, x_2) - k_{ngsm}(x_1, x_2) = \frac{1}{4} \int \Gamma(x_1, x_2; w_1, w_2) \, [p_{t}(w_1, w_2) - p_{ngsm}(w_1, w_2)] \, dw_1 dw_2.$$

Taking the absolute value and applying the triangle inequality:

$$|k_{t}(x_1, x_2) - k_{ngsm}(x_1, x_2)| \le \frac{1}{4} \int |\Gamma(x_1, x_2; w_1, w_2)| \, |p_{t} - p_{ngsm}| \, dw_1 dw_2.$$

Since each exponential term in $\Gamma$ has modulus 1 (i.e., $|e^{i\theta}| = 1$), and $\Gamma$ is a sum of 4 such terms, we have $|\Gamma(x_1, x_2; w_1, w_2)| \le 4$. Thus,

$$|k_{t}(x_1, x_2) - k_{ngsm}(x_1, x_2)| \le 1/4 \cdot 4 \| p_{t} - p_{ngsm} \|_{L_1}$$

This implies that the kernel approximation error is bounded by the $L_1$ distance between $p_t$ and $p_{ngsm}$, uniformly over all input $(x_1, x_2)$.

Conclusion

As $Q \to \infty$, $p_{ngsm} \to p_{t}$ in $L_1$, implying $k_{ngsm} \to k_{t}$ uniformly. Therefore, NG-SM can approximate any continuous kernel arbitrarily well.

[1] N. Kostantinos. Gaussian mixtures and their applications to signal processing. In Advanced Signal Processing Handbook: Theory and Implementation for Radar, Sonar, and Medical Imaging Real Time Systems, 2000.

C2: Although the method aims to learn informative shared latent variables across views, the paper does not offer enough interpretability or visualization analysis (e.g., t-SNE, clustering structure, or downstream task results).

We agree that visualization and interpretability analysis is important. Thus, we already included both qualitative and quantitative evidence in the original submission: Section 4.1 (Figure 1) visualizes the learned 2D latent variables and compares them with the ground-truth S-curve; Sections 4.2 and 4.3 evaluate downstream tasks using KNN, SVM, and other classifiers on multiple datasets.

Following your suggestion, we further visualize the clustering structure of the learned representation to enhance interpretability. Since the latent space is 2D, we directly visualize it for MNIST and CIFAR, and observe that the clearer cluster structure corresponds to higher classification accuracy - consistent with the results in Section 4.2.

C3: The paper mentions that the complexity is reduced to O(NL²) using RFF-based kernel approximation, but it lacks empirical runtime comparisons with other methods.

We appreciate the reviewer’s suggestion. As shown in Figure 2 (Section 4.2), we have included runtime evaluations on the MNIST dataset, demonstrating the efficiency of our RFF-based kernel approximation. However, since the analysis is embedded within the overall performance discussion and Figure 2 is not explicitly referenced in the runtime-related text, this result may appear unclear. To address this, we will revise the experimental section to explicitly refer to Figure 2 and better highlight the runtime advantage in practice.