Functional Bayesian Tucker Decomposition for Continuous-indexed Tensor Data

We greatly thank and appreciate the reviewer's time and effort in offering insightful and detailed comments and reviews. We address the comments below:( C: comments; R: responses)

C1: "not enough emphasis on related similar work BCTT (Fang, et al. Bayesian Continuous-Time Tucker Decomposition, ICML 2022)", "Missing seminal work: "Bayesian Tensor Regression" by [Guhaniyogi et al., JMLR 2017]"

R1: Good point. We acknowledge the use of similar techniques BCTT, specifically the utilization of state-space Gaussian Processes (GP) to model the latent dynamics in Tucker decomposition. As this similarity has been noted in short in the related works section of our paper, we do plan to follow your suggestion and highlight this more prominently in subsequent versions on their significant differences in formulation and inference. Please refer to the response (R1) to reviewer MPRT for more detailed elaborations. For the "Bayesian Tensor Regression"[Guhaniyogi et al., JMLR 2017], we will add it in the citations and related works section.

C2: "For the synthetic experiments, it would be useful to see the tensor reconstruction looks as a function of the number of samples?"

R2: Good suggestion! We add reconstruction RMSE with different numbers of samples, here are the results(we generate 1300 samples on the surface of the 3D function, and randomly select parts of samples with Gaussian noise (STD=0.02) for training, evaluate on the remaining samples:):

As we can see, the RMSE decreases with the increase of the number of training samples. With 650 samples, the RMSE is less than 0.03, which is similar to the noise level and can be seen as a good reconstruction. We will add the results in the latest version.

C3: explain if non-uniform core shapes were explored in the Beijing Air experiments with "Approximately Optimal Core Shapes for Tensor Decompositions" by [Ghadiri et al., ICML 2023]

R3: Thanks for raising the great reference! We will add the discussion on non-uniform core in our experiments with the latest work [Ghadiri et al., ICML 2023].

C4: "Is Tucker decomposition really a more compact low-rank representation?"

R4: Good point! We do agree that CP is more compact and that statement may be not accurate enough. We will polish it in the latest version. For a more detailed discussion on the compactness between Tucker CP and TT, please refer to the response (R4) to reviewer MPRT.

C5: By "successive derivatives of m-th order", do you instead mean to write $d^m f(x)/ dx^m$?

R5: Yes, we mean $d^m f(x)/ dx^m$. We will polish it to make it more clear.

C6: "The trained model cannot handle new objects with never-seen indices" -- this is not necessarily true, and is the core idea behind tensor completion.""

R6: Our statement was intended to highlight that traditional tensor models typically rely on grid-structured data characterized by discrete indices, and thus may struggle with grid-free interpolation for new objects identified by real-valued indices. We apology for any confusion caused and will revise our wording to clarify this point more effectively.

C7: "why use $\tau^{-1}$ instead of $\sigma^2$ for the Gaussian noise?"

R7: Good question. In the Bayesian framework we employ, the most common modeling approach for Gaussian noise involves assuming that its inverse: $\tau$ —also known as 'precision'— follows a Gamma distribution(see statement under equation 11). This convention is widely used in Bayesian analysis for Gaussian likelihood with observation noise."

C8: Is the preset (scalar) mode rank $R$ for a given Tucker core of shape $(r_1, \ldots, r_K)$ the product? I.e., $R = \prod_{k=1}^K r_k$?

R8: No, actually the preset (scalar) mode rank $R$ is rank for each mode and the shape of core is $(R, R, \ldots, R)$. We will add the clarification in the latest version. For the detailed statement for linear cost with rank $R$, please refer to the response (R4) for reviewer BN7i

C9: Typos and suggestions

R9: Thanks for the correction! We do greatly appreciate the reviewer's time and effort in offering such detailed help. We will fix them in the latest version.

I have read the author response and am willing to increase my rating from 3 to 5.

We thank you so much for reading our response and increasing the score!

We thank the reviewer for the careful review! We address the comments below:( C: comments; R: responses)

C1: "Relation and Differences Between Functional Tensor Models and Traditional Regression Tasks and Models [1,2,3,4,5].

R1: We are grateful to the reviewer for posing such an insightful question and for the excellent references provided, particularly CPNN [5]. We will certainly cite these and expand our discussion on this topic in the latest version of our manuscript.

A brief and high-level response is that, in addition to learning a good mapping from indexes to values—the focus of traditional regression tasks and models—functional tensor models emphasize learning robust representations of latent dynamics. This approach not only facilitates more flexible and robust prediction but also enhances interpretability and utility for downstream tasks.

Consider the US-TEMP dataset as an illustrative example. Each observation entry represents a temperature value corresponding to specific spatial-temporal indexes, such as (time, latitude, longitude). This is a quintessential spatial-temporal dataset. Various model types, ranging from simple linear regression to advanced deep models like CNNs, GNNs, VAEs, and diffusion models, can handle this task. However, these models are typically black-box in nature, offering limited insight into latent dynamics and lacking flexibility for potential downstream tasks. For instance, predicting temperature at a new location with additional features, like altitude, would typically require retraining the model from scratch.

Conversely, our proposed method is capable of learning latent dynamics and providing a low-rank, dynamic representation of key factors within the data. As demonstrated in Figure 2, the learned latent dynamics not only yield accurate predictions but also help elucidate the evolving processes underlying the data. We further emphasize the potential of these representations for downstream tasks. Taking the example of incorporating altitude as a new feature, our method allows for the addition of a new mode to the tensor. We can reuse the learned representations of existing features and only train the new mode dynamics. These reusable representations are crucial for achieving more flexible and general modeling capabilities."

C2: Compare with some nonlinear or GP-based tensor decompositions and continuous-time tensor decompositions.

R2: Good suggestion! We add the comparison with GP-based tensor decomposition: GPTF(Zhe, et al. Distributed flexible nonlinear tensor factorization, 2016) and continuous-time tensor decomposition: BCTT(Fang, et al.Bayesian Continuous-Time Tucker Decomposition, 2022) on the three datasets of BeijingAir. The mean of RMSE over 5 runs are shown:

As we can see, for every rank, FunBaT is with the best performance. The BCTT gets the second-best performance, as it can model the continuous dynamics for time mode, but not for other modes. The GPTF is with the worst performance. The reason is that FunBaT is a with-nature model for functional tensor data, as we discussed in C1-R1.

C3: Why the construction of Tucker is helpful compared with GP regression?

Good question! We claim that our model actually wraps multiple GP regressions with a tensor structure(Tucker or CP). In the case of a $K$-mode tensor with rank $R$, our approach involves managing $K \times R$ separate GP regressions. This is in stark contrast to a conventional GP regression, which typically relies on a single kernel to capture the entirety of multi-view data patterns. The tensor-based design splits the high-order data and multi-view interactions into low-rank, distinct subspaces. This decomposition significantly simplifies the learning process for each individual GP. The concept of decomposing and representing complex, high-dimensional data in a low-rank format is a broadly adopted strategy in the field of machine learning."

C4: Why is the time complexity linear with the rank $R$?

R4: Great point! The state-space GP is assigned to each dimension of the mode independently (equation 7). Thus, for mode with rank $R$, we will have $R$ separate state-space GP, and each GP's state dimension—determining the matrix mat/inverse cost in KF/RTS— is only related to kernel-decided order $m$ ($m=1$ for $Matern-3/2$ kernel, $m=2$ for $Matern-5/2$ kernel). During the inference over specific mode, we run the KF and RTS over the $R$ separate state-space dynamics, saying, a loop over $R$ to run KF and RTS. Thus, the time complexity is linear with the rank $R$.

We do understand the confusion is that we use the compact notation $Z^k$ to refer the concatenated states of all $R$ dimensions. We will add the clarification in the latest version and thanks again for pointing out this point.

C5: Why not put a multi-resolution setting for US-TEMP?

R5: Good point! We do not add the multi-resolution setting for US-TEMP is because the data's original size is relatively small: (15 latitudes, 95 longitudes, 267 timestamps), compared with BeijingAir, whose size is (428 atmospheric pressure, 501 temperature, 1461 timestamps). Discretization will make the data even smaller, and not suitable for real-world applications. We will add the discussion on this in the latest version.

C6: Typo in the paragraph above Figure 1

R6: Thanks for the correction! We will fix it in the latest version.

We thank reviewer MPRT for their careful review! We address the comments below:( C: comments; R: responses)

C1: “FunBAT modeling approach is very similar to that of BCTT (Bayesian Continuous-Time Tucker Decomposition, ICML 2022)” , What is the primary innovation of FunBAT? What's the deeper level of differences between FunBAT and BCTT.

R1: Good point! We acknowledge the use of similar techniques BCTT, specifically the utilization of state-space Gaussian Processes (GP) to model the latent dynamics in Tucker decomposition. This similarity has been noted in short in the related works section of our paper, and we plan to highlight this more prominently in subsequent versions on their differences:

• Differences in target scenarios and formulations: BCTT focuses on time-series tensor data, whereas ,FunBAT is centered on functional tensor data. This difference in application leads to distinct challenges and modeling: BCTT involves one Tucker-core dynamic with static factors, FunBAT employs a static core with groups of mode-wise dynamics. This fundamental difference in formulation leads to varied inferences.

• Differences in Inference Algorithms: Due to the divergent formulations, the inference algorithms between the two methods show significant differences. For each observation, BCTT needs to infer only one state of the single temporal dynamic. In contrast, FunBaT requires inferring multiple states of multiple dynamics, depending on each mode's index of the observation. which is more challenging. This results in distinctly different procedures in terms of message approximation, passing, and merging. Specifically, in BCTT, all Tucker core time-varying functions share the same timestamp, and then can be treated as one temporal dynamic by stacking. Thus, we only need to approximate the message once, and then the dynamic inference is in a synchronous manner. However, in FunBaT, we will run a loop over tensor mode to do mode-wise conditional moment matching, and then get the messages factors fed to different functions. The inferece of multiple dynamics is in a asynchronous manner.

C2: Existing functional Tucker decomposition works e.g., [1,2]

R2: Thanks for the excellent reference. We will modify our claim. We will add them in citations and correct the statements.

C3: What are the advantages of this function tensor decomposition method compared to existing approaches? What's the primary motivation of this paper? Is it only because "there is no functional Trucker?"

R3: Great question! The main advantage of FunBAT is that it's a scalable and flexible Bayesian model, with inherent abilities for robust modeling and probabilistic inference. For example, it can offer uncertainty-aware prediction at any index (equation 17). However, most existing works on functional tensor taking deterministic approximation, saying, the weighted sum of basis functions(FTT series, [1]), or MLP ([2]), can not do this.

The Bayesian methods, featured for robustness and uncertainty quantification, are underused in this field. The only work in the early year [Schmidt,2009] that bridged the Bayesian method and function tensor, is not scalable and limited to CP. Thus, as there is no scable Bayesian method for functional tensor, we design FunBAT by utilizing advanced Bayesian learning tools, like CEP[Wang,2019] and state-space GP[Solin,2016]. That is our primary innovation and motivation.

We also want to claim the proposed Bayesian framework is not only limited for Tucker, it also supports CP(section 4.2). We highlight the "Functional Tucker" as Tucker formulation is more challenging. Furthermore, we believe the proposed framework can be extended to TT too, which will form a general scalable Bayesian framework fitting various decomposition methods. We will explore more in-depth in this direction in our future work.

C4: Why does the Tucker decomposition lead to a more compact and flexible low-rank representation compared with TT and CP?

R4: Good point! In our view, compared with CP, Tucker is with stronger capacity as it models all possible interactions. Compared to TT, Tucker's rank setting is more flexible, as there is no constraint for the mode rank. However, TT sets a chain of matrix multiplication to get tensor entry, constraining the size(rank) of adjacent mode to fit the valid matrix multiplication. Moreover, the compact core tensor of Tucker can provide insights of the interpretability of which interactions are crucial, it will help people better understand the data. On the other side, we acknowledge that the statement may be not accurate enough and ambiguous, and will polish it in the latest version.

We thank reviewer wp4E for the valuable comments and support! We address the comments below:( C: comments; R: responses)

C1: Explanation of MSE of the proposed method much lower than the baselines in Table 1.

R1: Good question! For the significant performence gain of the proposed method in Table 1, we think the main reasons are:

1. The three pollution tensor datasets have inherent and strong mode-wise continuity, as their modes are : (pressure, temperature, time). The proposed method can capture the mode-wise continuity by the mode-wise dynamics. However, the baselines, like all static tensor methods, can not model that fact.
2. The observations of the three datasets are sparse. The observation ratio is less than 0.001%. The FTT series baseline, with too many basis functions, is easy to get overfitting issue, as it is with The proposed method is with robust modeling, as it is with probabilistic inference. The Bayesian framework can be seen as a regularization to avoid overfitting.

C2: Equation 4 implies the smoothness of z(x)?

R2: "Good point! As equation 4 indicates, z(x) is a Linear Time-Invariant (LTI) Stochastic Differential Equation (SDE), so we cannot say it's smooth under the strict mathematical definition. In fact, we cannot label any SDE or stochastic process as smooth, as stochastic processes can be unbounded. However, we can state that z(x) is stationary. Given that z(x) is an LTI SDE, its mean and autocovariance function are time-invariant. Generally speaking, with its stationary characteristics, the curve of z(x) is unlikely to exhibit significant abrupt changes, meaning that its curve tends to be relatively smooth.

C3: Discussion on the computational complexity over rank for CP and Tucker.

R3: Good point! We do agree the trade-off between the efficiency of CP and the capacity of Tucker is tricky and crucial for real-world applications. We will add a discussion on this in the future version.