A Functional Extension of Semi-Structured Networks

We thank the reviewer for the thoughtful and detailed comments. Below we address the mentioned weaknesses and questions.

Weaknesses

[...] interpretability of the proposed [...] not demonstrated in their experiments

We agree that this is an important part of our work. We would like to point out that we have included result interpretations in Figures 1 and 3, and presented the estimated weight surfaces in Figure 7 in the Appendix, accompanied by a brief explanation. While placing Figure 7 with interpretability results in the Appendix might seem less ideal, it is essential to note that interpreting the estimated weight surfaces of function-on-function regression models is inherently complex. This complexity is further compounded by the need for domain knowledge in biomechanics to fully grasp the results. Therefore, we chose to provide Figure 1 with a toy example and place Figure 7 in the Appendix. Nonetheless, we can confirm that the results are highly plausible from a biomechanical perspective and offer valuable insights into the relationships between accelerations and moments in the analyzed joints.

Besides, the FFR $\lambda^+$ can only explain the linear part of the target function $y$, but most of the time, we hope to explain the nonlinear interaction in the neural network.

We agree with the reviewer that nonlinearities are not of lesser importance. However, we would like to point out that, although the $\lambda^+$ part of the model is linear in the coefficients of the basis functions, this implies non-linearity in the estimated function-on-function effects. Figures 1, 3, and 7 clearly illustrate this non-linearity, where certain combinations of input and output signals change nonlinearly over the two different time domains. In addition, while it is possible to include interactions between input signals in $\lambda^+$, this increases the complexity of interpretation. We believe it is up to the modeler to decide the appropriate order of interactions to assess in the structured part and what effects to move into the black-box mechanics of the deep network.

Questions

Q1: [...] Why do we not add the residual design to the FNN directly?

We thank the reviewer for this interesting question. While we agree that adding a residual connection directly into the FNN would also improve performance compared to the deep-only network, stacking multiple residual connections would complicate the interpretation. In such a case, we would need to understand the total effect of the structured (linear) part on the outcome in this residual network and, even more challenging, find a way to orthogonalize such a model. Our current proposal addresses this issue clearly, as there is a direct connection from the input to the output that is linear in the coefficients.

Q2: Under the discretization setting, what are the differences between time series modeling and functional networks?

This is an excellent question that often arises in functional data analysis (FDA). The primary difference between time series modeling (TSM) and FDA is that, in FDA, the curves or “time series” are observed repeatedly and are viewed as replications of the same underlying process (e.g., acceleration curves across the time of one stride). In contrast, TSM typically deals with time series that do not have replications (we cannot observe today’s stock prices multiple times with different noise processes; we only observe each stock price time series once). In FDA, we further predict entire curves on the same domain as the training data, while TSM often focuses on forecasting specific future values. Additionally, FDA usually assumes the smoothness of the process (acceleration profiles, at least theoretically, do not have discontinuities) and is concerned with the shape of the curve (e.g., the curvature and course of the signal over time). In TSM, the shapes of the time series are often not the primary focus.

[...] the authors have not stated the GPU specification in Appendix D.2

All models were trained on a CPU; we did not use any GPUs.

Please let us know whether we have addressed all weaknesses as well as questions.

Dear Reviewer Ta2r,

Thank you for reading and acknowledging our rebuttal. We would like to briefly comment on the statement

model interpretability requires domain knowledge in biomechanics to perceive, which refrained the general audience from intuitively understanding the power of the model.

While it is true that the interpretability of the results requires domain knowledge (which applies to every field, not just biomechanics), the interpretability of the model itself does not. The coefficient surfaces estimated by our model can be interpreted like any other bivariate effect (similar to those in spatial applications --- however, instead of latitude and longitude, we provide a relationship between $y(t)$ and $x(s)$).

We assume that this is what the reviewer meant, but we wanted to clarify this point to ensure nothing was left unaddressed while we still have the opportunity to respond to comments.

Thank you again for your time reading our paper and providing the review. Please let us know if there are any remaining concerns that we should address.

We thank the reviewer for the thoughtful comments and pointing out ways to improve our manuscript. Below we address the mentioned weaknesses and questions.

Weaknesses

Can the authors comment on whether this component is original to their work [...]

We thank the reviewer for bringing this up. While a functional autoencoder has been proposed in the past (e.g., Hsieh et al., 2021) and most recently by Wu et al. (2024), these studies focus on the representation learning (and reconstruction) of the same signal. Other papers focusing on function-on-function regression (e.g., Luo and Qi, 2024; Rao and Reimherr, 2023; Wang et al., 2020) suggest similar approaches to ours but without the option to jointly train a structured model and a deep network.

We also believe that our idea to encode and decode signals using a general basis representation is to some extent novel. However, any mapping from function to scalar values and back (such as the functional PCA in Wang et al., 2020) can potentially be interpreted as such an encoding strategy. We will clarify this distinction and our contribution, and we thank the reviewer for highlighting this point.

Questions

[...] choice of basis functions for encoder and decoder. These should certainly be specified [...]

We thank the reviewer for pointing this out. In our experiments, we used thin plate regression splines but also obtained similar results with B-splines of order three and first-order difference penalties. We will provide this information in an updated version and will add a short discussion of possible other options with their pros and cons.

Fig2a and 2b present two different potential options for constructing this semi-structured model, might it be worth evaluating the performance of structure a)? Would we expect the performance to be similar, what are the drawbacks?

This is indeed an important question. We have included such a comparison in Appendix C in the original version of the paper, where we “compare the two different architectures suggested in Fig. 2(a) and Fig. 2(b)”. Using different real-world datasets, we found that both architectures perform quite similarly. In practice, our biomechanics project partners might be in favor of option b) as it allows them to use an established network for the deep model part and gives more flexibility, whereas from a practical point of view, option a) will be potentially more parameter-sparse and allows to jointly learn the function embedding, which in turn increases interpretability.

When fitting the parameters of the linear model, it is mentioned the large computational cost. I did not find the explanation of how the scalable implementation circumvents this to be particularly clear.

We thank the reviewer for this question. In Section 3.4, we have provided an explanation of how our implementation solves this problem with a focus on the space complexity (memory costs). In particular,

• memory costs are reduced by using mini-batch training and not having to cast our model into long-format as done by classical approaches. Our approach could indeed then be combined with the SVRG approach by Johnson and Zhang as the reviewer suggested.
• In contrast to classical implementations, we rely on array computations (as e.g. given in Equation 8 in the paper). Using the array model formulation, we never have to explicitly construct the Kronecker product of the basis matrices, which saves additional memory.
• In addition to the two previous solutions, we recycle the basis functions in the s-direction, which is non-trivial for other software implementations. In the neural network, however, this “simply” boils down to having multiple connections to the same matrix object in the computational graph.

References

Hsieh, T. Y., Sun, Y., Wang, S., & Honavar, V. (2021). Functional autoencoders for functional data representation learning. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM) (pp. 666-674). Society for Industrial and Applied Mathematics.

Luo, R., & Qi, X. (2024). General Nonlinear Function-on-Function Regression via Functional Universal Approximation. Journal of Computational and Graphical Statistics, 33(2), 578-587.

Rao, A.R. and Reimherr, M., 2023. Modern non-linear function-on-function regression. Statistics and Computing, 33(6), p.130.

Wang, Q., Wang, H., Gupta, C., Rao, A.R. and Khorasgani, H., 2020, December. A non-linear function-on-function model for regression with time series data. In 2020 IEEE International Conference on Big Data (Big Data) (pp. 232-239). IEEE.

Wu, S., Beaulac, C. and Cao, J., 2024. Functional Autoencoder for Smoothing and Representation Learning. arXiv preprint arXiv:2401.09499.

We thank the reviewer for the detailed and thorough review of our manuscript.

Weaknesses

improvement [...] rather mediocre [...] would be appreciated if error bars were reported

We only report results for a single train/test split as the application fixes this. To still provide a measure of uncertainty, we show individual joint performances in Fig. 5, and Tab. 2 summarizes findings with standard dev., showing a significant improvement for semi-structured models. However, it's not only about performance improvement but also being able to quantify the explained proportions by the structured and deep part.

I cannot find a substantial contribution to the [ML/DL] community [...] better be suited to a [biomechanics] journal

We thank the reviewer for the comment but politely disagree. 1) The paper is too technical for a biomechanics journal (a manuscript focusing on interpretation will be submitted to such a journal). 2) We believe that projects with a motivating application should be valued at least as much as those proposing methods without real applications. 3) We also think that we provide a substantial contribution to the ML/DL community as functional data is well represented at NeurIPS (e.g., Boschi et al., 2021; Gonschorek et al., 2021; Madrid Padilla et al., 2022), ICML (e.g. Yao et al., 2021; Heinrichs et al., 2023), etc., and our paper advances both orthogonalization and semi-structured models, also discussed in these communities (e.g., Lu et al., 2021; Vento et al., 2022; Rügamer et al., 2023, 2024).

Claim in l265 not supported with experimental results. Please provide error and dispersion metrics

We provide these metrics as boxplots in Fig. 6 (Appendix).

Questions

What are the additive structures [...] provide some examples

We wrote: “For example, neural additive models [...] such as generalized additive models”. We will revise the text to make these examples more clear.

proportional contribution of $\lambda^+$ and $\lambda^-$ [...] How [..] $\lambda^-$ is not solving the entire problem

The proportional contribution will depend on the application. $\lambda^-$ is not solving the entire problem due to the orthogonalization (see answer below).

What are the integration weights $\Xi$ concretely [...] can they be dropped

The integration weights are used to approximate the integral (trapezoidal Riemann weights in our paper). For time points on an equidistant grid, one could potentially drop them.

intuition what the orthogonalization implements [...] Is it some kind of regularization

The orthogonalization process subtracts everything that could have been explained through $\lambda^+$ from $\lambda^-$ and adds it to $\lambda^+$, ensuring that the two parts are orthogonal. This concept can be understood by analogy to applying multiple regression models. First, generate predictions by regressing the actual outcome on the features via $\lambda^+$ and $\lambda^-$. Next, regress these predictions on $\lambda^+$ to determine the portion that can be explained by $\lambda^+$. The residual term, representing what cannot be explained by the structured part, will be orthogonal to this part. The orthogonalization employs this approach using appropriate projection matrices and can be applied whenever the structured part is linear in the coefficients. While it may not be straightforward to see how this works for our model, Section 3.3 demonstrates how to do this using vector operations. It is not a regularization in the classical sense, as it exactly enforces orthogonality between the two parts.

Relating to Figure 4, how does runtime compare for the three implementations for $\lambda^-$?

Figure 4 investigates whether “without the deep part, [...] can [we] recover complex simulated function-on-function relationships [...] while scaling better than existing approaches”. Hence, no deep part was included in this experiment. Also, neither the boosting nor the additive model approach would allow the incorporation of the $\lambda^-$ part.

How does the memory compare if the other methods are implemented in a batched fashion

This would result in improved memory consumption w.r.t. number of observations, however, not w.r.t. the number of functional predictors.

intuition of why the neural network performs worse in high SNR regimes?

Full-batch optimization (additive model, boosting) is beneficial when there is less noise in the data (and vice versa, the neural network's stochastic optimization induces regularization and might thereby better deal with noise).

Could you include the results from reference [24]

As [24] used a different processing of functional predictors, we first had to adapt their code. We then ran their best model once using a deep-only variant and once using a semi-structured model. RelRMSE results are as follows:

References

• Boschi et al., 2021. A highly-efficient group [...] NeurIPS.
• Gonschorek et al., 2021. Removing inter-experimental [...] NeurIPS.
• Heinrichs et al., 2023, Functional Neural Networks [...] ICML.
• Lu et al., 2021. Metadata normalization. CVPR.
• Madrid Padilla et al., 2022. Change-point detection [...] NeurIPS.
• Rügamer, 2023. A new PHO-rmula [...] ICML.
• Rügamer, et al., 2024. Generalizing Orthogonalization [...] ICML.
• Vento et al., 2022. A penalty approach [...] MICCAI.
• Yao et al., 2021. Deep learning for functional [...] ICML.

Please let us know whether we have addressed all points and whether these answers are sufficient to re-evaluate your score.

The reviewer raises two points that we answer below. We thank the reviewer for these thoughtful comments, but in both cases would like to point out that these "weaknesses" have already been addressed in our original submission.

How much tuning has been done for the pure deep-network baseline? I would be a bit worried that the findings are very dependent on the choice of architecture of this model?

We agree that this is an important aspect that needs to be taken into account. Fortunately, some already-tuned architectures are widely adopted in biomechanics. As we write in the beginning of Section 4 “we use a tuned InceptionTime [16] architecture from the biomechanics literature [ 24]. As FFR and boosting provide an automatic mechanism for smoothing, no additional tuning is required.”. In other words, we don’t tune any of the models. While tuning might certainly benefit one or the other model, we think that it is difficult to make a perfectly fair comparison between all methods. In this light, we adopt a strategy that is closest to what practitioners in the field would likely do — use the tuned Inceptionnet with the hyperparameters found to be optimal in previous studies and take this architecture without further modifications.

We will make this more clear in a revised version of the manuscript.

The presentation could be made clearer. E.g. when $\lambda^+$ and $\lambda^-$ are introduced, they should be more clearly defined.

We thank the reviewer for this comment. We have defined $\lambda^+$ in the paragraph right after it was introduced ($\lambda^+(t) = \sum_{j=0}^J \lambda_j^+(t)$ with $\lambda_j^+(t) = \int_{\mathcal{S}_j} w_j(s,t) x_j(s)ds)$ and in the following paragraph (“Deep model part”) discuss choices for $\lambda^-(t)$. We are happy to include further details if there is anything else the reviewer is missing.

Please let us know whether we have addressed all weaknesses and whether these answers are sufficient to re-evaluate your score.

Hi, please see my updated review which hopefully makes it clearer what needs to be done for me to recommend acceptance

Dear Reviewer DyRG,

Thank you very much for reading our rebuttal and your prompt response. We also appreciate the revised review, which will help us to improve the clarity of our paper. Below we respond to the raised questions point-by-point:

How much tuning has been done for the pure deep-only network baseline from Section 4.2.2?

That is a very good question, and we can understand your concerns well. However, note that our previous response also applies to the application in Section 4.2. The quoted sentence “we use a tuned InceptionTime [16] architecture from the biomechanics literature [24]” is given in Section 4 and applies to both subsections. In other words, we did not tune the deep network part (for both the deep-only and the semi-structured model). We will make this more clear in a revised version of the manuscript.

I would be a bit worried that the findings are very dependent on the choice of architecture of this model? E.g. does it have any increased capacity over the deep part of the semi-structured network?

This is another good question that the reviewer raises. Note, however, that the deep network is the same for the “deep-only” model and the semi-structured network.

And does this architecture have suitable inductive biases for fitting linear functions (maybe residual connections which add a linear transformation of the input to the output)?

Good point. We can confirm that the architecture is suitable for fitting linear functions. A large part of the variation in these applications is explained by the (linear) function-on-function regression, which the InceptionTime model can also capture. This is also apparent from the fact that the orthogonalization has a large impact on the explained relationship (Figure 3), from predictions in Figure 8 in the Appendix and further confirmed in [24], where the deep network can represent a similar function space as the function-on-function regression (at least for this specific application).

We will make these points more clear in a revised version and thank the reviewer again for bringing up these points.

The presentation could be made clearer in a couple of areas. In particular, (1) Equation 3 is the first time that the “double function call” notation of e.g. $\lambda^-(X)(t)$ is used. Having a double function call makes sense given that there is a function which takes X as input and outputs another function. However it is not clear to me why this is then not used in Equation 1, where you could presumably replace $\mu(t)$ with $\mu(X)(t)$?

We thank the reviewer for pointing this out. The reviewer is absolutely correct and $\mu$ is also a double function with the first argument being the data $X$. In line 101, we defined $\mu$, writing “An FFR for the expected outcome $µ(t) := \mathbb{E}(Y(t)|X=x)$" and explicitly dropped the dependence (by considering $X$ to be fixed with realization $x$). We agree, however, that this might lead to confusion and will add the second argument where appropriate to be consistent.

And e.g. in the last part of Section 2.2, why write $h^{(L)}(t)$ instead of $h^{(L)}(X)(t)$? Would it be possible to make this clearer?

Again, the reviewer is correct. We simply tried not to overload the presentation but agree that this should be consistent and will update the notation. Thank you for pointing this out.

And also (2), when functional data analysis is introduced in Section 1 and 2, it would be helpful if an example application was described

We agree and will add an example from the field of biomechanics (predicting ground-reaction forces with acceleration curves) and neuroscience (predicting muscle movements from brain signals).

Dear Reviewer DyRG,

Thank you for following up on our response to your revised rebuttal. We appreciate that you have raised the score above the acceptance threshold and clearly outlined what was required from us to earn your recommendation for acceptance. We thank you again for your suggestions to change the notation, add details on experimental details, and include an illustrative example. We will incorporate these into the revised manuscript, which we believe will further strengthen our paper.