SAND: Smooth imputation of sparse and noisy functional data with Transformer networks

We thank the reviewer for the valuable and positive feedback.

We thank the reviewer for the valuable and positive feedback, which we have used to strengthen our paper. Please see our responses below.

[Weakness 1]

Answer: Thank you for the comment. We acknowledge that most of our theorems focus on SAND. However, our theorem does reveal one reason that SAND is better than the standard Transformer: See Corollary 1 and lines 223 to 226 of the manuscript, which explain how SAND effectively separates the smooth underlying process from random noise by limiting hidden nodes in the network. The standard Transformer is incapable of doing so due to its universal approximation property and the existence of noise in the training data (lines 149 to 154 in the manuscript). When the standard Transformer encounters noisy testing data, they often continue producing similar zigzag patterns, as they have learned to replicate the noise from the training phase.

[Weakness 2]

Answer: The traditional numerical derivative operator takes the form $f^\prime(a) \approx \frac{1}{b - a}[f(b) - f(a)]$, a linear combination of the inputs $f(a)$ and $f(b)$ with fixed weights (line 195). Our $\rm{diff}(\cdot)$ operator similarly calculates a weighted sum of these inputs, with weights learned by the network. Our decision to design the $\rm{diff}(\cdot)$ operator to resemble the attention module was driven by two main considerations:

• Cohesion with the Transformer Architecture: Our SAND module is a simple augmentation of the standard Transformer. SAND modifies the conventional attention module by omitting the softmax operation, thereby aligning its computational complexity with that of the original attention mechanism, as detailed in Table 1 of Vaswani et al. (2017). Compared to recurrent and convolutional networks (other architectural techniques for achieving smoothing), SAND enjoys major computational advantages (Vaswani et al., 2017).
• Leveraging Proven Mechanisms: The attention mechanism has demonstrated effectiveness across numerous tasks. By minimally adapting this component, we aim to preserve this versatility in our operator. The $\rm{diff}(\cdot)$ operator retains the core benefits of attention while extending its application to approximate derivatives and enforce smoothness.

[Weakness 3]

Answer: We have indeed considered the approach of post-processing imputations from a standard Transformer. As discussed in lines 164 to 168 and in Table 1 under the rows labeled “GT1P”, “GT1S”, “GT2P”, and “GT2S”, we applied both PACE and kernel smoothing as post-processing methods to imputations from a standard Transformer model. The effectiveness of these methods is discussed in lines 302 and 307, where we note that while the total variation of the estimated function generally decreases, the improvement in MSE is marginal, and in some cases, the MSE actually increases after post-processing. These findings underscore the limitations of simply adding some smoothing post-processing to standard Transformers, regardless of which smoothing technique is used. We chose PACE as a popular imputation method for noisy functional data, and we chose kernel smoothing due to its simplicity and historical prominence as the canonical smoothing method. Following your suggestion, we have also applied trend filtering to the output of standard Transformer, by using trendfilter from the package genlasso in R. The results are in the PDF file attached to our global rebuttal. Although the improvement is greater than using PACE or kernel smoothing as post-processing, SAND remains the best overall method. Our revised paper incorporates and discusses these results, as well as accounting for the other valuable feedback.

[Question 1]

Answer: Yes, there are positional embeddings inside bolded $\tilde{T}$ and $\tilde{T}_c$. The notation is defined in lines 185 and 186: bolded $\tilde{T}$ (or $\tilde{T}_c$) are matrices with the first row being $T$ (or $T_c$), and the rest of $p$ rows are the positional encoding of the output grid. Thank you for the suggestion. Due to the limited space in the main text, we will add a notation paragraph to in the appendix.

[Question 2]

Answer: The subtraction of the first element serves multiple purposes:

• Geometrical Justification: This adjustment ensures that two curves, which may have identical shapes but different intercepts (the first element in $T$), are treated equivalently by the $\rm{diff}(\cdot)$ operator. By subtracting the first element, we effectively remove the absolute positioning, focusing solely on the shape of the curve.
• Mathematical Rationale: Our approach is inspired by the first fundamental theorem of calculus, which expresses a function as $f(b) = f(a) + \int_a^b f’(x),dx$ (refer to line 201 in the main text). In the context of SAND, $f(a)$ represents the first element of $T$, serving as the initial value. The $\rm{Intg}$ function acts as "$\int$", and $\rm{diff}$ approximates the derivative. Subtracting the first element aligns with this conceptual framework, where $f(a)$ is reintegrated post-differentiation to reconstruct the curve accurately (see Equation 5 between lines 189 and 190).
• Algorithmic Efficiency: Subtraction normalizes the input data, enhancing the uniformity of the inputs. This normalization can lead to faster convergence of the $\rm{diff}(\cdot)$ operator’s parameters during training. Omitting the shift could potentially require more training epochs for SAND to achieve similar convergence, due to the variability in initial values across data samples.

We thank the reviewer for the valuable and positive feedback, which we have used to strengthen our paper. Please see our responses to the weakness below. Our responses to the questions are listed in the global author response.

[Weakness 1]

Answer: Thank you for highlighting this aspect. We have not scaled up the simulation for three main reasons:

• Simulation Constraints: As detailed in line 267, each simulated case requires one day for processing on an Nvidia GeForce GTX 1080 Ti. With a total of 88 cases outlined across Tables S2-S10 in the supplementary material, completing all simulations under these conditions requires nearly three months. Even with early-stopping techniques, it still takes up to two months to complete the simulation. These time constraints make larger simulations impractical under our compute budget.
• Relevance to Real-World Data: The sample size of 10,000 in our simulations is substantial, particularly when compared to real datasets we have used, such as the UK electricity dataset with 5,600 samples and the Framingham Heart Study with 870 samples. Simulating data at a scale comparable to real-world datasets ensures that our findings are both realistic and applicable.
• Effectiveness of SAND at current scale: Despite perceptions that 10,000 samples may not seem extensive, our simulations demonstrate a significant advantage of SAND over nine competitors. This indicates robust performance even at the current scale, supporting the effectiveness of SAND on datasets of similar size to popular real-world functional imputation tasks.

[Weakness 2]

Answer: Thank you for highlighting the importance of providing a detailed efficiency analysis. SAND modifies the conventional attention module by omitting the softmax operation, thereby aligning its computational complexity with that of the original attention mechanism, as detailed in Table 1 of Vaswani et al. (2017). Let $m$ denote the number of observations in a subject, $h_d$ be the representation dimension, and $k$ be the kernel size of convolutions. We summarize the computational efficiency of SAND as follows:

• Compared to Recurrent Modules: SAND requires $O(1)$ sequential operations, regardless of sequence length, facilitating faster computation and better suitability for parallel processing. In contrast, recurrent layers inherently require $O(m)$ sequential operations due to their dependency on previous outputs for current computations, which significantly slows down processing and limits scalability.
• Compared to Convolutional Modules: The computational complexity of SAND is O(m^2\cdot h_d), whereas for convolutional modules, it’s $O(k\cdot m\cdot h_d^2)$. In our simulation and data applications, the number of observations per subject $m$ is at most 30 while the representation dimension is 128. Given these parameters, convolutional operations become computationally intensive, leading to slower performance compared to SAND. SAND, analogous to a single attention module, introduces minimal additional computational overhead compared to standard transformers, which typically comprise hundreds of alternating layers of attention and feed-forward modules. We acknowledge the significance of this aspect and our revised paper clearly articulates the computational complexity of SAND as well as its computational advantages.

Regarding SAND’s prediction efficiency (in the sense of statistical estimation): Our evaluation of SAND’s prediction efficiency, measured through relative MSE, is detailed in Table 1. The results indicate that SAND is approximately 9% more efficient than a standard transformer and 11% more efficient than the best non-transformer benchmark, which is PACE. This demonstrates not only SAND’s computational advantages but also its superior accuracy in predictive tasks.

This multi-dimensional approach to assessing efficiency—encompassing computational, estimation, and prediction aspects—provides a robust evaluation of SAND’s performance across various metrics.

[Weakness 3]

Thank you for pointing out the need for a clearer discussion on the benefits of incorporating the attention mechanism in SAND. Here are the key advantages:

• Performance Enhancement: In section 5.1, our extensive experiments demonstrate that SAND significantly improves imputation accuracy, as evidenced by reductions in mean square error and total variation compared to standard methods. This performance boost underscores the effectiveness of the attention-based approach in handling functional data.
• Addressing Non-linearity: Traditional post-processing techniques (see Table 1 in the manuscript) in functional data analysis, such as PACE, are typically linear and may fail to capture complex, non-linear features in the data (lines 101 to 102 in the manuscript). SAND effectively bridges this gap by utilizing the non-linear modeling capabilities of the attention mechanism, allowing for a more nuanced and powerful analysis of functional data.
• Computational Efficiency: Leveraging the attention module within SAND capitalizes on the computational efficiencies discussed in response to [Weakness 2] and as detailed by Vaswani et al. (2017). Our revised paper more clearly lists these advantages of utilizing the attention module in SAND.

[Question 1]

Answer: Please see our response to the integrated question in the global author rebuttal for this question.