Multiple Descents in Unsupervised Auto-Encoders: The Role of Noise, Domain Shift and Anomalies

We appreciate the reviewer for highlighting the main strengths of our paper, especially our rigorous analysis of the double descent phenomenon under different settings and the fact that the paper is well-written and easy to follow.

Weakness 1:

We conducted extensive empirical research to examine how a model's behavior changes with variations in its size, number of training epochs, and training samples. This investigation demonstrated the impact of factors such as noise, domain shift, and anomalies on the test loss curve. Our findings revealed, for example, that over-parameterized models are more effective in minimizing distribution shifts and detecting anomalies, even without being explicitly trained for these tasks.

In terms of vision tasks using CNNs, we believe tabular data offers fascinating opportunities for research, which remains relatively underexplored. We appreciate the suggestion to include recent applications of autoencoders (AEs), such as sparse AEs. As a response, we have added a discussion in Appendix E.5, lines 1615-1619 and Figure 56 showcasing the emergence of double descent in sparse AEs trained on noisy MNIST samples. Due to time constraints, we have verified results specifically for 80% sample noise, with a bottleneck size of 550 and $k = 500$. Further experiments are underway to evaluate different noise levels, bottleneck sizes, and values of $k$.

Additionally, we have incorporated the citations recommended in the review. Thank you for your valuable suggestions, which have enriched our work.

W2 Theoretical results:

We appreciate the reviewer's insightful comments. We would like to emphasize that our work presents several novel findings that should be regarded as independent contributions to the field. Specifically, we have identified and thoroughly investigated multiple descent phenomena in the context of deep unsupervised learning, which had not been previously observed or systematically analyzed. While the concept of double descent has gained attention in supervised learning, its extension to unsupervised learning settings—particularly in autoencoders—represents a significant and original contribution. Our experiments revealed model-wise, epoch-wise, and sample-wise multiple descent behaviors under various conditions, including noisy data, domain shifts, and anomalies. These observations challenge existing assumptions about over-parameterization in unsupervised tasks and open new avenues for understanding the dynamics of generalization within this domain.

In addition to empirically discovering these phenomena, we demonstrated their practical implications using real-world datasets. For example, over-parameterized autoencoders not only improved reconstruction performance but also showed considerable utility in downstream applications such as anomaly detection and domain adaptation. These findings highlight the real-world relevance of our work and its potential to inform the design of robust unsupervised models in complex data environments. While we acknowledge that a deeper theoretical analysis of these observations is a valuable direction for future research, the novelty and scope of our empirical contributions provide a strong foundation for advancing the understanding of over-parameterized models in unsupervised learning.

However, as suggested by the reviewer, the theoretical foundations of these observations deserve further exploration. Building on the works cited in [3] and [4], future research could formalize the conditions under which multiple descent occurs in unsupervised settings. For instance, extending the parameter-counting frameworks from [3] to unsupervised tasks might reveal how the effective number of parameters changes with varying data distributions and model architectures (or latent dimension of the AE). Additionally, by leveraging insights from [4] regarding the limitations of the bias-variance tradeoff in fixed designs, theoretical studies could examine how random designs or data perturbations affect generalization and descent behaviors in unsupervised models. We have included these citations and ideas as potential directions for future work in the conclusions section, line 539.

Supervised vs unsupervised learning:

"To make the point for tabular data, the authors should study the behaviour of such (complex) models, e.g., one of the provided references. "

The reviewer’s claim is incorrect. Our architectures are, in fact, larger and more complex than the models employed in [E-G], which achieve state-of-the-art (SOTA) performance. While those specific architectures are not explicitly used in our work, the cited references were included to demonstrate that such architectures have been effective in related contexts, contradicting the reviewer's argument. Furthermore, our approach employs unsupervised autoencoders (AEs), which are well-suited for tabular data and align with the goals of our study.

“tabular data is for sure not the main interest for the neural network community, with a whole line of papers showing that classical statistical learning, especially (gradient boosted) tree-based methods, are still on par or even outperform NNs on tabular data”

[E] Gorishniy, Yury, Akim Kotelnikov, and Artem Babenko. "TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling." arXiv preprint arXiv:2410.24210 (2024).

[F] Gorishniy, Yury, et al. "TabR: Tabular Deep Learning Meets Nearest Neighbors." The Twelfth International Conference on Learning Representations. 2024.

[G] Holzmüller D, Grinsztajn L, Steinwart I. Better by default: Strong pre-tuned mlps and boosted trees on tabular data. arXiv preprint arXiv:2407.04491. 2024 Jul 5.

Classical Double Descent:

"People used over-parametrized before the authors' paper. So, what of the double descent, specifically, are the authors suggesting to be helpful for downstream applications?"

The reviewer suggests that since over-parameterized models were already used in practice prior to the authors' work, studying the double descent phenomenon is unnecessary. However, this perspective overlooks the significant impact that understanding double descent has had on the field. The discovery of double descent challenged traditional beliefs about the bias-variance tradeoff, which suggested that increasing model complexity would always lead to overfitting. Double descent revealed that beyond a certain level of over-parameterization, performance can improve again, defying these expectations.

This insight has reshaped the understanding of model behavior, particularly for over-parameterized models like deep neural networks. It has influenced how we think about capacity, training data, and generalization, and has led to new research on optimizing model architectures and regularization strategies. In short, research on double descent has directly informed practical machine learning applications, advancing generalization performance and robustness.

CNNs:

"Figure 35 and 36, showing a double descent in the completely under-specified case (#of channel < # of output classes)":

This comment further indicates a misunderstanding, as the reviewer appears unfamiliar with unsupervised learning using autoencoders, where the notion of 'output classes' does not apply in the traditional sense.

Additionally, the reviewer precisely wrote, "And again, why does it not happen in CNNs?" which demonstrates either a lack of thorough reading or oversight of our work. We have explicitly addressed this point in the paper, showing that double descent indeed occurs in CNNs. Additionally, on top of Figures 35 and 36, Figure 34 also presents model-wise double descent for different levels of sample and feature noise, and Figure 37 for the presence of domain shift.

Furthermore, we have conducted additional experiments based on the reviewer's earlier requests, yet the reviewer now moves the goalpost, implying that more experiments are still needed. We believe the current set of experiments sufficiently demonstrates the occurrence of double descent in CNNs across a range of settings, and we have made every effort to address previous concerns adequately.

MNIST:

To support the applicability of our work, we have not only included MNIST but also utilized complex datasets, including single-cell RNA sequencing data, which presents significant challenges such as high-dimensionality, sparsity, and noise. These datasets are representative of real-world tabular applications and align well with the scope of our study. Most of the works cited in our rebuttal also focus on such datasets, highlighting their relevance to sparse autoencoders and the broader research context.

We respectfully disagree with the reviewer’s assertion that findings on MNIST are inherently limited by its data complexity. We have cited several recent publications from top-tier venues [A-D] to substantiate the fact that MNIST is commonly used for demonstrating interpretability and for evaluating autoencoders in literature. Additionally, our results on single-cell RNA data further validate the robustness of our approach and the relationship between model size and generalization in challenging, high-dimensional tabular domains.

The suggestion to use vision or language benchmarks, while valuable in certain contexts, appears to shift the goalpost unnecessarily. Our study focuses on tabular data and interpretable sparse autoencoders, and the datasets chosen—MNIST and single-cell RNA data—are appropriate for addressing the goals of our work. We believe these choices provide a strong foundation for demonstrating the effectiveness of our methodology.

“ensure that findings are not due to low data complexity”

Our paper includes single cell RNA-seq data which is much more complex than all vision datasets. We also include results for anomaly detection on CelebA (attributes).

[A] Balın, M.F., et al., 2019, May. Concrete autoencoders: Differentiable feature selection and reconstruction. In International conference on machine learning (pp. 444-453). PMLR.

[B] Kim, C., et al., Discovering Features with Synergistic Interactions in Multiple Views. In Forty-first International Conference on Machine Learning. 2024

[C] Jethani, N., et al. "Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

[D] Svirsky, J., et al.. Interpretable Deep Clustering for Tabular Data. In Forty-first International Conference on Machine Learning. 2024

P2 - It is essential first to define the performance metric in the context of unsupervised learning:

We appreciate the reviewer's emphasis on defining performance metrics when discussing the double descent phenomenon in unsupervised learning. In our work, we explicitly define and employ key metrics suitable for the unsupervised context. Specifically, we use the test loss, measured by the Mean Squared Error (MSE) between the input and the reconstructed output, as a central metric. This is particularly appropriate for autoencoders, as it directly reflects the model’s ability to capture the underlying data structure rather than merely overfitting noise—crucial for understanding double descent behavior.

Beyond MSE, we also include other relevant metrics to demonstrate the practical implications of double descent. For anomaly detection, we use the ROC-AUC score to evaluate the model's ability to differentiate between normal and anomalous samples. In our domain adaptation experiments with single-cell RNA-seq data, we use the K-nearest Neighbors Domain Adaptation Test (KNN-DAT) to assess how well latent representations align across different domains. By leveraging these well-defined performance metrics, we ensure a comprehensive characterization of double descent patterns that is relevant to real-world unsupervised learning tasks.

P3 - If unsupervised learning is limited to distribution learning, I suggest defining the performance metric based on the distance to the target distribution:

Unsupervised learning is by no means limited to distribution learning. Our paper is specifically focused on under-complete autoencoders (AEs), which are non-generative models. AEs serve multiple purposes in machine learning, including dimensionality reduction, denoising, compression, anomaly detection, feature selection, and domain adaptation. Unlike Variational Autoencoders (VAEs) or Generative Adversarial Networks (GANs), which are often used for density estimation, our focus is on leveraging AEs for practical applications that do not require direct distribution modeling.

We have chosen performance metrics that align with the objectives of these tasks, such as Mean Squared Error (MSE) for reconstruction quality, ROC-AUC for anomaly detection, and KNN-DAT for assessing domain adaptation. Details about these metrics and their relevance to our analysis are provided in Subsections 4.1 and 5.1 of the paper and addressed in our response to point 2 above.

Q1 - Is it possible to define the double descent phenomenon for unsupervised learning models other than autoencoders?

A study has investigated the phenomenon of double descent in Generative Adversarial Networks (GANs). The researchers demonstrated double descent in both supervised and semi-supervised GANs; however, they did not observe a double descent curve for unsupervised GANs [1]. This study is also referenced in our paper.

[1] Luzi, Lorenzo, Yehuda Dar, and Richard Baraniuk. "Double descent and other interpolation phenomena in gans." arXiv preprint arXiv:2106.04003 (2021).

Q2 - How can the use of test loss to define predictive performance in unsupervised learning be justified?

Please read P3. Unsupervised learning is not limited to distribution learning and we are working with under-complete AEs, which is not a part of distributional learning.

We thank the reviewer for acknowledging the main strengths of our paper, particularly the encouragement for further research on model complexity in unsupervised learning.

P1: To clarify the distinction between our study of double descent in unsupervised learning and traditional regression tasks:

Our experiments focus on autoencoders (AEs), which fundamentally differ from regression models in both objective and structure. Unlike regression, where models map inputs to a specific target variable, AEs are designed to learn a compressed representation of the input data itself without supervision, with the goal of capturing inherent structure rather than fitting an external target. The "target" in an AE is simply the input itself, and the model's reconstruction loss reflects how well it retains underlying patterns rather than directly mapping to a response variable as in regression. By examining AEs under varying noise conditions, domain shifts, and anomalies, our study provides insights specific to the unsupervised setting, where multiple descent arises due to challenges in learning these representations rather than fitting labels. Moreover, regression is a completely supervised task, which assumes access to the target data, while our setup is completely unsupervised.

The fact that AE training and regression are completely different tasks is exemplified by the extensive theoretical and practical research that studies these two tasks separately. There is no use case of regression in which the target variable is $X$ ($X$ is the data). This difference involves not just a specific choice of target variable but also emphasizes that all types of contamination should be applied to both the input and output. This contrasts significantly with label/ target noise in a supervised setting, which is only applied to the output variable (y). Here is a list of studies discussing double descent in regression tasks, mostly linear regression: [1, 2, 3, 4, 5].

We would like to highlight that research into similar or related phenomena across different data models and architectures is a common practice in machine learning. For example, the behavior of L1-norm regularization has been studied separately in regression [6] and in autoencoders (AEs) [7]. These independent investigations underline the fundamental distinctions between these tasks; if they were inherently identical, distinct analyses of L1-norm behavior would not be necessary.

Our findings extend the understanding of double descent to unsupervised tasks and highlight unique implications for model selection in scenarios like anomaly detection and domain adaptation, which are not covered by standard regression-based double descent observations.

[1] Harmless interpolation of noisy data in regression.

[2] Benign overfitting in linear regression.

[3] On the number of variables to use in principal component regression.

[4] Dimensionality reduction, regularization, and generalization in overparameterized regressions.

[5] More data can hurt for linear regression: Sample-wise double descent, 2019.

[6] Regression shrinkage and selection via the lasso.

[7] Sparse L^1- Autoencoders for Scientific Data Compression

P2 - discovering multiple descent patterns may not offer a new perspective based solely on the current simulation results:

In contrast to the reviewer's claim, our work provides, for the first time, evidence of double descent in unsupervised learning using real-world data, moving beyond purely synthetic simulations. Specifically, we demonstrated the occurrence of double descent not only in synthetic settings but also on established benchmark datasets like MNIST, as well as in several real-world single-cell RNA-seq datasets. This extension to real-world datasets significantly enhances the relevance of our findings, as it validates the existence of multiple descent phenomena under realistic conditions that practitioners often encounter.

The inclusion of datasets such as MNIST and single-cell RNA-seq highlights the generalizability of our results across different types of data, ranging from image data to highly complex biological data with inherent noise and domain shifts. This real-world evidence contrasts with prior work that primarily relied on simulations and highly controlled environments, which may not fully capture the complexities of practical applications. Our work addresses this gap by systematically exploring these double descent behaviors in settings where noise, anomalies, and domain shifts are naturally present, providing valuable insights that are directly applicable to real-world scenarios.

Additionally, our analysis illustrates the implications of double descent for practical tasks like anomaly detection and domain adaptation, particularly in unsupervised settings.

Q4 - Determine SNR in real-world data:

One approach for estimating SNR from noisy data without making specific assumptions is using the Marchenku Pastur (MP) law [1]. First, we calculate the covariance matrix of our data. second, we perform eigenvalue decomposition to obtain the eigenvalues of the covariance matrix. Thirdly, we identify the eigenvalues that fall within the MP distribution (eigenvalues probably associated with noise) and the ones that do not (considered to be associated with the signal). Finally, we use the ratio of the signal-related eigenvalues to the noise-related eigenvalues to estimate the SNR. This approach assumes that the noise is i.i.d.

Another approach is using manifold learning, where the assumption is that the signal lies on a lower-dimensional manifold. This method can estimate the intrinsic dimension of the data (d) using existing schemes such as [2, 3]. After that, we can estimate the power of the signal by calculating the variance retained by the lower-dimensional manifold (for example, in PCA, we will calculate the variance based on the first k eigenvalues), and the rest of the eigenvalues will be proportional to the variance of the noise. Finally, the ratio between them yields the SNR.

Other methods can be applied for time signals, such as power spectrum analysis. This involves applying a Fourier transform to calculate the power spectral density (PSD) of the signal. By separating the power associated with the signal from that of the noise, we can compute their ratio to estimate the SNR. However, this method provides an approximation since some noise components may overlap with the signal's frequencies. Using techniques like low-pass filtering to separate the signal from the noise can also be effective. It’s essential to understand the data’s characteristics, domain knowledge, and relevant model assumptions for better accuracy.

[1] Gavish, Matan, and David L. Donoho. "The optimal hard threshold for singular values is $4/\sqrt {3}$." IEEE Transactions on Information Theory 60.8 (2014): 5040-5053.

[2] Ceruti, Claudio, et al. "DANCo: dimensionality from angle and norm concentration." arXiv preprint arXiv:1206.3881 (2012).

[3] Facco, Elena, et al. "Estimating the intrinsic dimension of datasets by a minimal neighborhood information." Scientific reports 7.1 (2017): 12140.

Q5 - KNN-DAT experiment (Figure 12):

Thank you for raising this important question. As you suggested, we utilized the KL-divergence metric to assess the distribution shift between our source data ('Baron') and the target datasets ('Segerstolpe', 'Xin', 'Mutaro', and 'Wang'). We have added a discussion about this in lines 449-452 of the revised manuscript and modified the legend of Figure 12 to include the KL-divergence results. Our findings indicate a relationship between the height of the test loss and the domain shift of the data, which aligns with the results from the simulated experiment.

Q2 - Hypothesis on why multiple descents occur:

We observed that for the double descent curve to manifest in the presence of sample noise, the noise must be predominant—characterized by a high proportion of noisy samples and low SNR values. In such cases, models in the critical regime, which have sufficient capacity, prioritize memorizing the noise over learning the underlying signal, leading to increased test loss. In contrast, when the SNR is high, even if all samples are noisy, the model behaves similarly to one trained on clean data. With high SNR or a small fraction of noisy samples, model-wise double descent does not occur, as minimal noise allows models with sufficient capacity to learn the underlying signal rather than memorize noise, thus avoiding the critical regime. Over-parameterized models exhibit a second descent because they have enough capacity to memorize noisy samples while still capturing the characteristics of the signal. A similar pattern holds for anomalies with low SAR, as illustrated in Figure 6.a, since noisy samples can be interpreted as a form of anomaly.

In the context of domain shift, models operating in the critical regime may memorize the source domain in order to reduce training loss. This can lead to an increased test loss when applied to target data. Over-parameterized models, on the other hand, can overfit the source data even as they minimize the model’s weight norm. These models, characterized by a lower L2 norm, tend to generalize better across different domains, which helps to lower the test loss. For a visual representation of this concept, refer to [1] Figure 3 and [2] Figures 2.d and 2.g. An explanation of the critical regions can be found in W2.

[1] Belkin, Mikhail, et al. "Reconciling modern machine-learning practice and the classical bias–variance trade-off." Proceedings of the National Academy of Sciences 116.32 (2019): 15849-15854.

[2] Dar, Yehuda, Vidya Muthukumar, and Richard Baraniuk. "A farewell to the bias-variance tradeoff." An Overview of the Theory of Overparameterized Machine Learning. ArXiv abs/2109.02355 (2021).

Q3 - Practical ways to predict the occurrence of multiple descents based on data characteristics or noise conditions:

We appreciate the reviewer’s question regarding practical ways to predict multiple descent phenomena. We first note that it is possible to estimate key data characteristics, such as intrinsic dimensionality and structural properties (explained in the next answer), which can provide initial indications of whether multiple descent is likely to occur. For instance, when noise levels are low and the dimensionality of the data is high, the phenomenon is less likely to arise. Conversely, high noise levels and low dimensionality will increase the likelihood of observing a double descent. However, validating these hypotheses requires further theoretical analysis.

Our study highlights that the critical factors influencing multiple descent include the number of training samples, signal-to-noise ratio (SNR), noise levels, the presence of anomalies, and domain shifts. Identifying and quantifying these dependencies can help refine predictions about when double descent behavior might emerge. Additionally, practical approximations, such as using proxy metrics for SNR estimation or monitoring training dynamics in real-time, could provide valuable insights for practitioners dealing with noisy or shifted datasets. We believe that these are great questions for future work.

We thank the reviewer for acknowledging the main strengths of our paper, particularly the importance of understanding the generalization behavior of over-parameterized models, how well the paper is written, and the thorough experiments providing insightful observations.

W1 - Providing further theoretical insights:

We agree that additional theoretical insights would deepen the understanding of this phenomenon. Our primary goal, however, was to provide comprehensive empirical evidence demonstrating the existence of the double descent phenomenon in unsupervised learning and to address the inconsistencies in the literature. We believe that our extensive empirical experiments, covering various contamination setups such as sample noise, feature noise, domain shift, and anomalies, help offset the absence of theoretical analysis.

Furthermore, significant contributions like [1] and [2] have advanced the understanding of double descent in supervised learning despite lacking theoretical backing. We hope that our extensive empirical findings across different contamination setups and the confirmation of double descent in unsupervised tasks will inspire researchers to investigate the theoretical aspects of this phenomenon in greater depth, both qualitatively and quantitatively.

[1] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849–15854, 2019.

[2] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021.

W2 - Defining critical regions for multiple descents more explicitly:

As mentioned in the first section addressing the weaknesses, our research is entirely empirical. While some papers explore the mathematical relationships governing the double descent curve in supervised learning—often identifying the critical regime based on the number of training samples and noise energy—these studies typically focus on linear, sparse models with few layers, substantiating their findings through synthetic data whose characteristics they can carefully control [1, 2, 3, 4]. In contrast, our study employs non-linear models in an unsupervised context using real-world datasets.

Our results indicate that the critical regimes deviate from the scaling laws reported in those theoretical papers. Moreover, we observe that data characteristics significantly influence the double descent behavior. For instance, while single-cell RNA data uniquely displays a triple descent pattern, other datasets consistently exhibit double descent.

Thank you for suggesting a definition of scaling law, which we are currently working on.

[1] Jacot, Arthur, et al. "Implicit regularization of random feature models." International Conference on Machine Learning. PMLR, 2020.

[2] Chen, Lin, et al. "Multiple descent: Design your own generalization curve." Advances in Neural Information Processing Systems 34 (2021): 8898-8912.

[3] Liang, Tengyuan, Alexander Rakhlin, and Xiyu Zhai. "On the multiple descent of minimum-norm interpolants and restricted lower isometry of kernels." Conference on Learning Theory. PMLR, 2020.

[4] Belkin, Mikhail, Daniel J. Hsu, and Partha Mitra. "Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate." Advances in neural information processing systems 31 (2018).

Q1 - Noise is not always Gaussian:

For your request, we have conducted an experiment involving the single-cell RNA and linear subspace datasets for the case of sample noise with Laplace distributed noise. The experiment was held exactly as the one in Figure 3 in order to compare the results with different types of noise distribution. We have added a discussion in lines 249-250 in the main paper, and in the new Appendix E.5 (lines 1610-1613) and Figure 55.

Dear reviewer, we appreciate the time and effort you have dedicated to reviewing our paper. As the discussion period is limited, we kindly request that the reviewer evaluate the new information provided in the rebuttal. We are eager to improve our paper and resolve all the concerns raised by the reviewer. If there are any remaining concerns that have not been addressed, we would be happy to provide further explanations.

I apologize. When submitting, I put the review of a different paper. Here is the correct review:

Summary: This paper investigates the double descent phenomenon about three key training parameters, training epochs, model size, and data sample size, within an unsupervised learning framework. To illustrate the phenomenon, consider training epochs: initially, test error decreases as the number of training epochs increases. However, after reaching a certain threshold, test error begins to increase. After a period of error rising, the test error decreases again, known as, double descent. Additionally, the authors conduct experiments using synthetic data and small-scope models to empirically examine the impact of customized noise, domain shifts, and anomalies on double descent behavior. Finally, the study assesses how model size influences domain adaptation and anomaly detection performance.

Soundness: 2: fair Presentation: 2: fair Contribution: 2: fair

Strengths

1. This work provides empirical evidence that the double descent phenomenon exists in various training parameters under unsupervised learning settings, and the non-linear relationship observed between test error and training parameters may be of interest.

2. Unlike previous studies, to verify the influential factor model size, this work proposes utilizing specific layers in different dimensions to simulate the changing of model size. This modification is well-suited for experiments with small-scope neural networks by avoiding introducing extra variables.

Weaknesses

$**Weakness in method motivation and novelty**$

1. This work needs more motivation to study the phenomenon of double descent. Particularly, this work does not provide possible theoretical improvement or help better understand the unsupervised learning problems. Additionally, the patterns identified lack a compelling connection to practical real-world (deep) model training. To summarize, this paper does not show enough evidence that the revealed pattern can improve existing optimization problems theoretically or practically.

2. While the study emphasizes that its unsupervised learning focus is distinct from prior work, the observed double descent patterns align with those documented in supervised learning. This difference in setup, while noted, does not seem to substantively alter the properties of the optimization process.

3. The empirical verification of double descent with respect to model size (via specific layer dimensions), sample size, and training epochs lacks novelty.

$**Weakness in evaluation**$

A substantial portion of the paper, particularly Section 4, examines double descent behavior under conditions of customized noise, domain shift, and anomalies in the training data. However, these customized conditions appear to have been selected without a clear rationale, and the resulting findings are unsurprising.

Questions

1. Could you elaborate on the fundamental differences between supervised and unsupervised learning that warrant separate investigations into the double descent phenomenon in each scenario?

2. What insights or practical applications do you anticipate arising from studying double descent under different data conditions, such as Gaussian noise and domain shift?

3. Section 5 is somewhat unclear. The conclusion drawn here—that over-parameterized models improve training performance under varying conditions—seems straightforward. Additionally, how does this finding relate to the double descent phenomenon?

Flag For Ethics Review

No ethics review needed.

Rating

3: reject, not good enough

Confidence 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Q3 - The conclusion drawn here—that over-parameterized models improve training performance under varying conditions:

If you are referring to: “Over-parameterized models also lead to reduced train and test losses, resulting in improved reconstruction of both the source and target data”, we have modified these lines to be: ”They also lead to reduced test loss, resulting in improved reconstruction of the target data.”, lines 465-466. You are right, the improvement in train loss as the model’s size rises is straightforward.

Weakness in evaluation:

Our study introduces four contamination setups: sample noise, feature noise, domain shift, and anomalies, each with varying noise levels, domain shifts, and SNRs.

Section 4 presents results using synthetic or semi-synthetic datasets, while Section 5 focuses on real-world data. These experiments are both relevant and novel, as real-world data often contains contamination, and our findings illustrate how this impacts test loss.

For instance, in many cases, datasets are inherently noisy, and we identify a critical regime that should be avoided. Additionally, in anomaly detection applications, it is not straightforward to receive pre-sorted data divided into clean and anomalous samples. We find that the ROC-AUC and test loss of the clean samples display non-monotonic behavior and double descent, respectively, as the training data is contaminated with anomalies.

Additionally, the double descent phenomenon itself is particularly complex in the unsupervised setting, especially when the training set contains more noisy samples than clean ones. Intuitively, one might expect the test loss to keep increasing as bigger models memorize the noise instead of learning the underlying structure of the signal.

Q1 - elaborate on the fundamental differences between supervised and unsupervised learning:

The fundamental differences between supervised and unsupervised learning indeed justify separate investigations into the double descent phenomenon, primarily due to the nature of the tasks, the types of assumptions involved, and how noise and data structure influence learning behavior in each regime.

1. Task Objectives: Supervised learning tasks typically involve predicting discrete or continuous labels, such as classification or regression. The model aims to map input features to a known target output. In contrast, unsupervised learning, such as autoencoders (AEs), aims to learn meaningful representations of the input data without labeled targets, often focusing on tasks like reconstruction. This difference fundamentally alters how errors and model performance are evaluated. In supervised learning, performance is measured in terms of predictive accuracy, while in unsupervised learning, metrics such as reconstruction loss are used.

2. Data Assumptions: Supervised learning commonly introduces noise as "label noise," where the labels may be flipped or altered, affecting the model's predictive capability. Unsupervised learning deals with noise in the input data itself, influencing the model's ability to learn useful representations. For instance, adding noise to an AEr’s training data can lead to memorization of noise rather than learning the underlying structure. This distinction changes the conditions under which double descent manifests, as unsupervised models need to balance learning useful features against overfitting to noise, while supervised models focus on the trade-off between bias and variance in predicting labels.

3. Model Complexity: Supervised and unsupervised learning differ in their underlying model assumptions. Supervised models often use assumptions about the relationship between input features and labels, such as linear separability. In unsupervised settings, models like autoencoders must capture complex data structures without explicit supervision, making them more sensitive to changes in data distribution and model complexity. As a result, the double descent behavior in unsupervised models may arise under different conditions, such as varying noise levels, domain shifts, or data anomalies.

Q2 - insights or practical applications arising from studying double descent under different data conditions:

Studying double descent under different data conditions, such as Gaussian noise and domain shift, can lead to several valuable insights and practical applications:

1. Noise Resilience: Understanding how models behave under different noise levels can help in designing more robust models that maintain performance despite noisy data.

2. Generalization: Insights into how over-parameterized models can generalize better even when trained on noisy data, challenging traditional views on overfitting.

3. Double Descent Curves: Identifying the points of double descent can help in selecting the optimal model complexity that balances bias and variance effectively.

4. Guidelines for Practitioners: Providing guidelines on selecting model sizes and training regimes based on the expected noise levels and domain shifts in the data.

5. Denoising: Insights from studying noise can lead to better denoising techniques in image and signal processing.

6. Transfer Learning: Improved understanding of how models handle domain shifts can enhance transfer learning techniques, making models more adaptable to new, unseen data.

7. Anomaly detection: Understanding the behavior of model capacity under different anomaly levels in the training data can help enhance anomaly detection capabilities.

Weakness 2 - This difference in setup, while noted, does not seem to substantively alter the properties of the optimization process:

Our work differs from prior research in several important ways. Firstly, while most existing studies focus on supervised learning, our research is entirely unsupervised, specifically utilizing autoencoders. In supervised learning, noise is typically added to the labels or targets, commonly referred to as "label noise." Mathematically, this can be represented as perturbing the labels $y$, such that $y' = y + \epsilon$, where $\epsilon$ is random noise. In contrast, our unsupervised setup involves adding noise directly to the input data, represented by $x' = x + \epsilon$, where $x$ is the input and $\epsilon$ is noise. This model distinction means that the optimization process in our work centers on reconstructing noisy input data rather than minimizing a loss with noisy target labels.

Secondly, we investigate scenarios that extend beyond standard noise settings, such as the presence of anomalies in the training data. This scenario, characterized by introducing anomalous samples alongside regular data, is fundamentally different from typical supervised noise settings and, to our knowledge, has not been explored in the context of double descent in supervised learning. The impact of such anomalies on the double descent behavior, particularly in an unsupervised context, provides new insights that are absent from prior work.

Finally, we identify the unique role of the latent dimension in determining the emergence of double descent in our unsupervised autoencoder setting. Unlike prior work that focuses on model width or parameter count, we show that the size of the latent space significantly influences the balance between learning meaningful representations and overfitting noise. Smaller latent dimensions constrain model capacity, leading to higher test loss, especially with noise or anomalies. As the latent dimension increases, the model transitions to an over-parameterized regime, resulting in the characteristic double descent curve, where the test loss first rises due to noise memorization and then decreases as generalization improves. This unique focus on the latent dimension highlights its critical impact on performance in unsupervised learning, offering insights that are particularly useful for tasks like anomaly detection and domain adaptation.

Weakness 3 - The empirical verification of double descent lacks novelty:

We believe that empirically verifying double descent concerning model size, number of epochs, and number of samples is novel for four main reasons:

1. [1] argues that double descent does not occur in unsupervised learning using AEs. Our work disproves this claim by demonstrating multiple instances of double and triple descent curves across various contamination setups. Additionally, our paper addresses and resolves the debate on the existence of double descent in unsupervised learning, as discussed in the related work section. The works in the related work section show that this phenomenon is of interest in unsupervised learning.

2. To expand on the second weakness: while model-wise, epoch-wise, and sample-wise double descent have been observed in supervised learning, one of our key contributions is demonstrating that this phenomenon also occurs beyond the supervised regime—a point that was previously uncertain.

3. We decompose the model's size into two interconnected components, the hidden and bottleneck layers, demonstrating how each layer influences the test loss and the critical regime. Figure 2 displays interesting results: as the bottleneck dimension increases, a larger hidden layer size is required to overfit (i.e., the critical regime shifts to larger hidden dimensions as the bottleneck layer size gets bigger). Intuitively, one might expect the opposite: that the critical regime would shift downwards as the bottleneck size increases, allowing an interpolating model to be achieved with a smaller hidden layer size while keeping the overall model size the same. This is a richer evaluation than supervised examples (see [2]) since we display that certain layers of an autoencoder are very influential for the reconstruction results and the double descent curve.

4. We show that anomalies also influence the double descent curve, an observation that was not observed in supervised learning.

[1] Lupidi, Alisia, Yonatan Gideoni, and Dulhan Jayalath. "Does Double Descent Occur in Self-Supervised Learning?." arXiv preprint arXiv:2307.07872 (2023).

[2] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021.

We thank the reviewer for recognizing the key strengths of our paper, particularly the empirical evidence demonstrating the existence of the double descent phenomenon across various training parameters in unsupervised settings and the relationship between test error and these parameters. Additionally, we appreciate the acknowledgment of our analysis on model size, which highlights the influence of both the bottleneck and hidden layers on the double descent curve.

Weakness 1 - motivation:

Our primary motivation lies in extending the understanding of double descent—previously studied mainly in supervised settings—to unsupervised learning scenarios, which are critical in many real-world applications. While the phenomenon of double descent has provided substantial insights into the generalization capabilities of over-parameterized models in supervised tasks, similar insights for unsupervised learning remain limited. Our work aims to address this gap by demonstrating how double descent behaviors occur with under-complete autoencoders across various challenging scenarios, including noise, domain shifts, and anomalies [1, 2, 3]. These scenarios are highly relevant in practical deep-learning tasks involving real-world data, particularly where labeled data is unavailable.

Regarding practical impact, our findings provide important empirical insights into optimizing unsupervised model training. For instance, we demonstrate that over-parameterized autoencoders can achieve lower test loss and better generalization, even in the presence of substantial noise or domain shift, which directly affects model selection in tasks like anomaly detection and domain adaptation. Our results suggest that over-parameterized models can mitigate the adverse effects of noise and improve feature extraction, thereby enhancing downstream performance in anomaly detection and domain adaptation.

While our work is primarily empirical (similar to the impactful works in [4, 5], it lays a foundation for future theoretical exploration by characterizing double descent patterns in unsupervised contexts. The identified behaviors provide an empirical basis for further theoretical analysis, particularly regarding how model capacity interacts with noise and domain shifts in unsupervised learning. Thus, our study not only contributes practical insights but also sets the stage for future theoretical advancements that can deepen our understanding of optimization and generalization in unsupervised models.

[1] Vincent, Pascal, et al. "Extracting and composing robust features with denoising autoencoders." Proceedings of the 25th international conference on Machine learning. 2008.

[2] Zhou, Chong, and Randy C. Paffenroth. "Anomaly detection with robust deep autoencoders." Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 2017.

[3] Deng, Jun, et al. "Autoencoder-based unsupervised domain adaptation for speech emotion recognition." IEEE Signal Processing Letters 21.9 (2014): 1068-1072.

[4] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849–15854, 2019.

[5] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021.

Dear reviewer, we appreciate the time and effort you have dedicated to reviewing our paper. As the discussion period is limited, we kindly request that the reviewer evaluate the new information provided in the rebuttal. We are eager to improve our paper and resolve all the concerns raised by the reviewer. If there are any remaining concerns that have not been addressed, we would be happy to provide further explanations.

Q2 - Some curves are missing:

In Figure 3(b), we only showed noise levels ranging from 30-60% since the figure was too cluttered. We followed your suggestion and also added 70-90% noise levels. Lines 246-248 discuss the final ascent phenomenon, which we also observed in single-cell RNA data for cases with 0-20% sample noise. These lower noise levels are presented in Appendix E.4, Figure 54(b), resulting in all the noise levels also presented in Figure 3(a). Please note that the single-cell RNA data is acquired from real-world experiments that already include natural noise, so even if we did not add any synthetic noise (presented by 0%), the data is still affected by noise.

Q3 - A paragraph to summarize the relation between the double descent and the four variables:

That is a great suggestion. In the revised paper, we have added a paragraph concluding the results of all the contamination setups in lines 344-349. As contamination setups become more severe, such as higher noise levels for sample and feature noise, significant domain shifts, many anomalies, or low SNR, the double descent phenomenon becomes more pronounced, and the test loss increases. In some instances, these noise levels also cause the critical regime to shift to the right.

Q4 - compare the double descent behaviors in unsupervised learning against the supervised learning regime:

Since double descent was initially observed in supervised learning, you are right that a comparison between our unsupervised results and those from supervised learning is essential. We have added Table 1 in Appendix A (page 17) in the revised paper, summarizing which contamination setups have been studied for their impact on the double descent curve in both unsupervised and supervised learning. Our findings indicate that the overall results are similar across the setups examined in both learning regimes [1, 2, 3]. The primary difference we observed is the necessity of low SNR values for the unsupervised regime.

Although the double descent phenomenon appears in both supervised and unsupervised regimes, we would like to clarify that their tasks differ significantly. For instance, most research on double descent in supervised learning focuses on regression or classification tasks, which come with different assumptions, such as using linear models and distinct methods of introducing noise, like label noise (where some labels are randomly altered). In contrast, our work addresses unsupervised autoencoder tasks with a different set of assumptions, such as learning meaningful representations in a latent space, and the noise is added to the samples. Therefore, comparisons between the behaviors in supervised and unsupervised regimes should be made cautiously.

[1] Nakkiran, Preetum, et al. "Deep double descent: Where bigger models and more data hurt." Journal of Statistical Mechanics: Theory and Experiment 2021.12 (2021): 124003.

[2] Li, Zhu, Weijie J. Su, and Dino Sejdinovic. "Benign overfitting and noisy features." Journal of the American Statistical Association 118.544 (2023): 2876-2888.

[3] Tripuraneni, Nilesh, Ben Adlam, and Jeffrey Pennington. "Overparameterization improves robustness to covariate shift in high dimensions." Advances in Neural Information Processing Systems 34 (2021): 13883-13897.

We appreciate the reviewer for highlighting the main strengths of our paper, especially our discovery of double descent in unsupervised learning and the analysis of various contamination setups, including sample noise, feature noise, domain shifts, and anomalies. We are grateful for the recognition that although there are four contamination setups, most of them share the same intuition regarding the height and horizontal position of the test loss and critical regime, as well as the connections we established between synthetic and real-world data observations.

Weaknesses - the paper is not smooth to read:

Due to the numerous experiments involving different datasets and contamination setups, the main paper contains many figures, which may impact the flow of reading. We have made efforts to reduce the number of figures, including moving all training loss figures to the appendix section. For your request, we have arranged some of the figures so the paper will be easier to follow. Figures 7-13 in our revised manuscript were modified to be at the top and bottom of the paper. Details regarding under-complete AEs: The rationale for using under-complete AEs (instead of over-complete ones) is explained in lines 203-205 in the “Results” section.

Additionally, in response to your suggestion, we have included more details explaining our choice of under-complete AEs in lines 70-72 of the attached manuscript, adjacent to the first mention of the term “under-complete AEs”. Appendix A, Table 2 details the model sizes, including the bottleneck layer size, hidden layer size for MLPs, and the number of channels for CNNs. Additionally, Figures 17 and 18 illustrate the MLP and CNN models and their parameter counts.

Q1 - How is Fig.2 different from the traditional figure of test loss against the number of parameters? Does it mean the relation between the two in unsupervised learning is the same as supervised learning?

In contrast to supervised learning, the bottleneck layer plays a significant role in many applications that use AEs (i.e., dimensionality reduction, denoising, compression, anomaly detection, feature selection, and domain adaptation). This motivates breaking down the model’s size into the bottleneck layer and other layers.

Figure 2 differs from traditional test loss versus number of parameters plots because it also presents the variation in the bottleneck size. In all the other figures in our paper, the bottleneck size remains constant, and we vary the hidden layer size. The bottleneck and hidden layer sizes play a crucial role in the outcomes of an autoencoder, which is why we broke down the model’s size into these parts. For example, suppose we have two models, one with a larger bottleneck layer and smaller hidden layers compared to the second, and both of them result in the same number of parameters. The first model might achieve better reconstruction results since the bottleneck layer is more significant, leading to less information loss in the latent space, making it easier for the decoder to extract a better output.

Looking only at the model’s size and not breaking it down to these two factors will result in noisy test loss curves as a function of the number of parameters since two models with similar capacities result in completely different test loss outcomes. Figure 2 shows that it’s not enough just to select the model’s capacity; we also need to consider the hidden and bottleneck layer size, as they influence the position of the critical regime. In all the figures in which the bottleneck layer size is constant, we reveal relations between the test loss and the model’s size that are similar to the supervised regime.

Moreover, Figure 2 displays interesting results: as the bottleneck dimension increases, a larger hidden layer size is required to overfit (i.e., the critical regime shifts to larger hidden dimensions as the bottleneck layer size gets bigger). Intuitively, one might expect the opposite: that the critical regime would shift downwards as the bottleneck size increases, allowing an interpolating model to be achieved with a smaller hidden layer size while keeping the overall model size the same. This is a richer evaluation than supervised examples (see [1]) since we display that certain layers of an autoencoder are very influential for the reconstruction results and the double descent curve.

[1] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021.

Dear reviewer, we appreciate the time and effort you have dedicated to reviewing our paper. As the discussion period is limited, we kindly request that the reviewer evaluate the new information provided in the rebuttal. We are eager to improve our paper and resolve all the concerns raised by the reviewer. If there are any remaining concerns that have not been addressed, we would be happy to provide further explanations.