Just How Flexible are Neural Networks in Practice?

Dear Reviewer,

Thank you for your thorough and insightful review of our submission. We appreciate the time you've dedicated to evaluating our work and providing feedback. We address your points below:

1. Measurement of EDD:
   a. Multiple random seeds: We appreciate the reviewer's input and concur with their observation. To ensure the robustness of our results to random initialization, in the original paper, where our networks did not successfully reach 100% training accuracy, we reran the networks three more times with different random seeds. This was done to ensure that it not fit all the examples. We thank you for highlighting this aspect
   b. Consideration of optimization: We agree that we want our study to capture the properties of different optimizers, but in our experiments, none of the optimizers we tried got caught in saddle points. We chose to check the Hessian spectrum because practitioners train with different optimizers, architectures, datasets, etc, for different numbers of epochs, and we need a way to ensure fair comparison. Fixing the number of epochs, therefore, would not yield a fair comparison, and simply terminating when the gradient norm is sufficiently low may not yield a fair comparison either because we could end up at a saddle point in the middle of training or a flat region which we would have escaped during the length of a standard training routine, but a gradient-norm based stopping condition would cause us to terminate the training routine very early.

2. Deeper Investigation of Observed Phenomena: We struggled to keep the experiments contained in this paper within the allotted space as is. Many of these areas will inspire deeper investigations, and in several cases, those deeper investigations will require their own stand-alone research papers. This paper aims to lay out the broad contours of an area that has been understudied, discovering unexplained phenomena. We believe that this investigation will contribute significantly to the conference and give the community food for thought.

3. About the Conclusions:
   a. Larger EDC for tabular datasets is due to simpler tasks: There are several ways to define simplicity level, with pros and cons. However, controlling different factors that may impact the EDC is essential. Inspired by the reviewer's question, we conducted an experiment where we tried to control for properties such as class numbers. We transformed several datasets into binary classification problems, standardizing the number of classes. This modification allowed us to isolate the influence of the input distribution on EDC more accurately. Our results revealed that the EDC across various image datasets is smaller than tabular datasets (see full details in the the paper). Therefore, we can assert that models can accommodate more examples for each model size for tabular datasets and, by some criteria, can be considered 'simpler'.
   b. CNN has a larger EDC than MLP: CNNs are recognized for their superior generalization on vision problems, attributed to their robust inductive bias for spatial relationships and locality (Taye MM, 2023). However, our work observed that CNNs exhibit greater parameter efficiency even on randomly labeled data, suggesting that their enhanced capacity extends beyond better generalization alone. Regarding tabular data, in response to the reviewer's inquiry, we investigated the EDC of MLP and CNNs for tabular datasets. Our findings, detailed in the updated version of our paper, reveal that the EDC is similar for the MLP and CNN. This outcome is unsurprising, as tabular datasets lack the spatial relationships and locality found in computer vision problems. Consequently, we did not anticipate a CNN advantage in fitting more examples than MLP in this context.

4. Experimental Settings and Methodologies:
   a. Clarify the optimizer settings: For our experiments, unless otherwise specified, we utilized SGD for training. We highlighted this detail in the updated version.
   b. Scaling filter sizes: Yes, we increased the filter size proportionally with the input sizes.

5. Fitting labels to 100% influenced the experiments: In principle, it's true, and there might be outliers. During our experiments, we also checked our analysis without requiring 100% accuracy, allowing for a bit less (in order to accommodate outliers), which didn't significantly change our results.

Questions:
Regarding the Figures - Yes, the results imply that scaling the width leads to the best generalization.

Thank you again for your thoughtful review. We made a significant effort to address your feedback, and we would appreciate you raising your score in light of our response. Please let us know if you have additional questions we can address.

References

• Taye MM. Theoretical Understanding of Convolutional Neural Network: Concepts, Architectures, Applications, Future Directions. Computation. 2023

Dear Reviewer 14iz,

We appreciate your comments and thoughtful considerations regarding the theoretical aspects of our paper. We acknowledge the importance of theoretical exploration in the realm of deep learning; however, we intentionally adopted a different approach in this work.

Our primary motivation was to explore the empirical side of network capacity rather than relying solely on theoretical frameworks such as PAC-Bayes and VC dimension. Historically, attempts to assess the capacity of deep networks have often been theoretical and, at times, not entirely aligned with empirical observations. We aimed to bridge this gap by providing an empirical perspective on network capacity.

While we recognize the value of predefined hypotheses derived from established theory, our focus on drawing conclusions post-results stems from our commitment to an experiment-driven methodology. This approach enhances transparency and clarity in the progress of scientific inquiry. The "double descent phenomenon" (Preetum, et al. 2021) is a good example; its impact on the community was initially observed empirically, and the (partial) theoretical understanding followed in subsequent works.

Regarding the relevance of model architectures to random labels and frequent references to linear models, as mentioned in our paper, many theoretical works provide vague and impractical approximation theory bounds, and this is why it is so important to examine it in practice. We agree that modern neural networks do not align with traditional models, and our paper precisely demonstrates that the practical flexibility of modern networks is much higher than linear models. Concerning random labels, based on previous works (Zhang et al., 2016), we sought to distinguish between the input distribution and the label distribution, exploring how the network can fit the data regardless of the label’s distribution. Moreover, as demonstrated in our paper, the ability of a neural network to fit many more correctly labeled samples than incorrectly labeled samples predicts generalization.

Concerning Section 5, we acknowledge your feedback on its current format. We will work on improving the presentation to ensure a more cohesive exploration.

Questions:

EDC Metric Explanation:

Defining the capacity of a model encompasses various perspectives. Intuitively, it refers to model complexity—a measure of how intricate a pattern or relationship a model can express. In simpler terms, it quantifies the number of examples a model can learn. While VC dimension is one method to define capacity, it may exhibit a considerable gap from the model's actual ability to fit the data and, therefore, not used in practice. In our work, we introduce EDC as an alternative measure for empirical capacity.

In essence, EDC is the maximum number of training data points a model can fit, representing its empirical capacity—the number of patterns it can learn. To elaborate, EDC is defined as the largest sample size where the model can perfectly fit randomly sampled training subsets of that size when optimized until convergence.

As explained in our paper, to compute the EDC, we adopt an iterative approach for each network size. Initially, we start with a small number of samples and train the model. Post-training, we verify if the model has reached 100% training accuracy. If met, we re-initialize the model with random initialization and train it again on a larger number of samples randomly drawn from the entire dataset. This process iteratively increases the number of samples in each iteration until the model fails to fit all the training samples perfectly. The largest sample size where the model achieves a perfect fit is taken as the EDC for that particular network size. Please refer to section 3 in our paper for a detailed explanation.

Flexibility:

We define flexibility as the model's ability to fit the data in practice. Please refer to the paragraph titled "Capacity, flexibility, expressiveness, and complexity? What’s the difference?" on page 3 for a paraphrased explanation.

Thank you for your valuable insights, and we are committed to refining the paper to address these concerns while maintaining the empirical focus that distinguishes our work. We would greatly appreciate it if you raised your score in light of our response. Please let us know if you have additional questions we can address.

References:

Nakkiran, Preetum, et al. "Deep double descent: Where bigger models and more data hurt." Journal of Statistical Mechanics: Theory and Experiment 2021.12 (2021)

Dear Reviewer ftNw,

Thank you for your detailed evaluation of our submission. We appreciate your comments and will address your concerns and questions below:

1. Comparison with Effective Model Complexity (EMC): We appreciate the reviewer's insight on the conceptual resemblance between our Empirical Data Capacity (EDC) and the Effective Model Complexity (EMC) as explored by Nakkiran et al. (2021) and agree that we should focus on our unique contribution to the empirical investigation of the EDC: Nakkiran et al.'s work defines EMC primarily to dissect the double-descent phenomenon, with a focus on the training procedure and the train/test error of the data. Their exploration is deeply intertwined with the dynamics of under-parameterized and over-parameterized regimes, particularly in relation to test and train error. Conversely, our investigation of EDC takes a distinctive route. We concentrate on the empirical study of neural networks' capacity to fit training data, independent of generalization or double-descent implications. Our methodology for computing EDC includes rigorous measures to assure independence in each evaluation iteration and to prevent under-training. These measures include monitoring gradient norms, stabilizing training loss, and examining the loss Hessian. These methodological distinctions underscore that EDC, while sharing the same concept as EMC, diverges in application, implications, and the questions that it answers. Our focus even extends beyond merely determining the largest subset of data a model can interpolate; it delves into the empirical factors influencing this capacity under varying conditions like architecture and optimizer choices. In light of this discussion, we agree with the reviewer's suggestion and will incorporate a detailed comparison between EDC and EMC in the revised version of our manuscript. This addition will further clarify the unique aspects of our study and its contribution to the field.

2. Experimental Details and Fair Comparison of Architectures: We apologize for any confusion caused by comparing the average EDC of different architectures in the paper. To provide clarification, for all figures where we plotted the average logarithmic EDC, we have included the complete, unaveraged results in the appendix. Specifically, Figure 5 in the appendix offers a detailed presentation of the EDC across the full spectrum of parameter counts for various models, including VITs, CNNs, and MLP. Our decision to feature averaged results in the main text was driven by the consistency of the observed phenomena across different model sizes, where no significant differences were noted between smaller and larger models. This choice aimed to balance clarity and conciseness in the paper's main body while providing extensive data in the appendix for readers seeking more detailed insights. Regarding the reviewer’s concern about the impact of scaling laws on the comparison between different models, In our study, we scaled ViTs following the scaling laws proposed by Zhai et al., 2022. These laws advocate for scaling all aspects—depth, width, MLP width, and patch size—simultaneously and uniformly. We have expanded our revised results to include different scaling laws for VIT to address the reviewer's concerns and further elaborate on this aspect. Based on the reviewer's suggestion, we used both SoViT from Ibrahim et al. (2023) and laws where the number of encoder blocks (depth) and the dimensionality of patch embeddings and self-attention (width) in the ViT are scaled separately. Figure 22 in the appendix of our updated paper demonstrates that scaling each dimension independently leads to suboptimal results in line with our observations from the EfficientNet experiments and that SoViT has slightly better results to the ones from Zhai et al., 2022. This expanded analysis strengthens our original findings and offers a more nuanced perspective on how different scaling strategies influence the EDC of various architectures. This elaboration adequately addresses the reviewer's concerns and provides a more comprehensive understanding of our comparative analysis.

3. Impact of ReLU on EDC: Thank you for pointing out this concern, and we would share your concern that exploding and vanishing gradient problems could in fact cause lower EDC if we were not successfully minimizing the loss function. Whereas ReLUs were adopted to prevent exploding or vanishing gradient problems that can cause poor conditioning and hamper optimization, we actually ensure that we obtain a minimum of the loss function in each experiment. That is, we find that ReLUs boost EDC even when we would be able to optimize successfully and converge to minima using competing activation functions.

=>