We thank this referee for their careful, detailed feedback. We are pleased to see that they found our experiments “conducted with high quality”, the results “insightful”, “useful” and “relevant” to the community, and that they appreciated our discussion on limitations as a “role-model” on how these should be discussed. We now address the weaknesses identified.

W1. Readability. We acknowledge that our study involves a certain amount of previous background, which is deep and appears across several papers. We have done our best to summarize it in the clearest way possible given the page restriction. We understand that thismay be insufficient for the unfamiliar reader, we will improve the readability and accessibility by including an extended version of the background in the supplementary material, which aims to be self-contained and to serve as an introduction to this line of work. We will also revise the structure of the paper to minimize the number of cross-references, and to streamline some of the discussions.

W2. Hypothesis in permutation tests: We thank the reviewer for pointing this out, indeed there was a typo and insufficies in the way we had described the hypothesis. We would like to rewrite the beginning of this section to: Line 221: * $...$ we conduct a non-parametric permutation-type hypothesis test for the assumption that the partial correlation, i.e., the correlation coefficient between the regression residuals, is equal to zero. Transferred to our setting, the null hypothesis implies that the correlation observed between generalization and PH dimensions is explained by the influence of other hyperparameters.* Lines 228-230, where we explain how to interpret the p-values from the permutation test, will be rewritten to: Recall that a p-value lower than 0.05 implies the rejection of the null hypothesis, or equivalently, it implies that there is a correlation between PH dimension and generalization gap that cannot be explained by the influence of the hyperparameters; whereas a p-value bigger than 0.05 implies a significant influence of the corresponding hyperparameter in the apparent correlation. We have also included an interpretation of these results, motivated by the answer to reviewer garn, which can be consulted in the corresponding answer.

W3. Update abstract: We acknowledge that the abstract might have created an expectation for work of a different nature, so we are open to change it.

We now address the question posed by the reviewer.

Q1. We thank the reviewer for this question. We implement two hypothesis tests in our work: one to report the statistical significance of the partial correlation test and the other for the conditional independence test.

• The null hypothesis of the former is that there is an influence of the hyperparameters in the observed correlation between generalization and PH dimensions. The null hypothesis is specified above. In this case we are not considering the correlation between generalization gap and PH dimensions directly, we are rather trying to evaluate the effect of the learning rate on it.
• Concerning the conditional independence test, the null and alternative hypotheses are illustrated in the Appendices, Section F, Figures 3 and 4. Here the null hypothesis is conditional independence between PH dimensions and generalization gap conditioned on the hyperparameters of the network. We could move the images to the main text if this makes it more clear.

Q2. We now cover all the suggestions for improvement.

• We have fixed all typos and redundant information, and would like to reiterate our thanks to the reviewer for this very careful read of our work, which was really helpful in improving it.
• We agree with the suggestion about the composed loss function, and will change it in an updated version of the paper.
• We notice that we might have confused appendices and supplementary material; we will refer to the text coming after the references as appendices instead of supplementary material.
• We thank the reviewer for this observation, indeed, we forgot to specify the metric. We would like to change the sentence to: Let \\mathbf{x} \= \\{x\_1, \\, \\dots, \\, x\_n\\} be a finite subset of a metric space $(X, \\rho)$.
• We will include the definition of $w’$ in Eq. (2) and will think of a suitable change of notation for the metric.
• The justification for our architectural choices in the experimental section is mainly that we are closely imitating the configuration of the experiments in Dupuis et al. $2023$ and Birdal et al. $2021$ , to make the comparison fair, and we tried to justify them in the Limitations section, within Models and Settings. This involves the networks, batch sizes, learning rates and datasets in the correlation analysis. The cross-entropy loss is used because we need it to be continuous in both variables (the parameter and the evaluation point) and uniformly bounded (see Assumption 3.1 in Dupuis et al. $2023$ ). The other architectural choice is the “standard cnn” that is designed to showcase double descent, which we implemented because we wanted to test if this affected the PH dimension.
• We thank the reviewer for this observation, as we have realized that this sentence is a bit misleading, and will rewrite it in next iterations of the paper. We have the same stopping criterion than Dupuis et al $2023$ : for classification, achieving 100% train accuracy; for regression, having the relative training loss to be smaller than 0.5% within 2000 iterations.
• We have completely changed the line 233, we included the new one in a reply above.
• We thank the reviewer for the suggestion of bolding relevant results in tables, this is already reflected in the revised table 1 of the global response and will be implemented for all other tables.
• We thank the reviewer for the observation on the notation for probabilities in the CMI, we will change it to avoid confusion.

We thank this reviewer for the careful review provided. We are very happy to see that they also appreciated the importance of evaluating existing work to “build upon solid foundations”. We would like to begin by addressing the weaknesses they identify.

1. **Discrepancy between our correlation coefficients and those in Dupuis et al. $2023$ **: We appreciate the reviewer's observation, which prompted us to revise the table. The updated table, along with detailed adjustments and answers, is included in the global response. Additionally, we have provided plots for the correlation experiment similar to those in Dupuis et al. $2023$ to clarify the coefficients.
2. We acknowledge that our interpretations may have been conservative. Here is a summary, which will be included in the revised paper:
   • Extension of correlation analysis to other hyperparameter:
     1. Significant correlations with learning rate/batch size point out a potential confounding effect in the observed correlation between fractal dimension and generalization gap, which we closely examine in further tests.
     2. The norm is a simple but flawed generalization quantifier and is significantly easier to compute than the PH dimensions. Observing higher correlations with the norm underscores the need to justify the use of more complex and less effective measures, which has not been addressed before.
   • Partial correlation: there is a significant partial correlation conditioned on the learning rate, for fixed batch sizes, in most cases for the Euclidean PH dimension. This suggests that the observed correlation between generalization and PH dimension could be attributed to a correlation between PH dimension and learning rate. The trend for the loss-based dimension is less consistent, but it appears to be more robust to the partial correlation test.
   • Conditional independence test: different results arise depending on the training dataset, pointing out that this might be another limitation when using PH dimensions as generalization measures.
   • The adversarial initializations indicate that PH dimension fails to predict generalization. Models with poor generalization (higher generalization gap) due to adversarial training, show lower fractal dimensions. This directly contradicts the proposed theory that PH dimension should be lower for better generalizing models. A clear plot illustrating this effect is included in the global answer.
   • Finally, the double descent manifesting with PH dimensions suggests some connection between the convergence properties of models in the critical region of double descent with respect to the model width, itself a poorly understood phenomenon, and the Euclidean persistent homology dimension. It also shows the failure of PH dimension to correlate with generalization gap, which stays monotonic in this region, while PH dimension not.
3. We agree the number of seeds in the adversarial initialization (10) was limited and have increased it to 30 seeds for AlexNet CIFAR10 and FC MNIST. Due to computational constraints we were only able to increase CNN CIFAR10 to 20 seeds. We refer to the global response for the results and interpretation in the corresponding figure. We hope this will help enhance the results.

We now answer the questions posed.

• We believe there are two main reasons for our findings contradicting the generalization bounds:
  1. The assumptions required to prove generalization bounds and their implications on architecture choices and training data remain unclear. Different studies involve various technical assumptions about optimization trajectories and loss functions that may not be met in practice.
  2. The primary reason for the discrepancy is likely the term in the generalization bound involving conditional mutual information between the training data and the optimization trajectory, which is complex and less understood compared to the fractal dimension term. It is unclear if this term can be empirically tested or estimated. The assumption that the conditional mutual information does not dominate the bound has not been proven and has been scarcely explored. We believe some experiments may enter a regime where this term dominates, disrupting the expected correlation.
• PH Dimension Estimations: The reviewer's observation motivated us to implement suggested experiments. We refer to the global answer for further details.
• We acknowledge that the line concerning the integration of persistent homology into neural network training is incorrect and due to an oversight in our writing, we will rectify it in the revised version of the paper. We also thank the reviewer for providing useful references that we will incorporate when expanding on this explanation.
• We appreciate the question about the CMI and would like to elaborate on the brief outline previously provided.
  1. From the definition on page 6, the CMI $I(X; Y \vert Z)$ vanishes if and only if $X \perp Y \vert Z$, i.e. $X$ and $Y$ are conditionally independent given $Z$. Thus, while changes in $X$ might seem linked to changes in $Y$, the CMI allows us to isolate the effect of $Z$ and establish whether $X$ and $Y$ are independent when $Z$ is fixed.
  2. For our test, $X$ is PH dimension, $Y$ generalization error, and $Z$ the neural network hyperparameters. Given the discrete nature of our selected hyperparameters, we empirically determine the probability density functions.
  3. To assess the significance of the computed CMI, we generate a null distribution for the CMI under local permutations of $X$ or $Y$ for fixed hyperparameter values. The null hypothesis implies that $X$ and $Y$ are conditionally independent, so the CMI is invariant to permutations. We reject the assumption of conditional independence if the computed CMI lies in the extremes of the simulated null distribution.

We thank this reviewer for their feedback and are happy to see that they found our conclusions “well-supported” and the text “well-written and organized.”

We would like to first address the concern regarding the clarity of the PH dimension definition. We acknowledge that the concept of fractal dimension, particularly in relation to PH, is rather involved. Given the 9-page limit, our background section may not have provided sufficient detail for readers unfamiliar with this topic. To address this, we will include an extended background section in the appendix, making our work more self-contained and providing an introduction to the intersection of work on fractal dimension and generalization properties of deep networks.

Now, we would like to answer the specific questions raised.

1. In Table 1, the correlation coefficients could have different signs in the same line. Why would this happen? Does this come from different models and data? We appreciate the reviewer highlighting this point, and would like to elaborate further to clarify it.

   • Under closer inspection, we noticed that for models trained with CHD we were correlating with training loss minus test loss, rather than the absolute value of this difference, which is what Dupuis et al $2023$ use. Correlating with the absolute value changes the sign for the whole rows corresponding to CHD.
   • We have provided an updated table in the global answer and would like to further clarify that for models trained with CHD, the correlation is computed between the PH dimensions and the loss gap, whereas for models trained with MNIST, the correlation is computed between the PH dimensions and the accuracy gap as in Dupuis et al. $2023$ .
   • Having different signs in the correlations with different measures just means that the relation is direct or inverse. We have noticed in our experiments that the norm correlates positively with the CHD but negatively for the classification models. This could be due to the ranges of hyperparameters selected being more limited in the CHD experiments. We also note that the sign of the correlation flips for the LB ratio between regression and classification.
   • Overall the important observation of this table is regarding the absolute value of the correlation coefficient and whether it is closer to 1, indicating a stronger correlation, or not.

2. In Table 2, how to explain the influence of the batch size on the p-value?
   There are several things to be said about this influence.

   • In Table 1, a significant correlation between the generalization gap and the learning rate/batch size ratio suggests potential confounding effects. This is also visible in Figures 1 and 2 from Dupuis et al. $2023$ , where different correlation patterns can be seen depending on the batch size (different markers in the plots) and learning rate (color).
   • Previous methods to take into account this effect involved calculating the mean of the granulated Kendall rank coefficients for learning rate and batch size. To compute these granulated Kendall coefficients, the Kendall rank correlation for each fixed value of learning rate (or batch size) and seed is computed and then averaged over all possible values. While valid, this approach masks the observed distinct correlation patterns for fixed batch sizes and learning rates.
   • To address this, we developed a partial correlation test. This test is specifically designed to assess if the correlation observed between two variables can be explained by a common correlation with a third variable. Given the different patterns of correlation differ with batch size, we felt compelled to divide amongst these to evaluate partial correlation. The same reasoning applies to the conditional independence test.
   • Concerning the p-values for different batch sizes in Table 2 (in the updated version of the paper, the values with p > 0.05, indicating that the observed correlation between fractal dimension and generalization is explained by a common correlation with learning rate, will be bolded): a clear trend for the Euclidean PH dimension is that this explanation holds for almost all batch sizes, particularly the larger ones. The trend for the loss-based fractal dimension is not systematic. For FCN5 + MNIST, it fails for larger batch sizes, whereas for FCN5 + CHD, it fails for the smaller batch sizes. The correlation is more resilient in the FCN7 models.

3. I cannot figure out how Figure 1 manifests double descent.
   We acknowledge that the double descent behavior was not very clear from the previous plots. The reason for this is that we had to slightly modify the original procedure from Nakkiran et al. $2021$ to avoid learning rate annealing and batch normalization, as prescribed by the assumptions of Dupuis et al. $2023$ and Birdal et al. $2021$ , which involve training with a constant learning rate and without batch normalization. These changes made the double descent, already a difficult to produce effect, less clear. We managed to produce a clearer plot by re-running these experiments adjusting the constant learning rate to 0.01, which is included in the global answer. The new plots clearly show double descent manifesting in test accuracy and Euclidean PH dimension.

We thank the reviewer for their time and feedback on our work. We are pleased to see that they found our evaluation “critical;” our statistical analysis “rigorous”, and our demonstration of the shortcomings of the existing theory “convincing”. We also appreciate the recognition of the opportunities for further investigation stemming from our experiments.

While it was challenging to identify concrete areas for improvement given the reviewer did not provide any detailed questions or requests, we would like to address their main point of criticism, namely, the lack of theoretical results in our paper.

Indeed, our work is experimental: it makes the important contribution of uncovering new, negative insights on an existing theory, which has gained considerable traction in high profile outlets since its inception (according to Google Scholar, around 100 citations in total). We believe that scrutinizing existing theories, especially in rapidly evolving fields like ML, is crucial to ensure that subsequent knowledge is built on a solid foundation. The experimental focus of our approach is justified by three main points:

1. The nature of the problem is partly empirical, with the theory making predictions that can be directly tested. By replicating the experiments and developing further tests, as we have done in our paper, we provide clear and actionable insights into the theory’s applicability and limitations.
2. Our findings provide strong empirical evidence that challenges the current understanding. A robust experimental foundation is necessary to guide theoretical advancements, which might otherwise lack practical relevance or be misdirected.
3. As the reviewer noted, our results open up new opportunities for further research. We believe the empirical evidence presented in our paper could inspire and inform subsequent theoretical work, rectifying existing results or developing new ones.

That being said, we understand that our presentation may have created an expectation of new theoretical results. To rectify this, we are open to updating the abstract to emphasize the empirical and experimental nature of our study, as also suggested by another reviewer.

We thank the reviewer once again for their consideration and ask to provide concrete questions we can address in our rebuttal.