The Directionality of Optimization Trajectories in Neural Networks

 

Questions:

“cosine similarity exhibits a monotonically decreasing trend in both the horizontal and vertical directions …. introduces nothing new”

Our point is surely not to just say cosine similarity is monotonically decreasing. Rather, the information lies in being able to gauge how fast or slow this decrease in cosine similarity is.

And indeed, then trajectory maps are not just about a single row or column, but rather how it captures the information about the pairwise inter-relations of each of the points in the trajectory.

 

“the observed trends are linked to variations in model size rather than performance across different metrics, …. Could you elaborate further on the insights provided by Figure 7.a in this regard? figures 7.b and 7.c "fail to capture the expected monotonic trend ”

• Many of the other angular and norm-based metrics could also be expected to scale with model size, but their trends with increasing scale are non-monotonic. This is in opposition to the apex angle at the peak which has a clean monotonic trend.

• Even if it is just a more reliable indicator of model size, there is also value in that.

• And, yes these figures also serve to show that some of the most natural complexity measures and which are also related to the extent of feature learning (distance from initialization) do not work as well in this setting — which is not entirely evident from prior work

 

Adam and SGD, in regards to LLMs

We would like to offer insights into this, however SGD is well-known to work worse in the LM setting. However, in our submission we do have experiments with AdamW and SGD on a shared setting of ResNet50 ImageNet classification that enables comparison across optimizers. For example, we see shared similarities across optimizers in their trajectory maps in appendices E.3 and F.2, with regards to the effect of weight decay.

 

“What does the complexity measure refer to in this context?”

It is in the generalization sense, i.e., complexity of the function/model class.

 

---

We hope the above answers the questions you had, but please let us know if there is something you would like us to further elaborate. We'd be more than happy to address. Besides, we also hope that in light of the above response, you will give your score a second thought. Thanks!

We would like to thank you for your feedback and for voicing your concerns. We understand that you see the promise of our “relatively thorough validation” that can help towards “useful information about the optimization process and the network’s behavior”. There is merit in some of the raised questions which we seek to address below, and we would also like to offer clarifications at other places so that we are on the same page. We hope you will allow us the chance.

 

“connections with the existing understanding of the optimization of neural network”

We don’t think this is a fair assessment of our work.

• We draw our connections to the recently uncovered Edge-of-Stability (Cohen et al, 21) phenomenon, and adding evidence to its interplay with momentum and weight decay.
• We also showcase how many of the optimization hyperparameters have choices well and beyond their sole intended purpose (Andruishchenko et. al., 2023; Zhang et al. 2023. This can be seen clearly by the directional exploration encouraged by both momentum and weight decay.
• Our work also ties in intimately with the emerging paradigm of implicit regularization (Dherin et al., 2021; Gunasekar et. al., 2018), but from the perspective of directionality.

 

“projecting high-dimensional information into lower-dimensional space may result in significant information loss”

Trajectory maps, as you have mentioned yourself, transform extremely high-dimensional information into a manageable form. However, there is nothing that prevents us from honing into some specific parts of this information, when needed.

• Namely, we already allude in the main text and also discuss in detail in Appendix H that trajectory maps can also be defined at the level of layers or groups of parameters. So one would track simultaneously multiple trajectory maps that each focus on certain subsets of the parameter space.

• This can indeed over some additional insights or finer details beyond those available in the pan-network trajectory map, as evident in Figure 74. Gathering information about fringe parameters (like biases) in the pan-network trajectory map is not the most suitable place.

Importantly, however, the trajectory map successfully represents the shared information across the majority of parameter groups, as is evident in Figure 73 (as well as Figure 75). Thus we can be confident that the crucial information is being retained even in the simple pan-network trajectory maps.

 

“redundancy observed in low-dimensional representations may not necessarily imply that iterations during the later stages of training are redundant or unnecessary”

Yes, and neither do we ever claim that late-stage training is unnecessary or trivially redundant. This is also precisely why in Section 4, under Question 1 Initial Thoughts, we explicitly say that “A tempting first guess is that perhaps the trajectories are just linear in this phase. But, if that were the case, we would be just using line-search”.

• We dedicate the whole of Sections 4.1 and 4.2 to understand the nature of directional movement that occurs in this late-stage. In particular, we explain that a lot of the directional movement occurs in the form of oscillations and noise. And, of course, one cannot just omit this late phase.

• However, our crucial point in leveraging redundancy relies on the fact that the overall trajectory direction (as seen from the origin) is largely maintained after some point in training (which is what we call directional saturation). And moreover, certain interventions, such as only training the normalization layers, preserve this macro-level trajectory direction, as is evident from our experiments in Section 6.

In brief, we are saying that there is redundancy inherent at the macro-scale which can be utilized to improve the late-stage training process.

 

“While the image on the right side of Figure 8 presents straightforward patterns, these patterns do not clearly align with the left side of the figure”

The left subfigure is indeed not the cleanest figure. We now provide in Figure 78, a more clear-cut demonstration of this fact.

Discussion on random walks:

We have added similarity in Trajectory Maps over random walks: The added Figure 11 visualizes the difference between the trajectory map for the ResNet50 experiments and that obtained for a random walk, with the same learning rate schedule as used in practice.

• Interestingly, the biggest difference is the further influence of the initial training phase as opposed to the random walk scenario.

• We agree that this view of the trajectory map offers added insights; and we are happy to add similar plots for the rest of the experiments. In fact, we did think of this, but we chose not to do so since the random walk is a baseline model — one that makes assumptions that are perhaps not entirely accurate (e.g., isotropic covariance for the steps). Thus, as a foray into this line of work, we went for the trajectory map as it is – but, of course, it is not the definitive representation of a trajectory :)

 

Expected MDS:

That’s a great question, thanks. We have added some calculations in Appendix C.2 that do for a simple constant schedule, and whose corresponding trajectory map is the one in Figure 12.

• This analysis in C.2 is extendible to more general schedules, such as stepwise schedules. The general idea is to compute MDS between every pair of phases through similar calculations (which can get a bit lengthy and out of scope). When the later phases employ smaller step sizes, this leads to added MDS, as is also pictorially evident in the dark grids in Trajectory maps that correspond to the later parts of training.
• We use the precise (stepwise) learning rate scheduler used when training ResNet50, with the same hyperparameters. It would also be interesting to carry out a comparison with the exact lengths of steps, as in the grafting literature in optimization [1] in future work, as well as more broadly explore trajectory maps in this context.

[1] Agarwal et. al., 2022: Learning Rate Grafting: Transferability of Optimizer Tuning

We thank you for your thorough review and your appreciation of our work. In particular, we are grateful that we have been able to communicate the originality of our perspective as well as its significance. While we are glad to hear that we were able to address “almost all of the pertinent concerns that a reader may raise”, we would also like to iron out the last few wrinkles:

 

Effect of sampling rate on Fig 4a (angle between checkpoints), and more broadly.

This is a valid concern. To address this, in Appendix L, we present a detailed study of the effect of sampling rate, whose main findings are:

• Firstly, at a broad level, comparing Figures 85 (a) and 85 (b), the obtuse nature of the angle between successive steps can still be seen.
• The first 30 epochs have a very similar trend across both levels of granularities, and the setting with both momentum and weight decay enabled has on average $\sim 18^\circ$ bigger angle between successive samples in this duration of 30 epochs, than in the setting where both momentum and weight decay are absent.
• The last 30 epochs show a slight deviation between the granularities. Here the finer granularity, lets us see that the latter setting (without momentum and weight decay) results in a slightly bigger angle albeit $~\sim 0.85^\circ$ than in the former setting of enabling both these hyperparameters.
• While the above observation in the last 30 epochs does echo your intuition, given this significant gap between the angular differences across these settings ($18^\circ$ vs $-0.85^\circ$), our larger finding of momentum and weight decay leading to an overall larger directional movement stands.

Additional Comments:

• Lastly, we also recommend you to have a look at Figure 84 where both granularities are shown in the same plot for the prototypical experimental setting of having both momentum and weight decay.

 

Early Training Period: Trajectory Map and Angles b/w adjacent samples.

Interesting remark. In Appendix L.3, we perform this suggested experiment and detail the results.

Observations:

• The TMap in Figure 86 (a) at this ‘high resolution’ bears a coarse-grained resemblance to the (tiny) 5x5 grid at the top-left of Figure 2 (left). This seems to suggest that some trajectory characteristics could repeat at different scales.
• While TMap essentially presents an ‘aerial’ view of the trajectory (origin will be high up in the loss landscape), the angle between adjacent samples on the other hand presents a ‘ground’/surface level view of the trajectory.
• The latter trend in Figure 86 (b) shows the trend of the angle formed between every pair adjacent updates. The raw trend is a bit noisy, fitting in with your view about the less stable nature of the early training period, so we additionally visualize a smoothened version of it.

 

Effect of Different learning rate schedules.

Thanks for this question. We have added new experiments with cosine, linear, and polynomial schedulers in Appendix K.

• In subsection K.1, we explore the effect on trajectory maps, where we find that the phase in training where directional movement saturates is common to all these settings.
• In subsection K.2, we study how the angle between adjacent updates changes when using these different learning rate schedulers as opposed to the stepwise learning rate scheduler used for Figure 4a. We find that while there are no stepwise transitions (which would understandably be specific to stepwise scheduler), the angles remain obtuse for a significant part of training as well as that the setting with both momentum and weight decay enabled results in a more obtuse angle, especially in the initial training phase (~20-30 epochs).

Overall, given these similarities in results, we can rest assured that our findings are general and not sensitive to the choice of learning rate schedulers.

 

Relation between methods of representational similarity and trajectory maps:

That’s an excellent comment, and thanks for sharing this very interesting work. There are surely similarities and avenues for cross-fertilisation in both these research directions. Some interesting threads at the top of our heads:

• Kernel matrices, like trajectory maps in our work, and CKA for representations are the key objects of focus in these directions, so there are definitely parallels.
• It would be interesting to see how well trajectory-based quantities can indicate the evolution of representations for a given network, while being data-free. Of course, some interesting questions might explicitly hinge on the given inputs, but it would be worthwhile to investigate the extent of representational dependence contained within the trajectories.
• In the opposite direction, it would make sense to explore a different, more geometry-aware, metric to compare the trajectories, than the plain Euclidean that we use here.

 

Section 4

 

“"Initial Thoughts" are confounded by the lack of training loss”

This should now be addressed by the loss curves in each of the scenarios. The particular figure of relevance here is Figure 16. As can be seen, the loss is slowly but surely going down. Also, from Figure 3b, we can see that the trajectory length is still increasing. The gradient norm in this setting is also non-zero (actually much bigger), so technically it won’t quite yet be at a local minimum.

 

“Fig. 4 the updates are oscillating, but this might be expected for (relatively) small batch sizes”

That’s a valid concern. In Appendix M, we now showcase ImageNet experiments with a much larger batch size 16384, while previously we had a batch size of 256. When we increase the batch size, we also increase the learning rate by the same linear factor and warm it up in the first 5 epochs (as suggested in Goyal et al. 2017).

We can draw the following observations about Figure 87:

• Figure 87 (a) and 87 (b) match quite closely, despite the significant differences in batch size.
• In both cases, we see obtuse angles between updates for a significant part of training, and also that the presence of momentum and weight decay leads to still bigger angles.

This shows that the oscillation also occurs at large batch sizes.

 

“Certainly the stable Hessian assumption seems unfounded in the early stage of training”

We would like to clarify ourselves.

• We are not claiming the onset of obtuse angles from the very beginning of training, but after some time into training (like around 5-10 epochs in Figure 4a). Thus, we are more near the vicinity of EoS.

• Besides, historically a lot of intuition in NNs has been gained from toy models given complexity. E.g. noisy quadratic model (Zhang et al. 2019, https://arxiv.org/abs/1907.04164) predicts batch size-LR scaling. Even infinite width (Jacot et al. 2018) is "simplified" but provides a lot of intuition for practical NN training dynamics

 

“EoS is for gradient descent … no?”

Past work [1] and similar show that it is possible to extend EoS to mini-batch gradient setting, but with a slightly generalized EoS criterion that defaults to 2/\eta \approx \lambda_max in full-gradient setting.

[1] Lee & Jang et al., ICLR 2023: A new characterization of the edge of stability based on a sharpness measure aware of batch gradient distribution

 

---

 

Section 5

 

Directional measures predict.

We have switched to “correlate with”, but are also happy to tone it in a different way which you prefer.

 

Besides there is already work on practical generalization measures (https://arxiv.org/abs/1912.02178) and it would be nice to talk about how the stronger ones there connect

We appreciate the observation regarding existing work on practical generalization measures (https://arxiv.org/abs/1912.02178) and agree that it would be valuable to discuss how the stronger measures identified there relate to our findings. We will incorporate this discussion in the revised version.

• That said, we wish to clarify that our focus on the efficacy of the apex angle at the origin is primarily a side observation rather than a core claim or contribution of our work.

• Furthermore, we would like to highlight that many of the generalization measures explored in prior work have not been thoroughly studied in the context of language modeling.

• Among these, the sharpness-based measures, which perform particularly well in the cited work, are not data-free. In contrast, the quantities we consider here, while data-dependent in nature, are effectively data-free in practice.

 

---

 

Section 6

 

It would be nice to have experiments on Pythia here as well.

While we would love to pursue this, conducting training experiments with Pythia checkpoints would require a substantial compute budget, which is unfortunately beyond our current resources. That said, we believe this line of inquiry is an exciting and valuable offshoot that deserves a dedicated and thorough investigation. The promising results we have obtained on CIFAR-10 and ImageNet, combined with the generalizable nature of trajectory maps, make extending this work to Pythia a compelling direction for future research.

 

---

Thank you once again for your thoughtful suggestions. We hope we have been able to address your remaining concerns. Should you have more questions or concerns, we remain at your disposal.

Thank you for your very comprehensive review. We truly appreciate this!

Below, we address each of your comments section by section, and point by point. Some of the raised comments have already been incorporated to the submission pdf, so for brevity we’ll omit further discussion.

---

Overall

 

“ it's a bit difficult to make claims about the importance or role of hyperparameters like momentum or weight decay with only two settings”

While the main text includes only a handful of settings for the sake of readability, we would like to emphasize that our supplementary material already contains an extensive analysis of hyperparameters with multiple settings. For clarity, we briefly outline them here:

• Section E.3 on Effect of Weight Decay with AdamW, ResNet50 contains 5 different settings of weight decay.
• Section F.1: Effect of Weight Decay with AdamW, ViT contains 3 settings of weight decay. Likewise Section F.2 looking at the effect of weight decay with SGD, ResNet50 also contains 3 settings of weight decay.
• Section F.3 on SAM also contains experiments on three sharpness coefficients.
• Section F.4, on the effect of momentum shows three values (0.9, 0.6, and 0)
• Section F.9 contains 4 different values of batch sizes.

So it is definitely the case that we have multiple settings/values of hyperparameters. We’ll certainly make the presence of these experiments clearer in the main text.

Besides, if there are some specific hyperparameters where you’d like to see more exploration, please let us know.

 

“no sense of how they connect to performance … loss curves”

Thanks for this point. We have included the loss and accuracy curves/plots in Figures 16, 22, 28, 34, 40, 46 as well as correlation of MDS with test performance in Figures 17, 23, 29, 35, 41, 47.

 

---

Section 1

 

‘Decapacitation Hypothesis’ & prior work

Thanks for bringing this recent work to our attention. We now refer to it.

We do want to mention while it is similar to our particular instantiation of decapacitation, it is important to note that:

• Our work is more geared towards (pre-)training, and not as much fine-tuning. It could be argued the late-stage might be thought of as being similar to fine-tuning, but there can still be differences.
• Also, as we mentioned, tuning only (batch/layer) normalization layers is just one particular example of how we aim to preserve the broader trajectory direction, while taking away capacity or locking in certain parameters.

Motivation from linear interpolation works

We really like your idea of motivating this in the style of Vlaar & Frankle. We will add a detailed related work discussion in the appendix that covers this and other related work on the cross-sectional geometry of loss landscapes

 

---

Section 2

 

motivation for why to flatten the parameters is not very clear.

Thank you for raising this point. Our motivation for trajectory maps was to provide a method to compactly represent high-dimensional information in a more manageable form. While our current approach aggregates parameters across the network, it is important to note that nothing fundamentally prevents us from focusing on specific subsets of parameters when needed.

• Specifically, as we mention in the main text and elaborate further in Appendix H, trajectory maps can be defined at the level of layers or groups of parameters. This allows us to track multiple trajectory maps simultaneously, each tailored to specific subsets of the parameter space.

• Such an approach can indeed offer additional insights or capture finer details beyond those accessible through a pan-network trajectory map, as demonstrated in Figure 74. For instance, while the pan-network trajectory map provides a holistic view, it may not be the ideal tool for isolating dynamics specific to biases or other parameter types.

• Consistent with views from muP about bias, we observe that trajectory maps for bias parameters sometimes exhibit slight differences, but these differences do not appear to critically impact the overall network dynamics.

Importantly, the pan-network trajectory map effectively captures shared information across the majority of parameter groups, as shown in Figures 73 and 75. This indicates that crucial information about the network's dynamics is retained, even in the aggregated and compact representation.

 

---

Section 3

 

Directional movement saturates exactly when learning rate is decayed, so it'd be nice to have a hyperparameter setting without this additional confounding factor

That’s a great question. In Appendix K, we have additional experiments with cosine, linear, and polynomial schedulers. We show that the similar behaviour of the trajectory maps, as in similar onset of directional saturation, can also be witnessed for these other learning rate schedulers. Thus verifying that LR scheduler is not a confounding factor.

Continuing the response to the rebuttal below.

Section 5

Directional measures predict

Thank you, I think this is more appropriate.

Besides there is already work on practical generalization measures...

Regardless of whether it is a side observation or not, there is a long literature on generalization measures. Presenting a new one without any context is inappropriate. It is fine not to comprehensively compare, but I think this should be couched in the understanding that this may or may not be competitive as a generalization measure, and it is not currently presented that way.

Also, there are measures besides sharpness which perform well in that work, and there are certainly other data-free generalization measures. Without a comparison on this paper's settings it is impossible for me to judge how reasonable the proposed one is.

Thank you for the response. I appreciate the additional experiments and clarifications. I'll go point by point to address the rebuttals.

Overall

it's a bit difficult to make claims...

Thank you for pointing me to these results. I believe they should be highlighted in the main text, though I completely understand that it is impossible to include them all due to space constraints.

no sense of how they connect to performance...

Thank you for providing the loss/accuracy curves. I feel that these help in interpreting the current set of results. However, I don't see them mentioned in the text at all. It would be helpful to reference (perhaps in the figure captions), so as to make reasoning easier.

Section 1

Decapacitation Hypothesis & prior work

I don't believe that late-stage finetuning is spiritually different than pretraining -> finetuning, given lots of prior results show similar properties in both settings (LMC as a big one [1, 2]). If the claim is that tuning only normalization layers is just one instantiation, then it is necessary to provide other instantiations. I don't think there's anything in this rebuttal that has convinced me it is worthwhile to give this phenomenon a new name and I believe the original criticism still stands.

• [1] https://proceedings.mlr.press/v119/frankle20a
• [2] https://arxiv.org/abs/2008.11687

Motivation from linear interpolation works

The entire point of this was to make the introduction more clear to the reader. I was not providing this to pad the related work far from where readers would see it, but rather to provide a model introduction that I enjoyed reading on how to motivate such fundamental scientific work. I do appreciate the additional changes providing definition for implicit bias.

Section 2

motivation for why to flatten the parameters is not very clear

Two points:

1. certainly nothing prevents considering individual parameters as vectors, but that is not what the majority of the experiments are about, so I don't think this is a reasonable line of arugment
2. even when flattening individual parameters, the point of this criticism was to ask why it makes sense to flatten e.g. convolutional tensors, when the natural linear-algebraic operation is happening in more dimensions. Without backing out a connection from the vector to the structure, it's unclear to me why this is a good idea.

I still maintain that this motivation is quite unclear, and I don't think the paper has yet addressed this. The goal is not to change the results or provide restricted results, the goal is to justify this choice, instead of taking it as a given.

Section 3

Directional movement saturates exactly when learning rate is decayed, so it'd be nice to have a hyperparameter setting without this additional confounding factor

Thank you for the additional results. This helps clarify the original experiments.

I'm also not sure if the additional concerns here were cut off in the initial response, or what happened.

Section 4

"Initial Thoughts" are confounded by the lack of training loss

It would be good to reference the training loss figures at the site of this confusion. Otherwise I can see that it would be similarly confusing.

Fig. 4 the updates are oscillating, but this might be expected for (relatively) small batch sizes

Thank you for the additional experiment. I'm not sure this really addresses the original concern. The point of the original concern was that what oscillation may really be tracking is that each individual update could be directionally defined according to the few examples that comprise it. ImageNet has a much larger number of examples than 16k, so I'm not sure how close this behavior is to gradient descent, or if it's still (relatively) small. The natural control to this is to look at the gradient direction after each update, or to take an estimate of the gradient, but for a fixed sample of examples that is shared across all timesteps. Without this control, I'm not sure that the concern is answered.

Certainly the stable Hessian assumption seems unfounded in the early stage of training

Thank you for clarifying that the theory is intended to address late-stage. It would be helpful to modify the text near the statement of Lemma 1 to make this explicit.

Also I don't want to open this can of worms, but I'm not sure that I buy that infinite-width NNs provide "a lot of intuition for practical NN training dynamics." They do not perform like training, and they assume something that even this work shows is false: dynamics are not linear around the origin.

EoS is for gradient descent... no?

Thank you for the additional reference.

Thanks for your response. To answer pointwise:

 

Section 1

give this phenomenon a new name

We understand your concern. We'll remove this naming, and reframe the discussion without it.

 

Motivation from linear interpolation

Yes, as mentioned before, we like this idea and do want the motivation to appeal to a wider readership. We'll incorporate the elements of their motivation in the camera ready.

 

Section 2

why it makes sense to flatten

Thanks for further explaining your question. There are theoretical and empirical reasons why flattening makes sense:

• Mathematically, in a finite-dimensional vector spaces, all norms are equivalent in the sense that they define the same topology. Please see https://en.wikipedia.org/wiki/Norm_(mathematics) or https://kconrad.math.uconn.edu/blurbs/gradnumthy/equivnorms.pdf

• Empirically, the usage of flattening parameters has successful precedents in optimization, and even in the linear interpolation literature. Optimization convergence analyses in deep learning default to working with flattened vectors as parameters, e.g., see AdaGrad https://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf and they are directly lifted as such in applications. Also, more recently in the line of work on schedule-free methods by Defazio et al., like in D-Adaptation https://arxiv.org/pdf/2301.07733 (ICML'23, outstanding paper award) directly estimates the size of the domain $\|\theta_0 - \theta_\ast\|$ based on flattened calculations.

Also, in the weight-matching methods used in the extended linear interpolation literature, such as for model merging modulo permutations (https://arxiv.org/pdf/2209.04836, https://arxiv.org/abs/1910.05653), it is a common practice to flatten convolutional tensors into vectors for weight matching.

 

Section 3

In the interest of time, we didn't touch on some concerns that we perceived to be somewhat tangential, but we can discuss them here briefly.

"epochly" -> intermediate?

Has been already done.

 

I feel that convergence in direction should probably be considered as convergence given that CE loss can never be minimized because of the exponential.

• Convergence in direction is relative to the minimum, which is unknown, so it is not trivial to ascertain.
• What we show here as directional saturation occurs well before the interpolation regime, i.e., much before achieving zero-training error.
• As far as post-zero training error (terminal phase), often training is still carried out as some works show that it confers benefits (such as that on the Neural Collapse https://www.pnas.org/doi/10.1073/pnas.2015509117 and related works.)

With all this in mind, we're simply resorting to the standard training lengths as reported in the literature, like 90 epochs for ImageNet.

 

Training loss curves for Fig 2 + 3 are missing: is directional similarity mostly a result of the model failing to learn well at all?

We have added these loss curves. As far as the particular question is concern, No --- as clearly evident in the training loss plots.

 

It might be nice to rescale the trajectory maps to be on separate color scales. For example in the case with no weight decay, the increased similarity could be due to the fact that WD removes superfluous weights, but without a lot of the initialization sticks around that isn't used much for the actual prediction task.

This was unfortunately incomprehensible to us (see the part in bold which seems chopped off). Modifications to trajectory map visualizations would nevertheless be something to look at in future works; but we are confident that our trajectory map visualizations have been tested out in various settings and are highly robust.

 

Thanks for your concrete and helpful feedback. We’re pleased to hear that you found our work sheds light on the consistency and detailed behaviour of optimization paths, as well as the supporting empirical results to be convincing.

We understand the raised concerns. Please allow us the opportunity to address them in detail below.

 

Learning Rate Schedule Diversity:

In brief, we have added new experiments with cosine, linear, and polynomial schedules in Appendix K.

• Overall, we find that the broader nature of the trajectory map as well as the angular metrics is highly similar across these schedules (barring minor variations).

• Hence, our proposed approach reflects the underlying characteristics of optimization and is not overly sensitive to the learning rate scheduling.

 

Correlation of Mean Directional Similarity with Performance:

That’s a valid question. We have now added the correlation of MDS (as well as a relativized version of it) with test accuracy in all the ImageNet settings. These plots being, namely, Figure 17, 23, 29, 35, 41, 47.

Observations:

• We do find that MDS numbers correlate well with performance when models are of the same size and the hyperparameters are varied. In particular, as MDS decreases, and the over-reliance on a specific direction (or set of them) decreases, and therefore the performance increases. Hence, in most of these figures we find a strong negative correlation.

• On average across these settings, MDS obtains a Pearson correlation of $-0.71$. The magnitude of this average Pearson correlation coefficient further increases to $-0.90$ if we do not consider a couple of outlying cases where the network underfits severely due to high regularization in these cases. Thus, it demonstrates that MDS can serve as a promising indicator of performance.

• However, we believe further refinements of MDS could potentially also enable comparison across networks of different sizes. Presently, MDS as a notion depends on the loss landscape and trajectories that can be taken within it, thus it is better suited to comparing performance/generalization for a given sized network.

Nevertheless, the full-extent to which MDS and other angular notions (such as promising ones like the apex angle at origin) can serve as a performance predictor deserves a separate paper of its own. But given these initial results, this would form an excellent future direction.

 

Others:

• Accuracy and Loss Numbers have now been provided. In particular, we have provided loss and accuracy curves, as well as exact final test accuracy for all ImageNet experiments are available in the figure captions. Please see Figures 16, 22, 28, 34, 40, 46 for more.

• References have been fixed, thank you for pointing these out.

 

---

We hope that we have been able to address all your pending concerns. But please let us know, in case you still have some other questions or comments. Besides, in light of the above response, it would be great if you could reconsider your score. Thank you!