• The paper is not very well written. E.g. the authors write “was wildly interpreted” – they probably mean widely. There are also technical things that are not clear. E.g. section 2 is very handwavy and I don’t understand what the authors mean. I would encourage them to formulate their thoughts as mathematical theorems with proofs. The algorithms are also unclear.
• The metrics reported by the paper seem strange. The authors state that “To mitigate randomness, the mean of the best 10 Top-1 accuracy on the test set during training is adopted”. Firstly, it seems like this is in effect doing model selection via the test-set – a separate validation set should be used for that. Secondly, the different scores during a training run and not independent, so it’s a little misleading to use these to calculate error bars. Thirdly, it seems like each experiment is only done once. So we might just be looking at statistical noise.

Minor issues:

The authors state that “weight decay weakens the interconnections between network layers, ultimately resulting in a decline in performance”. This is not true in general. If it was, people would not use WD.

The authors state that “For deep neural networks, high-level features possess fewer samples” – this is true for CNNs where the feature maps shrinks, but not for e.g. ViTs.

• The paper indeed uses clear mathematical notation to illustrate the problem, but I don't think the two issues are theoretically identified. In particular, there is no claim or proof to show how the issue directly affects generalization. I think the way the author identifies the issue is by pointing out the extreme case in which the issue can fail, e.g. (2), or a very large decay rate. Also, through empirical study of Figure 2 and Figure 3.
• The second issue and the solution lack clarity, It is not clear how Figure 3a illustrates the issue of layer-wise connectivity and how the proposed method solves it.
• The paper is a bit hard to follow because the mathematical illustration of the issues and empirical demonstration are separated (I don't think the mathematical representation is enough to identify the issue), which makes Figure 2 on Page 3 first introduced and described on Page 7, and Figure 1 on Page 2 introduced on Page 8.
• Should the Algorithm 1 the red line be $\theta_{t+1}=\hat{\theta}_{t+1}-\lambda \hat{\theta_t}$?

My concerns are around the limited evaluation. It might lead to wrong motivation and conclusion that could be fatal errors.

1. The empirical validation is primarily based on Cifar-10 and Cifar-100. Testing on more diverse datasets might provide a clearer picture of LPWD's effectiveness. Also, the improvements are subtle.
2. If the authors only tested LPWD during fine-tuning, it might limit the generalizability of their claims. The behavior and benefits of LPWD during the entire training process (from scratch) might be different.
3. Fine-tuning on a specific task with a pre-trained model might introduce some bias, as the model has already learned general features from the pre-training dataset. This could affect how weight decay or any regularization method operates.

minor: Typo: The delay defect will lead larger weights after penalizing when the four current factors (learning rate, gradient, weight decay rate and weights) meet certain conditions. “Lead to”?

The reason why the two major findings hurt the performance is not clear. The first finding is that WD will sometimes drive the current weight away from 0. The paper claims that as a result of the large number of parameters and cascading effect of deep neural networks, this phenomenon will hurt generalization. The second finding is that WD distorts the distribution of features, and again due to the cascading effect, this phenomenon will hurt the performance.

This cascading effect of deep neural networks is used to explain all the effects in this paper, without specific reasoning process and verifiable arguments. Thus, the findings do not quite make sense to me as it is hard to see how the drawbacks of WD affect the performance.