Dear Reviewer 4J3D,

Thank you for your time and effort to evaluate our submission, as well as your interesting comments and questions. In the following, we first respond to the weaknesses and then address the limitations and questions.

Weaknesses

(W1): no overhead quantification

We agree that these experiments would corroborate the utility of our method in practical settings. To this end, we included several additional experiments on a) the wall-clock runtime during training for different models, batch sizes, and $D$, b) the GPU memory overhead for the same settings, and c) computed the parameter overhead in our models as you suggested. Generally, the parameter and memory overhead is negligible for any batch size, while the wall-clock runtime increases slightly with $D$ for some models, but this difference becomes irrelevant for larger batch sizes.

Parameter Overhead (factor=new params/old params -1)

Peak memory utilization during training (GB). Means and std. over ten runs are shown.

Wall-clock runtime overhead (training) for neuron, filter, and attention sparsity. Means and std. over ten runs are shown.

(W2): mismatch between theoretical and practical set-up

Thanks for pointing out this subtle point. Our main result, Theorem 1, establishes that our $D$-Gating objective is equivalent to the non-smooth sparsity-regularized base objective, irrespective of the optimizer. This equivalence in terms of the solution structure permits, in principle, using any optimizer to find those sparse solutions. Our convergence result, Lemma 4, quantifying how fast the $D$-Gated objective aligns with the non-smooth sparse objective, merely provides a clean upper bound for the rate under gradient flow. Hence, there is no mismatch between our theoretical and practical set-up, just that the convergence rate under gradient flow does not hold for, e.g., Adam. Intuitively, any common optimizer strives to find a balanced representation, since, at any point, the $D$-Gating parameters can be perturbed toward a more balanced solution, thus reducing the $L_2$ penalty, while leaving the unregularized loss constant. To corroborate this intuition, we repeated the experiment in Fig.3 using Adam, not SGD. We find that the imbalance also decays to zero (even faster than SGD), although the decay is not exponential. This indicates that although the rate may not be as cleanly described as for gradient flow, Adam also leads to balanced gating representations and thus a sparsity-inducing objective.

(W3): use of words in exposition

We agree that “structured sparsity” can also encompass, e.g., N:M sparsity, not covered by our group sparse framework, although it is understood as group sparsity in certain communities. The phrase overparameterization was used as our $D$-Gating approach is inherently a (regularized) overparameterization. We will diligently go through our submission and reword it to emphasize that our approach is for group sparsity in a revised version.

(W4): visualization of our method (Fig.2)

Thank you for pointing this out. We also spent quite some time deliberating the best way to visualize our approach in a simple way and decided on the presented input-sparse example with explicitly mentioning that this is only one possible use case in the caption. Note that our submission also includes more visualizations of $D$-Gating architectures in Figures 9, 10, and 13 of our Appendix. Yet, we will reconsider our example choice and try to come up with a better visualization for a revised version.

(W5): Connection between loss convergence, norm regularizers, and balancedness

For clarity, we have added a version of Fig. 3 to the Appendix that shows the difference between the $L_{2}$ and the $L_{2,2/D}$ regularizer on the y-axis, showing qualitatively identical results. This is because the loss difference is simply the difference of both norm-based regularizers scaled by $\lambda$. To see this, $\mathcal{L}(\boldsymbol{\omega}, \boldsymbol{\Gamma}) - \mathcal{L}\_{\mathbf{w}}(\mathbf{w}) = \mathcal{L}\_0(\cdot) - \mathcal{L}\_0(\cdot) + \lambda \big( (||\boldsymbol{\omega}|_2^2| + || \boldsymbol{\Gamma}||_F^2)/D - ||\mathbf{w}||\_{2,2/D}^{2/D} \big)$. Hence, convergence of losses is virtually identical to convergence of the regularizers, which happens iff all parameters are balanced.

(W6): The experiments are on a very small scale.[...] This suggests that D-Gating may introduce overheads that would make it impractical at larger scales. The paper also lacks strong baselines.

Regarding the overheads, please see our new experiments described in (W1), showing that they are small/negligible.

Regarding the scale of experiments, while we agree that large-scale LLM experiments would strengthen our evaluation, we argue that our contribution has a methodological focus, and the experiments serve to showcase versatility. Given the decision space {architecture types}x{data modalities}x{sparsity use cases} is quite large, we tried to best represent this in our selection of experiments, spanning fully-connected DNNs, CNNs, neural trees, and transformers, using numerical, image, text, and speech data of various sizes, and successfully demonstrates $D$-Gating for input sparsity, filter sparsity, tree sparsity, and attention sparsity.

Regarding baselines, we agree that they are solid but dated methods, particularly for transformers where alternatives like SparseGPT exist. However, our contribution is mainly methodological, proposing a differentiable formulation of $L_{2,2/D}$ regularization, with the experiments demonstrating its effectiveness for various architectures and data modalities. SparseGPT, for example, is a method that efficiently solves a weight reconstruction problem post-training under sparsity constraints, and is thus fundamentally different from our method. Hence, we selected more directly comparable baselines for our experiments, i.e., direct optimization of the non-smooth $L_{2,1}$ regularizer (besides Network Slimming and LassoNet/HSIC for the feature selection experiments), to better illustrate our method’s advantage over the naive formulation, instead of trying to establish SOTA performance for every application. Moreover, we note that most SOTA methods are multi-pass, pruning-and-regrowing, or post-training weight reconstruction strategies that make a fair comparison much harder (see [1]).

Limitations/Questions

(Q1): key distinction between "modular" and "versatile" ?

We agree it can sound redundant and changed the title to “Differentiable Sparsity via $D$-Gating: Simple and Versatile Structured Penalization”.

(L1): [...] theoretical assumptions that may be violated in practice, e.g., in the very experiments used in the paper!

We hope the previous discussion has cleared up this misunderstanding. The equivalence of the $D$-Gating objective holds for any optimizer, just the exponential rate we derived is for gradient flow specifically.

(L2): Discussion of computational overheads was inadequate, and it should be clearly quantified, rather than simply saying "negligible"

Please see response to (W1).

We thank you again and hope this adequately answers your questions and concerns. We also hope that our additional experimental and theoretical results strengthen our submission in your eyes.

References:

[1] Wang, Huan, et al. "Why is the state of neural network pruning so confusing? on the fairness, comparison setup, and trainability in network pruning." arXiv:2301.05219 (2023).

Dear Reviewer vx9E,

Thank you for your time and effort to evaluate our submission, as well as your interesting questions. In the following, we first respond to the weaknesses and then address the questions.

Weaknesses

(W1): how SGD reaches such a zero (imbalance) state, is unguaranteed

Please see our response to question (Q2) below. Further, note that all our experiments also support the claim that SGD finds sparse solutions. E.g., in Fig. 4, we show that $D$-Gating finds exactly the same global minima as the Group Lasso for linear models.

(W2): Additional computational cost can be large

We agree with you that the parameter count overhead could become considerable for extremely few weights, small groups, and many/all parameters being regularized. But in practice, the param overhead is negligible. Even for the simple linear regression with 200 features and 40 groups, $D$-Gating with $D=4$ adds only 120 new parameters to the model. In contrast, for our NanoGPT implementation, we use 6 layers with 12 heads, yielding 72 groups in total. For $D=4$ this amounts to 216 additional parameters in a model with 10.8 mio. parameters. To illustrate this better, we have included the following table in the Appendix:

Paramever Overhead (factor=new params/old params-1)

In addition, to verify that the computational overhead is not considerable, we measured the wall-clock runtime and GPU memory overhead for different models and batch sizes, summarized in the tables below. Memory overhead is negligible and runtime slightly increases in $D$ but only for small batches.

Peak memory utilization during training (GB). Means and std. over ten runs are shown.

Wallclock runtime overhead (training) for neuron, filter, and attention sparsity. Means and std. over ten runs are shown.

Questions

(Q1): no in-depth comparisons with SOTA methods

While we agree that a comparison with sophisticated sparsification pipelines would be interesting, our contribution is mainly methodological, proposing a differentiable formulation of $L_{2,2/D}$ regularization, with the experimental evaluation demonstrating its effectiveness for various architectures and data modalities. For example, the method SRigL proposed in your reference [1] is a specialized approach for N:M-type structured sparsity with a constant fan-in constraint and based on pruning and re-growing connections, which is fundamentally different from our differentiable sparsity regularization approach. Therefore, we chose to opt for more directly comparable baselines in our experiments, i.e., direct optimization of the non-smooth $L_{2,1}$ regularizer, Network Slimming, or LassoNet, to better illustrate our method’s advantage over the naive formulation, instead of trying to establish SOTA performance for every application. Moreover, as shown in [2], the comparison set-up is massively complicated for sophisticated pruning and retraining/regrowing pipelines, further corroborating our use of directly related methods, albeit they may not be SOTA.

(Q2): Can you provide theoretical guarantees for convergence of SGD to sparse solutions?

We thank you for this interesting question, which highlights a subtle difference between convergence of the objective to a solution and convergence of the objective $\mathcal{L}(\boldsymbol{\omega}, \boldsymbol{\Gamma})$ to the non-smooth objective ${\mathcal{L}}_{\mathrm{w}}(\mathrm{w})$. Whether a $D$-Gated network converges to a solution overall (i.e., to solutions with many zero groups) can not be answered simply, as it depends on many factors like the network architecture, data set, and optimization hyperparameters. It is similar to asking whether objectives with explicit Group Lasso regularization always converge, which can not be answered without specifying the concrete learning problem that is being regularized. A formal analysis of how $D$-Gating impacts overall convergence rates is beyond the scope of this paper, evidenced by the lengthy but still restricted derivations in [3], obtaining the Kurdyka-Łojasiewicz exponent for a much simpler (unstructured) overparameterization, from which a local convergence rate can be deducted. In contrast, our theoretical results establish that a) all solutions of the $D$-Gating problem are solutions to the sparse regularized problem, and b) under gradient flow, the surrogate objective exponentially fast becomes the sparse regularized objective, i.e., the optimization of $\mathcal{L}(\boldsymbol{\omega}, \boldsymbol{\Gamma})$ turns into $\mathcal{L}\_{\mathbf{w}}(\mathbf{w})$.

Other comments:

New result for discrete-time SGD dynamics

Additionally, we have now extended our gradient flow results to discrete-time gradient descent, establishing (for sufficiently small step sizes) exponential decay of the imbalance similar to gradient flow, and thus $\mathcal{L}(\boldsymbol{\omega}, \boldsymbol{\Gamma}) \to {\mathcal{L}}_{\mathrm{w}}(\mathrm{w})$. The novel Lemma reads:

Consider the $D$-Gated objective Eq. (5). Then, under (S)GD with learning rate $\eta>0$, (i) the pair-wise imbalances in Eq. (13) evolve as $\mathcal{I}\_{j,d,d'}^{(t+1)} = \big(1 - \frac{4\lambda}{D} \eta\big) \mathcal{I}\_{j,d,d'}^{(t)} + \eta^2 \Delta\_{j,d,d'}^{(t)}, \\: d \neq d', d,d' \in [D]$, with different second-order terms $\Delta\_{j,d,d'}$ for $\mathcal{I}\_{j,1,d'}$ and $\mathcal{I}\_{j,d,d'}$ for $d,d'>1$. In particular, for sufficiently small $\eta$, $\mathcal{I}_{j,d,d'}^{(t+1)} \approx (1-4 \lambda \eta/D) \cdot \mathcal{I}\_{j,d,d'}^{(t)}$ exhibits discrete exponential decay. Moreover, (ii) for balanced zero gating parameters, $(\boldsymbol{\omega}_j^{(t)},\{\gamma\_{j,d}^{(t)}\}\_{d=1}^{D-1})=\boldsymbol{0}$, the future iterates are confined to $\mathbf{w}_j^{(t')}=\boldsymbol{0} \, \forall \,t'>t$, and (iii), balancedness, i.e., $\mathcal{I}\_{j,d,d'}^{(t)}=0$, is conserved $\forall t'>t$ between any two scalar gates $\gamma\_{j,d}^{(t)},\gamma\_{j,d'}^{(t)}$.

Choice of notation

We agree with you that it is not straightforward to read, but after much deliberation, we chose this notation to emphasize the “gating” operation using the triangle. The key issue is to find a representation for “group-wise element-wise multiplication”, a problem that other authors have also faced. E.g., [4] introduces an additional matrix $D$ that contains $0$s and $1$s to “broadcast” the group factors to the same dimension as the primary weight, yielding the expression $(D \boldsymbol{\gamma}^{\odot} ) \odot \boldsymbol{\omega}$, which is arguably even more inconvenient, while [5] use the symbol $\boldsymbol{\omega }\odot_{\mathcal{G}} (\boldsymbol{\gamma}^{\odot})$ for structured element-wise multiplication. But agreeing with your sentiment, we will think about how to best simplify the notation for a revised version, nevertheless.

We thank you again and hope this adequately answers your questions. We also hope that the additional experimental results, as well as our new theoretical result for the discrete-time evolution of imbalances, strengthen our submission in your eyes.

References

[1] Lasby, M., et al. "Dynamic Sparse Training with Structured Sparsity." The 12th ICLR.

[2] Wang, H., et al. "Why is the state of neural network pruning so confusing? on the fairness, comparison setup, and trainability in network pruning." arXiv:2301.05219 (2023).

[3] Ouyang, W., et al. "Kurdyka–Łojasiewicz exponent via Hadamard parametrization." SIAM Journal on Optimization 35.1 (2025): 62-91.

[4] Li, Jiangyuan, et al. "Implicit Regularization for Group Sparsity." 11th ICLR.

[5] Poon, C., and Gabriel P. "Smooth bilevel programming for sparse regularization." NeurIPS 34 (2021)

Dear Reviewer dzXs,

Thank you for your encouraging comments and the time and effort to evaluate our submission. In the following, we first respond to the question and then address the weaknesses.

Questions

(Q1): Can you quantify the memory/computation overhead of increasing D?

We agree that those experiments would help in evaluating practical implementation considerations. To this end, we have included several additional experiments on a) the wall-clock runtime overhead during training for different models, batch sizes, and $D$, b) the GPU memory overhead for the same settings, and c) computed the parameter overhead in our models. Generally, the parameter and memory overhead are essentially negligible for any batch size, while the wall-clock runtime increases slightly with $D$ for some models. However, this difference becomes irrelevant for larger batch sizes.

Wallclock runtime overhead (training) for neuron, filter, and attention sparsity. Means and std. over ten runs are shown.

Peak memory utilization during training (GB). Means and std. over ten runs are shown.

Parameter Overhead (factor=new params/old params-1)

(Q2): Can you outline a guideline for choosing the gate depth in practice?

In the general non-convex regularization literature [1], it is established that $L_{2,0.5}$ or $L_{2,2/3}$ regularization ($D=3,4$) usually yields the best trade-off between performance and increased optimization issues as we approach group $L_0$ regularization.

We conducted ablation studies on $D$ for group sparse linear regression, an input-sparse LeNet-300-100 trained on ISOLET, and a filter-sparse VGG-16 trained on CIFAR-10. As expected, some settings showed increased numerical instability for large $D$ (a jagged/“zig-zag” regularization path instead of a smooth one) and all settings confirmed that $D=4$ essentially reaps the full benefits of non-convex regularization while retaining stable training. The new plots are summarized as tables below.

Ablation studies for gating depth $D$ at various sparsity levels s (either test acc. or RMSE)

Depth ablation for numerical instability.

Note that the plot shows the jagged “zig-zag” regularization path for large $D$ much better. The table contains RMSE values, and bold values indicate instability.

Weaknesses

(W1): restricted model/data scope

While we agree that large-scale LLM or ImageNet experiments would strengthen our empirical evaluation, we argue that given the decision space {architecture types}x{data modalities}x{sparsity use cases} is quite large, we tried to best represent this in our selection of experiments, especially given our contribution has a methodological focus and the experiments serve to showcase its versatility. Our selection spans fully-connected DNNs, CNNs, neural trees, and transformers, using numerical, image, text, and speech data of various sizes, and successfully demonstrates D-Gating for input sparsity, filter sparsity, neural tree sparsity, and sparse attention heads.

(W2): baselines are not SOTA

We agree that MP and NS are solid but dated methods, particularly for transformer-based architectures where alternatives like SparseGPT exist. However, our contribution is mainly methodological and proposes a differentiable formulation of $L_{2,2/D}$ regularization, with the experimental evaluation demonstrating its effectiveness for various architectures and data modalities. SparseGPT, for example, is a method that efficiently solves a weight reconstruction problem post-training under sparsity constraints, and is thus fundamentally different from our sparse training method. Therefore, we chose to select more directly comparable baselines for these experiments, i.e., direct optimization of the non-smooth $L_{2,1}$ regularizer (besides Network Slimming and LassoNet/HSIC for the feature selection experiments), to better illustrate our method’s advantage over the naive formulation, instead of trying to establish SOTA performance for every application.

(W3): Theo. speed-up unverified in practice

We again agree with you that these experiments would be interesting. But as a submission that proposes mainly a methodological contribution and empirically validates it on a variety of applications, measuring acceleration other real hardware is an interesting avenue for future research.

(W4): Missing ablation studies on runtime/memory/parameter overhead and instability)

Please see our new experiments listed above on all these items.

We hope this adequately answers your questions and that the additional results on the computational/memory/parameter overhead, instability, and performance for larger $D$.

References

[1] Hu, Yaohua, et al. "Group sparse optimization via $\ell_{p,q}$ regularization." Journal of Machine Learning Research 18.30 (2017): 1-52.

A choice of $D$ equal to 3 or 4 appears to yield good results without adding significant computational cost, especially at high batch sizes. The author addressed my questions satisfactorily with the new, simple experiments. I understand that not all experiments can be conducted due to the exponential scaling, but hopefully we will see them in future work. I am increasing my rating to 6

Dear Reviewer YPR6,

We thank you for your time and for your encouraging comments, as well as for the interesting questions you posed. Below, respond to the raised limitations and respond to the questions point-by-point.

Questions

(Q1): Does the equivalence of objectives still hold when $L_2$ regularization is not hard-coded into the objective but a decoupled weight decay, as in AdamW, is applied?

Thank you for this question. Our proposed method turns a non-smooth objective with structured sparsity regularization into a fully differentiable surrogate with $L_2$ regularization. Therefore, the explicit $L_2$ penalty is necessary for our main equivalence result in terms of minima. This equivalence is baked into the objective structure and does not rely on a specific optimizer. Consequently, we may attempt to use (S)GD, Adam, or even AdamW, as long as the objective is the gated objective with $L_2$ penalty. Your question, if understood correctly, however, is rather if the equivalence holds if we additionally replace the $L_2$ penalty in our differentiable $D$-Gating objective by decoupled weight decay (WD) in AdamW, similar to how we can exchange explicit $L_2$ regularization using WD updates in (S)GD. As updates using decoupled WD do not correspond to updates under $L_2$ regularization, the straightforward answer is (likely) no. However, although no exact equivalence to an $L_{2,2/D}$ regularizer holds, it is plausible that sparsity-promoting effects also emerge for AdamW, but less cleanly. Supporting this, [1] introduce decoupled WD for $\ell_q$ quasinorms and successfully train sparse DNNs with AdamW. How $L_2$ regularization and decoupled WD interact is a non-trivial question and an active area of research (see, e.g., [2]). In Appendix E, the authors of [3] discuss links between AdamW and $L_2$ regularization under certain simplifications, suggesting qualitatively similar dynamics to SGD/Adam could manifest.

Limitations:

(L1)/(L4): no large-scale experiments/SOTA baselines.

While we agree that large-scale experiments would strengthen our evaluation, we argue that our contribution has a methodological focus, and the experiments serve to showcase versatility. Given the large decision space {architecture types}x{data modalities}x{sparsity use cases}, we tried to best represent this in our experiments, spanning dense DNNs, CNNs, neural trees, and transformers; using numerical, image, text, and speech data, and successfully demonstrating $D$-Gating for input/filter/tree/attention sparsity.

Regarding baselines, we agree that some are solid but dated methods, particularly for transformers where alternatives like SparseGPT exist, a method that efficiently solves a weight reconstruction problem post-training under sparsity constraints, and is thus fundamentally different from our method. Hence, we chose more directly comparable baselines, e.g., direct optimization of the non-smooth $L_{2,1}$ regularizer, to better illustrate our method’s advantage over the naive formulation, instead of trying to establish SOTA performance for every application.

(L2): No overhead/stability/depth ablation experiments.

We agree that those experiments would improve our submission. Thus we included several additional experiments, measuring param./memory overhead, runtime, num. stability and ablations for large $D$. We show that the param./memory overhead is negligible for all models, while runtime increases slightly with $D$ for some models but becomes irrelevant for larger batches. Large $D>4$ does not improve performance but can become unstable. We summarize the new results plots below as a table:

• Parameter count overhead for $D$-Gating (factor=new params/old params-1)

• We also measured the memory overhead of $D$-Gating, finding that the utilization over all settings are almost indiscernible from the vanilla memory usage (see reply to dzXs for table).

• Wall-clock runtime (training) for neuron, filter, and attention sparsity. Means and std. over ten runs are shown

• We further included ablations on how the gating depth $D$ influences performance and stability for an input-sparse LeNet-300-100, VGG-16, and a group-sparse linear model (contained in reply to dzXs in "Lin. Mod." table). In line with the sparsity regularization literature, $L_{2,q}$ penalties are usually only helpful up to $q=0.5$ or $q=2/3$, corresponding to $D=3,4$. In our experiments, we observe no performance gains for large $D$.

Ablation on depth $D$ at various sparsity levels s (test acc.)

• Due to character restrictions, further depth ablations and instability results for the linear model are contained in the reply to dzXs ("Lin. Mod." tables). We find that large $D$ exhibits increased instability but no performance benefit.

(L3): Proofs use only continuous-time gradient flow. The paper will be strengthen if it extends the theory to discrete SGD.

We agree that discrete-time results would strengthen our analysis beyond the existing (S)GD result in Lemma 4. We have revised the Lemma to derive the discrete-time imbalance dynamics and prove exponential decay for sufficiently small steps:

Consider the $D$-Gated objective Eq. (5). Then, under (S)GD with learning rate $\eta>0$, (i) the pair-wise imbalances in Eq. (13) evolve as $\mathcal{I}\_{j,d,d'}^{(t+1)} = \big(1 - \frac{4\lambda}{D} \eta\big) \mathcal{I}\_{j,d,d'}^{(t)} + \eta^2 \Delta\_{j,d,d'}^{(t)}, \\:d \neq d', d,d' \in [D]$, with different second-order terms $\Delta\_{j,d,d'}$ for $\mathcal{I}\_{j,1,d'}$ and $\mathcal{I}\_{j,d,d'}$ with $d,d'>1$. In particular, for sufficiently small $\eta$, $\mathcal{I}_{j,d,d'}^{(t+1)} \approx (1-4 \lambda \eta/D) \cdot \mathcal{I}\_{j,d,d'}^{(t)}$ exhibits discrete exponential decay. Moreover, (ii) for balanced zero gating parameters, $(\boldsymbol{\omega}_j^{(t)},\{\gamma\_{j,d}^{(t)}\}\_{d=1}^{D-1})=\boldsymbol{0}$, the future iterates are confined to $\mathbf{w}_j^{(t')}=\boldsymbol{0} \, \forall t'>t$, and (iii), balancedness, i.e., $\mathcal{I}\_{j,d,d'}^{(t)}=0$, is conserved $\forall t'>t$ between any two scalar gates $\gamma\_{j,d}^{(t)},\gamma\_{j,d'}^{(t)}$.

• Besides, we also include an experiment using Adam instead of SGD in Fig. 3, showing imbalance also decays to zero (even faster than SGD), although the decay is not exponential. This suggests that Adam also leads to balancedness and a sparsity-inducing objective. The feasibility of sparse DNN training using Adam is also demonstrated in our language experiment.

We hope this adequately answers your questions and that the additional new results on the computational overhead, instability, and performance for larger depths, as well as our new theoretical discrete-time analysis of the imbalance evolution, further strengthen our submission in your eyes.

References:

[1] Outmezguine, N.J., and Noam L.. "Decoupled Weight Decay for Any p Norm." arXiv:2404.10824 (2024).

[2] Xie, S., and Zhiyuan L.. "Implicit Bias of AdamW: L∞-Norm Constrained Optimization." 41st ICML.

[3] Kobayashi, S., Yassir A., and Johannes v.O.. "Weight decay induces low-rank attention layers." NeurIPS 37 (2024).