We thank the reviewer for their constructive feedback. We appreciate their assessment of the novelty of the method and the soundness of the theory.

1. (Q Cross influence of regularization and compression) From an analytical perspective, we only target the spectrum to enable a well-conditioned low-rank matrix. However, we provide results on the effect of the regularizer on the compression rate in Figure 4,5,7 in the Appendix, which show a full result matrix of RobustDLRT-trained low-rank networks with different regularization strenghts $\beta$ and their compression rate (and adversarial accuracy under different attacks). It becomes apparent that the compression rate of the networks is very similar and does not deepened on the regularization strength.

   We show the impact of compression on robustness in all Tables in the main manuscript, where it is apparent that the adversarial robustness of non-regularized, compressed networks suffers. It can be recovered by application of the proposed regularizer as we show in multiple test cases, e.g. Table 1-4. This is the main contribution of this work.

2. (Q Clarity of regularization vs compression) We note that the case of Figure 1, where the DLRT spectrum is a widened version of the baseline spectrum and the regularizer straightens the spectrum, is not present across all layers. In some layers, the spectrum of DLRT and robustDLRT is far larger than the baseline model; this fact could explain the lack of strong correlation between condition number and adversarial accuracy. As an aside, a portion of the above comment is present in a footnote in the original work, but was not properly referenced in the caption of Figure 1.

   Additionally, in the cases where the baseline is acceptable already, RobustDLRT shares similar adversarial accuracy.

   We finally comment that creating a robust network will require several defenses combined, and this paper is to demonstrate the capabilities of the proposed method.

3. (Q Evaluation Method) In Table 2, we evaluate the baseline, DLRT, and RobustDLRT models with the FGSM, Jitter, and Mixup adversarial attacks. Jitter and Mixup, described with references in Sections A.1.3 and A.1.4 respectively, are both iterative and gradient-based adversarial attacks that are quite strong.

   To eleviate the concerns of scalability we provide numerical results for the ImageNet-1k (1.2 Million images), where we compress the ViT-L32 vision transformer (304 Million parameters). We first conduct a hyperparameter sweep to determine standard hyperparameters as learning rate, weight-decay and batch size, which we report in Appendix C.

   We compare then the baseline model, DLRT and the regularized RobustDLRT adversarial performance in the table below and observe similar trends as in the Vision Transformer experiments of Table 2 in the main manuscript.

Table: ImageNet, Top-1 Accuracy under various attacks

Table: ImageNet, Top-5 Accuracy under various attacks

Finally we wish to remark that the slightly lower compression rate is expected since the hidden dimensions of ViT-L32 of 1024 is close to the number of ImageNet classes (1000), thus there is less redundancy in the model compared to other reported benchmarks.

4. (Q Impact of the compression in derivation of Sec. 4)

   1. The compression of the model is largely achieved through the rank-augmentation and rank-truncation steps of Algorithm 1. The regularizer influences the spectrum of the latent space map $\hat{S}$ which affects the truncation criterion of line 251. The regularizer may negatively affect compression by raising small singular values that would be truncated if not regularizer were used; however, we find in our experiments that the compression rate is not affected in practice.

      We feel the two follow up questions is addressed in the response to your second question (Q) above.

5. (Q Full rank regularization) We have conducted this experiment in Table 8 in the original manuscript in the Appendix, where we explore the effects on regularization of a full-rank network, were we have observed: "The regularizer is able to increase the adversarial robustness of the baseline training network, at the cost of some reduction of its clean validation accuracy."

   We wish to remark, that using $\mathcal{R}$ on the full-rank weight layer is computationally expensive, and that the regularizer is designed for low-rank networks.

We thank the reviewer for their constructive review. We appreciate that they found the proposed method practical, efficient and easy to implement.

1. (Q Lipschitz definition) We added the definition of Lipschitz continuity in the appendix. For completeness: A function $f(x)$ is Lipschitz continuous on a domain $D$ if there exists a constant $L \geq 0$ such that for all $x, y \in D$,

The smallest such $L$ is called the Lipschitz constant.

2. (Q orth operator) As described in line 235 of the original manuscript We need to check all line numbers before resubmission, $Q=orth( Z )$ is defined as the $Q$ factor in the QR-factorization, i.e, $QR=qr(Z)$. Additionally, $[A|B]$ for $A,B\in\mathbb{R}^{n\times r}$ concatenates $A,B$ in the second dimension, i.e. $[A|B]\in\mathbb{R}^{n\times 2r}$.

3. (Q method part) We wish to remark that the main contribution is the construction of the Regularizer $\mathcal{R}$ in Section 4. We state in line 153 that the method is compatible with any low-rank training scheme, e.g. [48, 30, 31, 29], that provides orthonormality of the basis $U,V$.

   To facilitate the implementation we choose the scheme of [30] ("FeDLRT") as the backbone scheme. We do not claim invention of FeDLRT here, but incorporate our regularizer into it. The applicability of the regularizer to a wide range of low-rank training methods is an advantage, as a user does not need to change their implementation to use our regularizer.

4. (Q Baselines) We denote by "Baseline" the default training of the full model (not low-rank factorized) with the training hyperparameters specified in Appendix A2; this is stated in Section 6, Line 293. DLRT and RobustDLRT use the two-step scheme of [30], which is described in detail in section 5. As denoted in the caption of Table 2, RobustDLRT with regularization parameter $\beta=0$ equals DLRT. The same methods are used in all other experiments. All other compared methods are cited.

5. (Q Adversarial training) In all experiments presented in this paper, we train solely on clean data without employing adversarial training. This choice is intentional: the proposed method acts as a regularizer and is conceptually orthogonal to data-driven approaches like adversarial training. To clearly isolate and illustrate the effect of the regularizer, we evaluate it in the unaltered, clean-data training setting.

6. (Typos) We fixed the typos - Thank you for noticing!

Thanks for the clarification of the definition and the answers by the authors. However, I still have the following concerns.

1. Clarification of Section 5. While I understand the motivation to highlight the compatibility of the proposed method, the current presentation may unintentionally give readers the impression that certain techniques (originally from prior work) are proposed in this paper. Even though the authors do not explicitly claim these as their own contributions, the phrasing and structure make the boundary less clear. I recommend explicitly clarifying which techniques are adopted from prior work and which are original contributions. For example, adding sentences such as “Following [Author et al., Year], we incorporate …” could help. Additionally, to better demonstrate compatibility, it may be more convincing to compare against multiple strong baselines rather than integrating many techniques from different papers.

2. Choice of Baselines and Robustness Concerns. The current evaluation relies on baselines that are not adversarially trained. This raises concerns, as the reported improvements in robustness might largely stem from the poor adversarial robustness of the chosen baselines rather than the effectiveness of the proposed approach. Since the technique is orthogonal to the training paradigm, I strongly suggest including experiments on adversarially trained baselines. This would clarify whether the improvements are intrinsic to the method itself rather than an artifact of weaker baseline robustness.

We thank the reviewer for their constructive review. We appreciate that they found our paper intriguing and the robust dynamical low-rank training idea interesting.

1. (Q scalability) To eleviate the concerns of scalability we provide numerical results for the ImageNet-1k (1.2 Million images), where we compress the ViT-L32 vision transformer (304 Million parameters). We first conduct a hyperparameter sweep to determine standard hyperparameters as learning rate, weight-decay and batch size, which we report in Appendix C.

   We compare then the baseline model, DLRT and the regularized RobustDLRT adversarial performance in the table below and observe similar trends as in the Vision Transformer experiments of Table 2 in the main manuscript.

Table: ImageNet, Top-1 Accuracy under various attacks

Table: ImageNet, Top-5 Accuracy under various attacks

We have measured the runtime of training one epoch on ImageNet of both DLRT and RobustDLRT on an A100 80GB. DLRT takes 26min 07 sec and RobustDLRT (with the regularizer) 27min 51 sec, so the overhead is roughly 3%. We remark that this overhead can be further reduced with an optimized implementation. We conclude that the method is scalable.

Finally we wish to remark that the slightly lower compression rate is expected since the hidden dimensions of ViT-L32 of 1024 is close to the number of ImageNet classes (1000), thus there is less redundancy in the model compared to other reported benchmarks.

2. We highlighted the best performing method in each setup of Table 2 to improve readability in the revised manuscript.

3. We changed $\hat{\mathcal{L}}$ to $\tilde{\mathcal{L}}$ in Fig.2 in the revised manuscript.

I thank the authors for taking the time to address the scalability with more experiments. After reviewing all of the other reviews and comments I feel that the current score still reflects the rating of the paper.

Scalability

We are happy to compare the scalability of our approach to prior methods from the literature, specifically SVDPrune, LoRA, and CondLR, which we also evaluated in Table 3 of the manuscript.

RobustDLRT (see Line 256 in the manuscript)

Let $s_*$ be the number of local iterations and $r$ the current rank of the $n \times n$ weight matrix.

• In each iteration, the cost for evaluating the low-rank layer is: $\mathcal{O}(2nr + r^2)$
• Every $s_*$ iterations, additional costs occur:
  • Basis update (QR decomposition): $\mathcal{O}(nr^2)$
  • Singular value truncation (SVD): $\mathcal{O}(r^3)$
• In every of the other $s_*-1$ iterations, the cost of regularizer evaluation: $\mathcal{O}(r^3)$
• The optimizer update in all the $s_*-1$ iterations are only applied to the coefficient matrix $S$, costing: $\mathcal{O}(r^2)$

Cumulative cost w/o optimizer over $s_*$ iterations: $\mathcal{O}\left(nr^2 + 2nrs_* + r^3s_* \right)$ where the terms are sorted by cost, assuming r is sufficiently small.

Optimizer cost: $\mathcal{O}(r^2s_*)$

CondLR

• In each iteration, a gradient update of the matrices $U,V$ is projected onto the Stiefel manifold via QR decomposition: $\mathcal{O}(nr^2)$
• The coefficient matrix $S$ is updated and projected onto a manifold with bounded singular spectrum via SVD: $\mathcal{O}(r^3)$
• The optimimzer is applied to U,S,V in each iteration, costing $\mathcal{O}((2nr + r^2))$ per iteration

Cumulative cost w/o optimizer over $s_*$ iterations: $\mathcal{O}\left(nr^2s_* + 2nrs_* + r^3s_*\right)$

This increases the cost by $(s_* - 1)nr^2$ over RobustDLRT (the leading term in asymptotic notation).

Optimizer cost: $\mathcal{O}((2nr + r^2)s_*)$

This increases the cost by $(s_* - 1)2nr^2$ over RobustDLRT (the leading term in asymptotic notation).

SVDPrune

• Instead of QR projection, the method regularizes $U,V$ to be orthogonal using the term $\|I - U^T U\|$ (analogously done for V), which has the same asymptotic cost as QR decomposition: $\mathcal{O}(nr^2)$
• The treatment of the $S$ matrix is identical to CondLR from a computational perspective.

Thus the asymptotic cost is the same as CondLR

LORA

• No regularization or orthonomalization is done, so no compute overhead for the method is reportable.
• Optimizer cost: The optimizer cost amounts to updating USV in $\mathcal{O}((2nr + r^2)s_*)$

We summarize that the asymptotic cost in terms of memory and compute of our method is superior to prior approaches as they are presented in the respective papers. Furthermore, the experiments in Table 2 have shown that the method is able to compress networks better than the compared methods (SVDPrune and CondLR) at a higher validation accuracy (clean and attacked) in most setups.

For transparancy and completeness, we wish to remark that CondLR and SVDPrune can be transformed to an staggered scheme with s_* local steps following the logic in our Section 5. In this case, the asymptotic cost is similar to our approach.

Parallelizability

Our method is natively compatible with data-parallel distributed training schemes. Furthermore, since the method is compatible with the federated low-rank training framework of [a], it is scalable in a tensor-parallel sense, where U,V are held on a central server and S is distributed across federated/distributed clients, e.g. GPU nodes or edge devices. We remark that this extension is not natively applicable to SVD prune, since orthogonality of U,V is only weakly enforced.

[a] Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees; Steffen Schotthöfer, M. Paul Laiu

We hope that the reviewer takes this additional information into consideration when providing a final score for our work. Furthermore we thank the reviewer for their thoughtful review and for helping us improve the paper.

We thank the reviewer for the constructive feedback and appreciate that they find the theoretical analysis and algorithmic description of the method clear.

1. (Q about motivation) The cited papers [1,2,3,4,5,6] state - just like us - that adversarial robustness correlates with the parameter count of an otherwise untreated, pre-trained neural network. To the best of our understanding, the cited works [1-5] investigate the effects of sparsity pruning on models with robust defenses in mind, i.e., adversarial training. The last reference [6] gives empirical evidence that a pretrained and then pruned network has increased adversarial robustness.

   We state in line 30 "Moreover, it has been observed that compressed networks can exhibit increased sensitivity to adversarial attacks [29]", where [29] analyzes the degrading effect of raw low-rank compression on robustness, which we confirm in, e.g., Table 2, and explain in Figure 1. However, we do not make an explicit statement about the direct effects of sparsity postprocessing, i.e. pruning, on the adversarial robustness of neural networks.

   We acknowledge however the lack of clarity of the statement and will change it to "Moreover, it has been observed that low-rank compressed networks can exhibit increased sensitivity to adversarial attacks [29]"

2. (Q about PDG attack) In Table 2, we evaluate the baseline, DLRT, and RobustDLRT models with the FGSM, Jitter, and Mixup adversarial attacks. Jitter and Mixup, described with references in Sections A.1.3 and A.1.4 respectively, are both iterative and gradient-based adversarial attacks that are quite strong.

   To address the reviewers concerns, we also ran the same experiment in Table 3 but with the $\ell^2$-PGD attack. Overall, we see that robustDLRT is competitive with the other robustness-improving methods when the compression rate is factored in.

   The first three rows list the computed mean over 10 random initializations.
   The values of all other methods, given below the double rule, are taken from [Savostianova 2023; Table 5]. We remark that RobustDLRT outperforms CondLR with τ=0.1 for $\epsilon$<=0.2, and CondLR with τ=0.5 for $\epsilon$<=0.26.

   We wish to remark that opposed to the experiments in Table2, we do not sweep over β but take the value that is obtained from the $l_1$ FGSM results of Table 2.

RobustDLRT has competitive adversarial accuracy to all methods with a compression rate ≥ 80%.

3. (Q about sparsity vs low-rank) Pruning[1-6], i.e. sparsity postprocessing, of pretrained [6] or pretrained on adversarial examples [1] networks can maintain adversarial robustness. We remark that this requires the computational expense of (a) full network training, (b) sparsification and (c) finetunning.

   The distinguishing factor of our work is that RobustDLRT is able to adaptively compress and influence the spectrum of a neural network during training. Moreover, direct access to the spectrum during training is computationally viable when a low-rank factorization of the layer weights is used. This puts the contributed work in the beneficial zone of low compute/memory cost during training and inference and high clean accuracy and high adversarial accuracy/robustness; these properties required for the target application of resource constrained edge devices.

   You are correct in your assessment that on many hardware architectures, low-rank based compression increases throughput compared to sparsity compression formats.

   We include a clarifying statement in the related work section of the revised manuscript to disambiguate the scope of this work from sparsity pruning scenarios, where (to the best of our knowledge) the full-model pretraining is treated as an offline cost.

4. We will highlight all runs where RobustDLRT outperforms the full-rank baseline model. We remark that these runs are surpassing the theoretical guarantees of DLRT of achievieng close-to-baseline performance, see [a]. In all runs the regularized method outperforms the unregularized low-rank baseline "DLRT".

   [a] Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations; Schotthöfer et al. 2022 NeurIPS