A ROBUST DIFFERENTIAL NEURAL ODE OPTIMIZER

We would like to thank the reviewer for their positive remarks, as well as their useful comments and further suggestions for improving our work. In the following we address each of the concerns and suggestions of the reviewer.

1. Computational overhead

• While we admit that our method introduces a second set of weights, we highlight that most adversarial training methods typically include nested optimization schemes that are computationally demanding. The main advantage of our methodology is that the antagonizing weights are updated through a single backpropagation, which reduces computational complexity while also achieving robustness even with natural training bypassing the computational burden of perturbing the input. In addition, note that we also enhance computational efficiency by exploiting efficient matrix decompositions.

2.Experiments on CIFAR-100 and Tiny ImageNet and a more extensive comparison with various state-of-the-art optimizers that enhance robustness

• Following the suggestions of the reviewer, we present further comparisons against the state-of-the-art optimizers that enhance robustness such as Lyanet[1], and FI-ODE[2]. Additionally, we also evaluate the all optimizers in additional datasets: CIFAR100 and TinyImageNet. These results in the Table below further indicate the robustness of our optimizer as it outperforms state-of-the-art robust optimizers in multiple datasets. We also highlight the significantly shorter training time required by GTSONO in contrast to Lyanet and FI-ODE.

  Optimizer	CIFAR100-PGD_20^0.03	Tiny-ImageNet PGD_20^0.03	Training TinyImageNet (min:sec)	# of Parameters
  Adam	$10.7\pm1.1$	$1.9\pm0.1$	5:37	1.35 M
  SNOpt	$19.7\pm0.6$	$3.6\pm0.9$	5:41	1.35 M
  SGD	$9.6\pm0.5$	$0.8\pm0.2$	5:20	1.35 M
  LyaNet^*	$20.1\pm0.5$	$5.4\pm0.4$	313:21	19.6 M
  FI-ODE^*	$15.4\pm0.4$	$4.0\pm0.2$	607:56	2.4 M
  c-GTSONO	$23.5\pm0.1$	$9.1\pm0.8$	6:35	1.47M

  CIFAR10-Optimizer	PGD_20^0.03	PGD_20^0.05	Time (min:sec)	# of Parameters
  LyaNet^*	$46.1\pm0.4$	$32.7\pm0.6$	32:44	19.6 M
  FI-ODE^*	$48.5\pm0.3$	$33.4\pm0.6$	143:17	2.4 M
  c-GTSONO	$50.6 ± 0.3$	$35.0 \pm 0.2$	3:53	1.35M

3. Insights into the stability and convergence properties of GTSONO, especially under varying hyperparameters and different neural network architectures?

• The convergence guarantees are agnostic of the architecture and hyperparameter configuration. The only necessary assumption for the convergence guarantees to hold should be that the function of our accumulated loss $Q$ is locally convex-concave - which can be imposed through appropriate selection of the regularization parameters $R_u, R_v$

4. In general, deep learning optimizers have used mini-batch size to obtain a partial batch of datasets, which means that random noise will be introduced, so the optimizer's parameter update rule should correspond to a stochastic differential equation instead of an ordinary differential equation. Therefore, why do the authors use an ODE rather than an SDE for their theoretical analysis?

• We would like to kindly emphasize that the stochasticity comes from the input, however this does not affect the dynamics which remain deterministic. Therefore, the appropriate representation is through an ODE instead of an SDE. This allows us to employ deterministic optimal control which naturally leads to a deterministic update law.

As we are approaching the end of author-reviewer discussion, we would appreciate the reviewer to clarify if our responses has sufficiently addressed the raised concerns, and, if so, we kindly appreciate the reviewer to re-consider their rating.

We truly appreciate the encouraging comments of the reviewer, as well as their useful remarks. In the following we address the points raised by the reviewer.

1. Gap why the optimization of min-max optimal control can be beneficial to the robustness problem in the adversarial training problem

• In our approach, we introduce the antagonizing set of controls to represent the disturbances in the system, inspired by game theoretic optimal control. This key representation enables us to utilize computationally efficient and scalable trajectory optimizers such as min-max Differential Dynamic Programming (DDP). In the realm of Optimal Control (OC), it has been demonstrated that game-theoretic formulation enables the model handling uncertainties and external disturbances [1].
• The key advantage of incorporating as adversarial control inputs lies in the fact that instead of typically obtaining nested optimization schemes that are computationally demanding, our algorithm updates the antagonizing weights with a single backward pass which reduces computational complexity while also achieving robustness even with natural training bypassing the need of perturbed input. **

2. Complexity comparison between GTSONO and other Neural ODE optimizers: first-order adjoint and SNOpt.

• We updated the paper, in order to present better the complexity comparison between GTSONO and other Neural ODE optimizers, adding a pertinent table in the main part of the paper. For an extensive comparison with regards to the computational requirement of the optimizers, we refer the reviewer to Tables 8-12 in Appendix D, which show training time and memory consumption.

3. Clarification on the selection to demonstrate the effectiveness of our optimizer on input-robustness

• While we mostly refer the reviewer in weakness 1, regarding our motivation to recast the optimization problem with input disturbance to a game theoretic formulation. We chose to demonstrate the effectiveness on input-robustness, given that this is the most widely used form of attack. We also appreciate the suggestions of the reviewer to include studies on systems weights which we intend to explore in our future work.

4. In the experiments, it seems that only having adversarial control on convolution layers is much better than having them on all of the layers. From the theory parts it is not clear why this is the case. The authors should provide more analysis on this.

• We provide some intuition for this phenomenon. To quantify the similarity between the feature vector of the models from attacked and clean images, we calculate the cosine similarity. We find that the model with adversary weights in the convolution layers evaluated with attacked images is significantly closer to the feature vector from the clean images (cosine_similarity=0.92), compared to the feature vector from attacked images in the model without adversary control in the convolution layers (cosine_similarity=0.21). Therefore, feeding a robust feature vector to the classifier is similar to feeding the feature vector from the clean images, which eliminates the need to include antagonizing weights in the last linear layers, as it is well known that they drop the natural accuracy of model. Therefore, since the disturbance is in the input, this implies that it suffices to add adversary control in the convolution layers in order to capture robust features for robustifying the model. We intend to to investigate more thoroughly the underlying theory in future work.

5. When combing with adversarial training (table 4), why CW accuracy drops? This may indicate some gradient obfuscation issue, it will be good to include some stronger attacks in the evaluation (like AutoAttack, Square) as suggested in weakness 3.

• Based on the reviewer’s recommendations we have also added AutoAttack and Square attacks to assess our optimizer. In the tables below, we compare the robustness of GTSONO against the baseline classifiers on CIFAR-10 with natural training, and with TRADES adversarial training. These results provide further evidence for the robustness of our optimizer against multiple attack methods, both with natural and adversarial training. We also appreciate the insight of the reviewer regarding the issue with gradient obfuscation, which we intend to explore more thoroughly in our future work.

References

1. Sun, W., Pan, Y., Lim, J., Theodorou, E. A., & Tsiotras, P. (2018). Min-max differential dynamic programming: Continuous and discrete time formulations. Journal of Guidance, Control, and Dynamics, 41(12), 2568-2580.

As we are approaching the end of author-reviewer discussion, we would appreciate the reviewer to clarify if our responses has sufficiently addressed the raised concerns, and, if so, we kindly appreciate the reviewer to re-consider their rating.

4. GTSONO natural accuracy in CIFAR-10

• We acknowledge that indeed GTSONO demonstrates a decrease in natural accuracy towards achieving high robustness. Note that this is a well-known trade-off between robustness and accuracy, as studied in [3]. Therefore since our focal focal point is the increased robustness of our optimizer it follows that a decrease in the natural accuracy of our optimizer may be observed. Another interpretation of this phenomenon is that the value function of GTSONO converges to a saddle point, due to the effect of the antagonizing control . This can be viewed as a compromise in the natural accuracy of the optimizer when compared to other optimizers which converge to a local minimum. It should be noted that the value functions of GTSONO and the baseline optimizers are not the same, therefore this can not serve for direct comparisons, however it provides us with an intuition for this phenomenon.

References

1. Rodriguez, I. D. J., Ames, A., & Yue, Y. (2022, June). LyaNet: A Lyapunov framework for training neural ODEs. In International Conference on Machine Learning (pp. 18687-18703). PMLR.
2. Huang, Y., Rodriguez, I. D. J., Zhang, H., Shi, Y., & Yue, Y. (2022). Fi-ode: Certified and robust forward invariance in neural odes. arXiv preprint arXiv:2210.16940.
3. Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., & Jordan, M. (2019, May). Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning (pp. 7472-7482). PMLR.

We would like to thank the reviewer for their positive comments, as well as their constructive criticism and further suggestions for improving our work. In the following we address the points raised by the reviewer.

1. Novelty of Proposed Approach

• We first emphasize that our approach differs considerably from conventional min-max methods for adversarial learning. While most min-max adversarial training methods consider the adversary input, our proposed method instead consider the disturbance being injected through the auxiliary weights. This, as also recognized by Reviewer gukw, offers a novel perspective by recasting the Neural ODE optimization as a robust trajectory optimization through the lens of min-max optimal control theory. This key reformulation enables computationally efficient and scalable trajectory optimizers (such as min-max Differential Dynamic Programming presented in our work) that is otherwise absent in prior adversarial formulations. We urge the reviewer to recognize the distinction.
• The experiments empirically verify our choice as our algorithm converges to a saddle point even when trained with clean images, which is translated to an increased robustness of our model against adversarial attacks, even with natural training.

2. Convergence proof

• While we do acknowledge that our convergence analysis is partly inherited from prior theoretical optimization tools, we emphasize that it holds merit to show that our optimizer has theoretical convergence guarantees to a local saddle point. This kind of (local) convergence analysis is crucial in characterizing training stability, especially in deep learning-based algorithms, yet are absent in those inspired by optimal-control methodologies. Furthermore, we highlight that the convergence of our algorithm is still not a direct application of the known convergence results as we need to leverage the formulation of our update law, and the preconditioning with the inverse of the Jacobian of G, for proving that the spectral norm of the Jacobian of our update law is guaranteed to be less than 1 - which implies stable convergence to the saddle point.

3. Experiments on CIFAR-100 and Tiny ImageNet and comparison with other methods

• Following the suggestions of the reviewer, we present further comparisons against the suggested optimizers Lyanet[1], and FI-ODE[2], as well as on the additional datasets: CIFAR-100, and TinyImageNet. We observe that GTSONO outperforms all baselines , in every tested dataset, while taking slightly longer time to train than SGD. Furthermore, we observe that the next best in performance are LyaNet and FI-ODE, which however have significantly more parameters, leading to dramatically longer training time. Finally, we would like to emphasize that the experiments on the optimizers LyaNet, and FI-ODE are conducted with the architectures and hyperparmeter configurations selected by [1], and [2].

As we are approaching the end of author-reviewer discussion, we would appreciate the reviewer to clarify if our responses has sufficiently addressed the raised concerns, and, if so, we kindly appreciate the reviewer to re-consider their rating.