Deep Sturm–Liouville: From Sample-Based to 1D Regularization with Learnable Orthogonal Basis Functions

We would like to thank the reviewer for carefully reading our paper, providing thoughtful feedback and recognizing the novelty of our approach.

I — In response to the reviewer's main concerns:

1 — 1D Regularization vs. 0D Regularization: By 0D regularization, we refer to regularization applied at discrete data points. For example, a term like $\sum_i \lvert \nabla_x F(x_i) \rvert$ computes the gradient of the predictor $F(x)$ only at the sampled points $x_i$. Ideally, one would compute a continuous regularization such as $\int_{\Omega} \lvert \nabla_x F(x) \rvert dx$, but this is generally intractable. Our method bridges this gap by computing regularization along one-dimensional trajectories defined by a vector field $a(x)$ for each data point $x$. This 1D regularization provides a more informative approximation of the continuous case by integrating along these paths. Within this framework, we introduce two new types of regularization: spectral regularization and implicit regularization. These are complementary to traditional techniques (such as L1 or L2 regularization), which we do not aim to replace. In future work, our framework could also incorporate other forms of regularization or even modify the loss function itself.

2 — Implicit Regularization: The key idea behind implicit regularization in our framework is to retain only the first $n^\text{th}$ basis functions. By controlling the number of retained basis functions, we also control their level of oscillation—a direct consequence of the Sturm–Liouville theorem (see lines 173–180). This is analogous to the Fourier basis, where higher-order components exhibit more oscillation than lower-order ones. However, Sturm–Liouville provides a more general formulation. This property allows us to construct smoother function approximators, which have a known theoretical connection to generalization. Moreover, sample efficiency is inherently tied to generalization performance (see [1, 2] ). Our experiments on sample efficiency were specifically designed to assess generalization behavior. By limiting the number of basis functions, we directly control the expressiveness and smoothness of the learned function, thus influencing the regularization effect.

II — Regarding the reviewer's concerns on "Experimental Design and Analyses":

1 — Architectural Ambiguity for the Baseline: The reviewer is right, and we will include a dedicated section to clarify this point in the appendix. For MNIST and CIFAR-10, we report the results from Massaroli et al. (2020), and we will explicitly reference this in the appendix. For the tabular datasets, we used an architecture similar to the one used for the function $a(x)$. The source code is accessible via an anonymous GitHub https://rb.gy/n5iz02. Upon acceptance, the source code will be made publicly available to support the reproducibility of our work.

2 — Incomplete Hyperparameter Analysis: We conducted an ablation study in Figure 4(b) of the paper to analyze the impact of the spectral regularization coefficient on training accuracy. However, the coefficient $\alpha$ was missing from the figure and no reference was made to Equation (9). We will update the figure and its legend to include both the coefficient and the appropriate reference, as their absence is indeed misleading.

3 — Computational Fairness: The reviewer is right to highlight that computational budget is a critical factor for fair comparisons. For MNIST and CIFAR-10, we chose to report the results from Massaroli et al. (2020), a recognized reference. For the tabular datasets, we used similar architectures for both the baseline and the DSL model to ensure a fair comparison in terms of parameter count and model complexity.

Lastly, thank you for pointing out several typos in the manuscript—we will correct them accordingly.

[1] Zhang, Chiyuan, et al. "Understanding deep learning requires rethinking generalization." arXiv preprint arXiv:1611.03530 (2016).

[2] Arpit, Devansh, et al. "A closer look at memorization in deep networks." International conference on machine learning. PMLR, 2017.

The authors would like to thank you for your detailed review, for carefully examining the demonstrations and for recognizing the novelty of our approach.

I — Demonstration 2

First, we would like to address the mistake you identified in Demonstration 2, specifically in lines 728–732. The reviewer is absolutely correct, and we sincerely thank you for pointing this out. This demonstration is a key element of our paper, and we greatly appreciate your attention to its details. Fortunately, the mistake can be corrected without affecting the main results. Below, we explain how to fix it. As a reminder, the objective of this part of the demonstration is to construct a change of variables such that the first row of the Jacobian matrix of the mapping function $(t, y = F(x))$ is the vector $a(x)$ which is required to be orthogonal to the remaining rows of the Jacobian.

(i) — As the reviewer correctly pointed out, $a(x)$ must be the gradient of a scalar function. We will explicitly add this as a condition in the theorem. This is directly related to lines 251–252 of the paper, where we assume the absence of singular points.

(ii) — The reviewer also rightly pointed out an issue in the construction of the orthogonal basis: the Gram-Schmidt process alone is not enough as the resulting vectors also need to be gradients of scalar functions. To address this, we will leverage tools from differential geometry by interpreting the change of variables as defining a one-dimensional bundle generated by the vector field $a(x)$. We will add the following explanation to the paper:

``Let $E$ be a one-dimensional real vector bundle over a manifold $M$. It is known (see Milnor & Stasheff's Characteristic Classes) that $E$ is trivial if and only if it admits a global nowhere‐vanishing section—equivalently if its first Stiefel–Whitney class $w_1(E)$ vanishes. In this case, a nowhere‐vanishing section $a$ provides a canonical trivialization $E \cong M \times \mathbb{R}$. Furthermore, if we choose local coordinates adapted to this trivialization and if there exists a smooth function $V : \mathbb{R}^n \rightarrow \mathbb{R}$ such that $\nabla V(x)=a(x)$, the Jacobian of the corresponding local map $T_x$ can be arranged so that its first row is precisely $a(x)$. This reflects the fact that the coordinate system is chosen to align the fiber direction (generated by $a(x)$) with the first coordinate axis. With an additional Riemannian metric on $M$, the orthogonal complement Bundles lemma (see John M. Lee, Introduction to Smooth Manifolds) then provides an orthogonal decomposition $T_xM = \operatorname{span}\{a(x)\} \oplus \operatorname{span}\{a(x)\}^\perp,$ completing the geometric picture. Consequently, the first row of the Jacobian is exactly $a(x)$, aligning the coordinate system with the direction defined by $a(x)$. Combined with the orthonormality of the remaining coordinates, this ensures that the first row of the Jacobian matrix is orthogonal to all other rows.''

II — Ablation study

To evaluate the role of the vector field in our approach, the reviewer suggested an ablation study that removes its influence. We agree this is a valuable experiment and would like to describe it in more detail. In this ablation, we apply the proposed Sturm–Liouville formulation over the time interval $[0, 1]$ and evaluate the basis functions at $t = 0.5$. The input to each MLP is formed by concatenating the data point $x$ with the time variable $t$.

For the sample efficiency results and the accuracy performance on tabular data, please find below the updated plot and table that includes the ablation model https://rb.gy/xjx9tk. Our results show that DSL achieves higher sample efficiency and comparable accuracy compared to the ablation model.

III — Weakness and other questions

1 — Code Availability: The source code is accessible via an anonymous GitHub https://rb.gy/n5iz02. Upon acceptance, the source code will be made publicly available.

2 — Use of Neural Networks in DSL: We acknowledge the dependency of the DSL on neural networks. As a direction for future work, we are exploring ways to design a version of the DSL that does not rely on neural networks. DSL represents a foundational block that we aim to develop further.

3 — 0D vs 1D Regularization: Since this question was also raised by another reviewer, we addressed it in the rebuttal of Reviewer cdFW (1).

4 — DSL Coverage of $\Omega$: The reviewer is right to point out that, for a general vector field, the trajectories might not cover the entire domain $\Omega$. To ensure full coverage, the ODE must satisfy certain constraints. These are discussed in the paper between line 215 (second column) and line 253 (first column), where we explain the uniqueness of $\gamma(t_-)$ and $\gamma(t_+)$. However, we realize this section may not clearly state that these constraints also guarantee coverage of $\Omega$. We will add an explicit sentence to highlight this point.

We thank the reviewer upBa for taking the time to review our paper in detail and for recognizing the novelty of our approach. The novelty of our approach was highlighted also by reviewer 2moq, who described our article as "highly original with no directly related techniques in a published literature". We appreciate your concern regarding the scalability of our method to large-scale datasets or real-time applications. We fully agree that this limitation may hinder the immediate widespread deployment of our framework in such contexts.

However, as the reviewer noted, the core contribution of our work lies in introducing a new and principled approximator framework for regularization, grounded on theoretical results. As such, we believe it should be viewed as a foundational step—laying the groundwork for future improvements and extensions.

Improving scalability is a clear direction for future work. Potential avenues include designing more efficient approximations of the Sturm–Liouville basis, new methods to compute gradients through the eigenvalue process that avoid relying on the implicit differentiation theorem, as well as exploring alternative ODE solvers that are better suited to Sturm–Liouville-type equations. Additionally, we plan to explore new forms of regularization beyond the spectral and implicit ones introduced here. We see this work as the first step in a broader research agenda that combines structure, theory, and learning in a unified framework. We will update the paper to make this limitation clearer—especially in the limitations section—and outline these directions as promising areas for future work.