Improved Particle Approximation Error for Mean Field Neural Networks

Thank you for the positive feedback and useful suggestions.

The Authors do not really motivate why one should expect that the particles approximation does not depend on the LSI constant.

The particle approximation error is due to the nonlinearity of $F_0(\mu)$ with respect to $\mu$. In fact, Langevin dynamics, a special case of MFLD for the linear functional $F_0$, can be simulated with only one particle. On the other hand, LSI-constant is affected by the regularization strength $\lambda, \lambda'$ and the boundedness or Lipschitz continuity of $F_0( \mu_\mathbf{x})$ or $\delta F_0( \mu_\mathbf{x}) /\delta \mu$ w.r.t $\mathbf{x}$. For instance, in the learning mean-field neural network setting (Eq. (14)), LSI is satisfied as long as the activation $h$ is bounded. In this way, these two concepts (approximation error and LSI) come from different factors and do not seem to have a direct relationship, and hence we expect that an LSI-constant free approximation error can hold.

Q: line 61: How does the LSI constant deteriorate with the regularization parameter? Could you provide a sketch to give some more information? This could be also worth to be included and referenced in the manuscript.

Let us consider the Gibbs distribution proportional to $\exp( f(x) - g(x) )$, where $g$ is $c$-strongly convex and $f$ is uniformly bounded by $B$. Then, combining techniques of [Bakry and Émery (1985)] and [Holley and Stroock (1987)], we can prove LSI with an estimated LSI constant: $c \exp(-4B)$. Please also refer to Appendix A.2 of [Nitanda, A., Wu, D., and Suzuki, T. (2021)]. In our case of the proximal Gibbs distribution, $f$ and $g$ correspond to $- \frac{1}{\lambda} \frac{\delta F_0(\mu)}{\delta \mu}$ and $g = \lambda' \|x\|\_2^2$, respectively. Hence we can conclude that LSI constant is $\frac{2\lambda' }{\lambda} \exp(- 4B/\lambda )$, as described on line 61 if $\|\frac{\delta F_0(\mu)}{\delta \mu}\|_\infty \leq B$. This estimation is often used in the literature. For instance, please see [Nitanda et al., 2022; Chizat, 2022; Suzuki et al. 2023b]. Moreover, $f$ could be Lipschitz continuous function because we can reformulate "the sum of Lipschitz continuous function and strongly convex function" into "the sum of bounded function and strongly convex function" by using Miclo's trick [Bardet et al., 2018]. We will mention these techniques in Appendix.

[Bakry, D. and Émery, M. (1985)] Diffusions hypercontractives. In Seminaire de probabilités XIX 1983/84, pages 177–206. Springer, 1985.

[Holley, R. and Stroock, D. (1987)] Logarithmic Sobolev inequalities and stochastic ising models. Journal of statistical physics, 46(5-6):1159–1194

Other suggestions

Thank you for your thorough reading. We will update the manuscript and fix typos according to your comments.

Thank you for the positive feedback and helpful comments.

Besides Theorem 1, the remaining results seem to be a straightforward application. The convergence rate $\exp(-\alpha \lambda\eta k)$ and error bound $\eta/\alpha \lambda$ still suffer from the LSI constan.

As the reviewer pointed out, the results in Section 3.2 are basically obtained by the combination and modification of existing results. However, we would like to emphasize that our main aim is to derive an improved particle approximation error, and the results in Section 3.2 serve to demonstrate the effectiveness of our result. The fact that we can improve the approximation error of existing results by combining them with our result would be a point worth noting.

The Wasserstein error bound suffers from the LSI constant. Q: however is there a possibility that the time discretization error or Wasserstein error can also be made independent of the LSI constant?

At the moment, it is nontrivial whether the time discretization error of MFLD and Wasserstein propagation chaos can be independent of the LSI constant. However, it is worth mentioning that we can derive a propagation of chaos regarding TV norm, which is free from the LSI constant. This can be proven by replacing Talagrand's inequality with Pinsker's inequality $\mathrm{TV}(\mu , \mu') \leq \sqrt{2 \mathrm{KL}(\mu \| \mu')}$. The convergence in terms of TV norm is also important as it implies the weak convergence of distributions. We will add this comment in the revision.

The standard assumptions of MFLD apply, for example the activation must be Lipschitz smooth as well as bounded to ensure that Assumption 1 holds. These implications should be explained in the text alongside Assumption 3.

Thank you for the suggestion. Indeed, the smoothness and boundedness of the activation function can be used for verifying Assumption 1. We will update the manuscript accordingly.

Thank you for your careful reading and feedback.

Verification of Assumption 4 and Lemma 22 of [Kook et al., 2024]

Thank you for bringing up this point. Indeed, the authors of [Kook et al., 2024] updated their manuscript due to an error in Lemma 22 of the previous version. As a result, a new $N$-independent LSI constant (Corollary 25 of [Kook et al., 2024]) requires a strong assumption on $N$. Therefore, we plan to simply replace it with the LSI constant: $\alpha \geq \frac{\lambda'}{\lambda}\exp(-NB/\lambda)$, supported by Holley-Stroock argument under boundedness of $F_0$. Although this LSI constant depends on $N$, it is useful in practice because we basically want to use the minimum number of $N$ to achieve a given accuracy.

In fact, an LSI constant $\frac{\lambda'}{\lambda}\exp(-NB/\lambda)$ is rather better than the original constant in between lines 204-205 of the submission when $N \leq 1/\lambda'$, which aligns with practical machine learning settings. Specifically, for a given number of examples $n$ and $\gamma>0$ (e.g., $\gamma=1/2, 1$), we usually set $\lambda' = 1/n^\gamma$, and our particle approximation bound also suggests an error of $1/n^\gamma$ when $N=n^\gamma$, which indeed satisfies $N \leq 1/\lambda'$. In other words, under this typical machine learning setting, the estimated LSI-constant $\alpha \geq \frac{\lambda'}{\lambda}\exp(-NB/\lambda)$ is better than the LSI constant (lines 204-205) based on the previous version of [Kook et al., 2024].

Moreover, we confirmed that our proof technique can be used to improve their LSI result (Corollary 25 of [Kook et al., 2024]) so that it holds any $N$. Indeed, the assumption on $N$ of Corollary 25 stems from Wasserstein propagation of chaos (Theorem 26), which inherently requires $N \geq 1/\alpha$ (see also Theorem 5). On the other hand, our propagation of chaos result holds for any $N$, and hence, we can remove this condition from Theorem 26. However, our aim is not to improve the LSI but to derive an LSI-constant free particle approximation error. Therefore, the improvement of the LSI will be studied in future work with further refinement, and we will just utilize $N$-dependent LSI for our submission.

The main challenge in handling unbounded activations

The boundedness is due to several factors: Lipschitz smoothness of the first variation (Assumption 1), the LSI-constant by Holley-Strook argument, and the evaluation of Bregman divergence. That being said, we can relax the boundedness for each factor. For instance, the boundedness of $h(\cdot,z)$ can be replaced with the Lipschitz continuity for deriving the LSI-constant and with the bounded second moment of $h(X,z)~(X\sim \mu_*)$ for evaluating Bregman divergence. The most critical point to be careful about is the relaxation for Assumption 1. Indeed, Assumption 1 with an unbounded activation function introduces another limitation: the output layer must be fixed, and the derivative of the loss function $\ell$ must be bounded (e.g., logistic loss).

We also would like to note that the boundedness of the first variation in Assumption 1 simply says the differentiability of $F_0$. This seems redundant, so we will omit it in the revision.

In Line 162, the authors refer to Equation (11) as the continuous-time propagation of chaos error, while Equation (11) only considers the error at optimality

Thank you for pointing it out. Indeed, Eq. (11) only considers the error at the solution. We will simply reward this sentence as follows: "The finite-particle approximation error $O(\frac{\lambda}{\alpha N})$ appearing in (11), (13) means ..."

Moreover, we will update the manuscript according to your suggestion and fix the typo. Thank you for pointing them out.

Thank you for the positive feedback and thoughtful comments.

About bounded activations

In fact, the boundedness of the activation function is commonly assumed in the literature. Hence, it is not a critical limitation. That being said, we can relax the boundedness for each factor. For instance, the boundedness of $h(\cdot,z)$ can be replaced with Lipschitz continuity for deriving the LSI constant and can be replaced with the bounded second moment of $h(X,z)~(X\sim \mu_*)$ for evaluating Bregman divergence. The most critical point to be careful about is the relaxation for Assumption 1. Indeed, Assumption 1 with an unbounded activation function imposes another limitation: the output layer must be fixed, and the derivative of the loss function $\ell$ must be bounded (e.g., logistic loss).