Scaling laws for learning with real and surrogate data

Essentially all weaknesses pointed by the referee concern literature review. We will expand the comparison with literature.

Regarding the specific suggestions provided:

1. Data augmentation. We feel this is a significantly different setting. In data augmentation, the new data is typically obtained by perturbations/transformations of the original one. As such, the dependence between original and surrogate data need to be carefully modeled.
2. Distributionally robust optimization. This is indeed an interesting connection that is worth exploring.
3. We agree that “introducing weighted ERM” is not accurate. We will replace it with “investigating weighted ERM” or similar.
4. We agree that the content of Theorem 3 is not easy to parse. We will complement it with some simplified/approximate expressions that are easier to parse and interpret.

Questions:

Q1: The symbol $\asymp$ indicates equivalence up to (possibly large) multiplicative constants. In several of our mathematical examples (and in simulations) constants seem to be close to 1 (possibly with some small additive error). We use $\approx$ to indicate that we interpret the scaling law in this stronger sense.

Q2: $\beta$ is a scaling exponent that quantifies how the test error scales with the sample size. Scaling laws have been widely popular among AI practitioners and in those cases this exponent is fitted to empirical data. In these examples we compute the scaling exponent explicitly.

Q3: Indeed, we assume $n=Q^d$ for mathematical simplicity. It is standard to assume such “regular designs” in nonparametric regression because this simplifies the analysis, without changing the final result.

We agree that it would be important to generalize our theory to classification loss. At the same time, a substantial number of results suggest that the two settings are not as different in high dimension. Among others

1. Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and Anant Sahai, Classification vs regression in overparameterized regimes: Does the loss function matter?, Journal of Machine Learning Research 22 (2021),
2. Montanari, A., Ruan, F., Sohn, Y. and Yan, J., 2019. The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime.

These papers show that the risk of max margin classification is either identical (paper 1) or very similar (paper 2) to that of least square regression in high dimension. We also would like to point out that some of our simulations are for square loss and some for classification loss. The results are qualitatively very similar.

We also agree that investigating the sensitivity to the regularization value is an important research question. At the same time, we would like to point out that:

1. Theorems 1 and 2 keep holding (up to constants) if the regularization parameter is changed by a multiplicative constant.
2. Theorem 3 captures the entire dependence on the regularization parameter. We did not explore this dependence in the plots presented in the manuscript, but we can do it straightforwardly (it is just a matter of changing a parameter in a function that evaluate the formula).

The setting in Section 3.1 is really motivated by nonparametric regression and other nonparametric settings (eg the Sobolev regression problem of Section 3.2). In these cases, the theta_k are Fourier or wavelet coefficients and $\Omega$ encodes the prior smoothness information about the signal. For instance, cubic smoothing splines correspond to the case $\Omega_k$ proportional to $k^4$.

The reviewer mentions that the paper does not take “adequate care to ensure that non-expert readers” do not “add junk data to their training”. We would like to emphasize that, to the contrary, this is exactly the problem that we address in our paper. If the weight \alpha is optimized through a consistent procedure (eg cross-validation), then the resulting learning procedure is very robust to the use of bad surrogate data.

Dear Reviewer 3jgb,

Thank you very much for submitting your review report. The author(s) have posted responses to your review. Could you kindly provide comments on whether your concerns have been adequately addressed?

AC

1. Thanks for the positive feedback!

2. We agree that studying classification and other losses would be an important next step.

3. Thanks for pointing out the typos. We will correct them in the final version. We will also add the definitions of the symbols/concepts pointed out by the reviewer.

4. The “closeness” enters implicitly in the first term on the right-hand side of Eq (4), namely the excess population error of training on (infinitely many) surrogate samples. In this formula we assume that the population error of training on (infinitely many) original samples is 0. Hence this term is non-negative and equal to 0 if and only if surrogate distribution coincides with the original one.

We agree: we will expand the related-work discussion.

At the same time, part of this criticism is unfair. The distinction between “technical report” and “research paper” depends on the subcommunity. Mathematical papers tend to be more technical and less focused on positioning.

Review: ‘The word 'unrelated' makes it a very strong argument. Surprisingly, I did not find it clearly defined in the main paper.’

Response: The abstract and introduction try to convey our results in an intuitive way. This sentence means that the surrogate data have a distribution very different from the original one. In Section 2 we give a very simple mathematical example in which this claim is formalized and formally proven. Subsequent sections provide more sophisticated examples.

So the claim is both rigorously stated two pages after it is introduced, rigorously proven, and, judging from the referee report, very surprising.

Review: Actually, I don't think learning with unrelated surrogate data will always reduce model loss.

Response: Actually Section 2 already gives a toy example in which adding surrogate data (and proper data-driven weighting) will yield a non-zero improvement, regardless of the distance between original and surrogate data!

In practice, this improvement can be negligibly small, when the distribution of surrogate data is very different from the original one. However, we find that the improvement is often non-negligible in practice. Further, a data-driven weighing strategy yields significant improvements even in cases where the common practice of naively mixing real and surrogate data with equal weight hurts rather than helps.

Review: The work does not have a standalone discussion for its limitations.

Response: Section 5 is a standalone discussion of the paper’s limitations.

Dear Reviewer XfMG,

Thank you very much for submitting your review report. The author(s) have posted responses to your review. Could you kindly provide comments on whether your concerns have been adequately addressed?

AC

In our paper we provide two recommendations for selecting $\alpha_*$

1. Compute the error on a validation split from the original data for various values of $\alpha$, and optimize among those. (This can be improved using cross validation)
2. Use the scaling law expression, with free parameter fitted to empirical error at $\alpha=0$ and $\alpha=1$. The first technique can be proved to be consistent under standard assumptions. The second one is computationally faster. Of course, we can think of intermediate strategies as well. We will emphasize these recommendations.

Indeed, the result for high-dimensional ridge regression uses Gordon inequality. This approach can be generalized to cover cases with non-identity covariance at the price of becoming technically heavier. (See, e.g. Celentano, M., Montanari, A. and Wei, Y., 2023. The lasso with general gaussian designs with applications to hypothesis testing. The Annals of Statistics, 51(5), pp.2194-2220.)

The connection to model collapse is indeed worth pursuing. One interesting remark is that there exist weighting schemes that will prevent model collapse.

Thanks for your questions:

a. The analysis of this type of regularization (l2 penalty centered at theta**) could be carried out in the same setting as the one we do. Indeed, it can sometimes be viewed as a special case of the analysis we do. However, there are three important differences:

1. We consider the surrogate sample size m to be finite. The effect of finite m in the scaling laws we derive (Eq (4), Eq (10), Eq (11), etc) is important, and would not be modeled by the setting proposed by the AE: the first error term in the square brackets in Eq. (4) would disappear. On the other hand, we believe it is very important to capture this effect. In practice, the scaling law (4) can be used to decide how much surrogate data to acquire, and this is only possible if we model the effect of finite m.

2. The regularization effect produced by surrogate data depends of course on the distribution of the surrogate data itself. While thinking of it as an l2 penalty can provide useful intuition, it is not generally correct even in very simple models. For instance, in the Gaussian sequence model of Section 3.1, if the model is heteroscedastic, then the regularization will be a weighted l2, depending on the noise distribution. Similarly for our other theoretical results. These examples can be treated as corollaries of our results, and we plan to add them to our revision.

3. Finally, the choice of theta** is of course very important, and can significantly change the behavior of the weighted ERM scheme. While in very simple models and/or theoretical work, we can pretend that we have a good guess, in large high-dimensional models, there is typically no practical way to come up with a meaningful guess without surrogate data.

b. Thank you very much for these points. Indeed the $\Lambda(\alpha)$ is a typo: this should be $R_{test}(\alpha)$. The uniformity interval $[\epsilon_0,1-\epsilon_0]$ can be replaced by $[0,1]$ when $\delta,\delta_s>1$. This is also confirmed visually in Figure 4. If $\delta$ or $\delta_s<1$, then a `phase transition’ can occur at $\alpha=0$ or $\alpha=1$, which is why we focus on the interval $[\epsilon_0,1-\epsilon_0]$. While this phase transition phenomenon is quite interesting, we do not have space to elaborate on it in this submission. We know however that when this phase transition occurs, the test error is typically higher at $\alpha=0$ or $\alpha=1$. Therefore, from a practical point of view, there is no loss of generality in focusing on $[\epsilon_0,1-\epsilon_0]$. Finally, the test error $R_{test}(\theta)$ is defined in line 54, and for the specific case at hand in line 199.