A New Neural Kernel Regime: The Inductive Bias of Multi-Task Learning

We thank the reviewer for their careful review of our work and the concerns brought up. Below, we address each of the weaknesses brought up in a point-by-point manner.

1. [Multi-variate case] The results for the multivariate case (which also apply to the univariate setting) are more of an approximate argument. This is partially due to the fact that far less is known about the solution sets in the multivariate case (see e.g. [3],[4], [8]) compared the the complete characterization which is known for univariate solutions ([2]). The fact that multivariate solutions can also display variety in orientation, as evidenced in the example of Figure 5, hint at the potential richness of the solution sets. Under the iid task model used in our analysis, we believe that the approximations assumed in the analysis can be made more rigorous, which is of interest for future work. We disagree that the results are underwhelming. The fact that multi-task neural network training behaves like an $\ell^2$ minimization over linear combinations of the learned neurons, whereas single-task training is equivalent to $\ell^1$ minimization over the neurons, implies that the nature and properties of functions the functions learned by multi-task vs. single-task training can be profoundly different. This is, in our opinion, novel and interesting.

2. [Interpolation] We note that the more approximate argument presented in the multivariate case (which also applies to the univariate case) characterizes non-interpolation training problems which minimize a data-fitting term plus a penalized loss (for any penalty strength). Neural network interpolation problems are also a setting of practical interest, since many overparameterized networks trained in practice are able to achieve zero or near-zero training error. Moreover, standard multi-task learning scenarios generally assume that all tasks share the same set of input data points, and these tasks are learned jointly, which is exactly the setup we consider.``

3. [Relevance of kernel regime] The fact that multi-task learning behaves like a kernel method is important/interesting because it shows that the solutions to each task can be profoundly different compared to those obtained by training a separate network for each, even if the tasks are statistically independent as assumed in our analysis. In other words, multi-task training can have a major effect on the learned functions for each task even in situations where one might not anticipate such an effect. This insight may be valuable in practice, since this effect of multi-task training may be desirable or undesirable depending on the application. Regarding generalization properties, see the general comment made to all reviewers above.

4. [Non-convex objective] As discussed in Appendix 8, all experiments except that in the bottom right of Fig. 5 are done by training multi-output neural networks using the Adam optimizer with MSE loss and weight decay parameter 1e-5 (univariate experiments) or 1e-3 (multivariate experiments). Because the data-fitting term is non-convex and lambda is very small, the resulting problem is always non-convex. For the experiments in Fig. 4 showing that multi-task training with different random initializations leads to the same solution, it is not necessary to check the objective values since we know a priori (by our result in Theorem 3.1) that the learned functions depicted in the figures are global minimizers. However, if it is helpful to the presentation, we can include these numbers. For the multi-variate experiments in Fig. 5 the global optimality of these solutions were verified using the convex neural networks approach in Tolga & Pilanci [8].

Regarding the minor weaknesses:

1. We agree that the use of the word "unlikely" here is imprecise since no distributional assumptions are given on the data/labels. We note that Corollary 1 (Appendix 7.2) of Theorem 3.1 implies that, if the task labels are sampled i.i.d. from an absolutely continuous distribution (e.g. Gaussian) and the number of tasks T is greater than 1, then those labels admit only the connect-the-dots solution with probability one. (This follows from Corollary 1 in Appendix 7.2.) So for example, if the labels are given by ground truth values plus some additive Gaussian noise—even a small amount of noise—the unique multi-task solution is connect-the-dots. We will mention this explicitly and update the phrasing to be more precise.
2. Agreed.
3. We are unsure of what the reviewer is referring to for citations [1,2,3,4] but we can include some more discussion/mention of existing multi-task learning research which focuses on the benefits of unrelated tasks, and how our work relates to these upon the reviewers clarification.
4. Agreed.

[2] Hanin, Boris. "On The Implicit Bias of Weight Decay in Shallow Univariate ReLU Networks."

[3] Ongie, Greg, et al. "A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case" ICLR 2020

[4] Zeno, Chen, et al. "How do minimum-norm shallow denoisers look in function space?." NeurIPS 2024.

[8] Pilanci, Mert, and Tolga Ergen. "Neural networks are convex regularizers: Exact polynomial-time convex optimization formulations for two-layer networks." ICML 2020.

We thank the reviewer for their careful review of our work and for their feedback. Below, we address each of the comments individually.

1. [Clarification on Theorem 3.1] We thank the review for this clarifying question about Theorem 3.1. Lemma 3.2 says that, given any solution which interpolates the data, we can remove all knots located away from the data points ("extraneous knots") without increasing the representational cost of the network. Removing these "extraneous knots" may introduce new knots at the data points which were not originally present, but Lemma 3.2 says that the total cost $R(0) = \|b-a\|_2 + \|c-b\|_2$ of the new solution with knots at the data points (see Fig. 3) is no greater than the cost $R(\delta) = \| \delta + b-a \|_2 + \frac{1}{1-\tau} \| \delta \|_2 + \| c-b + \tau \delta/(1-\tau) \|_2$ of the original solution, which may or may not have knots at the data points. Therefore, the question does not present a gap in the proof. Please let us know if this clarifies the concern.
2. [Generalization] See the general response to all reviewers above. When we referenced generalization, we specifically meant to reference the "tempered overfitting" result in [3], which is specific to the univariate case.
3. [Experiments] See the general response for details on new experimental results.

Regarding questions:

1. We address this question in the rebuttal to weaknesses above.
2. In the univariate case, the intuition is that multi-output weight decay regularization encourages neuron sharing across outputs ([6]) while also encouraging the weights of each output individually to be "small," i.e., encouraging each output individually to be a min-norm univariate interpolant (which are characterized in [2]). In general, the only way to accomplish this is for each output to perform connect-the-dots interpolation, with the shared knots located at the shared data points. The multivariate case is more subtle. The example in Figure 5 was constructed specifically to illustrate the existence of multiple solutions with signficantly different "directions" of variation. We agree that it is difficult to argue that any particular one of these solutions is better, but the more symmetric solution obtained by multi-task training does stay closer on average to the data points (we verified this computationally). We used pure noise tasks here for illustration and convenience, but similar results are obtained if the multiple tasks are randomly perturbed versions of the two-squares task. Intuitively and more generally, since the multi-task solutions are approximately equal to the kernel solution, these functions will be weighted combinations of all neurons in the model and rarely strongly aligned with a subset of the neurons. Thus, the multi-task solutions tend to be smoother and have more symmetry.
3. In the new experiments conducted we do not rely on any special structure for the data samples. In this case we still observe a striking difference between solutions learned by multi-task task networks vs. single task networks trained individually.
4. This is an interesting theoretical question for future work. Our initial experiments (not shown) indicate a similar kernel-like effect when regularizing biases in multi-task settings.

[2] Hanin, Boris. "On The Implicit Bias of Weight Decay in Shallow Univariate ReLU Networks."

[6] Shenouda, Joseph, et al. "Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compression." JMLR 2024.

We thank the reviewer for their feedback, but it seems the reviewer has a misunderstanding about the results of our paper. Below, we address each of the comments individually.

1. [1-task vs. 2-task solution] Our result does indeed demonstrate the difference in solution between learning 1 task individually versus 2 tasks jointly. The description in Theorem 3.2 of conditions under which multi-task solutions are non-unique holds for any T > 1.
2. [Relation to NTK] Our results are based on established exact characterizations of the global minimizers of training neural networks with weight decay ([2], [1], [3], [4], [5], [6]). The global minimizers are non-unique in general, and in the univariate case, it is well known that the connect-the-dots solution is just one of (potentially infinitely) many solutions. The NTK regime is a different, approximate analytical framework that relies on a number of assumptions which we do not employ in this work. Namely, our analysis does not require that the learning rate be extremely small ("lazy training") or that the network have infinite width, both of which are necessary for the NTK regime and it is now well established that such assumptions are insufficient for explaining the success of neural networks ([8],[9],[10]]). Additionally, the paper [7] cited by the reviewer does not address connections between neural kernel regime(s) and multi-task training/"task augmentation," which is one of the main novel contributions of our paper. Finally, the conclusions about the results in [7] seem to be incorrect, as [7] states that the solutions are almost linear interpolation under additional assumptions on the data. In contrast our result states that the solutions are exactly connect-the-dots linear interpolation with very weak assumptions on the data which are almost always satisfied.
3. [Generalization] See the general response to all reviewers above. When we referenced generalization, we specifically meant to reference the "tempered overfitting" result in [3], which is specific to the univariate case.
4. [Multi-variate case] The kernel will depend on the specific neurons learned in training, as indicated in the paper. These will depend on the training data collectively across all tasks and the random initialization of the training process. However, the function learned for each task will be a standard kernel solution in terms of this kernel. In contrast, training separate networks for each task will, in general, produce different solutions that cannot be viewed as kernel solutions. Specifically, the solutions to single-task training with weight decay effectively minimize the $\ell_1$ norm of the output weights, as discussed in the paper and many of the referenced prior works.

Regarding Question #2, this is an interesting setting, but our main focus in this paper is the standard multi-task learning scenario, in which all tasks share the same data points x.

[1] Joshi, Nirmit, Gal Vardi, and Nathan Srebro. "Noisy interpolation learning with shallow univariate ReLU networks." ICLR 2024.

[2] Hanin, Boris. "On The Implicit Bias of Weight Decay in Shallow Univariate ReLU Networks."

[3] Ongie, Greg, et al. "A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case." ICLR 2020.

[4] Parhi, Rahul, and Robert D. Nowak. "What kinds of functions do deep neural networks learn? Insights from variational spline theory." SIAM Journal on Mathematics of Data Science 2022.

[5] Savarese, Pedro, et al. "How do infinite width bounded norm networks look in function space?." COLT 2019.

[6] Shenouda, Joseph, et al. "Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compression." JMLR 2024.

[7] Lai, Jianfa, et al. "Generalization ability of wide neural networks on $\mathbb{R}$."

[8] Jacot, Arthur, et al. "Feature Learning in $L_2$-regularized DNNs: Attraction/Repulsion and Sparsity." NeurIPS 2022

[9] Arora, Sanjeev, et al. "On exact computation with an infinitely wide neural net." NeurIPS 2019.

[10] Damian, Alex, et al. "Neural networks can learn representations with gradient descent." COLT 2022.

We thank the reviewer for their review and positive score. Our analysis does focus on two-layer networks, but the conclusions of the multivariate analysis can be used to reason about the behavior of functions learned at layers within a deeper network. A ReLU layer in a deep network can be viewed as a shallow network (with inputs coming from the previous layer and outputs going into the next layer). If the outputs of this layer are sufficiently diverse, then the behavior can be similar to the kernel-like behavior indicated by our analysis. We have now also included new experiments in higher dimensions beyond the Fig. 5 example (see the general response above).

Also, we agree that when training neural networks in multi-task settings, the solutions will not be exactly the same across different initializations, but this is not at odds with our results. In our setting, the input weights will depend both on the training data (collectively across all tasks) and the random initializations, but whatever input weights (and hence neurons) are learned, the output weights for each task are effectively minimizing a common weighted ridge regularization objective. We will update the language of the paper to make this point more clear.