Dropout Enhanced Bilevel Training

This paper points out that overfitting is a common challenge in bilevel training task. To address this issue, the author proposes the dropout mask for the bilevel optimization problem. Specifically, the author proves the convergence of dropout bilevel methods theoretically and empirically shows that proposed method mitigates the overfitting issue.

The paper uses dropout methods for bilevel training tasks. Bilevel optimization problems consist of two intertwined optimization problems and can be particularly sensitive to small changes, especially when data is limited. The study introduces a bilevel optimization model that considers the distribution of dropout masks and examines how varying dropout rates impact the hypergradient of this model. The authors adapt an existing bilevel method to incorporate dropout and provide theoretical convergence guarantees for this new approach. Empirical tests on data cleaning problems show that this method can mitigate overfitting.

This paper brings dropout to bilevel optimization to avoid overfitting issues. The author forms a statistical bilevel optimization problem that includes the distribution of the dropout masks and propose a dropout method to solve it. To analyze the convergence of the proposed method, they delicately quantify the bias induced by the dropout. The empirical performance of the proposed method is tested on the data hyper cleaning task and shows that dropout efficiently improve the test accuracy and is more stable.

This paper proposes a dropout method to address the issue of overfitting in bilevel training tasks. The authors provide theoretical convergence guarantees from an optimization perspective and demonstrate its effectiveness through experiments, using data cleaning as an illustrative example.

We thank the reviewers for their efforts in reviewing this paper. We address the common points mentioned by several reviewers in this global response.

1. Additional experiments

Suggestions from reviewers indicate that we should add more experiments, beyond data cleaning, to demonstrate the generalizability of the dropout bilevel method.

In response to this comment, we have applied our method to another type of bilevel problem: meta-learning, as discussed by Franceschi et al. [2018]. Specifically, we focus on few-shot learning, following the experimental protocols of Vinyals et al. [2016]. The bilevel formulation is intrduced in (2) of the revised version of the work and the experiment details are added in Section 5. Figure 3 in the revised version shows the accuracy of the method when adding dropout at different rates to the $i$th layer. Figure 3 also shows the accuracy against the iteration for each training process, and shows adding appropriate dropout to any layer improves the testing accuracy.

2. Relation between the dropout bilevel model and the original bilevel model

It is natural to consider the relationship between the stationarity of the dropout bilevel optimization problem and that of the original bilevel optimization problem. We have found that the stationary point of the model employing dropout is not the same as that of the original one.

Let's consider a simple 2-dimensional case. Let $f(x_1,x_2): = \frac12 x_1^2x_2 + 1000x_1$. Then $\nabla f(x) = (x_1x_2 + 1000,\frac12 x_1^2)$. Let $m= diag(m_1,m_2)$ with $m_1 \approx Bernoulli(p)$ and $m_2 \approx Bernoulli(p)$. Then $\\mathbb{E}_m f(mx) = p^2f(x_1,x_2) + (1 - p)pf(0,x_2) + (1 - p)p f(x_1,0) + (1-p)^2f(0,0)$ and

Let $x^k=(x_1^k,x_2^k)$ be with $x_1^k = -\frac{1000p^2 + 2000(1-p)p}{k}$ and $x_2^k = k$. Then ${lim}_k\nabla \mathbb{E}_m f(mx^k) = 0$. However, ${lim}_k\nabla f(x^k) = ( -1000p^2 - 2000(1-p)p+1000,0)$. Therefore, the convergence of $\nabla \mathbb{E}_m f(mx^k)$ does not necessarily imply the convergence of $\nabla f(x^k)$. And the error between $\nabla f(x^k)$ and $\nabla \mathbb{E}_m f(mx^k)$ can not be closed unless $p$ approaches $1$. Therefore, there are additional errors brought only by the dropout. This example implies that the dropout bilevel method may not optimize the original bilevel problem. This is expected because when the training loss is optimized, the model fits the training data too well and this will increase the chance of overfitting.

3. Why not early stopping

We would like to clarify early stopping cannot always resolve the issue of overfitting. As shown in Figure 2 (c) of the revised version of the paper (Figure 4 in the original version), the testing accuracy of the method with dropout (indicated by the red line) is significantly higher than the peak testing accuracy of the method without dropout (the green line). Additionally, our new experiments on another bilevel problem, as illustrated in Figure 3 of the revised version, demonstrate that the testing accuracy does not decrease when dropout is not applied. However, the application of proper dropout leads to an increase in testing accuracy. These findings suggest that early stopping may not always be a preferable alternative to dropout. Therefore, it is still worthwhile to explore the application of dropout.

In order to mitigate overfitting in bilevel optimization, this paper studies dropout, both theoretically and empirically. The main motivating application is data cleaning, framed as weighing of the training data points. The reviewers criticized this strong focus as not general enough, and during the rebuttal, the authors added an application in meta-learning. Overfitting materializes a lot less in this second application, but it can still be seen. Strengths of the paper include clarity, the detailed theoretical analysis, and studying a relatively new aspect of bilevel optimization. Weaknesses include too much focus on the data cleaning application (partially mitigated during the rebuttal by a second application, but this necessarily did not receive as much love as the first application) and not clear enough delineation from methods that already use dropout in bilevel optimization. Overall, this is a good contribution to the field of bilevel optimization and I recommend acceptance.

Accept (spotlight)