Efficient Machine Unlearning for Deep Generative Models by Mitigating Optimization Conflicts

1. The method is a bit straightforward and simple. There is not much surprise in the gradient-based parameter selection and two objective functions for forgetting and remembering. Since the theoretical contributions are also weak, as shown below, the paper would be stronger if there was more strong theoretical results or more experimental analysis including new insights.

2. The mathematical notation is unclear, and the correctness and significance of the theoretical results are unclear. In lines 138-141, it appears that $\theta_r$ and $\theta_f$ are part of the model parameter $\theta$. However, in equation (3), the proposed method discusses the model $G_{\theta_r}$ consisting only of $\theta_r$ or $theta_f$, which does not match the previous explanation. I If $_r$ and $_f$ do not represent part of the parameters but represent a state of the parameters, such as the optimal state for some objective, then their definition is necessary. I could not judge the correctness and significance of Proposition 4.1 due to the above unclearness and the little explanation of proofs in the appendix. The contribution of the derivation of Proposition 4.2 seems a bit trivial since it can be obtained immediately from the assumption of Lipschitz continuity.

3. The relationship between $\theta_r$ and $theta_f$ is also unclear in Figure 1. In this figure, $\theta_r$ and $theta_f$ appear subspaces in the possible parameter space. However, originally, $theta_r$ and $theta$ and defined as a vector, not as a set of vectors. If $\theta$ represents a set of vectors, what is $G_theta$? If this paper justifies the proposed method using mathematical formulas, I think that it requires a clear definition and a discussion.

1. The writing is confusing. Here is one example, in lines 138 - 140, are $\theta_f$ and $\theta_r$ different models or subsets of one single model? Based on later analysis they seem to be subsets of one model. Besides, the authors wrote ''Fast forgetting of $D_f$ can be achieved by keeping $θ_r$ consistent with its original values and leaving $θ_f$ unchanged''. Where does the unlearning happen if keeping both $θ_r$ and $θ_f$ either consistent with their original values or unchanged?
2. $\mathcal{L}_f$ and $\mathcal{L}_r$ are not defined until one page later than mentioned in Sec. 4.1, making the method section hard to follow.
3. In Eq. 2, $\theta$ is the trainable parameter, while in Eq. 7, $\theta$ becomes the original model parameters.
4. The experiment section is unpolished. For example, in Table 1's caption: ''The best results are bolded and the second best results are underlined.'', I failed to find any underlines in Table 1.
5. The author define ''UT'' as the ''unlearning time'' but never report it. Thus, I fail to tell whether the proposed method is more efficient or not.

Some statements should be revised to improve clarity. For example, in the Abstract, the statement "EBU only preserves the parameters related to data to remember" is confusing, because EBU also trains all model parameters. In Line 102, it says "We identify these shared parameters by analyzing corresponding weight saliency maps during the unlearning process"; but the gradient information is used instead.

The notations should be carefully revised to make it easier to understand. For example, in Line 142, both $\theta_f$ and $\theta$ in $\theta_f \cap \theta=\theta_f$ are weights, not sets.

The parameters selection procedure is the foundation of the proposed method. However, empirical experiments justifying its effectiveness are lacking.

The writing and presentation of the paper are not clear enough and confusing in many places. It looks to me different sections are written by different authors and they are not consistent with each other. For instance, $\sigma$ in section 4 is $\delta$ in section 5. $\zeta$ in section 4 is $\delta$ in C.1. $L$ in eq 5 is $M$ in C.1.

The mathematical problem of unlearning is ill-defined in this paper. This is because $\theta_f$ and $\theta_r$ are not formally defined in section 3.2. There is also no justification for why only a subset of weights are used for unlearning some data, and I do not agree with this assumption unless there is proof of existence and uniqueness of $\theta_f$ and $\theta_r$. Consequently, the losses and gradient computations are not valid to me.

The theory in section 4.1 is just simple implementation of PAC bounds and is not presented correctly. In eq 2 there should be separate averages for the two sums, and in eq 3 the $N_f+N_r$ should be the min of these two. Prop 4.1 is missing the with high probability statement. More importantly, the theory has nothing to do with the proposed loss in eq 6,7,9.

The mandatory target distribution in eq 6 is Gaussian, but there is no justification why it is the most effective one, especially when the model has to learn the same Gaussian for all $y\in Y_f$.

As for experiments in section 5.3, they are not extensive enough. While the results on the single nudity prompt look better, there is no systematic proof of unlearning of this concept because other natural language prompts or even adversarial prompts might trigger the concept to be generated. There is also lack of quantitative study for art styles and other tasks such as the I2P dataset in SLD.