Manipulation Inversion by Adversarial Learning on Latent Statistical Manifold

We wish to sincerely thank the reviewer for the valuable comments and insightful suggestions.

[Q1 Weakness 1: Editing Performance]

Indeed, as pointed out by Lemma 4.2 of our manuscript, the performance of editing is in accordance with the accuracy of manipulation inversion. Our best performances on manipulation inversion thus verify the superiority for image editing of our method. We also newly conduct extensive evaluations on image editing, in which our method consistently achieves the best editing performances. We refer to our response to Reviewer#1qCg [Q3] for more details.

[Q2 Weakness 2: Experiment of Inverting $w$]

Yes. We agree with the reviewer that the distance between the ground-truth and inverted latent codes is also crucial to evaluate the effectiveness of our method. As illustrated in Figure 2, we thus conducted new evaluations regarding the latent space MSE, and report the results in Table 4. As can be seen from this table, our method, by effectively optimizing the manipulation inversion with our VAT strategy, consistently achieves the best accuracy. We further evaluate the MSE between editing images, by choosing the smile direction, and also report the results in Table 4, which again achieves the superior performances.

[Q3 Weakness 3: Clarifying Section 4]

Indeed, the random manipulation is sampled by our VAT strategy proposed in Section 4, in which one manipulation direction is optimized at each iteration. More specifically, for each latent code $\mathbf{w}$, we compute the perturbation loss $\psi(\mathbf{w}, \mathbf{v})$, and iteratively solve for the worst-case direction $\mathbf{v}^*$ via power iteration on the Hessian of $\psi$, i.e., (Golub & der Vorst, 2000) of our manuscript, ensuring maximal disruption to the inversion consistency. The worst-case direction $\mathbf{v}^*$ is then used to optimize our encoder to achieve the manipulation inversion. We further provide Algorithm 1 to depict the overall pipeline of our method.

Moreover, $\mathcal{L}\_{j}$ and $\mathcal{L}\_{s}$ are the losses that correspond to the semantic and Jacobian terms. More specifically, by combining Equation (6) and (8) in our manuscript, we are able to calculate the Riemannian gradient given the target loss $\psi(\mathbf{w},\mathbf{v})=\mathbf{d}^T\mathbf{d}$, where $\mathbf{d}=f(g(\mathbf{w}+\beta\frac{\mathbf{v}}{||\mathbf{v}||_2}))-\beta\frac{\mathbf{v}}{||\mathbf{v}||_2}-\mathbf{w}$. In this way, we are able to move a step further based on Equation (6) of our manuscript, as follows

$$\nabla_r\phi(\mathbf{w})=\nabla_e\phi(\mathbf{w})^T(\mathbf{S}^T\mathbf{S}+\mathbf{J}^T\mathbf{J})=2\mathbf{d}^T(\mathbf{S}^T\mathbf{S}+\mathbf{J}^T\mathbf{J}),$$

where $\phi(\mathbf{w})=\psi(\mathbf{w},\mathbf{v})$. Therefore, the achieved Riemannian gradient via our established manifold is equivariant to calculate the Euclidean gradient by the modified target loss $\psi(\mathbf{w},\mathbf{v})=\mathbf{d}^T(\mathbf{S}^T\mathbf{S})\mathbf{d}+\mathbf{d}^T(\mathbf{J}^T\mathbf{J})\mathbf{d}$, and we thus denote $\mathcal{L}\_{j}=\mathbf{d}^T(\mathbf{J}^T\mathbf{J})\mathbf{d}$ and $\mathcal{L}\_{s}=\mathbf{d}^T(\mathbf{S}^T\mathbf{S})\mathbf{d}$, such that the manipulation inversion loss can be effectively optimized via our final loss given by Equation (9) of our manuscript.

Furthermore, $\mathbf{J}$ is essentially the Jacobian of the pre-trained generator, which can be computed efficiently by the gradients during each iteration, namely, via backpropagation with respect to pixel outputs (following Ramesh et al., 2018). On the other hand, $\mathbf{S}$ consists of semantic directions, obtained by either supervised or unsupervised manners, which formulates the semantic linear space in the latent space; this is obtained by Shen et al., 2020. We wish to thank the reviewer for pointing out the ambiguity of our Section 4, and we will elaborate more on this, including the calculation on $\mathbf{S}$ and $\mathbf{J}$, in the revision.

Many thanks for the insightful suggestions.

[Q1 Weakness 1: Quality and Accuracy of the Editing Results]

Indeed, the realism and accuracy of image editing are tightly related to the accuracy of manipulation inversion, which has been proved in our manuscript. We further conducted comprehensive evaluations on the editing performances. We may refer to our response to Reviewer#1qCg [Q3] for more details. Notably, for several target attributes with distinct semantics, we calculate the Clip scores to assess the semantic accuracy of edited images, in which our method achieves superior performances. Our method also obtains the lowest FIDs and highest ClipIQA scores, verifying the superior quality of edited images. This is due to the manipulation inversion oriented optimization, in which advanced domains are searched and optimized in the latent space of StyleGAN.

[Q2 Weakness 2: Clarifying $\mathcal{L}_j$ and $\mathcal{L}_s$]

Many thanks for pointing out this. Indeed, by combining Equation (6) and (8) in our manuscript, we are able to calculate the Riemannian gradient given the target loss $\psi(\mathbf{w},\mathbf{v})=\mathbf{d}^T\mathbf{d}$, where $\mathbf{d}=f(g(\mathbf{w}+\beta\frac{\mathbf{v}}{||\mathbf{v}||_2}))-\beta\frac{\mathbf{v}}{||\mathbf{v}||_2}-\mathbf{w}$. In this way, we are able to move a step further based on Equation (6) of our manuscript, as follows

$$\nabla_r\phi(\mathbf{w})=\nabla_e\phi(\mathbf{w})^T(\mathbf{S}^T\mathbf{S}+\mathbf{J}^T\mathbf{J})=2\mathbf{d}^T(\mathbf{S}^T\mathbf{S}+\mathbf{J}^T\mathbf{J}),$$

where $\phi(\mathbf{w})=\psi(\mathbf{w},\mathbf{v})$. Therefore, the achieved Riemannian gradient via our established manifold is equivariant to calculate the Euclidean gradient by the modified target loss $\psi(\mathbf{w},\mathbf{v})=\mathbf{d}^T(\mathbf{S}^T\mathbf{S})\mathbf{d}+\mathbf{d}^T(\mathbf{J}^T\mathbf{J})\mathbf{d}$, and we thus denote $\mathcal{L}\_{j}=\mathbf{d}^T(\mathbf{J}^T\mathbf{J})\mathbf{d}$ and $\mathcal{L}\_{s}=\mathbf{d}^T(\mathbf{S}^T\mathbf{S})\mathbf{d}$, such that the manipulation inversion loss can be effectively optimized via our final loss given by Equation (9) of our manuscript.

Many thanks for the insightful comments.

[Q1 Theoretical Claims: Multiple Directions]

Questions for Authors 1: Handling Multiple Directions in Manifold Construction: Indeed, our manifold is established based on the semantic direction $\mathbf{S}$ and Jacobian matrix $\mathbf{J}$. The Jacobian matrix $\mathbf{J}$ is defined by excessive sampling in the latent space. However, we effectively employed the gradients of the generator to obtain $\mathbf{J}$, following (Ramesh et al., 2018) of our manuscript; this avoids the excessive sampling procedure. Our manipulation inversion requires sampling $\mathbf{v}$ when calculating the loss of Equation (1) in our manuscript, which is intractable. We thus propose the VAT strategy, by choosing the worst-case $\mathbf{v}^*$. The worst-case $\mathbf{v}^*$ can also be effectively solved by power iteration on the Hessian of $\psi$, i.e., (Golub & der Vorst, 2000) of our manuscript. We also provide our detailed pipeline in the newly added Algorithm 1.

Questions for Authors 2: Clarification on Equation (7) and Maximization of $\mathbf{v}$: Eq. (7) aims to maximize the perturbation loss $\psi(\mathbf{w},\mathbf{v})$, by choosing the worst-case $\mathbf{v}^*$. By constraining $\mathbf{v}$ to a unit vector ($\frac{\mathbf{v}}{||\mathbf{v}||_2}$), we enforce the maximization by focusing on directions, instead of infinitely increasing the scales in the latent space. The scalar $\beta$ governs the perturbation strength instead. This way, the resulting $\mathbf{v}^*$ identifies the direction that most disrupts inversion consistency. This way, our VAT-related loss in Equation (7) of our manuscript essentially formulates an upper bound of the primary manipulation inversion loss given by Equation (1) of our manuscript. Based on this, minimizing $\psi(\mathbf{w},\mathbf{v}^*)$ thus ensures the robust and effective convergence of our encoder $f$ to arbitrary perturbations.

[Q2 Experimental Designs Or Analyses: Editing Performance and Method Complexity]

Indeed, as pointed out by Lemma 4.2 of our manuscript, the performance of editing is in accordance with the accuracy of manipulation inversion. Our best performances on manipulation inversion thus verify the superiority for image editing of our method. We also newly conduct extensive evaluations on image editing, in which our method consistently achieves the best editing performances. We refer to our response to Reviewer#1qCg [Q3] for more details.

Regarding the complexity, instead of excessively sampling multiple directions, the proposed VAT strategy effectively reduces the computational complexity of our method. However, our method needs to calculate the Jacobian of the generator, thus requiring to forward twice during training. Consequently, our method was trained on a single NVIDIA GeForce RTX 4090 GPU, with a total training time of about $80$ hours. Regarding the FSE baseline, it consumed 50 hours as comparison. Both methods consumed comparable memory requirements. Inference with our method does not increase computational complexity or memory cost, compared to existing state-of-the-art GAN inversion methods.

[Q3 Questions for Authors 3: Comparing Methods]

Ref. [1] operates in Z space for better inversion accuracy and editing. However, we did not find the publicly available codes within the tight rebuttal period. Ref. [2] (HFGI) and Ref. [3] (FSE) are two state-of-the-art comparing methods and have been already compared in our manuscript. For Ref. [4], the proposed PTI is an optimization-based method, which fine-tunes the generator to allow for improved accuracy for each image, instead of the encoder-based methods for all images including Refs. [2,3] and our method. This requires excessive computation to search for the best per image during the inference. In the rebuttal, we also report the new comparisons of PTI in Table 1, Table 2, and Table 3, in which our method also achieves the best performances for both reconstruction and editing.

[Q4 Questions for Authors 4: Hubness Problem]

Indeed, we did not observe the hubness problem empirically. This may due to two possible aspects: (a) Pretrained Encoder Initialization: The fixed encoder maps inputs to semantically stable regions in latent space, avoiding hubness-prone initialization that commonly appears in random sampling. (b) Multi-Constraint Optimization: Jacobian regularization stabilizes gradients, while semantic constraints anchor optimization to plausible latent regions, jointly preventing convergence to hubness-dominated local optima. We believe further analysis on hubness problem could improve inversion, which is left for future work.

Many thanks for the valuable and insightful comments!

[Q1 Claims and Evidence: More Visual Examples on Analysis]

Yes. We have conducted further analysis on the generating latent space of StyleGAN, revealing the possible limitations for GAN inversion. More specifically, Figure 2-(a) of our manuscript essentially depicts our Finding 1, and we further add new visual examples in Figure 1-(a), which further highlights the multi-domain characteristics.

Regarding our Finding 2, Figure 2-(b) of our manuscript emphasizes the anisotropy property within the latent space. We also provide more visual examples in Figure 1-(b), by additionally calculating the identity (ID) values. From this figure, we can conclude that for the same scale, different manipulation directions exhibit distinct anisotropy across IDs and semantics.

Moreover, our Finding 3, together with Figure 3-(c) of our manuscript, reveals the inconsistent variation. We also clarify the details and provide more visual illustrations in Figure 1-(c), which demonstrate the non-monotonic trends of MSE in the image space, when consistently increasing the editing scale. Therefore, the above analysis reveals the characteristics of the latent space, in which existing GAN inversion methods may be limited during the optimization. For example, the majority of existing GAN inversion methods optimize the image MSE, which may not ensure the semantic consistency and accuracy within the latent space as pointed out by our Finding 3.

[Q2 Methods And Evaluation Criteria: Lack Implementation Details]

Indeed, the random manipulation is sampled by our VAT strategy, in which one manipulation direction is optimized at each iteration. More specifically, for each latent code $\mathbf{w}$, we compute the perturbation loss $\psi(\mathbf{w}, \mathbf{v})$, and iteratively solve for the worst-case direction $\mathbf{v}^*$ via power iteration on the Hessian of $\psi$, i.e., (Golub & der Vorst, 2000) of our manuscript, ensuring maximal disruption to the inversion consistency. The worst-case direction $\mathbf{v}^*$ is then used to optimize our encoder to achieve the manipulation inversion. We further provide Algorithm 1 to depict the overall pipeline of our method.

[Q3 Experimental Designs Or Analyses: Editing Performance]

Indeed, for GAN inversion, the evaluations on editing performances are ad hoc due to the lack of ground-truth, as also pointed out by the reviewer. This also motivates us to establish a new proxy to evaluate the editing realism, i.e., the accuracy of manipulation inversion, with proved equivalence via Lemma 4.2 of our manuscript. Therefore, Fig. 4 of our manuscript demonstrates that for various editing directions, our method is able to precisely invert back, thus equally proving the superior editing performances of our method.

We further conducted new evaluations based on the suggestions from the reviewer, i.e., using widely applied FID, ID, and Clip scores to evaluate the editing performance in face scenarios, together with the ClipDiff and ClipIQA scores for church scenarios. Please note that for face scenarios, we are aware of the semantics of editing attribute, thus capable of calculating Clip scores as suggested by the reviewer. For church scenarios, however, we cannot access the ground-truth semantics to edit attribute, thus infeasible to calculate IDs and Clip scores. Instead, we employ the editing directions by GANSpace and calculate the difference of Clip features between unedited and edited images (named as ClipDiff), to verify the effectiveness of editing. This is calculated by

$$\mathrm{ClipDiff} = 1 - \mathrm{cos}\left(\phi(\mathbf{x}\_\mathrm{original}),\phi(\mathbf{x}_{\mathrm{edited}})\right)$$

where $\phi(\cdot)$ denotes Clip image encoder. Moreover, since church scenarios only contain 300 test images, the FIDs can significantly vary, and we calculate ClipIQA scores instead to evaluate the subjective quality of edited images.

We report the results in Table 1, in which our method consistently achieves the best editing performances, across different scenarios and attributes. Our superior editing performance is also in accordance with the best performance of manipulation inversion, in which their equivalence has been pointed out in our manuscript.

[Q4 Relation to Broader Scientific Literature]

Yes, using human feedback or reinforcement learning instead of our VAT mechanism is expected to be useful when extending our manipulation inversion method to large-scale generative models, in which the latent space becomes further complicated. We leave this as our interesting future work.