Video2StyleGAN: Disentangling Local and Global Variations in a Video

We thank the reviewer for the constructive comments and suggestions on our paper. We address each of the concerns as follows:

1.The word "video manipulation" is a little ambiguous.

Our method does not propose a new generator for videos. We use the pre-trained SG3 generator trained on images. So, technically, it is designed to perform image manipulation. In our work, we combine different latent spaces of StyleGAN3 ($W+$, $S$, $F_f$, and filter weights) and hierarchically combine them in a ‘non-trivial’ manner. This enables us to effectively control the local and global features in a reenacted video, and hence, we can perform ‘’video manipulation’’. We will make it more clear in a subsequent revision.

2.Masked $W+$ Space

In the StyleGAN domain, w/w+ latent codes are input to a number of layers at different resolutions. Each layer controls the characteristics of the output image. In general, the earlier layers are responsible for the coarse features, and the later ones are responsible for finer details. Works like StyleFlow, PSP, etc, show that the identity-preserving edits can be made in the early layers of the StyleGAN. Hence, we ‘mask’ the latter layers so that there is no significant change in the style of the image and only the coarse semantic attributes, in our case, the pose is manipulated.

3.BiSeNet Mask upsampling and significance of co-driving video.

While the upsampling of the masks may bring in more errors, we actually do not need accurate masks to identify the layers responsible for a given attribute. The masks can be coarse, and we can identify the layers that have maximum overlap in terms of IOU scores. In order to show that the rotation and translation motion in the video can be inputted from a different video and hence is disentangled, we use co-driving video to extract this information and apply it to our final videos.

4.Texture sticking artifact, background changes, and the generator used.

Yes, SG3 sometimes shows jitters in the background compared to non-StyleGAN methods, i.e., FOMM and TPS; we would like to argue these methods are keypoint-based and are designed to explicitly avoid jittering. In contrast, our method does not require keypoints to compute the motion, so some jitters are expected with complex backgrounds. However, it is relatively simple to remove background artifacts by using the method in Local Facial Feature Encoding of the main paper on the background in addition to mouth, nose, and eyes. Instead of modeling the local motion, the corresponding $s$ activations of the background feature maps can be suppressed, hence avoiding motions in the background. In general, the artifacts observed in the video stem from the StyleGAN3 generator. It's crucial to note that this particular aspect of our video quality is subject to enhancement with improvements in StyleGAN-based generators. We use the R configuration of StyleGAN3.

5.SG2 vs SG3

A notable distinction between SG3 and SG2 lies in SG3's ability to reposition the head, a feature crucial for a video generation pipeline. This capability proves beneficial in creating realistic motions for talking heads, as demonstrated in our supplementary video. While we acknowledge the presence of certain artifacts inherited from the SG3 generator, we maintain that our method attains commendable video generation quality at a resolution of 1024x1024. This is particularly noteworthy as many alternative methods primarily train on videos and are designed to support a lower 256x256 resolution. We recognize the existing artifacts and attribute them to the current state of the SG3 generator. However, we are optimistic that with the ongoing advancements in GAN generators, especially continuous improvements in terms of disentanglement, our method will progressively overcome these artifacts and further enhance its overall performance.

6.What is the modified baseline?

The modified baseline method is only applying the edits to layers 3 - 7 of the w+ space. This is explained in section 3.3 of the paper. We will mention it more explicitly.

7.Deformations in the baselines.

In the example shown in the supplementary video, specifically, the head shape is changed. Notice in the baseline, it creates the unnatural bloating of the head region or increases the head size, as in the example in the first row. In comparison, our method maintains the features of the head while performing the pose change.

8.Related works, Figure 2 and moving some parts from the supplement.

Thanks for pointing out additional related works. We included the related works in the paper. We will improve the figure and move the relevant discussion from the supplement to the main paper.

We thank the reviewer for the constructive comments and suggestions on our paper. We address each of the concerns as follows:

1.SG2 vs SG3

A notable distinction between SG3 and SG2 lies in SG3's ability to reposition the head, a feature crucial for a video generation pipeline. This capability proves beneficial in creating realistic motions for talking heads, as demonstrated in our supplementary video. While we acknowledge the presence of certain artifacts inherited from the SG3 generator, we maintain that our method attains commendable video generation quality at a resolution of 1024x1024. This is particularly noteworthy as many alternative methods primarily train on videos and are designed to support a lower 256x256 resolution. We recognize the existing artifacts and attribute them to the current state of the SG3 generator. However, we are optimistic that with the ongoing advancements in GAN generators, especially continuous improvements in terms of disentanglement, our method will progressively overcome these artifacts and further enhance its overall performance.

2.Explanation of user study and ablation.

In the supplementary material, we conduct two distinct user studies. The first study assesses the reconstruction quality of identity, and our method is preferred approximately 63% of the time. This outcome underscores the paper's contribution to generating high-quality videos at a superior resolution. The second user study focuses on evaluating the motion in the video, revealing that our method demonstrates comparable performance to competing methods in this aspect. To ensure the robustness of our approach and to affirm that a naive baseline does not yield high-fidelity and temporally consistent videos, we perform an ablation experiment. Specifically, we evaluate the Local Feature Encoding in conjunction with the baseline. The rationale behind this ablation is to demonstrate the contribution of Local Feature Encoding to temporal smoothness and keypoint accuracy.

3.Normalizing the images for expression changes.

Yes, we normalize the images before computing the expression edits. We do this by standard landmark detection and cropping using the code in the StyleGAN repo.

4.Pick-a-Pic score for the videos.

Unfortunately, there are currently no widely adopted large-scale video evaluation methods comparable to the one mentioned for assessing temporal consistency. The reviewer proposed a promising avenue for future work that could greatly benefit the community. In response to this suggestion, we already computed a temporal metric, FVD (Fréchet Video Distance), for our videos, and the results are presented in Table 4 of the supplementary material. In this comparison with competing methods, it's important to note that due to the smooth nature of the GAN latent space we employ, there are no discernible jumps or flickering artifacts in the video. Consequently, the FVD metric aligns closely with methods that are trained explicitly on video data.

5.Pose change results.

We have incorporated a video on the supplementary webpage, showcasing various poses derived from distinct frames alongside the corresponding face reenactment videos. It is essential to observe that our method demonstrates an ability to disentangle pose information with localized edits.

We thank the reviewer for the constructive comments and suggestions on our paper. We address each of the concerns as follows:

1.The motivation of the work.

The paper consistently presents a clear and well-defined motivation supported by evidence both in the main text and the supplementary materials. Firstly, our objective is distinct: we address the challenge of high-quality video generation and face reenactment at a resolution of 1024x1024 using a GAN without relying on any video training. Notably, competing works typically achieve generation at a lower resolution (256x256) and are trained on video data. Second, our focus extends beyond resolution; we aim to enhance fidelity in the results at this higher resolution. The paper demonstrates substantial improvements in reconstruction quality and the preservation of details, accompanied by comparable performance in motion and expression transfer, as evidenced in Tables 1 and 2 in the main paper. While we acknowledge that further research could enhance motion and expression transfer, our overall approach outperforms prior art. Third, we emphasize the importance of a facial reenactment system that provides control over both local and global features in video generation. Our StyleGAN3-based framework, as illustrated in the paper and supplementary video, effectively disentangles various local and global properties, including expressions in a video, and independently controls them. This capability is not demonstrated in previous video editing works trained on videos, let alone in a network solely trained on images. Finally, given the absence of high-quality video datasets and other network limitations, our rationale for choosing StyleGAN3 is based on the need for solutions that use a smooth latent space to model videos. The generator is not inherently video-based; the innovation lies in our exploration and combination of regularized latent spaces to make it suitable for video editing. To achieve this, we meticulously examine different latent spaces of StyleGAN3, including $W+$, $S$, $F_f$, and filter weights, and we are the first to hierarchically combine them in a 'non-trivial' manner.

2.StyleHEAT and other methods naturally support videos, and why did we not compare with StyleHEAT?

We firmly disagree with the reviewer's assessment. StyleHeat utilizes the pretrained latent space of StyleGAN2 for video generation. In our paper, we have conducted a thorough comparison of our results with the competing methods. It is important to highlight that StyleGAN3 offers support for head repositioning, a capability that StyleGAN2 lacks. Our method is compared specifically with StyleHEAT, and we have already presented the comparison results for this method in Supplement Table 3, Figure 8, and with a video-based evaluation accessible on the attached webpage.

3.Results of Pose changes.

We have incorporated a video on the supplementary webpage, showcasing various poses derived from distinct frames alongside the corresponding face reenactment videos. It is essential to observe that our method demonstrates an ability to disentangle pose information with localized edits.

4.Other datasets?

We want to emphasize that our approach is not trained on extensive video datasets. Our comparison with the provided dataset remains fair, as all the methods considered are impartial to the training samples within the MEAD dataset. This dataset encompasses diverse actors expressing emotions with varying intensities, enabling the capture of micro-expressions and finer details. This diversity is crucial for evaluating high-resolution video generation, such as the one we propose. Consequently, the MEAD dataset serves as a robust test set for all methods, including ours.

5.Artifacts in the video (teeth).

The artifacts observed in the video stem from the StyleGAN3 generator. It's crucial to note that this particular aspect of our video quality is subject to enhancement with improvements in StyleGAN-based generators.

While the author claims to present a method for video editing, a significant portion of the experimental results leans towards reenactment aspects. If positioned as a reenactment method, it fails to comprehensively demonstrate pose control along the row, yaw, and pitch axes. Upon closer examination of the videos, it is evident that the accuracy of single-frame expression driving is relatively low. Regardless of the research direction, the experiments conducted with this method are incomplete, and the comparisons are inadequately presented. The author has not fully addressed my concerns. Regarding the dataset comparison issue, shuffling the image sequence in video datasets can easily address the stated reason for not training on video datasets. MEAD fails to showcase the diversity of pose and expression. Throughout the entire paper, the author neglects to demonstrate pose control along other axes, which is a perplexing omission.

We thank the reviewer for the constructive comments and suggestions on our paper. We address each of the concerns as follows:

1.Figure is unclear

Thanks for the suggestion. We will improve the figure in an eventual final version of the paper.

2.The method uses fine-tuning/optimization. Any approach to minimize it?

The most time-consuming phase in our Video2StyleGAN method is the GLOBAL POSE ENCODING step, which encompasses pose optimization utilizing the w space of StyleGAN3. Although this component of the method relies on optimization, a viable alternative could involve adopting a learned pose regressor, akin to approaches seen in the StyleGAN2 domain, such as StyleFlow. Exploring this alternative is on our agenda for future research.

3.Counterparts of our components to Diffusion models

One can compare the counterparts of our method with h-space and the skip connections in the Diffusion domain. For example, the” Training-free Style Transfer Emerges from h-space in Diffusion models” paper discovers that DMs inherently have disentangled representations for content and style of the resulting images: h-space contains the content, and the skip connections convey the style.

4.The strategy of using 3 - 7 layers and pose edit applied to only the first 8 layers.

In the StyleGAN domain, it is a common practice to utilize a subset of the w+ space for identity-preserving edits. Numerous works in the StyleGAN domain, such as StyleFlow and StyleCLIP, employ various strategies to identify the layers responsible for these edits. In our approach, we determined the layers that, after the edit, do not alter the identity and style of the person. This determination is achieved by leveraging ArcFace embeddings and comparing the original image before and after the edits introduced by these layers.