SeeClear: Semantic Distillation Enhances Pixel Condensation for Video Super-Resolution

Q1. Why do the comparison methods in Table 1 use different numbers of frames? If the same frame is used, what is the performance like?

The selection of numbers of frames for training depends on the architecture, such as sliding-window-based (e.g., EDVR-M) and recurrent-based (e.g., IconVSR) methods. The longer temporal information the model can access during training, the better the performance could be generally attained [1, 2]. Nevertheless, the increase in the number of frames will also lead to huge computational overhead. Therefore, we propose to parallelly align inter-frame within the short clip and maintain long-term temporal consistency via additional texture memory.

[1] Shuwei Shi, Jinjin Gu, Liangbin Xie, Xintao Wang, Yujiu Yang, and Chao Dong. Rethinking Alignment in Video Super-Resolution Transformers. In NeurIPS, 2022.

[2] Kelvin C.K. Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy. Investigating Tradeoffs in Real-World Video Super-Resolution. In CVPR, 2022.

Q2. In the ablation study (model 2, Table 2), how to use semantic conditions without MFSA, InCAM, and CaTeGory?

The semantic conditions are utilized for the generation of each frame via cross-attention, enabling the SeeClear to be cognizant of the content to be generated. Specifically, the query and key/value are respectively projected from features of SR and instance-centric semantic embedding of segmentation. It embeds the semantic priors into the pixel-level features, similar to the position embedding.

Q3. In InCAM, what is the value of k in top k, and how is it determined? Is there experimental support for this choice?

On the one hand, the resolution of the LR frames constrains the value range of k. On the other hand, there is saturation in the choice of k, as shown in Table 3 of the rebuttal document. As long as the instances and backgrounds in the frames can be retrieved completely, a larger k won't lead to performance improvement while increasing the computational cost of the network.

Q4. The method uses pre-trained models to extract semantic information, introducing significant additional computation, which limits the method's applicability. Meanwhile, the paper lacks comparisons of complexity and parameter counts.

Albeit an additional segmentation network is introduced, the auto-encoder is replaced by DWT simultaneously, making it possible to increase the network scale and improve the performance. Additionally, the option of the segmentation network is highly flexible, as long as it is transformer-based architecture. Especially with the continuous development of segmentation tasks in recent years, it is highly beneficial for performance gain of SeeClear. The semantic-assisted generative super-resolution comes forth in recent advanced works [1, 2]. Some of them not only require additional training for the semantic extraction network but also add ControlNet as a condition control mechanism, bringing more parameters. To further demonstrate the effectiveness and efficiency of the SeeClear, we also compare it with other representative diffusion-base SR methods in terms of the number of parameters and inference time, please see Q2 of the global response for more information.

[1] Haoze Sun, Wenbo Li, Jianzhuang Liu, Haoyu Chen, Renjing Pei, Xueyi Zou, Youliang Yan, and Yujiu Yang. CoSeR: Bridging Image and Language for Cognitive Super-Resolution. In CVPR, 2024.

[2] Rongyuan Wu, Tao Yang, Lingchen Sun, Zhengqiang Zhang, Shuai Li and Lei Zhang. SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution. In CVPR, 2024.

Q5. The paper proposes using wavelet transform to improve UNet but lacks experimental justification for why simple downsampling and upsampling wouldn't be more efficient.

Please see the answer to Q1 of the global response.

Q6. Figure 1, while aesthetically pleasing, is challenging to understand. It would be better to clearly explain the network structure (e.g., Figure 8) and the inference process.

The Figure 1 illustrates the two main networks (i.e., semantic distiller for semantic extraction and pixel condenser for denoising) as a conceptual sketch, rather than a main figure. To further demonstrate the network architecture of the network and the its connection with the devised components, we supplement detailed illustration and explains, as you seen in the Figure 8 and subsequent text in the Section B of supplemental materials.

A video clip consisting of five frames is parallelly sampled during the inference process. These LR frames are first fed into the semantic distiller to extract semantic tokens and then corrupted by random noise as the input of the pixel condenser. The pixel condenser iteratively generates the corresponding HR counterparts from noisy LR frames under the condition of LR frames and semantic priors. The specific information flow of the pixel condenser is elaborated in Section B of supplemental materials. Due to the space limitation, we would like to supplement the pseudo-code of the inference in the final edition.

Q1. Diffusion-based models usually show poor performance on PSNR (e.g., StableSR and Reshift), but SeeClear demonstrates a significant improvement. Could you analyze which parts of SeeClear contribute to this improvement? Please refer to the weaknesses part above.

Diffusion-based image super-resolution acquires remarkable perceptual quality thanks to the generative capabilities of the models, yet the consistency with low-resolution images is somewhat overlooked. However, the highly correlated content within adjacent frames of a video supplies more references and constraints for the generation of individual frames. Coupled with InCAM, which activates and associates semantic-related pixels among adjacent frames, it's possible to enhance the perceptual quality of the reconstructed video and strengthen the constraints on fidelity, as shown in Table 2 of the paper. Similar experimental results can also be observed in the comparative method SATeCo. Besides, the pixel condenser employs wavelet transform to change the resolution of features and continually reinject significant information into downsampled features from LR frames, guaranteeing the content fidelity. And the split high-frequency components of LR frames are transmitted to the decoder via skip connections, guiding the generation of details and textures. Additionally, some works [1, 2] also provide theoretical support for this experimental result. In conclusion, we hold the belief that the constraints of multiple frames of the video and the appropriate module designs jointly constrain the solution space of the model.

[1] Theo Adrai, Guy Ohayon, Michael Elad, and Tomer Michaeli. Deep Optimal Transport: A Practical Algorithm for Photo-realistic Image Restoration. In NeurIPS, 2023.

[2] Dror Freirich, Tomer Michaeli, and Ron Meir. A Theory of the Distortion-Perception Tradeoff in Wasserstein Space. In NeurIPS, 2021.

Q2. Lack of computation analysis.

Please see the answer to Q2 of the global response.

Q3. Lack of an ablation study on the wavelet transform.

Please see the answer to Q1 of the global response.

Q4. The Other baselines such as VRT and IconVSR are also evaluated on Vimeo-90K-T and UDM10 datasets. Could you complete it for a fair comparison?

Due to the lack of diffusion-based VSR evaluated onVimeo-90K-T and UDM10 datasets, we can only just provide the quantitative result generated via our proposed method.

Q5. Figure 7 needs more explanation.

Please refer to the answer to the Q3 of the global response.

We reformulate Eq. (16)-(18) as follows:

where $\bar{C}_j$ and $\bar{T}_j$ respectively denote zero-initialized channel-wise semantics and textures of j-th group. $\widehat{T}_j$ is the texture embedding that possesses one-to-one correspondence with semantic embedding via matrix multiplication.

Thanks for your careful reading and detailed comments. We will rectify some confusing statements and formulas in the subsequent edition. Nevertheless, we deem it necessary to highlight our novelty and restate the proposed method. Different from the text in the realm of T2I and segmentation mask in existing generative super-resolution, we pioneer the exploration of the utilization of temporal information based on the semantic similarity for diffusion-based VSR. The instance-centric semantic embeddings and channel-wise semantics are employed to determine the conditional pixels with high quality and semantic similarity from adjacent frames and long-term temporal information, enhancing the generative quality and temporal consistency.

Q1. What is the "conditional video generation"?

As depicted in Figure 1 of the rebuttal document, SeeClear consists of a forward diffusion process and a reverse process for VSR. In the diffusion process, patch-level blurring and residual shift mechanism are integrated to degrade HR frames based on the handcrafted time schedule. During the reverse process, a transformer-based network for open vocabulary segmentation and a U-Net are employed for iterative denoising. The former is responsible for extracting semantic embeddings related to instances from LR videos, similar to the process of distillation in physics, and is therefore named the semantic distiller. The latter is utilized to filter out interfering noise and retain valuable information from low-quality frames, similar to the condensation process. All of them are tailored for image processing, and SeeClear takes diverse semantic embeddings as conditions to enable the network to be aware of the generated content and determine the aligned pixels from adjacent frames for the temporal consistency of the whole video. More details of U-Net for denoising are elaborated in Section B of the supplementary material.

Q3 & Q4. Are the "semantic tokens" actually text embeddings? How to divide channels into different groups?

The conditions in SeeClear refer to instance-centric semantic embeddings, semantics within the channel dimension and discriminative textures, rather than text or segmentation mask. The semantic distiller comprises three components, i.e., backbone, encoder and decoder. The encoder takes multi-scale features from backbone as input and models the relationship among pixels via self-attention. Randomly initialized tokens serve as the input of the decoder together with the dense features yielded by the encoder, which integrate instance-related semantic information through cross-attention. Eventually, those semantic tokens are utilized to predict classes, bounding boxes, and masks by simple modules like MLP.

Besides, we dig abundant semantics within the channel dimension via CaTeGory. For example, different channels response to edges/corners in shallow layer, and deeper layers further extract complex patterns by combining the information of different channels in the previous layer. CaTeGory consists of semantic and texture embeddings, which are zero-initialized as parameters of network. Throughout the training process, semantic embeddings iteratively aggregate the channel semantic from instance-centric semantics via cross-attention, and texture embeddings formulate representative textures from the multi-scale features of the U-Net. To establish the one-to-one correspondence between channel-wise semantics and textures, CaTeGory correlates semantic and texture embeddings through element-wise multiplication.

Q2 & Q6. What is the role of "randomly initialized tokens"? And what is specific role of the encoder-decoder module? In InCAM, the way of introducing semantic features seems similar to SFT.

To effectively utilize these semantic conditions, InCAM is inserted before the spatial self-attention of U-Net, aiming at performing inter-frame alignment in the semantic space implicitly. Distinct from SFT, which merely generates coefficients for feature modulation based on class priors derived from the segmentation mask, InCAM utilizes two types of semantic priors, i.e., semantic embeddings and dense features. Among them, dense features and those of VSR are jointly employed for the generation of modulation coefficients, enabling the modulated features to lie in the domain between the segmentation and the super-resolution, narrowing the gap between semantic priors and pixel features. Moreover, the semantic embeddings are not only used as conditions for the generation of each frame, but also can locate pixels related to specific instances within the frame. Therefore, InCAM incorporates the instance encoder-decoder (L206-207) to yield clip-wise semantics based on the instance-centric semantics of adjacent frames, which associate the same instances and align the semantically related pixels between adjacent frames. The random initialized tokens input in the decoder will serve as clip-wise semantics after the iterative interaction along temporal axis. From this perspective, InCAM also possesses the ability of inter-frame alignment that SFT lacks.

Q5. It is hard to understand Eq. (16), (17) and (18). From (17) and (18). Please elaborate.

Please see the official comment.

Q7 & Q8. Please provide comparison with the following related methods. I think it is hard to reach the given conclusion from [8]. Please elaborate.

Due to the unavailability of Upscale-A-Video’s code and the relevant datasets (e.g., REDS30), a fair comparison cannot be conducted. But we are willing to carry out the comparison in our subsequent versions once the code and related datasets are released. While SFT is an image-level SR method and lacks quantitative results, thus we add other representative methods in the response to Q2 for Reviewer NXC1.

Please refer to the answer to Q3 in the global response for the elaboration of the given conclusion.

Q3: The answer is still unclear. I still do not unstand the meaning of "semantic tokens" (O) and what are the relation with the visual features (F). Q4: I do not see any direct answer to the question of how to divide channels. Q8: The answer to Q3 seems to have no relation with Q8.

I would like to maintain my current rating.

Q1. The performance of the proposed method is not significant. In Table 1, the improvement is very marginal or is worse than other methods. Moreover, in Figure 4, the generated texture is comparable to other methods.

For the sake of fair comparison, SeeClear is trained only on five frames and achieves PSNR/SSIM comparable to those models trained on longer sequences, along with the best LPIPS. SeeClear also acquires the superior PSNR on the Vid4 dataset, surpassing the runner-up [1] by 0.36 dB. It is worth highlighting that the novelty of SeeClear lies in its pioneering exploration of semantic tokens in the activation and association of semantic-related pixels across frames during the inter-frame alignment of diffusion-based VSR, rather than merely the improvement of performance metrics. The instance-centric semantic tokens directly extracted from frames not only stimulate the generation potential of the diffusion model but also avoid the cross-modal alignment between text and images. Besides, the inter-frame related conditional pixels determined by semantic similarity equips the diffusion-based VSR model with cognitive ability along the temporal dimension, reducing the interference of irrelevant pixels and motion estimation errors and enhancing the quality of the reconstructed videos. Moreover, the restoration of tiny objects or intricate structures and real-world VSR demands specific training datasets [2, 3, 4], which are not coincident with the configurations and benchmarks of prevailing VSR. These issues have been thoroughly discussed in Section D of the supplementary materials and are considered as our future undertaking.

[1] Zhikai Chen, Fuchen Long, Zhaofan Qiu, Ting Yao, Wengang Zhou, Jiebo Luo, and Tao Mei. Learning spatial adaptation and temporal coherence in diffusion models for video super-resolution. In CVPR, 2024.

[2] Fanghua Yu, Jinjin Gu, Zheyuan Li, Jinfan Hu, Xiangtao Kong, Xintao Wang, Jingwen He, Yu Qiao, and Chao Dong. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. In CVPR, 2024.

[3] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In ICLR, 2023.

[4] Jiezhang Cao, Yue Shi, Kai Zhang, Yulun Zhang, Radu Timofte, and Luc Van Gool. Deep equilibrium diffusion restoration with parallel sampling. In CVPR, 2024.

Q2. It would be better to compare more methods. (a) Transformer-based (e.g., VSR Transformer) or RNN-based method (e.g., BasicVSR++); (b) Diffusion-based image restoration methods (e.g., DDRM, DDNM, DeqIR, etc). (c) Compare methods trained with more frames (e.g., VRT-16).

We add more methods for comparison, including RNN-based (BasicVSR++), Transformer-based (RVRT, VRT) and Diffusion-based (DDNM) methods. All these methods except DDNM are trained with more frames. It can be observed that more complex architectures (e.g., second-order grid propagation) and longer sequence used for training can significantly enhance the quality of reconstructed video but inevitably result in substantial computational overload.

Q3. The authors should compare the efficiency, including model size, training/inference time and FLOPs. The efficiency comparisons can demonstrate the effectiveness of the proposed method.

Please see the answer to Q2 of the global response.