Temporally Consistent Atmospheric Turbulence Mitigation with Neural Representations

R4-Q1: Capability of handling longer video sequences and significant motion.

Our method effectively handles videos with significant motion. We've included additional experimental results and a detailed analysis in our global response to shared question A. This demonstrates our approach's robustness across various dynamic scenarios

R4-Q2: The related work section lacks completeness. More test-time methods should be carefully reviewed. Also, no SOTA methods were involved in the experimental section for comparison, which is unacceptable..

We'd like to highlight that our experimental section includes a comprehensive setup that SOTA and advanced methods in the related fields:

• TMT (TCI’23) [1]: The pioneering video-based ATM method utilizing transformer architecture and temporal attention.

• DATUM (CVPR’24) [2]: The current SOTA video-based ATM method, trained on an advanced turbulence simulator dataset, achieving 10x faster inference time and superior performance compared to TMT.

• TurbNet (ECCV’22) [3]: An effective image-based method trained on an advanced turbulence simulator.

• VRT (TIP’24) [4]: A notable video method for general deblurring.

Comprehensive qualitative and quantitative results are presented in Figure 1, Figure 5 and Table 1 of the main paper, with additional comparisons shown in attached file Figure 1. These results demonstrate our method's significant performance improvements in addressing the residual distortion (temporal inconsistency) that persists in current SOTA methods.

Table 1 and subsection 1.1 in the main paper present a comprehensive taxonomy of current ATM methods, analyzing their strengths and limitations relative to our approach.

We've added two more unsupervised and test-time optimization-based turbulence removal methods for comparison and discussion in our global response to shared question C.

[1] Zhang, Xingguang, et al. "Imaging through the atmosphere using turbulence mitigation transformer." IEEE Transactions on Computational Imaging (2024).

[2] Zhang, Xingguang, et al. "Spatio-Temporal Turbulence Mitigation: A Translational Perspective." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Mao, Zhiyuan, et al. "Single frame atmospheric turbulence mitigation: A benchmark study and a new physics-inspired transformer model." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[4] Liang, Jingyun, et al. "Vrt: A video restoration transformer." IEEE Transactions on Image Processing (2024).

R4-Q3 : In addition to VRT, more SOTA transformer-based methods, as well as the recent-popular diffusion-based models, should be added for analysis.

Regarding the comparison with diffusion-based models, we've prepared a comprehensive analysis highlighting our method's unique advantages. Please see our response to R2-Q5 for an in-depth discussion on this topic.

R4-Q4: Discussion about Efficiency.

We appreciate your suggestion on our model's efficiency. Our global response to R1-Q5 provides a detailed breakdown of our method's computational performance, including comparisons with other test-time optimization approaches for turbulence removal.

R4-Q5: Intermediate results.

These results are compiled in our global response to shared question D for a clearer understanding of our approach.

R3-weakness 1: Forward Model of Turbulence.

Indeed mitigating turbulence requires overcoming color aberrations, blurriness, and deformations and other effects. Fortunately, existing techniques have successfully addressed many of these challenges and arguably the largest limitation of existing approaches is that their results lack temporal consistency. Our method addresses this challenge.

Recent advancements in the field, from TMT (2023) to DATUM (2024), have leveraged increasingly large datasets created by advanced simulators and more complex networks. While these improvements have significantly enhanced sharpness in real-world cases, as evidenced in main paper Figures 1 and main paper Figure 5 of our main paper, poor temporal consistency remains a glaring issue. This inconsistency is not only noticeable to the human eye but also detrimental to downstream tasks such as video segmentation.

This persistent problem suggests that video-based ATM methods struggle to improve temporal consistency relative to other types of degradation. The primary challenge lies in poor generalization to the random and complex deformations caused by real-world turbulence.

Our method addresses this critical gap by providing an effective refinement solution for video-based ATM methods when applied to real-world videos. We believe that tackling the temporal inconsistency issue represents a significant contribution to the video ATM field at this stage, as it targets a key weakness in current state-of-the-art approaches

R3-weakness 2: Qualitative result of ours trained on the distorted video (No Base Method).

Even without a base method to provide partially restored frames, our approach effectively improves temporal consistency by a significant margin, as demonstrated in attached file Figure 1. For clear evidence of this improvement, please observe the X-t slice presented there.

R3-weakness 2: Table 2 is confusing. Why do video-based ATM methods have worse PSNR and PSNR_x-t compared to image-based methods on the HeatChamber dataset?

Apologies for any confusion caused by the content in the main table. Below is a detailed explanation of our results. We will clarify in the revision.

When reconstructing static scenes, image-based reconstruction methods have a significant advantage over video-based methods. This is because image-based methods effectively impose a prior assumption that the scene is completely static, while video-based methods do not. Consequently, it's not surprising that image-based methods perform well on the static HeatChamber dataset.

Without this strong static prior, it's understandable that video-based methods are less effective on static datasets. However, this situation reverses for dynamic scenes, where the static scene prior would be invalid. In these cases, image-based methods perform much worse than video-based methods.

R3-weakness 2: Why do some metric scores (tv_slice) appear much worse?

This is a typo; a lower slice_tv indicates better performance. We will fix it in the camera-ready version.

R3-weakness 3: Missings comparisons two unsupervised methods

At present, no open source implementation of [3] is available. We have provided comparisons with Mao’s [1] Li's [2] in our global response to the shared question C. Our approach handles videos with moving objects, whereas Li's and Mao's methods cannot.

[1] Mao, Zhiyuan, Nicholas Chimitt, and Stanley H. Chan. "Image reconstruction of static and dynamic scenes through anisoplanatic turbulence." IEEE Transactions on Computational Imaging 6 (2020): 1415-1428.

[2] Li, Nianyi, et al. "Unsupervised non-rigid image distortion removal via grid deformation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[3] Thapa, Simron, Nianyi Li, and Jinwei Ye. "Learning to remove refractive distortions from underwater images." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

R3-Q1 : Capability on large motion.

Yes, our method can handle moving objects with large motion. For the qualitative results, please see our global response to the shared question A .

Thank you for your thoughtful review and for recognizing the potential of our paper.

In the revised version, we will include both quantitative and qualitative comparisons with unsupervised and test-time optimization methods ( Li et al. & Mao et al). Additionally, we will expand our discussion section to incorporate your insightful questions and our exchange, helping readers better understand our contribution in the atmospheric turbulence mitigation field.

Again, we greatly appreciate your valuable time and insightful suggestions.

R2-Q1 & R2-Q4: Capability of moving object on more cases

Yes, our method can handle moving objects. More than half of the cases in our main paper and supplementary materials are dynamic videos. This is evident as the lines in X-t Slice or Y-t Slice are not perfectly vertical or horizontal, indicating the objects are moving along the temporal dimension. Please also view the html file in the supplement which includes video reconstructions of our scenes.

We have also added additional experimental results, with large object motion, in our global response to shared question A. Our approach can effectively reconstruct scenes with significant object motion.

R2-Q2: Comparison with Li’s method

We have added comparisons with Li's [1] in our global response to shared question C. Our approach handles videos with moving objects, whereas Li's method cannot.

[1] Li, Nianyi, et al. "Unsupervised non-rigid image distortion removal via grid deformation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

R2-Q3: More ablation study.

We conducted an ablation study on temporal resolution (T) and the presence of L_temp, as shown in Table below, with corresponding qualitative results in attached file Figure 4.

The first row of the Table shows TurbNet's performance. Equipping TurbNet with our method yields a 0.76 dB improvement in PSNR, demonstrating the effectiveness of our representation field design in regularizing irregular turbulence motion. Decreasing T from 15 to 5 (implying stronger regularization) further improves PSNR by 0.94 dB, SSIM by 0.03, and PSNR_slicext by 1.28 dB. Adding our proposed loss function achieves additional performance gains.

Attached file Figure 4 illustrates qualitative results. Removing the proposed loss function and relaxing regularization leads to noisier and more jittery X-t slices compared to our full method

R2-Q5: Advantage of the proposed method over diffusion model.

Our method in video ATM offers several advantages over diffusion models: It is generalizable, physics-grounded, temporally consistent, and free from hallucinations.

Diffusion models can learn incredibly accurate priors on images and, more recently, videos. Long term, they could be an effective method to mitigate turbulence. However, they also come with severe limitations. Diffusion models require extensive training data and often suffer from hallucination biases inherent in that data. These models are purely data-driven and do not consider domain knowledge or the physics of turbulence. In contrast, our method addresses residual temporal inconsistency issues from existing physics-based turbulence removal models. By observing turbulence motion features, we design the mitigation based on temporal motion regularity, achieving effective results without any training. Moreover, our method can be applied to any existing physics-based ATM method, making it more grounded in the physics of turbulence rather than relying solely on training data quality.

To date, there has been no exploration of diffusion models for video-based ATM. The only diffusion model method, PiRN, for image-based ATM is a closed-source approach trained on limited synthetic image datasets. There is no evidence it can reconstruct temporally consistent dynamic videos. In contrast, our method significantly improves temporal consistency.

R2-Q6: Visualization of the temporal and spatial field.

Please see our global response to the shared question D

R1-Q1 / R1 - weakness 1 : Visualization of Canonical Spatial Field and Representation Field Design

The canonical spatial field C serves as a base spatial representation, containing all spatial content of the video. We can obtain a canonical image by deriving it from this field without applying delta x and y. This canonical spatial field functions similarly to a key frame in video compression, providing a reference from which other frames in the sequence can be derived. For more detailed results and discussion on this topic, please refer to our global response to shared question D

Our feature maps M and N, combined with the Hadamard product, provide an efficient representation strategy for 3D feature volumes represented by INR. This tensor decomposition strategy is commonly used to parameterize 3D volumes in INR, enhancing their ability to represent 3D signals while reducing the number of required parameters [1][2][3]. The Hadamard product in our two-stage architecture efficiently combines spatial and temporal information, offering a balance between computational efficiency and representational power, particularly suited for video ATM tasks.

R1-Q3 / R1 - weakness 3: More ablation study on loss function

Both representation and loss design are effective in our method. The neural representation design plays a more crucial role, bringing a 0.95dB improvement in performance, while the L_temp loss contributes an additional 0.15dB improvement, as demonstrated in our ablation study. For more details, please refer to our response to R2-Q3.

R1-Q4 / R1-weakness 4 : Camera shake

Thank you for the interesting question. Please see our global response to shared question B.

R1-Q5: Wall clock time

We conducted experiments on a 25-frame video with 540x540 resolution. Our total running time consists of two parts: the preprocessing using the ATM model (DATUM) and our test-time optimization approach. For comparison, the running time for other unsupervised test-time optimization methods (Li's [2] and Mao’s [1] methods) only includes the methods themselves, as they do not rely on preprocessing. As shown in the Table below, even with the additional DATUM preprocessing time, our method is more than 15 times faster than Mao’s method and around 30 times faster than Li's. Our method's efficiency stems from combining the advantages of supervised models with test-time optimization. We use faster supervised methods to address major issues in video turbulence mitigation and apply the slower test-time optimization only to residual temporal inconsistencies. This approach makes our method significantly more efficient than pure test-time optimization methods that must solve the entire problem independently.

R1-Q2: Missing details about MLP

Our network architecture consists of two MLPs: a content MLP with 3 fully-connected layers (input: Q1, hidden: 128, output: 2) and a temporal MLP with 8 fully-connected layers (input: Q2, hidden: 128, output: 3 RGB). We encode position information in the optimizable Canonical Spatial Feature Map C and Spatial Feature Map M, both with dimensions H x W matching the image. For each RGB image location, we query the corresponding Spatial Feature Map M location, perform a Hadamard product, and input to the MLP. In the Temporal Feature Map, multiple frames share neighboring features with different weights due to explicit regularization. We will provide additional illustrations in the revision to further clarify these concepts.

R1-Q6: The missing enhanced module

The “enhanced module” refers to a network feature that ablation studies caused us to remove. Referencing it in the manuscript was a typo—apologies for the confusion. This reference will be removed.

R1 - weakness 2: Add comparison with Mao’s method

Our method substantially outperforms Mao et al. [4] and [5] on dynamic video content. This is because their approaches are only effective for static scene turbulence removal and fail when applied to videos containing moving objects. For qualitative comparison and discussion, please see our global response to shared question C

R1 - weakness 6: Term involved in the loss function

At line 187, the loss is calculated between the output of our method and the output of the base method (TurbNet, TMT, DATUM, etc.). We will clarify this description in the camera-ready version.

R1 - weakness 5/7/8/9: Typos

Thank you for pointing out the typos. We will fix them in the camera-ready version.

[1] Chen, Anpei, et al. "Tensorf: Tensorial radiance fields." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

[2] Fridovich-Keil, Sara, et al. "K-planes: Explicit radiance fields in space, time, and appearance." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Zhou, Haowen, et al. "Fourier ptychographic microscopy image stack reconstruction using implicit neural representations." Optica 10.12 (2023): 1679-1687.

[4] Mao, Zhiyuan, Nicholas Chimitt, and Stanley H. Chan. "Image reconstruction of static and dynamic scenes through anisoplanatic turbulence." IEEE Transactions on Computational Imaging 6 (2020): 1415-1428.

[5] Li, Nianyi, et al. "Unsupervised non-rigid image distortion removal via grid deformation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

I am satisfied with these revisions and will raise my rating to weak accept for this paper. The main justification behind my rating is the experimental results shown in the paper (KLT tracking, enhanced temporal consistency) including some moving scenes in the supplemental html page.