Robust Video Perception by Seeing Motion

1. The main concern is the assumption of the paper may be too strong. Authors assume all the attacks will destroy the optical flow and that the flows can be recovered given the existing information. In Figure 5, the authors demonstrate how the motion consistency is different in clean, attacked, and defended videos. However, this may not be a general situation, for example, the rectangular occlusion attack (ROA) mentioned in [1]. There are many other types of common video attacks that have not been discussed in the paper.
2. I understand that the proposed method is a test-time optimization method, but it can not be an excuse to not compare with other relative existing methods in the paper.
3. Necessary ablations are missing. There are hyperparameters like step size, K, and bounds but the authors did not mention how sensitive is the framework towards those hyperparameters.

[1] Wu, T., Tong, L., & Vorobeychik, Y. (2019). Defending against physically realizable attacks on image classification. arXiv preprint arXiv:1909.09552.

• The experiments lack comparison with other existing solutions in this line. This makes the estimation of method significance hard.
• The proposed methods rely on the multiple terms of constraints, such as smoothness, brightness etc. However, all these constraints are studied for long in previous works. I thus have no confidence about the novelty of the proposed method in this paper.
• In the related works, the authors discuss the motion estimation methods, “Robustify Machine Learning Models.” (I believe “robustify” is a typo here) and “Adversarial Attacks and Defenses on Videos”. However, to someone not familiar with the domain, it is expected to discuss whether the used techniques in the method such as the involved consistency terms and the idea of “repairing then perception” has been adopted in related works. I would expect the authors to make more clear claim and elaboration about the novelty of the proposed method.

In my view, the main weakness of the paper is the empirical evaluation of the adversarial robustness of the model. However I do not have a background in adversarial robustness and would therefore be interested in the other reviewers' opinions about these points.

• The evaluation in section 4.3 seems to be irrelevant to me. The adversarial pertubation is optimized using the original, undefended model but tested on the adapted, defended model. From a high performance on these stimuli we do not learn anything about the robustness of the adapted model.
• In section 4.4, attacks against the adapted model are considered.
  • The first two attacks use knowledge of the test time loss in the optimization of the adversarial pertubation. The additional constraints however seem to be limiting. For example, an adversarial pertubation that hardly changes the optical flow estimate as optimized in "Adaptive Attack 2" will clearly be problematic for the defense. However, adversarial pertubations with this property are only a subset of all adversarial pertubations. There might be strong adversarial pertubations that cannot be found with this approach.
  • Using gradient approximation as in Adaptive Attack 3 is a much stronger approach. Consequently, much stronger pertubations are found in which only a small fraction of the performance is restored by the defense (Table 5 last row). As mentioned by the authors, the better performance of the multiple constraints defense seems to mainly stem from the computational cost of attacks.
• Only gradient-based attacks are considered in this paper. Since gradient-based attacks are computationally difficult for a defense relying on test-time optimization, it would be interesting to also consider a gradient-free attack (cf. Carlini et al. 2019).

In summary, I am not convinced that the adapted model is adversarially substantially more robust than the base model. The good results reported for many attacks rather seem to be due to the computational cd of optimizing adversarial pertubations due to the use of test-time optimization.

• This paper investigates the issue of robustness in video action recognition, but it lacks comparison with test-time adaptation (TTA) methods, such as [A-B]. These TTA methods also aim to adapt to out-of-distribution data when the input data is disturbed by noise. Although these TTA methods mainly focus on updating model parameters, and this paper primarily focuses on adjusting the input data, how to prove that data processing is superior to model parameter adjustment? I believe a comparison should be made based on experimental results.
• Under noisy conditions, many TTA methods can achieve desirable results, while the improvement brought by this paper's method is relatively low.
• In appendix A.2.1, under noisy conditions, the average performance improvement brought by this paper's method is very low and can even be counterproductive under certain noise conditions. Does this indicate an issue with the approach of changing input data?
• How to verify the reliability of the long-range photometric consistency in section 3.3? Are there any ablation study results reflecting the performance gain brought by each part?
• The explanation of the formula content in Algorithm 1 in the main body is not clear enough.

[A] Temporal Coherent Test-Time Optimization for Robust Video Classification. ICLR23 [B] Video Test-Time Adaptation for Action Recognition. CVPR23

As I'm not familiar with this research topic, I may not give a fair review:

1. What does the standard mean in different tables?
2. How about the time consuming? Is it better than those methods that need training?
3. Is this method suitable for those with videos with quick motion?
4. How does this method extend to general action recognition, since most of the current video backbones only use RGB frames?