FlowBench: A Robustness Benchmark for Optical Flow Estimation

Dear Reviewer GPBg,

Thank you very much for your review, we greatly appreciate your input and suggestions.

Indeed, FLOWBENCH does not provide a new method for optical flow estimation but is rather intended to facilitate research towards reliable and robust optical flow estimation methods and to provide insights and analysis to existing methods. While being “an engineering contribution” - and an analysis of key findings -, it is timely and important. This has also been acknowledged by the other reviewers.

This work, as confirmed by Reviewer MVeY, is “An extensive study on 89 released models/checkpoints over four widely used datasets shows in-depth findings and insights” (now 91 models). As further highlighted by Reviewer MVeT, “This paper is among the first attempts to evaluate optical flow methods on reliability and generalization ability, which are both important but often overlooked in the research community” and “The results on all latest models are insightful and worth sharing.” Reviewer hv2r further extends the strengths of this work by mentioning that “This paper provides an interesting analysis that methods with the best accuracy are not necessarily the most robust methods. Given comprehensive analyses, readers can understand the pros and cons of each method. I believe the benchmark will certainly benefit the community and help design a better and more robust method for future work” and Reviewer F85d emphasizes the importance of our work by mentioning, “This work … by focusing on robustness—vital for real-world applications like autonomous driving and medical imaging—it directly addresses the gap between research advancements and deployable technology, which is quite important”.

Metrics Beyond EPE:

Yes, we agree that apart from EPE, other metrics might also be interesting to different users. Thus, apart from EPE, FLOWBENCH also enables calculating a lot of other interesting metrics, such as outlier, 1-px error, 3-px error, 5-px error, and cosine distance between two vectors. These vectors are the same as that in the case of EPE calculations. We limited the analysis in this work to use EPE, since it is the most commonly used metric for evaluation, moreover, most works on optical flow estimation (such as CosPGD, PCFA, RAFT, MS-RAFT+, and many other works) show a very high correlation between performance evaluations using different metrics. Following your suggestion, we now better highlight this in Appendix C.

Writing:

Thank you very much for bringing this to our attention. we agree that the original submission might not have been easy to follow for researchers not familiar with adversarial and OOD robustness. Thus we now revise the text to better explain these (now available, revisions in blue color). For example, we now explain in Sec. 4.2 that an adversarial attack is a perturbation made on the input images to fool a method into changing its predictions while the input image looks semantically similar to a human observer. Most works that focus on the reliability of optical flow estimation methods perform adversarial attacks, however, these works either focus on targeted attacks or on non-targeted attacks, not both at the same time. The objective of targeted attacks is to optimally perturb the input image such that the method predictions are changed towards a specifically desired target, for example, a target can be a $\overrightarrow{0}$ flow i.e. attacking so that the flow prediction at all pixels should become zero. On the other hand, non-targeted adversarial attacks do not intend to shift the method's predictions to a specific target, they simply intend to fool the method into making any incorrect predictions.

Thank you for your detailed explanation! The rebuttal provides more information to address my concerns. However, I agree with Reviewer F85d's concern on architecture analysis. For example, Flowformer combines transformers with correlation. But Table 1 classified it as Attention. This makes me suspect the reliability of the analysis in Sec 5.3/5.4. Also, different methods may use different training data, which can be a factor that severely affects the robustness. (e.g. Flowformer++ uses more data than Flowformer. RAFT is trained from scratch, but Flowformer loads pre-trained weights). I think that is why I feel the analyses are messy. A better classification of methods should be provided.

Dear Reviewer GPBg,

Thank you very much for your reply. Thank you for bringing to our attention that we misclassified Flowformer and Flowformer++ in our initial assessment. We have now correctly classified it as “Attention + Cost Volume”. Nonetheless, the findings from our observations still hold. The biggest takeaway from our work is that irrespective of the classification method i.e. irrespective of the point-matching method used and irrespective of the number of learnable parameters, all the DL-based optical flow estimation methods are not reliable under adversarial attacks and do not generalize to image corruptions. Yes, there are minor differences in robustness performance based on such factors, however, compared to the errors on i.i.d. Data, the errors under attack and errors under image corruptions are significantly higher, making these differences hardly significant and thus demonstrating an alarming threat to these methods if deployed in the real world. To the best of our understanding, this point comes across well from the analysis in Section 5.3 and Section 5.4.

Yes, we agree that different methods use different training strategies, as also pointed out by Reviewer hv2r. We agree that the training setup can play a significant role in the robustness of a method. However, in the case of optical flow methods, they are hardly ever adapted to the target setting since the target application usually does not have ground truth leading one to rely on the available pre-trained models. Currently, the training strategies and architectural design choices of different methods are significantly intertwined. For example, MS-RAFT+ is trained significantly differently from FlowFormer++, FlowFormer++ can be evaluated on a dataset only if it is finetuned for that dataset (which we did), however, MS-RAFT+ is proposed with a single training strategy to work across datasets. To ensure fair comparison, we contacted the authors of MS-RAFT+ and they accepted that our evaluations would be fair according to them. In our current analysis, we attempt to make significant observations based on multiple models, keeping their training and evaluation settings intact. We agree that it would be difficult to disentangle if the performance difference is due to the architectural design choice or due to the training strategy, however, previous works also do not disentangle the two when comparing i.i.d. Performance. Given that, optical flow methods were not proposed to specifically overcome adversarial attacks or image corruptions, we hypothesize that training all optical flow with a different known setup would not help improve their reliability and generalization abilities and the current observations from this work would still hold. Thus, we believe that the robustness evaluations performed here are realistic. To improve the reliability and generalization abilities of optical flow methods, better approaches need to be proposed, that focus on these aspects, and studying these would be very interesting. Thank you for the suggestion, we now include a synopsis of this discussion in future work.

Please do let us know if we successfully answered your questions and concerns, or if you have any further questions or concerns, we would be happy to address them.

Best Regards

Authors of Paper #1055

Dear Reviewer GPBg,

Thank you, firstly for engaging in the discussion so far and secondly, for raising the score. We would additionally like to thank you for your suggestion, we have now incorporated this in our revised manuscript (now available).

We believe your suggestion further improves the quality of the manuscript, helping us better understand the differences in different families (representative methods and their variants) of optical flow methods. In revised Appendix B we provide a detailed justification for why a particular method has been considered within the chosen family.

In Section 5.3 we analyze the methods solely based on the family that is respectively analyzed, and the architectural design choices and training strategies that make different methods in a single family stand out.

Please do let us know if we successfully addressed your concerns, or if you have any further questions or concerns, we would be happy to address them.

Best Regards

Authors of Paper #1055

Answers to Questions:

FLOWBENCH currently covers the following three predictive analyses that help significantly lower the computational expenditure of future works in this direction:

1. In Sec. 5.1, we find that there is a high correlation in the performance of optical flow estimation methods against targeted attacks using different targets, thus saving compute resources for future works as they need to evaluate only against one target. Please note, that evaluations using adversarial attacks are extremely expensive both time-wise and computation-wise. Thus, the impact of this finding cannot be underscored as one can be used as a prediction for the other.

2. In Sec. 5.5, we show that white-box adversarial attacks on optical flow estimation methods can be independent of the availability of ground truth information, and can harness the information in the initial flow predictions to optimize attacks, thus overcoming a huge limitation in the field. Here, the initial predicted flow acts as a proxy for the ground truth flow, reducing the huge amount of work required to capture ground truth flow.

3. In Appendix A, we show that synthetic common corruptions included in FLOWBENCH serve as a reliable proxy for possible real-world domain shifts and corruptions, thus significantly reducing the effort to gather such data in the wild, and helping users predict the performance of their methods in real-world scenarios.

Lastly, please note that training optical flow methods is extremely expensive, and thus we use publicly available checkpoints. FLOWBENCH is built upon ptlflow, and any new method and checkpoint added to ptlflow in the future would be implemented using FLOWBENCH. Following your suggestion and given the time constraint, we extend our analysis to two recently proposed methods SplatFlow (February, 2024) [1] and NeuFlow (March, 20204) [2]. We urge the community to use FLOWBENCH upon acceptance and expand the analysis to future methods as they are proposed.

Please note that our current analysis contains evaluations against many targeted and non-targeted adversarial attacks: FGSM ($\ell_{\infty}$ and $\ell_2$-norm bounded), BIM ($\ell_{\infty}$ and $\ell_2$-norm bounded), PGD ($\ell_{\infty}$ and $\ell_2$-norm bounded), CosPGD ($\ell_{\infty}$ and $\ell_2$-norm bounded), PCFA ($\ell_2$-norm bounded), and Adversarial Weather, and 23 Common Corruptions for 91 models. Barring limited analysis on Adversarial Weather (sparks, rain, snow, fog) due to their high computational requirements, all other evaluations exist for each of the 91 models, i.e. 9123 (common corruptions) + 918 ( $\ell_{\infty}$ and $\ell_2$-norm bounded non-targeted adversarial attacks) + 91*18 ($\ell_{\infty}$ and $\ell_2$-norm bounded targeted adversarial attacks with zero target and negative flow) + 91 (i.i.d. evaluations), which is over 4500 evaluations total over the datasets: KITTI2015, Sintel (clean), and Sintel (final). Thus we believe that our claim of “comprehensive” is justified. To the best of our knowledge, this is the most comprehensive optical flow benchmark to date. Please refer to the appendix for all the results.

Please do let us know if you have any further questions or concerns, or if you feel that we missed addressing any of your questions or concerns, we would be happy to address them.

Best Regards

Authors of Paper #1055

References:

[1] Wang, Bo, Yifan Zhang, Jian Li, Yang Yu, Zhenping Sun, Li Liu, and Dewen Hu. "Splatflow: Learning multi-frame optical flow via splatting." International Journal of Computer Vision (2024): 1-23.

[2] Zhang, Zhiyong, Huaizu Jiang, and Hanumant Singh. "NeuFlow: Real-time, High-accuracy Optical Flow Estimation on Robots Using Edge Devices." arXiv preprint arXiv:2403.10425 (2024).

Dear Reviewer F85d,

Thank you very much for your review, we greatly appreciate your input and suggestions.

As you rightly point out, we might have initially missed the opportunity to better communicate the evaluation metrics such as GAE (previously GAM) in the main paper, by only providing details in the appendix. We therefore revised our submission to now better highlight that the work does consider realistic corruptions that an optical flow method can face in the wild similar to those suggested by you (revisions are in blue in the revised paper now available). We would like to take this opportunity to highlight these evaluations and the contributions of said evaluations:

Our analysis and all possible evaluations using FLOWBENCH, go beyond just theoretical worst-case scenarios of adversarial attacks. FLOWBENCH as a benchmarking tool, and our evaluated benchmark for analysis, both also consider synthetic image corruptions. We now discuss and empirically prove in Appendix A, that synthetic Common Corruptions serve as a reliable proxy for the real world. Unfortunately, there exists no dataset captured in the wild with corruptions and domain shifts that also contains ground truth for optical flow estimation, however, this effort has been made for semantic segmentation with the ACDC dataset. In Appendix A we discuss how for semantic segmentation, synthetic image corruptions (very similar to those used in FLOWBENCH) accurately represent the possible domain shifts and corruptions that can be captured in the wild.

We specifically use 2D Common Corruptions and 3D Common Corruptions for our evaluations. The calculated Generalization Ability, GAE (previously GAM) is the maximum EPE of a method across all 15 2D Common Corruptions: Gaussian Noise, Shot Noise, Impulse Noise, Defocus Blur, Frosted Glass Blur, Motion Blur, Zoom Blur, Snow, Frost, Fog, Brightness, Contrast, Elastic Transform, Pixelate, JPEG Compression, and eight 3D Common Corruptions: Color Quantization, Far Focus, Fog 3D, ISO Noise, Low Light, Near Focus, XY Motion Blur, and Z Motion Blur. All the common corruptions are at severity 3. There exist more 3D Common Corruptions, however computing them is extremely resource intensive. Please note that attempting to capture these corruptions in the real world, and annotating the ground truth flow is significantly more expensive.

Thus, we believe that our analysis is well-rounded and takes care of possible real-world scenarios suggested by you.

Regarding the suggestions for the analysis in this work, thank you very much for bringing it to our attention. We now address this in the revised conclusion of the work. We discuss that methods using attention-based pointing matching are marginally more reliable than methods using other matching techniques, while methods using CNN and Cost Volume-based matching have marginally better generalization abilities.

This in conjecture with a previous observation that there is no apparent correlation between generalization abilities and the reliability of optical flow estimation methods, helps us conclude that based on current works, different approaches might be required to attain reliability under attacks and generalization ability to image corruptions.

Please note, here we can merely talk about “improvements” as “marginal” because as seen from the TARE, NARE, and GAE (previously TARM, NARM, and GAM) values, which are essentially EPE values calculated over multiple evaluation methods, there is an absolute lack of robustness in all the optical flow methods. These observations are merely highlighting “the least bad among the worst”. While the EPE values of these methods for i.i.d. Evaluations are in the low single digits, their EPE values for evaluations under adversarial attacks and image corruptions are in the hundreds!

Dear Reviewer F85d,

Thank you very much for your reply.

Anticipating requests for data accessibility, we have included all individual experimental results in the appendix, additionally, we have included samples of perturbed and corrupted images in the appendix. We request the reviewer to please confirm if this is the data being referred to or if we have misunderstood the request?

Regarding the reviewer's suggestion to include “more state-of-the-art (SOTA) methods”, to the best of our knowledge we have now included all peer-reviewed state-of-the-art (SOTA) optical flow estimation methods available publicly. Would the reviewer have suggestions of any particular optical flow method that we might have overlooked?

As acknowledged by Reviewer MVeY, “The writing is overall in good quality.”, and Reviewer hv2r, “The paper is written clearly so that it's easy to follow and understand many graphs, metrics, and analyses in the paper.”, we have attempted to show that compared to the errors on i.i.d. Data, the errors under attack and errors under image corruptions are significantly higher, making the differences between methods hardly significant and thus demonstrating an alarming threat to these methods if deployed in the real world. Additionally, this work has more significant contributions, such as those highlighted in the “Summary to First Author Response”:

First, we demonstrate the need for FLOWBENCH, the lack of a framework to study the reliability and generalization abilities of optical flow methods alongside their i.i.d. performance has led to the unfortunate reality of methods proposed over time only improving the i.i.d. performance and not their reliability and generalization abilities. As commonly used, reliability is the robustness against white-box adversarial attacks, as they serve as a proxy for the worst-case scenario. Generalization ability is the robustness to image corruptions, specifically 2D and 3D Common Corruptions as they serve as a proxy for real-world domain shifts and signal corruptions. The lack of reliability and generalization ability is severely alarming, as the error of optical flow estimation methods in i.i.d. evaluations is in the low single digits, whereas the error of methods in evaluations against adversarial attacks and common corruptions is in the high hundreds!

Second, in Sec. 5.1, we find that there is a high correlation in the performance of optical flow estimation methods against targeted attacks using different targets, thus saving compute resources for future works as they need to evaluate only against one target. Please note that evaluations using adversarial attacks are extremely expensive both time-wise and computation-wise. Thus, the impact of this finding cannot be underscored.

Third, in Sec. 5.2, we observe the methods known to be SotA on i.i.d. samples are not reliable, and do not generalize well to image corruptions, demonstrating the gap in current research when considering real-world applications. Additionally, we observe here that there is no apparent correlation between generalization abilities and the reliability of optical flow estimation methods.

Fourth, in Sec. 5.3, we show that methods using attention-based pointing matching are marginally more reliable than methods using other matching techniques, while methods using CNN and Cost Volume-based matching have marginally better generalization abilities. Please note that observations three and four together help us conclude that based on current works, different approaches might be required to attain reliability under attacks and generalization ability to image corruptions.

Fifth, in Sec. 5.4, we show that, unlike image classification, increasing the number of learnable parameters does not help increase the robustness of optical flow estimation methods.

Sixth, in Sec 5.5, we show that white-box adversarial attacks on optical flow estimation methods can be independent of the availability of ground truth information, and can harness the information in the initial flow predictions to optimize attacks, thus overcoming a huge limitation in the field.

Such an in-depth understanding of reliability and generalization abilities to optical flow estimation methods can only be obtained using our proposed FLOWBENCH. We are certain that FLOWBENCH will be immensely helpful to gather more such interesting findings and its comprehensive and consolidated nature would make things easier for the research community. We would be happy to include any specific suggestions the reviewer might have to improve the manuscript to better put across our takeaways.

Please do let us know if we successfully answered your questions and concerns, or if you have any further questions or concerns, we would be happy to address them.

Best Regards

Authors of Paper #1055

Thanks for the discussion, both Reviewer F85d and authors.

I partially agree with Reviewer F85d. Here are some points that I agree with:

• Data accessibility

  Though Appendix H includes raw data, it's very hard to parse the information. Also, there are so many methods in Fig. 2 that it takes time to digest the information and find corresponding methods in the legend (it's good to be comprehensive, though). It will be much better to have a benchmark website where people can interactively sort methods by each criterion. Will it be possible to prepare an early version of an interactive benchmark website during the rebuttal period?

• Adding more recent SOTA methods

  When quickly looking at the Sintel benchmark, yes, some recent methods are missing from the paper, such as CroCo-Flow (ICCV 2023), FlowDiffuser (CVPR 2024), MemFlow (CVPR 2024), but there can be more. (CVPR 2024 papers can be considered as unpublished as the ICLR deadline was in May.) However, I believe that the new methods can be added easily by running the established library. Is that right?

However, I disagree with these points:

• Empirical report

  Yes, the paper is a bit closer to an empirical report. It doesn't discuss the causation of each design factor but rather talks about correlation at a method/checkpoint level. However, I think this is also one of the things that the paper teaches us: after looking at the results in the paper, now we know that we also need analyses at an architectural component level under a controlled setup. It's a bit hard to realize that before reading this paper. To have a better robustness benchmark, we gotta start from somewhere I think. In-depth robustness analyses under controlled configuration sound an interesting direction, by the way as a whole new paper.

• 4,500 experiments

  If I understood correctly, these are already done, and people don't have to run by themselves again. If a new method comes, it can only run the new one and include it in the benchmark I think.

• Adding non-DL methods

  Yes, it's really interesting to see how the non-dl methods behave. However, in terms of EPE, I don't remember what the best-performing one was on Sintel or KITTI. Maybe the last one can be around 2017/2018. Could you name some methods if they shouldn't be missed in the comparison? If needed, this paper can frame itself as a robustness benchmark that compares DL methods exclusively. Also, one could also write a new paper for a comprehensive robustness comparison between DL vs non-DL methods.

I hope Reviewer F85d has more time or bandwidth to discuss these points instead of concluding the evaluation early. I hope to hear your thoughts on those.

Dear Reviewer F85d,

As requested, following we share some model weights in FLOWBENCH and their respective sources. Please note, FLOWBENCH itself offers many more combinations. As we mentioned in the paper and mentioned by Reviewer hv2r, we have not trained the models ourselves, thus for the training implementation we share their respective GitHub repositories.

Please do let us know if this answers your concerns.

Best Regards

Authors of Paper #1055

Reviewer hv2r,

As reviewers, it is our responsibility to uphold the standards of the community and ensure the integrity of the reviewing process. Our role is to assess submissions critically and recommend disqualification for work that does not meet the rigorous standards of ICLR, a flagship conference in AI. While I welcome and respect differing opinions, it is not appropriate to instruct a fellow reviewer on how to proceed. Such decisions ultimately rest with the Area Chair (AC).

Regarding this particular submission, I have consistently pointed out in my comments that the core value of the work is undermined by the absence of critical elements such as data accessibility (adversarial samples, empirical weights, and logs). Without this information, it is impossible to reproduce or rigorously examine the results. The authors’ arguments have not convinced me otherwise, as the empirical data forms the foundation of the work’s value. Understanding factors that impact algorithmic robustness is important, but without access to supporting data, the validity of the conclusions cannot be verified.

I am trying to understand your motivation in encouraging me to change my judgment. Do you believe this is an exceptional work that must be shared with the community despite these shortcomings? If so, I would question your understanding of the task and the standards of ICLR. Alternatively, if there are other reasons, I am open to hearing them.

Until then, I firmly believe that this submission requires further refinement before it is ready for consideration.

Dear Reviewer hv2r,

Thank you very much, we deeply appreciate your efforts for not just providing a sound and thorough review with a positive evaluation of our work, but also positively engaging in further discussions.

Data accessibility

We agree that making meaningful observations with plots when performing a study as large scale as the one provided by us can be difficult. In our experience, including tables for the numbers would have been even more difficult to read thus we opted for plots. We completely agree that an interactive website is the way to go, anticipating this we already have an initial version of the website ready, and we would have it hosted with the camera-ready submission. The domain we have for the website unfortunately reveals our identity, and thus to safeguard the sanctity of the double-blind review process, we chose to not share the URL. We share a screenshot from a very initial prototype of this website in Appendix I (latest revision). Additionally, we now created an identity leak-proof anonymous Google sheet with multiple sheets that have values from our evaluations, that we shared with the chairs. Sharing these results publicly will allow anyone to easily reproduce and steal all our evaluations which is why we refrain from publicly sharing this document before acceptance. We believe that if required, one can already verify our evaluations using Appendix H.

Adding more recent SOTA methods

Yes, your understanding is correct, “new methods can be added easily by running the established library”. We agree that including these could enhance the comprehensiveness of the work, but as acknowledged by you (Reviewer hv2r) and Reviewer MVeY, there will always be new methods coming up. The only reason we could not include these in our evaluations is because they are currently not reliably supported by PTLFLow as they are fairly new and integrating new methods is a time-consuming process, and, PTLFlow mentions “PTLFlow is still in early development, so there are only a few models available at the moment, but hopefully the list of models will grow soon.” Post acceptance, we hope to publicly aid in this endeavor.

Empirical report

Thank you very much for acknowledging the contributions of this work.

4,500 experiments

Yes, we attest that your understanding is correct.

Adding non-DL methods

Thank you very much for acknowledging that “one could also write a new paper for a comprehensive robustness comparison between DL vs non-DL methods”. Currently, we attempt to frame the paper towards DL-based optical flow methods, while in favor of saving space, we often refer to methods as just “optical flow estimation methods” or “optical flow methods”, in some cases like the abstract and introduction, we call them “learning-based optical flow estimation methods” and “DL-based optical flow estimation methods”. We begin the motivation of this work by mentioning the vulnerability of DL-based methods towards adversarial attacks and common corruptions, and then we extend the motivation to have safer DL-based optical flow estimation methods. Nonetheless, if the reviewers suggest, we can try to align the paper more towards DL-based optical flow methods and mention so more exclusively.

Best Regards

Authors of Paper # 1055

Reviewer F85d,

My apology if my response sounds like instructing a fellow reviewer on how to proceed. My intention was more like trying to open a discussion with a fellow reviewer as "this concludes my evaluation" sounded like "the end of discussion" to me. I wanted to hear more details on the thoughts that are different from mine and doublecheck if there are some points that I missed. And I am glad that the discussion can be continued.

What are the adversarial samples, empirical weights, and logs? If I understood correctly, the paper grabbed over 90 pretrained checkpoints, performed each adversarial attack, and evaluated the metric. Does adversarial samples mean that corrupted/attacked images of each method? Do empirical weights are pretrained checkpoints of all methods used in the benchmark? Are logs the ones from evaluation processes? If that's the case, I think authors can simply attach those data in the supplementary so that we all can have a look. This will ensure transparency of the evaluation. If not, this concern will still remain.

Again, I am not encouraging you to change your judgment but just wanted to hear your thoughts and try to understand the shortcomings of the paper for a better judgment. I would just appreciate it if you are willing to discuss something even without changing the rating. I think this paper is not an exceptional work but rather a good seed work that people can have a look at the extensive benchmark, compare almost all (DL-based) existing optical flow methods, and hopefully design more robust architectures, and conduct more improved analyses or benchmarking experiments, based on this paper.

If only exceptional works that must be shared with the community are the papers accepted at ICLR, probably you are right that I have different standards on ICLR.

Best,

Reviewer hv2r

Dear Reviewer F85d,

Do these model weights clarify the issue? We would appreciate your letting us know whether more data is needed. Regarding plain evaluation files and logs, we hope you understand that we can not post them openly on openreview prior to acceptance. It would allow anyone to copy our entire paper. We have, however, made sheets with all plain evaluation outcomes available to the AC to prove that our results are valid and reproducible. Further, we will, of course, make everything publicly available upon acceptance. Screenshots of the respective website can already be viewed in the current revision.

Thank you for the discussion and for letting us know whether anything more is required from your side!

Best Regards,

Authors of Paper # 1055

Dear Reviewer MVeY,

Thank you very much for your score and for your review, we greatly appreciate your input and suggestions. We greatly appreciate the effort invested in also reading the appendix.

For the specific suggestions for improvement, we have now incorporated these in our revised version of the paper (available now, revisions are in blue), thank you very much for these suggestions.

We would like to address your concerns as follows:

1. Yes, we used EPE in different settings for TARE, NARE, and GAE (previously: TARM, NARM, and GAM) values. However, FlowBench also captures a lot of other metrics, some of them being Cosine Distance, outlier error, 1px error, 3px error, and 5px error (We now describe this in Appendix C). Users are free to choose which metric they would like to use. Regarding very high EPE values, yes indeed these very high values are EPE, as shown by Agnihotri, et. al. [3] in their work CosPGD, Section C.2 when evaluated against non-targeted adversarial attacks, the EPE values can exceed 100. This further highlights the extent to which optical flow methods are vulnerable against adversarial attacks. We now discuss the specific difference between targeted and non-targeted adversarial attacks in Section 4.2.

2. Yes, we agree this would indeed be very interesting. Our current understanding is that traditional non-DL-based optical flow methods should be more robust to adversarial attacks and most common corruptions when compared to DL-based optical flow methods. However, a few 3D common corruptions considered in this work, like z-blur, and xy-blur might pose a challenge to these traditional methods as well. Given that the community’s motivation behind moving to DL-based optical flow methods was purely i.i.d. Performance-based, it would be interesting to evaluate the robustness of the traditional methods to draw inspiration for future, more robust methods. Currently, given the short deadline for the rebuttal, and given that ptlflow currently does not support any traditional methods, it would be very difficult to evaluate them, however, this is a very interesting suggestion that we would definitely consider for the extension of this work and now mention as Future Work in Section 6 of our revised paper.

3. Thank you very much for this suggestion, we have now renamed the metrics as “Errors” where lower is better. For “TARE” (previously “TARM”), we report -1*Error, and thus now the lower the “TARE” value the more reliable the method and vice-versa. We tried using the reciprocal, however, this exhibits a weird scaling that might make it less intuitive for the readers. Please let us know whether the suggested TARE works better from your perspective!

Answer to Question:

We consider the reproducibility of our results to be of utmost importance. To the best of our understanding, there is no randomness in results. In the case of adversarial attacks such as PGD, and CosPGD the random noise added before optimization is the only point where some stochasticity can be introduced. However, using the seeds we used, all evaluations are completely reproducible. We provide these seeds in the code. For common corruption, the entire pipeline is completely deterministic.

Please do let us know if you have any further questions or concerns, or if you feel that we missed addressing any of your questions or concerns, we would be happy to address them.

Best Regards

Authors of Paper #1055

References:

[3] Agnihotri, S., Jung, S. &; Keuper, M.. (2024). CosPGD: an efficient white-box adversarial attack for pixel-wise prediction tasks. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:416–451 Available from proceedings.mlr.press/v235/agnihotri24b.html.

Thank you for your response, which resolves most of my concerns. On the randomness of evaluation, I don't think fixing random seeds is a great way to avoid randomness because different versions of packages may behave differently even if we use the same random seed, so maybe it will be better if we could have an analysis on the variance of the metrics to help better understand these metrics.

Overall, although there may be weaknesses of this benchmark, as pointed out by the other reviewers, I still see values of this work as the first benchmark on optical flow robustness and generalization. Also, the extensive experiment results are worth sharing and can help researchers better understand the status quo of optical flow estimation from new perspectives other than just accuracies. Therefore, I agree with Reviewer hv2r and recommend acceptance.

Dear Reviewer hv2r,

Thank you very much for your score and your review, we greatly appreciate your input and suggestions.

We have incorporated all your suggested improvements in our revised submission (available now, revisions are in blue), thank you very much for these suggestions.

We would like to take this opportunity to further discuss the concerns you shared:

Comparability of Evaluated Models: Yes, we agree that the training setup can play a significant role in the robustness of a method. However, in the case of optical flow methods, they are hardly ever adapted to the target setting since the target application usually does not have ground truth leading one to rely on the available pre-trained models. Currently, the training strategies and architectural design choices of different methods are significantly intertwined. For example, MS-RAFT+ is trained significantly differently from FlowFormer++, FlowFormer++ can be evaluated on a dataset only if it is finetuned for that dataset (which we did), however, MS-RAFT+ is proposed with a single training strategy to work across datasets. To ensure fair comparison, we contacted the authors of MS-RAFT+ and they accepted that our evaluations would be fair according to them. In our current analysis, we attempt to make significant observations based on multiple models, keeping their training and evaluation settings intact. We agree that it would be difficult to disentangle if the performance difference is due to the architectural design choice or due to the training strategy, however, previous works also do not disentangle the two when comparing i.i.d. Performance. Given that, optical flow methods were not proposed to specifically overcome adversarial attacks or image corruptions, we hypothesize that training all optical flow with a different known setup would not help improve their reliability and generalization abilities and the current observations from this work would still hold Thus, we believe that the robustness evaluations performed here are realistic. To improve the reliability and generalization abilities of optical flow methods, better approaches need to be proposed, that focus on these aspects, and studying these would be very interesting. Thank you for the suggestion, we now include a synopsis of this discussion in future work.

In-depth analyses and Question: Thank you very much for raising this question, we now attempt to address this question in the revised conclusion of the work. We discuss that methods using attention-based point matching are marginally more reliable under attack than methods using other matching techniques, while methods using CNN and Cost Volume-based matching have marginally better generalization abilities to image corruptions. This in conjecture with another observation, in this work, that there is no apparent correlation between generalization abilities and the reliability of optical flow estimation methods, helps us conclude that (based on current works) different approaches might be required to attain reliability under attacks and generalization ability to image corruptions. Please note, here we can talk about “improvements” as merely “marginal” because as seen from the TARE, NARE, and GAE (previously TARM, NARM, and GAM) values, which are essentially EPE values calculated over multiple evaluation methods, there is an absolute lack of robustness in all the optical flow methods. These observations are merely highlighting “the least bad among the worst”. While the EPE values of these methods for i.i.d. Evaluations are in the low single digits, their EPE values for evaluations under adversarial attacks and image corruptions are in the hundreds!

Regarding the comment on Fig. 1: We continue the discussion from the previous question. Indeed there are a few data points belonging to recently proposed methods that lie below the regression line, but we do not manually remove any data point. We observe in Fig. 1 (right), the EPE values across 2D and 3D common corruptions, denoted using GAE, are in the 100s, while the EPE values on i.i.d. datasets are in the low single digits, thus these optical flow methods are very far away from true generalization abilities. Ideally, the generalization ability of methods to corruptions would be such that they are completely robust of image corruptions, having the same EPE on i.i.d. and OOD evaluations, however, currently we are very far from this ideal scenario, and thus even with the regression line slightly improving, to the best of our understanding, the observations would still hold.

Regarding the comment on Fig. 2: Thank you very much, we indeed made a mistake in the text, we have now corrected this in the revised version.

Please do let us know if you have any further questions or concerns, or if you feel that we missed addressing any of your questions or concerns, we would be happy to address them.

Best Regards

Authors of Paper #1055

Here, we summarize the key takeaways of our paper, as highlighted in our revision.

First, we demonstrate the need for FLOWBENCH, the lack of a framework to study the reliability and generalization abilities of optical flow methods alongside their i.i.d. performance has led to the unfortunate reality of methods proposed over time only improving the i.i.d. performance and not their reliability and generalization abilities. As commonly used, reliability is the robustness against white-box adversarial attacks, as they serve as a proxy for the worst-case scenario. Generalization ability is the robustness to image corruptions, specifically 2D and 3D Common Corruptions as they serve as a proxy for real-world domain shifts and signal corruptions. The lack of reliability and generalization ability is severely alarming, as the error of optical flow estimation methods in i.i.d. evaluations is in the low single digits, whereas the error of methods in evaluations against adversarial attacks and common corruptions is in the high hundreds!

Second, in Sec. 5.1, we find that there is a high correlation in the performance of optical flow estimation methods against targeted attacks using different targets, thus saving compute resources for future works as they need to evaluate only against one target. Please note that evaluations using adversarial attacks are extremely expensive both time-wise and computation-wise. Thus, the impact of this finding cannot be underscored.

Third, in Sec. 5.2, we observe the methods known to be SotA on i.i.d. samples are not reliable, and do not generalize well to image corruptions, demonstrating the gap in current research when considering real-world applications. Additionally, we observe here that there is no apparent correlation between generalization abilities and the reliability of optical flow estimation methods.

Fourth, in Sec. 5.3, we show that methods using attention-based pointing matching are marginally more reliable than methods using other matching techniques, while methods using CNN and Cost Volume-based matching have marginally better generalization abilities. Please note that observations three and four together help us conclude that based on current works, different approaches might be required to attain reliability under attacks and generalization ability to image corruptions.

Fifth, in Sec. 5.4, we show that, unlike image classification, increasing the number of learnable parameters does not help increase the robustness of optical flow estimation methods.

Sixth, in Sec 5.5, we show that white-box adversarial attacks on optical flow estimation methods can be independent of the availability of ground truth information, and can harness the information in the initial flow predictions to optimize attacks, thus overcoming a huge limitation in the field.

Such an in-depth understanding of reliability and generalization abilities to optical flow estimation methods can only be obtained using our proposed FLOWBENCH. We are certain that FLOWBENCH will be immensely helpful to gather more such interesting findings and its comprehensive and consolidated nature would make things easier for the research community.

We would additionally address reviews from each reviewer for finer details and questions asked.

Thank You,

Regards

Authors of Paper #1055

References:

[1] Wang, Bo, Yifan Zhang, Jian Li, Yang Yu, Zhenping Sun, Li Liu, and Dewen Hu. "Splatflow: Learning multi-frame optical flow via splatting." International Journal of Computer Vision (2024): 1-23.

[2] Zhang, Zhiyong, Huaizu Jiang, and Hanumant Singh. "NeuFlow: Real-time, High-accuracy Optical Flow Estimation on Robots Using Edge Devices." arXiv preprint arXiv:2403.10425 (2024).