3D Visual Illusion Depth Estimation

Response to Reviewer hfin

We sincerely thank the reviewer for the insightful comments and constructive suggestions. Below, we provide detailed responses to each of the concerns.

Ambiguities in Virtual Data Generation

Due to the lack of established methods and annotated datasets for detecting 3D visual illusions, we currently rely on manual identification of illusion regions. Prior detection or segmentation methods are easily misled by our 3D visual illusion data. Additionally, the complex content, often involving multiple overlapping elements within a single illusion, requires multiple extensive manual interactions using SAM2 to fully annotate each single illusion region.

---

Insufficient Details on VLM Fine-Tuning

We apply LoRA to fine-tune the last layer of QwenVL2-7B and the QV projection layer of FLUX, as detailed in lines 100–116 of the supplemental materials.

---

Generalization to Unseen Illusions

This is an interesting and challenging suggestion.

1. Since VLMs often struggle with semantically ambiguous scenes, such as inpainting or printed illusions, we currently restrict their use to identifying mirror illusions, which are especially problematic for stereo matching. As long as a novel illusion does not deceive the VLM or both the monocular and stereo models, our approach can effectively handle it. However, when encountering a composition of different illusion types, our method may fail.

2. Our experiments on Booster represent a cross-dataset validation setting. Our model is fine-tuned on our dataset and evaluated directly on the Booster training set. Results indicate that our method achieves over 10 points of improvement in the bad x metric.

---

Automation in Data Generation

This is a promising direction and one we are actively exploring. However, given the near absence of existing datasets for 3D visual illusions, current general-purpose detection and segmentation tools (e.g., SAM, DINOv2) fail to perform reliably. In fact, even VLMs are often fooled by these illusions, and humans ourselves can be misled by complex illusions, as illustrated in Figure 1 of both the main text and the supplemental materials.

Thus, while unsupervised learning alone may currently be insufficient, collecting more diverse and representative data is a practical step forward. Developing automated detection methods for 3D visual illusions remains an exciting direction for future research.

Response to Reviewer c4xs

We appreciate the reviewer’s thoughtful feedback on our model computation, dataset generalization, evaluation fairness, and the role of VLM driven fusion. Below, we address each concern with detailed explanations, additional analyses, and planned revisions.

---

High Model Complexity and Computational Cost

We acknowledge that the current system exhibits high complexity due to the integration of monocular, binocular, and vision-language models. However, we do not claim to have perfectly solved the 3D visual illusion challenge. Rather, our primary contribution lies in systematically investigating where and why different model types fail, a question of growing importance for downstream applications such as embodied AI, VLM, and automous driving, especially in light of the rapid development of 3D foundation models.

Based on our observations and analyses, we propose a reference baseline that serves as a conceptual and empirical guide to mitigate the impact of this challenge. While we anticipate that future methods will definitely achieve higher efficiency and better performance, we believe that at this stage, raising awareness of the 3D visual illusion challenge and identifying failure modes across different model paradigms is a more urgent and foundational task.

To address concerns regarding computational cost, we will include detailed inference metrics (runtime and memory consumption) in the revised experimental section. All results were obtained on a single NVIDIA H100 GPU with input resolution of $1920 \times 1080$:

---

Potential Unfairness in Baseline Comparisons

We ensure fair and consistent comparisons across baselines. The models in the ablation study presented in Table 3 of the main text are all fine-tuned on the 3D-Visual-Illusion dataset and evaluated on the Booster training set, as noted in Lines 235–237. The first row of Table 3 represents our stereo-only baseline model, which significantly underperforms compared to our full model. Its performance is even worse than RAFT-Stereo without fine-tuning, as shown in Table 2, highlighting the limitations of prior architectures on large-scale illusion data.

Regarding the monocular-only model, DepthAnything V2 failed to converge on the 3D-Visual-Illusion dataset. Despite extensive attempts, including data augmentation (e.g., reducing non-illusion regions), multi-stage training (e.g., pre-training on partial regions), and modified loss functions (e.g., weighted loss for illusion areas), the model consistently crashed during training and produced noisy, unusable outputs. Consequently, we chose not to report meaningless results in Table 3. We will include this explanation in the revised manuscript.

For fusion effectiveness, we evaluated multiple fusion strategies and demonstrated that our VLM-driven fusion yields more than a 10-point improvement in the bad x metric, confirming its effectiveness.

---

Dependence on a Closed-Domain Dataset

We appreciate the reviewer’s emphasis on evaluating generalization beyond the proposed dataset.

As shown in Table 2 of the main text, we have already reported the zero-shot performance of our model on the non-transparent regions of the Booster dataset, which represent mundane, non-illusory scenes. Our model consistently outperforms all baselines across all metrics.

To further minimize the potential influence of adjacent illusory regions on nearby non-illusion regions in Booster, we additionally conduct a zero-shot generalization evaluation on the well-known Middlebury dataset, where scenes are predominantly free from any 3D visual illusions. We compare our model with several SOTA stereo-based approaches using metric disparity space over the entire image, without restricting the maximum disparity range.

As shown in the table below, our model does not degrade in performance on these mundane scenes. On the contrary, it achieves improvements, particularly in terms of EPE:

These results confirm that our model maintains or improves performance in standard, non-illusory environments, supporting its generalizability and robustness across domains.

---

Limited Ablation on VLM's Reasoning

We thank the reviewer for highlighting the need for a deeper analysis of the VLM’s reasoning capabilities.

In response, we provide a quantitative evaluation of the VLM’s confidence by comparing the predicted confidence maps with the disparity error maps, as defined in Equation 10 of the supplemental material. This analysis is conducted on the Booster dataset under a zero-shot generalization setting. The results show that the error rate of our confidence estimation is approximately 20%, demonstrating a strong generalization ability, even in previously unseen illusory scenes.

---

Clarity of Model Checkpoint in Table 1

We appreciate the reviewer’s request for clarification on the comparison setup in Table 1.

The methods listed in Table 1 are all pretrained models evaluated without any fine-tuning on our dataset. All model weights are obtained from their respective official repositories. Specifically, the checkpoint for DepthAnything V2 (metric) is taken from the official GitHub repository under the Indoor setting. We will include detailed model checkpoint sources and configurations in the revised version for full transparency and reproducibility.

---

More Ablation Study on Model Candidates

We acknowledge the reviewer’s interest in additional ablation studies across different model candidates. However, as noted in Line 116 of the supplemental material, the training time for each model variant is nearly 20 days, making it impractical to conduct further large-scale ablations within the current project scope. We hope the following clarification is satisfactory.

In our study, we carefully selected well-established and widely adopted architectures for both monocular and binocular baselines. For the vision-language component, we employed the most powerful open-source model available from Qwen at the time of experimentation.

---

Semantically Ambiguous Objects

The suggestion regarding semantically ambiguous failure cases is quite interesting. And we will add more failure case analyses in the supplemental material.

Notably, VLM models struggle in semantically ambiguous scenes, such as inpainting or printed illusions. These illusions can mislead even VLMs. Therefore, we currently restrict the use of VLMs to identifying mirror illusions, which are particularly problematic for stereo matching.

The solution of our model currently focuses on real-world illusions with potential risks in downstream applications, e.g., a printed road image on a wall. We do not explore highly artistic or contrived illusions (e.g., simulated 3D windows behind iron frame), which lie outside the current scope but are an important direction for future exploration.

Thank you for the clarification. The new information is helpful, but it has revealed a significant contradiction in your results and has not resolved our core concern about fair comparisons.

1. Regarding the Need for a Clear and Fair Comparison

The comparisons in Tables 1 and 2, as they stand, remain methodologically unsound for making performance claims.

• On your stated goal of "raising awareness": Simply placing your fine-tuned model next to zero-shot baselines in Table 1 is confusing, not illuminating. A more effective way to demonstrate the challenge would be to restructure the table to directly compare baseline models' performance on illusion vs. non-illusion regions.
• On performance attribution: Because the baseline models in Table 2 were not fine-tuned on your illusion dataset, it is impossible to attribute your model's superior performance to its architecture. The advantage more likely stems from the specialized training data, a benefit the baselines were not given.

---

2. Regarding the Contradictory RAFT-Stereo Results

You've argued that fine-tuning on your dataset demonstrates its value. However, the results you provided show the exact opposite for the RAFT-Stereo baseline:

As shown, fine-tuning the baseline on your dataset significantly degraded its performance across all metrics (e.g., EPE increased from 9.55 to 15.11).

• Could you please explain this major performance drop? This result seems to suggest that standard architectures like RAFT-Stereo are fundamentally unable to learn effectively from your dataset's distribution. This is an important finding and should be stated more clearly with evidence and experiments.

---

I believe that addressing these final points will resolve the remaining ambiguities and allow for a more accurate assessment of your work's important contributions.

Response to Reviewer bw6S

Limited Novelty

Our key contributions include:

• Identification of new and significant 3D visual illusion challenges to diferent 3D foundation model.
• Construction of a large-scale dataset featuring diverse simulated and real-world data.
• Introduction of an effective baseline for addressing these challenges.

As mentioned in Lines 5, 8, and 38 of the main text, our focus is on monocular and binocular depth estimation, not multi-view methods. We provide a comparison with VGGT in Tables 1 and 2, which leverages richer monocular information and demonstrates significantly better performance than classical multi-view methods.

Your referenced methods fuse monocular priors into volume, which introduces a strong monocular bias. This early fusion makes the matching process vulnerable to monocular-related illusions such as inpainting, picture, replay, and holography illusions.

---

No Need for VLM-based Monocular Depth Fusion

The comparable performance of our model to stereo-based methods on the 3D-Visual-Illusion dataset arises because its test set comprises real-world data, which lacks mirror illusions. Since stereo-based methods are more sensitive to such illusions, we evaluate generalization on the Booster dataset, which includes challenging mirror-like regions.

As shown in Table 2, our method improves the bad x metric by over 10 points in transparent regions, clearly outperforming stereo-based alternatives. Further validation through the ablation study in Table 3 confirms the effectiveness of the VLM-based monocular depth fusion approach.

---

Missing Zero-shot Performance Evaluation

The performance shown in Table 1 reflects a generalziation evaluation of our model from virtual data to real-world data. Table 2 further demonstrates zero-shot generalization on the Booster dataset, as described in Lines 235–236 of the main text.

---

Improvements in Non-Visual-Illusion Regions

Contrary to concerns, our model also exhibits 3–5 point improvements in the bad x metric on non-transparent (non-visual-illusion) regions, as presented in Table 2. This confirms that the proposed method not only excels in illusion scenarios but also generalizes well to standard depth estimation settings.

Utilization of Specific Stereo GRU Model

We use RAFT-Stereo as it is a classic and well-known stereo baseline. To investigate the effect of different GRU architectures on fusion performance, we conduct comparisons in Table 3. A detailed explanation of these experiments can be found in Lines 265–272 of the main text.

---

Any Improvement by Using SOTA Stereo-Based Methods?

Using additional SOTA stereo-based methods is indeed a valuable suggestion. However, due to current time constraints, it is not feasible at this stage. Nevertheless, our existing experimental results already validate the effectiveness of the proposed method. Incorporating more SOTA stereo methods remains a promising direction for future work.

---

Color of Image and Text Embedding in Figure 4

Thank you for your suggestion. We will change the color of the second text embedding to yellow in Figure 4 to improve visual clarity and distinguishability.

---

Performance of Existing Stereo-Based Methods on 3D Visual Illusions

Current stereo-based methods continue to struggle with mirror illusions, particularly those involving near mirror surfaces. What's more, without appropriate fusion strategies, monocular bias may be injected into the stereo matching process, leading to degraded performance under visual illusion scenarios.

Response to Reviewer ud34

We appreciate the reviewer’s thoughtful concerns regarding the generation of ground truth for the virtual data and the potential paradox raised. Below, we provide detailed responses to each point.

---

1. Purpose of Initial Depth Model Use

We adopt DepthAnything V2 as our initial depth predictor because it is the only monocular foundation model that consistently produces generalizable and dense depth predictions across diverse scenes. In contrast, classical multi-view reconstruction methods fail to generate satisfactory dense depth for our web-sourced monocular videos, primarily due to the following challenges:

• Presence of dynamic objects
• Complex lighting conditions
• Fixed or limited camera motion, which degrades the quality of multi-view reconstruction

As mentioned in Lines 108–111, these characteristics make classical multi-view approaches unsuitable for our setting.

---

2. Manual Annotation of Illusion Masks

Since no existing method or dataset provides reliable detection of various 3D visual illusions, manual annotation is currently the only viable solution for creating illusion masks, which are crucial for both training and evaluation.

While we agree that developing an automatic monocular 3D illusion detector would enhance scalability, such work is beyond the scope of this paper and warrants future investigation.

---

3. Co-Planar Assumption and Systematic Bias

We acknowledge the concern regarding potential systematic bias introduced by the co-planar assumption. However, we argue that this effect is limited due to two key reasons:

1. Deformation reduces illusion effectiveness: In real-world scenarios, daily-life illusions lose their illusory effect when displayed on curved or irregular surfaces. Moreover, illusions specifically designed for curved surfaces are extremely rare, less than 800 images in our dataset that contains more than 200,000 images.

2. Robustness in binocular disparity learning: In rare cases where surface is curved and planar fitting is inaccurate, the pixel-wise matching mechanism in stereo models remains robust, because learning is minimally affected as long as matching is correct, no matter the surface is curved or not in zero-shot generalization testing.

---

4. Link Between Illusions and Specific Algorithms

We thank the reviewer for this insightful suggestion. The following explanation will be added to the Introduction and Method sections:

(1) Monocular Depth Estimation:

Monocular methods rely on depth cues such as shape, perspective projection, shadow, and defocus, which are embedded in RGB textures. These models learn fixed mappings from texture to geometry. However, when visual illusions simulate such texture cues on a 2D surface (e.g., via inpainting, printed pictures, replays, or holograms), the learned mappings are misled, resulting in incorrect depth predictions, which manifests as an illusory effect.

(2) Stereo-Based Depth Estimation:

Stereo methods depend on pixel-wise matching across views. Reflective or transparent surfaces cause ambiguous correspondences as both the real surface (e.g., mirror) and virtual background are projected onto the same pixels. Since most RGB signals come from the distant background, the near mirror surface is often ignored, leading to incorrect depth and a visual illusion.

(3) More Analysis:

• For inpainting, picture, replay, and holography illusions, stereo-based methods will find right and great matching between left and right images according to the rich texture.
• For mirror illusions, monocular methods will predict correct geometry due to scene understanding abilities learned from large-scale datasets.

---

5. Connection Between the Problem and the VLM-Based Solution

As illustrated in Figure 5 of the supplementary materials, Vision-Language Models (VLMs) can reliably recognize mirror illusions and provide reasoned judgments. Motivated by this, we use VLMs with carefully designed prompts to provide commonsense-level reasoning for detecting mirror illusions.

This enables the generation of a confidence map, indicating regions where binocular-based methods are likely to fail. The implementation details are discussed in Lines 202–206 of the main text and Lines 137–140 of the supplementary materials.

---

We hope these clarifications effectively address the reviewer’s concerns and help improve the understanding of our work. Thank you again for the valuable feedback.