Instruct2See: Learning to Remove Any Obstructions Across Distributions

We sincerely appreciate Reviewer zzKc's valuable feedback. Our responses to the weaknesses and questions are listed below.

R-W1 Minor performance improvements: We would like to emphasize that the primary goal of our method is not to maximize performance for any specific obstruction type, but to achieve robust zero-shot generalization across a wide range of unseen obstructions. From this perspective, our approach shows a clear advantage over existing methods, including those that claim to be category-agnostic and those we adapted to operate in zero-shot settings.

Instead of tailoring the model to known obstruction types, our distribution-agnostic formulation treats all obstructions using a unified approach. This applies regardless of their appearance, transparency level, or spatial structure. As a result, our method delivers consistent performance across both seen and unseen categories, something existing methods cannot achieve without retraining or manual tuning.

R-W2 Selection of more representative cases: Thank you for your suggestion. We will replace more representative experimental samples for visualization.

We sincerely appreciate Reviewer 4Rdn's valuable feedback. Our responses to the weaknesses and questions are listed below.

R-W1 Limited scope of experimentation: Thank you for your comment. We believe some of our experimental results may have been inadvertently overlooked. In both the main paper and supplementary materials, we present extensive zero-shot evaluations across a wide range of obstructions, including rain streaks, snow, strokes, power cables, spots, scratches, yarn, shadows, watermarks, and complex multi-obstruction scenarios. These results collectively validate the effectiveness and generalization capability of our method.

Unlike object removal, where datasets are abundant and well-defined, obstructions are often irregular in shape, appearance, and distribution, making comprehensive evaluation more challenging. To address this, we have curated and tested on all publicly available obstruction data that aligns with our problem definition. We will revise the manuscript to highlight these results more prominently. Additionally, we welcome suggestions on any other publicly available datasets we may have missed and are happy to include further evaluations as needed.

R-W2 Limited technical contribution: Thank you for your feedback. We respectfully believe this concern may arise from an underappreciation of the conceptual novelty behind our approach. Our method introduces a new formulation of obstruction removal as a distribution-agnostic soft masking problem, which fundamentally unifies the treatment of both opaque and semi-transparent obstructions within a single framework.

This perspective shifts away from traditional category-specific pipelines and reframes obstruction removal as a context-aware reconstruction task, where the model learns to reason about partially visible content regardless of obstruction type. Prior methods typically rely on known obstruction categories or retraining per class, whereas our method enables zero-shot generalization to unseen obstruction types—an ability that existing approaches lack.

Moreover, our flexible soft-mask recovery strategy, combined with multi-modal prompt integration, allows the model to adaptively handle obstructions with varying degrees of transparency, shape complexity, and semantic ambiguity. We also show that even with access to ground-truth masks, conventional methods fail to generalize across distributions, underscoring the need for our proposed formulation.

This unified, distribution-agnostic perspective and the demonstrated generalization across diverse and unseen scenarios represent a meaningful advancement in both the theoretical framing and practical capabilities of obstruction removal.

R-Q1 CLIP fine-tuning strategy: In our framework, the primary role of the text encoder is not to align text with visual features in the conventional sense, but rather to help the model interpret user instructions, particularly in understanding the transparency attributes of obstructions described in the prompt. To achieve this, fine-tuning the text encoder is essential.

For example, before fine-tuning, the cosine similarity (after softmax) between the user instruction "There are raindrops in the image, please remove them" and the two core commands "remove opaque obstructions" and "remove transparent obstructions" were 0.5374 and 0.4626, respectively. After fine-tuning, these shifted to 0.00004 and 0.99996, respectively, indicating a dramatic improvement in semantic alignment. This refinement is critical for guiding the model’s recovery strategy based on the nature of the obstruction.

Additionally, Section 5.3 (Ablation Study) in the main paper provides experimental evidence for the importance of textual and visual conditioning. We further support this with a qualitative case study in Appendix D.2, illustrating the performance gains achieved under different conditioning strategies.

We sincerely appreciate Reviewer 91FV's valuable feedback. Our responses to the weaknesses and questions are listed below.

R-W1 Domain restributions: Our model is trained on natural image datasets, and as such, it performs well across a wide range of domains within the natural image space, such as urban scenes, landscapes, wildlife, and daily objects, without restriction. However, for domains outside this distribution, such as medical imaging, the model cannot be expected to produce meaningful restorations, as the training data does not include such content.

However, our framework itself is domain-agnostic. By training on appropriately curated domain-specific data (e.g., medical images), the same architecture and methodology can be applied to obstruction removal in those domains. We will clarify this point in the revised manuscript to better reflect the scope and potential of our approach.

R-W2 Explanation of running efficiency: Thank you for your question. The efficiency of our method is largely due to the encoder-decoder architecture of the restoration model. Although the model has a relatively high parameter count, this does not translate into high computational cost. This is because deeper modules operate on low-resolution feature maps, allowing for higher channel dimensions without significantly increasing FLOPs or runtime. This design decouples parameter size from computational load, enabling efficient resource use while maintaining strong performance.

R-W3 More details of experiment: Thank you for the suggestion. We will revise the Experiment Settings section to include additional implementation and training details for improved clarity and reproducibility.

R-W4 Minor spelling error: Thank you for pointing this out. We will carefully review the manuscript to correct any spelling errors.

R-Q1 Overlapping obstruction removal: Thank you for your comment. We would like to clarify that our original submission includes experiments on images with multiple overlapping obstructions, with results presented in Appendix C.2 and Figure 11. These experiments demonstrate that our method can accurately identify and represent overlapping obstructions using multimodal prompts and masks, and that our model effectively removes them, further validating the robustness of our approach.

We sincerely appreciate Reviewer 17U3's valuable feedback. Our responses to the weaknesses and questions are listed below.

R-W1 Lack of key details: We emphasize that our method targets a fundamentally different problem than prior approaches such as Restormer and PromptIR. These existing restoration methods assume prior knowledge of obstruction characteristics and are designed to handle specific, predefined categories (e.g., rain, snow, flare). As a result, they lack generalization ability: a model trained for snow removal, for example, fails on rain without retraining, even when the distributions are similar.

In contrast, our work breaks this constraint through two key innovations: (1) a distribution-agnostic obstruction formulation, and (2) a flexible soft-mask recovery strategy integrated with multimodal obstruction representation. Together, these enable zero-shot generalization to diverse and unseen obstructions. This direction is both novel and underexplored, marking a significant advancement in the theory and practice of obstruction removal.

We will further strengthen the distinction between our method and existing work in the Introduction and Related Work sections to ensure clarity.

Additionally, we clarify that our original submission already includes: (1) a detailed rationale for dataset selection (see Section 3), and (2) complete details on prompt/text conditions, including their integration in Algorithm 1 (Appendix A) and illustrative examples in Figure 8 (Appendix B). As most of this information is provided in the appendix and may be easily overlooked, we will revise the main text to reorganize the structure or better guide readers to these supporting materials.

R-W2 Unreasonable distribution between main text and appendices: Thank you for your suggestion. We will reorganize the content distribution of the main text and appendices.

R-W3 & Q2 Application scope and problem definition: Thank you for your feedback. All obstruction categories, both seen and unseen, are sourced from publicly available datasets to ensure reproducibility.

The seen classes (Fences, Raindrops, Flares) were selected for their diversity in shape and structure, providing a strong foundation for generalizing to irregular obstructions. These include grid-like, point-based, and diffuse patterns, offering broad coverage of common obstruction forms.

The unseen classes (Rain Streaks, Snow, Stroke, Power Cables, Yarn, Scratches) were chosen to introduce appearance-level and geometric shifts from the training distribution. For example, snow and yarn differ in opacity and texture; rain streaks and power cables differ in orientation and continuity. This setup enables a meaningful evaluation of zero-shot generalization across varied obstruction types.

We will expand Section 3 to clarify these distinctions and include additional descriptions of seen/unseen differences.

Regarding limitations: Indeed, the limitation of our method under intended usage has already been discussed in Section 6 of the paper. Our approach is specifically designed for small, spatially sparse obstructions and is not suitable for cases where large regions are occluded, as this exceeds the model’s implicit semantic completion capacity. To make this limitation more explicit, we will include additional visual examples and further elaborate on typical failure modes, even within the intended application scope.

R-Q1 Comparison with diffusion-based method: Some important comparisons may have been overlooked. As shown in Figure 12 and Table 4 of Appendix C.3, our method demonstrates superior zero-shot generalization compared to inpainting baselines. This includes Repaint, a representative diffusion-based approach. In addition, in response to Reviewer CuLC’s Question W2, we extended our analysis to include DiffEdit, a diffusion-based image editing method. Together, these results provide comprehensive evidence of the advantages of our method over diffusion-based baselines.

We sincerely appreciate Reviewer CuLC's valuable feedback. Our responses to the weaknesses and questions are listed below.

R-W1 Mask detector design: This concern appears to stem from a misunderstanding. We have already clarified the role and details of the mask detector in Appendix A.1 of the submission. The mask detector is an off-the-shelf component and not central to our contribution. Our primary focus is a novel, distribution-agnostic framework for obstruction removal, which introduces (1) adaptive soft/hard recovery strategies for both opaque and semi-transparent obstructions, and (2) multi-modal prompt integration for precise obstacle representation. The framework is fully plug-and-play and can seamlessly incorporate more advanced mask detectors to further enhance performance without modification.

R-W2 Comparison with diffusion-based method: Indeed, our submission demonstrates that diffusion-based models struggle with irregular and complex obstructions. In Table 4 and Figure 12 of Appendix C.3, we compared our method against Repaint, a representative diffusion-based inpainting approach, and validated the strong zero-shot generalization ability of our framework through both qualitative and quantitative results.

As requested, we further evaluated DiffEdit for unseen obstacle removal (PSNR↑/SSIM↑):

The results show that DiffEdit performs poorly. This stems from its design as an image editing tool, which strictly constrains changes to masked regions and lacks the flexibility to handle irregular-shaped holes. Even when provided with detailed VLM-generated prompts, DiffEdit fails to achieve meaningful completions and often hallucinates entirely new objects. These limitations highlight the inadequacy of diffusion-based editing methods for this task and underscore the effectiveness of our proposed framework.

R-W3 Usage of input strategy like Instruct-Pix2Pix: We conducted a direct comparison between our final strategy and the channel-wise concatenation approach (original + masked image), as in Instruct-Pix2Pix. The results are shown in the table below.

Incorporating the raw image as input significantly degrades performance rather than improving it. This is because our training data includes three types of degradations, and direct access to the original image encourages the model to memorize specific degradation patterns, which harms zero-shot generalization.

While we agree that conventional inpainting methods struggle with semi-transparent obstructions due to the occluded content being partially visible yet unknown, our method addresses this effectively through a soft-masking strategy. Moreover, original image features are already embedded implicitly via CLIP’s encoder, enabling semantic completion without overfitting to training-specific obstructions.

R-W4 Description of the restoration model: The restoration model follows a standard encoder-decoder architecture, a widely adopted design in image processing tasks. As this component is not the core focus of our work (the key is the distribution-agnostic framework), we provided only a brief overview in the main text. However, to ensure methodological transparency and reproducibility, we will include a detailed description of the network architecture in the appendix.

R-W5 More evaluation metrics: As suggested, we incorporated three additional evaluation metrics—LPIPS↓, CLIP Score↑, and User Study (US)↑ (0–1)—to provide a more comprehensive assessment of obstacle removal performance across both seen (Fence, Flare, RainDrop) and unseen (Rain Streak, Snow, Stroke) scenarios. Due to space limitations, we reported results for three representative methods. Even under these expanded metrics, our method consistently outperforms baselines. Full comparisons across all metrics and methods will be included in the final submission.

R-W6 Too small image size: Thank you for your suggestion. We will modify the experimental images to enhance the comparability between methods.