GeoComplete: Geometry-Aware Diffusion for Reference-Driven Image Completion

[W1/Q2: Runtime Comparisons]

The Table A below summarizes the computational cost (using four 24G GPUs) of Paint-by-Example [32], RealFill [30], and our method. Both RealFill and our method are based on per-scene optimization. For a fair comparison, we adopt identical experimental settings, including batch size, number of optimization steps, and number of GPUs. Although our approach introduces slightly higher overhead than RealFill, it achieves significantly better reconstruction quality. Notably, our method reaches promising results within 500 steps (18 minutes), outperforming RealFill even at 2000 steps (50 minutes).

Compared to one-shot models such as Paint-by-Example, per-scene optimization methods generally provide more accurate content restoration but are less suitable for time-sensitive or real-time applications. As shown in Figures 4 and 5, our method produces more faithful results by leveraging scene-specific optimization, whereas Paint-by-Example often fails to transfer fine-grained content from the reference images due to the absence of such tuning.

Table A. Computational Cost and Performance Comparison.

---

[Q1: Clarification on Dynamic Regions]

To identify dynamic regions, we use an LLM (GPT-4o) with the target image and all reference images as input, prompting it to detect any dynamic content. The exact prompt used is provided in our supplementary material (Lines 8–11).

We acknowledge that LangSAM may fail, particularly in highly complex scenes. Such failures can degrade the quality of the generated masks and, consequently, the reconstructed point clouds. However, our method does not heavily rely on the projected point cloud for image completion. Instead, we introduce two key components: Conditional Cloud Masking (CM) and the Joint Self-Attention (JSA) module with masking, which are designed to prevent the model from over-relying on potentially noisy or inaccurate geometric information.

During training, although the projected point cloud may already contain inaccuracies (e.g., due to VGGT), the conditional cloud masking module further masks portions of it. Supervised by ground-truth RGB images, the model learns that not all point cloud information is reliable, allowing it to adaptively interpret and utilize geometric cues under challenging conditions.

In addition, our joint self-attention module with masking (JSA) can potentially mitigate this issue. It establishes explicit token-to-token links between the target and cloud branches. This design prevents noisy cloud tokens from globally influencing all target tokens through cross-attention, especially when noisy tokens are more numerous.

To support our claim, we present the following experiments for verification. We manually select 13 scenes with significant dynamic objects from RealBench to evaluate the robustness of our method. We conduct two experiments: (1) removing the Langsam masks entirely, and (2) simulating segmentation errors by randomly adding 10% additional masked regions to each Langsam mask. The results are shown in Table B. When Langsam fails, our method exhibits only a slight drop of approximately 0.3 dB in PSNR. In contrast, the variant without CM and JSA shows a substantially larger drop of approximately 1.4 dB, highlighting the robustness of our full method to segmentation errors introduced by Langsam.

Table B. PSNR under Different Langsam Mask Conditions on Dynamic Scenes.

[W1/Q1: Robustness for Upstream Modules]

We acknowledge that during both training and inference, the upstream modules (LangSAM and VGGT) may consequently generate inaccurate projected point clouds, as noted in Lines 115–116.

To address this issue, we introduce a conditional cloud masking (CM) strategy (Lines 150–159) that prevents the model from over-relying on potentially noisy or erroneous geometry. During training, although the projected point cloud may already contain inaccuracies from VGGT, we further mask portions of it. With supervision from ground-truth RGB images, the model learns that not all information from point clouds is trustworthy, enabling it to adaptively interpret noisy geometry cues.

In addition, our joint self-attention module with masking (JSA) can potentially mitigate this issue. It establishes explicit token-to-token links between the target and cloud branches. This design prevents noisy cloud tokens from globally influencing all target tokens through cross-attention, especially when noisy tokens are more numerous.

Consequently, at inference time, even when VGGT or LandSAM outputs are inaccurate, our model can still adaptively integrate information from both projected point clouds and RGB signals to produce reliable results.

To support our claim, we present the following experiments for verification.

1. Noisy Point Cloud Simulation: To simulate degraded or poor-quality geometry, we randomly add Gaussian noise to a subset of points in the generated 3D point cloud. The experimental results are shown in Table A. Although the performance of our method slightly degrades with increasing noise levels, it consistently outperforms the baseline method RealFill across all settings.

Table A. PSNR under Different Ratios of Noisy Points in the Generated 3D Point Cloud.

2. Sparse Point Cloud Simulation: We randomly drop a certain ratio of points from the point cloud before projection to simulate sparse projected point clouds. The experimental results are in Table B. It can be observed that when the point cloud becomes sparse, even with only 25% of points dropped, the method without CM and JSA experiences a significant performance drop. In contrast, our full method remains more robust under these conditions.

Table B. PSNR under Different Drop Ratios of Points in the Generated 3D Point Cloud.

3. Ablation Study on LangSAM: We manually select 13 scenes with significant dynamic objects from RealBench to evaluate the robustness of our method. We conduct two experiments: (1) removing the Langsam masks entirely, and (2) simulating segmentation errors by randomly adding 10% additional masked regions to each Langsam mask. The results are shown in Table C below. When Langsam fails, our method exhibits only a slight drop of approximately 0.3 dB in PSNR. In contrast, the variant without CM and JSA shows a substantially larger drop of approximately 1.4 dB, highlighting the robustness of our full method to segmentation errors introduced by Langsam.

Table C. PSNR under Various Langsam Mask Conditions for Scenes with Challenging Dynamic Objects.

---

[W2/Q2: Computational Cost and Potential Acceleration Strategies]

The Table D below summarizes the computational cost (using four 24G GPUs) of Paint-by-Example [32], RealFill [30], and our method. Both RealFill and our method are based on per-scene optimization. For a fair comparison, we adopt identical experimental settings, including batch size, number of optimization steps, and number of GPUs. Although our approach introduces slightly higher overhead than RealFill, it achieves significantly better reconstruction quality. Notably, our method reaches promising results within 500 steps (18 mins), outperforming RealFill even at 2000 steps (50 mins).

Compared to one-shot models such as Paint-by-Example, per-scene optimization methods generally provide more accurate content restoration but are less suitable for time-sensitive or real-time applications. As shown in Figures 4 and 5, our method produces more faithful results, while Paint-by-Example often fails to preserve fine-grained content from the reference images.

To explore potential acceleration strategies, we first identify that the primary computational bottleneck in our framework lies in the 2,000-step per-scene fine-tuning process. One potential solution is to pre-train the LoRA parameters of the diffusion model on a large-scale, task-specific dataset. This would serve as a strong initialization for subsequent per-scene adaptation, thereby significantly reducing the number of required optimization steps while preserving the quality of generated results. We consider this an important direction for future work.

Table D. Computational Cost and Performance Comparison.

---

[W3: Dynamic and Complex Scenes]

We acknowledge that LangSAM may struggle in dynamic or cluttered environments, especially when accurate or complete textual prompts are unavailable. These limitations can affect the quality of the generated masks and, consequently, the reconstructed point clouds. However, our method treats the 3D point cloud as an auxiliary input rather than a rigid intermediate representation. While the geometric cues from the point cloud may be imperfect, our model selectively leverages reliable information through a conditional cloud masking mechanism and a joint self-attention module. This enables the model to focus on trustworthy regions while ignoring potentially misleading geometry.

Moreover, we would like to emphasize that the benchmark datasets RealBench and QualBench encompass a wide range of complex scenes, including significant dynamic content, multiple moving objects, and varying lighting conditions. The corresponding experimental results are presented in the Table C. These results demonstrate the strong robustness and generalization ability of our method, even in dynamic and complex scenarios.

---

[W4/Q3.1: Potential Baselines]

Following the suggestion, we evaluate existing reference-guided image generation methods, including OmniGen, Step1x-Edit, and Bagel. We initially follow the official instructions to run these baseline models. For scenes where the models fail to perform adequately, we employ ChatGPT to generate prompts and manually refine them as needed. However, these methods fail to handle all scenes. In some cases, the restored results become completely white or visually meaningless. In summary, only 28 scenes from OmniGen, 13 scenes from Step1X-Edit, and 28 scenes from Bagel produce valid outputs. PSNR and SSIM are computed only on these successfully restored results. The results suggest that under the reference-based image completion setting, existing reference-guided image generation methods still perform suboptimally.

---

[Q3.2: Rationale of Baselines]

Our baseline methods are primarily selected from the RealFill paper [30], which serves as a strong and recent benchmark in this field. Specifically, we include prompt-based approaches such as SD Inpaint [27] and Generative Fill [2], as well as reference-based completion methods like Paint-by-Example [32], TransFill [37], and RealFill [30]. Notably, both TransFill and RealFill are trained on dedicated image completion datasets, making them particularly suitable for comparison. Overall, these baselines constitute a representative and diverse set of state-of-the-art techniques, enabling fair and meaningful evaluation.

[W1: One-shot Geometry vs. Multiple Homographies]

The limitations of multiple homographies have been previously identified and evidenced by RealFill [30], which demonstrates catastrophic failure when the scene’s structure cannot be accurately estimated. In contrast, extensive experiments in VGGT validate the superiority of one-shot geometry estimation. Due to this year’s rebuttal policy, we are unable to include additional comparison figures. However, some TransFill results are already available in the RealFill paper [30]. We also commit to including direct qualitative comparisons with TransFill in the revised version.

---

[W2: Clarification of Workflow]

Initially, we process all reference images and the target image using VGGT and Langsam to obtain the point cloud.

During training, given a reference image $I_{ref}$ and its corresponding projected point cloud $I_{cloud}$, we use our conditional reference masking to mask the reference image, obtaining the augmented reference image $I_{aug-ref} \in \mathbb{R}^{3 \times H \times W}$ and the mask image $I_{mask} \in \mathbb{R}^{1 \times H \times W}$. We also use our conditional cloud masking to augment the projected point cloud, resulting in $I_{aug-cloud} \in \mathbb{R}^{3 \times H \times W}$. We then encode these images into the latent space using a VAE. This results in a latent reference image $I_{ref}^{latent}$ (used as the ground truth), a latent masked reference $I_{aug-ref}^{latent}$, and a latent projected point cloud $I_{aug-cloud}^{latent}$, all in $\mathbb{R}^{4 \times h \times w}$, where $h = H / 8$ and $w = W / 8$. The mask is also downsampled to $I_{mask}^{latent} \in \mathbb{R}^{1 \times h \times w}$.

To fine-tune the diffusion model, we add noise to the ground truth latent $I_{ref}^{latent}$ to obtain a noisy latent $I_{noisy}^{latent}$. The input to the target branch is the concatenation of $I_{noisy}^{latent}$, $I_{mask}^{latent}$, and $I_{aug-ref}^{latent}$, resulting in a tensor of shape $\mathbb{R}^{9 \times h \times w}$. Similarly, the input to the cloud branch is the concatenation of $I_{noisy}^{latent}$, $I_{mask}^{latent}$, and $I_{aug-cloud}^{latent}$, also in $\mathbb{R}^{9 \times h \times w}$. The objective is to estimate the added noise, which has shape $\mathbb{R}^{4 \times h \times w}$.

During inference, given the target image $I_{tar}$, its corresponding projected point cloud $I_{cloud}$, and the mask image $I_{mask}$, we directly process them into the latent space. This results in a latent target $I_{tar}^{latent}$ and a latent projected point cloud $I_{cloud}^{latent}$. The mask is also downsampled to $I_{mask}^{latent}$. We initialize $I_{noisy}^{latent}$ using standard Gaussian noise. Then, we concatenate the corresponding latent tensors to construct the inputs for the target and cloud branches. After an iterative denoising process and using the VAE decoder, we obtain the final output.

---

[W3: Comparison with Recent Work]

Following the suggestion, we provide a discussion comparing our method with Geometric-aware 3D Scene Inpainter [R1]. This method conditions a diffusion model on multi-view images and scene geometry to reconstruct accurate 3D structure, which in turn supports high-quality inpainting. However, it assumes that all multi-view inputs share the same underlying scene geometry, which limits its applicability under challenging conditions such as dynamic objects or varying lighting. In contrast, our approach uses 3D geometry only as auxiliary guidance, rather than a reconstruction target. By leveraging the generative capabilities of the diffusion model, our method focuses on completing missing image regions with greater flexibility and robustness, even when the geometric cues are imperfect.

[R1] Salimi, Ahmad, et al. "Geometry-aware diffusion models for multiview scene inpainting." arXiv preprint arXiv:2502.13335 (2025).

---

[Q1: Clarification of TransFill [37]]

The quantitative results of TransFill are directly taken from the RealFill [30] paper, and we follow the same evaluation setting to ensure a fair comparison. Since the official code of TransFill is not publicly available, we, like the RealFill team, contacted the authors to obtain the qualitative results.

---

[Q2: Robustness of GeoDiff]

Following the suggestions, we conduct ablation studies on the scenario with reduced overlap. As the point cloud covers less of the image, the number of points decreases, leading to reduced pixel information in the projected point cloud. To simulate this scenario, we perform the following experiments:

1. Pixel Drop in the Projected Point Cloud: We randomly drop pixels within the target regions of the projected point cloud, directly reducing the available point cloud information. The experimental results are presented in Table A.

2. Region Drop in the Projected Point Cloud: We randomly remove multiple rectangular blocks within the target regions of the projected point cloud until the total number of removed points reaches a fixed ratio of the points in these regions. This block-based removal simulates region-level information loss. The experimental results are presented in Table B.

3. Sparse Point Cloud Simulation: We randomly drop a certain ratio of points from the point cloud before projection to simulate sparse projected point clouds. The experimental results are in Table C.

It can be observed that our method remains more robust under these conditions, consistently outperforming the baseline method RealFill. This robustness arises from the fact that our method does not heavily rely on the projected point cloud for image completion. Instead, we introduce two key components: Conditional Cloud Masking (CM) and the Joint Self-Attention (JSA) module with masking, which are designed to prevent the model from over-relying on potentially noisy or inaccurate geometric information.

During training, although the projected point cloud may already contain inaccuracies (e.g., due to VGGT), the conditional cloud masking module further masks portions of it. Supervised by ground-truth RGB images, the model learns that not all point cloud information is reliable, allowing it to adaptively interpret and utilize geometric cues under challenging conditions.

In addition, our joint self-attention module with masking (JSA) can potentially mitigate this issue. It establishes explicit token-to-token links between the target and cloud branches. This design prevents noisy cloud tokens from globally influencing all target tokens through cross-attention, especially when noisy tokens are more numerous.

Furthermore, in Tables A, B, and C, we present the ablation studies for our CM and JSA modules. When the point cloud or projected pixels become sparse, even with only 25% of the points dropped, the variant without CM and JSA exhibits a significant performance drop. In contrast, our full method remains robust under these conditions.

Table A. PSNR under Different Drop Ratios of Pixels in the Projected Point Cloud.

Table B. PSNR under Different Drop Ratios of Regions in the Projected Point Cloud.

Table C. PSNR under Different Drop Ratios of Points in the Generated 3D Point Cloud.

[W1: Literature Review]

Thank you for highlighting these relevant works. Below, we clarify the differences between our method and prior approaches:

• GeoFill: We would like to clarify that our manuscript already discusses GeoFill [36] in lines 24–30 and 75–79. In particular, as noted in the RealFill [30] paper, GeoFill heavily relies on accurate 3D scene representations, making it sensitive to reconstruction errors. Moreover, RealFill highlights several limitations of GeoFill, including its difficulty in handling complex scene geometry, appearance changes, and scene deformation. In contrast, our method treats the 3D point cloud as an auxiliary input rather than a rigid intermediate representation. While the geometry cues from the point cloud may be imperfect, our model selectively leverages reliable information through a conditional cloud masking mechanism and a joint self-attention module. This design enhances robustness and enables more accurate and flexible image completion across diverse scenarios.

• Geometric-aware 3D Scene Inpainter: This method conditions a diffusion model on multi-view images and scene geometry to reconstruct accurate 3D structure, which in turn supports high-quality inpainting. However, it assumes that all multi-view inputs share the same underlying scene geometry, which limits its applicability under challenging conditions such as dynamic objects or varying lighting. In contrast, our approach uses 3D geometry only as auxiliary guidance, rather than a reconstruction target. By leveraging the generative capabilities of the diffusion model, our method focuses on completing missing image regions with greater flexibility and robustness, even when the geometric cues are imperfect.

• OccludeNeRF: OccludeNeRF distills knowledge from a pre-trained diffusion model for 3D scene inpainting using Score Distillation Sampling. In contrast, our method directly conditions the diffusion model on projected point clouds and target cues to complete missing regions. While both approaches incorporate geometry within a diffusion framework, the underlying mechanisms and task objectives are fundamentally different.

We note that Geometry-aware 3D Scene Inpainter (2025.02) and OccludeNeRF (2025.04) are very recent arXiv preprints released around the same time as our submission. We will include the above discussion to clarify the distinctions between these works and ours in the revised version.

---

[W2: QualBench & RealBench]

We would like to clarify that both QualBench and RealBench are not newly created datasets, but directly derived from the RealFill dataset [30], covering all evaluation data used in RealFill. We follow the same evaluation settings, including all datasets and metrics, as established in RealFill to ensure fair and consistent comparison.

---

[Q1: Contribution]

Beyond integrating diffusion with geometry, our key innovations lie in the dual-branch diffusion architecture and the target-aware masking strategy, both specifically designed for reference-based image completion.

Our dual-branch diffusion comprises a target branch and a cloud branch. The target branch conditions the diffusion model on the masked image to generate missing content, while the cloud branch conditions it on the projected point cloud to provide geometric cues. A joint self-attention module is introduced to fuse information from both branches.

This design enables the model to adaptively integrate visual and geometric signals. However, since most regions in the target image are masked, the resulting latent features often lack meaningful content. While masked tokens can attend to the cloud branch, they may still struggle to extract effective guidance.

To address this, we modify the attention mask to explicitly link each masked token in the target branch to its corresponding token in the cloud branch, ensuring that the target branch receives direct geometric cues even in the absence of visual information.

Our target-aware masking guides the model to focus on useful and non-redundant reference cues. It consists of two components: conditional reference masking and conditional cloud masking. Unlike RealFill [30], which randomly masks reference images, conditional reference masking selectively masks informative regions to encourage the model to learn from content that complements the target view. Conversely, conditional cloud masking randomly applies white padding to the projected point maps, masking redundant regions while preserving the informative ones, thereby guiding the model to effectively utilize geometric cues. Additionally, masking the projected point cloud informs the model that not all information from the point cloud is trustworthy. This prevents the model from over-relying on potentially noisy, sparse, or inaccurate geometry, thereby enhancing its robustness.

All these task-specific designs enable our method to selectively leverage geometric cues, even when they are imperfect. This distinguishes our approach from prior work, which often relies on highly accurate 3D reconstructions or treats geometry as a rigid intermediate representation.

---

[Q2: Robustness of GeoDiff]

We agree that during both training and inference, the output of VGGT may contain inaccuracies, as noted in Lines 115–116.

To address this issue, we introduce a conditional cloud masking (CM) strategy (Lines 150–159) that prevents the model from over-relying on potentially noisy or erroneous geometry. During training, although the projected point cloud may already contain inaccuracies from VGGT, we further mask portions of it. With supervision from ground-truth RGB images, the model learns that not all information from point clouds is trustworthy, enabling it to adaptively interpret noisy geometry cues.

In addition, our joint self-attention module with masking (JSA) can potentially mitigate this issue. It establishes explicit token-to-token links between the target and cloud branches. This design prevents noisy cloud tokens from globally influencing all target tokens through cross-attention, especially when noisy tokens are more numerous.

Consequently, at inference time, even when VGGT outputs are inaccurate, our model can still adaptively integrate information from both projected point clouds and RGB signals to produce reliable results.

To support our claim, we present the following experiments for verification.

1. Noisy Point Cloud Simulation: To simulate degraded or poor-quality geometry, we randomly add Gaussian noise to a subset of points in the generated 3D point cloud. The experimental results are shown in Table A. Although the performance of our method slightly degrades with increasing noise levels, it consistently outperforms the baseline method RealFill across all settings.

Table A. PSNR under Different Ratios of Noisy Points in the Generated 3D Point Cloud.

2. Sparse Point Cloud Simulation: We randomly drop a certain ratio of points from the point cloud before projection to simulate sparse projected point clouds. The experimental results are in Table B. It can be observed that when the point cloud becomes sparse, even with only 25% of points dropped, the method without CM and JSA experiences a significant performance drop. In contrast, our full method remains more robust under these conditions.

Table B. PSNR under Different Drop Ratios of Points in the Generated 3D Point Cloud.

3. Ablation Study on LangSAM: We manually select 13 scenes with significant dynamic objects from RealBench to evaluate the robustness of our method. We conduct two experiments: (1) removing the Langsam masks entirely, and (2) simulating segmentation errors by randomly adding 10% additional masked regions to each Langsam mask. The results are shown below. When Langsam fails, our method exhibits only a slight drop of approximately 0.3 dB in PSNR. In contrast, the variant without CM and JSA shows a substantially larger drop of approximately 1.4 dB, highlighting the robustness of our full method to segmentation errors introduced by Langsam.

Table C. PSNR under Various Langsam Mask Conditions for Scenes with Challenging Dynamic Objects.

---

[Q3: Details of Baseline Methods]

For SD Inpaint [27] and Generative Fill [2], we follow the instructions from RealFill, which generate long descriptions for each scene with the help of ChatGPT. For RealFill [30], we follow their official setting by fixing the text prompt to a sentence containing a rare token, i.e., “a photo of [V]”. For a fair comparison, our method also adopts this setting.

Dear Reviewer 4jUZ,

Thank you for your earlier comments! They were insightful and have helped us strengthen the paper. I just wanted to check if our rebuttal has fully addressed your points. If there are any remaining concerns or areas that would benefit from further clarification or additional supporting results, we would be happy to provide them. We greatly appreciate your feedback and hope that our clarifications have resolved your concerns.

Sincerely,

Authors

Dear Reviewer 4jUZ

Please engage in the discussion with the authors and other reviewers as soon as possible.

Thank you.

Best,

AC