MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generation

We thank all your valuable comments of our paper. The detailed responses are shown below.

W1: Assumption of Perfect Object Alignment

Response: Thank you for your valuable suggestions. We acknowledge that our method requires an initial 3D alignment of the object. For example, we can align the bottom surface of the object with the plane of the scene.

However, we clarify that the core contribution of our paper is illumination harmonization and consistent ligthing and shadow generation, not reconstruting the background scene or aligning the composited object, as described in Section 3.1.

While diffusion-based models can often blend a misaligned 2D object into a single image, this generative flexibility becomes a critical weakness in a multi-view 3D context. Running a diffusion model independently on each view would almost certainly produce inconsistent object geometry, lighting, and shadows, shattering the 3D illusion. Our method is explicitly designed to solve this multi-view consistency problem, which necessitates a consistent 3D position for the object as a starting point.

W2: Impact of Sparse Geometry or View Coverage

Response: This point relates to the scope of our work. Our method assumes a pre-existing, high-quality 3D representation of the background scene (e.g., a pre-trained 3DGS). Consequently, the depth maps required by our model are rendered from this representation, rather than being generated from sparse views. Since 3D representation is known, the input views can be picked, thus we did not analyze the varying view coverage or the sparse geometry in the main text.

To explore this boundary condition, our supplement (Section A.7) includes an experiment where we use VGGT to estimate geometry from unposed images. The results show that our adjustments to the Gaussian reconstruction process effectively mitigated the impact of noisy input geometry on the harmonization quality.

W3: Motivation for Hilbert Curve vs. Standard Transforme

Response: The motivation is to preserve spatial locality. A standard transformer requires the input sequence order to be meaningful. For an unordered set of 3D points like Gaussians, an arbitrary ordering provides no useful spatial bias for the attention mechanism. The Hilbert curve imposes a meaningful order, ensuring that Gaussians that are close in 3D space are also close in the 1D sequence, which aids learning.

For the "w/o Hilbert transform" ablation in Table 3, the architectural change is in the input preparation. Instead of Hilbert ordering, we simply take the list of N Gaussians and perform a naive reshape into a pseudo-image. This is equivalent to ordering them by their arbitrary initial index. The resulting performance drop (e.g., PSNR in the 3D task falls from 30.29 to 28.99) provides clear empirical evidence for the Hilbert curve's benefit. We will add this clarification to Section 4.4.

W4: Evaluation Metrics and User Study

Response: We agree that pixel-wise metrics have limitations for evaluating generative models. We use them because they are the most established quantitative standards for image restoration and translation tasks. To provide a more complete picture, we supplement these metrics with two crucial elements:

1. Extensive Qualitative Results: We provide numerous visual comparisons in Figure 47 and Figure 46 in the supplementary material., which allow for direct perceptual judgment of quality.
2. Formal User Study: The user study was conducted via an online questionnaire and we collected 374 valid responses from human participants. The study presented participants with side-by-side comparisons and asked them to choose which result demonstrated more realistic and consistent lighting and shadows. The two questions corresponded to the single-view and multi-view compositing tasks, respectively, with the results summarized in the pie charts on Figure 3 in the supplementary material.

W5: Training/Test Datasets and Zero-Shot Evaluation

Response: Thank you for the reminder, and we will make it explicit in Section 4.1.

• Training Data: The model was trained exclusively on the "simple synthetic scenes" from our proposed DTC-MultiLight dataset except 50 scenes for evaluation.
• In-Domain Testing: A held-out split of 50 "simple synthetic scenes" from DTC-MultiLight was used for in-domain evaluation, as shown in Figure 4&5 and Figure 5 in the supplementary material.
• Zero-Shot Testing: All other evaluations were conducted in a zero-shot setting on entirely unseen datasets to test generalization. This includes:
  • our "complex synthetic scenes" (Figure 4&5 and Figure 5 in the supplementary material).
  • our "real captured scenes" (Figure 6).
  • FOSCom and Objects With Lighting (OWL) dataset (Figure 4 and Figure 4 in the supplementary material).

Thank you for your detailed feedback.

W1: We understand your concerns. Our method does indeed require the assumption of a pre-aligned composite in the 2D application, which is a limitation. While earlier diffusion-based methods could adjust object-scene alignment through their generative capabilities, they also modified the object's shape, texture, and size. Our feed-forward approach, however, aims to efficiently and stably generate highlights and shadows for inserted objects while keeping them unchanged, which is more user-friendly for inserting objects in AR scenes.

W3: We agree that our ablation study on the Hilbert curve was insufficient. Following your suggestion, we will use a pre-trained point transformer to extract positional and color features from the Gaussian primitives, then merge them with the image features as input to the network. We will provide the results of this comparative experiment before the end of the discussion period.

We thank your recognition and suggestions for our paper. The detailed responses are shown below.

Q1: Visualization of Video Results

Response: Due to time constraints, we were unable to produce the final videos for the initial submission. We have since rendered several high-quality videos showcasing multi-object composition with smooth camera motion in complex scenes. These will be contained in the camera-ready version of the paper.

Q2: Details of Hilbert Curve Mapping

Response: Thank you for these insightful questions. We will add a more detailed explanation of the mapping process to.

1. Space Occupancy and Collisions: The purpose of constructing the Hilbert curve is to preserve the spatial information of sparse Gaussians and we do not explicitly prevent multiple Gaussians from mapping to the same discrete point on the curve. Our process is as follows: we build a KD-Tree on the Hilbert curve's points for efficient lookup. Each Gaussian is then mapped to its nearest neighbor on the curve. The final 1D sequence is ordered first by position along the curve. If a "collision" occurs (multiple Gaussians map to the same point), their relative order is resolved using a stable sort based on their original index in the input list.
2. Empty Curve Points: Curve points that have no assigned Gaussians are simply ignored. The final sequence passed to the transformer consists only of the ordered Gaussian features, with no gaps.

We thank all your valuable comments of our paper. The detailed responses are shown below.

W1: Motivation for Individual Components

Response: We clear the motivation for two key components below.

• Swin Transformer: The Swin Transformer introduces shifted-window attention, achieving linear computational complexity while maintaining hardware efficiency through its hierarchical pyramid structure. It demonstrates superior performance over the original transformer across multiple vision tasks (+2.7 box AP on COCO detection and +3.2 mIoU on ADE20K segmentation), while also requiring fewer FLOPS (-45.5%) and parameters (-35.8%) at the same input resolution. Consequently, it serves as the backbone for several widely-used models like Grounded DINO and DINO-X. In our work, we adopt the Swin Transformer as the backbone for both our 2D and 3D object compositing models. (Quantitative comparison is from the paper "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows")
• Hilbert Curve Mapping: Inspired by recent advancements in processing point clouds with transformers, like Point Transformer V3, we utilize the Hilbert curve maximally preserves spatial locality, which means points that are close in 3D remain closing in the 1D sequence. Then, the ordered sequence is then folded into a 2D grid, allowing our transformer to effectively process the geometric structure of the scene. The ablation study (e.g. PSNR in the 3D task falls from 30.29 to 28.99) provides clear empirical evidence for the Hilbert curve's benefit.

W2: Concerns about Gaussian Initialization

Response: We appreciate the reviewer's concern regarding efficiency. We clarify two key points:

1. The Gaussian initialization is a one-time, per-scene preprocessing step, not a per-inference overhead. We clarify that the one-minute training for the Gaussians was chosen to achieve the highest reconstruction quality and we can trade off this training time for quality. To demonstrate this, we conducte a comparison experiment below. Once this 3D scene representation is constructed, our framework can perform harmonization for any number of viewpoints or even for newly inserted objects at a rapid speed (e.g., 0.07 seconds per frame). This initial cost should be viewed as analogous to loading or preparing a 3D asset, after which real-time interaction becomes possible.
2. Our method is fully compatible with the latest advancements in feed-forward 3D reconstruction. Recent geometry foundation models can generate 3DGS representations directly from sparse inputs in a single feed-forward pass. To demonstrate this, we conduct experiments using AnySplat, which generates a 3DGS scene from input frames without per-scene optimization, drastically reducing the initial setup time.

The results reveal a clear trade-off between reconstruction paradigm, speed, and scene complexity. AnySplat achieves the fastest reconstruction time at under one second and delivers quality nearly on par with the highest-quality, per-scene optimized 3DGS in complex, forward-facing scenes. However, its performance significantly degrades in simple, surround-view scenes, likely due to the inherent challenge of jointly estimating camera poses. Meanwhile, by reducing training iterations of the vanilla 3DGS to 1k, it produces comparable results in just a few seconds, offering a practical compromise between the instant inference of AnySplat and the lengthy optimization of a full 10k-iteration run.

W3: Training Times and Model Complexity

Response: We acknowledge that the training time (15+3=18 days on 16 A100 GPUs) is quite long. However, this is standard for training large-scale, feed-forward vision models from scratch on a new, large dataset. For context, comparable large-scale models like VGGT requires 9 days of training on 64 A100 GPUs and Fast3r requires 6.13 days of training on 128 A100 GPUs. In terms of total GPU-hours, our training budget is in a similar range.

Regarding complexity, while our pipeline involves two stages, the core transformer architecture is nearly identical for both the 2D and 3D models, as illustrated in Figure 2 of our supplementary material. We propose a unified architecture for both 2D and 3D tasks with different data representations.

W4&Q3: Image Resolution

Response: Our model supports inference at a resolution of 2K, as shown in Figure 7. Furthermore, we are also conducting training with higher resolution (512x512) and will release the pretrained models with different resolutions.

W5: Text Clarity and Supported Inputs

Response: We thank the reviewer for these actionable suggestions. We will incorporate all proposed textual edits, including adding a definition for object compositing in the abstract and correcting the typo in Section 3.1. We will also clarify the supported input modalities and the training data details.

• Supported Inputs:
  1. Single-view image: For a single view input, only the 2D Object Compositing model is used.
  2. Multi-view images with known poses: This is the primary use case for our full pipeline, where the 3D model leverages the 3DGS representation to enforce multi-view consistency.
  3. Multi-view images without known poses: This is also supported as an extension. As demonstrated in our supplementary material (Section A.7), camera poses can first be estimated with VGGT, after which our pipeline can proceed.

The training dataset we used is simple synthetic scenes from the DTC-MultiLight dataset we proposed. The complex synthetic scenes and real captured scenes are used for out-of-domain evaluating.

W6&Q1: Generalization to Arbitrary Scenes

Response: We apologize if this was unclear. While we train models on tabletop scenes (simple synthetic scenes), our evaluation is deliberately performed on a much wider variety of out-of-domain content to test generalization:

• Complex Indoor Scenes: Figure 4 and Figure 4&5 in the supplementary material show strong performance on the Objects With Lighting (OWL) dataset and complex synthetic scenes of DTC-MultiLight dataset, which features complex indoor environments.
• Diverse Outdoor Scenes: Figure 4 in the supplementary material shows qualitative results on the FOSCom dataset, which includes varied scenes such as outdoor streets.
• Real-World Captures: Figure 6 demonstrates robust performance on casually captured real-world scenes that differ significantly from our synthetic training data.

Q2: Clarification of "Iterative Bias"

Response: I'm certain it refers to exposure bias. While these models produce high-quality textures, their stochastic and iterative nature can lead to results that are visually plausible but physically inconsistent, especially for precise effects like shadows and highlights.

Q4: Use of Pre-trained Components

Response: Thank you for your suggestion. We agree that using more available pre-trained components is a method to accelerate training like VGGT. However, Our network backbone is a specific architecture that takes a multi-channel input tensor concatenating the composite image, background, and depth map. Standard pre-trained weights from datasets like ImageNet are trained on 3-channel RGB images and would not be directly applicable. Meanwhile, the hyperparameters of the network architecture of our model are also different.

Q5: Perceptual Loss Details

Response: The perceptual loss Lp used during training is a VGG19-based feature loss, not the LPIPS metric. While both are derived from VGG features, LPIPS involves an additional learned linear layer on top of the features to better align with human perceptual judgments. Our training loss uses a standard L2 distance in the VGG feature space, a common and effective practice for image-to-image tasks.

Q6: Monocular Depth Input

Response: For all our experiments on public datasets like FOSCom and OWL where ground-truth depth was unavailable, the depth maps were generated by Depth Anything v2.

Q7: User Study Details

Response: The user study was conducted via an online questionnaire and we collected 374 valid responses from human participants. The study presented participants with side-by-side comparisons and asked them to choose which result demonstrated more realistic and consistent lighting and shadows. The two questions corresponded to the single-view and multi-view compositing tasks, respectively, with the results summarized in the pie charts on Figure 3 in the supplementary material.

We appreciate your valuable suggestions and will revise in the final version:

• We will add a summary of the motivations to the beginning of the method section.
• Our model was trained from scratch, not fine-tuned. We will add the justification for our training effort to the implementation details section.
• We also agree that user studies are an important indicator, so we will incorporate the results into the main text and elaborate on the specific details in the supplementary materials.

We thank your appreciations and suggestions for our paper. The detailed responses are shown below.

W1: Discussion of Failure Cases

Response: Thanks for your excellent suggestion. We agree that a transparent discussion of limitations is essential. Due to space constraints, the limitation is presented in Section A.8 of the supplementary material. We also summarize some major failure cases as follows:

• Shadow Precision: Our model can learn the general direction of highlights and shadows from the scene, but sometimes it fails to generate accurate and hard-edged shadows, as shown in the last column of Figure 5. The challenge lies in the fact that the information in the picture cannot determine the position of the light source, so creating sharp and accurate shadows is difficult.
• Reflective Surfaces: Our model, trained primarily on diffuse materials, struggles to generate physically accurate reflections of composited objects on specular or mirrored surfaces. The challenge lies in capturing the characteristics of the materials from the images, and reflections and shadows are often easily confused.

Q1: Training Data for Luminous Objects

Response: Thank you for highlighting this. The original training dataset did not contain a large number of luminous objects. To specifically probe our model's ability to generalize to complex light transport phenomena, we curated a targeted dataset (about 1/3 of the scenes in the original dataset) where some objects were assigned emissive material properties. The strong performance on this task, as shown in Figure 7, even with relatively limited exposure during training, demonstrates the powerful generalization capability of our framework. We will clarify this in Section 4.3.

Thank you for your clarification. After reading the other reviews and seeing the additional discussion necessary on the initial 3D alignment, I would slightly lower my score to 5.