DRIP: Unleashing Diffusion Priors for Joint Foreground and Alpha Prediction in Image Matting

W1: Absence of Baselines for Foreground Color Estimation

Thank you very much for highlighting the need for a proper baseline in foreground color estimation. We agree that this aspect warrants further discussion and comparison.

1. Related Work Discussion In the related work section, we initially focused on methods that simultaneously predict alpha and foreground RGB, as well as those that output alpha followed by post-processing to obtain foreground RGB (L79-83). However, we acknowledge that the two works mentioned by the reviewer that adopt a sequential approach where foreground colors are predicted before alpha. This is indeed a significant method, and we will include a detailed discussion of these works in the revised related work section.
2. Evaluation with Baselines To address the reviewer's concern regarding the absence of a proper baseline, we have added an evaluation of the Learning-based Sampling (LBS) method as a foreground estimation technique. The dataset and metrics for foreground estimation are consistent with those in the paper (L206-212, L245-249). The qualitative results are presented in Figure.A of the rebuttal PDF, and the quantitative results are provided in the table below. The evaluation shows that our method, when leveraging the powerful priors from latent diffusion models (LDM), significantly outperforms other methods, achieving state-of-the-art (SOTA) performance in foreground color estimation.

   Method	SAD	MSE
   LBS	49.7	8.6
   Ours	34.1	4.3

W2: Technical Novelty

Thank you for your valuable feedback. Our approach addresses significant challenges in the ill-posed problem of image matting by incorporating prior knowledge through latent diffusion models. Here are our main technical contributions (L64-69):

1. Transforming Generative Models for Matting: We modified LDMs, which are generative models, into discriminative models for matting by modeling the task as conditional generation, making this the first LDM-driven matting method.
2. Joint Prediction of Foreground and Alpha: To bridge the gap between LDM's RGB output and our RGBA output, we introduced a switcher and cross-domain attention mechanism. These facilitate mutual information exchange, ensuring high consistency and accurate joint predictions.
3. Mitigating VAE Reconstruction Errors: We proposed a latent transparency decoder to address the VAE's inherent reconstruction error, aligning RGBA predictions with the input image to preserve necessary details for high-quality matting.

Q1: Different Trimap Shapes

We acknowledge the importance of trimap accuracy and have ensured our model's robustness. During training, we use data augmentation (L227-234), varying the size and number of dilation and erosion kernels to enhance robustness to different trimap shapes. For the Composition-1k and AIM-500 datasets, predefined trimaps facilitate method comparisons.

To validate our model's robustness, we conducted additional experiments with different trimap shapes, adding two new trimaps with medium and large shapes in the Composition-1k dataset, using 10 and 20 iterations with kernel sizes of 50 for dilation and erosion.

The qualitative results of these tests are shown in Figure B of the rebuttal PDF. As expected, the larger trimap includes more unknown regions, increasing the prediction challenge. However, our model still performs well. The quantitative results in the table below show that although alpha prediction accuracy decreases with larger unknown regions, our model maintains high precision, demonstrating its robustness.

Q2: Video Matting

We supplemented our evaluation with tests on the VideoMatte240K dataset [1] using the model trained on Composition-1k. For each frame, we generated a trimap with erosion and dilation (kernel size 10, iterated five times) and conducted tests at a resolution of 512×288. Evaluation metrics included MSE(L242-244) and dtSSD for temporal coherence.

The quantitative results are presented below. We included results from a human video matting method, MODNet[2]. Since our model uses trimaps for each frame, our results generally outperform those of human video matting methods.

The qualitative results, shown in Figure C of the rebuttal PDF, indicate that our model maintains good performance on unseen video test data. This also demonstrates the generalization capability of our model.

[1] Real-time high-resolution background matting. CVPR’21

[2] Modnet: Real-time trimap-free portrait matting via objective decomposition. AAAI’22

Q3: End-to-end Training

Thank you for the insightful suggestion. End-to-end training with unfrozen encoder weights presents several challenges. Firstly, optimizing the latent space while treating it as an optimization target can destabilize the training process. Additionally, training both the VAE encoder and decoder simultaneously increases memory usage significantly.

This approach would also require balancing three types of loss functions: latent space constraints, pixel space constraints, and latent space regularization, necessitating an extensive hyperparameter search and increasing computational costs exponentially.

While end-to-end training could offer benefits, the costs and complexities currently outweigh those of our two-stage approach. Implementing this during the rebuttal period is impractical. However, we appreciate the suggestion and acknowledge its potential for future work.

W1: Computational Complexity

Thank you for your valuable comments regarding the computational complexity of our model. We understand the importance of addressing computational demands, particularly for real-time or resource-constrained applications. In the limitation section of our paper, we discussed the issues related to model complexity and deployment, specifically noting that the use of latent diffusion models substantially increases the architectural complexity of our approach (L323-L326). Below, we provide detailed information about the computational complexity:

The table above shows the computation time and memory usage for processing images of various resolutions on a standard GPU (NVIDIA RTX A6000) using float16 precision. While the latent diffusion models increase the computational load, the performance remains within acceptable limits for many practical applications.

W2: Real World Scenarios

Thank you for raising this important point. In the original paper, we demonstrated our model’s generalizability by training on synthetic datasets and evaluating on unseen real-world datasets, where it achieved outstanding results (L283-284). To further validate our model’s robustness, we have supplemented our evaluation with tests on the VideoMatte240K dataset [1], also using the model trained on the Composition-1k dataset. The quantitative results are as follows:

Additionally, the qualitative results are provided in Figure C of the rebuttal PDF. These results show that our model performs well on unseen video test data, further demonstrating its generalization capability.

[1] Real-time high-resolution background matting. CVPR’21

W3: Biases From Pretrained Model

Thank you for raising this important concern. As mentioned in the limitations section of our paper, our approach indeed relies on pre-trained Latent Diffusion Models (LDMs), which may carry biases from their training data (L326-331). Regarding rare objects that are not well-represented by the original SD model, this is an area that requires further research, and currently, there is no suitable dataset for evaluation. However, it is undeniable that the training dataset used for LDMs, such as LAION-5B, is extremely large. This extensive dataset is a significant reason why our method demonstrates better generalization capabilities compared to other approaches (L266-284).

W4: Textual Input Feature

Thank you for raising this question. Our approach indeed builds on the Stable Diffusion (SD) model (L147-149), but we have customized it to focus specifically on the image matting task, intentionally omitting the textual input feature of the SD model. This decision was made to simplify the model's architecture and optimize it for the specific requirements of image matting.

By doing so, we ensure that the cross-attention layers fully capture the intricate details and dependencies in the visual input without being influenced by text-based guidance. Our state-of-the-art (SOTA) results across multiple datasets (L266-284) demonstrate that omitting the textual input does not lead to a degradation in performance.

W5: Classifier Free Guidance

In our paper, we clarified that our approach uses an empty text input (L222-223). Classifier-free guidance is typically employed to enhance the ability of a model to follow textual descriptions by combining the predictions from a given text prompt with those from an empty text prompt. However, in our method, the text input is deliberately left empty, which means that CFG is not utilized.

W6: Fine-Tuning Based on Layer Diffusion

We appreciate the suggestion to consider fine-tuning based on layer diffusion. Our current approach leverages pre-trained latent diffusion models (LDMs) for robust performance (L7-9). However, we recognize the potential benefits of layer diffusion fine-tuning. We are exploring various fine-tuning strategies to improve our model's robustness and accuracy and plan to incorporate these techniques in future iterations. Thank you for your valuable feedback.

W1: Contribution to Image Matting

Our primary contributions and insights are not focused on the neural network architecture or block design. Instead, we concentrate on exploring how to leverage the priors learned by well-scaled image generation models to address the ill-posed problem of image matting. Here are the unique aspects of our work:

1. Joint Estimation of Foreground and Background: In previous works, foreground extraction required complex post-processing. Our experiments demonstrate that leveraging the priors from generative models facilitates foreground image prediction. Furthermore, the alpha map and foreground have a close domain relationship, and joint estimation benefits both.
2. Addressing Domain Gaps with Generative Models: We tackle a significant technical challenge by adapting image generation models for alpha and foreground estimation. This involves bridging the severe domain gap between the RGB images encoded and decoded by VAEs and the RGB-A images required for matting.

We believe that our approach of integrating generative model priors into the matting process brings a novel perspective and advances the field of image matting. Thanks for your valuable feedback.

W2: Inadequate Discussion of Model Limitations

Thank you for your feedback. We acknowledge that the discussion of limitations is crucial. In the original paper, we have addressed the limitations of our model in the "Conclusion and Future Work" section (L322-331). We have outlined several aspects where the model might face challenges or limitations.

If there are specific areas or additional limitations you believe were not adequately covered, we would greatly appreciate your insights. We are open to discussing and incorporating any further considerations you might suggest during the discussion phase and are willing to make necessary revisions to address these points comprehensively.

W1: Training Dataset Size and Model Generalization

Thank you for raising concerns about our dataset size and model generalization. Our training dataset is substantial, comprising 431 foreground images and a background library of 82,783 images. Each foreground image is paired with 100 different backgrounds (L206-L209), resulting in at least 43,100 training samples. Additionally, we applied data augmentation techniques such as random horizontal flipping, cropping, and photometric distortion, further increasing the effective training dataset size (L227-L229). Thus, our dataset is quite extensive.

For generalization, our model trained on the Composition-1k dataset was evaluated on the independent AIM-500 real-world dataset. The results showed that our model generalizes well to unseen data (L283-284).

We fully acknowledge the importance of dataset size for generalization. However, matting is a dense annotation task that is extremely challenging to label. This is precisely why we incorporated the strong prior knowledge from the latent diffusion model trained on LIAON, which significantly aids in improving generalization capabilities.

W2: Clarification on Prior Knowledge and Fine-Tuning with LoRA

When we refer to 'prior' in our paper, it indeed pertains to the pre-trained Latent Diffusion Model (L147-148). Regarding your question about using LoRA to fine-tune the model on a smaller dataset, we have experimented with this approach. Our findings indicate that fine-tuning with LoRA resulted in poor performance and difficulty in convergence. Consequently, there was no issue of overfitting or performance degradation over extended training periods. We hope this clarification addresses your concerns.

W3: Evidence for Cross-Domain Self-Attention Advantages

In our paper, we conducted an ablation study (L288-305) to evaluate this component. The quantitative results demonstrate that cross-domain self-attention improves both alpha prediction and foreground color prediction accuracy. This improvement is attributed to the high correlation between alpha and foreground color, as both describe information about the foreground object. Cross-domain attention enhances the interaction and consistency of this information, leading to better overall performance.

W4: Switcher

The switcher is indeed an embedding designed to help the network distinguish between the two outputs: foreground color and alpha value. Both outputs are predicted simultaneously. Our ablation study (L306-309) demonstrates that providing this additional embedding information, rather than merely differentiating between foreground and alpha by channel order, enhances the neural network’s ability to distinguish and effectively utilize information from the two modalities.

Q1: Transparent Latent Decoder

We appreciate the suggestion to fine-tune the VAE. However, solely fine-tuning the VAE is not ideal for several reasons:

1. Consistency with Pre-trained Weights: To leverage the pre-trained weights in the LDM, especially in the U-Net, it is crucial to maintain the consistency of the latent distribution output by the VAE encoder. Over-finetuning the VAE encoder could disrupt this consistency, negatively impacting performance.
2. Compression: The VAE inherently introduces a non-negligible reconstruction loss due to its compression nature, which is problematic for high-resolution tasks like matting that require low-level detail. This limitation, as mentioned in the paper (L189-190, L310-314), necessitates the use of a transparent latent decoder to meet the high precision demands of matting tasks.

Our current approach, combined with the transparent latent decoder, offers a balanced solution that addresses the high-resolution requirements of matting tasks while leveraging the strengths of pre-trained LDM components.

Q2: Steps of Sampling

In the appendix of our paper, we have included results and analyses regarding the impact of denoising steps on performance (L507-510). The specific values are shown in the table below.

Our findings indicate that increasing the number of timesteps generally enhances performance, although the improvement diminishes as the number of timesteps increases. Additionally, our experiments show that setting the sampling steps to at least 5 generally yields satisfactory results.

Q3: Scaling Up Training Data

Thank you for your comments and for inquiring about potential methods to scale up the training data for more robust model training. We have plans for future research that involve using methods such as layerdiffusion to generate additional data. This generated data will then be screened with a human-in-the-loop approach to ensure quality. In our current study, to maintain a fair comparison with other methods, we used the same training dataset.

Q4: Necessity of Trimap

Thank you for your comment regarding the use of trimap in our model. We understand the desire for a model that can directly perform RGB-A extraction without requiring a trimap. However, we have found that the use of a trimap is still necessary for achieving accurate results, especially when multiple foreground objects are present in an image.

To address this concern, we conducted a qualitative experiment where the trimap was set to an unknown region across the entire image. This situation introduces ambiguity about which foreground object to extract, as illustrated in Figure D of the rebuttal PDF. For example, with a trimap indicating unknown regions, the model faces difficulty in deciding whether to extract the rabbit or the flower, leading to potential confusion and reduced accuracy.

Additionally, to ensure fair comparison with previous methods, we have maintained consistent settings and trimaps across our experiments(L251-257).

Dear Reviewer YFFw,

We sincerely appreciate your time and effort in reviewing our submission and providing valuable comments. We have provided a detailed response to each concern you raised and hope they have adequately addressed your concerns. As the author-reviewer discussion phase is coming to an end (Aug 13 11:59pm AoE), we would like to confirm whether our response has effectively addressed your questions. If you have any other questions or concerns, please do not hesitate to contact us.

Best regards,

Authors

Dear reviewers, a gentle reminder of the participation of the reviewer-author discussion. Thank you!