Where's Waldo: Diffusion Features For Personalized Segmentation and Retrieval

Thank you for finding our approach novel and the ides of our method interesting. We address your comments below.

Q1: The logical flow before the method section needs to be adjusted. The authors should rearrange the related works and the motivation of their work. The authors should list their contributions.
A1: We appreciate the reviewer's valuable feedback and their contribution to improving our paper. We will make sure to rearrange the related works and the motivation of our work to improve the logical flow leading up to the method section. We will make a clear distinction between supervised and self/weakly supervised foundation models in the text and tables. We will also explicitly list our contributions to highlight the key aspects and innovations of our work.

Q2: It is hard to understand what the authors want to do and the problem they want to address.
A2: This paper addresses Personalized Retrieval and Personalized Segmentation, where a user provides a reference image and either a mask or the object's class name (lines 1-2, 101-102 in the paper). Personalized retrieval aims to find database images with the exact object from the reference image, while personalized segmentation seeks to segment the specified object in new images and videos. We demonstrate that features from pre-trained diffusion models achieve high zero-shot performance on these tasks without training or fine-tuning (please refer to Section 3.1 for details). Lastly, we further point to major drawbacks with current personalized retrieval and segmentation benchmarks and offer two new alternative benchmarks. We show that our approach (PDM) suppresses all supervised and self-supervised methods on all benchmarks. We will make sure that these issues are better emphasized in the revised version.

Q3: What is the “short description” mentioned in the paper?
A3: The “short description” is the “class name” (as we use in lines 1-2, 101-102, 124, 157). We will make sure to clarify this in the final version.

Q4: Visualized results do not present the situation where a desired instance appears together with other instances of the same class.
A4: Due to space constraints, we cropped the images in the figures to focus on the relevant objects among other similar objects. We will include more examples without cropping and enlarge the images in the final version.

Q5: Why using diffusion features and not other models?
A5: The fundamental difference between diffusion models and the other models listed by the reviewer is that they are generative, in the sense that they focus on p(image|text). The diffusion training objective (i.e., coarse-to-fine reconstruction loss) requires the model to generate informative features for every object. This is different from other, non-generative models, some of which use image-level contrastive learning objectives (e.g. CLIP), and others use image-level discriminative objectives (Resnet, ViT). Such image-level discriminative or contrastive objectives often remove information about specific instances and objects [4]. Furthermore, current diffusion models contain a self-attention component which controls how some parts of the image depend on other parts. This allows the model to generate coherent objects and makes the model contain an explicit representation of object instances. For additional details, please refer to recent papers on diffusion features [1,2,3]. We will clarify and add this discussion to the final version.

Q6: Figure 3 is hard to understand. Why "dog" is not fed into the target branch?
A6: We thank the reviewer for their feedback. We will make sure to simplify the figure in the final version. This figure illustrates the flow of our algorithm and aligns with the algorithm described in Section 3. Colors are used to differentiate between "Appearance" (green) and "Semantic" (yellow) features. In this figure, a reference image of a dog labeled "dog" and a target image are provided. Both images and the class name undergo feature extraction (see L155-156). The reference and target branches follow the same process, making it irrelevant which branch receives the class name.

Q7: What information is used to calculate the cross attention with the target image?
A7: Semantic similarity is calculated as the cross attention between class name token $C$ and the semantic feature map $F^{S}_{t}$ as detailed in lines 170-172 and Eq. 6.

[1] Tang et al. (2023) “Emergent Correspondence from Image Diffusion”.
[2] Lue et al. (2023) “Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence”.
[3] Zhang et al. (2023) “A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence”.

Thank you for your feedback. Regarding the robustness of diffusion features, our experiments show they are effective across a wide range of concepts, outperforming both supervised and self-supervised methods on various benchmarks. What tools would the reviewer suggest to further validate the robustness? We believe that as better and faster diffusion models are released, PDM can be applied to achieve more robust features and thus better performance. Introducing PDM will indeed allow further research in this direction. As for the inference speed, it is indeed a limitation as noted in our paper. We currently achieve feature extraction in half a second, making it applicable to a variety of tasks. As inversion methods improve, our approach will also speed up.

Thank you for finding our approach insightful, technically sound with great transferability and our paper to be well motivated and easy to follow. We kindly address your comments below.

Q1: Cite relevant work.
A1: Thank you for your feedback. We will make sure to cite these papers in the final version.

Q2: Can the method be extended to DiT and stable video diffusion?
A2: We thank the reviewer for their suggestion. We are optimistic that PDM can be applied to DiT and video diffusion models. First, recent studies [1] identified structural and appearance features in vision transformer-based models. Second, it was not hard to find instance-features in several Unet models, SDXL and SDv2.1. Thus, we assume that other diffusion models will also exhibit comparable instance features. We appreciate the reviewer's insightful question, and we leave this for future work.

[1] Tumanyan et al. (2022) “Splicing ViT Features for Semantic Appearance Transfer”.

Thank you for your feedback. We are encouraged that you find our approach innovative, the challenge novel and our paper to be excellently written. We address your comments below.

Q1: Have you considered more complex ways of combining appearance and semantic features?
A1: We thank the reviewer for their suggestion. In the paper, we chose feature averaging to avoid training or hyperparameter tuning on labeled data. Following the reviewer's suggestion, we tested a weighted combination of semantic and appearance features: $wF_{appearance} + (1-w)F_{semantic}$. We split the PerMIR and ROxford-Hard [1] datasets into training (20%) and test sets (80%), and optimized $w$ on the training sets. The table below shows that when a training set is available, weighted fusion improves results. We also plan to explore learnable fusion methods with advanced techniques in future work. We will include this analysis in the final version.

Q2: In the proposed multi-instance benchmarks, selected frames may lack feature diversity or suffer from quality issues. How do you ensure fairness and consistency across methods despite randomness?
A2: We appreciate the reviewer’s comment. Random frame selection was done once during dataset preparation to ensure fair comparisons among all methods. We manually inspected the frames for quality and diversity, finding them acceptable and adequate given the BURST [2] dataset’s quality and video length. In response to this comment, we further quantified frames quality and diversity. Using the CLIP model, we found an average cosine similarity of 0.17 between frames, indicating low similarity (compared to 0.31 for adjacent frames) and thus high diversity. For quality, the mean SSIM between dataset frames and a random ImageNet subset was 13.2 (compared to 11.8 for ImageNet samples, higher values indicate better quality). We will clarify and include this analysis in the final version.

Q3: Missing ablation study for Personalized Segmentation.
A3: Following this comment, we now added the following ablation study for personalized segmentation:
(1) Object Mask Instead of Class Name: In this scenario, we considered the case where the class name is not provided, but an object mask is available. We tested this configuration on PerMIS, resulting in a mIOU of 45.0% compared to the original 42.3% when using the class name. The bIOU was 89.2% compared to the original 86.8% when using the class name. This shows that using an object mask leads to improved segmentation performance, indicating its potential as a valuable alternative when class names are not available.

(2) Appearance vs. Semantic Maps: We examined the individual contributions of the Appearance and Semantic maps to the final similarity map. For this experiment, we used each map independently as the final similarity map, ignoring the other. When using only the Appearance Map, we achieved a mIOU of 30.2%, compared to 24.9% when using only the Semantic Map. Both results are significantly lower than our original mIOU of 42.3% when using both maps and averaging them. These findings underscore the necessity of integrating both maps to achieve optimal performance in the final similarity map, and eventually in personalized matching.
We thank the reviewer for this feedback. We will make sure to add this ablation study to the final version.

Q4: How did you manage to enhance image reconstruction quality?
A4: We simply build on recent advances in inversion with diffusion models [3, 4]. For SDXL-turbo, using [3] for inversion results in a PSNR of 24.1, close to the upper bound of 26.3 set by the diffusion VAE. This high PSNR indicates good reconstruction quality and hence high-quality features.

Q5: Could the authors suggest ways to address the limitations in future work?
A5: Image inversion is the primary limitation of our method, and thus it falls into a different line of work. We did not focus on developing new inversion techniques within the scope of this study. Note that our method is agnostic to the inversion approach, allowing us to adopt faster and more accurate methods developed by the community in the future.

[1] Radenovic et al. (2018) “Revisiting oxford and paris: Large-scale image retrieval benchmarking”.
[2] Athar et al. (2023) “Burst: A benchmark for unifying object recognition, segmentation and tracking in video”.
[3] Pan et al. (2023) “Effective real image editing with accelerated iterative diffusion inversion”.
[4] Garibi et al. (2024) “ReNoise: Real Image Inversion Through Iterative Noising”.

Thank you for your detailed and insightful feedback. We believe we address all your concerns below.

Q1: It is not fair to compare PDM to Dinov2 or SAM since those models were not trained with text supervision.
A1: We thank the reviewer for the comment. The paper compared two types of baselines: (1) Baselines trained with text supervision (CLIP and OpenCLIP; See Table 2, Figure 5b and Figure S2) and (2) baselines without supervision, to be consistent with the literature [1,2,3]. Following this comment, we add comparisons with more methods trained with text supervision. Specifically, we evaluated personalized retrieval with features from BLIP2 [4], GLIP [5], and SLIP [6], on the PerMIR dataset. The table below shows that PDM outperforms all text-image models by a large margin. We will add this experiment and additional details in our revised manuscript/suppl.

Q2: How does PDM performance and quality vary with the underlying diffusion model?
A2: Thank you for the insightful question. Indeed, it is not clear a-priori that similar features with comparable quality can be found in other diffusion models. To answer this question, we repeated the feature finding process in two diffusion models: SDXL, and SDv2.1. We then ran new experiments for the Personalized Image Segmentation task using two datasets, PerSeg [7] and PerMIS. The table below reports segmentation performance using three diffusion models: (1) SDXL-turbo (results taken from the paper), (2) SDXL, and (3) SDv2.1. We also report the run time for feature extraction per image and the mean PSNR of images to measure inversion-reconstruction quality.

The results show that one can find PDM features for other diffusion models (SDv2.1 and SDXL) that produce better results than SDXL-turbo in terms of PSNR, mIoU and bIoU. Their inversion reconstruction time is 10x slower because they require more inversion steps. We will report and discuss these results in the final version.

Q3: The method seems only applicable to Unet diffusion models.
A3: The paper focused on UNet-based diffusion models because they are currently the most widely-used text-to-image models. We are optimistic that similar features can be found in other diffusion models, for the following reasons. First, recent studies [8] identified structural and appearance features in vision transformer-based models. Second, it was not hard to find instance-features in several Unet diffusion models. Thus, we assume that other diffusion models (such as DiTs) will also exhibit comparable or better instance features. We appreciate the reviewer's insightful question, as it highlights a promising avenue for future research.

Q4: What happens if you compute OpenCLIP+DINOv2+PDM?
A4: Following this question, we tested a combination of DINOv2 and PDM. Naturally, there are many ways to combine several features, and here we followed the protocol from [9,10]. We first used OpenCLIP to select the 400 most similar images to each query image. Then, we selected 100 images with highest DINOv2 scores, and then ranked them using PDM. We tested the combined method for personalized retrieval in two datasets: PerMIR and ROxford-Hard. In PerMIR, this method achieved a mAP of 70.2%, compared to 69.9% for OpenCLIP + PDM and 70.8% for Dinov2 + PDM. In ROxford-Hard, it achieved a mAP of 58.3%, compared to 57.7% for OpenCLIP + PDM and 62.1% for Dinov2 + PDM. This approach shows a slight improvement over OpenCLIP + PDM but underperforms compared to Dinov2 + PDM, likely due to OpenCLIP's less effective retrieval capabilities.

[1] Tang et al. (2023) “Emergent Correspondence from Image Diffusion”.
[2] Lue et al. (2023) “Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence”.
[3] Zhang et al. (2023) “A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence”.
[4] Li et al. (2023) “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models”.
[5] Li et al. (2022) “Grounded Language-Image Pre-training”.
[6] Mu et al. (2022) “SLIP: Self-supervision meets Language-Image Pre-training”.
[7] Zhang et al. (2023) “Personalize Segment Anything Model with One Shot”.
[8] Tumanyan et al. (2022) “Splicing ViT Features for Semantic Appearance Transfer”.
[9] Shao et al. (2023) “Global Features are All You Need for Image Retrieval and Reranking”
[10] Zhu et al. (2023) “R2Former: Unified Retrieval and Reranking Transformer for Place Recognition”