Personalized Representation from Personalized Generation

Thank you for your insightful, helpful feedback. We have addressed your comments below; please let us know if you have any further questions.

Distillation v. Personalized finetuning

Thank you for raising this point – it is important to isolate the source of performance gains (to clarify, we do not introduce a new loss function, and rely on the existing infoNCE function).

To separate the effects of the contrastive loss from the synthetic data, we add cross-entropy, a simple non-contrastive loss, to Table 6. We initialize a linear layer with a sigmoid function that projects a backbone's output (CLS token concatenated with avg-pooled patch tokens) to a binary prediction (1 indicates the presence of the target object, 0 otherwise). We then supervise with cross-entropy and fine-tune the backbone with LoRA while fully updating the linear layer. On the DF2 validation set, cross-entropy leads to worse results than all contrastive losses. This indicates that a contrastive loss is important for performance gains.

As you note, however, the generative prior in Stable Diffusion is important for generating positives in novel poses and backgrounds. For instance, in Sec 5.3 we show that pose diversity in generations results in downstream robustness to pose variation. This could indeed be considered a form of distillation, and is an advantage of using a T2I generator.

Contrastive learning setup/objectives

Our objective is to improve a representation's performance for personalized tasks. We chose a contrastive framework for the following reasons:

• It does not require extra labels. We formulate positive pairs as $(X, g(X))$, where $g$ is a generator, and obtain negatives from the random distribution of images. We then train to enforce representational invariance to the changes (pose, background, etc) introduced by $g$.
• It does not require us to learn additional decoders or heads, which is useful in our low-data regime.
• Contrastive losses are robust to noise [1]. DreamBooth often fails to preserve fine-grained details, so it is useful to choose a loss that can still extract signal even if the positive pair does not exactly match

As detailed in the previous section, following your suggestion we compare to straightforward cross-entropy supervision (non-contrastive). Our results suggest that contrastive losses are indeed important for our objectives.

Typos/formatting

Thank you for pointing out these typos; we have corrected them in our revision.

Ethics review

Below we give details on ethics/privacy guidelines for the datasets. Note that for DF2 and Dogs, our reformulation solely entails resplitting the existing published datasets and annotating with masks. We also provide the licenses that both DF2 and Dogs were released under in Sec. A.1.

• PODS: The majority of PODS objects belong to the authors. For all additional objects, we obtained explicit consent from the owners. We chose objects with no personal-identifying attributes (such as the owner's name). We also only include images of objects, with no human subjects.
• DF2: The DeepFashion2 images are taken from the original DeepFashion dataset [2], and online shopping websites [3]. These two benchmarks are widely used/cited in vision literature. To access the dataset we signed an agreement that it only be used for non-commercial research purposes.
• Dogs: The DogFaceNet are images captured by the authors, and downloaded from publicly-available dog photos on adoption websites [4]. The authors do not, to our knowledge, provide further details regarding privacy/consent measures.

[1] Yihao Xue et al. "Investigating why contrastive learning benefits robustness against label noise". ICML, 2022.

[2] Yuying Ge et al. "DeepFashion2: A Versatile Benchmark for Detection, Pose Estimation, Segmentation and Re-Identification of Clothing Images." CVPR, 2019.

[3] Ziwei Liu et al. "DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations." CVPR, 2016.

[4] Mougeot, G. et al. "A Deep Learning Approach for Dog Face Verification and Recognition." PRICAI 2019: Trends in Artificial Intelligence, 2019.

Thank you for your detailed feedback. We have addressed your comments below; please let us know if you have any further questions.

High computational cost & cheaper alternatives

You raise a good point on cost -- in Sec. 1 of our Global Response we discuss new evaluations of cheap alternatives to diffusion models.

We now cover:

• A limited set of real images
• A larger set of real images
• Synthetic positives without T2I models
• Synthetic positives with T2I models
• Incorporating internet-available images

We investigated using GANs, but did not find suitable customized generation methods. However, we hope our new comparisons (see Global Response) provide examples of achieving good performance with cheap alternatives to T2I models.

In Sec. 2-3 of our Global Response we discuss method runtimes and tradeoffs between using T2I models and cheaper approaches.

Baseline comparisons

Thank you for your suggestions. We address this above and in Sec. 1 of our Global Response; please let us know if further follow-up is needed.

Limited task scope / real-world validation

We completely agree that testing robustness in diverse/complex real-world scenarios is critical. We highlight some ways that our datasets test this:

• Subjects: They cover a wide range of subjects: fashion (DF2), animals (Dogs), and household items (mugs, screwdrivers, bags, bottles, shoes).
• Robustness/generalization:
  • DF2 is designed to include distribution shift. Training images are store/gallery images, and test images are in-the-wild consumer photos with a variety of settings, poses, distractors, and quality/occlusion.
  • PODS: In Sec. A.2.2 we explain the distribution shifts in PODS. Training images are taken in controlled, simple settings featuring the object one pose/location. Test sets evaluate combinations of variations in pose, background, # objects. We also vary camera angle and occlusion/visibility.
• Complex/real-world scenarios:
  • DF2/Dogs: Images from DF2 and Dogs are collected in the wild, in various environments. These naturally collected images reflect complex, uncurated settings, with no specific control over backgrounds, lighting, or other factors.
  • PODS: The “Both” test subset includes complex/real-world scenarios (see end of A.2.2). For example, mugs are captured on full dish racks, and screwdrivers captured in toolboxes.

We have added examples from DF2/Dogs (Fig. 10-11) highlighting these aspects.

Biases/limitations in synthetic data

You are correct that synthetic data introduces bias/limitations. For example, Stable Diffusion struggles with certain objects, such as screwdrivers, leading to worse performance on those categories. It also exhibits a well-known bias towards single-object images, potentially causing lower performance on distractor variations in Fig. 9. In Section 5.3, we analyze limitations introduced by DreamBooth and Cut/Paste (which uses real images of the object).

Diversity analysis

Thank you for your feedback – following your suggestion, we added a quantitative analysis of fidelity / diversity across synthetic datasets (Fig. 12), where we plot fidelity v. diversity, colored by performance, for various DreamBooth datasets and Cut/Paste. Non-filtered DreamBooth datasets lie at the extremes (high-diversity, low-fidelity or vice versa); the filtered dataset achieves a better balance of the two, and thus higher performance.

How do synthetic-data representations compare to real-data representations in terms of generalization?

We updated Fig. 9 to explicitly compare synthetic-learned representations (Masked DreamBooth, Combined DreamBooth + Cut/Paste, Cut/Paste) to real-data-learned representations (Real-Image) on the PODS distribution shifts. Synthetic-augmented datasets generalize significantly better across OOD test sets.

Ethical/Privacy concerns

Thank you for raising this important concern. We acknowledge the potential privacy implications of instance recognition tasks, particularly in sensitive areas (i.e., surveillance). To address this, we deliberately focused on non-human ID tasks and did not collect any human data in the PODS dataset. We recommend that practitioners prioritize ethical deployment by using anonymized data and adhering to standard privacy-preserving guidelines. We also believe that there are significant positive applications when deployed responsibly (see Global Response Section 2).

Thank you for your helpful feedback and comments. We have carefully addressed your comments in detail below. Please let us know if you have any further questions.

Novelty of the approach

Thank you for your feedback. We acknowledge that the individual concepts of synthetic data generation and contrastive learning are not novel. However, our work makes several unique contributions outside of the methodology components that we believe are valuable.

• We empirically show that personalized representations (using synthetic data to handle lack of real data) significantly outperform pretrained representations, and that they are adaptable across different instance-level tasks.
• We introduce new evaluation mechanisms for personalized representations in vision and present the PODS dataset, which we expect will be a useful benchmark for personalized representation, instance-level detection, and personalized generation research.
• We highlight tradeoffs (computational cost v. downstream performance) of different approaches to personalized data generation, and the unique biases/limitations of different methods.

These contributions offer new perspectives on personalized learning and synthetic augmentation in vision, with practical implications for future research. We revised our introduction to better clarify our contributions.

Computational cost

Thank you for your insightful feedback on computational cost. We understand this is an important consideration and we address this question in Sec. 2-3 of the Global Response. We also add updates to Section 5.2 of the paper to study the tradeoffs between efficiency and performance. We compare methods involving image generation to those using solely real images, which is explained in Sec. 1 of the Global Response.

Real-world scenarios where benefits outweigh training costs

You make a great point; it is important to consider real world cost-benefit tradeoffs. We discuss such scenarios in Section 2 of our Global Response.

Challenging cases

Thank you for your suggestion -- we add examples of challenging cases in Sec. D.2 of our revision (Figs. 16-17).

Given a target image, we identify hard negatives as test negatives with a high DINOv2 cosine similarity to the target image, and hard positives as test positives with a low cosine similarity. As you note, hard negatives are typically different dogs of the same breed; hard positives are often the same dog in very different contexts or poses.

• Success cases: For hard positives DINOv2-P has higher cosine similarities than DINOv2, even under significant lighting, pose, and context shifts; this highlights its capabilities.
• Failure cases: DINOv2-P also sometimes has higher cosine similarity for hard negatives – this could be because of noisy positives in the synthetic training dataset, causing the personalized representation to associate the target object with subtly incorrect features.

Decline in performance for certain tasks

In the Global Response Sec. 4 we explain some updates we have made to our main results. Personalization with DreamBooth now outperforms non-personalization in 33/36 cases in Table 1.

However, you are right that in a few cases we see a decline in performance. We highlight some potential reasons:

• State-of-the-art generators do not always achieve necessary levels of diversity/fidelity. For instance, with vanilla DreamBooth (Table 1), performance declines on classification for Dogs, but after incorporating masked training and filtering (Masked DreamBooth), performance increases (Table 2). We plot the diversity/fidelity of these datasets v. performance in Fig. 12 of our revision.
• We fine-tune each backbone on the same data, however in practice different backbones may have unique strengths/weaknesses. For instance, performance on PODS/DF2 retrieval increases for DINO/CLIP, but declines for MAE.

Overall, this shows there is room to improve for this novel problem formulation/setting beyond our proposed method.

Thank you for your detailed review and valuable suggestions. We address your feedback below; please let us know if there are further comments or concerns.

Additional baselines/comparisons

Thank you for your insightful comments regarding additional baselines and comparisons. Please refer to Sec. 1 of our Global Response, where we present results on the following additional comparisons:

• Training with real negatives (rather than generated negatives) from the internet, i.e. from publically available vision datasets. This enables real-data-only baselines.
• Training with an extended pool of real images, as an upper bound to performance, as well as Cut/Paste and Masked DreamBooth when trained/sampled from this extended pool.

Cut/Paste vs Masked DreamBooth trade-off: When scaling up the real dataset to n=20, Cut/Paste begins to outperform DreamBooth (performance of 74% vs 70%, both outperform real-image only finetuning). However, sampling from the combined pool still performs best (76%), which suggests that T2I models contribute important additional information even with more real data. In domains where accuracy is key (see Sec. 2 of global response), this method may be worth the extra compute.

Computational cost and runtime

We agree that the computational cost is an important consideration. Please refer to Sec. 2 of our Global Response, where we address computational cost in detail. We have added the runtimes of our methods and comparisons in Appendix Table 3, with brief discussion in Sec. 5.2 of the paper.

Multi-object personalization

Thank you for your suggestion on multi-object personalization - it reflects a real-world use case of personalization, and an interesting extension to our work. We experimented with personalizing to multiple (2-5) objects on the PODS dataset by jointly training on triplets for multiple objects. With our current training pipeline, we can achieve strong improvements over pretrained models on global tasks (on par with single-object-personalized models). However, dense-task performance significantly decreases. Qualitatively, we found artifacts/noise in the patch-level similarity maps, causing distortions in the threshold detections/segmentations. Naiively training on triplets from multiple objects seems to negatively affect dense features, and require more careful exploration of the composition of triplets.

Filtering synthetic data

For our results in Table 1 (i.e. personalization without access to additional data/annotations) we do not filter our synthetic datasets, however we do filter the Masked DreamBooth dataset (i.e., when segmentation masks are available).

You are correct that many works, such as [1], apply a filtering step. In our preliminary experiments we found that filtering without segmentation masks of the target object did not meaningfully improve performance. This is likely because we have a very small reference train set (3 images) so filtering based on global embeddings is affected by spurious signals, such as keeping images with the same backgrounds as training images. This negatively affects diversity, outweighting improvements in fidelity.

When segmentation masks are available, we are able to decouple the object and background in our training images. We explain our filtering procedure in Sec. A.3.2.

In Fig. 12 of our revision (Appendix Sec. C.4), we plot the relationship between fidelity, diversity, and resultant performance for DreamBooth datasets (with/without filtering and LLM captions) and Cut/Paste. While non-filtered DreamBooth datasets lie at the extremes (high-diversity, low-fidelity or vice versa), the filtered dataset achieves a better balance of the two, and thus higher performance.

Performance on real-image personalization

As you have noted, training on real-images only sometimes causes performance to decline. We suspect that this is due to the limited size of the real dataset (only 3 images). Even when applying classic augmentation, the dataset lacks diversity, and as shown in Fig. 12 of our revision, maintaining both fidelity and diversity are key to significant performance gains. We note that in our updated results, with a small training set, training on real-images improves on the pre-trained backbone in 8/12 cases.

[1] He et al. "Is synthetic data from generative models ready for image recognition?." ICLR 2023.

[2] Zhang et al. “Personalize segment anything model with one shot”. arXiv, 2023.

[3] Fu et al. “DreamSim: Learning new dimensions of human visual similarity using synthetic data”. NeurIPS, 2023.

Thank you for your constructive and insightful feedback. We address your comments in detail below. Please let us know if you have any further questions.

Inclusion of real negatives

You make an important point about including real negatives. We initially excluded real negatives because we envisioned a setting in which a user can obtain personalized representations without any additional real data or upfront effort, besides taking a few positive images.

We agree that in practice, real negatives may be available via open-source datasets. In our Global Response, Sec. 1, we explain updates to Sec 5.2 of the paper, including results on allowing access to additional data. We use real negatives from open-source datasets (ImageNet, LAION-RVS-Fashion [1]). Please refer to the Global Response for details/results, and let us know if any follow-up is needed.

Revision of Figure 1

Thank you for your constructive feedback. We have updated the figure following your suggestions and added a reference to it in the introduction.

Real-image baseline

We appreciate your point about the “real-aug” experiment. To clarify: in this experiment, we solely use the three real training images (i.e, $\mathcal{D}_R$) as positives. As suggested, we have revised the name to “real-image baseline”.

We always automatically apply on-the-fly classic augmentations (random horizontal flips, random rotations, and random resized crops) during training. These are applied to all positive/negative images, and can be considered a standard part of the contrastive training pipeline. We have updated Sec. 4.2 of our paper to clarify this. We do not report results on real-image or synthetic training without augmentations, as these are standard practice for vision pipelines.

As you have noted, real-image training sometimes causes performance to decline. We suspect that this is due to the limited size of the real dataset (only 3 images). Even when applying classic augmentation, the dataset lacks diversity, and as shown in Fig. 12 of our revision, maintaining both fidelity and diversity are key to significant performance gains. We note that in our updated results, with a small training set, training on real-images improves on the pre-trained backbone in 8/12 cases.

Access to segmentation masks

Thank you for pointing out the ambiguity. The distinction between our results in Sec. 5.1/Table 1 and Sec. 5.2/Table 2 is that Table 2 assumes access to additional resources including segmentation masks for training, while Table 1 only assumes access to the initial three real images. We have added clarifications to Table 1 and Table 2.

We note, though, that for downstream evaluation of segmentation and detection, we always use segmentation masks, as we consider evaluating/probing the learned representations to be separate from actually training the representations.

Dense task performance

To probe how well representations localize, we extract dense predictions directly from the patch embeddings using thresholding. This provides an unmodified assessment without introducing external factors such as pre-trained segmentation modules or additional post-processing.

However, for weaker backbones like MAE and CLIP, while the object of interest is often identified as a high-confidence region, the confidence values are usually not distinct enough from the background, leading to very poor performance with simple thresholding, and thus marginal dense performance changes in Table 1. MAE features often require supervised finetuning to transfer to dense prediction tasks [2]. We point to DINOv2, which achieves good dense performance out-of-the-box, as a better assessment of how personalization affects dense performance. Future work could explore using personalized features to train segmentation heads (as in [3]); this would require generating segmentation masks for synthetic images.

We also evaluate with state-of-the-art segmentation pipeline PerSAM in Table 9, which better reflects the practical use of these representations, and has more consistent performance gains.

[1] Lepage, Simon et al. "LRVS-Fashion: Extending Visual Search with Referring Instructions." arXiv (2023).

[2] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." CVPR, 2022.

[3] Oquab, Maxime, et al. "Dinov2: Learning robust visual features without supervision." TMLR, (2023).

References

[1] Morrison, et al. “Understanding Personalized Accessibility through Teachable AI: Designing and Evaluating Find My Things for People who are Blind or Low Vision.” SIGACCESS Conference on Computers and Accessibility, 2023.